I understand what you mean, sometimes it just happenens..that's how the mind works. Pray to God continuously any time it happens; God knows it's not your fault but continue to pray. You could tell your pastor if you want, I am unsure how it works in your church/holy place. Hang in there!
God Bless.