I have been a member of The Church of Jesus Christ of Latter-day Saints since I was baptized at 8. My wife and I have been married nearly 30 years, and have had numerous visits by different bishops and other leaders (we don't call them pastors or ministers). Never once during that time, has any of them visited with my wife while she was alone with them in the house, let alone had any type of sex with any one of them.
With the LDS faith, a married man and woman can have sex. Sex outside of marriage, which would include extramarital sex or sex before marriage, goes against such beliefs. LDS leaders are generally married themselves.
Until this question, I'd never heard of such a possibility. Abstinence before marriage is taught, and complete fidelity while married is also taught. That would go for everyone, regardless of the position they hold.
Now, members are human beings, too, and I have heard of situations that aren't in accordance with gospel teachings, not just regarding sexual relationships, but any other of God's laws or commandments, but in every instance, the actions are not condoned, or brought into general practice.
There are groups which call themselves offshoots of the main church. I have no idea what they practice, so I can't comment on them. If a bishop or other leader were having sex with married women of their congregation, the moment it was discovered, and then proven, the bishop would be removed from his position, most likely excommunicated, and potentially so would the wives, if they were doing so willingly and knowingly (as in, they knew it was wrong to do but they did it anyway, without blackmail or some other form of coercion).
There are many rumors out there about the Church, its members, what they do or don't do, and while some of it is bound to be true, or even partly true, a lot of it is not, or taken out of context.
Man is fallible, makes mistakes, and members of the Church are no different. That's why the gospel of Jesus Christ and His atonement exist. To teach us how we should live, and then provide a way back when we fall. However, wantonly going against God's laws, which this would be in any case, brings consequences, and would not be even remotely okay for any leader and woman not his wife to do.
NOTE: This answer may show up twice. I'm having a hard time getting it to post, but for some reason it's already being upvoted.