Depending on the quality of the life you are living and your lifestyle choice, the number of hardest facts or parts of the day varies. For a person who is on diet resisting the food that he loves would be the hardest. For an alcoholic coming back home and resisting the first drink is the hardest. For a heartbroken person the hardest part is not to feel sad when thinking about the special someone. Same goes million of other life related things too.
But, I think the hardest part of the day for anyone is to not offend/ hurt the person he is interacting with.
So Let me skip the obvious penis joke that I was dying to make. (I am trying not to offend anyone here, But I think I failed)
In our everyday life we interact with peoples. And that involves a lot of word exchanges. And being humans we tends to add lot of unnecessary information to the interaction, and most of us thinks that they are easy but they are not. Imagine that you made choice that you are in company of friends who are lets say not nice. But you developed a liking to them and to stay in that niche you say things that you don't mean to. (That's not much of a help) Lets just say you called someone fat, mean and some horrible things during one of the conversation, you didn't mean too but you said it anyways. Let's say you were trying to be funny. Most of the time we don't feel bad because we are having fun at the moment but you subconscious mind is altering your personality at the very same time. This still may not affect you, But when the person who was the target confront or says something back, that stirs an inner conflict. Which is never good. (That is not much either)
Let me try to put in more simpler words, It is the 'I shouldn't have said that' feeling. Admit it you get it at least once a day. Where you use the word intentionally or otherwise but ultimately resulted in you feeling sorry about. And even more troublesome fact is you don't know how the person would react to your words, Many a times the words can be misunderstood and can cause you pain. There are cases in my life where I had to explain the other person that I was actually complimenting them on their achievement.
I am no expert in this matter, but the answer to the question 'Am I looking fat in this dress' is always No. I guess what I am trying to say is 'Chose your words wisely', The pain that I mentioned earlier doesn't necessarily has to be mental, You could get stabbed/shot or worse getting kicked into a bottomless pit. :-D