|Are we talking about your sexuality or company policy right now? Either way, I hate it.|
The use of this foul phrase communicates, whether it is your intention or not, that what you're comfortable with is more important that right or wrong, correct or incorrect, truth or error. Even more insultingly, it says that what you're comfortable with is more important that what I'm comfortable with, which is why it's just the very douchiest when a superior says it to someone under him.
If you want to indicate preference, say "I'd rather you didn't" or "I'd prefer you didn't".
If you want to convey that it would be against policy or procedure, please just say so.
If you wish to express disapproval, say "That's not a good idea", or better yet, "That's a terrible idea".
We're grown-ass men. Don't talk to other grown-ass men like that. Certainly don't speak to me that way. You could say that I'm not comfortable with it. Instead, speak to me as if we were both mature adults. You can do it, little buddy. You can do it.