Who hasn’t had a co-worker over the years that drove them crazy—constantly coming in late, avoiding work, foisting their assignments on someone else’s shoulders? Working with a co-worker who is disrespectful can take your job from okay to awful in no time. And with a third of our lives spent at work, that can start to feel like a very heavy burden.

But what about the co-worker who not only makes your workplace feel like drudgery, but who also makes you feel uncomfortable—so much so that you are this close to quitting? Someone who makes you feel like just about anywhere would be a better place to work.

Picture going to the copier and finding lewd images printed out, or someone circulating an off-color joke via email or, worse still, the individual—male or female—who makes a point of frequently telling you just how attractive your new haircut is and how it makes you look like “a hot movie star.”

What exactly constitutes harassment?

Under federal law, you are protected from harassment based on: Your gender, your race, the religion you practice, your age, any disability you have and your sexual orientation. These protections were codified in 1964 with the adoption of the Civil Rights Act.

So someone at work calling you a racial epithet would be grossly inappropriate, and it is unlikely it would be tolerated by upper management. But bear in mind, too, that one comment may not be enough to constitute harassment, especially if the offender is disciplined and/or never repeats the behavior.

So how do you handle a difficult workplace?

The best thing you can do when work has become miserable, is to talk to human resources. Keep a log of the times you call, email or otherwise alert HR to your concerns. Most employers want not only to avoid a potential lawsuit, they also don’t want to lose the time and money they have invested in an employee.

If the behavior—whether it is just plain disrespectful or flat out harassment—doesn’t stop, it might be wise to seek legal counsel. And keep in mind that, while being a jerk is not illegal, as Matt Verdecchia notes in a blog for the Society for Human Resource Management: “The line between disrespect and harassment is very thin.” Thus, what may start out as simple disrespect can quickly morph into harassment and that’s when legal action comes into play.