"Privileged" is not an insult.
When someone points out that you are privileged in some way, they are not calling you a bad person. They are not calling you unintelligent or uncaring. They are not saying that you have had an easy life, or even that you have had an easier life than every member of the marginalized population being discussed.
What they are saying is that there are certain things you've simply never had to think about because of you are. These are things that other people have to think about because of who they are.
Let me give you an example.
I was a campus tour guide in college. One day, I had a prospective student in my group who used a wheelchair. The tour started off normally. But something went wrong just after we passed the Student Health Center. The end of a long sidewalk didn't have a slope. There was no way for the high schooler in the wheelchair to get down from the sidewalk to cross the street. He had to travel all the way back along the sidewalk to the side that did have a slope, then come back down the street to rejoin the group.
I was mortified. It was such an obvious sign of the campus not being fully wheelchair-accessible, and right by the Student Health Center of all places. This student could have been embarrassed to have everyone have to wait for him, and he couldn't have had a great impression of the school.
But you know what? I had probably crossed the street at that location hundreds of times prior to this incident. And I had never noticed the lack of a slope in the sidewalk.
That is my able-bodied privilege.
Since I don't have mobility concerns, I simply did not register that the sidewalk at that particular intersection wasn't wheelchair-accessible. I just stepped over it without thinking.
So when someone calls you out on your privilege, something like this is what they're talking about. The curb that you've metaphorically stepped over without realizing it is a much bigger problem to someone else. The very best thing that you can do is to realize that things exist that you don't notice simply because you don't need to, but that other people do. Just listen to them.