It’s not just you. Love handles—those pockets of excess fat over the lower sides of your torso—aren’t exactly easy to, well, handle. In fact, even devoted gym-goers often struggle to shed these seemingly omnipresent pockets of fat, no matter how many workouts they do.

But fear not: Though love handles may be stubborn, they aren't invincible. Here's what you need to know about defeating this common foe.

Why you get love handles in the first place

This side fat exists because of sheer physiology—and there's not much you can do to defeat that.

“For men in particular, there are more fat cells in this region, making it a convenient depot for the body to dump excess triglycerides, the storage form of fat,” says Jeffrey M. Willardson, Ph.D., C.S.C.S., a professor of kinesiology at Eastern Illinois University.

Put another way, "fat wants to go to its house—to the fat cells—so that’s the place where you’re likely first going to put your fat,” adds Wayne L. Westcott, Ph.D., an instructor of exercise science at Quincy College in Quincy, MA.

Of course, love handles develop over time not overnight. Blame a lack of workouts, empty calories, lots of stress, and/or poor sleep, which can up cortisol and adrenaline levels, encouraging fat storage, says Willardson. And remember: “Some people, due to age or genetic predisposition, will always have a little more fat in that area regardless," Willardson says.

Why love handles are so hard to lose

The disadvantage? Once love handles appear, biology makes them tough to ditch. “You lose fat in the reverse order that you deposit it,” says Westcott. “And typically for men, the last place they lose fat would be that midsection area—the first place they put it on.”

What gives? If you think about fat storage from an evolutionary standpoint, it could be that we put fat on in our midsection first because it’s easily accessible energy your body could grab if it was fasting or faced with a shortage of food. So when you aim to lose fat, your body first removes it from arguably ‘less essential’ areas like the face, Westcott says.

Areas with the highest concentration of fat cells are also the most difficult to firm up, says Willardson. The love handles in particular are problematic because the muscles under them—your obliques—are thin, so they won’t bulk up, explains Westcott. Think of your obliques as sections of cable rather than a 6x6 piece of sheetrock—there's not a lot of muscle to stack on.

How to get rid of love handles

So what can you do? Overall, a plan of muscle-conditioning exercises, done regularly and with the right nutritional background, will improve your functional fitness, posture, and metabolic after-burn, notes Willardson. Consider total-body moves like squats, upper body pulling and pressing, or core work (planking type movements) every other day, plus high-intensity interval training on alternate days, he says.

Don’t obsess on your core, though. “You don’t necessarily lose fat in the area you train,” reminds Westcott. The majority of research suggests ‘spot reducing’—working specific body parts to lose fat there—doesn’t work. It’s worth noting, however, that specific targeted exercises—when combined with cardio, strength training, and a healthy diet—can indeed impact fat loss, he says.

His suggestion? Incorporate a few moves three times a week that target the obliques. Consider the bicycle crunch: Lie on your back, extending one leg, while bringing the opposite knee toward your chest. With your hands behind your head, touch your opposite elbow to the close knee. Twist your body and touch the other elbow to your other knee. Or make one of these 30 best abs exercises of all time part of your regular routine.