Actually it's not.
Hunting, that is, so called sports hunting - aims - quite literally, for the biggest bucks with the biggest racks. They don't go for smaller males, males with deformed antlers, or females. Since females are the ones directly reproducing, the more that there are, the more of them there will be.
Smaller males take up less room and might be less challenging to other males, so therefore after the hunting season not only is there proportionately more of them, more of them can fit in a given area.
More females and smaller males thus increase population density. This makes them more prone towards disease spread.
Larger males, being territorial take up more room, and drive others away. This helps to relieve environmental pressure. They also improve the general strength of the herd by passing on genes for larger & stronger males. So what happens with regular hunting by people is that you end up with runts that are less likely to be in prime physical condition.
Over time you end up with populations with an increase in deformities, more readily observable with their antlers, as well as increased susceptibility to disease. This is a phenomenon that's easily observable in a place like Britain where predators had been eliminated long ago, and sports hunting has produced such deer as a result.
Natural predators, of course, weed out the weaker and the smaller, improving the overall health and strength of the breed, reducing pressure on the habitat, as well as forcing deer and other prey to wander farther away in order to keep away from their predators, reducing habitat degradation even more.
There also has been a general decline in hunting, so while some killed, overpopulation tends to be an increasing problem, outcompeting smaller species of animals as well as not allowing for proper regeneration of their habitat as they basically eat small saplings and anything else they can consume, leaving what looks like a healthy forest, but one that is actually in general decline as it ages.