You're more likely to have problems with 2-legged varmints than 4-legged. This isn't necessarily true, depending on location. Since the original question is "while you're hunting," it's a relevant point.
I don't hunt much, but I live out in the boonies and we're hip-deep in everything from coyotes to black bear. We get mountain lions, feral dog packs, several sorts of poisonous snake, even an occasional wolf thanks to the government. Everybody I know carries a handgun around here.
After a few less than impressive through-and-through hits with my .45, I found somebody to sell me a beater Taurus 431 .44 Special. A year ago I used it on a coyote that was sneaking down on my chickens, and it seemed to work all right.
But guys around here carry everything from Ruger LCRs to single actions in .45 Colt. Four summers ago there was a big fire south of us that drove a lot of wildlife our way. A neighbor was home in the evening when a big bear broke through his glass door. I never knew they'd do that. My neighbor didn't want to take the time to go for his rifle, so he shot it three times in the side with the compact 9mm he had with him. That same evening the same bear - wounded in the side - showed up at another neighbor's 4-5 miles away. That guy put it down with a shotgun. So while the nine solved the first guy's immediate problem, I'd call it underpowered for bear. But we don't actually see all that many bear.

Getting back to the original point: Most of us who live out here have shot a predator at some time or other, but I don't know anybody who's been in a gunfight. So it seems logical to let that guide handgun selection.