Ever think about a pistol??? Less recoil on the shoulder than any rifle. Check out the classifieds here. Guy has a
"Remington XP100R in 223 comes with Leupold 2 pc. mount (dove tail front and windage rear), Redfield rings, Leupold M8 4X scope, Timney trigger plus factory trigger, box and papers."
Just a thought.
However, if you want a rifle (advantages/dis for both) I think any of the calibers below the 22-250 would be comparable. Especially if you get a "Varmit special" gun that weighs 8+lbs.
The .17 rem would be the mildest because of bullet weight/created energy.