25-06 would be a great rifle for an all around with little recoil and great performance. I say go for it!!
I've used them since I was 12 on all sorts of critters in Oregon. Only thing I haven't seen shot with it is sheep,mt goat, and cougar. Everythign else has fallen to the 25 without hesitation.
Got into the long range thing when I was about 14-15 and it was my primary rifle. Loaded with 115g NBT's at 3200fps, I took coyotes, whitetail, muleys, and black bear out to 600 yards with no problems.
Personally, it was never my "elk rifle" though. I knew it would work in a pinch, but I always toted a 7mm or 300 mag of some sort for elk. However, 1 year my sister needed a rifle to use so I set her up with my 25-06 and some 100g NP's at 3200fps (modest load) and I watched as she shot her cow elk at about 250 yards through the shoulder once and behind the shoulder for the 2nd and it was done for. The little 100g NP's did quite well, they didn't exit, but they still went a good 3/4 of the way through. Damage was massive. I believe if I would have set her up with some 115g or 120g partitions, they would have went to the far side, if not fully exiting. I just didn't wanna buy any since I had the 100s on my bench and knew they would work, shot placement is still the key factor, while using a good quality bullet for elk.
I also know a couple guys who use the 25-06 and one of them killed a raghorn bull with the 110g AB at around 525 yards. Bull didn't go much more then 10 yards, and fell over dead.
All in all, I believe the 25-06 is a truly great dual purpose rifle. You have 75-85's for varmints, 100's for antelope, and 115-120's for deer, bear, and elk. Load up the correct weight and type of bullet for the task at hand and have fun killing stuff. THey dont kick much at all, they are always accurate, and they kill very very good at a modest range for there size.