I had a Remington 700 ADL that would walk off the target by the fifth or sixth shot. When let cool, it would return to zero after about an hour. It was rocking between the screws. My gunsmith bedded and floated it, and after that it was my go to deer rifle.
I wouldn't condemn your barrel as temperature sensitive without checking the bedding and barrel channel. I just don't understand getting a Boyds, and not glassing the action. I would glass bed it, then, if that doesn't help your problem, you can still screw a new tube on later.