If you already own a .204 then you have a good gun for close to medium (400?) range. The Swift will push that farther but not by much since with a 12" twist about the heaviest bullet you can shoot will be a 60 or maybe the Berger 64 HP.
As the owner of a .22-250 AI (duplicates the Swift ballistics) with a 10" twist barrel, I shoot the 60 VMax. Great bullet for accuracy and terminal effect, but gets blown around pretty badly at the longer distances (past 450 or so).
To me, the lack of a really good varmint-specific .224 bullet in the heavyweight range (70-80) coupled with the need for a 10" twist at least (to stabilize the Berger 70 HPBT LTB bullet; the 70 VLD requires 9") means you can't wring the top performance from a .22 without a custom fast-twist barrel which nobody makes on a factory bolt gun, and makes me think the .243/6mm is the way to go for a factory long-range gun. Load the 87 VMax with .400 BC and don't look back. You'll have two good rifles whose performance complement each other.
At distance, wind drift matters less than trajectory, since the latter is fixed and unchanging (per stationary target) and will have to be figured correctly anyway no matter what it is, while wind is constantly changing and shifting in both speed and direction, and cannot be figured exactly. So you want to narrow your margin of error with the bullet that has the least drift. The heavier weights of .243 bullets are not necessary to kill the varmint, but rather to resist wind drift beyond 400 yards. The need for this resistance grows radically with increased range. So, shooting a 58 or 65 grain bullet in the .243 is really a waste of time for the longer shots - go with the 75 or 87 VMax or the 105 AMax match bullet for the really tough shots.
Remember you will always have to compute drop compensation no matter what it is, but with the wind you will be forced to guess at something constantly changing.
Just my $.02.