Measure your neck on the .204 of a factory round and one that has been fired one time, as well as one that you have just sized...The amount of 'spring back' is your first sign of needing to anneal if that is a concern...
There are three things with small caliber rounds that can affect your seating process...Case neck lubrication, brass brittleness, and your neck sizing ball adjustment...
Failure to have adequate lube in the case neck will cause a 'hard feel' when seating a bullet, brass that has been reloaded too many times will start to get brittle and there is a factor difference between full length sizing and neck sizing as to the amount the brass is worked, and if your sizing ball is not correctly set in the sizing process, the seating process can be affected...
Most of my sizing dies are adjusted with the neck sizing ball positioned where the pressure relief hole is located in the die, as that is when the ball and the die wall is sizing the neck at the same time..If the sizing ball is too far down, or too high in the sizing die, the case neck will not be at it's optimum tension/size.