First off, make sure your neck tension is good to start. You should be able to strongly push the nose of the bullet of the loaded cartridge against a wood block and not have it move back. If you can set the bullet back, your neck tension is lacking and your expander ball may need corresponding adjustment.
Secondly, don't continue to feed from the same round off the magazine again and again. What I do when I first load the rifle for a day of calling is strip the round off the top of the magazine by hand. Lock the bolt to the rear, point the muzzle down and just drop the round into the chamber. Ride the bolt forward and use the forward assist to put the bolt fully into battery. Lock your magazine into place.
When unloading, drop the magazine, and slowly pull back on the charging handle. Keep the palm of your support hand over the bottom of the magazine well with the fingers covering the ejection port. As you retract the bolt the round will drop right into your palm. Slip the round into a shirt pocket and use that as the round you load on your next stands until fired.
This accomplishes two things: A) It saves the round from the constant rigors of of being chambered from the magazine. B) It is a whole lot quieter in the loading process, usually just a "click" from the bolt locking up when pressing the forward assist.
Something else to check is if your lower has the M4 style feed ramps cut into it. I'm not to keen on the feed angles of the .204, but it may be part of the problem if they are needed and your lower doesn't have them.