I seem to recall that you use a lead sled Hidalgo? If that's the case then it (probably) isn't the shooter.
Is the "1 out of 5" a generalization meaning about 20% of the shots are flyers randomly distributed in your strings, or is it specific as in "the first shot of every five shot string I shoot is a flyer"? If you change to 10 shot strings would the incidence of flyers change to 1 in 10 shots?
Is there a consistency to the flyer of any kind, i.e. is it the first shot out of a cold barrel, always the second shot, always the last shot, etc?
Will the flyer always go in the same place?
You groups sucked with factory ammo, but was there consistantly a flyer there too?