Your accuracy is dependent on the fit of the barrel, barrel nut, bolt and bolt carrier, and of course, the shooter.
If you could remove the upper, lock it in a holder like a Ransom Rest, and activate the firing pin without the lower, you would still have the same degree of accuracy.
Any looseness in the fit between the upper and lower is just an annoyance. I've installed the accu-wedge in a couple of mine that have a lot of 'slop' and I have two that fit tight enough that it's not noticeable.