A foam earplug will do the same thing.
In theory, play between the upper and lower won't affect accuracy.
However it can have a mental effect on the shooter, if it's bothering you it will mess with your group.
If the wiggle is so bad that it causes the scope to look canted, that would be more than I could stand.
If you don't mind me asking what brand of upper and lower do you have.
