Did you pattern test off of a rest such that you know an exact point of aim so you can tell where the center of the pattern matches to the POA? Many--and quite possibly most--shotguns do NOT center the pattern on the POA at all distances. My Stoeger, for example, shoots the center of the pattern slightly high at 30 yards so I have to aim as though I'm trying to shoot the bottom of the shoulder. This is very common with shotguns and it varies with loads, so while #4 shot might center, BB may not.
If you missed or nearly missed with BBs at 20 yards, then the load isn't likely to be the problem. I use Hornaday Dead Coyote in BB shot and it flattens them even at 30 yards. At 20, you just didn't get them in the center of the pattern for some reason.
Also, keep in mind that as shot sizes go up, pellet count goes down. So you might hit them harder with buck, but you hit them less as well.
Grouse