This question is difficult to answer for a couple of reasons.
The first reason is that group howls, by their very nature, are little coyote vocal contests, each egging the others to better effort. Unless they respond from different directions, hard to say how many are involved. But, nevertheless, I will guess that most times it is groups of four to possibly six? When I have had them come into howling...after howling, it has mostly been one or two. A couple times, as many as five, which happened to correspond to my estimate, at the time. But, there are ways to find out how many are involved, if you really need to know.
I have had as many as nine coyotes respond, single file, on a day stand, but it was to a distress call, which is beside the point except for the fact that occasionally, you do have that many concentrated in one place, and they might decide to respond to a howl.
But, the other reason I think it is difficult to guess how many animals are out there is that very often, they come in without any answer, at all. Sure, you might get a response from a southerly direction from a group, and then you watch a single approach from the opposite direction, knowing that he had nothing to do with it. This is in those areas where you have the advantage of good visibility, and can say such a thing with a high degree of certainty.
In parts of the country where there is a lot of cover...Texas comes to mind, you would have great difficulty determining just how many animals you have responding to your howl, and getting a glimpse of an animal, or is it several(?) passing behind all that brush. I have been hunkered down in mesquite choked washes, and had coyotes drifting around all over the place, impossible to keep track of them.
In those situations where you hear a response to your invitation, and you know that they don't always respond....who's to say that you have three groups out there, two responding vocally, and one, not wasting your time, they are coming in? Unless you are in an aircraft, or something like that, observing, you will never know. And, that's okay, ain't it?
Good hunting. LB