In a game tonight, my Chaos Warriors were grinding their way through a unit of Dwarfs including Thane BSB and Runesmith. Obviously neither character was really geared for combat since, each turn, my opponent was refusing my obligatory challenge, allowing me to send his Thane to the rear ranks (from where he was annoyingly helping to keep the Dwarfs in combat...)
I had just ground him down to one rank, which would normally prevent him from refusing a challenge, because he wouldn't be able to move either character out of base to base contact. However, at that point he flanked me with his Quarrellers, who also had a champion and Runesmith.
So, I issue a challenge. Now what? He has four models who could accept: Thane [BSB], Runesmith 1, Runesmith 2, and champion. Some of these models could also refuse, by being able to retreat to a rear rank of the Quarrellers, but neither the Thane nor Runesmith 1 could get out of contact. The trouble is that my challenge isn't directed at anyone in particular, so we weren't sure what to do and couldn't find any rules to cover it.
What I think would make sense is that, after I challenge, either Runesmith 2 or the Quarreller champion have the option of accepting (because they can still run and hide) but, if neither of them accept, either the Thane or Runesmith 1 has to accept. In other words, I think that he cannot refuse the challenge unless all of his characters are able to refuse. However, there is the alternative possibility that he can refuse because there is at least one model that can be banished to the back.
What happened is that we rolled a dice to see whether or not he could refuse, decided not, and then he took it with his champion.