It's the usual GW "kinda here, kinda there" type rules, but the majority of it comes from page 128. If you use the glossary at the back of the BRB, under 'Challenges', there is a page number given for Challenges in Building Assaults. The relevant rules are:
pg128 - Allocating Attacks - last paragraph
Note that casualties are taken fromthe "back" of the unit as normal. If one side suffers casualties before it fights, extra models areassumedto step up and join the assault party in place of fallen comrades. So ten modelswillalways get to fight, assuming there are enough left to do so!
That's the big that explains that if he wastes your Champion in the challenge, you still get to 'Step Up' and fill in the Champ's spot. Assuming of course that the Champ hasn't already struck his blows (for instance, because he has an ASF weapon, or because he's a Character with better Initiative). The underlined bit is what's important here: 10 models will
always fight. So you always get to nominate 10 models to attack.
pg128 - Challenges - single paragraph
A single challenge may be issued and accepted as normal, but only models chosen as part of the two sides' assault parties can issue, accept, and refuse challenges - you can't use the challenge to increase the number of models fighting.
The part where they explain that the challenge is fought as normal sends you back to the Challenges section of the book. Basically, they're fought exactly the way they normally are - only models engaged in the challenge can strike one another, etc etc.
The most ambiguous thing about Building Assaults, actually, is the step-up rule. In one particular game, I had a rather cheese-mongering opponent stick the following unit into a building:
19 Dwarf IronBreakers w/ Full Command
1 Thane BSB w/ GreatWeapon
For his assault team, he picked 9 'Breakers, and their Champion.
I sent in 9 Chaos Warriors, and my Champion against them. He accepted the Challenge with his Champ.
Once I murdered the Champion, he was entitled to 'Step Up'. He elected to add his Thane to the assault party, and proceeded to
mutilate my Warriors.
There are no rules in the book which prevent this.
Why? Because there is nothing which says that only "Rank and File" models are allowed to 'step up' into the fighting ranks. Players have naturally assumed that models are assumed to step directly forward - if you kill the front rank of a unit, characters who somehow found themselves in the second rank will step up and become vulnerable.
In reality however, there is nothing in the book that says this is how it works, and in a building assault, the abstract nature of the "assault party" means that it is entirely viable to put that Thane into the Assault Party
after the Champion is killed.
Now, if he were joining the party after his own initiative then he would
probably have to fore-go his attacks (we've struck all the I5 blows, so your I9 Elf missed his turn), but there's not even an 'official' rule specifying
that.
So yeah - figure
that out, once.