I think that's a pretty good plan. The only issue is going to be working out the 'roof' of your irregular octogon shaped towers. Maybe using some plasticard cut to fit would do.
It's obvious, but patterning the walls to make them symetrical is always good. So having a doored wall section in the middle of your walls, then having the scroll piece on either side of it, 2 matching pieces on the other side of each of those continuing from the center looks best. You're going to need quite a few sets. I can't wait to see it.
-T