To destroy walls u need use manogels/ cannons/ upgraded rams. Any other unit must destroy 8 walls, not 1 :D
So that walls can be destroyed quickly only in imperial age(15 century in Chivalry).
If u would find more units that can destroy all walls by one time - pls, tell me about that(name of unit, required upgrades).
Have fun :)