The point is that we called it Barnbrack. Not Barmbrack. Now, barm is the foam, or scum, that forms on the top of fermented alcoholic beverages and is used as a raising agent in bread. And it's perfectly true that a brack requires a raising agent. But it's name is Barnbrack, not Barmbrack. Besides, we always used yeast.
I rest my case.