I was worried at first when I saw ours, but got the binoculars out and could see a little while tail end. They were only there a couple of monthsMason bees eat away at the mortar and can cause a lot of damage. The enter through gaps in the mortar. You may have had tree bumblebees which are quite rare and a different proportion altogether. Google them, the males do a dance outside the nest around May, unmistakable. They usually get in through the eaves or a gap in the slates.