I did a similar thing, planted rose bushes near the side of the house and fence. Then I was really happy with the bumble bees until I found-out they were wood boring bees and they had made their nests in the wood under my roof and fence posts. I had to replace the fence and the wood under the roof. I also uprooted the rose bushes :( Just chalk it up to bad luck, you should be fine.