I have a similar issue with the wasps in my neck of the woods... It all starts with the birds... once they damage fruit on the trees, the wasps swarm in and begin eating all the fruit (while still on the trees). I have no idea where all the wasp nests are (not on the house or we'd have removed them), but I had to be very plucky to get out and harvest the fruit while avoiding the feeding wasps this year.
I'd be a lot happier if it were bees getting that delicious harvest.