It's entirely reasonable to expect people look out for traffic and not make people stop in order to pull out of their driveway. I do it every morning. I'm not more important than anyone else on the road, nor if your neighbor.
I would like to know the rule, official or not, on this. People often honk at me, or more bewildering, pass behind me as I am backing out. And these are always non-neighbors. I believe if at least be axle was in the street when they showed up, they can wait a literal minute.