@Jacob @Andy: I was hung up on that for a moment too, but Archie Whitman surely frequented the local brothel, and likely brought his son along for the ride on many an occasion. Sure, regular visits may not quite constitute growing up in such a place, but add in our protagonist's lingering shame over his origins and it's understandable that Don would perceive himself as being (uncomfortably) at home, and even identify himself as such in the presence of the madame whom he'll likely never see again.