First it was all, "you can't even order anything if you don't speak French". Which sounds horrible...so horrible as to be almost unreal. Then, once he gets called out it's, "I'm just pointing out that it would be difficult".
It's almost like he was using hyperbole, bordering on xenophobia and fear mongering, to make a point and then immediately backpedaling once it became clear that A) the point is invalid, and B) we're not as dumb as he thinks we are.
Here's the thing Frank, every single place has some drawbacks. Even your beloved Newport Beach for example. The chances of you experiencing an earthquake, traffic congestion, smog, or urban decay in California are probably much greater than a millionaire being unable "to order anything because he doesn't speak French" in Montreal.
Last time you were in the Los Angeles area did you not notice any of the substantial problems it has? Or do you avoid going there (and don't plan to) because of all the Hispanic people speaking a language you don't understand?