By definition, rich depends on poor for contrast, otherwise the concept has no meaning.
What is well off is relative. There are people living in private homes on acres of areable land throughout the world that would rather live in a slum in NYC because even the poor here are richer than them lot there. Most people in America came from people like that.
Houses in areas with the highest real estate taxes are no less subject to speculation. In fact, they became that way because of it. The smallest, crappiest, ugliest house under the #4 Train El on Jerome Ave in the Bronx is worth more than most of the nicer ones in Oklahoma. And despite have far lower property taxes, Oklahomo's homes will continue to be far cheaper.
Poverty is a product of uneven wealth distribution, and that would be the result of a relative few making most of the money. As the masses lack the means to individually generate massive cash flow, they end up having less of it.
Taxes aren't the cause, but the result, determined by what citizens can afford relative to their level of wealth.