A new push by states to tax the real estate of the wealthy has sparked a backlash among brokers and potential buyers, who say the taxes punish the most important local spenders.

From tax hikes on pricey second homes in Rhode Island and Montana to Cape Cod’s proposed transfer tax on homes over $2 million and the L.A. mansion tax, state and local governments see a revenue gold mine in the pricey properties of the wealthy.

“It’s a smack in the face to people who just spend money here,” said Donna Krueger-Simmons, sales agent with Mott & Chace Sotheby’s International in Watch Hill, Rhode Island.

The tax hikes are being driven by tighter state budgets and populist anger over housing costs. States are looking to offset budget cuts expected from the new tax and spending bill in Washington. At the same time, the housing market has become a tale of two buyers, with the middle class and younger families struggling to afford homes while the luxury housing market thrives from wealthy all-cash buyers.

The solution for many states: tax the homes of the rich.

Rhode Island’s new levy, nicknamed “The Taylor Swift Tax,” is among the most extreme. The popstar bought a beach house in the state’s elite Watch Hill community in 2013.

The measure imposes a new surcharge on second homes valued at more than $1 million. For non-primary residences, or those not occupied for more than 182 days a year, the state will charge $2.50 for every $500 in assessed value above the first $1 million. That charge is on top of existing property taxes and will add up to big increases for luxury homes in Newport, Watch Hill and other well-heeled, summer communities in the state.

A version of this article appeared in CNBC’s Inside Wealth newsletter with Robert Frank, a weekly guide to the high-net-worth investor and consumer. Sign up to receive future editions, straight to your inbox.

Swift’s house, for instance, is assessed at around $28 million, according to local real estate records. Her current property taxes are estimated at around $201,000 a year. The new charges will add another $136,442 to her annual taxes, bringing her yearly total to $337,442 — even though locals say she rarely visits.

Real estate brokers say the increase targets the very taxpayers who already contribute the most. Wealthy second-homeowners pay hefty property taxes but don’t use many local services, since their primary residences are in New York; Boston; Palm Beach, Florida; or other locales. Their kids typically don’t attend the local schools, and they’re infrequent users of the police, fire, water and other municipal services since most stay for only 10 to 12 weeks out of the year.

“These are people who just come here for the summer, spend their money and pay their fair share of taxes,” said Krueger-Simmons. “They’re getting penalized just because they also live somewhere else.”

Brokers and longtime residents say the summer residents of Newport, Watch Hill and other seasonal beach towns are the economic engines for local businesses, restaurants and hotels.

“You’re just hurting the people who support small business,” said Lori Joyal, of the Lila Delman Compass office in Watch Hill. “You’re chasing away the people who spend most of the money in these towns.”

Rhode Island is also hiking its conveyance tax on luxury real estate starting in October. The tax on real estate sales will be an additional $3.75 for each $500 paid above $800,000 for a real estate purchase. At the same time, the state’s steep estate tax deters many of the ultra-wealthy from living there full-time.

Brokers say some second-home owners are considering selling and many would-be buyers are pausing their purchases. While the tax hike alone isn’t expected to lead to any significant wealth flight, Joyal said potential buyers in Rhode Island are already looking at coastal towns in Connecticut as alternatives.

“It’s always about choices,” she said. “At the end of the day it’s about how they can choose to spend their discretionary dollars. Connecticut has some beautiful coastal towns without some of these other high taxes.”

Montana has passed a similar tax. The influx of Californians and other affluent newcomers who poured into the state during Covid has led to soaring home prices and growing resentment over gentrification. Meanwhile, the state’s low income tax rate and lack of a sales tax has left it little room for revenue increases to handle the necessary increase in services.

In May, the state passed a two-tier property tax plan, lowering rates for full-time residents and raising taxes on second homes and short-term rentals. For primary residences and long-term rentals valued at or below the state’s median home price, the tax rate will be 0.76%. Homes worth more than that will face a tiered-rate system of up to 1.9% on any value over four times the median price.

The Montana Department of Revenue expects the changes, which will start next year, will hike second-home taxes by an average of 68%. Brokers say some buyers are waiting to see the tax bills next year before making any decisions about whether to buy or sell.

“I’ve heard about some buyers who have put on the brakes to wait for the dust to settle and see what happens,” said Valerie Johnson, with PureWest Christie’s International Real Estate in Bozeman, Montana.

Johnson said that while the tax was touted by legislators as hitting wealthy second-home owners, it will also hit longtime locals who own investment homes and rent them out for income.

“These are small businesses for many people,” she said.

Manish Bhatt, a senior policy analyst at the Tax Foundation, said tax hikes aimed at wealthy second-home owners may be popular politically, but they rarely make for successful or efficient tax policy. Real property tax reform should be broad based, rather than focused on taxpayers who are singled out just because they don’t live in a community full-time, he said.

“There is a grab to find revenue right now,” he said. “But taxing second-home owners could have the opposite impact — dissuading people from owning a second home or continue to own in those communities.”

While the new taxes alone might not drive out the wealthy, “we do know that taxes are important to businesses and individuals and could cause people to make a decision to buy in another nearby state,” Bhatt said.

The projected revenue from the new taxes may also disappoint. When Los Angeles passed its so-called “mansion tax” in 2022, proponents touted revenue projections of between $600 million to $1.1 billion a year. The tax, imposed on real estate sales over $5 million, has only raised $785 million after more than two years, according to the Los Angeles Housing Department.

Higher interest rates that hurt the housing market have played a role, experts say. Yet Michael Manville, professor of urban planning at the UCLA Luskin School of Public Affairs, said wealthy buyers and sellers also reduced transactions in response to the tax.

“The lower revenue is a reason to be concerned because it suggests that the tax might actually be reducing transactions, which in turn can reduce housing production and property tax revenue,” he said.