5 secrets of successful community surveys

Anyone who’s been to a typical “town hall” meeting in a city, with one resident after another grabbing the microphone for a few minutes, knows its shortcomings. The angriest voices often get the most airtime, and usually it’s the same small group of people who show up again and again. A similar pattern often plays out in online forums and in comments on cities’ social media posts.

That’s one of the reasons a growing number of cities are turning to community surveys. By reaching out directly to hundreds or even thousands of residents and asking for feedback by phone, mail, or online, city leaders are hoping to get a more accurate read on what the public thinks is — or isn’t — working.

Surveys don’t replace other forms of public engagement, of course, but cities often find they produce actionable insights. Two years ago, city leaders in Kansas City, Mo., went to voters with the largest bond authorization in the city’s history — an $800-million plan to fix up city roads, sidewalks, and stormwater system. City leaders did so with confidence that the issue would resonate: Time and again on surveys, residents had put crumbling infrastructure atop their list of concerns. The package required three separate ballot questions to pass with supermajorities. They all cleared it successfully

Surveys are nothing new for local governments, said Michelle Kobayashi, who wrote the book on this topic and consults with hundreds of U.S. cities on their surveys as vice president of National Research Center, Inc. What has changed over the past decade, she said, is that surveys — whether narrowly focused on a single issue or more broadly focused on community-wide satisfaction — have evolved into a widely adopted best practice.

[Get the latest innovation news from Bloomberg Cities! Subscribe to SPARK.]

“A good survey will require only 10 to 15 minutes for people to share their perspectives, and that’s attractive to a different person than those who can spend three hours with you on a Tuesday night,” Kobayashi said. “Surveys provide more diverse opinions into the decision-making process.”

If surveys help with public engagement, they also can give city leaders something else they need for good governance: data. Increasingly, cities are connecting resident survey data to strategic planning and performance measurement, to ensure that department budgets and priorities are closely aligned with what matters most to residents.

To find out more about how cities are using surveys, Bloomberg Cities spoke with the people behind leading community-wide surveys in Kansas City, Scottsdale, Ariz., and Tulsa, Okla., which recently overhauled its resident survey in partnership with the research firm Gallup. Here are five things they pointed to that make for a successful community survey.

1. Make it matter.

While a community survey can be a quick way to take the public’s current pulse, its value grows in time, when results become stitched into the way communities govern themselves.

Kansas City has run its Citizen Satisfaction Survey for nearly two decades — long enough for the survey to become a pillar of local governance that transcends administrations. The survey, administered on a quarterly basis, is intertwined with other hallmarks of the city’s famously data-driven approach to delivering better results for residents.

[Read: How Kansas City cemented data-driven management into city law]

For example, survey questions are used as outcome measures in Kansas City’s strategic plan. That means survey data regularly come up in monthly KCStat meetings, where Mayor Sly James and City Manager Troy Schulte sit down with department heads and assess how they’re doing at meeting their goals.

In other words: The survey is no one-off.

That’s also true in Scottsdale, which has conducted resident surveys since 1986 and shifted to its current survey format in 2003. “We’re able to talk with our council and community about trends over 15 years,” Assistant City Manager Brent Stockwell said. “You want to know not only how you’re doing at a point in time, but over time.”

Scottsdale runs its survey every two years after municipal elections. The timing is meant to provide the incoming city council with data they can govern with. “We get into a cadence where we ask the survey, have a conversation with council about the results, and then build that into our organization’s strategic plan and start working on those things,” Stockwell said.

2. Get buy-in from survey stakeholders.

Kansas City tailors its survey questions in ways that make the answers actionable for city departments. But that wasn’t always the case, according to Deputy Performance Officer Kate Bender. In the early days, she said, city staff saw the survey “as this external report that came out once a year, and mostly something to be defensive against.” Staff were brought more deeply into the process to ensure their buy-in. “You need to make sure your survey questions align with your organization’s interests and priorities,” she said. “That sounds basic. But being able to tell people in the Parks Department, ‘Here’s what people prioritize,’ and to tell people in the Public Works Department, ‘Here’s what people prioritize,’ makes it much easier to work with.”

[Read: Three tips on cross-sector collaboration]

In addition to engaging key city leaders in creating its new survey, Tulsa sought help from outside City Hall — from local nonprofits, universities, and philanthropy. Tulsa already relies heavily on cross-sector collaboration to deliver key services, and Chief Finance Officer James Wagner said applying that approach to the survey helped build community-wide buy-in for the effort. It’s also resulted in nonprofits and others using the survey to guide their own work. “The city is in the best position to ask questions and be the convener of stakeholders who need this data to inform their work,” Wagner said. “But we recognize that city services are just one piece of the puzzle.”

3. It’s worth the work to get a representative sample.

There’s a risk with surveys, Kobayashi said: If you’re not talking to enough people — and if you’re not talking with people who truly reflect the diversity of your community, then “you might end up with the same folks who are at the town halls — and then giving them an even bigger microphone.”

Cities need to make diversity a priority, Kobayashi said. One way to do that is to create surveys that take 10 to 15 minutes or less, to make answering the questions a light lift. Another is to follow up with frequent reminders to take the survey. “If you don’t follow up with the same person multiple times, you get a skewed sample,” she said. “The person who will answer your phone call the first time, or the person who will return the survey in the mail right away, is a different kind of person than someone who’s busy and thinks they’ll get to it later.”

[Read: 9 Ways cities can use data for results]

Changing technology, and the way people use it, is a factor, too. What the decline of landlines means for telephone surveys has been a hot topic among researchers for more than a decade. So is the question of whether online surveys can be trusted. For local leaders, those questions are good reasons to hire a survey research firm with a solid track record. “We’re fortunate to have a good partner in survey administration,” said Kansas City’s Bender, who works with another leading firm in the space, ETC Institute. “We rely on them to follow the shifting trends.”

4. Separate the signal from the noise.

To some local leaders, resident surveys produce information overload — it can be hard to make sense out of all the numbers and what they mean. Meanwhile, the data nerds will always want more.

For those who just want the big picture, a good place to start is by benchmarking your results against other cities. That’s another benefit of working with survey firms that do a lot of work with cities — they ask enough common questions that cities can see how their results stack up against others. That’s vital context. Once you find out that 80 percent of your community says your city is a “good place to live,” the first thing you’ll want to know is: Is that good?

For those who want to dive deep, Kansas City’s Bender has another suggestion: In addition to the standard report survey firms will typically produce, ask them for the raw data. “It allows us to do a lot more analysis and overlay it with other information,” she said. For example, Kansas City overlays its survey data with service calls coming in through the city’s 311 line. Doing this gave city leaders a clear view into recent troubles with trash pickup — and is now helping inform discussions around what to do about it.

Tulsa’s survey last summer received almost 4,500 responses, which Wagner said he can drill down into with some detail. For example, it’s possible to look at results by respondents’ race, age, and ZIP code. “We can draw conclusions about whether they think their neighborhood is getting better or worse versus the city as a whole,” Wagner said. “That’s not something we were able to do in previous surveys.”

[Read: In Tulsa, volunteers mine data to help government make better decisions]

At the same time, city leaders have to be mindful of what their survey results are — and aren’t — mathematically capable of showing them. Scottsdale’s survey reaches a few hundred people, which leaves a 6-percent margin of error. It’s important for city leaders to not overreact to findings that aren’t statistically significant. “There can be a lot of noise in survey results,” Stockwell said. “You want to focus on where something significant is happening.”

5. Use the results to spark policy change and conversation.

The surest sign that cities see the value in surveys is that they’re making policy decisions based on the results. In Tulsa, a plan under development to create an independent police monitor to follow up on citizen complaints and use-of-force incidents grew in part from survey results showing that less than 20 percent of black Tulsans say they trust the police. In Scottsdale, poor survey results related to drinking water pointed city leaders to dissatisfaction with the taste of the water; investments in the water plant have turned that around.

Then there’s Kansas City, where the 2017 vote on issuing bonds to fix the city’s infrastructure grew directly out of resident feedback via the survey. The survey results also informed the strategy city leaders used to build support ahead of Election Day.

“It was a testament to using what residents had told us,” Bender said. “And during the education period before the vote, we were able to communicate: ‘Here’s what we plan to do, and it’s based on what you told us is important to you.’”

The value of surveys isn’t always so direct. It will more often pay off in the form of better-informed discussions that draw on data about what residents really think, really want, and really care about.

“Without objective data and evidence like citizen surveys, all people have to rely on is their personal opinions or political beliefs,” said Scottsdale’s Stockwell, noting that the city’s survey forms a launch point for many conversations between the city manager’s office and city council. “Once you have data, it starts to inform those opinions and beliefs, and you’re adding objective information into what is otherwise a subjective discussion.”

With their focus on obtaining survey data, Kansas City, Scottsdale, and Tulsa all were recently recognized by What Works Cities, a Bloomberg Philanthropies initiative that helps cities use data to achieve better outcomes for residents. To get a check-up on your own city’s data practices and learn more about how your city can get What Works Cities Certified, click here.