Today Sat Sun
It is forecast to be Partly Cloudy at 11:00 PM EDT on August 22, 2014
Partly Cloudy
24°/18°
It is forecast to be Chance of Rain at 11:00 PM EDT on August 23, 2014
Chance of Rain
21°/18°
It is forecast to be Partly Cloudy at 11:00 PM EDT on August 24, 2014
Partly Cloudy
23°/17°

13 Comments

news

Toronto’s Budget Survey Deeply Flawed


Since long before he was elected mayor, Rob Ford has championed the idea that Toronto was spending its way towards fiscal disaster. Believe that or not (and many don’t), Ford swept into office on a wave of anti-gravy promises, so it’s no surprise that he’s launched a massive review of City-run services—via a series of roundtable discussions and an extensive online survey—with the aim of determining which are Torontonians’ greatest priorities, and which might be suitable for spending cuts. That is, after all, what people voted for.
However, despite a lot of noise about broad public consultation, the review is not likely to generate much meaningful public input. For one thing, the roundtables have been crammed into a whirlwind two-week schedule, with only limited participation available due to registration limits (there is one remaining session, taking place tomorrow night at 7 p.m. at the Scarborough Civic Centre). Moreover, City has chosen to purchase a DIY survey tool rather than commissioning a qualified polling firm to design its questions properly.


The massive survey asks people to weigh in on which services Toronto should drop or contract out to close a $774-million budget gap. City Hall could have handed this critical part of the review to any of Toronto’s numerous research firms with strong track records in public affairs—a shortlist would include Ipsos Reid, Harris/Decima, Polaris, Environics, and Vision Critical. Instead, they opted to purchase a simple tool which allowed them to design the survey themselves, from a company called Qualtrics.
The document resulted, according to Glenys A. Babcock, a former VP at Ipsos Reid who now works as a consultant, is poorly designed and suffers from inherent political biases.
20110606survey2.jpg

You Can Say Anything We Want

Do you want to tell City Hall that public transit is an important issue for Toronto as a whole? Well, you can’t. The survey lists seven broad issues and asks respondents to rate them by importance; transit is not among them. Your only option is to tick of “infrastructure,” which includes everything from water to roadways. Are affordable daycare, support for the elderly, or universal accessibility important to you? We can’t even guess which category those fall under. “Meeting the basic needs of vulnerable people” seemed likely, but later on in the survey it becomes clear that “vulnerable people” is a code-word for “crime-prone youth in poor neighborhoods.” A blank space after the question allows for write-ins, but it doesn’t let you rate issues by importance, and provides little-to-no basis for comparison.
“We have to ask why [...] such an obviously lousy survey was sent out,” Babcock says. “This is about Rob Ford and accountability. Where is the accountability here?”
As she goes through the survey, Babcock’s frustration grows. She chalks up most of its flaws to inexperience, but some oddities make her suspicious. The online survey lets you click through almost every screen without ever answering a question, but you must provide your postal code. Already annoyed at the survey’s weak privacy policy, Babcock was less than thrilled by this. “I thought, ‘Isn’t that interesting—they want to know what ward I’m in.’”
Though it aims to sort respondents, to discern the different needs and opinions of various demographics, the survey’s categories seem illogical. You wouldn’t normally group 15-year-old high-schoolers with 24-year-old university grads who live and work on their own, would you? Or, to give another example, if you rode the TTC once over the past year, would you put yourself in the same ridership group as people who rely on it every day, or those who buy tickets for their children? If you were designing this survey, apparently, you would. “How are these the same people?” Babcock wants to know.

Garbage In, Garbage Out

When it offers more than a handful of possible responses, the survey goes overboard and sabotages itself. It asks for in-depth feedback on 35 different service categories, each of which is subdivided into “activities,” to create a laundry-list of decisions on what services the City should provide, farm out, cut, or improve. This single question from the online survey fills almost three printed pages. Included are such items as the police, the fire department, and Emergency Medical Services (all as separate entries), and a single entry encompassing all “arts, culture, and heritage programs” but another one for “city-run live theatres.” Garbage collection is there, and public health, and “funding and programs for vulnerable groups,” and the Toronto Zoo.
It’s a little surprising to see bedrock services like health and firefighting on this list at all—what would happen if everyone said that the City should drop them? Presumably, the City would ignore those responses and keep providing the services. So why are they padding out this list? With so many choices in front of them, many people would be reticent to rate every service highly, which means that on an overcrowded list, some entries will get bumped down arbitrarily.
In Babcock’s view, the main consequence of such overcrowding will be a tendency for respondents to answer randomly, seeing a wall of options to get through, instead of a set of core services that need case-by-case evaluation.
“The results are likely to have an enormous random element to them and not provide meaningful input,” she says. Or, as she also put it: “garbage in, garbage out.”
20110606survey3.jpg

More = Less

After deciding which of the services should be City-run, respondents are asked to choose only three from the large list for further discussion. At that stage, it’s not hard to see how things that matter a little bit to a lot of people will overshadow those that are crucial to a few.
Asked to choose between police services and “community-run heritage programs,” how many will voice their opinions on the latter? The survey, in other words, can push respondents towards thinking in terms of the bare minimum level of acceptable service. Is the outcome of such an exercise likely to be something most Torontonians will be happy with?
One of the most striking features of the survey: respondents are asked, for any given service, whether “maintaining the quality is more important” or “lowering the cost to the City is more important.” Think the service should be improved? There’s no check-box for that. It provides another misleading set of choices when it asks respondents how they would choose to pay for any cost increases—via increased property taxes, higher user fees, or a combination thereof. Conspicuously absent: the array of other revenue-generating tools the City has at is disposal, such as the now-cancelled Vehicle Registration Tax or the Land Transfer Tax Ford has promised (but cannot afford) to cut. The survey simply chooses from among the full range of options the City could consider, and presents only some of these to the public for deliberation.
There are other issues. The survey ascertains respondents’ nationality, but never mentions settlement or newcomer services. It asks whether they have received a “university diploma” instead of a degree. Some of these things would be funny, if they didn’t point to an unsettling lack of attention to detail in such an important document. Overall, the whole thing is a toxic blend of incompetence and self-assurance, delivered with a populist spin and a political agenda.

Ford has enjoyed a honeymoon of sorts, with his popularity buoyed by hopes that he will steer Toronto’s economy towards the right without causing much damage. The structure of this review makes it clear that these are fantasies indeed. The survey, in the guise of speaking to the public, does little more than steamroll over Toronto’s diversity of perspectives.
Although Babcock filled out the survey, she did so with a deepening sense of futility. As that feeling spreads, it will poison future attempts to connect City Hall with Toronto residents. Rather than squeezing savings from the budget using faulty analysis, those behind the review should ask themselves whether the City, in the long run, can afford to burn so much public trust over a manufactured panic and some poorly chosen questions.

CORRECTION: June 8, 2011 When we originally published this post we attributed the poor design of the survey to Qualtrics, the firm the City chose to help facilitate this part of the consultation process. However, Qualtrics sells do-it-yourself survey tools and did not itself design the survey; the company is therefore not accountable for the survey’s poor design. We have amended this article to reflect this, and send our apologies ot Qualtrics.

Comments

  • isyouhappy

    Thank you for this article! Keep this up!

  • Nick

    Everyone's Toronto is different, some people value this service more than that (daycare vs. ice rinks), and I'm wondering what at all could be the value of this survey…I did it last week and felt the same as you Andre – presumably the service that gets the most votes wins, at the expense of others, which is the tyranny of the majority, and not what a civilized society is about (looking after all its citizen). Toronto has a revenue problem, exacerbated by tax cuts and freezes implemented by Ford, and by his failure to go to higher levels of government to get back our tax money that flows out of the city (Torontonians pay more in taxes to higher levels of government than we get back in services). He's turned Miller's surplus into a $775 million deficit, and in spite of his promise of no service cuts, this is sure to happen. Or, there will be user fees for the “unpopular” services, which is taxation in another guise, probably on people who can least afford it.

  • 24601

    Just taking the survey now.  It's pretty bad.  So bad that I doubt anyone will get any usable data out of it.  Which is probably the point, they'll be able to draw whatever conclusions they like, and point to this mess of a survey and say “We asked, you answered.”

    Seriously. How is it an option in this survey to select “Increase User Fees” for Police Services? What would a Police Services User Fee look like?

  • http://piorkowski.ca qviri

    Turning 911 into a 1-900 number, presumably

  • http://sueedworthy.ca/ Sue

    I attended the City Hall roundtable session – link to my thoughts and experiences here http://tinyurl.com/3rwwupp. Something else I pointed out and wrote down for the facilitator is the incredibly bad/suspicious optics of something as simple as the choices not being alphabetized – I immediately noticed as I was specifically looking for “arts and culture” and even though it starts with an A, it is nearly at the bottom of the form. The amount of scrolling to get to the bottom of the list is unreal – the list continues long after the choice boxes. Poorly designed, and as I said, the optics are terrible, like important things were arbitrarily chosen.

  • http://sueedworthy.ca/ Sue

    I attended the City Hall roundtable session – link to my thoughts and experiences here http://tinyurl.com/3rwwupp.
    Something else I pointed out and wrote down for the facilitator is the
    incredibly bad/suspicious optics of something as simple as the choices
    not being alphabetized – I immediately noticed as I was specifically
    looking for “arts and culture” and even though it starts with an A, it
    is nearly at the bottom of the form. The amount of scrolling to get to
    the bottom of the list is unreal – the list continues long after the
    choice boxes. Poorly designed, and as I said, the optics are terrible,
    like important things were arbitrarily chosen.

  • http://twitter.com/gilmourtaylor Geoff Gilmour-Taylor

    Wait, so Sports & Culture is a single category to be ranked? The “detailed” questions are ridiculous. The “city-run live theatres” are three theatre/concert venues (not named). I know this because I was asked to rank the live theatre responsibilities, (but there was only one). The responsibilities in the detailed categories read like an org chart, and sometimes overlap (describe in up to 5 sentences the duties and responsibilities of your department).

    I agree Babcock, this is very amateurish.

  • http://paul.kishimoto.name Paul Kishimoto

    Just to hammer the point: if you ran a business on such amateurish market research, you'd soon be out of it.

    This is the short-form census survey in miniature; or perhaps a slightly formalized version of the mayor's “Call me!” approach. It is not designed or intended to inform good government, or even figure out what kind of bad government people might want.

    It does two things: one, provide data that is insufficiently detailed to point out flaws in the things Ford & co. have already decided to do, yet which can be cursorily referenced to add legitimacy to same. Expect to hear the phrase, “People told us…” used in support of things the majority of citizens do not want. Second, it neutralizes and defuses opposition in advance. Expect to hear, “If you don't like this, why didn't you give your input earlier, when we offered you the chance? The time is past etc.”

  • Benssss

    Well done as usual, André.

  • http://twitter.com/little_potato Ambrose Li

    Don’t they even have one single French translator working in the entire city? Why is the online survey providing a French version that is essentially gibberish? (and Chinese versions that are only slightly less gibberish?)

    If they can’t provide the survey in “51 languages”, then don’t…

  • http://twitter.com/johnnydib Johnny Dib

    Thank you, very much what I thought about it. 
    I sent an email to the Mayor expressing my dissatisfaction with the survey and the terminology used by his entourage.
    http://johnnydib.tumblr.com/po… 
    I got two automated replies one saying I will read your email and the other saying I read it, and both were still referring to Residents as Customers and to Citizens as Taxpayers which is disgusting, the next day I read in the paper about plans to give corporate names to parks and community centers what a pathetic council. 
    And thank you for pointing out this: “it's not hard to see how things that matter a little bit to a lot of people will overshadow those that are crucial to a few”
    Basically the purpose of this survey is to say we need Police Fire and Emergency everything else is getting cut and privatized, there's a disgusting plan of slash and burn going on before our eyes, the most pathetic feature about it is that it's no where close to being fiscally responsible as they claim.
    What fiscally responsible person would sell irreplaceable assets for the sake of temporary cash! What fiscally responsible person would entrust the wolf with the chicken as is being done in Garbage collection.
    Pathetic Council!

  • http://www.facebook.com/people/Blake-Bryson/507433470 Blake Bryson

    Thumbs up for Qualtrics surveys.

    • http://twitter.com/CHINDYR Jordan Roach

      The apology to Qualtrics should be at the top, before people read the article and subsequently blame the survey tool for the designer’s shortcomings. Qualtrics is amazing.