Monthly Archives: June 2009

Faulty Feedback Forms

Recently I conducted a training seminar for managers from several organizations on “Managing Customer Satisfaction and Loyalty.” The participants came from SMEs, large businesses and the non-profit sector. I asked the 30 participants if their organization measured customer satisfaction on a regular basis – two-thirds did not! How does one run a successful company without knowing if we are indeed delivering value to our customers? Most of those who did not measure CS were SMEs, which was not surprising.

After the session was over, I started thinking about how well companies measure customer satisfaction. It is one thing to say we do it, and yet another to actually do it. I collected several customer satisfaction/feedback forms. My unscientific sample included: Subway (fast food/quick service), Pizza Delight and Boston Pizza (restaurant), Delta Hotels (hospitality), Dell Canada (computers & technology), Sears Canada, Leon’s and Kent (Retail). For Delta and Dell, I got their online feedback forms.

My analysis led me to conclude that most forms are poorly designed. They either do not ask the right questions or do not measure using the right scales, which would diminish the quality of data obtained. Let me elaborate on this conclusion. Specific problems I identified included the following.

  1. Response Scaling or Categories. Leon’s uses a YES/NO (dichotomous) scale. Often response to questions like “where you satisfied with the service at the front desk” is not black or white. It is useful to know the degree to which customers like or dislike a particular aspect of the service. Subway uses a 3-point scale (excellent, satisfactory and unsatisfactory) and Boston Pizza uses a 4-point scale (excellent, good, fair, needs improvement), which is interesting because I would think that any response that is less than “excellent” is in need of improvement. The response scales convey the impression that for Boston Pizza “fair” is good enough. Kent, a home improvement chain which competes with Home Depot, has 10-point scales, which in my view is better than all of the previous examples. A scale with more response categories provides more variance, and will help in identifying consumers who have varying levels of satisfaction. I consider 2, 3 and 4 point scales to be clearly inadequate.
  2. Scale versus Item. Customer satisfaction is a construct that needs multiple measures. Anyone trained well enough in survey research will tell you that using multiple items leads to more reliable measurement.
  3. Measures. There is a great deal of variance in the actual measures found in these surveys. Some like Dell Canada, Kent and Sears (in the latest version I saw) don’t have an overall satisfaction question or an intent to revisit question. Dell Canada, instead, chooses to focus on four specific aspects of online user experience (interesting Dell.com asks only the “overall satisfaction” question and does not probe on individual drivers of satisfaction). Some firms like Leon’s include loyalty measures and others like Boston Pizza have a “willingness to recommend” question. It just appears that not a lot of thought has gone into the creation of these comment cards. Most of the feedback surveys or cards focus on ratings of individual performance attributes (like knowledge of staff, friendliness of staff, appearance of the place etc.) and don’t have measures which can be used to a create satisfaction/loyalty index. The scales are also not suitable for predictive modeling. I would like to see surveys, even short feedback cards, measure attribute-level satisfaction as well as overall satisfaction and loyalty.
  4. Wording. Some of the customer feedback cards are poorly worded. There are two types of errors:
    • Two-in-one Questions. Delta’s questionnaire consists of several two-in-one questions like “comfort and layout.” Comfort of a room may be determined by several factors with layout being one of them. Even if layout was perfect, overall comfort could have been low due to several other factors.
    • Use of Jargons. Dell uses the term “usability” in their Canadian site. According to Wikipedia:
  5. Usability is a term used to denote the ease with which people can employ a particular tool or other human-made object in order to achieve a particular goal. Usability can also refer to the methods of measuring usability and the study of the principles behind an object’s perceived efficiency or elegance.

    Wouldn’t the term “user-friendliness of the site” be easier for the average consumer to understand than the term “usability”? Opt for language that is easy to understand.

  6. Data Collection. The vast majority of these are self-reporting type of instruments. Leon’s has a shorter version of their form that is implemented via phone (using IVR or interactive voice response survey). Delta Hotels offers both a paper form and an online form. Little thought is given to increasing the response rates for these surveys, which is typically very low. Leon’s again does a nice job offering a chance to win a $50 gift certificate and Kent has a similar offer. In other places, the comment cards are tucked away near the counter or some place and the onus is on the customer to request them. Among the firms with multiple locations, some like Boston Pizza don’t ask for the location of the restaurant. It would be a good idea to ask this question so that different locations can be compared on customer satisfaction. In my recent experience, Leon’s was the only place where I didn’t have to ask for it. They included it with my purchase receipt. Consumers are bombarded with surveys these days. Make it easy for customers to complete the survey and provide an incentive to respond. Without a good response rate month after month, it would not be possible to track key metrics.
  7. The Objective. These comment cards and surveys seem to measure different things. It is unclear how much thought has gone into what these surveys are supposed to accomplish. What analysis will be done with the data? What managerial insights will be generated by such analysis? Thinking about what analysis will be done and what managerial questions will be answered and what gaps will be identified makes it easier to identify the right questions and ask them using appropriate scales. If some really useful insights have to emerge from this exercise, then more thought has to go into the design and implementation of these feedback cards and systems.

In addition to methodology issues, it should be borne in mind that asking the customer for feedback provides the firm two great opportunities:

  • Make Customer Feel Important. By asking for opinion, you are letting your customers know that you care about them. If you can also show how you have used customer feedback in the past to improve your service, it adds to your credibility.
  • Reinforce Brand Identity and Brand Experience. The survey process should be treated as another critical touch point, where the customer comes in contact with the brand. I was surprised to see the varying quality of the comment cards. Sears was the worst. Sears used to have a fairly decent card with English on one side and French on the other side (in Canada), with the Sears logo at the bottom. A couple of weeks ago, I went to Sears and asked for the feedback form. After searching several messy drawers behind the counter, the clerk in the children’s clothing section gave me poorly photocopied form which did not even have the company name or logo! In terms of presentation, Pizza Delight and Kent were very good. Pizza Delight has a neatly folded 2″x4″ card which showed the logo and graphics in colour and they also attached a little ball-point pen to the comment card… a nice touch. Kent has a professional looking card with pre-paid mail and even a signed letter from the general manager, which suggests they take customer feedback seriously. Dell Canada’s online comment card did not have its logo (they may fix this soon), while its US counterpart did. When customers are asked for feedback, that’s a great opportunity to reinforce the brand identity. Clearly, some companies, even big ones, are not capitalizing on this opportunity.

The problem is everyone thinks they can design a survey. After all, what’s so difficult about asking a few questions? As evident from my limited sample, a lot of things can go wrong if the expertise, attention and resources required to do it well are not allocated. The worst thing is that a poorly presented survey or feedback card can present the brand in bad light.

In closing, while I don’t excuse SMEs that do not ask their customers for feedback, it is not surprising that many SMEs are not measuring customer satisfaction. But among the firms that do measure, even many larger firms are executing it poorly both from a methodological and branding standpoint. That’s a shame.

As always, I welcome your comments and feedback.

(This post originally appeared on November 5, 2007)