How we report the results of survey data is an important and subtle component of how we evaluate program and association success. While lots of different satisfaction scoring scales exist (1 to 5, 1 to 7, or 1 to 10, etc.) most associations and PD providers use one of two methods for interpreting and reporting data: the arithmetic mean (or average score) and the top box score(s) (or percentage of respondents who gave the highest ratings). What you get from each method, and the consequences of that analysis, can be quite different, so choosing the method that best aligns with your survey and organizational objectives deserves some consideration.
Top box scores. Top box scores represent the percentage of respondents who gave the best responses (on a scale of 1 to 10, either a 10, or a 9 or 10). Possible percentage scores range from 1 to 100. Top box scores are easily understandable because they clearly identify how many people fall into a certain category, for example, very happy or happy. Organizations understand the difference between 76% of respondents being very happy or happy, and 69% of respondent being very happy or happy. The downside of top box scores is that they throw some data away – the middle – but also, if the bottom box numbers are not shared, important low box data. High top box scores, by themselves, can eclipse low box scores that may signal a problem. For example, if 20% of respondents were very unhappy, wouldn’t you want to know that?
Average scores. With average scores, the mean is calculated by summing all responses and dividing by the number of responses. On a scale of 1 to 10, possible scores range from 1 to 10. Average scores are easy to calculate, and take into account the full range of responses, from very unhappy to very happy, so provide the best overall statistic of the typical rating (especially valuable for year to year comparisons). On the other hand, average scores provide less understandable data (especially within a single event) ie. what’s the difference between an 8.1 and an 8.5?
So which metric to use? Ultimately, the answer depends on your priority – statistical precision or audience comprehension? A combination of the two metrics is the most honest, understandable and useful approach, often presented as a description of top box scores in the highlights or executive summary, with average scores later on. This blended approach may require more effort but it ensures that the organization and the reader receive all the data, and are therefore better able to act on the responses to make improvements.
Regardless of which interpretation and reporting method you choose one thing is certain: alternating between approaches when it suits your purpose – especially if your goal is validation versus positive change – will raise long-term credibility problems. Objective analysis and reporting of survey results are just as important as proper survey construction.