Ranking Questions in Online Surveys

Ranking and rating questions are used for a wide range of survey objectives from customer satisfaction, to product testing, and opinion surveys. Rating questions seek to record a measure against a predefined scale (e.g. verbal or numerical scale). They could be used to elicit brand performance, customer satisfaction, or an importance rating to reflect how a product or service is evaluated.  By contrast, a ranking question asks the respondent to compare all attributes together and place them in order, to reflect their considered priorities of the question.
Advantages and Disadvantages of Ranking Questions
Arguably, the main advantage of using a ranking question is that it forces the respondent to compare and make a clear choice between the attributes presented. This can be particularly useful for when it is necessary to prioritise issues (e.g. new product features on a software product most required, or in polling surveys where the researcher may be hoping for some distinctive results). However, by the same token, it can be argued that in circumstances where a respondent would more realistically place a multiple of attributes at a similar level of importance, the ranking question is less useful (e.g. a ranking question provides a unique rank for each attribute, but does not allow a shared ranking of more than one attribute).
If there are only a few attributes in the ranking question (e.g. up to 5), they are relatively fast and easy to complete but beyond 8 – 10 attributes, they can become complicated and time consuming for the survey participant. 
Its important to ensure the list of attributes are randomized, and that the question makes clear that a ranking of 1 is the highest rank.
Some surveys present ranking questions as a grid, with the ranking order across the top (columns) and the attributes in the rows. This approach can get very complicated for the respondent when tying to rank the attributes. A much better question type is the drag and drop ranking question style. The respondent is asked to drag the attributes into a box, and then can, as necessary, reorder each attribute by moving it up or down. Its visually more interesting, and encourages the respondent to consider very carefully each answer.  For this reason, arguably it delivers better quality data than using grid format ranking questions.
Stacks Image 15028
How to analyse ranking data
While the idea of using a ranking question seems simple enough, it is not unusual for people new to using this question type to be unsure of how to analyse and interpret the data. There is no single best approach to analyse ranking data. However, here are a few pointers. In our experience the first few ranking positions, as well as the lowest ranking positions are of interest to the analyst. It is a good idea to arrange your data to show the % scores for all the ranking results, side by side.

As you can see in this example, we are now showing all 9 ranking positions for all the attributes in the question. From Surveygoo data tables, the % scores for each rank were pasted into a new spreadsheet for ease of comparison.
Stacks Image 15033
What we don’t do is to add up the % in each row. This would be misleading. However, we have added some colour codes for each rank: yellow for relatively lower scores and red for relatively higher scores.
Immediately we can see in this example that “Managing a team working on the survey”, followed closely by “Deciding when to use internal resource or use outside consultants” are ranked higher. A higher % of respondents identified these as rank 1, compared to other attributes. “Evaluating survey software” had the least highest score in rank 1 position, and also in rank 2 and 3 positions. 
By taking the same approach for rank 2 and 3 positions, overall we see that attributes in the top three include: 
• Managing a Team working on the survey
• Deciding when to use internal resource or use outside consultant
• Planning Research Objectives
• Planning Research Design
• Questionnaire Design
At the other end of the spectrum (ranks 7,8 and 9) two attributes receive relatively higher scores, and therefore, are seen to be ranked lower:
• Evaluating Survey software
• Scripting online surveys via a Survey tool
The middle ranking attributes (ranks 4,5 and 6) include:
• Managing a Team working on the survey
• Planning research design
• Analysis and reporting
• Scripting online surveys via a Survey tool
Now we have a summary of the overall and relative rankings of the attributes. Ranking questions often do not generate as clear cut results as some would like. For example, “Managing a team working on the survey” features as both higher and middle ranking attributes. 
The fact that ranking questions can produce more nuanced results means that it takes more time to analyse the data. But with a bit of time, it is possible to draw out some clear patterns in the data. Its clear in this example, that in relative terms issues around planning and managing are generally ranked higher; evaluation of software and using software are lower ranking attributes, while analysis and reporting are middle ranking. Its not as obvious which % to highlight, but the patterns in the data are clear enough.

Surveygoo Books Available on Amazon and iBooks Store

Stacks Image 31423
Visit Amazon to buy | More info on Book
Stacks Image 31428
Visit iBooks Store to download Free

Start Using Our Survey Tool Now!

See Price Plans