I'm currently attending the annual conference of the American Educational Research Association (AERA) in Chicago. One of my presentations was about "understanding university faculty perceptions about innovation in teaching and technology." A tall order, to say the least. My research colleagues (TJ Kopcha and Brandy Walker) and I used something called the Q methodology for the research project. The interesting part of this methodology is the use of an activity called a Q Sort to collect data. The cool thing about this activity is that people seem to find it an intriguing and fun thing to do. It is not a game, but it has some game-like qualities due to some of its intrinsic challenges. I have built an initial prototype of the Q Sort activity using LiveCode. Before I show that, let me provide some brief background about the Q Sort activity itself.
To explain how a Q Sort works, let's use an example that is near and dear to my heart - pizza. So, consider for a moment how much you like pizza. Think about how important certain kinds of toppings are to you. Are you a meat lover, a vegetarian, or are you someone who just wants as many toppings that will fit on top? Consider the cost of pizza. Are you a gourmet who doesn't care about cost, or someone on a budget who is always looking for those "2 for 1" deals.
OK, now imagine being asked to complete a survey about your pizza preferences. It's easy to imagine a typical survey that asks you to rate the importance of statements like "The crust is thick and chewy" using a scale from 1 (least important) to 5 (most important). Now, if you are a pizza lover, you might not think too much about the question and just answer "4" or "5" for any particular question.
In contrast, a Q Sort uses a special "game board" that resembles something like this:
All of the survey questions are then presented on separate cards, such as these:
The challenge in a Q Sort is that you have to move each statement card to one and only one slot on the Q Sort board. So, you are forced to choose which items are truly the most and least important to you, at least in comparison to the other items. That can be difficult, but in an intriguing sort of way. When you are done, you have a measure of your subjective point of view of pizza, at least as represented by the arrangement of these particular statements.
By the way, it is not a coincidence that the Q Sort board kind of looks like an inverted normal curve, that is, the classic bell curve (but upside down). This is actually an important part of the theory behind the Q Methodology, but we won't get into that here.
The Q Sort activity we used in our research had 33 statements, a rather typical number for a Q Sort, so it was much more involved and difficult for the participants to complete. Yet, they all seemed to enjoy the activity, with a few people reporting really liking it.
The biggest disadvantage of the Q Sort activity is that it is a bear to administer and score. You have to prepare a bunch of Q sorting boards and sets of cards. Participants have to accurately write down on their sorting board which statement is where before they hand it in. And then, you have the tedious task ahead of you of manually entering the data into a spreadsheet or statistical software package. This just cries out for an electronic version.
Yes, it would be great to have an electronic version of the Q Sort activity. Several are available, the most notable is Q-Assessor. This looks like an excellent tool, at least as shown in a YouTube video demonstration of it. However, it is costly. They charge researchers $500 a month for a subscription to use it. We have not found any suitable free or low cost versions to use. Producing a fully supported electronic version is challenging because the limited amount of space available on a computer screen makes it very difficult to design the activity appropriately. Indeed, when completing the paper version participants typically need a large table surface available for laying out and sorting their statements. There is the possibility to redesign the activity so as to promote a multi-step Q sort procedure, which could easily allow 30-50 statements. Q-Assessor uses a two-step sorting procedure and it looks very effective. However, their design also requires that not all statements are fully visible to participants at the same time. They report that their design approach has been fully validated, but I think this remains an issue. I have certainly not solved this problem, but I recently had a design breakthrough that has helped tremendously. More about that shortly.
First, check out this brief video demonstration of the current prototype:
You will obviously notice in the video how different the Q Sort board is in my prototype from the example illustrated above. Here's a partial screen shot for those who did not watch the video:
This change in the visual layout of the board was my design breakthrough. At first, I had it stuck in my head that the gameboard had to be exactly like that used in the paper versions. Then, I was introducing the survey application Qualtrics to my doctoral students. If you have never used or seen Qualtrics, suffice it say that there are some very creative question types available. One of question types allows the user to sort a list of items using a drag and drop interface. This triggered the insight that I could use a more linear visual layout using indentations to represent the different rating levels. This visual design allows for a much more compact visual arrangement of the game board. One reason is that the game board slots become "docking locations" for the statements. This means that the slots don't have to be the same size as the statements. I was also able to use LiveCode to dynamically resize the statements based on their lengths. Doing so opens up yet more screen space. My early try-outs of this design convince me that this is the way to go for an electronic version.
But before I congratulate myself too much, let me quickly also point out that my prototype only currently allows for about 20 total statements, which is far below what would be needed for serious research using the Q methodology. Yet, I think I'm on the right track and may be able to squeeze out more space to allow for a few more statements. I may also try the multi-step approach in later versions, similar to that used in Q-Assessor.
As shown in the video, I designed the prototype in a very special way so as to facilitate the dynamic design and delivery of the Q Sort activity. I'll explain the details of this in a subsequent post. Suffice it to say that this approach allows for an unlimited number of Q Sorts to be designed using the same client software. For example, although designed as a research tool, I see some tremendous potential of the Q Sort activity as a teaching tool. A teacher could use the Q Sort activity throughout a course to engage students and provoke discussion and reflection. The topic of the Q Sort can be changed as often as one wishes just by changing the contents of the download URL entered into the software. To realize this potential, I still need to figure out an easy way to score the activity so as to make the results available more or less right away. I think there are some ways to do that without doing a full Q analysis. I'll be working on that.
Yes, I am quite smitten with the Q Sort activity. One thing I like about it is that it promotes "mindfulness" in completing the activity. That is, you really have to think about what you value as you complete it. You can't just quickly click a bunch of 4s or 5s on a survey. You are fully engaged in activity as you complete it. The other thing, and this is not trivial, is again the fact that people seem to really enjoy doing it. I know I'm repeating myself, but I think it is because it has a game-like feeling about it. Also, I think the engaging, mindful nature of it is just naturally motivating. I also find the instructional potential of Q Sorts very, very exciting. I think a Q sort would be an excellent classroom activity leading to motivating and reflective discussions.
There is much more to explain and talk about here, but I hope this gives you a good sense of what a Q Sort is and the advantages of creating an electronic version. This project really needs a web-only delivered solution, so this will be a great candidate for exporting LiveCode to HTML5 when that feature becomes available in the (hopefully) not too distant future. On that note, I send my best wishes and vibes to the LiveCode team currently working on this.
What is a Q Sort?
To explain how a Q Sort works, let's use an example that is near and dear to my heart - pizza. So, consider for a moment how much you like pizza. Think about how important certain kinds of toppings are to you. Are you a meat lover, a vegetarian, or are you someone who just wants as many toppings that will fit on top? Consider the cost of pizza. Are you a gourmet who doesn't care about cost, or someone on a budget who is always looking for those "2 for 1" deals.
OK, now imagine being asked to complete a survey about your pizza preferences. It's easy to imagine a typical survey that asks you to rate the importance of statements like "The crust is thick and chewy" using a scale from 1 (least important) to 5 (most important). Now, if you are a pizza lover, you might not think too much about the question and just answer "4" or "5" for any particular question.
In contrast, a Q Sort uses a special "game board" that resembles something like this:
All of the survey questions are then presented on separate cards, such as these:
The challenge in a Q Sort is that you have to move each statement card to one and only one slot on the Q Sort board. So, you are forced to choose which items are truly the most and least important to you, at least in comparison to the other items. That can be difficult, but in an intriguing sort of way. When you are done, you have a measure of your subjective point of view of pizza, at least as represented by the arrangement of these particular statements.
By the way, it is not a coincidence that the Q Sort board kind of looks like an inverted normal curve, that is, the classic bell curve (but upside down). This is actually an important part of the theory behind the Q Methodology, but we won't get into that here.
The Q Sort activity we used in our research had 33 statements, a rather typical number for a Q Sort, so it was much more involved and difficult for the participants to complete. Yet, they all seemed to enjoy the activity, with a few people reporting really liking it.
The Downside to Q Sorts
The biggest disadvantage of the Q Sort activity is that it is a bear to administer and score. You have to prepare a bunch of Q sorting boards and sets of cards. Participants have to accurately write down on their sorting board which statement is where before they hand it in. And then, you have the tedious task ahead of you of manually entering the data into a spreadsheet or statistical software package. This just cries out for an electronic version.
Creating a Q Sort Activity with LiveCode
Yes, it would be great to have an electronic version of the Q Sort activity. Several are available, the most notable is Q-Assessor. This looks like an excellent tool, at least as shown in a YouTube video demonstration of it. However, it is costly. They charge researchers $500 a month for a subscription to use it. We have not found any suitable free or low cost versions to use. Producing a fully supported electronic version is challenging because the limited amount of space available on a computer screen makes it very difficult to design the activity appropriately. Indeed, when completing the paper version participants typically need a large table surface available for laying out and sorting their statements. There is the possibility to redesign the activity so as to promote a multi-step Q sort procedure, which could easily allow 30-50 statements. Q-Assessor uses a two-step sorting procedure and it looks very effective. However, their design also requires that not all statements are fully visible to participants at the same time. They report that their design approach has been fully validated, but I think this remains an issue. I have certainly not solved this problem, but I recently had a design breakthrough that has helped tremendously. More about that shortly.
First, check out this brief video demonstration of the current prototype:
My Design Breakthrough
You will obviously notice in the video how different the Q Sort board is in my prototype from the example illustrated above. Here's a partial screen shot for those who did not watch the video:
This change in the visual layout of the board was my design breakthrough. At first, I had it stuck in my head that the gameboard had to be exactly like that used in the paper versions. Then, I was introducing the survey application Qualtrics to my doctoral students. If you have never used or seen Qualtrics, suffice it say that there are some very creative question types available. One of question types allows the user to sort a list of items using a drag and drop interface. This triggered the insight that I could use a more linear visual layout using indentations to represent the different rating levels. This visual design allows for a much more compact visual arrangement of the game board. One reason is that the game board slots become "docking locations" for the statements. This means that the slots don't have to be the same size as the statements. I was also able to use LiveCode to dynamically resize the statements based on their lengths. Doing so opens up yet more screen space. My early try-outs of this design convince me that this is the way to go for an electronic version.
But before I congratulate myself too much, let me quickly also point out that my prototype only currently allows for about 20 total statements, which is far below what would be needed for serious research using the Q methodology. Yet, I think I'm on the right track and may be able to squeeze out more space to allow for a few more statements. I may also try the multi-step approach in later versions, similar to that used in Q-Assessor.
Using a Web-Based Approach for Populating the Q Sort Activity and Collecting Data
As shown in the video, I designed the prototype in a very special way so as to facilitate the dynamic design and delivery of the Q Sort activity. I'll explain the details of this in a subsequent post. Suffice it to say that this approach allows for an unlimited number of Q Sorts to be designed using the same client software. For example, although designed as a research tool, I see some tremendous potential of the Q Sort activity as a teaching tool. A teacher could use the Q Sort activity throughout a course to engage students and provoke discussion and reflection. The topic of the Q Sort can be changed as often as one wishes just by changing the contents of the download URL entered into the software. To realize this potential, I still need to figure out an easy way to score the activity so as to make the results available more or less right away. I think there are some ways to do that without doing a full Q analysis. I'll be working on that.
Closing Thoughts
Yes, I am quite smitten with the Q Sort activity. One thing I like about it is that it promotes "mindfulness" in completing the activity. That is, you really have to think about what you value as you complete it. You can't just quickly click a bunch of 4s or 5s on a survey. You are fully engaged in activity as you complete it. The other thing, and this is not trivial, is again the fact that people seem to really enjoy doing it. I know I'm repeating myself, but I think it is because it has a game-like feeling about it. Also, I think the engaging, mindful nature of it is just naturally motivating. I also find the instructional potential of Q Sorts very, very exciting. I think a Q sort would be an excellent classroom activity leading to motivating and reflective discussions.
There is much more to explain and talk about here, but I hope this gives you a good sense of what a Q Sort is and the advantages of creating an electronic version. This project really needs a web-only delivered solution, so this will be a great candidate for exporting LiveCode to HTML5 when that feature becomes available in the (hopefully) not too distant future. On that note, I send my best wishes and vibes to the LiveCode team currently working on this.
Good luck. Post your progress please!
ReplyDeleteHi Dr. Rieber, is this LiveQ-Sort technology complete and accessible for purchase or use for educational research? Thank you, Cailin
ReplyDeleteHi Cailin. Actually, I'm happy to report it is. I'm finishing up a blog post to make the announcement. But, you can go to http://nowhereroad.com/qsort and submit a request for an instructor account.
ReplyDelete