Creating a Custom Tool to Analyze Q Sort Data

December is finally here. It's been a wonderful semester at the University of Georgia. They seem to get better the longer I'm here. Although I haven't written many posts lately, I have been doing quite a bit of work with LiveCode. In particular, I've done much more work on my Q sort tool. In case anyone wants some background, here are links to my previous three posts about my Q sort tool:

1. Creating a Q Sort with LiveCode
2. Lloyd's Q Sort Project: Importing Data from an Internet File
3. Latest on My Q Sort Prototype: Enhancing the User Experience and Inventing an Instructional Strategy

In short, a Q sort is a quantitative way to measure subjectivity. I know, that sounds completely contradictory. Think of it as a ranking procedure with a few twists. It's a procedure that's been around since the 1930s. One of my recent accomplishments is creating a much improved Q sort tool. The design improvements have been significant enough to warrant a full step in the version number - I'm now up to version 3.1. However, this post is not about that. Instead, it's about creating a completely new tool that focuses on analyzing the data that results from the Q sort activity.

As I've explained in previous posts, I've been trying to come up with an instructional strategy using a Q sort as the main class activity. In my early field trials, I really struggled to provide the class participants with the sorting results as quickly as possible. The reason is that I feel that the Q sort activity promotes a very active, "minds on" experience for participants. In order to take full advantage of their thinking and engagement, I need to compile, analyze, and report to the class participants the results of the Q sort activity as soon after they complete it as possible. In this post, I'll explain why that was such a challenge and what I've built to make this task much, much easier.

The Q Sort Raw Data

When a person completes a Q sort activity, a data string is created and uploaded to a text file stored on the Internet. For example, here is a sample based on a recent Q sort activity completed by an undergraduate class on the topic of favorite vacation destinations using the prompt "Sort these vacation destinations from most to least favorite." Here is the list of vacation destinations they sorted:

Aspen, Colorado
Branson, Missouri
Cancun, Mexico
Hawaii
Home
London
Miami
Myrtle Beach
New York City
Orlando
Paris
Pittsburgh
Rome
San Francisco

I found a web site that listed 10 of the top vacation spots in the United States and I added the rest. (Yes, adding Pittsburgh was rather mischevious of me. But hey, I think it's a fantastic place to spend one's vacation. I also added "home," which I thought was rather insightful.) A lot of the students never heard of Branson, Missouri, which might be a good thing.

Each line of data is an individual's data separated by commas. Here are a few lines of the data to illustrate:

 vacation-abd,Tue, 1 Dec 2015 12:56:26 -0500,Favorite Vacation Destinations?,PERSON1,0,-2,+1,+3,0,+1,0,-1,-1,-3,+2,-1,+1,0,Summary Statement Results,2,Time (seconds),59  
 vacation-abd,Tue, 1 Dec 2015 12:56:31 -0500,Favorite Vacation Destinations?,PERSON2,+1,-1,0,+3,-3,+1,0,-1,0,-2,0,-1,+2,+1,Summary Statement Results,+3,Time (seconds),78  
 vacation-abd,Tue, 1 Dec 2015 12:57:51 -0500,Favorite Vacation Destinations?,PERSON3,+3,-3,+1,+2,0,+1,+1,-1,-1,-1,0,-2,0,0,Summary Statement Results,+3,Time (seconds),131  
 vacation-abd,Tue, 1 Dec 2015 12:57:51 -0500,Favorite Vacation Destinations?,PERSON4,0,-3,-1,+1,0,+3,0,-1,0,+1,+2,-2,+1,-1,Summary Statement Results,+3,Time (seconds),159  

Each line starts with the unique Q sort code, then a date/time stamp, then the name of the Q sort, then the name of the person (I obviously substituted PERSON for each name). This is all followed by the sorting data where each statement's rating is provided. I also ask participants to rate a "summary statement," something I explain briefly below, so those results are next. Finally, I collected the time it took the person to complete the Q sort (in seconds).

OK, great, I have collected data. How does one quickly make sense of these data quickly and accurately in order to trigger some class discussion?

My First Idea: Use Excel to Analyze the Data

First, it is important to recognize that the analysis I wanted to perform was very simple in comparison to a true Q sort analysis. An actual Q sort analysis is best thought of as an upside-down factor analysis, meaning that instead of reducing the number of measures (i.e. statements) down to a smaller number of common factors, the idea is to reduce the number of people into a smaller number of profiles comprising those people. This type of analysis can take weeks and requires a sophisticated statistical package to pull it off. No, what I wanted was a straightforward analysis that I could do quickly, with results that the students and I could understand and react to. The goal of the analysis is just to trigger discussion and more reflection. That is, this is a learning goal, not a research goal. So far, I've settled on computing the sum and standard deviation for each statement based on all of the participant scores. The sum gives an overall sense of the importance of each statement for this group of people in comparison to the others. The standard deviation gives a quick sense of how much the group "agrees" with the ranking of that statement.

My basic plan is for participants to have a short small-group discussion in class - about 10 minutes - about their Q sort results immediately after completing it while I quickly analyze the data and prepare some slides of the overall results. That's a tall order in 10 minutes. And yes, my first idea was to use Excel to analyze the data. This is actually a good idea, if there was more time. I was able to successfully create an excel file on the fly that would analyze the data, but it was a challenge to do so quickly without making any mistakes. It's hard to explain what it feels like to be in the "heat of the moment" when teaching, but it can be stressful to try to focus on a task such as this requiring attention to detail. Such an Excel file begins with the following:

It's easy to get the data into neat and tidy columns using the "Text to Columns" option in Excel. This option allows you to split a line of data into columns based on some delimiter, such as the handy dandy comma. As you can see, I deleted a bunch of the columns to just focus on the people and their statement ratings. I summed the results for each statement. I also computed the standard deviation for each statement (again, as a rough measure of "agreement"). This is all well and good, but I really need the data in this form:

Fortunately, Excel has a transpose option. You first copy the data cells, then you choose "Paste special." Transpose will be one of the options. I then added a column and pasted in the statement labels. Then I sorted the rows in order of sum (largest to smallest). I did the same thing for the standard deviation, though I sorted that data from smallest to largest so that the statements the participants were "most agreement with" are at the top.

So, sure, I could do this in Excel, but allow me to repeat: Doing this in about 10 minutes without making a mistake is quite a challenge. And, if I wanted to do several Q sorts in a single class session, I would really be struggling. I'm good, but I'm not that good. I played around with the idea of creating an Excel template to facilitate the process, but that didn't work so well. What I wanted was a one-click solution.

Creating a One-Click Q Sort Analysis Tool with LiveCode

To meet these challenges, I created yet another LiveCode project that takes the Q sort data and produces the results quickly as described above. Here's the main screen (which uses all of the undergraduate's responses):

All I need to do is paste in the raw data into the large field on the left and the statements on the right. The bottom two left fields give me to double-check the data to be sure everything is ready for the analysis. For example, if all is good, a person's ratings will sum to 0. (I have noticed that data is sometimes missing, such as one of a person's statement ratings. Other quirky errors in the data have happened too, such as a person's data being duplicated. I'm not really sure why this is happening. Fortunately, it's a relatively rare occurrence, and I can usually make the needed corrections. Still, it's a cause of some concern.) I also drew that long arrow from the right side of the screen to the left to remind me to check that the raw data is being parsed correctly. I built in an auto-refresh feature so that after the mouse leaves each field after pasting, the key data fields update automatically.

If all is good, I simply click the red button "Analyze" and voilà! Here is the output:

The data are all nicely computed and sorted just the way I want.

If I now click on the button "Copy All to Clipboard" I can then paste the data into Excel and use the "Text to Columns" option to produce the following formatted spreadsheet:

This is more than adequate, but a few more edits within Excel will produce the following output:

I usually just project this final spreadsheet on the large screen for students to review. But, I could easily copy and paste this into a PowerPoint slide as well.

The main point to all of this is that I can go from raw data to PowerPoint in about 2 minutes without needing any intense concentration to do so. While this isn't exactly a one-click solution, it's pretty close!

I should explain the "Summary Statement Average" of 2.41. After participants complete the Q sort, I ask them to rate one more question. In this case, the statement was "It is important to go on a real vacation at least once a year." I used the same rating scale as that used in the Q sort, which in this case ranged from -3 to +3 (a seven point scale). A summary rating of 2.41 clearly indicates that this group believes in the idea of yearly vacations. This summary statement gives me a Likert-like item that I'm considering using as a kind of weight to modify the raw data. These results are shown above under the orange headings ("adjusted"). I have some research questions related to this. Perhaps I'll explain more about that later.

Are You Like Me?

You might have noticed another button just below the Analyze button titled "Are You Like Me?" This is rather interesting and I will explain it further in a follow-up blog posting. Basically, this button will produce an analysis where each person is compared to the rest of the group. I think this has much potential for both instruction and research. It leads to some very interesting results that really seem to grab the participants' attention and interest.

Final Thoughts

I really wasn't wanting to create this Q Sort Analysis Tool, but it was needed and I'm obviously now glad it's done. It makes the analysis process extremely easy for me. And, even though I compute some simple statistics (i.e. sum and standard deviation), it would be very easy to add other statistics as well, should they prove necessary.

So, how did these undergraduates sort this list of favorite vacation destinations? As you probably already noted, Hawaii topped the list with very little disagreement among the group. Somewhat surprisingly, London was next (I would have bet on Paris). I find it very interesting that "Home" garnered the most disagreement and this result could be used to initiate some interesting discussion. Alas, Pittsburgh was second to last and there seemed to be little disagreement about this among the students. I'll feel sorry for them the next time I'm enjoying a delicious fish sandwich with pierogies on the side at Cupka's Cafe on Pittsburgh's southside. But, Pittsburgh did rank higher than Branson, Missouri. Perhaps "Thank God for Branson, MO!" should become Pittsburgh's new motto.