QPL Reference Manual

The content analysis functions may be used by project administrators to systematically analyze responses made to STING and OPENEND questions. Typically, the Data Administrator will lead the analysis process by defining topic tags and monitoring the inter-rater reliability scores as coding progresses. Two or more Administrators will independently code responses using the tags that the Data Administrator has defined. The Administrators may not change the codes or see how another Administrator has coded particular responses. Generally, more than one pass through the data is required to get sufficiently high inter-rater reliability scores to be confident that the summary of the coded results accurately reflect the responses.

Performing a content using the QPL software is a seven-step process:

1. Estimate Maximum Number of Tags Needed

The maximum number of topic tags that may be used in a specific project is set when you build build your web site files. By default, you can use up to 256 topic tags. If you expect to use more tags in your content analysis, you may increase the maximum number in blocks of 32 to a grand maximum of 3,200 tags when you build your project files.

You should not increase the grand maximum to more than you really need because having many unused tags may adversely affect the performance of your system with no analytical benefit.

While it is possible to increase the maximum number of allowable tags by writing a custom SQL script to alter the database, it is much easier to consider your analysis needs now and build the project with enough room to store the tags you will need.

You may increase the maximum allowable number of tags by using the "/c" converter option as demonstrated below.


HomeSite: ** CONVERT Options:  /d /f /l /u /c 512

 Console: C:\myjob> qpl_convert myfile.qpl /d /f /l /u /c 512 /php

The number of tags specified with the "/c" option must be evenly divisible by 32. (This command tells the qpl_convert.exe program how many 32-bit integers should be allocated to store your maximum number of tags.) In the above examples, the "/c" option is used to allow a maximum number of 512 topic tags. (See Converter Command Line Options for more information.)

2. Deploy Your Project

Now you may follow the normal steps to deploy your project on a web server.

3. Decide Who Will Define the Topic Tags

Ideally, one Data Administrator will be responsible for managing a content analysis project, including defining the topic tags and training the Administrators on how to correctly apply them to responses.

A Data Administrator, however, may choose to use the "Content analysis > Delegate tasks" function to let one of the Administrators enter the tags. The Data Administrator may permanently delegate this task throughout the course of the content analysis process or may use this function again to remove an Administrator's ability to edit topic tags.

This function may also be used to let an Administrator see two of the project management reports that he or she would normally not see: the Reconciliation report and the Summary report. They do not see these reports by default because they show how any Administrator has coded responses to a particular question. Letting Administrators view these reports would compromise the independence of the coding process so granting access is a decision that should be carefully considered.

4. Define the Topic Tags

When the web site has been deployed, the Data Administrator may begin entering topic tags using the "Content Analysis > Define topic tags" function on the main administrative page. Click on the "New" link to add a new tag, or an existing tag link to modify that tag. The Data Administrator may also enter a detailed definition of the tag and specify which questions to which the tag may be applied.

The Data Administrator, of course, should take care when modifying the settings for a tag after the content analysis process has begun. While changes may be made to a tag, please keep in mind that some Administrators may have already applied that tag to one or more responses. The "Update decisions" option on the "Define Topic Tags: Step 2" page can be used to clear a tag that may have been applied to any responses by any Administrator, but be warned, this cannot be undone! Generally, it is much safer to add a new tag rather than attempting to change the meaning of an existing tag. If you must alter an existing tag, you should add a note to the tag definition that clearly documents the change, including when you made the change.

When an Administrator logs into the survey, the Define topic tags function will be renamed "View topic tags," and he or she will only be able to view the list of topic tags and definitions. They will not be able to make any changes in this list.

If you do not already have a set of topics you are interested in studying, there are several ways you can view the responses that have been entered so you can start developing a topic list.

5. Apply Topic Tags

After the tags have been defined and the Administrators have been trained, the Administrators may begin the task of reading each response and selecting the appropriate tags. Administrators may also add their own comments about a particular response, which may be used to document questions about a particular response or to make notes about potential new topic tags. This is done using the "Content analysis > Assign topic tags" function.

The "Assign Tags: Step 1" page lets the Administrator choose which question to analyze. The actual coding is done by looking at all the responses for a particular question.

The "Assign Tags: Step 2" page gives the Administrator various ways to select a batch of responses for analysis. For example, the Administrator may choose to look at only responses that have not been coded, or responses that contain certain key words, or responses that contain certain comments made by the Administrator himself.

The "Assign Tags: Step 3" page shows the list of responses the Administrator selected for analysis and the tags and comments that have already been assigned to each response. The Administrator may open any of the responses to assign tags or change previous tag assignments.

The "Assign Tags: Step 4" page shows one response and the tags and comment that the Administrator has currently applied. The Administrator may page forwards and backwards use the "Next" and "Previous" buttons to navigate through the current list of responses. Any changes to the tags or comments are stored each time the Administrator goes to a new page.

6. Review Inter-Rater Reliability Scores

After the response Administrators have completed a pass through all the responses for a particular question, the Data Administrator should review the "Inter-Rater Reliability Report" to determine how well the Administrators have agreed on the tags that they have applied. Generally, a Cohen's Kappa or a Krippendorff's Alpha Reliability score of 0.7 or better indicate an acceptable degree of reliability. The closer these scores are to 1.0, the better the reliability.

After identifying tags that are not being consistently applied by the Administrators, the Data Administrator's next step should be to use the "Reconciliation Report" to review responses where the Administrators disagreed. After identifying reasons why the Administrators may have assigned tags differently, the Data Administrator may choose to update the definitions of problem tags and then ask the Administrators to make another pass through the responses using the revised definitions (i.e., Step 5).

Steps 5 and 6 should be repeated until acceptable reliability scores have been obtained.

7. Report Results of Content Analysis

After you have achieved acceptable reliability scores, you can summarize the results of this content analysis several ways. All of these let you select which Administrator's tags to use and, if you select multiple Administrators, how to merge tags used to code the same response. Here, you may choose to use the union of responses (i.e., keeping all tags applied by any Administrator) or the intersection of responses (i.e., keeping only tags applied by all Administrators).