1 Background on catSurv.com

1.1 What is catSurv.com?

catSurv.com is a webservice, based on the OpenCPU system, that hosts a stable and quick version of our catSurv R package.

1.2 Why host catSurv online?

We host catSurv online so that any web-enabled survey software can query our server in order to administer an adaptive inventory (see our published work using computerized adaptive testing in political science here and here).

1.3 How do you interface with catSurv.com when using survey software?

It depends. In theory, anyone well-versed in web development should be able to fully integrate catSurv operations into the operations of a given survey software. But because most researchers are not trained in web development, we’ve worked out how to interface with catSurv.com when using the Qualtrics survey software.

We provide the basic procedure and all necessary JavaScript needed to fully implement an addaptive inventory within a Qualtrics survey project.

1.4 How can I ask a question about catSurv?

We’ve created a catSurv Google Group forum to ask questions—to us and to the computerized adaptive testing community—relating to all things catSurv and adaptive survey implementation. We encourage you to post questions or problems that arise when implementing your adaptive inventory to the Google Group, but we ask you to please report any catSurv software bugs as a GitHub issue.

2 Interfacing with catSurv.com from Qualtrics

2.1 Getting started with catSurv

First, you’ll need to do some work in R using catSurv to set up your adaptive inventory. Make sure the latest version of catSurv (v1.2.0) is installed and loaded into R.


In what follows, we assume some working knowledge about the catSurv package. See our package documentation and the supplementary materials of our paper for detailed explanations and examples of the catSurv software and its functionality. Here, we’ll simply go over a few important steps necessary to interface with catSurv.com with your own adaptive inventory.

2.2 Example data and Cat object

For this example, we use the 20-item agreeableness battery from the 100-item IPIP representation of Costa and McCrae’s Five Factor Model. We’ve fit a graded response model (GRM) with 774,410 response profiles collected by the myPersonality Project and 1500 response profiles collected by YouGov in July 2018 for the full 20-item battery. See ?grmCat for more information about item parameter estimation. Our goal is to administer a 4-item adaptive inventory.

While we cannot share all of this raw sample data, the estimated item parameters for the model are stored in a Cat object available in the catSurv package. In addition to item parameter estimates, the Cat object stores all research-chosen features of the adaptive algorithm such as the statistical routine for item selection. In sum, the Cat object needs to contain all information governing and used by the adaptive algorithm. We have chosen defaults for all options catSurv makes available to the researcher, but researchers may want to customize their algorithm. See ?'Cat-class' for a detailed description of all of the customizable features stored in the Cat object.

This Cat object can be loaded into your R environment with the data() function. See ?agree_cat for more details about this battery and item parameter estimation.

agree_cat@discrimination # GRM discrimination parameters
##        q86         q6        q66        q46        q36        q26        q56 
##  1.3963428  1.3325832  1.8201505  1.2308572  0.9626278  1.2288147  1.1692755 
##        q76        q13        q96        q82         q9        q22        q32 
##  1.2768165  0.8477620  1.3225958 -0.9240555 -1.6871667 -0.9958325 -1.2405438 
##        q92        q42        q52        q62         q2        q72 
## -1.7112276 -0.9724382 -1.0603658 -1.0066092 -1.0676238 -0.9820362
agree_cat@selection # minimum expected posterior variance item selection
## [1] "EPV"

2.3 Basic procedure from Qualtrics

With Cat object in hand, you now need to set up your Qualtrics survey. The following six steps outline the basic procedure. We discuss each step in detail below.

  1. Hidden data—Add question wordings and response options for the full battery to Qualtrics as “hidden data” in the “header” for the survey. This allows Qualtrics to access this data while hiding it from respondents.
  2. Qualtrics question—Set up, in a very specific way, a Qualtrics question that includes code we have written to interface cleanly with the Qualtrics survey software.
  3. Cat object—Add the actual Cat object into the Qualtrics survey as “embedded data.”
  4. Test your adaptive battery—Testing should include many trial answer profiles, ensure the survey flow appears correctly to respondents, and downloading the resulting data to ensure the answer profiles are retrievable.
  5. Run the survey!—If tests come back clean, administer the Qualtrics survey.
  6. Obtain results—Use the readQualtrics function in catSurv to clean and obtain the answer profiles.

2.4 Battery items as hidden data

The first step is to set up “hidden data” in Qualtrics which holds all question wordings and response options for your full battery. To do so, first navigate to your institution’s or your personal instance of the Qualtrics software. You can add an adaptive inventory to either a new survey or an existing project. We recommend starting with a new survey, ensuring the adaptive inventory works smoothly, and then building additional survey features around the adaptive battery. We create a new survey called “Agreeableness CAT” for this example. To set up your battery items as hidden data, you’ll need to define a custom header. To do so, click the “Look and Feel” button which is located on Qualtrics webpage toolbar when you are on the “Survey” tab.

A new window should appear. Within it, navigate to the “General” tab. Click the “Edit” button just below the “Header” label.