Welcome!
1 Background on catSurv.com
1.1 What is catSurv.com?
catSurv.com is a webservice, based
on the OpenCPU system, that hosts
a stable and quick version of our catSurv R package.
1.2 Why host
catSurv online?
We host catSurv online so that any web-enabled survey
software can query our server in order to administer an
adaptive inventory (see our published work using computerized
adaptive testing in political science here and here).
1.3 How do you interface with catSurv.com when using survey software?
It depends. In theory, anyone well-versed in web development should
be able to fully integrate catSurv operations into the
operations of a given survey software. But because most researchers are
not trained in web development, we’ve worked out how to interface with
catSurv.com when using the Qualtrics
survey software.
We provide the basic procedure and all necessary JavaScript needed to fully implement an addaptive inventory within a Qualtrics survey project.
1.4 How can I ask a
question about catSurv?
We’ve created a catSurv Google
Group forum to ask questions—to us and to the computerized adaptive
testing community—relating to all things catSurv and
adaptive survey implementation. We encourage you to post questions or
problems that arise when implementing your adaptive inventory to the
Google Group, but we ask you to please report any catSurv
software bugs as a GitHub issue.
2 Interfacing with catSurv.com from Qualtrics
2.1 Getting started with
catSurv
First, you’ll need to do some work in R using catSurv to
set up your adaptive inventory. Make sure the latest version of
catSurv (v1.2.0) is installed and loaded into R.
install.packages("catSurv")library("catSurv")In what follows, we assume some working knowledge about the
catSurv package. See our package documentation
and the supplementary
materials of our paper for detailed
explanations and examples of the catSurv software and its
functionality. Here, we’ll simply go over a few important steps
necessary to interface with catSurv.com with your own adaptive
inventory.
2.2 Example data and
Cat object
For this example, we use the 20-item agreeableness battery from the
100-item IPIP representation of Costa and McCrae’s Five Factor Model.
We’ve fit a graded response model (GRM) with 774,410 response profiles
collected by the myPersonality Project and 1500 response profiles
collected by YouGov in July 2018 for the full 20-item battery.
See ?grmCat for more information about item parameter
estimation. Our goal is to administer a 4-item adaptive inventory.
While we cannot share all of this raw sample data, the estimated item
parameters for the model are stored in a Cat object
available in the catSurv package. In addition to item
parameter estimates, the Cat object stores all
research-chosen features of the adaptive algorithm such as the
statistical routine for item selection. In sum, the Cat
object needs to contain all information governing and used by the
adaptive algorithm. We have chosen defaults for all options
catSurv makes available to the researcher, but researchers
may want to customize their algorithm. See ?'Cat-class' for
a detailed description of all of the customizable features stored in the
Cat object.
This Cat object can be loaded into your R environment
with the data() function. See ?agree_cat for
more details about this battery and item parameter estimation.
data(agree_cat)
agree_cat@discrimination # GRM discrimination parameters## q86 q6 q66 q46 q36 q26 q56
## 1.3963428 1.3325832 1.8201505 1.2308572 0.9626278 1.2288147 1.1692755
## q76 q13 q96 q82 q9 q22 q32
## 1.2768165 0.8477620 1.3225958 -0.9240555 -1.6871667 -0.9958325 -1.2405438
## q92 q42 q52 q62 q2 q72
## -1.7112276 -0.9724382 -1.0603658 -1.0066092 -1.0676238 -0.9820362agree_cat@selection # minimum expected posterior variance item selection## [1] "EPV"2.3 Basic procedure from Qualtrics
With Cat object in hand, you now need to set up your
Qualtrics survey. The following six steps outline the basic procedure.
We discuss each step in detail below.
- Hidden data—Add question wordings and response options for the full battery to Qualtrics as “hidden data” in the “header” for the survey. This allows Qualtrics to access this data while hiding it from respondents.
- Qualtrics question—Set up, in a very specific way, a Qualtrics question that includes code we have written to interface cleanly with the Qualtrics survey software.
- Cat object—Add the actual
Catobject into the Qualtrics survey as “embedded data.” - Test your adaptive battery—Testing should include many trial answer profiles, ensure the survey flow appears correctly to respondents, and downloading the resulting data to ensure the answer profiles are retrievable.
- Run the survey!—If tests come back clean, administer the Qualtrics survey.
- Obtain results—Use the
readQualtricsfunction incatSurvto clean and obtain the answer profiles.
2.5 Creating the Adaptive Question
The first step is done! Now, you’ll need to create a Qualtrics question. All items administered from an adaptive battery via Qualtrics will utilize the same Qualtrics question. Since you just specified the question and response option wordings elsewhere as hidden data, a single Qualtrics question will be modified dynamically using that information as the participant takes the survey. Esentially, we’ll creating the structure of the questions to be asked, and the appropriate question wording and response options will be populated in real-time.
This question needs to be a multiple choice question with the number
of options equal to the maximum number of response options
possible across all items in your battery. It does not matter if the
options are vertically or horizontally oriented, but vertical listings
are more accessible on mobile devices. We suggest editing the question
text and response options with default language that will be visible to
respondent’s only while catSurv is loading.
The goal is to have this question repeat and adaptively populate new questions from your battery. To do so, we’ll configure “Loop & Merge”. This feature allows a Qualtrics question to repeat multiple times. Under the “Block Options” menu, which can be found in the top-right corner of the block in which your question resides, select “Loop & Merge” in the third demarcated section of the menu.
Turn Loop & Merge on by clicking the blue button.
You’ll notice that turning on Loop & Merge reveals a grid with
multiple rows and columns. You’ll need to edit this grid for your
adaptive inventory. Importantly, implementing an adaptive battery on
Qualtrics requires the researcher to use a survey length stopping
rule. In this example, all respondents will answer four items from
the agreeableness battery. This length stopping rule is indicated by the
number of rows in this grid. Add or remove rows until the number of rows
equals the length stopping rule for the adaptive battery as indicated in
the lengthThreshold slot of the Cat
object.
To add rows, just click once in the last row’s box. To remove a row, click the “-” button next to it. In our example, we must remove one row by clicking the “-” symbol next to row five. Then, remove all columns but the first using the red “-” button. Click “Save” to exit.
As we mentioned, this Qualtrics question needs to be set up in a very
specific manner. Next, we’ll use custom JavaScript to enable this
question to interface with catSurv via
catSurv.com. First we need to navigate to the question’s
JavaScript editor. To do so, click the gear symbol beside the
question.
You should then see a menu of options. Click the “Add JavaScript” option.
The JavaScript editor should open as a new window. There will already be some JavaScript in the editor. Delete all of it, and replace it with the code we’ve provided here. Click “Save” and exit the editor.
2.6 Cat
object as embedded data
Now that the adaptive question is configured, the final step before
testing your survey is to set up our embedded data objects. Embedded
data enables survey developers to store data that persists between
questions. We use embedded data to we store the Cat object
that updates with the respondent’s answers as they take the adaptive
battery. Click “Survey Flow” in the survey toolbar.
Find any item in the flow and click “Add a New Element Here” or “Add Below”.
Next we’ll add three pieced of embedded data. In the new survey flow element, click “Embedded Data”.
Then, add the following field-value pairs:
| Field | Value |
|---|---|
next_item |
-1 |
last_item |
-1 |
last_answer |
-1 |
To do so, click “Add a New Field” to type in a field name, then click “Set a Value Now” to set the value as -1.
In addition to these three entries, we also store the
Cat object as embedded data. After creating the
Cat object, convert it to a JSON representation using the
toJSONCat() function. As mentioned above, specify all slots
in the Cat object that inform the adaptive algorithm (e.g.,
the @selection slot indicates the adaptive inventory’s item
selection routine) before converting it to a JSON string. Double check
that the lengthThreshold slot is specified with the desired
length stopping rule (here, 4) and that the @answers slot
has only null values.
setLengthThreshold(agree_cat) <- 4toJSONCat(agree_cat)Then, create a new embedded data object. The field name should be
catObj, and the associated value is the JSON string
returned from the toJSONCat() function.
Before leaving the survey flow, ensure that the adaptive inventory block (called “Default Question Block” in this example) appears after all of these embedded data objects. We recommend moving the embedded data item so it appears first in the survey flow. You can do this by clicking “Move” in any survey flow item.
Lastly, make sure at least one Qualtrics survey block follows the adaptive inventory question block. The embedded data will not reflect the respondent’s last answer if the survey exits from the adaptive inventory question block. We include a final message in this example, but you could also include additional survey items.
Now, the adaptive inventory is ready! Test your Qualtrics survey from start to finish (do not preview only the adaptive question) to make sure the adaptive battery is incorporated correctly. We highly recommend taking the survey a multiple times and downloading the data from Qualtrics (which we will discuss next) before building any additional question items or features into your survey.
2.7 Retrieving the data
Finally, you’ll need to make sense of the respondent’s answers to the
adaptive inventory. Because different respondents receive different
adaptive inventories, their answers to the battery are not stored as
usual as separate columns in a dataset. Rather, the respondents’ answers
to the adaptive battery are saved as a column of Cat
embedded data objects. We’ve built functionality into
catSurv to transform the data into a format useful for
subsequent analysis.
First, navigate to the “Data & Analysis” tab.
Once there, click “Export & Import” then “Export Data.” We recommend downloading the data as a .csv file.
Then, use the readQualtrics() function in the
catSurv package. We demonstrate how to use the
readQualtrics() function with example data saved in the
catSurv package. You can follow along with this example
after read in your .csv file using the read.csv function in
R.
The catObj column holds the embedded data
Cat object for each respondent. These embedded data objects
hold what questions were asked to the respondent and their answers.
Because the first two rows of the Qualtrics data contain information
about the question items, delete these rows to create a vector holding
the embedded data corresponding to each respondent. We similarly create
a vector of respondent identifiers.
data(ex_qualtrics_results) # loads example results
cat_vect <- ex_qualtrics_results$catObj[-c(1,2)] # Cat embedded data objects
ids <- ex_qualtrics_results$ResponseId[-c(1,2)] # respondent identifiersThen, pass the Cat objects and respondent identifiers to
the readQualtrics() function which will clean the adaptive
inventory responses and return a data frame easier to use in subsequent
analyses. Each row of the cleaned data holds a respondent’s answer
profile. With the data in this format, it is now simple to use
functionality in catSurv to estimate the respondents’
positions on the latent trait using the estimateThetas()
function.
clean_df <- readQualtrics(catObj = cat_vect, responseID = ids)
clean_df## q86 q6 q66 q46 q36 q26 q56 q76 q13 q96 q82 q9 q22 q32 q92 q42 q52 q62 q2
## resp1 1 NA 2 NA NA NA NA NA NA NA NA 4 NA NA 3 NA NA NA NA
## resp2 NA 5 NA NA NA NA 5 NA NA NA NA NA NA NA 1 NA 1 NA NA
## resp3 3 NA 3 NA NA NA NA NA NA NA NA 3 NA NA 4 NA NA NA NA
## resp4 2 4 NA NA NA NA NA NA NA NA NA 1 NA NA -1 NA NA NA NA
## q72
## resp1 NA
## resp2 NA
## resp3 NA
## resp4 NA# estimate respondents' positions
estimateThetas(catObj = agree_cat, responses = clean_df)## [1] -2.01361634 2.77553590 -1.37453394 0.099475452.8 Including multiple adaptive inventories
We anticipate many researchers might want to include multiple adaptive inventories on their Qualtrics survey. This example walks through the general procedure for how to do so. Including multiple adapative inventories is a relatively straightforward extention of the procedure outlined above. Here were detail the important ways the procedure differs.
Broadly, including multiple adaptive batteries requires the researcher to repeat the steps outline above for each inventory. The crucial difference is that the JavaScript for each adaptive inventory must be set up to call on the appropriate hidden and embedded data. As we will demonstrate, we recommend doing this by giving the variables in the hidden and embedded data unique names for each adaptive inventory. The trickier task is then editing the JavaScript to call on the appropriate variables. We provide example scripts to facilitate this editing task. We will demonstrate including more than one adaptive battery on a survey using 20 item agreeableness battery (used above) as well as the 20 item neuroticism battery.
The first task, like before, is specifying the text for the question
items and response options by defining a custom header. The variable
name qtext now must be unique for each battery, here we
append an identifier for the battery on the variable name, resulting in
the variables qtextAGREE and qtextNEURO. The
script here is an abbreviated version of the both batteries. The full
script is available here.
<script>
var qtextAGREE = {
1: {
"text": "Have a good word for everyone.",
"responses": {
1: "Very inaccurate",
2: "Moderately inaccurate",
3: "Neither inaccurate nor accurate",
4: "Moderately accurate",
5: "Very accurate",
}
},
...
20: {
"text": "Am out for my own personal gain.",
"responses": {
1: "Very inaccurate",
2: "Moderately inaccurate",
3: "Neither inaccurate nor accurate",
4: "Moderately accurate",
5: "Very accurate",
}
}
};
var qtextNEURO = {
1: {
"text": "I often feel blue.",
"responses": {
1: "Very inaccurate",
2: "Moderately inaccurate",
3: "Neither inaccurate nor accurate",
4: "Moderately accurate",
5: "Very accurate",
}
},
...
20: {
"text": "I rarely lose my composure.",
"responses": {
1: "Very inaccurate",
2: "Moderately inaccurate",
3: "Neither inaccurate nor accurate",
4: "Moderately accurate",
5: "Very accurate",
}
}
};
var potentials = null;
function setPotentials(pots) {
potentials = pots
}
function getPotentials() {
return potentials;
}
</script>Next you must create an adaptive question for each battery. Each battery needs its own question block. Create each battery’s question block by following the instructions in Section \(\ref{adaptive_question_section}\) up until pasting the JavaScript code. The JavaScript is the only thing that changes when using multiple adaptive batteries in the same survey. We will return to the JavaScript later on. For now, leave the JavaScript as given.
Next, set up the embedded data objects. Each battery needs its
own embedded data objects. For simplicity, we recommend
appending the names of the embedded data objects with a unique
identifier for the battery. For example, we create
next_itemAGREE, last_itemAGREE, and
last_answerAGREE for the agreeableness battery and
next_itemNEURO, last_itemNEURO, and
last_answerNEURO for the neuroticism battery. Note the key
value for all of these fields remains -1. We follow a similar naming
pattern for the Cat object embedded data, naming the fields
catObjAGREE and catObjNEURO. The key values
for these embedded data objects are their respective JSON formatted
Cat objects.
data(agree_cat)
data(neuro_cat)
setLengthThreshold(agree_cat) <- 4
setLengthThreshold(neuro_cat) <- 4
toJSONCat(agree_cat)
toJSONCat(neuro_cat)
Now that we’ve created the hidden and embedded data, we can return to
the issue of editing the JavaScript for each adaptive battery’s question
block. A section of the script you will need to edit is shown in the
following JavaScript. The full script is available here.
It is not important to understand what this script is doing—you only
need to know how to edit it. Notice how BATTERY is appended
to last_item, last_answer, and
catObj in the following JavaScript. BATTERY is
a placeholder for the unique identifier of your battery that
you used when creating the hidden data in the custom header and the
embedded data.
Qualtrics.SurveyEngine.addOnload(function()
{
var domain = "https://catsurv.com";
var url = domain + "/ocpu/library/catSurv/R/processAJAX";
var last_item = +Qualtrics.SurveyEngine.getEmbeddedData('last_itemBATTERY');
var last_answer = +Qualtrics.SurveyEngine.getEmbeddedData('last_answerBATTERY');
...
var payload = "\""
+ Qualtrics.SurveyEngine.getEmbeddedData("catObjBATTERY").replace(/"/g,
'\\"') + "\"";
...The BATTERY placeholder appears many times throughout
the script. We recommend using “find and replace” tools in a text editor
to replace the placeholder text with your battery’s unique identifier.
The following JavaScript shows how the script should appear after
editing it for the agreeableness adaptive inventory. After editing the
JavaScript, paste it into the appropriate JavaScript editor as explained
in Section \(\ref{adaptive_question_section}\). Repeat
this process for each adaptive battery—edit the JavaScript by replacing
the BATTERY placeholder with your battery’s unique
identifier and add the JavaScript to the battery’s question block.
Qualtrics.SurveyEngine.addOnload(function()
{
var domain = "https://catsurv.com";
var url = domain + "/ocpu/library/catSurv/R/processAJAX";
var last_item = +Qualtrics.SurveyEngine.getEmbeddedData('last_itemAGREE');
var last_answer = +Qualtrics.SurveyEngine.getEmbeddedData('last_answerAGREE');
...
var payload = "\""
+ Qualtrics.SurveyEngine.getEmbeddedData("catObjAGREE").replace(/"/g,
'\\"') + "\"";
...After testing or fielding your survey, the
readQualtrics() function in the catSurv
package can again be used to clean the response profiles to each
adaptive battery. We demonstrate this process using example data
available in the catSurv package. First, read the raw data
into the R environment. Now there will be a column of embedded data
Cat objects for each adaptive battery, and the names of
those columns will correspond with the unique identifier you gave to the
Cat embedded data object. Here the catObjAGREE
and catObjNEURO columns hold the respondent’s answers to
each adaptive inventory.
# load example data
data(ex_qualtrics_results_multiple)
# columns now match battery unique identifier
colnames(ex_qualtrics_results_multiple)[c(13,17)]## [1] "catObjAGREE" "catObjNEURO"# agreeableness Cat embedded data objects
cat_vect_agree <- ex_qualtrics_results_multiple$catObjAGREE[-c(1,2)]
# neuroticism Cat embedded data objects
cat_vect_neuro <- ex_qualtrics_results_multiple$catObjNEURO[-c(1,2)]
# vector of respondent identifiers
ids <- ex_qualtrics_results_multiple$ResponseId[-c(1,2)]As outlined in Section \(\ref{retrieving_data_section}\), we can
then separately clean the adaptive batteries using the
readQualtrics() function and estimate the respondent’s
positions on the latent traits using the estimateThetas()
function.
clean_df_agree <- readQualtrics(catObj = cat_vect_agree, responseID = ids)
clean_df_neuro <- readQualtrics(catObj = cat_vect_neuro, responseID = ids)
# estimate respondents' positions
estimateThetas(catObj = agree_cat, responses = clean_df_agree)## [1] -0.5143049 1.3438497 -1.7000434 0.1345768estimateThetas(catObj = neuro_cat, responses = clean_df_neuro)## [1] 1.0232555 -2.1631739 0.3474408 0.85849883 Technical note
For obvious reasons, the item selection routines in
catSurv need to be fast. Given the value of survey time, it
is unacceptable for users to have to wait for even three or four seconds
for the next item to be chosen. Moreover, the routines for item
selection are often highly iterative. While R is a very useful
programming environment for many things, completing iterative
calculation at speed is not one of them. Therefore, the true guts of
catSurv are written in C++ and integrated into R using
Rcpp (Eddelbuettel et al.
2011). We achieved further speedups by parallelizing some
functions with RcppParallel (Allaire
et al. 2019) and using the very efficient numerical integration
routines in the GNU Scientific Library (GSL) via RcppGSL
(Eddelbuettel and Francois 2019). The
result is an implementation of CAT that executes in less than one
one-hundredth of a second for even the most computationally expensive
routines.
For most users, these technical details are irrelevant as they can
install the compiled packages from the Comprehensive R Archiving Network
(CRAN) simply using the install.packages() function.
However, for users who need to compile R packages on their own machines
(e.g., users with Ubuntu operating systems) some additional steps are
required to ensure that C++ and GSL are installed and configured as R
expects. We recommend starting with this vignette: https://cran.r-project.org/web/packages/RcppGSL/vignettes/RcppGSL-intro.pdf.
However, additional steps may be necessary depending on how your machine
is configured.