IMS Logo IMS Question & Test Interoperability:
ASI Outcomes Processing

Final Specification Version 1.2
Copyright © 2002 IMS Global Learning Consortium, Inc. All Rights Reserved.
The IMS Logo is a trademark of IMS Global Learning Consortium, Inc.
Document Name: IMS Question & Test Interoperability: ASI Outcomes Processing
Date: 11 February 2002

IPR and Distribution Notices

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS's procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Copyright © IMS Global Learning Consortium 2006. All Rights Reserved.

If you wish to distribute this document or use this document to implement a product or service, you must complete a valid license registration with IMS and receive an email from IMS granting the license. To register, follow the instructions on the IMS website: http://www.imsglobal.org/specificationdownload.cfm.

This document may be copied and furnished to others by Licensee organizations registered on the IMS website provided that the above copyright notice and this paragraph are included on all such copies. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to IMS, except as needed for the purpose of developing IMS specifications, under the auspices of a chartered IMS work group.

Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/question/qtiv1p2/qtiv1p2speclicense.html.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

Table of Contents


1. Introduction
     1.1 Question & Test Interoperability Overview
     1.2 Scope & Context
     1.3 Structure of this Document
     1.4 Nomenclature
     1.5 References

2. Information Model
     2.1 Exchanging ASI Objects
     2.2 Use Cases
     2.3 ASI Issues
     2.4 Outcomes Processing Solution
           2.4.1 The In-built Scoring Algorithms
           2.4.2 Defined Proprietary Algorithms
     2.5 Tabular Representation
           2.5.1 QTI Outcomes Processing Data Objects

3. XML Binding
     3.1 outcomes_processing> Elements
           3.1.1 <qticomment> Element
           3.1.2 <outcomes> Element
           3.1.3 <objects_condition> Element
           3.1.4 <processing_parameter> Element
           3.1.5 <map_output> Element
           3.1.6 <outcomes_feedback_test> Element
     3.2 <outcomes> Element
           3.2.1 <qticomment> Element
           3.2.2 <decvar> Element
           3.2.3 <interpretvar> Element
     3.3 <objects_condition>
           3.3.1 <qticomment> Element
           3.3.2 <outcomes_metadata> Element
           3.3.3 <and_out> Element
           3.3.4 <or_out> Element
           3.3.5 <not_out> Element
           3.3.6 <objects_parameter> Element
           3.3.7 <map_input> Element
           3.3.8 <objectscond_extension> Element
     3.4 <outcomes_feedback_test> Element
           3.4.1 <test_variable> Elements
           3.4.2 <displayfeedback> Element
     3.5 <outcomes_metadata> Element
     3.6 <and_objects> Element
           3.6.1 <outcomes_metadata> Element
           3.6.2 <and_objects> Element
           3.6.3 <or_objects> Element
           3.6.4 <not_objects> Element
     3.7 <or_objects> Element
           3.7.1 <outcomes_metadata> Element
           3.7.2 <and_objects> Element
           3.7.3 <or_objects> Element
           3.7.4 <not_objects> Element
     3.8 <not_objects> Element
           3.8.1 <outcomes_metadata> Element
           3.8.2 <and_objects> Element
           3.8.3 <or_objects> Element
           3.8.4 <not_objects> Element
     3.9 <test_variable> Element
           3.9.1 <variable_test> Element
           3.9.2 <and_test> Element
           3.9.3 <or_test> Element
           3.9.4 <not_test> Element
     3.10 <and_test> Element
           3.10.1 <variable_test> Element
           3.10.2 <and_test> Element
           3.10.3 <or_test> Element
           3.10.4 <not_test> Element
     3.11 <or_test> Element
           3.11.1 <variable_test> Element
           3.11.2 <and_test> Element
           3.11.3 <or_test> Element
           3.11.4 <not_test> Element
     3.12 <not_test> Element
           3.12.1 <variable_test> Element
           3.12.2 <and_test> Element
           3.12.3 <or_test> Element
           3.12.4 <not_test> Element

4. Best Practice & Implementation Guide
     4.1 Overall Data Model
     4.2 Relationship to the Other IMS Specifications
     4.3 Basic Example XML Instances
           4.3.1 Example ('Number Correct' and 'Number Correct Attempted')
           4.3.2 Example ('Weighted Number Correct', 'Weighted Number Correct Attempted',
'Parameter Weighted Number Correct' and 'Parameter Weighted Number
Correct Attempted')

           4.3.3 Example ('SumofScores' and 'SumofScoresAttempted')
           4.3.4 Example ('Weighted Sum of Scores', 'Weighted Sum of Scores Attempted', 'Parameter Weighted Sum of Scores' and 'Parameter Weighted Sum of Scores Attempted')
           4.3.5 Example ('Best K from N')
           4.3.6 Example ('Guessing Penalty' and 'WeightedGuessingPenalty')
           4.3.7 Example (Remapped Output 'Sum of Scores')
           4.3.8 Example (Remapped Input 'Sum of Scores')
           4.3.9 Example (Parameterized 'Sum of Scores')
     4.4 Advanced Example XML Instances
           4.4.1 Example (Multiple-choice Quiz)
           4.4.2 Example (True/false Quiz)
     4.5 Implementation Guidance
           4.5.1 The In-built Scoring Algorithms
           4.5.2 Defining Proprietary Scoring Algorithms
           4.5.3 Using Meta-data
           4.5.4 Consideration of Selection & Ordering
     4.6 Example XML Instances
     4.7 Proprietary Extensions
     4.8 V1.x/V2.0 Issues & Compatibility
     4.9 IMS Harmonization

Appendix A - Glossary of Terms
     A1 - General Terms
     A2 - Elements & Attributes

Appendix B - In-Built Scoring Algorithms
     B1 - 'Number Correct' Scoring Algorithm
     B3 - 'Weighted Number Correct' Scoring Algorithm
     B4 - 'Weighted Number Correct (Attempted)' Scoring Algorithm
     B5 - 'Parameter Weighted Number Correct' Scoring Algorithm
     B6 - 'Parameter Weighted Number Correct (Attempted)' Scoring Algorithm
     B7 - 'Sum of Scores' Scoring Algorithm
     B8 - 'Sum of Scores (Attempted)' Scoring Algorithm
     B9 - 'Weighted Sum of Scores' Scoring Algorithm
     B10 - 'Weighted Sum of Scores (Attempted)' Scoring Algorithm
     B11 - 'Parameter Weighted Sum of Scores' Scoring Algorithm
     B12 - 'Parameter Weighted Sum of Scores (Attempted)' Scoring Algorithm
     B13 - 'Best K from N' Scoring Algorithm
     B14 - 'Guessing Penalty' Scoring Algorithm
     B15 - 'Weighted Guessing Penalty' Scoring Algorithm

Appendix C - Logic Rules

About This Document
     List of Contributors

Revision History

Index

1. Introduction

1.1 Question & Test Interoperability Overview

The IMS Question & Test Interoperability (QTI) specification describes a basic structure for the representation of question (item) and test (assessment) data and their corresponding results reports [QTI, 02i]. Therefore, the specification enables the exchange of this item, assessment and results data between Learning Management Systems, as well as content authors and, content libraries and collections. The IMS QTI specification is defined in XML to promote the widest possible adoption. XML is a powerful, flexible, industry standard mark-up language used to encode data models for Internet-enabled and distributed applications. The QTI specification is extensible and customizable to permit immediate adoption, even in specialized or proprietary systems. Leading suppliers and consumers of learning products, services and content contributed time and expertise to produce this final specification. The IMS QTI specification, like all IMS specifications, does not limit product designs by specifying user interfaces, pedagogical paradigms, or establishing technology or policies that constrain innovation, interoperability, or reuse.

The 'Outcomes Processing' specification contains the description of how the aggregated scores at the Assessment and Section levels can be derived. These scoring outcomes are based upon the child Sections and/or Items. A wide range of scoring algorithms is supported through the usage of a predefined set of parameterized instructions; these avoid the realization of the algorithms within the XML. This document contains the relevant information model, XML binding and best practices guidance but it should be read in the context of the core ASI documents.

1.2 Scope & Context

This document is the IMS Question & Test Interoperability ASI Outcomes Processing Specification and must be read in conjunction with the core documents:

It defines the outcomes processing features that are to be applied to the Section and/or Assessments. These new Assessment and Section features are not backwards compatible with the other versions of the IMS QTI specifications. These new features have no effect on the IMS QTILite specification [QTI, 02h] because that refers only to the Item data structure.

1.3 Structure of this Document

The structure of this document is:

2. Information Model
The description of the information model of the outcomes processing component for the full IMS QTI ASI;
3. XML Binding
The description of the XML binding of the outcomes processing component for the full IMS QTI ASI;
4. Best Practice & Implementation Guide
The description of the best practices and implementation guide for the outcomes processing component of the full IMS QTI ASI;
Appendix A - Glossary of Terms
A brief description of all of the elements and attributes that have been used to support outcomes processing;
Appendix B - In-Built Scoring Algorithms
A description of the operation of each of the default scoring algorithms;
Appendix C - Logic Rules
The logic rules that are supported by the or_object, and_object and not_object elements.

1.4 Nomenclature

API
Application Programming Interface
ASI
Assessment, Section, Item
CBT
Computer Based Training
DTD
Document Type Definition
QTI
Question & Test Interoperability
VLE
Virtual Learning Environment
W3C
World Wide Web Consortium
XML
Extensible Mark-up Language

1.5 References

[IMS, 01]
IMS Persistent, Location-Independent Resource Identifier Implementation Handbook, M.McKell, Version 1.0, IMS, April 2001.
[QTI, 02a]
IMS Question & Test Interoperability: ASI Information Model Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02b]
IMS Question & Test Interoperability: ASI XML Binding Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02c]
IMS Question & Test Interoperability: ASI Best Practice & Implementation Guide, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02d]
IMS Question & Test Interoperability: ASI Selection & Ordering Specification, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02e]
IMS Question & Test Interoperability: Results Reporting Information Model, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02f]
IMS Question & Test Interoperability: Results Reporting XML Binding, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02g]
IMS Question & Test Interoperability: Results Reporting Best Practice & Implementation Guide, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02h]
IMS Question & Test Interoperability: QTILite Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02i]
IMS Question & Test Interoperability: Overview, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.

2. Information Model

2.1 Exchanging ASI Objects

The possible advanced Assessment structures that can be exchanged using QTI are shown in Figure 2.1:

Possible <assessment> structure (a)

(a)

Possible <assessment> structure (c)

(c)

Possible <assessment> structure (b)

(b)



Figure 2.1 Possible <assessment> structures.

In the case of Sections, some of the possible data structures that can be exchanged are shown in Figure 2.2:

Possible <section> structure (a)

(a)

Possible <section> structure (b)

(b)

Possible <section> structure (c)

(c)

Possible <section> structure (d)

(d)

Possible <section> structure (e)

(e)

Figure 2.2 Possible <section> structures.

The wide range of different data structures that can be exchanged is especially significant when developing mechanisms that support 'Outcomes Processing'. 'Outcomes Processing' is responsible for:

It is important to note that the type of scoring algorithm that is to be used is not mandated by the IMS specification. Instead, the specification is capable of supporting most of the available scoring algorithms. Also, the algorithms themselves are not encoded within the specification itself. Instead the type of algorithm is specified and the necessary parametric information supplied to enable the assessment engine to execute the corresponding algorithm.

2.2 Use Cases

The following representative use-cases have been identified for support within QTI ASI outcomes processing V1.2:

  1. Multiple-choice quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete (single response) questions whose outcomes are a dichotomous variable e.g. multiple-choice questions. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "CORRECT" for each Item. This takes on the value "True" if the response was correct and "False' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the percentage of total score obtained by a candidate and (3) the percentage of items actually attempted which were correct;
  2. True/false quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete true/false. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "isCorrect" for each Item. This takes on the value '1' if the response was correct and "-1' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the weighted total score;
  3. Multiple-response quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are multiple-response questions (two answers for each) with partial scores being awarded when at only one correct answer is given. For the sake of being definite we assume there are 5 Items and that we have a defined outcome, "SCORE" for each Item. This takes on the value "+1" for each correct response "-1' for each incorrect response. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the percentage of items actually attempted which were correct;
  4. End of chapter test - this is the same as use-case (a) except that all of the items have different weights. We wish to see (1) the number of correct items, (2) the percentage of total weight answered correctly by the candidate and (3) the "weighted" percentage of questions attempted correctly of those questions answered;
  5. English comprehension/composition test with essay - this is a placement test designed to assess English language ability. The test consists of two sections: (1) Short answer items (multiple-choice) and (2) the Essay. Short answer items come in two kinds. The first are 15 discrete vocabulary, grammar and usage Items that all produce a single dichotomous outcome "isCorrect". The second are the reading comprehension Items (although they may actually test other skills than just reading) that consist of a stimulus followed by five discrete questions. We assume that these are done as Items with five responses and hence five outcomes: these are Boolean variables whose meanings are as above. All of these Items can be given different weights (as can the 5 outcomes of the two reading comprehension items). The essay question is graded "A,B,C,D,E,F" by either a human or computer rater. The divisions do not necessarily represent equal spacings. The essay represents 1/3 of the total grade. Out of ten points, the mapping of the grades is as follows A=10, B=8, C=7, D=5, E=2, F=0. The total score for the assessment is required;
  6. Diagnostic scoring - this is an additional score report which has been retrofitted onto the 'English comprehension test ((e) above). In addition to the overall score, "diagnostic" scores are going to be given in four different sub-areas: "Reading", "Writing", "Vocabulary" and "Grammar" (the dependencies among these sub-scores and the overall score are ignored for computational simplicity). The 15 discrete items and each of the 10 outcomes from the reading items all load onto these sub-scores differently. Some outcomes are completely irrelevant to some of the sub-scores. An Item/outcome can have a high "vocabulary" load and a low "grammar" load, or vice-versa. The essay is also added onto each of the sub-scores, but with a different loading;
  7. Diagnostics complex response biology lite - this is a classroom assessment designed to assess the students understanding of the scientific method as it applies to Biology. The assessment returns not a single score but three scores reflecting "Domain Knowledge" (knowledge of the biological subjects in the assessment), "Methodological Knowledge" (understanding of the scientific method) and "Integrated Knowledge" (the ability to apply the scientific method to a given problem). The items are a series of complex "tasks". Each task gives a piece of scenario of a scientific investigation and has the students work through several steps of the process, answering questions about what they are doing at each step. Each task has many complex responses, and hence many observable outcomes. The outcomes are related to the number of steps in the task. Each is a variable given one of the values "Low", "Medium", or "High" depending on how the student did on one part of one step. The outcomes are labelled "OUT***" where '***' is a sequence number unique to the item. For more definite, assume that there are five tasks: Task1 has 7 outcomes, Task2 has 5, Task3 has 8, Task4 has 6 and Task5 has 13. For each of the reporting variables, the scoring model should add an appropriate "weight" if the outcome has reached an appropriate level (either "Medium" and above or "High"). The weight and the level will, in general, be different for each of the reporting variables. Not all outcomes are used for all of the reporting variables;

2.3 ASI Issues

The core features within the ASI structures that are related to the process of selection and ordering are shown in Figure 2.3.

An ASI outcomes processing structure

Figure 2.3 An ASI outcomes processing structure.

The relationship of these features to outcomes processing are:

An ASI outcomes processing exampl

Figure 2.4 An ASI outcomes processing example.

The scoping of the scoring algorithms that are supported is summarized using the example Section/Item combination shown in Figure 2.4. The scope of the scoring algorithm is:

It is the responsibility of the scoring engine that is implementing the scoring algorithm to maintain the scoping of the variables. If the default variable names are used then all of the variables will be named 'SCORE'. For instance, the scoping should ensure that:

Section_Ident_2.SCORE = Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE

At the level of the 'Section_Ident_1' then the variables 'Item_Ident_2.1.SCORE' etc. are only perceived through the variable 'Section_Ident_2.SCORE'.

2.4 Outcomes Processing Solution

The outcomes processing capability is based upon two complimentary mechanisms:

2.4.1 The In-built Scoring Algorithms

The algorithms that are to be supported are:

  1. Number of right answers (including multivariate responses) - this is identified by the algorithm name "NumberCorrect" and has associated with it the default integer variable 'COUNT'. This is the store for the number of correctly answered objects. An object is defined as correctly answered if the value assigned to the default Boolean variable 'COUNT' for the object is 'True'. The number correct algorithm can only operate on Boolean scoring variables. The default integer scoring variables COUNT.max (equal to the number of objects attempted), COUNT.min (set to zero) and COUNT.normalized (a value of between zero and one) are also available. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  2. Number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'NumberCorrectAttempted';
  3. Weighted number right (including multivariate responses) - this is similar to the 'number of rights answers' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedNumberCorrect". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  4. The weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedNumberCorrectAttempted';
  5. Parameterized weighted number correct - name 'ParameterWeightedNumberCorrect' this is similar to the 'WeightedNumberCorrect' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of CORRECT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  6. The parameterized weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedNumberCorrectAttempted';
  7. Percentage correct (including multivariate responses) - this score is derived from both the "NumberCorrect" and "NumberCorrectAttempted" algorithms in which the percentage correct is defined by the equation-

  1. Sum-of-scores - this is identified by the algorithm name "SumofScores" and has associated with it four default integer variables SCORE, SCORE.min, SCORE.max and SCORE.normalized. These variables store the sum of the scores, the minimum possible score (set to the sum of the 'minvalue' attributes for all of the corresponding objects), the maximum possible score (set to the sum of the 'maxvalue' attributes for all of the corresponding objects) and the normalized score (a value of between zero and one) respectively for the set of objects (the minimum, maximum and normalized values are only valid if they have been explicitly defined for all of the corresponding objects). The value of 'SCORE' is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  2. Sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'SumofScoresAttempted';
  3. Weighted sum of scores (including multivariate responses) - this is similar to the 'sum of scores' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedSumofScores". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  4. The weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedSumofScoresAttempted';
  5. Parameterized weighted sum of scores - name 'ParameterWeightedSumofScores' this is similar to the 'WeightedSumofScores' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  6. The parameterized weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedSumofScoresAttempted';
  7. Best 'K' out of 'N' (including multivariate responses) - this is identified by the algorithm name "BestKofN" and is used to calculate the outcome by using the highest 'K' scores from the set of presented objects generically defined as 'N' objects. The values are placed in the default variables 'SCORE', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  8. Negative scores (including multivariate responses) - called 'GuessingPenalty'. Each object has three associated variables for the values of the number of correct answers ('COUNT.correct'), the number of incorrect answers ('COUNT.incorrect') and the number of unattempted answers ('COUNT.unattempted');
  9. Item Response Theory (IRT) - supported using the processing parameter mechanism;
  10. Time remaining - for further study in V2.0;
  11. Par scores - for further study in V2.0;
  12. Cut scores (pass/fail, right/wrong and including multivariate responses) - for further study in V2.0;
  13. Multiple forms/equating (table based) - for further study in V2.0;
  14. Partial credit/Graded response - for further study;
  15. Critical Item - for further study in V2.0;
  16. Critical Outcome - for further study in V2.0;
  17. Factor analysis/ Multivariate IRT (research) -supported through extensions. For further study in V2.0;
  18. Bayes net (research) - supported through extensions. For further study in V2.0;
  19. Computer Adaptive Testing - for further study in V2.0.

2.4.2 Defined Proprietary Algorithms

Proprietary algorithms can be used but these must be characterized using the parameter definition mechanisms supported in the specification i.e. the <objects_parameter> and <processing_parameter> elements. Any number of parameters can be passed using these elements and there is no restriction on the naming convention used for these parameters - all parameters are specific to an algorithm. The names of the proprietary algorithms (passed in the scoremodel attribute) should not clash with the intrinsic names and some form of naming convention based upon the creating organization should be adopted.

2.5 Tabular Representation

The tables in this Section provide a conceptual, informative description of the elements in the data objects. The columns in these tables refer to:

No:
The number of the data element. An element may be composed of sub-elements. The numbering scheme reflects these relationships.
Name:
The descriptive name of the element.
Explanation:
A brief functional description of the element.
Required:
Indicates if the element is required:
  • M = Mandatory Element that must be included in the data object, if the element at the higher level is included;
  • C = Conditional Element. Existence is dependent on values of other Elements;
  • O = Optional Element.
Multi:
Multiplicity of the element:
  • Blank = single instance;
  • Number = maximum number of times the element is repeatable;
  • n = multiple occurrences allowed, no limit;
  • Repeatability of an element implies that all sub-elements repeat with the same element.
Type:
A description of formatting rules for the data element. Type includes the maximum length of the element:
  • ID = element used to uniquely identify an object;
  • Code = element value from a list of codes;
  • Description = descriptive element, human language
  • Flag = binary flag
  • Enumerated = list of predefined non-numeric options i.e. the definitive list of objects
  • The international character set specified by ISO 10646 will be used for all fields.

The type will also include a description of the set of valid values for the sub-element:

  • Coding schemes using numerical values;
  • The set of values as defined in the Domain i.e. making it closed. The list of values cannot be extended to include values not defined in the specification. If there is a need for values not included in the domain set of values then the extension should be done defining a new element under the Extension element that is a part of each data object definition.

Note: Additional descriptive information about the element.

2.5.1 QTI Outcomes Processing Data Objects

Table 2.1 describes the data objects that are used in the construction of the QTI outcomes processing elements.

Table 2.1 QTI outcomes processing data objects detailed description.

No
Name
Explanation
Reqd
Mult
Type
Note
1
outcomes_processing
Accumulated outcomes processing and feedback applied within Sections and Assessments.
O
n




1.1
scoremodel
The type of scoring model being adopted.
O


CDATA string describing the model.
String 32 chars.
Default string is "SumofScores".
1.2
qticomment
The comments used to annotate the XML file.
O




Comments should be used to aid human readability of the XML file itself.
1.2.1
xml:lang
The language that is being used for the information.
O


String 32 chars.
The language entries will be defined as per the ISO639 and ISO3166 standards.
1.3
outcomes
To create the variables required for the assessment accumulated scores.
M




The assessment accumulated processing variables group.
1.3.1
qticomment
As per structure 1.2.
1.3.2
decvar
Declaration of a variable to be used for scoring.
M
n


Each type of variable must be declared before it is used.
1.3.2.1
varname
The name of the variable to be declared.
M


String
256 chars.
Default is set as 'SCORE".
1.3.2.2
vartype
The type of variable.
M


Enumerated:
String
Integer (default)
Decimal
Scientific
Boolean
Enumerated
Set
Default is set to 'Integer'.
1.3.2.3
defaultval
The default value for the variable.
O


Numerical 32 chars
String 32 chars
True/False
Can be set to any value. Default is set to '0'.
1.3.2.4
minvalue
The minimum value permitted for a numeric score.
O


String 32chars.
Applies to the value of the score after all of the item processing has been completed.
1.3.2.5
maxvalue
The maximum score permitted for a numeric score.
O


String 32chars.
Applies to the value of the score after all of the item processing has been completed.
1.3.2.6
cutvalue
The value above which the participant will have been defined to have mastery of the subject.
O


String 32chars.
The type of this cut value is set by the variable type.
1.3.2.7
members
The set of enumerated values.
O


String 1024chars.
This is a comma separated list without enclosing parentheses.
1.3.3
interpretvar
The interpretation to be applied to the variable in terms relevant to an actor.
O
n


At present this element will be a comment string however it will be further developed in version 1.2.
1.3.3.1
varname
The name of the input variable being described.
O


As per structure 1.3.2.1.
1.3.3.2
view
The view to which the interpretation is applied.
O
n
Enumerated:
All (default)
Administrator
AdminAuthority
Assessor
Author
Candidate
Invigilator
Proctor
Psychometrician
Scorer
Tutor
The 'All' view is the default value.
1.4
objects_condition
This contains the conditions that are applied to define the ways in which the outcomes variables are combined to create the aggregated value.
O
n




1.4.1
qticomment
As per structure 1.2.
1.4.2
outcomes_metadata
Contains the rules that are applied to the IMS QTI-specific and/or IMS Meta-data meta-data fields of the object to decide if the object scoring is to be aggregated.
C


String
1-256 chars.


1.4.2.1
mdname
Identifies the IMS QTI-specific or IMS Meta-data field that is to be used for the aggregation rule.
M


String
1-64 chars.


1.4.2.2
mdoperator
Identifies the nature of the meta-data field comparison that is to be applied.
M


Enumerated list:
EQ
NEQ
LT
LTE
GT
GTE


1.4.3
and_objects
Contains the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.
C




See Appendix B for the logic rules.
1.4.3.1
outcomes_metadata
As per structure 1.4.2
1.4.3.2
and_objects
As per structure 1.4.3
1.4.3.3
or_objects
As per structure 1.4.4
1.4.3.4
not_objects
As per structure 1.4.5
1.4.4
or_objects
Contains the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.
C




See Appendix B for the logic rules.
1.4.4.1
outcomes_metadata
As per structure 1.4.2
1.4.4.2
and_objects
As per structure 1.4.3
1.4.4.3
or_objects
As per structure 1.4.4
1.4.4.4
not_objects
As per structure 1.4.5
1.4.5
not_objects
Contains the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.
C




This element contains only ONE of the sub-elements.
See Appendix B for the logic rules.
1.4.5.1
outcomes_metadata
As per structure 1.4.2
1.4.5.2
and_objects
As per structure 1.4.3
1.4.5.3
or_objects
As per structure 1.4.4
1.4.5.4
not_objects
As per structure 1.4.5
1.4.6
objects_parameter
Contains the value of a particular parameter that is to be used by the corresponding scoring algorithm variable selection. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters.
O
n


These parameters only apply to the objects within the containing <objects_condition> element.
1.4.6.1
pname
The name of the parameter that is being defined.
M


String
256 chars.
Parameters of the same name may have different significance to different algorithms.
1.4.7
map_input
This element is used to re-map the default input variable to another variable.
O
n




1.4.7.1
varname
The name of the input variable whose name is to be remapped.
O


As per structure 1.3.2.1.
1.4.8
objectscond_extension
The objects condition extension facility.
O
n
ANY


1.5
processing_parameter
The value of a parameter that is passed to the scoring algorithm that is being used.
O
n
String
128 chars.
These parameters apply to the algorithm as a whole and not to specific objects operated on by the algorithm.
1.5.1
pname
The name of the parameter that is being defined.
M


String
256 chars.
Parameters of the same name may have different significance to different algorithms.
1.6
map_output
This is used to remap the result of the scoring algorithm from the default/named variable to another named variable.
O
n
String
256 chars.
The new variable name should have been declared in the <outcomes> element of the parent object. This remapping operates for all of the associated variable names e.g. the *.min , *.max and *.normalized variables.
1.6.1
varname
The name of the output variable whose name is to be remapped.
O


As per structure 1.3.2.1.
1.7
outcomes_feedback_test
Contains the tests to be applied to determine if any and the type of feedback to be presented.
O
n




1.7.1
title
The title of the feedback test.
O


String
256 chars.


1.7.2
test_variable
The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.
M






1.7.2.1
variable_test
The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.
C






1.7.2.1.1
varname
The name of the variable whose state is to be tested.
O


As per structure 1.3.2.1.
1.7.2.1.2
testoperator
Identifies the nature of the variable comparison that is to be applied.
M


Enumerated list:
EQ
NEQ
LT
LTE
GT
GTE


1.7.2.2
and_test
This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.
C






1.7.2.2.1
variable_test
As per structure 1.7.2.1.
1.7.2.2.2
and_test
As per structure 1.7.2.2.
1.7.2.2.3
or_test
As per structure 1.7.2.3.
1.7.2.2.4
not_test
As per structure 1.7.2.4.
1.7.2.3
or_test
This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.
C






1.7.2.3.1
variable_test
As per structure 1.7.2.1.
1.7.2.3.2
and_test
As per structure 1.7.2.2.
1.7.2.3.3
or_test
As per structure 1.7.2.3.
1.7.2.3.4
not_test
As per structure 1.7.2.4.
1.7.2.4
not_test
This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.
C




Only one of the contained sub-elements can be used within each usage of the NOT operator.
1.7.2.4.1
variable_test
As per structure 1.7.2.1.
1.7.2.4.2
and_test
As per structure 1.7.2.2.
1.7.2.4.3
or_test
As per structure 1.7.2.3.
1.7.2.4.4
not_test
As per structure 1.7.2.4.
1.7.3
displayfeedback
The trigger for displaying feedback.
M
n




1.7.3.1
feedbacktype
The type of feedback to be displayed.
M


Enumerated:
Response (default)
Solution
Hint
The default value is 'Response'.
1.7.3.2
linkrefid
The identifier of the material to be referenced.
M


String 32 chars.
Consistency checking is beyond the scope of this specification. Usage rules are given in the Q&TI Best Practice Guide.

3. XML Binding

3.1 outcomes_processing> Elements

Description: The <outcomes_processing> element is the container for all of the outcomes processing instructions for Assessments and Sections. Multiple outcomes processing containers can be use when multiple scoring algorithms are to be applied to produce the aggregated outcomes. If multiple outcomes_processing elements are supplied, it is the intention that all should be run and the outcomes from all of them together should be reported as the outcomes of the enclosing Section or Assessment. The outcome variables defined by each <outcomes_processing> element should be unique across all outcomes_processing elements defined by a section. In particular, it is an error for multiple <outcomes_processing> elements to set the same outcome variable and the results will be undefined.

The <outcomes_processing> element structure

Figure 3.1 The <outcomes_processing> element structure.

Multiplicity: The <outcomes_processing> occurs zero or more times within the <assessment> and <section> elements.

Attributes:

Elements:

Example:

<outcomes_processing scoremodel="SumofScores">
<outcomes>
<decvar/>
</outcomes>
</outcomes_processing>

3.1.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes processing structure as a whole.

Multiplicity: Occurs zero or once within the <outcomes_processing> element.

Attributes:

3.1.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0).

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: See sub-section 3.2.

3.1.3 <objects_condition> Element

Description: This element contains the conditions that are applied to define the ways in which the outcomes variables are combined to create the aggregated value using the in-built set of algorithm definitions.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes: See sub-section 3.3.

3.1.4 <processing_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes:

3.1.5 <map_output> Element

Description: This element is used to re-map the named variable to another named variable (given in the body of the element). The target variable name must have been declared using <decvar> in the <outcomes> element of the enclosing <outcomes_processing> element. When a variable is remapped, all of its derived variables are to be remapped as well. Thus if remapping 'SCORE' to 'myScore', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized' would be remapped to 'myScore.min', 'myScore.max' and 'myScore.normalized' respectively.
Data-type = string (1-256 chars).

Multiplicity: This occurs zero or many times within the <outcomes_condition> element.

Attributes:

3.1.6 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: See sub-section 3.4.

3.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0). The declaration of each variable is accompanied by the implicit declaration of the <vname>.min, <vname>.max and <vname>.normalized variables. These are used to store the minimum value, maximum value and normalized value (in the range zero to one) of the scoring variable.

<outcomes> elements

Figure 3.2 <outcomes> elements.

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: None.

Elements:

3.2.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes.

Multiplicity: Occurs zero or once within the <outcomes> element.

Attributes:

3.2.2 <decvar> Element

Description: The <decvar> element permits the declaration of the scoring variables. The default name is 'SCORE' but the variables SCORE.min, SCORE.max and SCORE.normalized are also declared and made available from the scoring algorithms. A <decvar> with no attributes is assumed to define the integer variable 'SCORE' with a default value of zero.

Multiplicity: This occurs one or more times within the <outcomes> element.

Attributes:

3.2.3 <interpretvar> Element

Description: The <interpretvar> element is used to provide statistical interpretation information about the associated variables.

Multiplicity: This occurs zero or more times within the <outcomes> element.

Attributes:

Another description of this element and its sub-components is given in IMS QTI: ASI XML Binding V1.2
[QTI, 02b].

3.3 <objects_condition>

Description: Each <objects_condition> element defines a subset of the objects (Items and Sections) selected by the selection algorithm that are to be used in this score. This allows for the construction of subscores. If no <outcomes_metadata> element is present within the <objects_conditions> element to select a subset of objects, then the <objects_conditions> applies to all objects selected by the selection algorithm. If multiple <objects_conditions> are given within a single <outcomes_processing> element, then the algorithm is applied to the union of all objects selected. The <objects_parameter>, <map_input> and <objectscond_extension> elements within the <objects_condition> element apply only to those objects selected in the condition. If an object is selected by more than one <objects_condition> element, it should receive parameter assignments and input mapping as if it were the first element so selected. These conditions include the identification and mapping of the variables for input.

<objects_condition> elements

Figure 3.3 <objects_condition> elements.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: None.

Elements:

3.3.1 <qticomment> Element

Description: This element contains the comments that are relevant to the presentation of the Item.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes:

3.3.2 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the QTI-specific and/or IMS Meta-data meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule. This data is a string of up to 64 characters length.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.5.

Elements: None.

3.3.3 <and_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.6.

3.3.4 <or_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.7.

3.3.5 <not_out> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.8.

3.3.6 <objects_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm variable selection. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters. These parameters are applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outomes_condition> select the same object, the parameters are taken from the first <outcomes_condition> that selects the element.

Multiplicity: Occurs zero or more times within the <objects_condition> element.

Attributes:

3.3.7 <map_input> Element

Description: This element is used to re-map the named input variable to another variable. The default variable names are derived from the type of scoring algorithm identified in <outcomes_processing> element. The target variable name must have been declared in the evaluation objects that undergoing aggregation and must be of the same type as the default variable. When a variable is remapped, all of its derived variables are to be remapped as well. The input mapping is applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outcomes_condition> select the same object, the input mapping used is the one contained in the first <outcomes_condition> that selects the element.
Data-type = String (1-256 chars).

Multiplicity: This occurs zero or many times within the <objects_condition> element.

Attributes:

3.3.8 <objectscond_extension> Element

Description: This element contains the proprietary extensions that can be used to extend the functionally capabilities of the <outcomes_condition> element.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: None.

3.4 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

<outcomes_feedback_test> elements

Figure 3.4 <outcomes_feedback_test> elements.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes:

Elements:

3.4.1 <test_variable> Elements

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.