IMS Question & Test Interoperability: ASI Outcomes Processing Final Specification Version 1.2

Table of Contents

1. Introduction

1.1 Question & Test Interoperability Overview

The IMS Question & Test Interoperability (QTI) specification describes a basic structure for the representation of question (item) and test (assessment) data and their corresponding results reports [QTI, 02i]. Therefore, the specification enables the exchange of this item, assessment and results data between Learning Management Systems, as well as content authors and, content libraries and collections. The IMS QTI specification is defined in XML to promote the widest possible adoption. XML is a powerful, flexible, industry standard mark-up language used to encode data models for Internet-enabled and distributed applications. The QTI specification is extensible and customizable to permit immediate adoption, even in specialized or proprietary systems. Leading suppliers and consumers of learning products, services and content contributed time and expertise to produce this final specification. The IMS QTI specification, like all IMS specifications, does not limit product designs by specifying user interfaces, pedagogical paradigms, or establishing technology or policies that constrain innovation, interoperability, or reuse.

The 'Outcomes Processing' specification contains the description of how the aggregated scores at the Assessment and Section levels can be derived. These scoring outcomes are based upon the child Sections and/or Items. A wide range of scoring algorithms is supported through the usage of a predefined set of parameterized instructions; these avoid the realization of the algorithms within the XML. This document contains the relevant information model, XML binding and best practices guidance but it should be read in the context of the core ASI documents.

1.2 Scope & Context

This document is the IMS Question & Test Interoperability ASI Outcomes Processing Specification and must be read in conjunction with the core documents:

• IMS QTI: ASI Information Model [QTI, 02a];
• IMS QTI: ASI XML Binding [QTI, 02b];
• IMS QTI: ASI Best Practice & Implementation Guide [QTI, 02c].

It defines the outcomes processing features that are to be applied to the Section and/or Assessments. These new Assessment and Section features are not backwards compatible with the other versions of the IMS QTI specifications. These new features have no effect on the IMS QTILite specification [QTI, 02h] because that refers only to the Item data structure.

1.3 Structure of this Document

The structure of this document is:

2. Information Model	The description of the information model of the outcomes processing component for the full IMS QTI ASI;
3. XML Binding	The description of the XML binding of the outcomes processing component for the full IMS QTI ASI;
4. Best Practice & Implementation Guide	The description of the best practices and implementation guide for the outcomes processing component of the full IMS QTI ASI;
Appendix A - Glossary of Terms	A brief description of all of the elements and attributes that have been used to support outcomes processing;
Appendix B - In-Built Scoring Algorithms	A description of the operation of each of the default scoring algorithms;
Appendix C - Logic Rules	The logic rules that are supported by the or_object, and_object and not_object elements.

1.4 Nomenclature

API	Application Programming Interface
ASI	Assessment, Section, Item
CBT	Computer Based Training
DTD	Document Type Definition
QTI	Question & Test Interoperability
VLE	Virtual Learning Environment
W3C	World Wide Web Consortium
XML	Extensible Mark-up Language

1.5 References

[IMS, 01]	IMS Persistent, Location-Independent Resource Identifier Implementation Handbook, M.McKell, Version 1.0, IMS, April 2001.
[QTI, 02a]	IMS Question & Test Interoperability: ASI Information Model Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02b]	IMS Question & Test Interoperability: ASI XML Binding Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02c]	IMS Question & Test Interoperability: ASI Best Practice & Implementation Guide, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02d]	IMS Question & Test Interoperability: ASI Selection & Ordering Specification, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02e]	IMS Question & Test Interoperability: Results Reporting Information Model, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02f]	IMS Question & Test Interoperability: Results Reporting XML Binding, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02g]	IMS Question & Test Interoperability: Results Reporting Best Practice & Implementation Guide, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02h]	IMS Question & Test Interoperability: QTILite Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02i]	IMS Question & Test Interoperability: Overview, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.

2. Information Model

2.1 Exchanging ASI Objects

The possible advanced Assessment structures that can be exchanged using QTI are shown in Figure 2.1:

• The Assessment could contain a single Section block (Figure 2.1a);
• The Assessment could consist of a reference to an external Section block (Figure 2.1b);
• The Assessment could consist of a mixture of contained and externally referenced Section blocks (Figure 2.1c). There is no preferential order in the blocks and references.

(a)	(c)
(b)

In the case of Sections, some of the possible data structures that can be exchanged are shown in Figure 2.2:

• The Section could contain one or more Items (Figure 2.2a);
• The Section could contain one or more references to external Items (Figure 2.2b);
• The Section could contain one or more Sections (Figure 2.2c);
• The Section could contain one or more references to external Sections (Figure 2.2d);
• The Section could contain a mixture of blocks and references to Section and Items. There is no constraint on the order in which the Sections and Items are declared/referenced i.e. interleaving is supported.

(a)	(b)
(c)	(d)

(e)

The wide range of different data structures that can be exchanged is especially significant when developing mechanisms that support 'Outcomes Processing'. 'Outcomes Processing' is responsible for:

• The aggregation of scores assigned to the Items and/or Sections to create a score or set of scores for the parent Section. The ways in which this roll-up is achieved reflects the different scoring algorithms;
• The aggregation of scores assigned to Sections to create a score or set of scores for the parent Assessment. The ways in which this roll-up is achieved reflects the different scoring algorithms;
• Any assignment of scores based upon more than one Item and/or Section.

It is important to note that the type of scoring algorithm that is to be used is not mandated by the IMS specification. Instead, the specification is capable of supporting most of the available scoring algorithms. Also, the algorithms themselves are not encoded within the specification itself. Instead the type of algorithm is specified and the necessary parametric information supplied to enable the assessment engine to execute the corresponding algorithm.

2.2 Use Cases

The following representative use-cases have been identified for support within QTI ASI outcomes processing V1.2:

1. Multiple-choice quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete (single response) questions whose outcomes are a dichotomous variable e.g. multiple-choice questions. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "CORRECT" for each Item. This takes on the value "True" if the response was correct and "False' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the percentage of total score obtained by a candidate and (3) the percentage of items actually attempted which were correct;
2. True/false quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete true/false. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "isCorrect" for each Item. This takes on the value '1' if the response was correct and "-1' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the weighted total score;
3. Multiple-response quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are multiple-response questions (two answers for each) with partial scores being awarded when at only one correct answer is given. For the sake of being definite we assume there are 5 Items and that we have a defined outcome, "SCORE" for each Item. This takes on the value "+1" for each correct response "-1' for each incorrect response. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the percentage of items actually attempted which were correct;
4. End of chapter test - this is the same as use-case (a) except that all of the items have different weights. We wish to see (1) the number of correct items, (2) the percentage of total weight answered correctly by the candidate and (3) the "weighted" percentage of questions attempted correctly of those questions answered;
5. English comprehension/composition test with essay - this is a placement test designed to assess English language ability. The test consists of two sections: (1) Short answer items (multiple-choice) and (2) the Essay. Short answer items come in two kinds. The first are 15 discrete vocabulary, grammar and usage Items that all produce a single dichotomous outcome "isCorrect". The second are the reading comprehension Items (although they may actually test other skills than just reading) that consist of a stimulus followed by five discrete questions. We assume that these are done as Items with five responses and hence five outcomes: these are Boolean variables whose meanings are as above. All of these Items can be given different weights (as can the 5 outcomes of the two reading comprehension items). The essay question is graded "A,B,C,D,E,F" by either a human or computer rater. The divisions do not necessarily represent equal spacings. The essay represents 1/3 of the total grade. Out of ten points, the mapping of the grades is as follows A=10, B=8, C=7, D=5, E=2, F=0. The total score for the assessment is required;
6. Diagnostic scoring - this is an additional score report which has been retrofitted onto the 'English comprehension test ((e) above). In addition to the overall score, "diagnostic" scores are going to be given in four different sub-areas: "Reading", "Writing", "Vocabulary" and "Grammar" (the dependencies among these sub-scores and the overall score are ignored for computational simplicity). The 15 discrete items and each of the 10 outcomes from the reading items all load onto these sub-scores differently. Some outcomes are completely irrelevant to some of the sub-scores. An Item/outcome can have a high "vocabulary" load and a low "grammar" load, or vice-versa. The essay is also added onto each of the sub-scores, but with a different loading;
7. Diagnostics complex response biology lite - this is a classroom assessment designed to assess the students understanding of the scientific method as it applies to Biology. The assessment returns not a single score but three scores reflecting "Domain Knowledge" (knowledge of the biological subjects in the assessment), "Methodological Knowledge" (understanding of the scientific method) and "Integrated Knowledge" (the ability to apply the scientific method to a given problem). The items are a series of complex "tasks". Each task gives a piece of scenario of a scientific investigation and has the students work through several steps of the process, answering questions about what they are doing at each step. Each task has many complex responses, and hence many observable outcomes. The outcomes are related to the number of steps in the task. Each is a variable given one of the values "Low", "Medium", or "High" depending on how the student did on one part of one step. The outcomes are labelled "OUT***" where '***' is a sequence number unique to the item. For more definite, assume that there are five tasks: Task1 has 7 outcomes, Task2 has 5, Task3 has 8, Task4 has 6 and Task5 has 13. For each of the reporting variables, the scoring model should add an appropriate "weight" if the outcome has reached an appropriate level (either "Medium" and above or "High"). The weight and the level will, in general, be different for each of the reporting variables. Not all outcomes are used for all of the reporting variables;

2.3 ASI Issues

The core features within the ASI structures that are related to the process of selection and ordering are shown in Figure 2.3.

The relationship of these features to outcomes processing are:

• <qtimetadata> - contains the QTI-specific meta-data about the object. This meta-data may be used to decide which objects are selected to support a particular aggregation mechanism;
• <outcomes_processing> - the set of aggregation processing instructions that are to be applied to the child objects. The variables to contain the aggregated scores are declared using the <outcomes> element;
• <feedback> - the feedback that is to be presented if the conditions defined within the <outcomes_processing> occur;
• <selection_ordering> - the selection and ordering rules that are to be applied to the set of child objects contained within the parent. Only those objects that are presented to the participant can contribute to the scoring algorithm but these may or may not be attempted by the participant;
• Internal and externally referenced Section and Item objects - the set of contained and referenced objects which can be selected and ordered;
• External metadata - the external meta-data descriptions that are linked to the object and which conform to the IMS Meta-data specification.

The scoping of the scoring algorithms that are supported is summarized using the example Section/Item combination shown in Figure 2.4. The scope of the scoring algorithm is:

• For Section 'Section Ident_1' the set of children objects is - 'Item Ident_1', 'Section Ident_2', 'Section Ident-3' and 'Item Ident_2'. This means that the scoring algorithm identified in Section 'Section Ident_1' is applied to the default variables associated with each child object;
• For Section 'Section Ident_2' the set of children objects is - 'Item Ident_2.1', 'Item Ident_2.2' and 'Item Ident_2.3'. The Section-level default variables are formed from the aggregation of these child objects;
• For Section 'Section Ident_3' the set of children objects is - 'Item Ident_3.1', 'Item Ident_3.2' and 'Item Ident_3.3'. The Section-level default variables are formed from the aggregation of these child objects;
• If for all of these three Sections only some of the objects are selected then the corresponding aggregation is based upon those objects presented.

It is the responsibility of the scoring engine that is implementing the scoring algorithm to maintain the scoping of the variables. If the default variable names are used then all of the variables will be named 'SCORE'. For instance, the scoping should ensure that:

Section_Ident_2.SCORE = Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE

At the level of the 'Section_Ident_1' then the variables 'Item_Ident_2.1.SCORE' etc. are only perceived through the variable 'Section_Ident_2.SCORE'.

2.4 Outcomes Processing Solution

The outcomes processing capability is based upon two complimentary mechanisms:

• In-built scoring algorithms - in which the appropriate scoring algorithm is just named and so the corresponding assessment engine is responsible for applying the algorithm to the default input and output variables;
• Defined proprietary algorithms - in which the relationship between the input and output scoring variables is defined using a parameterized approach.

2.4.1 The In-built Scoring Algorithms

The algorithms that are to be supported are:

1. Number of right answers (including multivariate responses) - this is identified by the algorithm name "NumberCorrect" and has associated with it the default integer variable 'COUNT'. This is the store for the number of correctly answered objects. An object is defined as correctly answered if the value assigned to the default Boolean variable 'COUNT' for the object is 'True'. The number correct algorithm can only operate on Boolean scoring variables. The default integer scoring variables COUNT.max (equal to the number of objects attempted), COUNT.min (set to zero) and COUNT.normalized (a value of between zero and one) are also available. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
2. Number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'NumberCorrectAttempted';
3. Weighted number right (including multivariate responses) - this is similar to the 'number of rights answers' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedNumberCorrect". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
4. The weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedNumberCorrectAttempted';
5. Parameterized weighted number correct - name 'ParameterWeightedNumberCorrect' this is similar to the 'WeightedNumberCorrect' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of CORRECT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
6. The parameterized weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedNumberCorrectAttempted';
7. Percentage correct (including multivariate responses) - this score is derived from both the "NumberCorrect" and "NumberCorrectAttempted" algorithms in which the percentage correct is defined by the equation-

1. Sum-of-scores - this is identified by the algorithm name "SumofScores" and has associated with it four default integer variables SCORE, SCORE.min, SCORE.max and SCORE.normalized. These variables store the sum of the scores, the minimum possible score (set to the sum of the 'minvalue' attributes for all of the corresponding objects), the maximum possible score (set to the sum of the 'maxvalue' attributes for all of the corresponding objects) and the normalized score (a value of between zero and one) respectively for the set of objects (the minimum, maximum and normalized values are only valid if they have been explicitly defined for all of the corresponding objects). The value of 'SCORE' is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
2. Sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'SumofScoresAttempted';
3. Weighted sum of scores (including multivariate responses) - this is similar to the 'sum of scores' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedSumofScores". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
4. The weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedSumofScoresAttempted';
5. Parameterized weighted sum of scores - name 'ParameterWeightedSumofScores' this is similar to the 'WeightedSumofScores' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
6. The parameterized weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedSumofScoresAttempted';
7. Best 'K' out of 'N' (including multivariate responses) - this is identified by the algorithm name "BestKofN" and is used to calculate the outcome by using the highest 'K' scores from the set of presented objects generically defined as 'N' objects. The values are placed in the default variables 'SCORE', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
8. Negative scores (including multivariate responses) - called 'GuessingPenalty'. Each object has three associated variables for the values of the number of correct answers ('COUNT.correct'), the number of incorrect answers ('COUNT.incorrect') and the number of unattempted answers ('COUNT.unattempted');
9. Item Response Theory (IRT) - supported using the processing parameter mechanism;
10. Time remaining - for further study in V2.0;
11. Par scores - for further study in V2.0;
12. Cut scores (pass/fail, right/wrong and including multivariate responses) - for further study in V2.0;
13. Multiple forms/equating (table based) - for further study in V2.0;
14. Partial credit/Graded response - for further study;
15. Critical Item - for further study in V2.0;
16. Critical Outcome - for further study in V2.0;
17. Factor analysis/ Multivariate IRT (research) -supported through extensions. For further study in V2.0;
18. Bayes net (research) - supported through extensions. For further study in V2.0;
19. Computer Adaptive Testing - for further study in V2.0.

2.4.2 Defined Proprietary Algorithms

Proprietary algorithms can be used but these must be characterized using the parameter definition mechanisms supported in the specification i.e. the <objects_parameter> and <processing_parameter> elements. Any number of parameters can be passed using these elements and there is no restriction on the naming convention used for these parameters - all parameters are specific to an algorithm. The names of the proprietary algorithms (passed in the scoremodel attribute) should not clash with the intrinsic names and some form of naming convention based upon the creating organization should be adopted.

2.5 Tabular Representation

The tables in this Section provide a conceptual, informative description of the elements in the data objects. The columns in these tables refer to:

No:	The number of the data element. An element may be composed of sub-elements. The numbering scheme reflects these relationships.
Name:	The descriptive name of the element.
Explanation:	A brief functional description of the element.
Required:	Indicates if the element is required: M = Mandatory Element that must be included in the data object, if the element at the higher level is included; C = Conditional Element. Existence is dependent on values of other Elements; O = Optional Element.
Multi:	Multiplicity of the element: Blank = single instance; Number = maximum number of times the element is repeatable; n = multiple occurrences allowed, no limit; Repeatability of an element implies that all sub-elements repeat with the same element.
Type:	A description of formatting rules for the data element. Type includes the maximum length of the element: ID = element used to uniquely identify an object; Code = element value from a list of codes; Description = descriptive element, human language Flag = binary flag Enumerated = list of predefined non-numeric options i.e. the definitive list of objects The international character set specified by ISO 10646 will be used for all fields. The type will also include a description of the set of valid values for the sub-element: Coding schemes using numerical values; The set of values as defined in the Domain i.e. making it closed. The list of values cannot be extended to include values not defined in the specification. If there is a need for values not included in the domain set of values then the extension should be done defining a new element under the Extension element that is a part of each data object definition.

Note: Additional descriptive information about the element.

2.5.1 QTI Outcomes Processing Data Objects

Table 2.1 describes the data objects that are used in the construction of the QTI outcomes processing elements.

No	Name	Explanation	Reqd	Mult	Type	Note
1	outcomes_processing	Accumulated outcomes processing and feedback applied within Sections and Assessments.	O	n
1.1	scoremodel	The type of scoring model being adopted.	O		CDATA string describing the model. String 32 chars.	Default string is "SumofScores".
1.2	qticomment	The comments used to annotate the XML file.	O			Comments should be used to aid human readability of the XML file itself.
1.2.1	xml:lang	The language that is being used for the information.	O		String 32 chars.	The language entries will be defined as per the ISO639 and ISO3166 standards.
1.3	outcomes	To create the variables required for the assessment accumulated scores.	M			The assessment accumulated processing variables group.
1.3.1	qticomment	As per structure 1.2.
1.3.2	decvar	Declaration of a variable to be used for scoring.	M	n		Each type of variable must be declared before it is used.
1.3.2.1	varname	The name of the variable to be declared.	M		String 256 chars.	Default is set as 'SCORE".
1.3.2.2	vartype	The type of variable.	M		Enumerated: String Integer (default) Decimal Scientific Boolean Enumerated Set	Default is set to 'Integer'.
1.3.2.3	defaultval	The default value for the variable.	O		Numerical 32 chars String 32 chars True/False	Can be set to any value. Default is set to '0'.
1.3.2.4	minvalue	The minimum value permitted for a numeric score.	O		String 32chars.	Applies to the value of the score after all of the item processing has been completed.
1.3.2.5	maxvalue	The maximum score permitted for a numeric score.	O		String 32chars.	Applies to the value of the score after all of the item processing has been completed.
1.3.2.6	cutvalue	The value above which the participant will have been defined to have mastery of the subject.	O		String 32chars.	The type of this cut value is set by the variable type.
1.3.2.7	members	The set of enumerated values.	O		String 1024chars.	This is a comma separated list without enclosing parentheses.
1.3.3	interpretvar	The interpretation to be applied to the variable in terms relevant to an actor.	O	n		At present this element will be a comment string however it will be further developed in version 1.2.
1.3.3.1	varname	The name of the input variable being described.	O		As per structure 1.3.2.1.
1.3.3.2	view	The view to which the interpretation is applied.	O	n	Enumerated: All (default) Administrator AdminAuthority Assessor Author Candidate Invigilator Proctor Psychometrician Scorer Tutor	The 'All' view is the default value.
1.4	objects_condition	This contains the conditions that are applied to define the ways in which the outcomes variables are combined to create the aggregated value.	O	n
1.4.1	qticomment	As per structure 1.2.
1.4.2	outcomes_metadata	Contains the rules that are applied to the IMS QTI-specific and/or IMS Meta-data meta-data fields of the object to decide if the object scoring is to be aggregated.	C		String 1-256 chars.
1.4.2.1	mdname	Identifies the IMS QTI-specific or IMS Meta-data field that is to be used for the aggregation rule.	M		String 1-64 chars.
1.4.2.2	mdoperator	Identifies the nature of the meta-data field comparison that is to be applied.	M		Enumerated list: EQ NEQ LT LTE GT GTE
1.4.3	and_objects	Contains the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.	C			See Appendix B for the logic rules.
1.4.3.1	outcomes_metadata	As per structure 1.4.2
1.4.3.2	and_objects	As per structure 1.4.3
1.4.3.3	or_objects	As per structure 1.4.4
1.4.3.4	not_objects	As per structure 1.4.5
1.4.4	or_objects	Contains the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.	C			See Appendix B for the logic rules.
1.4.4.1	outcomes_metadata	As per structure 1.4.2
1.4.4.2	and_objects	As per structure 1.4.3
1.4.4.3	or_objects	As per structure 1.4.4
1.4.4.4	not_objects	As per structure 1.4.5
1.4.5	not_objects	Contains the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.	C			This element contains only ONE of the sub-elements. See Appendix B for the logic rules.
1.4.5.1	outcomes_metadata	As per structure 1.4.2
1.4.5.2	and_objects	As per structure 1.4.3
1.4.5.3	or_objects	As per structure 1.4.4
1.4.5.4	not_objects	As per structure 1.4.5
1.4.6	objects_parameter	Contains the value of a particular parameter that is to be used by the corresponding scoring algorithm variable selection. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters.	O	n		These parameters only apply to the objects within the containing <objects_condition> element.
1.4.6.1	pname	The name of the parameter that is being defined.	M		String 256 chars.	Parameters of the same name may have different significance to different algorithms.
1.4.7	map_input	This element is used to re-map the default input variable to another variable.	O	n
1.4.7.1	varname	The name of the input variable whose name is to be remapped.	O		As per structure 1.3.2.1.
1.4.8	objectscond_extension	The objects condition extension facility.	O	n	ANY
1.5	processing_parameter	The value of a parameter that is passed to the scoring algorithm that is being used.	O	n	String 128 chars.	These parameters apply to the algorithm as a whole and not to specific objects operated on by the algorithm.
1.5.1	pname	The name of the parameter that is being defined.	M		String 256 chars.	Parameters of the same name may have different significance to different algorithms.
1.6	map_output	This is used to remap the result of the scoring algorithm from the default/named variable to another named variable.	O	n	String 256 chars.	The new variable name should have been declared in the <outcomes> element of the parent object. This remapping operates for all of the associated variable names e.g. the *.min , *.max and *.normalized variables.
1.6.1	varname	The name of the output variable whose name is to be remapped.	O		As per structure 1.3.2.1.
1.7	outcomes_feedback_test	Contains the tests to be applied to determine if any and the type of feedback to be presented.	O	n
1.7.1	title	The title of the feedback test.	O		String 256 chars.
1.7.2	test_variable	The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.	M
1.7.2.1	variable_test	The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.	C
1.7.2.1.1	varname	The name of the variable whose state is to be tested.	O		As per structure 1.3.2.1.
1.7.2.1.2	testoperator	Identifies the nature of the variable comparison that is to be applied.	M		Enumerated list: EQ NEQ LT LTE GT GTE
1.7.2.2	and_test	This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.	C
1.7.2.2.1	variable_test	As per structure 1.7.2.1.
1.7.2.2.2	and_test	As per structure 1.7.2.2.
1.7.2.2.3	or_test	As per structure 1.7.2.3.
1.7.2.2.4	not_test	As per structure 1.7.2.4.
1.7.2.3	or_test	This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.	C
1.7.2.3.1	variable_test	As per structure 1.7.2.1.
1.7.2.3.2	and_test	As per structure 1.7.2.2.
1.7.2.3.3	or_test	As per structure 1.7.2.3.
1.7.2.3.4	not_test	As per structure 1.7.2.4.
1.7.2.4	not_test	This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.	C			Only one of the contained sub-elements can be used within each usage of the NOT operator.
1.7.2.4.1	variable_test	As per structure 1.7.2.1.
1.7.2.4.2	and_test	As per structure 1.7.2.2.
1.7.2.4.3	or_test	As per structure 1.7.2.3.
1.7.2.4.4	not_test	As per structure 1.7.2.4.
1.7.3	displayfeedback	The trigger for displaying feedback.	M	n
1.7.3.1	feedbacktype	The type of feedback to be displayed.	M		Enumerated: Response (default) Solution Hint	The default value is 'Response'.
1.7.3.2	linkrefid	The identifier of the material to be referenced.	M		String 32 chars.	Consistency checking is beyond the scope of this specification. Usage rules are given in the Q&TI Best Practice Guide.

3. XML Binding

3.1 outcomes_processing> Elements

Description: The <outcomes_processing> element is the container for all of the outcomes processing instructions for Assessments and Sections. Multiple outcomes processing containers can be use when multiple scoring algorithms are to be applied to produce the aggregated outcomes. If multiple outcomes_processing elements are supplied, it is the intention that all should be run and the outcomes from all of them together should be reported as the outcomes of the enclosing Section or Assessment. The outcome variables defined by each <outcomes_processing> element should be unique across all outcomes_processing elements defined by a section. In particular, it is an error for multiple <outcomes_processing> elements to set the same outcome variable and the results will be undefined.

Multiplicity: The <outcomes_processing> occurs zero or more times within the <assessment> and <section> elements.

Attributes:

• scoremodel (optional. Default='SumofScores'). The type of the scoring algorithm that is to be used.
  Data-type = String (max of 32 chars).

Elements:

• qticomment
• outcomes
• objects_condition
• processing_parameter
• map_outcome
• outcomes_feedback_test

Example:

<outcomes_processing scoremodel="SumofScores">
  <outcomes>
    <decvar/>
  </outcomes>
</outcomes_processing>

3.1.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes processing structure as a whole.

Multiplicity: Occurs zero or once within the <outcomes_processing> element.

Attributes:

• xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
  Data-type = string.

3.1.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0).

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: See sub-section 3.2.

3.1.3 <objects_condition> Element

Description: This element contains the conditions that are applied to define the ways in which the outcomes variables are combined to create the aggregated value using the in-built set of algorithm definitions.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes: See sub-section 3.3.

3.1.4 <processing_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes:

• pname (mandatory). The name of the proprietary parameterized value that is to be used by the scoring algorithm.
  Data-type = string (1-64 chars).

3.1.5 <map_output> Element

Description: This element is used to re-map the named variable to another named variable (given in the body of the element). The target variable name must have been declared using <decvar> in the <outcomes> element of the enclosing <outcomes_processing> element. When a variable is remapped, all of its derived variables are to be remapped as well. Thus if remapping 'SCORE' to 'myScore', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized' would be remapped to 'myScore.min', 'myScore.max' and 'myScore.normalized' respectively.
Data-type = string (1-256 chars).

Multiplicity: This occurs zero or many times within the <outcomes_condition> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is normally used by the scoring algorithm e.g. 'SCORE', 'CORRECT', etc. The default name is 'SCORE' but the variables SCORE.min, SCORE.max and SCORE.normalized are also to be remapped.
  Data-type = String (max of 256 chars).

3.1.6 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: See sub-section 3.4.

3.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0). The declaration of each variable is accompanied by the implicit declaration of the <vname>.min, <vname>.max and <vname>.normalized variables. These are used to store the minimum value, maximum value and normalized value (in the range zero to one) of the scoring variable.

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: None.

Elements:

• qticomment
• decvar
• interpretvar

3.2.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes.

Multiplicity: Occurs zero or once within the <outcomes> element.

Attributes:

• xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
  Data-type = string (1-2048 chars).

3.2.2 <decvar> Element

Description: The <decvar> element permits the declaration of the scoring variables. The default name is 'SCORE' but the variables SCORE.min, SCORE.max and SCORE.normalized are also declared and made available from the scoring algorithms. A <decvar> with no attributes is assumed to define the integer variable 'SCORE' with a default value of zero.

Multiplicity: This occurs one or more times within the <outcomes> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is to be declared.
  Data-type = String (max of 256 chars).
• vartype (required. Enumerated list: String, Decimal, Scientific, Boolean, Integer, Enumerated, Set. Default=Integer). The type of the variable declared.
  Data-type = Enumerated list.
• defaultval (optional). The default value to which the variable is to be initialized.
  Data-type = String (max of 16 chars).
• cutvalue (optional). The value/grade above which the subject is considered to have been mastered.
  Data-type = String (max of 16 chars).
• minvalue (optional). The minimum value permitted for a numeric score.
  Data-type = String (max of 32 chars).
• maxvalue (optional). The maximum value permitted for a numeric score.
  Data-type = String (max of 32 chars).
• members (optional). The set of enumerated values that constitute the member.
  Data-type = String (max of 1024 chars).

3.2.3 <interpretvar> Element

Description: The <interpretvar> element is used to provide statistical interpretation information about the associated variables.

Multiplicity: This occurs zero or more times within the <outcomes> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable whose interpretation details are to be described. The default name is 'SCORE'.
  Data-type = String (max of 256 chars).
• view (optional with selection from the enumerated list of: All, Administrator, AdminAuthority, Assessor, Author, Candidate, InvigilatorProctor, Psychometrician, Scorer, Tutor. Default=All). The view defines the scope for the display of the associated information i.e. to whom the material can be presented.
  Data-type = Enumerated list.

Another description of this element and its sub-components is given in IMS QTI: ASI XML Binding V1.2
[QTI, 02b].

3.3 <objects_condition>

Description: Each <objects_condition> element defines a subset of the objects (Items and Sections) selected by the selection algorithm that are to be used in this score. This allows for the construction of subscores. If no <outcomes_metadata> element is present within the <objects_conditions> element to select a subset of objects, then the <objects_conditions> applies to all objects selected by the selection algorithm. If multiple <objects_conditions> are given within a single <outcomes_processing> element, then the algorithm is applied to the union of all objects selected. The <objects_parameter>, <map_input> and <objectscond_extension> elements within the <objects_condition> element apply only to those objects selected in the condition. If an object is selected by more than one <objects_condition> element, it should receive parameter assignments and input mapping as if it were the first element so selected. These conditions include the identification and mapping of the variables for input.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: None.

Elements:

• qticomment
• outcomes_metadata
• and_objects
• or_objects
• not_objects
• objects_parameter
• map_input
• objectscond_extension

3.3.1 <qticomment> Element

Description: This element contains the comments that are relevant to the presentation of the Item.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes:

• xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
  Data-type = string (1-2048 chars).

3.3.2 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the QTI-specific and/or IMS Meta-data meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule. This data is a string of up to 64 characters length.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.5.

Elements: None.

3.3.3 <and_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.6.

3.3.4 <or_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.7.

3.3.5 <not_out> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.8.

3.3.6 <objects_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm variable selection. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters. These parameters are applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outomes_condition> select the same object, the parameters are taken from the first <outcomes_condition> that selects the element.

Multiplicity: Occurs zero or more times within the <objects_condition> element.

Attributes:

• pname (mandatory). The name of the proprietary parameterized value that is to be used by the scoring algorithm.
  Data-type = string (1-64 chars).

3.3.7 <map_input> Element

Description: This element is used to re-map the named input variable to another variable. The default variable names are derived from the type of scoring algorithm identified in <outcomes_processing> element. The target variable name must have been declared in the evaluation objects that undergoing aggregation and must be of the same type as the default variable. When a variable is remapped, all of its derived variables are to be remapped as well. The input mapping is applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outcomes_condition> select the same object, the input mapping used is the one contained in the first <outcomes_condition> that selects the element.
Data-type = String (1-256 chars).

Multiplicity: This occurs zero or many times within the <objects_condition> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is normally used in the scoring algorithm e.g. 'SCORE', 'COUNT', etc.
  Data-type = String (max of 256 chars).

3.3.8 <objectscond_extension> Element

Description: This element contains the proprietary extensions that can be used to extend the functionally capabilities of the <outcomes_condition> element.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: None.

3.4 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes:

• title (optional). The title of the feedback test.
  Data-type = String (max of 256 chars).

Elements:

• test_variable
• displayfeedback

3.4.1 <test_variable> Elements

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.