From Bunderson 2003, pp. 12-13
USER CENTERED DESIGN
1. Overall appeal.
2. Usability. The instrument will be easy to use, understandable, quick and efficient.
3. Perceived value to the target users, perceived positive consequences.
CONTENT and CONSTRUCT evidence of validity
4. Content coverage and appropriateness.
5. Substantive processes -- The important but typically invisible mental processes
used by those whom we would wish to score as more successful on an instrument,
or affective attributes of persons such as their beliefs, attitudes, and values. It is
only through theories of the cognitive, linguistic, affective or perhaps psychomotor
processes that we can design appropriate questions or performance tasks to get at
different degrees of these usually invisible processes.
6. Structure of the constructs. The starting number of questions or tasks is expected
to collapse into a smaller number of separate unidimensional measurement scales.
The scales we design should correspond with an hypothesized, then increasingly
validated structure.
CRITERION-RELATED
7. Generalizability. Evidence that the scoring methods and scores are reliable, and
generalize to different genders, racial groups, national groups, etc.
8. External. Evidence that the scores predict other valid criteria of what is being
measured; also, evidence that other instruments correlate or do not correlate as
would be expected by the nature of their constructs.
9. Consequential. Evidence that positive results (consequences) do occur over time,
and that unexpected but negative consequences do not occur over time. This is an
extension of the perceived positive consequences listed under category I, above. In
this aspect of validity, we obtain evidence of the actual occurrence of positive or
negative consequences.
USER CENTERED DESIGN
1. Overall appeal.
2. Usability. The instrument will be easy to use, understandable, quick and efficient.
3. Perceived value to the target users, perceived positive consequences.
CONTENT and CONSTRUCT evidence of validity
4. Content coverage and appropriateness.
5. Substantive processes -- The important but typically invisible mental processes
used by those whom we would wish to score as more successful on an instrument,
or affective attributes of persons such as their beliefs, attitudes, and values. It is
only through theories of the cognitive, linguistic, affective or perhaps psychomotor
processes that we can design appropriate questions or performance tasks to get at
different degrees of these usually invisible processes.
6. Structure of the constructs. The starting number of questions or tasks is expected
to collapse into a smaller number of separate unidimensional measurement scales.
The scales we design should correspond with an hypothesized, then increasingly
validated structure.
CRITERION-RELATED
7. Generalizability. Evidence that the scoring methods and scores are reliable, and
generalize to different genders, racial groups, national groups, etc.
8. External. Evidence that the scores predict other valid criteria of what is being
measured; also, evidence that other instruments correlate or do not correlate as
would be expected by the nature of their constructs.
9. Consequential. Evidence that positive results (consequences) do occur over time,
and that unexpected but negative consequences do not occur over time. This is an
extension of the perceived positive consequences listed under category I, above. In
this aspect of validity, we obtain evidence of the actual occurrence of positive or
negative consequences.
Comments