Developing Assessment Instrument
Criterion-Referenced Tests
Criterion-referenced tests are part of the formative evaluation and are important for evaluating both the learners progress and the instructional quality. Tests are used to examine a person's or group's achievement in a carefully defined content area. They focus on specific goals and objectives with a given content area.
4 types of criterion-referenced tests
1. Entry Skills Test
2. Pretests
3. Practice Tests
4. Posttests
Criterion-Referenced Tests
Criterion-referenced tests are part of the formative evaluation and are important for evaluating both the learners progress and the instructional quality. Tests are used to examine a person's or group's achievement in a carefully defined content area. They focus on specific goals and objectives with a given content area.
4 types of criterion-referenced tests
1. Entry Skills Test
2. Pretests
3. Practice Tests
4. Posttests
Norm-Referenced Tests
Norm-referenced tests are used to compare relative performance of learners in larger areas of content such as a year's worth of content within a specific subject area. These tests are part of the summative evaluation.
Test Design
Verbal information: short-answer, alternative response, matching, or multiple choice
Intellectual skills: objective-style test items, creation of product (e.g. widget), or live performance of some type
Attitudes: require either that a learners state their preferences or that the instructor observes the learners' behavior and inters their attitudes form their actions.
Psychomotor: typically requires the learner to perform a sequence of steps that collectively represents the instructional goal.
Objective tests: include test items that are easy for learners to complete and designers to score (e.g. short answer, true/false, matching, and multiple choice).
Alternative assessment instruments: used to measure performance products and attitudes.
5 steps to developing instrument
1. Identify the elements to be evaluated
2. Paraphrase each element
3. Sequence the elements on the instrument
4. Develop response format: checklist (yes or no), rating scale (e.g. poor, adequate, good), frequency count
5. Determine how the instrument will be scored
Portfolio assessments: collections of criterion-referenced assessments that illustrate learners' work.
5 criteria
1. should be very important and warrant the increased time required for assessment format
2. work samples must be anchored to specific instructional goals and performance objectives
3. work samples should be criterion-referenced assessments that are collected during instruction process
4. assessments are the regular pretests and posttests
5. each regular assessment is accompanied by its rubric with student's response evaluated and scored, indicating strengths and weaknesses.
Test Item Criteria
There are four types of test item criteria.
1. Goal-centered
2. Learner-centered
3. Context-centered
4. Assessment-centered
Norm-referenced tests are used to compare relative performance of learners in larger areas of content such as a year's worth of content within a specific subject area. These tests are part of the summative evaluation.
Test Design
Verbal information: short-answer, alternative response, matching, or multiple choice
Intellectual skills: objective-style test items, creation of product (e.g. widget), or live performance of some type
Attitudes: require either that a learners state their preferences or that the instructor observes the learners' behavior and inters their attitudes form their actions.
Psychomotor: typically requires the learner to perform a sequence of steps that collectively represents the instructional goal.
Objective tests: include test items that are easy for learners to complete and designers to score (e.g. short answer, true/false, matching, and multiple choice).
Alternative assessment instruments: used to measure performance products and attitudes.
5 steps to developing instrument
1. Identify the elements to be evaluated
2. Paraphrase each element
3. Sequence the elements on the instrument
4. Develop response format: checklist (yes or no), rating scale (e.g. poor, adequate, good), frequency count
5. Determine how the instrument will be scored
Portfolio assessments: collections of criterion-referenced assessments that illustrate learners' work.
5 criteria
1. should be very important and warrant the increased time required for assessment format
2. work samples must be anchored to specific instructional goals and performance objectives
3. work samples should be criterion-referenced assessments that are collected during instruction process
4. assessments are the regular pretests and posttests
5. each regular assessment is accompanied by its rubric with student's response evaluated and scored, indicating strengths and weaknesses.
Test Item Criteria
There are four types of test item criteria.
1. Goal-centered
2. Learner-centered
3. Context-centered
4. Assessment-centered
Congruence of Design Process
It is imperative that the skills, objectives, and assessments all refer to the same skills.
5 Procedures for Evaluating the Design
1. Organize and present the materials to illuminate their relationships.
2. Use goal-centered criteria to judge the congruence among materials.
3. Compare materials and the characteristics of the learners.
4. Judge congruence between the performance and learning contexts with the performance objectives and test items
5. Judge clarity of all materials
Reflections
In public health, there are process and outcome evaluations to health initiatives. There is a lack of process evaluations either because they don't know how to be measured or because they are in such a hurry that they forget. I can see how the criteria-referenced tests could be missed or done poorly, especially when there is a time crunch. I see the importance of evaluating along the way especially when it could help with the development of the instruction. Without the formative/process evaluation, it can be a waste of time for the learner and the instructor if the training does not meet the instructional goals and objectives.
It is imperative that the skills, objectives, and assessments all refer to the same skills.
5 Procedures for Evaluating the Design
1. Organize and present the materials to illuminate their relationships.
2. Use goal-centered criteria to judge the congruence among materials.
3. Compare materials and the characteristics of the learners.
4. Judge congruence between the performance and learning contexts with the performance objectives and test items
5. Judge clarity of all materials
Reflections
In public health, there are process and outcome evaluations to health initiatives. There is a lack of process evaluations either because they don't know how to be measured or because they are in such a hurry that they forget. I can see how the criteria-referenced tests could be missed or done poorly, especially when there is a time crunch. I see the importance of evaluating along the way especially when it could help with the development of the instruction. Without the formative/process evaluation, it can be a waste of time for the learner and the instructor if the training does not meet the instructional goals and objectives.