Norming, or calibration, is the process of faculty members rating students’ work together while applying the rubric. This practice helps to assess students’ work in a consistent and accurate manner regardless of who is assessing the work. Norming is normally led by a facilitator, someone from the department or outside, who is experienced in the norming process.
Suggested materials for a norming session:
- Copies of the sample rubric
- Copies of students’ work that exemplify each level of mastery. Student work should include pieces that fall under low, medium, and high level of mastery
- Score cards
- Extra paper, pens, post-its, paper clips, stapler, etc.
Have the norming session in a space that:
- Enables the scorers to comfortably rate students’ work
- Has a white board, markers, flip chart and/or smart board
The norming process entails:
- Background: Participants should be informed about the purpose of the activity, the use of gathered data, how the session is connected to the program assessment plan, the anticipated time for the session, and the expected outcomes from the session. Inform participants that the objective is to assess the program, not individual students or faculty, and discuss the ethical guidelines, including matters of confidentiality and privacy.
- Materials: Describe the nature of the products that will be assessed. Provide each participant with copies of several student products that exemplify each level of mastery. Ask each participant to independently apply the rubric to each of these products, writing their ratings on a piece of paper.
- Rubric Sample: Explain the scoring rubric and its categories. Give details on how it was developed. Participants should take time to familiarize themselves with the rubric. Clarify the type of rubric that will be used for assessment and whether the norming session is for one, some, or all elements of the rubric.
- Individual Ratings: Participants should take their time to individually read and assess the pieces of work. When everyone is done, gather everyone’s ratings and display them so that everyone can see the degree of agreement. This can be done on a white board, with each person stating their ratings as they note them down on the board. Otherwise, the facilitator could ask raters to raise their hands once their rating category is stated, which would allow to view the extent of agreement between everyone and it would help to spot raters who normally give unusually high or low ratings.
- Discussion and Consensus: Have a discussion on the ratings. There will be differences in opinion. However, the discussion is useful to establish standards. Try to reach an agreement about the suitable rating for each of the products being assessed by asking participants who provided different ratings to explain their decision. Raters should try to explain their decision by referring to the rubric. Reaching a consensus upon rating is possible; however, sometimes a split decision occurs (e.g., the group agrees that a product is a “1-2” split since it has qualities of both categories). This should not be an issue. You may let the group review the rubric to clarify its use, but preclude them from deviating from the rubric and learning outcome(s) being assessed.
- Repeat as Needed: When the group is comfortable with the rubric, the rating process is repeated. Inform participants how to log ratings using the score cards and explain the procedures. Reviewers begin scoring. After the scoring process, summarize the scores and present a brief summary to the group. You may conclude the meeting with a discussion of five questions:
- Are results consistent?
- What do the results reveal? What do we think about the results in relation to students’ learning?
- Who should the results be shared with?
- How are the results related to curriculum, pedagogy, or student support services, e.g. the Writing Center?
- How can the assessment process improve?
References
University of Hawai’i, Mānoa. (2017, August 29). Creating and using rubrics. Retrieved from https://manoa.hawaii.edu/assessment/howto/rubrics.htm
Washington State University. (2016, June). Quick guide to norming on student work for program-level assessment. Retrieved from https://atl.wsu.edu/documents/2015/03/rubrics-norming.pdf/