Annex G

Post-hoc Mark Scaling Guidance for APACs

Definition

Scaling is the systematic adjustment of a set of marks for a module/assessment in order to ensure that they properly reflect the achievements of the students in meeting the module learning outcomes.

This document provides guidance on the application of post-hoc mark scaling as a corrective measure, as part of the marking/moderation process. It applies to scaling subsequent to the final marking and moderation process, when applied by the Programme/Discipline APAC (Assessment, Progression and Awarding Committee).

Although scaling may be applied in other circumstances (such as translating marks from overseas or converting raw assessment marks into standard university assessment marks) those circumstances are not covered by this guidance.

Scaling, as a corrective measure, is different to mitigation which provides corrective action for an individual student.

Principles

In applying scaling, disciplines should adhere to the following principles:

1) Scaling should be used only in exceptional circumstances. When circumstances have been identified that lead to the need for scaling to be applied, it is expected that action is taken to avoid these circumstances recurring. Scaling should not be relied upon to address recurring problems. When assessment marks differ from expectations, effort should be made to identify a cause for this, and this should be a part of the consideration when an appropriate adjustment is agreed. Also see 1.4.5g within the Exceptional Circumstances Handbook.

2) Scaling should be used to correct instances where student marks do not accurately reflect their achievement and learning outcomes. It is not intended that scaling is used to apply a norm-based marking system. Scaling decisions are made through academic judgement. Marks after scaling should be a more accurate reflection of student performance in relation to the intended learning outcomes, than the marks before scaling.

3) Scaling should not advantage students, but simply correct any unfair disadvantage or unfair advantage.

4) The action of scaling should be applied equally to all students within the relevant group. Normally this will be all students on the module, but it may be all students within a particular teaching group, or taking a particular assessment. The algorithm used for scaling may lead to different changes at different mark levels, but the algorithm should be applied to all students equally.

5) Scaling should be applied anonymously. Only the following information is expected to be relevant to determining a new mark for a student:

a) Whether they are part of the cohort to be scaled
b) The original mark

6) The application of scaling must not change the rank order of students within the assessment (or module, if applied at the module level). However, it is accepted that when scaling is applied at the assessment level, it may change the rank order within the module as a whole.

7) Scaling does not replace moderation. Where concerns have been identified with the marking process at moderation, corrective action should be taken at this stage.

8) Scaling should be applied at the appropriate level to correct the original issue. It is expected that this would usually mean assessment level, whenever possible. Assessments or modules should not be scaled to compensate for problems in other assessments or modules.

9) In applying scaling, disciplines should be mindful of the size of the cohort, and whether any variation from expected performance is statistically significant. In very small cohorts scaling is unlikely to be appropriate unless supported by other evidence to show that the marks do not appropriately reflect the students’ academic performance. Disciplines should also bear in mind that the scaling of one module may affect the information used to determine the appropriateness of scaling in other modules.

10) External examiners must be consulted about whether to scale and how that scaling should be achieved

11) Clear records of scaling decisions must be kept. Scaling decisions should be recorded within APAC minutes, and reported to the Faculty APAC. Details of what should be recorded are outlined below.

How to identify assessments/modules where scaling would be appropriate

Disciplines are expected to review module assessment data as part of the APAC process.

It is important that the process of deciding upon and applying scaling is determined by academic judgement. Consideration must also be given to the circumstances that led to the original marks being different to expectations. The guidance below suggests some possible triggers that may lead to further investigation of a module. However, just because a module meets the relevant trigger, it does not necessarily mean that scaling is appropriate.

Examples of reasons why scaling may not be appropriate, despite the module meeting one of the triggers listed below, include:

• The size of the module means that the difference in performance between modules/years is not statistically significant
• The performance of the cohort is not comparable to previous years as a result of different entry standards
• Professional or regulatory requirements mean that specific modules require a particular standard to be attained and so scaling is not permissible
• The module assesses particular skills (e.g. statistics/data analysis) that are different to the other modules taken, so performance across modules is not comparable.
• The module is typically taken by students from other disciplines. Therefore their performance in their home discipline is not comparable.

Modules, or assessments within modules, might be identified for consideration as to whether scaling might be appropriate in the following situations: 

1. The performance of students in the assessment/module is significantly different to their performance in other modules taken in that academic stage.

2. The average assessment/module performance of students is significantly different to the average of student performance in the same module taken in previous years.

3. The distribution of marks is highly unusual. This might include an abnormally large number of fails, an abnormally large number of marks in the 1st class/distinction range, or a pattern of marks (such as bunching) which is abnormal.

4. Other information leads us to believe that the marks given for this assessment/module do not reflect the actual performance of students in the assessments.

How to apply scaling

It is important that the principles outlined above are adhered to when scaling is applied.

Examples of approaches to scaling marks include:

1) Addition or subtraction of a fixed number of marks.
2) Multiplication of all marks by a particular factor (eg. 1.02 or 0.97).

The same alteration does not need to be applied throughout the range, as long as care is taken to ensure that the rank order of students is not affected.

Care should also be taken with respect to the following:
• No mark can be given that is greater than 100 or less than zero.
• No mark that was originally greater than zero should be reduced to zero, unless the discipline is satisfied that there are clear reasons to do so.
• If an assessment mark was capped due to late submission or academic misconduct then the uncapped mark should be scaled, and then the capping reapplied. If scaling is being applied to the whole module then this does not apply.
• If a module was capped due to referral or academic misconduct then the uncapped mark should be scaled, and then the capping reapplied.
• If the reason for scaling would apply equally to referral/deferral assessment marks, then they should also be scaled accordingly. For example, if the reason for scaling related to an issue with the original teaching provision, then the referral/deferral assessments would be equally affected, and so should also be scaled. If the issue was related to the delivery or content of an exam, and the issue did not affect the referral/deferral paper, then scaling would not need to be applied to the referral/deferral assessment mark.

When to apply scaling

Post-hoc mark scaling should be applied after the marking and moderation process is complete. It should be undertaken as part of the discipline/programme APAC process, either in the APAC meeting or in a pre-meeting before the APAC.

Scaling should be undertaken in the academic year in which the module was delivered. Once marks have been finalised by the APAC it is not anticipated that any scaling would be subsequently applied.

Who should be consulted?

Post-hoc mark scaling is undertaken by discipline/programme APACs. In the event that scaling decisions are made outside of the APAC meeting, then they require the approval of the Chair (or person to whom ‘Chair’s action’ powers have been delegated).

External examiners should be consulted on any scaling action, either as part of the APAC, or in a separate communication.

It is strongly recommended that the module convener is consulted about decisions to scale their module. Their input is likely to be required in understanding the reason why scaling is necessary. However, there may be times when they are not available to be consulted, so their approval is not essential.

Directors of Education and Student Experience or Programme Directors should be able to put the performance of students on the module within the context of programmes overall. Therefore their input is likely to be valuable. However, they would normally be invited to the APAC meeting, or should send a delegate if there are unable to attend. Therefore their approval is not essential.

Record of Scaling

Accurate records should be kept of scaling decisions. The following should be recorded within the APAC minutes, or, if decided outside of an APAC through Chair’s action, be reported to the next appropriate APAC meeting:

• What was scaled, ie. which module/assessment, and whether it applied to all students on the module or a sub-cohort
• The rationale for the scaling
• Exactly what was done, with enough detail that the action could be repeated on the original set of results and achieve the same outcome
• Any future actions to prevent the need for scaling on this assessment/module again
• Any specific communications required to students or staff
• Confirmation that the decision to scale was approved by the external examiner, and any other relevant APAC members as detailed in the TQA manual.

Discipline/Programme APACs must report their scaling decisions to the Faculty APAC. A report on scaling decisions within the Faculty will also be required for the University APAC.

Records must also be kept of the original marks prior to scaling.

Referral/Deferral APACs must be informed of scaling at the original APAC, so that a decision can be made on whether it is appropriate to apply the same scaling to the referral/deferral marks.

Informing Students

Students should be informed of the final module and assessment marks following the APAC, as part of the normal mark release process.

Students should be made aware that any marks released prior to the APAC are provisional, and therefore potentially subject to change.

When agreeing on scaling, and discussing a rationale, the APAC should also discuss whether any specific communications should be sent to students to explain the action taken. Where an exceptional year has meant that multiple modules have been scaled a single communication to all students covering all scaling decisions may be sufficient. Similarly, in an exceptional year when students might anticipate scaling, it may be appropriate to communicate why the decision has been taken not to scale.

It is the responsibility of the Chair of the APAC to oversee any communications to students regarding scaling.