Comparing the NFER paper-based tests with the NFER Online Assessments

Modified on Tue, 20 Aug at 2:00 PM

Some schools planning to use the NFER Online Assessments (NOA) with their pupils may have been using the NFER paper-based tests (NFER Tests) or may also decide to continue using NFER Tests alongside NOA.

NFER Tests results are displayed in reports on the Analysis Tool; teachers may also use the Teacher Guide to turn raw scores into standardised scores or to see which of their pupils are deemed to have met age-related expectations. NOA uses its own separate platform and the NOA reports can be accessed in the NOA reporting tool. For the reasons outlined in more detail below, learner scores should not be directly compared between the two test suites. Instead, interpretation should focus on comparisons of performance within the individual test suites. We recommend:

comparing performance of pupils to the relevant standardisation cohort
monitoring progress over time within one assessment suite
comparing pupils relative to each other on an individual assessment.

Teachers should also bear in mind that individual learner performance may be affected by individual differences in taking a paper test versus an online test (for example, some pupils may find it easier to demonstrate their learning in one medium over the other).

Learner scores should not be compared between different tests

Raw scores or the percentage of correct responses from two tests made up of different questions should never be compared directly. Although both assessments cover the same curriculum areas, new questions have been written specifically for NOA. It is very difficult to create two different assessments with exactly the same level of difficulty, so one test might be slightly more difficult than another and therefore a particular raw score on one test will not mean precisely the same level of attainment as achieving that same raw score on another test.

Changes in learner attainment between trials impact standardised scores

Prior to the development of NOA, NFER Tests were similarly developed and standardised between 2015 and 2018. NOA was standardised during the academic year 2022/23 for years 3 and 4 and will be standardised during 2024/25 for years 5 and 6 ahead of their launch in September 2025.

Standardised scores reflect the performance of an individual relative to the sample that the test was standardised on. Learner attainment in the population is likely to have changed between the paper-test standardisation trials and the NOA standardisation trials due to factors such as the current National Curriculum becoming more embedded over time, and the impact of the Covid-19 pandemic.

This means that we might not expect average learners for the paper-test trials to have performed at the same standard as average learners for NOA trials. If the same learner were to take two assessments, the first standardised on a lower-attaining sample and the second standardised on a higher-attaining sample, then we would expect them to be awarded a higher standardised score on the first assessment than on the second, even if their actual attainment had not changed (ignoring small changes in performance that would happen regardless with repeated attempts at any test). If a pupil receives a lower standardised score on a NOA assessment than on a previous NFER paper test this does not mean that they have not made sufficient progress, just that they are being compared to a different cohort of pupils.

Age-related expectations

The NOA age-related expectations (ARE) report, due to be launched in summer 2025, will indicate to teachers whether or not learners are considered to be achieving or exceeding ARE at the end of the relevant academic year based on their performance in the NOA summer assessment. In summer 2024, NFER carried out a standard setting exercise to determine the ARE thresholds for the NOA year 3 and 4 summer tests. The exercise, involving a representative panel of teachers, also included a review of the existing thresholds for ARE on the equivalent paper-based tests using the same performance descriptors. The performance descriptors are a set of statements defining what a learner is expected to know and understand at the boundaries for achieving and exceeding age-related expectations. This allowed us to validate the ARE thresholds for NOA in comparison to those on NFER Tests and will help us to ensure that schools are supported in making appropriate interpretations of the results from summer 2025.