Uneven learning, but not necessarily unknown: Insights from the DfE research and Star Assessments

In our recent blog post, you will have seen an overview of the latest findings from our joint project with the Department for Education (DfE) and Education Policy Institute (EPI). This innovative research into the impact of COVID-19 on pupil attainment in England uses Renaissance Star Assessment data and information from the National Pupil Database to provide a robust estimate of the learning loss experienced by pupils in years 3-9 over this unique year.

We know that many Renaissance users have already taken actions to assess the impact of this disruption on their pupils. This latest release offers an insight into the national picture and provides a valuable benchmark with which to compare your pupils, with breakdowns provided by age, socio-economic status, and by region. So, if you are looking to get an idea of where your cohort sits, where should you begin? This blog aims to provide you with the practical steps needed to gauge the impact on your pupils and benchmark against the findings of this research.

The power of the Scaled Score

Much of the focus around these reports has spoken of the months of learning loss experienced by pupils, but such estimates are all based on the Renaissance Star Scaled Score. Many Star users will be comfortable with one or more of the scores provided after an assessment, whether that’s the Reading Age, NRSS, PR or SGP, but in many ways the Scaled Score is the unsung hero that ties them all together. From this raw score, both our criterion-referenced and normed-referenced scores are developed offering relative and actual attainment information for teachers. This single scale across primary and secondary is also linked to the national curriculum through Renaissance’s learning progression, showing teachers where a pupil is working and what they are ready to learn next (more on that to come!).

Now, this Scaled Score can also be used to compare your Star Reading and Maths results with the findings from the latest reports, giving you a vital indication on how pupils have fared following the disruption to their learning since March 2020. Much like the Renaissance KS2 SATs linking study, this research provides a Scaled Score benchmark which you can apply to your Star results. Of course, no one can know the precise impact of the pandemic on children’s learning, and these estimates should not be viewed as finite as there is some uncertainty around the estimates of learning loss. However, they do provide a robust indication at a country-wide level of how pupil’s learning was impacted, and we hope they can give some reassurance and provide a new perspective to your assessments.

What does this research tell us?

The graphs below show the mean Scaled Scores in reading and maths in the first half of the autumn term in 2019/20 and 2020/21. These are presented for year groups 3, 6 and 9 in reading and 3 and 6 for maths, but the intervening years are also available in the Annex tables of Report 2, and at the bottom of this blog. As the Star attainment data was matched with pupil records in the National Pupil Database, you will also see breakdowns by several key characteristics and ethnicities, although sample sizes are smaller for these breakdowns so do interpret the results with some caution.

You will notice variations in the proportion of the change between the first bar to the second bar for each year group, as well as in the starting points from the pre-pandemic baseline. As is the pattern seen in national outcomes, historically pupils taking the Star Assessments perform better if they are female for reading or male for maths, worse if they are FSM Ever 6 or SEND, and EAL pupils who have recently arrived in the state school system in the last two years are at a disadvantage to other EAL pupils. The similarity in the pattern of these outcomes compared with those of national assessments, increases the confidence that the findings from this research on learning loss provide an accurate picture of the impact of time out of the classroom.

As well as these mean Scaled Scores by year group and characteristic, EPI also calculated the average difference in Scaled Score points by sub-group. These average differences provide a rough measure by which you can compare changes in your own results year on year, and these are available in the Annex tables A8, A10 and A11.

Overall, EPI found that for primary aged pupils in years 3 and 6, the average results in reading were between 10 and 30 Scaled Score points lower in the first half of the autumn term in 2020/21 across all characteristic groups when compared with results in 2019/20. For year 9 pupils in secondary schools, the average results in reading in 2020/21 were broadly the same as in 2019/20.

In mathematics, results in 2020/21 were substantially lower than in 2019/20 across all characteristic groups – apart from EAL recent arrival pupils. On average, year 3 pupils scored around 35 Scaled Score points lower in 2020/21 than in 2019/20, year 6 pupils scored around 20 points lower than in 2019/20.

How to find the Scaled Score for your pupils

The estimates provided in this research are based on very large sample sizes and this means there may have more variation if you were to look at an individual pupil or small group. Looking at these averages across a year group will give you the best comparison and you can find the average Scaled Score for a year group by using the Star Reading and/or Star Maths Summary Report. Currently, the average Scaled Scores are provided for Autumn 1, so you will want to view the report for this period only by narrowing down the dates in the Reporting Period. Then choose the option to Group By year group. You could also select yes to Summary Only to save generating all pupil results.

As with all Renaissance reports, you can easily filter for a specific demographic using a Reporting Parameter Group. You will first need to ensure that characteristics of interest are assigned to pupils, either by assigning them manually to a single pupil or by adding them to multiple pupils through an import or through Users > Edit multiple pupils. From there you can build the reporting parameter groups required.

If your pupils used Star Assessments in the previous academic year and you would like to benchmark them against the national average from 2019/20 or view the change between these two periods, you can use the Star Growth Report, or change the dates of the Summary Report to the corresponding period in the previous year. Alternatively, you can go to School Years > Work in a Different School Year and select 2019-2020 and then follow the above instructions to pull the report.

What next?

Now you have a sense of where your pupils are compared to the national picture, you may well be asking how you can best accelerate their learning and hasten the recovery from the time lost to learning. As mentioned above, Renaissance Star Assessments are built around an empirically validated learning progression which was created in collaboration with the NFER. This not only pinpoints where a pupil is on the national curriculum based on their Scaled Score for reading and maths, but even highlight the Focus Skills™ that are essential to a pupil’s progression.

A personalised report is available via the Star Reading and Maths instructional planning reports, for both individual pupils, as well as groups of pupils, and Renaissance have also released free teacher workbooks of the Focus Skills to give everyone the opportunity to prioritise the most important skills in the curriculum. It should be noted that it is not that other skills do not have value, but rather these Focus Skills are essential pre-requisites that open up the curriculum to pupils and help to accelerate their progression. A full blog to help explain this can be found here.

Want to know more?

We have provided responses to some questions below, but if you want to know more about any of the information discussed in this blog, please don’t hesitate to reach out! We will also be holding a panel discussion to discuss this research further on Thursday 1^st July at 12.30pm. We hope to see you there.

Register

FAQ

Why am I not able to calculate the learning loss in months for my pupils?

Unfortunately, it is not possible to replicate the process of finding the learning loss in months for your students as the models used would not be accurate at the smaller scale of a single class or year group from one school.

A breakdown of the methodology used to create the estimates of learning loss in months is provided in the first Interim Report which was released in February. This methodology consisted of taking millions of historic data points from the 2017/18 academic year onwards to plot attainment trends across different year groups and demographics. This prior attainment information was used to create estimates of the expected outcomes for the 2020/21 academic year if there had been no disruption to learning.

This was all done using the Renaissance Star Scaled Score as discussed above, and regression models were created to allow for different rates of progress. The below equation was then used to create the learning loss in months.

What about the Reading Age and months progress seen in this score?

As above, the months learning loss discussed in these reports is specific to this research and should not be confused or compared with months change which might be seen via the Renaissance Star Reading Age, as this is a very different measure.

The Renaissance Reading Ages provided following a Star Reading assessment are a criterion-referenced score representing a bracket of Scaled Score points, and as such, they are not the most robust means of measuring student progress. Reading Age does not take into account the pupil’s age and a pupil could grow by several scaled score points but not change in terms of Reading Age. If you are looking to measure granular progress, the Scaled Score is the most sensitive score, whilst the normed-referenced scores such as the NRSS and PR will give a stronger indication of whether a student is on track based on their age and year group. The SGP will also compare students with their national peers, and let you know whether the progress they are making is not only good for their age, but also for their ability. If you would like to know more about Star Assessments there is information here or reach out to our team.

What about looking ahead?

Not only can the average Scaled Scores provided in this research be used to benchmark your students against the national picture in this unique year, it also affords the opportunity to compare where your year groups are now ahead of next autumn. If, for example, you are assessing Year 4 this term you could compare their average Scaled Score with that of Year 5 in Autumn of either 19/20 or 20/21 to see if they are on track.

As star assessments are not used by all students, is this study representative of the wider student population?

Yes. Matching with the National Pupil Database allowed EPI to examine the demographic breakdown of pupils who sit Star Assessments, and found that overall, the characteristics of Renaissance pupils corresponded to pupils nationally. The proportions of Renaissance pupils who were boys, who were eligible for free school meals, who had English as an additional language, or who had an identified special educational need or disability, were very close to the proportion of pupils nationally that had each characteristic. This was true amongst both primary and secondary aged pupils.

Renaissance pupils were slightly more likely to be from white backgrounds and had a range of prior attainment in national curriculum assessments at Key Stage 1 and Key Stage 2. They were slightly less likely than average to be assessed as being below the expected standard at Key Stage 1 and slightly less likely to be either below or above the expected standard at Key Stage 2.

There is more detail provided in Report 2, and such analysis increases the confidence that the patterns of results we see in this data are likely to reflect outcomes for the pupil population as a whole.

Can I see if my region was more affected than others?

Yes. The average loss by Scaled Score was broken down into regions for Autumn 1 and Autumn 2 and this information is provided in Report 2 in figures 3.3, 3.4, 4.5 and 4.6, or in annex tables A8, A10 and A11. Again, smaller sample sizes do play a role here, so please do interpret with caution.

Data Tables

Table A1: Mean scaled scores in reading in the first half of the autumn term 2019/20 – 2020/21 for all year groups by characteristics for figure 2.1

Table A2: Mean scaled scores in mathematics in the first half of the autumn term 2019/20 – 2020/21 for all year groups by characteristics for figure 2.2

Table A8: Estimated learning loss, in scaled score points, and pupil numbers by sub-group for figures 3.3 and 3.4

Note: Asterisks indicate sub-groups where the sample is less than 500 pupils and as a result some caution should be taken with interpreting the estimate.

Table A10: Estimated learning loss, in scaled score points, and pupil numbers for reading by sub-group for figures 4.5

Note: Asterisks indicate sub-groups where the sample is less than 500 pupils and as a result some caution should be taken with interpreting the estimate.

Table A11: Estimated learning loss, in scaled score points, and pupil numbers for mathematics by sub-group for figure 4.6

Note: Asterisks indicate sub-groups where the sample is less than 500 pupils and as a result some caution should be taken with interpreting the estimate.