From a policy perspective, it is understandable that politicians would want to highlight how well education has been doing under their leadership.
This is why you might have heard chancellor Jeremy Hunt describing how “we have risen nine places in the global league tables for maths and reading in the last seven years” in the autumn budget statement.
His claim is not untrue – but, if we zoom out to see the big picture, it starts to look a bit different.
Hunt based his statement on the 2018 Programme for International Student Assessment (Pisa) results for attainment in reading, science and mathematics among 15-year-olds. International rankings are often used in this way: to make the argument that a country’s education is improving.
But these rankings can be a bit problematic. For instance, the Pisa data shows that the UK did indeed rise up the rankings for mathematics and reading. In science, though, the UK’s scores actually fell and the only reason we didn’t drop in the rankings was because other countries also did worse.
Problems with international education rankings
Another complication is that it’s unclear whether Hunt’s claim refers to the whole of the UK or only to England. Pisa data is meant to represent the whole of the UK, but, as researcher John Jerrim has pointed out, there is uncertainty surrounding Pisa results, with clear evidence of bias in the sample for certain parts of the country.
Things also look different when we consider short- and long-term improvements. The Pisa data shows that the UK’s mean performance did not change significantly for reading and science between 2015 and 2018, but it did improve for mathematics.
When we look at prior attainment, we can see that much of that improvement was down to stronger students becoming stronger, while for lower-prior-attaining pupils, there was no significant change.
In fact, none of the subjects showed a long-term significant trend in the share of lower-prior-attaining students. Meanwhile, the share of top-performing students in reading increased significantly, and the share of top-performing students in science decreased significantly.
In other words, it’s quite a mixed picture.
We run into similar problems when we attempt to use Progress in International Reading Literacy Study (Pirls) data.
For example, we sometimes see Pirls cited as evidence that introducing phonics into primary schools has been one the most successful education reforms made since 2010.
Indeed, according to the 2016 Pirls study of the reading ability of nine-year-olds in 50 countries, England had risen to joint eighth place, from joint tenth place in 2011. That rise was largely attributed to improvements in reading for boys and low-prior-attaining children.
But, as we saw with Pisa, it’s hard to show any causal effects with correlational data.
If you look at the Pirls data in more detail, you will see that between 2006 and 2011, England’s scores actually showed a bigger increase than they did between 2011 and 2016. To make any causal claims of the increase being down to more effective education policy, then, you would need to recognise earlier decisions that might not be related to the current government – Jim Rose’s 2006 Independent review of the teaching of early reading comes to mind here.
Having said that, boys did improve more between 2011 and 2016, as did those with low prior attainment.
But scores also come with a measurement error (a level of uncertainty about the score), and if you take that into account, England’s 2011 tenth place could actually have been anything from sixth to fourteenth, while its 2016 eighth place could actually have been anywhere between eighth and fourteenth.
On the whole, then, international rankings are not very meaningful for proving policy is effective. Even when politicians’ claims are based on data, we need to consider how selective or exaggerated they might be.
As I have said before, general claims should be seen as the starting point for exploring a phenomenon, not the final conclusion.
Christian Bokhove is associate professor in mathematics education at the University of Southampton and a specialist in research methodologies
References
