Why international tests don't show the full picture of Manitoba student achievement

"International comparisons … are tricky business, and simple conclusions such as these are unwarranted." 

Such was the lament of Harvard education professor Daniel Koretz in his book Measuring Up: What Educational Testing Really Tells Us, in response to a state education department's surface analysis of PISA scores.

Big data is great, but we need to be careful about how we use it and in what context — especially in light of what the latest results from the standardized international test suggest about Manitoba students.

Every three years, the Organization for Economic Co-operation and Development, or OECD, administers the Programme for International Student Assessment (PISA) test in approximately 70 countries.

The PISA tests, since 2000, have sought to help the OECD take the intellectual temperature of 15-year-olds throughout the world in terms of mathematics, reading and science competencies.

Canada has done exceedingly well in these tests, despite being the only OECD partner without a federal education system. In the 2018 results, released earlier this week, Canada ranked sixth in the world.

Drilling further, Manitoba's data ranks near the bottom of the list, although our province is still above the OECD average, on par with the Netherlands, France, and the Czech Republic.

As we sift through the hefty 2018 PISA results, it is important to acknowledge a few things.

First, all sectors in Manitoba, including education, have a tremendous amount of work to do to ensure that all Manitobans have access to high-quality social services. If we are to improve the outcomes for our young people, we need to ensure that all families have the means to a decent life, not just some.

Second, PISA assesses competencies, not the outcomes of learners — curricular outcomes are not assessed specific to a province or country, nor is the whole child considered. The test does not highlight how learners are flourishing or their well-being, nor does it follow learners as cohorts.

It's a snapshot that does not tell the whole story. 

One snapshot

For instance, despite Singapore's incredible results on the PISA tests, the subsequent OECD survey on student well-being reveals that its learners suffer from intense anxiety brought about by school. 

Any test is a snapshot, including the provincial Grade 3, 7, and 12 assessment currently employed in Manitoba. We have assessment at these points, including the early development instrument, to help us look at cohorts of learners in school divisions. We can observe the development of a group of learners from preschool to graduation.

The data is paramount to informing our teacher moves, as we can follow the progress of learners from kindergarten all the way to post-secondary. But one data set is just that. One snapshot.

There is an oft-spoken adage that 'things were a lot better when I was in school.' In fact, education in Manitoba is better now.

PISA also does not assess critical thinking, historical thinking, ecological literacy, empathy, communication skills, and soft skills — all characteristics explicitly stated by an RBC report called Humans Wanted, focused on how today's young people can succeed in the workforce they'll step into.

The report argues that with the changing nature of work, we need a citizenry that can collaborate, communicate, solve complex problems and imagine the plight of others. As our society becomes more automated, we need thinkers.

Lastly, as educators, we need to continue to get better at our craft. Any teacher worth his or her salt is perpetually trying to improve practice.

We know from the evidence what matters in terms of student learning: the day-to-day interaction between educator and learner, as evidenced by British education expert Dylan Wiliam, and the collective efficacy of teachers, as argued by New Zealand's John Hattie.

The evidence from researchers like Wiliam and Hattie shows what works is strong formative assessment, coupled with groups of educators who fundamentally believe they are making a difference in the lives of their learners.

More standardized testing, according to meta-analyses, has a minimal effect size on learning.

Education is getting better

Public education is a relatively novel enterprise on this planet, including in Canada. Originally an option for the rich and intended for moral and religious instruction, public education since Confederation has evolved from a utilitarian practice for fuelling industry in the early 20th century, to a reaction to the atrocities of the Second World War, to learner-centered stances in the 1960s.

Now, we've reached a convergence of competing values — filling talent pools on one hand, the development of global citizens on the other.

But the quality of our public school system is a reflection of the strength of our democracy and society. This should be at the forefront of our conversations.

What's important in the wake of the latest PISA results is that public education has not become worse. There is an oft-spoken adage that "things were a lot better when I was in school." 

In fact, education in Manitoba is better now.

Since the inception of public education in the 19th century in Manitoba, we have asked learners to learn more and perform more complex skills. 

We are asking learners to launch high-altitude balloons into space to conduct experiments, to design and operate robots, to head out into internships, to connect with experts all over the world, to write and publish their own books, and to think like scientists, historians, anthropologists, philosophers, and mathematicians — all to prepare them for some existential crises which affect the whole planet, and to create sustainable communities.

And we now include everyone. Only a few generations ago, we asked learners entering Grade 10 to pay for their own books. We used to relegate learners with special needs to off-campus programs, and the marginalized and vulnerable got the message that school was not for them. 

No longer. Schools now mitigate all levels of social problems and have become the community hubs for essential services and connection. Every teacher, administrator, and school endeavours to ensure that all our learners, no matter who they are, belong.

An all-hands-on-deck approach needs to be considered by Manitobans. If we are truly interested in ameliorating the outcomes of our youth, then we need to ensure that all sectors and systems are working together. 

To rely purely on international tests to inform our collective work simply is inadequate and will not create the sustainable and democratic society will all want.

This column is part of CBC's Opinion section. For more information about this section, please read this editor's blog and our FAQ.

• Read more opinion pieces published by CBC Manitoba.