The results for Portugal indicated a very sizable improvement, when compared to 2006 and earlier results (link). For instance, the reading score increased from 472 to 489, placing Portugal at the OECD mean - and above countries such as Spain, the Czech Republic, Slovakia or Austria. The government hailed these findings as vindication for all the reforms introduced since 2005; while many teacher blogs (link) raised questions about the validity of the results. These questions became more vocal when the education ministry refused to release the list of presumably randomly-selected schools in the survey (link).
As someone that has studied the effects of one of the reforms introduced by the former government - the new teachers' assessment and incentives structure -, I too am curious about the validity of these 2009 PISA results for Portugal. Especially, as I found evidence that those teacher reforms had a detrimental effect on student achievement (link). So to shed some light on this issue I analysed the PISA micro data (available from here) from 2006 and 2009 but splitting the weighted results between public and private schools and not simply looking at their mean.
In the case of reading, this is what I found:
First, perhaps unsurprisingly, the results indicate that students in private schools appear to do better than students in public schools, both in 2006 and 2009. Second - and more important -, the gap between private and public schools remains virtually the same over the three-year period.
The results are very similar for sciences and maths:
According to my previous research, I actually expected the public-private gap to widen. On the other hand, it's difficult to reconcile the government view that the improvement in Portugal's PISA results was due to its reforms with the fact that private schools see their PISA scores increase by a similar level than public schools. For instance, and as far as I know, the teachers' assessment structure in private schools was not subject to the same adjustments as in public schools.
If one looks at the case of Spain, the pattern there is much more mixed - improvements in reading, as in Portugal, but stable results in sciences and maths, at both public and private schools:
To wrap up, I am not sure what explains this massive improvement in the perceived achievement of Portuguese students in the two types of schools. Some hypothesis are:
-successful reforms implemented by the government matched by equally successful reforms implemented by private schools, even if in a more decentralised manner;
-composition changes between private and public schools, in such way that both means improve over time;
-more generous marking.
It would be really important to clarify this - otherwise we can end up vindicating the wrong policies - or not vindicating the right ones...
Hi Pedro. Very nice to see the beginning of serious efforts to explore the causes of the 2006-2009 improvement.
ReplyDeleteA quick question: what do you mean by "more generous marking"? More generous grading of the results of the skills' tests in PISA? I'm sure there's a lot of measurement problems that can be raised in terms of comparability across PISA waves and it's possible that tests were "easier" this time. But if that's the case, wouldn't that apply to all countries, or do you think there are problems in cross-national validity too?
BTW, I think another argument that has been advanced is related with changes in school retention practices. That's perhaps a more serious argument that the govt could advance, actually.
Hi Pedro
ReplyDeletefrom your numbers we can also see that a larger sample was used, is there is coding for the place where the sample was collected to check there was some selection bias in the sampling for 2009?
I think it is worth to underline two issues that might clarify trends of different countries for the pisa results - even disregarding potential sampling problems.
ReplyDeleteThe first one is related with considering the tests results as completely standardized in a year to year basis and, therefore, conclude that an increase in the average score represents necessarily an increase in the quality of the education system. It may be instead that for a particular year the exams are overall easier. Thus it might be more informative to just present the relative position of one country against all the other.
Second (and related with the first), is how to correspond the share of, say, an improvement of the (relative) test scores to improvements in the education system as opposed with other factors that fall outside the direct reach of government policies. One of such other factors that might be important is the number of subjects sampled with an immigration background. One can easily imagine that these subjects are naturally handicapped in the performance of the pisa test when compared with native subjects.
Given these two points it seems to me abusive to consider, as some do, that the Portuguese educational system is as good as the Swedish just because the test scores are statistical equivalent between the two countries. Immigrants in Portugal typically speak the same language and have a closer culture to the host country than the ones in Sweden; the share of subjects with immigration background is higher in Sweden than in Portugal; equally, the migration inflow has been stronger in Sweden than in Portugal during the last decade. All these factors should have an impact in the relative test scores that is not easily imputable to the quality of the educational system of each country.
A more transparent way of displaying the trends would show the relative results of the test scores for both groups of subjects - with and without immigration background. Actually the OECD documents show some of these results, despite, in my opinion, not in the most intuitive way. Nevertheless, it seems to me that most of our relative improvement in the pisa results should be related with these trends in the migration flows. And of course that what I've claimed here for the comparison Sweden/Portugal applies also for any other pair of countries.
Tiago,
ReplyDeletethe improvement that the Portuguese results was both absolute and relative. If you compare the increase in the score, Portugal had the 3rd in reading, 4th in maths and the 2nd in sciences, highest increase of all. The lack of standardization argument doesn't hold.
I can't follow your comment on immigration, for two reasons. First 2006-2009 was not a period of big immigration inflows to Portugal. Second, if anything, immigration would drive the results downwards - as it is shown in any economical paper on education - not upwards.
A more general comment. I love scepticism, I just don't understand why this dormant Portuguese scepticism is only woken up whenever there are good results.
Thank you all for your comments:
ReplyDelete1. Yes, by "more generous marking" I meant disparities in the marking standards by country. I'm sure the OECD try to minimise these but I suppose they can't be ruled out completely.
2. Indeed retention apparently did change. While the (weighted) mean age of the students surveyed stayed constant (15.7) in the 2006 and 2009 samples, the (weighted) mean grade went up (from 9.2 to 9.5, in which 9 is "9.o ano" and 10 is "10.o ano"). Given the strong relationship between PISA scores and grade (at least from the simple bivariate regression I ran), the increase in the score can be explained entirely by the higher grades of the 2009 sample. The question then is: Is this because students are doing better or because the sample criteria changed?
3. The PISA micro data does not include any sort of geographical information for Portugal (nor for several other countries). And the Education Ministry declined to provide these data, although I don't really know why.
4. My understanding is that the OECD try to keep the results comparable over time - but at the same time they standardize them and focus on cross-country comparisons. In any case, I certainly agree that many factors do change from one to next PISA test - and a lot of those aspects are outside the control of the government. On the other hand, if one has some evidence to say that "Portuguese kids are now learning more than they used to", this will inevitably be important per se and gain a political dimension. The problem is all the question marks surrounding that statement.
5. Good point on the "asymmetric scepticism". Let's make it as symmetric as possible.
"First 2006-2009 was not a period of big immigration inflows to Portugal. Second, if anything, immigration would drive the results downwards"
ReplyDeleteMiguel, this is exactly my point. If immigration has a downward impact on the test results and if immigration was less important in Portugal than in other countries, then aggregate test results for Portuguese kids should be overestimated in a relative basis.
Showing the results just for native kids would increase the comparability of the relative quality of each educational system.
Pedro, curious your point about the mix of "9º ano" and "10º ano", explanation may well lie in that change in the mix - overall, students do not pay much attention to "9º ano", but they become more serious on the "10º ano", as the grades from that point onwards will count for their grade point average to enter a public university; "9º ano" grades are just harmless on that respect. Pressure from parents to have good grades also intensifies on 10º ano.
ReplyDeleteCan you just simulate a sample in 2009 with 9.2 grade average? and see the average results (by simulation, random sampling; or just by adjusting average values to the different composition mix)?
cheers.
Tiago,
ReplyDelete1. That only explains the relative improvement, not the absolute.
2. I really doubt that all OECD countries had a strong migration inflow across those years. OECD is not only Sweden and Denmark. Actually I recall seeing numbers about the immigration to EU in the last 5 years, and apart from Italy and Spain, the numbers were very low.
I have another hypothesis to explain this increase, related to kindergarten. These kids were in the 9th grade in 2009, meaning they would have entered kindergarten around 1996.
ReplyDeleteThis was exactly the period in which there was a boom in kindergarten coverage, one of the key policies of the first Guterres' government.
Given the known positive impact of kindergarten on later school results, this would explain the increase in both public and private schools.
This comment has been removed by the author.
ReplyDeleteMiguel Carvalho, tem razão, a hipótese que coloca é perfeitamente credível. Usando dados desagregados por regiões, sabendo o grau de cobertura do pré-escolar, deverá ser possível testar essa hipótese.
ReplyDeletePeople as Svein Sjøberg, University of Oslo, and others had pointed out that the test is based on a contradiction: testing education whereas it cannot test on subjects actively studied in a given country and not in other.
ReplyDeleteAt the same time they show examples of questions in which the level of difficulty is greatly reduced or enhanced
http://www.scienceinpublic.com.au/blog/wp-content/uploads/Svein-Sjoberg-PISA-tests_La-Revue_no14-Sept-2012.pdf
The value for money of the full education is reduced by politicians to PISA point per invested dolar!! Many politician claim they have to kill funding of arts and sports because the PISA rank shows that countries without arts have a better pisa position!! "a better education".
Also when a country did good in the PISA test having on average less overall budget in education, less teachers and reduced salaries the politicians of the other countries use that PISA result to kill the budget of education claiming that they need to DO BETTER in the Pisa test: they need to slash the salaries to improve education!!!
Please take a look at this excellent paper:
http://www.scienceinpublic.com.au/blog/wp-content/uploads/Svein-Sjoberg-PISA-tests_La-Revue_no14-Sept-2012.pdf