from 30 January 2023 to 1 February 2023
Complesso S. Marcellino e Festo
Europe/Rome timezone
The Political Sciences and Physics "E. Pancini" Departments of Federico II University organize the 1st International Conference on Measurement in STEM Education (MESE1).

A Counterfactory Approach To The Evaluation Of Educational Policies: Methodology, Variables And Results

31 Jan 2023, 15:05
G4 (Complesso S. Marcellino e Festo)


Complesso S. Marcellino e Festo

Largo S. Marcellino, 80138 Napoli NA


Andrea Bendinelli (INVALSI)


The fight against early school leaving, the prevention of school failure, the raising of basic skills levels and support for learning are among the main objectives of the interventions financed through projects activated in favor of schools. The study is aimed at quantifying the effect on the levels of competence achieved by students in the INVALSI tests through interventions aimed at (i) improving school performance and (ii) reducing early school leaving through the construction of a longitudinally anchored and counterfactual analytical system able to evaluate the progression and/or changes in students' skills over time in three areas: Italian, Mathematics and English. This evaluation is carried out in a comparative way between the students of the schools involved in the Projects (hereinafter "treated cases") with respect to those not involved (hereinafter referred to as "untreated cases"). The purpose of effect evaluation is to verify the ability of an intervention to change the behavior or conditions of a specific target population in the desired direction. The method used, the Difference-in-Difference (Angrist & Pischke, 2009; Keele, 2020), makes it possible to estimate the value of the outcome variable of the students treated in the event that they would not have received the treatment (counterfactual). The net effect of the treatment is therefore obtained as the difference between the score observed on the "treated" students and the score that would have been observed in the absence of the treatment. To estimate the effect of interventions on the results of standardized tests, a quasi-experimental evaluation design was introduced by adopting a counterfactual approach. In counterfactual analysis, three groups are referred to: (i) control group, (ii) factual group, and (iii) counterfactual group. The control group, built using the Propensity Score Matching technique (Martini A., 2011), is made up of students who have not received the treatment and who present characteristics (socio-demographic, such as gender, age, geographical location, citizenship, socio-economic status and culture of the family of origin; and scholastic, previous level of competence and regularity with respect to the course of studies) completely similar to the characteristics of the students who received the treatment and who make up the current group. Finally, the counterfactual group, constructed with the help of a control group, is the one for which it is estimated what the variation over time of the students' competence in the treated group would have been if they had not received the treatment. The data used selected from the national surveys to which INVALSI administers to students in the various school levels covered by the Survey, thanks to the presence of a longitudinal student code, it was possible to follow the students over time to evaluate the variation in the outcomes following specific interventions. It was also possible to proceed with a clustering of the group of treaties to evaluate the effectiveness of policy interventions on sub-populations on the basis of the different characteristics present within them. Thanks to the subdivision into subpopulations, it was possible to conclude that the efficacy of the treatment had different outcomes according to the identified subgroup. The results will also be discussed based on the literature on the subject.

Primary author

Presentation Materials

Your browser is out of date!

Update your browser to view this website correctly. Update my browser now