Dott. Onofrio Rosario Battaglia
University of Palermo, Palermo, Italy
Clustering methods to study student reasoning lines: theoretical aspects and experimental results
Studying reasoning lines students deploy when dealing with problematic situations is, in Education Research, one of the most important aims. The reasoning lines can be achieved by analysing the answers that students give to a questionnaire, although it becomes increasingly complicated as the number of students to be analysed increases. In my speech I focus on a quantitative method based on clustering by discussing theoretical and methodological aspects. The method can allow a researcher to analyse a set of answers given to a questionnaire, even in the case of a large sample of students. Moreover, clustering is not presently common to the physics education research community. I describe in detail two different clustering methods, a hierarchical one and a non-hierarchical one. I introduce a binary coding that makes the answers quantitatively analysable. Moreover, a correlation coefficient and a metric suitable for measuring student similarity in the case of binary coding are presented. Then, criteria for choosing the optimal number of clusters for both the clustering methods are discussed. For the same purpose, a new coefficient is introduced to measure the total amount of information one can obtain from a clustering solution. I show that each cluster can be characterised by its centroid. It summarizes the most frequent answers given by students in a given cluster. An example of a clustering procedure for experimental data is given. The comparison between the results obtained through the two clustering methods shows a good agreement exhibiting robustness in the proposed method.
Prof. William Boone
Program in Learning Science and Human Development, Miami University, Oxford, Ohio USA
What I wish I had known as a Rasch beginner….
This talk will present a summary of what I wish I had known 30+ years ago as I began my Ph.D. studies with Ben Wright and first learned Rasch methods. In this talk I will present key errors beginners commonly make when they start to use Rasch. Naturally I will briefly discuss the ins and outs of such beginner Rasch errors…my goal..to aid Rasch novices so that they can confidently, quickly and correctly use Rasch. I will also share and discuss what my family calls “Bill’s Rasch elevator speech”…such a speech is the one I can give very quickly (60 seconds) in an effort to help others understand why Rasch is important. Quickly explaining Rasch to those who do indeed need your Rasch help (but they do not know they need your assistance) is a critical skill that one must develop. It is an art to succinctly explain Rasch to others, be it in person, or in papers. Even as a beginner with Rasch, by mastering your own elevator speech you can increase the number of people who seek your Rasch expertise, be it in academia or the private sector.
Prof. Milos Kankaras
Faculty of Philosophy, Department of Psychology, University of Montenegro, Montenegro
Assessment of social and emotional skills in cross-national settings: an OECD approach
Dott. Michele Marsili
INVALSI, Rome, Italy
AUTOMATED ASSESSMENT OF OPEN-ENDED QUESTION OF INVALSI TESTS
This work describes the new procedures of automated corrections of freeform answers given by the 8th, 10th and 13th grade students to open-ended questions in CBT Computer Based Test) INVALSI tests. INVALSI team, composed of statistical and computer scientists, responsible of open-ended question correction, has implemented an algorithm to process text strings of different complexity. Before survey distribution, the correction team and the items authors group discuss to define the correction criteria, that is a set of rules to determine the correct or incorrect classification for each answer given by the students for a specific item. The discussion produced, moreover, the indications to remove useless elements for the classification, then translated in operations of the algorithm on the textual data such as punctuation detection and removal, special characters, articles, conjunctions, word lemmatisation, etc. The answer strings were subsequently processed by a “data cleaning” operation, that was focused on the automated correction of spelling and typing errors, by detection and substitution of “out-of-vocabulary” words (OOV words). After the “data cleaning” phase, the correction criteria fixed by the experts have been translated in logical IT patterns, aiming to uniquely defining the set of admissible ways to give a correct answer. The last test phases of the algorithm were characterized by a constant exchange of information about the encoding, among the authors’ team and the correction team, this passage being critical to refine the logical rules used for correction and to get more consistency and precision between the encoding produced by the algorithm and the authors’ indications. The final test of the algorithm ends with a comparison between the manual encoding by video correction and the one processed by the algorithm on a set of items already processed in a former test: the algorithm is accounted as accurate enough and aligned to the indications of authors’ team when the complete accordance of the two encoding was achieved. The methodological approach, countable as a method of supervised automated correction, represents a valid compromise between a manual encoding and a totally automated one, typical of the machine learning algorithms. This method has indeed the benefit of considerably reduce the hours/man needed to correct the open-ended answer items, when compared to a manual procedure, and get a better accuracy reducing the wrong encoding matches, when compared to a non-supervised automated procedure. A comparison between supervised and non-supervised automated procedure has been eventually done to evaluate the distance between the two methodological approaches.
Prof. Stefania Mignani
Department of Statistical Sciences, University of Bologna, Italy
Assessing The Gender Gaps In Maths Competence: An Overview Of What We Known From Invalsi Data
Amongst the extensive research conducted in the context of gender gaps in educational context, an issue that is frequently examined is the role of pre-university education and of the study of STEM subjects, in particular mathematics. The impact of how this discipline is taught, along with the relative attitudes to and interest in this subject, seem to be the main factors pushing young people, and girls in particular, away from the study of STEM subjects. In Italy several studies have been conducted to investigate the gender gap in mathematics using standardized large-scale data from INVALSI tests. The presentation will discuss a review on recent learning results in mathematics of young Italian students both in a cross sectional context and in a longitudinal perspective. For cross-sectional data the focusing on the differences in top and low performances may confirm or not whether the gender differences are more evident within groups of students characterize by different achievement level. The cross-section analyses provide only snapshots of the level of competencies acquired at a specific grade in a given year. On the contrary, a longitudinal approach could evaluate how the performances change over time. Besides, INVALSI makes data available including not only the test responses, but also a set of variables dealing with socio-demographic and economic characteristics, the educational path and, for some grades, student self-reported information about individual and emotional aspects. In the literature, an interesting issue deals with the emotional aspects induced by the test administration. The emotional component behaviour can partially explain the gender gap performance. The results of the illustrated studies offer room to enrich the debate around the international recommendations to promote gender equality in education by ensuring equal opportunities in making educational choices between boys and girls and making the study of STEM subjects equally inclusive and attractive
Prof. Giuseppe Pellegrini
Observa Science in Society, Vicenza, Italy
Evolution acceptance and high school students. Methodological considerations and new perspectives of investigation
The idea of biological evolution is not accepted by many people around the world, with a large disparity amongst countries. Some factors may act as obstacles to the acceptance of evolution, such as religion, a lack of openness to experience, and not understanding the nature of science. Although the strength of the association between evolution acceptance and non-scientific factors varies among studies, it is often assumed that resistance to evolution is the by-product of a religious background. Some studies are even more specific and try to associate the acceptance of evolution with precise religious affiliations.
In my speech I will propose an introduction of the main tests used internationally to measure evolution acceptance and some statistical tools that allow our research team to verify the relevance of sociocultural factors in predicting such acceptance.
Starting from an in-depth reflection on the factors that influence the knowledge and acceptance of evolutionary theories, the Italian-Brazilian research group formulated a research question to address this complex picture considering the same religious affiliation in two different countries with deep sociocultural differences. Catholic Christians in Italy and Brazil have several similarities, including many family connections owing to immigration history. Brazil is the country with the highest number of Catholic Christians in the world, and Italy is the hub of Roman Catholicism.
These conditions allowed to conduct a survey on the adolescent population with two statistically significant national samples in 2014. We adopted a clear definition of evolution acceptance despite the complexity of the discussion on the subject in different languages, taking acceptance as the expression of explicit recognition of the objective validity of known scientific statements about evolution under absolute anonymity. This definition considers two steps. The first is associated with scientific statements about evolution, which must be clear and well known, avoiding issues under discussion, for instance, about the origin of life. Students must show not only a positive attitude towards evolution but also express clearly that a statement based on biological evolution is considered a valid premise to construct a judgement about the real world. The second step refers to objective conditions in which a person may admit his/her positive judgment about a certain scientific statement. We aimed to explore the strength of associations among nationality, religion, and the acceptance of evolution by students using multiple correspondence analysis (MCA) and statistical tools, with nationwide samples from two different countries. In our research we found that wider sociocultural factors predict the acceptance of evolution to a higher degree than a religious background. Roman Catholic students showed significant differences between the two countries, and the gap between them was wider than between Catholics and non-Catholic Christians within Brazil. Our conclusions support those who argue that religious affiliation is not the main factor in predicting the level of evolution acceptance. The sociocultural environment and the level of evolutionary knowledge seem to be more important in this regard. These results open up new interpretative perspectives and provide a better understanding of attitudes towards evolution.
Prof. Martin Rusek
Charles University, Faculty of Education. Prague, Czech Republic
The black box is not that black anymore: The use of eye-tracking in STEM education research
Despite many attempts to measure pupils and students' learning outcomes, their performance in problem solving has largely remained something of a black box. This talk will therefore focus on the potential of using eye-tracking data in combination with the think-aloud method. Case studies will be used to show how to analyze data from actual science education research. Finally, further possibilities of using eye-tracking in STE(A)M education research will be outlined.
