Familiarity, Test Performance and Implications for Fairness

In this paper, we study whether students' familiarity with test topics affects their performance and whether familiarity disparities in groups lead to unfair examinations. Consider a student who is asked to write, in a foreign language, about their experience at a museum. What is evaluated is their writing proficiency in that language. Does it matter if they have ever been in a museum for the quality of their writing? What if most students from a given social group have never been to a museum? Is the test unfair towards that group due to its choice of topic? Through an experiment, we increase the familiarity level of randomly chosen individuals with a given topic and test whether they perform better on a writing task on the same topic than their less informed counterparts. Through a survey, we obtain the distribution of familiarity with several topics used in a worldwide standardised English test and test whether the test is unfair towards any group in three categories: gender, culture and socioeconomic status.

Zoom link: