Subjectivity in Stereotypes Against Migrants in Italian: An Experimental Annotation Procedure

The presence of social stereotypes in NLP resources is an emerging topic that challenges traditionally used approaches for
the creation of corpora and resources. An increasing number of scholars proposed strategies for considering annotators’
subjectivity in order to reduce such bias both in computational resources and in NLP models. In this paper, we present
Open-Stereotype, an annotated corpus of Italian tweets and news headlines regarding immigration in Italy developed through
an experimental procedure for the annotation of stereotypes aimed to investigate their different interpretation. The annotation
is the result of a six-step process, where annotators identify text-spans expressing stereotypes, generate rationales about
these spans and group them in a more comprehensive set of labels. Results show that humans exhibit high subjectivity in
conceptualizing this phenomenon, and that the prior knowledge of an Italian LLM leads to more consistent classifications of
specific labels that do not depend on annotators’ background.