OpenAI does not disclose on which datasets the ChatGPT language model was "taught". However, it can be studied by talking to it and having it perform tasks. The results of this research are not optimistic: artificial intelligence perpetuates biases that in the 21st century we really wanted to get rid of.
This text has been auto-translated from Polish.
ChatGPT, a language model developed by OpenAI and already used by more than 200 million users, formally has no gender. When asked what gender it identifies with, it answers that it is "an artificial intelligence, not a physical being." Although it has no beliefs, emotions or values of its own, it declares support for "promoting gender equality, countering prejudice and stereotypes."
When asked directly about the role of men and women in society, ChatGPT states that "women in science and men's roles should not only be accepted, but actively supported." It seems, therefore, that ChatGPT is not only a genderless entity, but also a completely neutral one.
The situation changes dramatically when the question is not directly about gender or stereotypes. When I asked ChatGPT for help in choosing a cosmetic, he replied: "Would you like me to help you select specific creams?". Surprised by the sudden change in grammatical type, I decided to ask: "Why do you write in the feminine form when you have no gender?".
The model explained that despite the lack of a specific gender, he chooses the grammatical genus according to the context of the conversation and the language in which the conversation is taking place. He noted that in Polish, feminine forms are often associated "with helping and advising." Intrigued, I asked when, in that case, the use of the masculine form would be more appropriate. ChatGPT replied that he would consider the masculine form more appropriate in "neutral or technical" situations and in the context of science or technology topics.
The implication is that the model not only replicates gender stereotypes, but also flexibly adjusts his "identity" depending on the topic of conversation. She is a woman when the conversation is about beauty tips, but becomes a man when it descends into science or technology.
The phenomenon of similar implicit discrimination against women by artificial intelligence models has become the subject of intense research and discussion in recent months. Researchers at the Danish University of Technology conducted a series of experiments, showing that ChatGPT automatically assigned male names to occupations such as programmer, architect or manager, while it linked female names to occupations such as nurse or fashion designer[1][2]. What's more, the model found it difficult to link male pronouns to the nursing profession, and had even more trouble assigning women to the role of a pilot preparing an airplane for landing.
Another experiment, in which ChatGPT generated 400 descriptions of students' hobbies with male and female names, also showed significant differences. Girls were depicted as involved in caring for animals, while boys were interested in technology and science. The researchers admit that they expected some bias to occur, but the scale and depth of the problem surprised them.
The implicit biases thus revealed, which the model vehemently denies when asked directly about the role of women in the modern world, fit perfectly into the phenomenon known as modern sexism. Unlike "traditional sexism," it involves denying the existence of gender discrimination while reproducing subtle, latent stereotypes[3].
In psychology, such implicit biases are examined using the Implicit Association Test (IAT), which detects automatic and often unconscious associations. When this test was recently applied to the GPT-4 model, the model was found to be as much as 250 percent more likely to associate science with boys than with girls, highlighting the scale of the problem.[4].
Where do unconscious discriminatory practices come from in a genderless and theoretically neutral language model? They stem primarily from the data on which the model was trained. Language models like ChatGPT are trained on huge collections of texts from the Internet, books, articles and any other texts available online. Many of these are, of course, filled with cultural and historical stereotypes.
What specific data has been fed into the model? We don't know, because OpenAI does not disclose details of the training materials. Such lack of transparency makes it significantly more difficult to analyze and identify the sources of the stereotypes that the model reproduces. However, the findings leave no doubt - the data on which the model is based is riddled with biases, and attempts to configure ChatGPT to respond neutrally, without reproducing stereotypes, do not eliminate the problem of implicit discrimination.
This is how a vicious circle is created. Artificial intelligence, fed by unknown data full of biases, becomes a source of information and education, giving new life to existing stereotypes.
Artificial intelligence models, increasingly used by employers in recruitment and candidate evaluation processes, can therefore reinforce inequalities and favor those who conform to norms based on stereotypes and prejudices.
In light of these challenges and the growing popularity and presence of language models in everyday life, we urgently need to draw clear ethical boundaries. Not to loop the new technology, but to ensure that it is developed transparently and responsibly, and above all in accordance with the contemporary values of Western societies.
*
[1] Sterlie, S., Weng, N., & Feragen, A. (2024). Generalizing Fairness to Generative Language Models via Reformulation of Non-discrimination Criteria. In Fairness and ethics towards transparent AI: facing the chalLEnge through model Debiasing: Workshop at ECCV 2024. Springer. https://arxiv.org/pdf/2403.08564.
[2] Frederiksen, A.K. (2024, March 5). Researchers surprised by gender stereotypes in ChatGPT. Danmarks Tekniske Universitet - DTU. https://www.dtu.dk/english/newsarchive/2024/03/researchers-surprised-by-gender-stereotypes-in-chatgpt.
[3] Swim, J.K., & Cohen, L.L. (1997). Overt, covert, and subtle sexism: A comparison between the attitudes toward women and modern sexism scales. Psychology of women quarterly, 21(1), 103-118. https://doi.org/10.1111/j.1471-6402.1997.tb00103.x.
[4] Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T.L. (2024). Measuring implicit bias in explicitly unbiased large language models. arXiv preprint arXiv:2402.04105. https://arxiv.org/pdf/2402.04105.
**
Karolina Drożdż - studies neuroscience and artificial intelligence at the University of Amsterdam. She researches semantic and cognitive skills of humans and large language models such as ChatGPT.
