Summary

During this PhD project, Amalie Pauli studied how to measure and evaluate style-related aspects of text, particularly in text generated by artificial intelligence. With the rapid development and increasing use of large language models, such as chatbots, it has become important to assess not only the information these systems generate, but also how it is expressed—for example, whether the text contains persuasive language. However, measuring such language use is inherently difficult, as style is subjective, nuanced, and dependent on context.

Amalie Pauli focused on measuring large language models’ capabilities to generate persuasive language and examined how different conditions affect both the type and degree of persuasive language produced. The research shows, for example, how generated messages can reflect gender-stereotypical language patterns in persuasive language. In addition, the project critically evaluated existing assessment methods for a style-related task, demonstrating that commonly used approaches do not always measure what they are intended to measure. The work highlights the need for evaluation methods that are reliable (consistent), valid (accurately capturing the intended concept), and efficient to apply at scale.

The findings contribute to a better understanding of how to evaluate nuanced and subjective aspects of AI-generated text. This is important for developing trustworthy AI systems and for identifying potential risks related to persuasion, bias, and misinformation in automated communication.

This summary was prepared by the PhD student.