Stance detection with LLMs

Promises and pitfalls of using large language models to identify actor stances in political discourse

Viviane Walker, Mario Angst, Gerold Schneider

University of Zürich

Should you use LLMs for stance detection?

The challenge

General task

In a text, identify

  • the stance (support, opposition, irrelevant)

  • of a (any) named entity

  • toward a given (any) statement

  • based on the text.

Rationale

  • identification of stances of actors highly relevant to social science research
  • this type of stance detection task still a challenge
  • very general task -> LLMs are very general models

Research Question

Can zero-shot stance classification with LLMs achieve adequate performance in an applied, empirical social science research setting?

State of the Art

  • zero-shot stance detection: by an LLMs > by a fine-tuned BERT (Zhang, Ding, and Jing 2022)
  • prompt engineering for zero-shot stance detection (Liu et al. 2023)
  • little research on real-world task examples
  • most of the world does not speak English

Methods

Prompt chain example: s2

Prompt chain example: is2

Prompt chain example: nise