arXiv | Github | Data

Abstract

This paper introduces misinfo-general, a benchmark dataset for evaluating misinformation models’ ability to perform out-of-distribution generalisation. Misinformation changes rapidly, much quicker than moderators can annotate at scale, resulting in a shift between the training and inference data distributions. As a result, misinformation models need to be able to perform out-of-distribution generalisation, an understudied problem in existing datasets. We identify 6 axes of generalisation-time, event, topic, publisher, political bias, misinformation type-and design evaluation procedures for each. We also analyse some baseline models, highlighting how these fail important desiderata.

Citation

1@misc{verhoeven2024yesterdaysnewsbenchmarkingmultidimensional,
2      title={Yesterday's News: Benchmarking Multi-Dimensional Out-of-Distribution Generalisation of Misinformation Detection Models},
3      author={Ivo Verhoeven and Pushkar Mishra and Ekaterina Shutova},
4      year={2024},
5      eprint={2410.18122},
6      archivePrefix={arXiv},
7      primaryClass={cs.IR},
8      url={https://arxiv.org/abs/2410.18122},
9}