POSQA: Probe the World Models of LLMs with Size Comparisons (EMNLP 2023 Findings)

This repository is based on our publication POSQA: Probe the World Models of LLMs with Size Comparisons (PDF).

@misc{shu2023posqa,
      title={POSQA: Probe the World Models of LLMs with Size Comparisons}, 
      author={Chang Shu and Jiuzhou Han and Fangyu Liu and Ehsan Shareghi and Nigel Collier},
      year={2023},
      eprint={2310.13394},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Overview

• Scripts for running experiments in scripts/
• Dataset in datasets/
• Generated Knowledge in knowledge/
• Experiment results in outputs/
• Human Annotations in annotations/

Setup

pip install -r requirements.txt

More detailed instructions are in the corresponding folders.

Data

• Dataset can also be founded in Huggingface Datasets: POSQA