Intermediate · Evaluation
CommitmentBank (CB)
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A dataset used to evaluate an LLM's ability to discern if a passage's author believes a specific stated clause.
Technical Definition
A dataset used to evaluate an LLM's ability to discern if a passage's author believes a specific stated clause.
How it works
CommitmentBank (CB) is a dataset designed to assess a Large Language Model's (LLM) capability in understanding authorial belief. Each data point consists of a passage, a target clause extracted from that passage, and a true/false label indicating whether the passage's author holds the belief expressed in the clause. For example, it checks if the author truly believes 'she had a sense of humor' in a given text.
Related Concepts
- Dataset — An organized collection of examples used to train, validate, or test a model.
- Natural Language Understanding (NLU) — The ability of a computer program to understand human language as it is spoken or written.