Behavioral Testing Lab
Test a sentiment model the way CheckList does: with templates, word lists and expectations you can state in a sentence.
Accuracy is one number; behaviour is a list
A model can score well on a test set and still fail sentences a child would get right. CheckList (Ribeiro et al., 2020) tests a model the way software is tested: write a template, fill it from word lists, and state what the model should do. Three kinds of test:
- MFT
- Minimum functionality. Each case has a right answer:
- INV
- Invariance. A change that should not matter must not change the label:
- DIR
- Directional. A change that should push one way must not push the other (by more than ):
A probability of exactly 0.5 means the model found no word it knows; it is reported as neutral. Generation is seeded, so everyone sees the same cases.
The word-weight model from the Explainability Lab: loading… The second option is an ablation of the same model, not a different one: its negators are read as ordinary words.
Failure rate by capability
Failed cases ÷ cases, pooled over the tests in each cell
Tests
Open a test to read its failing cases
Write your own test
Put a word list in braces: {pos_adj}.
Use {pos_adj2} for a second, independent slot from the same list.
Your tests travel in the page link, so a team can share one suite.