Use SHAP and LIME to understand how models make
predictions.
edit_note Input Text
Prediction
Negative
Positive
LIME — local surrogate
The model is treated as a black box.
Words are deleted at random, the model is re-queried on each variant, and a weighted linear
model is fitted to those answers. These colours are that surrogate's coefficients: green pushes positive, red pushes negative.
Compare with the SHAP panel. LIME samples; SHAP here is computed
analytically from the model's structure. Where they disagree, that disagreement is
information — it usually means the model is not locally linear.
bar_chart SHAP
Feature Importance
category Aspect-Based
Sentiment
compare_arrows
Counterfactual Explanations
What minimal changes would flip the
prediction?
Try:
tune Applicant
Features
$60,000
680
35%
5
$50,000
35
waterfall_chart
SHAP Waterfall
tune Patient
Features
45
26
125
210
3
waterfall_chart
SHAP Feature Importance
gavel Defendant
Profile
Modelled on ProPublica's 2016 COMPAS analysis. The scoring
function below is race-blind — race is not one of its inputs, just as it is not
an input to COMPAS. Watch the disparity appear anyway.
30
2
0
Where is the race selector? There isn't one, because
the model has no race input. The chart below instead scores each group's typical
criminal-history profile through the same race-blind function. The gap it produces is
not something the model was told to create.
warning Disparate Impact Without a Race Input
waterfall_chart
SHAP: Why This Score?
compare_arrows
Fairness Comparison: Same Profile, Different Race
school Why This Matters
ProPublica's 2016 investigation found that COMPAS was biased against Black defendants — they
were
nearly twice as likely to be falsely flagged as high risk compared to White defendants.
Explainability tools like SHAP help reveal when race acts as a hidden
driver
of predictions, even when the model claims to be "fair."