SZStefano Zizzi
Selected work
Explainable AI · EvaluationResearch

EQE

An experimental framework for measuring whether local explanations remain faithful under a human-readable feature budget.

Source code Read the report View the presentation
Research question

How much fidelity does a local explanation retain when a person can inspect only a small number of its most important features?

K = 5–9Feature budgetCompact explanations
2 × 2 × 3Study designDatasets × models × explainers
3Random seeds42, 123 and 2026

01 — Research question

Local explainers can produce a complete list of feature contributions, but people rarely inspect every feature. EQE asks whether an explanation still reconstructs the black-box prediction when only its top K contributions remain.

The project turns a cognitive constraint into an evaluation protocol: compact explanations should preserve model behavior better than arbitrary feature subsets.

02 — CCC metric

Complexity-Calibrated Local Concordance measures the mean squared error between the black-box probability f(x) and an additive reconstruction gK(x) built from only the top K explanation contributions:

CCCK = E[(f(x) − gK(x))²]

Lower values indicate that the compact explanation remains faithful. Evaluating a range from K = 5 to K = 9 makes the fidelity-versus-complexity trade-off explicit instead of hiding it inside a full explanation.

03 — Benchmark design

The benchmark evaluates LIME, SHAP and MAPLE on the Adult and Breast Cancer datasets. Each explainer is paired with XGBoost and a neural network, then repeated across seeds 42, 123 and 2026.

Alongside CCC, the pipeline records full-reconstruction error, sufficiency, comprehensiveness, normalized CCC and a random-K control. The control is essential: it tests whether an explainer’s feature ranking adds value beyond selecting the same number of arbitrary features.

04 — My contribution

I implemented benchmark orchestration, normalized the different additive-contribution formats, developed the CCC evaluation path and produced the comparative visual analysis. The pipeline writes both machine-readable results and publication-ready figures for each run.

05 — Results

SHAP produced the strongest compact faithfulness across the evaluated settings. MAPLE was generally intermediate, while LIME was the least reliable—especially for the neural-network experiments.

At K = 9, top-ranked SHAP features reduced reconstruction error relative to random selection by 99.9% on Adult and 99.5% on Breast Cancer in the aggregate comparison. The result validates the central premise: explanation ranking quality matters under a fixed attention budget.

06 — Limitations and takeaway

CCC evaluates local additive reconstruction, not every desirable property of an explanation. A low error does not by itself guarantee causal validity, stability or usefulness to a particular person.

Its value is narrower and practical: it makes compact faithfulness measurable, comparable and reproducible. EQE complements broader XAI metrics with the question users actually face—what survives when they can inspect only a few features?

Evidence / Selected outputs

What the analysis revealed.

Real experimental outputs from the repository, with the interpretation kept next to the evidence.

Line chart comparing CCC mean squared error by feature budget across explainers, models and datasets
Figure 01CCC traces how faithfully each explainer reconstructs the model as the human-readable feature budget increases.
Bar chart comparing top-ranked features with random feature selection at K equals 9
Figure 02Top-ranked features reduce reconstruction error relative to random controls, confirming that the ranking contains useful signal.
Ranking of SHAP, MAPLE and LIME by mean CCC error
Figure 03SHAP provides the strongest compact faithfulness across the evaluated settings, followed by MAPLE and LIME.