SZStefano Zizzi
Selected work
NLP · Political DiscourseComplete

CLARITY

Detection and classification of evasive political answers across transformer encoders, prompted LLMs and QLoRA fine-tuning.

Source code Read the report SemEval task
Research question

Does fine-grained supervision about how a political answer is evasive improve the coarser prediction of whether the answer is clear?

0.608Macro-F1Best mapped clarity
0.721AccuracyFine-tuned mapped clarity
308Test pairsShared held-out evaluation

01 — Research question

Political answers can sound fluent and relevant while still avoiding the question. CLARITY studies whether an answer is a clear reply, an ambivalent reply or a clear non-reply—and whether first identifying the underlying evasion strategy improves that prediction.

The project uses question-answer pairs from QEvasion and evaluates every model family on the same held-out test set with Macro-F1 as the primary metric.

02 — Three connected tasks

The experiment separates the problem into three settings:

  • Direct clarity: classify the answer into three clarity classes.
  • Direct evasion: identify one of nine fine-grained rhetorical strategies.
  • Mapped clarity: predict the evasion strategy first, then deterministically map it to a clarity class.

This design tests whether a finer intermediate representation helps or simply introduces another place for errors to propagate.

03 — Model tracks

The encoder track compares BERT-family models, with recorded report comparisons centered on DistilBERT and RoBERTa. It also evaluates imbalance-aware resampling, tone features and several loss functions.

The prompted track uses Llama 3.1 8B Instruct with Zero-Shot, Chain-of-Thought, taxonomy tables, RAG, rhetorical tone and Few-Shot variants. Dynamic example retrieval uses sentence embeddings and FAISS.

The fine-tuning track adapts Llama 3.2 3B Instruct with 4-bit QLoRA and DoRA. Separate adapters address direct clarity and fine-grained evasion, followed by the same deterministic mapping.

04 — My contribution

I co-developed the experimental pipeline and model comparisons, helped analyze the QEvasion task, and contributed to the reproducible environment used to run and record the experiments. Outputs are committed as machine-readable metrics, predictions and confusion matrices rather than only summarized in the paper.

05 — Results

The strongest prompted configuration was static Nine-Shot evasion classification followed by clarity mapping, reaching 0.608 Macro-F1. The fine-tuned mapped model reached the highest accuracy at 0.721, while the RoBERTa baseline remained the strongest direct-clarity system by Macro-F1 at 0.558.

Fine-grained supervision was therefore useful, but not universally so. Encoder errors in the nine-class task could propagate through the mapping, and semantically similar retrieved examples did not necessarily share the same rhetorical strategy.

06 — Takeaway

CLARITY’s main lesson is methodological: task decomposition can expose useful structure, but intermediate labels must earn their complexity through evaluation. The best result came from reasoning over the evasion taxonomy, while the error analysis shows exactly where that benefit stops.

Evidence / Selected outputs

What the analysis revealed.

Real experimental outputs from the repository, with the interpretation kept next to the evidence.

Comparison of zero-shot and chain-of-thought prompting strategies across clarity, evasion and mapped clarity tasks
Figure 01Prompt ablation across direct clarity, nine-class evasion and deterministic evasion-to-clarity mapping.
Macro-F1 comparison across one-shot, three-shot, nine-shot, dynamic and tone-aware prompting
Figure 02Static Nine-Shot prompting produced the strongest mapped-clarity Macro-F1; retrieval by topic did not reliably retrieve matching rhetorical strategies.
Confusion matrix for the fine-tuned direct clarity classifier
Figure 03The direct-clarity confusion matrix makes the remaining class imbalance and error structure visible.