Research Question
Federated LLM settings make calibration harder to explain because model behavior, client distribution, robustness, and fairness all interact. FR-FCLC needs a plain-language story: local clients produce calibration evidence, the server aggregates it, and the final thresholds are evaluated across non-identical clients.
The research question I worked from was practical: can robust and fairness-aware conformal calibration keep per-client coverage closer to target when clients are non-IID or Byzantine-corrupted?
What I Built
The project implements a post-hoc calibration pipeline with Adaptive Prediction Set scores, robust aggregation, fairness-aware threshold adjustment, and evaluation scripts. The public README documents a 100-client simulation, 5 seeds, 3 settings, and a Byzantine robustness sweep from 0% to 50%.
Project Overview
FR-FCLC is a public research implementation focused on fair and robust federated conformal logit calibration.
My Contribution
- Implemented client preprocessing, non-IID data partitioning, local APS score calculation, and robust server-side threshold aggregation.
- Added fairness-aware threshold adjustments, multi-seed experiment execution, and Byzantine attack sweeps.
- Produced statistical result analysis, JSON outputs, plots, and reproducible experiment artifacts.
System Architecture
Federated calibration pipeline
Clients compute local scores, the server aggregates thresholds, and experiments evaluate coverage and robustness.
The architecture follows the public repository structure and README experiment description.
Primary Result
Per-client coverage, honest setting
Real chart generated by the repository's evaluation script, copied from plots/coverage_comparison.png.
Baseline: the Naive method hits the 90% target on average (mean 0.9010) but its per-client coverage gap is wide (0.2632). Primary result: FR-FCLC keeps every client close to target with a mean of 0.9432 and a gap of just 0.0421 — the smallest of the four methods.
Robustness under a Byzantine deflate attack
Real chart generated by the repository's robustness sweep, copied from plots/byzantine_robustness_deflate.png.
As corrupted clients rise from 0% to 50%, the Robust baseline's coverage gap climbs past 0.45 and its coverage mean stays stuck near 0.78. FR-FCLC's coverage gap stays flat near 0.09 across the entire sweep.
Limitations
FR-FCLC is not a universal winner. The repository's paired significance tests show FR-FCLC's raw coverage is statistically comparable to the Fair baseline, not significantly better. Under both Byzantine settings, FR-FCLC's coverage gap is far smaller than Naive or Robust, but it does not fully clear the repository's strict 5% per-client fairness tolerance once the system is under attack. The honest claim is "most robust of the four methods in the published sweep," not "fair under every condition."
The validated results are BoolQ-only. The repository also preprocesses TruthfulQA, but full-vocabulary APS scoring for TruthfulQA has not been tested at the 100-client scale, so none of the coverage or fairness numbers above extend to it yet.
Technical Decisions
Use multi-seed experiments instead of reporting a single favorable run.
- A single run can overstate robustness or hide unstable behavior.
- Multi-seed evaluation adds compute time and result-management complexity.
- The comparison is more reproducible and more credible.
How I Work
Engineering Challenge
- Research results can sound more final than they are.
- The README was reviewed for what is actually claimed: simulation, seeds, settings, robustness sweep, and artifacts.
- The case study avoids publication claims and frames results as implementation evidence.
- The page communicates serious research engineering without overstating academic validation.
Quality & Testing
Quality Evidence
Product engineering evidence
- The README documents multi-seed evaluation, robustness sweeps, result JSON, and statistical analysis artifacts.
- No CI result is published for the research run; the public repository provides scripts and generated artifacts.
- The repository structure lists preprocessing, simulation, aggregation, and analysis modules.
Public page quality
- Result charts include descriptive captions and alt text.
- Diagrams and charts scale within the public case-study layout.
- Published claims are backed by the FR-FCLC evidence ledger.
Results & Impact
The safe result claim is implementation completion: a public repository documents the 100-client simulation setup, generated plots, result JSON files, robustness sweep, and statistical analysis artifacts.
What I Learned
Research engineering is both implementation and communication. A strong page explains what was tested, what was not proven, and where the reader can inspect the artifacts.