Research Question
FR-FCLC stands for Fair and Robust Federated Conformal Logit Calibration. It is a post-hoc calibration framework for federated large language models: each client computes Adaptive Prediction Set (APS) scores locally, the server aggregates them into a global conformal threshold with coverage guarantees, and Byzantine-robust trimmed-quantile aggregation and fairness-aware per-client threshold adjustment are layered on top.
The research question I worked from was practical: can robust and fairness-aware conformal calibration keep per-client coverage closer to target when clients are non-IID or Byzantine-corrupted?
What I Built
The project implements a post-hoc calibration pipeline with Adaptive Prediction Set scores, robust aggregation, fairness-aware threshold adjustment, and evaluation scripts. The public README documents a 100-client simulation, 5 seeds, 3 settings, and a Byzantine robustness sweep from 0% to 50%.
Project Overview
FR-FCLC is a single-author research implementation built for a graduate Distributed AI course, with a public repository, experiment scripts, and generated result artifacts.
My Contribution
- Implemented client preprocessing, non-IID data partitioning, local APS score calculation, and robust server-side threshold aggregation.
- Added fairness-aware threshold adjustments, multi-seed experiment execution, and Byzantine attack sweeps.
- Produced statistical result analysis, JSON outputs, plots, and reproducible experiment artifacts.
System Architecture
Federated calibration pipeline
Clients compute local scores, the server aggregates thresholds, and experiments evaluate coverage and robustness.
The architecture follows the public repository structure and README experiment description.
Primary Result
Per-client coverage, honest setting
Real chart generated by the repository's evaluation script, copied from plots/coverage_comparison.png.
Baseline: the Naive method hits the 90% target on average (mean 0.9010) but its per-client coverage gap is wide (0.2632). Primary result: FR-FCLC keeps every client close to target with a mean of 0.9432 and a gap of just 0.0421 — the smallest of the four methods.
Robustness under a Byzantine deflate attack
Real chart generated by the repository's robustness sweep, copied from plots/byzantine_robustness_deflate.png.
As corrupted clients rise from 0% to 50%, the Robust baseline's coverage gap climbs past 0.45 and its coverage mean stays stuck near 0.78. FR-FCLC's coverage gap stays flat near 0.09 across the entire sweep.
Limitations
FR-FCLC is not a universal winner. The repository's paired significance tests show FR-FCLC's raw coverage is statistically comparable to the Fair baseline, not significantly better. Under both Byzantine settings, FR-FCLC's coverage gap is far smaller than Naive or Robust, but it does not fully clear the repository's strict 5% per-client fairness tolerance once the system is under attack. The honest claim is "most robust of the four methods in the published sweep," not "fair under every condition."
The validated results are BoolQ-only. The repository also preprocesses TruthfulQA, but full-vocabulary APS scoring for TruthfulQA has not been tested at the 100-client scale, so none of the coverage or fairness numbers above extend to it yet.
Technical Decisions
Use multi-seed experiments instead of reporting a single favorable run.
- A single run can overstate robustness or hide unstable behavior.
- Multi-seed evaluation adds compute time and result-management complexity.
- The comparison is more reproducible and more credible.
How I Work
Engineering Challenge
- Byzantine-robust aggregation alone still leaves wide per-client coverage gaps once corrupted clients push the global threshold off target.
- I swept Byzantine corruption from 0% to 50% under both inflate and deflate attacks across 5 seeds, comparing the Naive, Robust, Fair, and combined FR-FCLC methods on per-client coverage mean and gap.
- FR-FCLC layers fairness-aware per-client threshold adjustment on top of Byzantine-robust trimmed-quantile aggregation, so the published threshold is corrected for both corrupted updates and client heterogeneity.
- Across the full 0-50% deflate sweep the FR-FCLC coverage gap stays near 0.09, while the Robust baseline's gap climbs past 0.45 and its coverage mean stays near 0.78.
Quality & Testing
Quality Evidence
Product engineering evidence
- The README documents multi-seed evaluation, robustness sweeps, result JSON, and statistical analysis artifacts.
- No CI result is published for the research run; the public repository provides scripts and generated artifacts.
- The repository structure lists preprocessing, simulation, aggregation, and analysis modules.
Public page quality
- Result charts include descriptive captions and alt text.
- Diagrams and charts scale within the public case-study layout.
- The page publishes only evidence from the public repository: README result tables, experiment scripts, and generated plots.
Results & Impact
The public repository documents a complete 100-client federated calibration study: the simulation setup, generated plots, result JSON files, the 0-50% Byzantine sweep, and paired t-test significance analysis. In the honest setting FR-FCLC is the tightest of the four methods on per-client coverage (0.9432 mean, 0.0421 gap) and it stays the most robust method across the full attack sweep, within the limitations noted above.
What I Learned
Research engineering is both implementation and communication. The results only mean something alongside what was tested, what was not proven, and a repository the reader can run.