Federated Learning for Your Thesis: A Practical Guide to Designing and Reporting Experiments

Federated learning (FL) is an attractive thesis topic: it combines machine learning, privacy and distributed systems, and it fits naturally with healthcare, IoT and mobile data where raw data cannot be pooled. It is also an area where reviewers are quick to spot weak experimental design. A model that "works" in one centralized run does not show much about how it behaves across many clients with different data. This guide explains how to design, run and report federated learning experiments so your thesis chapter or paper is reproducible and convincing.

1. Define the federated setting before you write code

Start by stating which kind of federated learning you are studying, because the design choices follow from it:

• Cross-silo FL: a small number of reliable clients, such as hospitals or organizations, each with a reasonably large dataset. Most clients usually take part in every round.
• Cross-device FL: a large number of unreliable clients, such as phones or sensors, each with little data. Only a fraction of clients participate in each round, and dropouts are normal.
• Horizontal vs vertical FL: in horizontal FL, clients share the same features but hold different samples; in vertical FL, clients hold different features about the same samples. Most thesis work is horizontal, so say so explicitly.

Write one short paragraph describing the real-world scenario you are simulating: who the clients are, why data cannot be centralized, and what the server is trusted to see. This framing tells the reader which results matter.

2. Simulate clients and non-IID data honestly

The heart of most FL experiments is how you split data across clients. An IID split (shuffling and dividing randomly) is a useful sanity check, but real federated data is usually non-IID, and results under IID splits can look much better than what you would see in practice. Common ways to create non-IID partitions include:

• Label skew with a Dirichlet distribution: each client's class proportions are drawn from a Dirichlet distribution with a concentration parameter (often called alpha). Smaller alpha means more skew. Report the alpha values you used and show results for more than one level of heterogeneity.
• Shard-based splits: sort data by label and give each client a few shards, so each client sees only a few classes.
• Natural partitions: split by a real attribute such as hospital, device, user or region. When such a split exists in your dataset, it is often more convincing than a synthetic one.
• Quantity skew: give clients different amounts of data, which affects weighting during aggregation.

Fix and report random seeds for the partitioning, and consider including a figure or table that shows the class distribution per client. Reviewers often ask for this, and it helps readers understand your heterogeneity.

3. Choose the right baselines

A federated method should be compared with at least three reference points:

1. Centralized training on the pooled data, as an approximate upper bound (it ignores privacy constraints, so it is a reference, not a competitor).
2. Local-only training, where each client trains on its own data, showing why collaboration is needed.
3. Standard FL algorithms: FedAvg as the basic baseline, and methods designed for heterogeneous data, such as FedProx (which adds a proximal term to limit client drift) or SCAFFOLD (which uses control variates). If your work is about personalization, include a personalized FL baseline as well.

Use the same model architecture, data splits and training budget for every method. An unfair budget, such as more local epochs for your method only, is one of the most common reasons reviewers question a result.

4. Report the training protocol in full

FL has more hyperparameters than centralized training, and missing details make work impossible to reproduce. In your methods section or appendix, list:

• Number of clients and the number (or fraction) sampled per round
• Number of communication rounds and local epochs per round
• Local optimizer, learning rate, batch size and any learning-rate schedule
• Aggregation rule and how client updates are weighted
• Algorithm-specific settings, such as the proximal coefficient for FedProx
• Framework and versions used, for example Flower, TensorFlow Federated, FedML or a custom PyTorch simulation

Plot accuracy (or your main metric) against communication rounds, not only final accuracy. In FL, how fast a method reaches a target performance is often as important as where it ends.

5. Evaluate beyond a single average number

Decide whether you evaluate on a global test set held by the server, on each client's local test set, or both, and say which. Then report:

• Mean and spread across runs: repeat experiments with several seeds and report mean and standard deviation or confidence intervals.
• Per-client performance: the worst-performing clients matter, especially in medical settings, where a model that fails for one hospital is a real problem. Fairness across clients is a valid result in itself.
• Communication cost: rounds to reach a target metric, model size or total bytes exchanged if your method claims efficiency.
• Task-appropriate metrics: for imbalanced medical data, use metrics such as AUC, sensitivity, specificity or F1 rather than accuracy alone.

6. Be precise about privacy

Keeping data on the client is not, by itself, a formal privacy guarantee; model updates can leak information. If your thesis claims privacy protection, be specific:

• If you use differential privacy, state whether it is at the client level or sample level, report the privacy budget (epsilon and delta), the clipping norm and the noise multiplier, and show the privacy-utility trade-off.
• If you use secure aggregation or encryption, explain the threat model: what the server and other clients can and cannot see.
• If you evaluate attacks, such as membership inference or gradient inversion, describe the attacker's assumptions clearly.

If your work does not add formal privacy mechanisms, say so plainly and present privacy as a limitation or future work. Overclaiming privacy is a common reason for critical reviews.

7. Common mistakes to avoid

• Testing only on IID splits and claiming the method handles heterogeneity
• Tuning hyperparameters on the test set, or tuning your method more carefully than the baselines
• Reporting a single run without variance
• Leaking data between clients through shared preprocessing statistics, such as global normalization computed on all data, without mentioning it
• Calling a single-machine simulation a "real deployment"
• Omitting the code or partition scripts when the venue encourages sharing them

8. A simple structure for your FL chapter

A clear order helps examiners and reviewers: problem setting and threat model, data and client partitioning, method, baselines, training protocol, results (convergence, final performance, per-client analysis, communication cost), privacy analysis, ablation studies, and limitations. If each section answers "what exactly did you do, and how can someone repeat it?", your federated learning work will be much easier to defend.