A Modular Federated Suite for Low-Rank, Expressive, and Efficient LLM Fine-Tuning

Vepakomma, Praneeth; Ponkshe, Kaustubh; Singhal, Raghav

Permalink

https://hdl.handle.net/2142/130308 Copy

Description

Title

A Modular Federated Suite for Low-Rank, Expressive, and Efficient LLM Fine-Tuning

Author(s)

• Vepakomma, Praneeth
• Ponkshe, Kaustubh
• Singhal, Raghav

Issue Date

2025-09-17

Keyword(s)

• LLM fine-tuning
• Federated LLM fine-tuning
• Pushing the efficiency-performance trade-off

Abstract

This works presents a modular federated suite of four methods—LoRA-SilverBullet (LoRA-SB), ABBA, Fed-SilverBullet, and FedEx-LoRA that together enable low-rank, expressive, and resource-efficient fine-tuning of large language models (LLMs). LoRA-SB approximates full fine-tuning within low-rank subspaces via a principled initialization strategy, provably preserving gradient directions and reducing trainable parameters by 27–90× without any additional hyperparameter tuning. ABBA reparameterizes weight updates as the Hadamard product of two low-rank matrices, and formally increasing expressivity under a fixed parameter budget. FedEx-LoRA introduces a lightweight residual correction to recover exact LoRA adapter updates under standard federated averaging, preserving efficiency with minimal overhead. Fed-SB leverages the LoRA-SB’s low-rank update with FedEx-LoRA’s exact aggregation in differentially private federated learning, and cuts communication costs by up to 230x. We will detail theoretical results on convergence, reconstruction bounds, communication complexity, and privacy loss, alongside empirical evaluations on reasoning and language benchmarks. This suite offers a principled path to deploy LLM fine-tuning in resource-constrained and privacy-sensitive federated environments.

Publisher

Allerton Conference on Communication, Control, and Computing

Series/Report Name or Number

2025 61st Allerton Conference on Communication, Control, and Computing Proceedings

ISSN

2836-4503

Type of Resource

Text

Genre of Resource

Conference Paper/Presentation

Language

eng

Handle URL

https://hdl.handle.net/2142/130308

Copyright and License Information

Copyright 2025 owned by the authors.

Owning Collections