CommuteProp: Decoupled Training for Communication Bound Split LLM Fine-Tuning

Published in arXiv preprint arXiv:2610.05105, 2026

CommuteProp accelerates split learning for large language model fine-tuning by running cross-block gradient propagation and local weight updates concurrently. This asynchronous schedule overlaps network transfers with computation, reducing the idle time imposed by synchronous client-server training. The work analyzes convergence and asynchronous errors, and introduces an NS preconditioner to address noise from stale information. Experiments demonstrate higher training throughput with comparable accuracy and compatibility with existing privacy-enhancing techniques.