%A Konečný,Jakub
%A Richtárik,Peter
%D 2018
%J Frontiers in Applied Mathematics and Statistics
%C
%F
%G English
%K Communication efficiency,Distributed mean estimation,Accuracy-communication tradeoff,Gradient compression,quantization
%Q
%R 10.3389/fams.2018.00062
%W
%L
%N 62
%M
%P
%7
%8 2018-December-18
%9 Original Research
%#
%! Randomized Distributed Mean Estimation: Accuracy vs Communication
%*
%<
%T Randomized Distributed Mean Estimation: Accuracy vs. Communication
%U https://www.frontiersin.org/article/10.3389/fams.2018.00062
%V 4
%0 JOURNAL ARTICLE
%@ 2297-4687
%X We consider the problem of estimating the arithmetic average of a finite collection of real vectors stored in a distributed fashion across several compute nodes subject to a communication budget constraint. Our analysis does not rely on any statistical assumptions about the source of the vectors. This problem arises as a subproblem in many applications, including reduce-all operations within algorithms for distributed and federated optimization and learning. We propose a flexible family of randomized algorithms exploring the trade-off between expected communication cost and estimation error. Our family contains the full-communication and zero-error method on one extreme, and an ϵ-bit communication and O(1/(∈n)) error method on the opposite extreme. In the special case where we communicate, in expectation, a single bit per coordinate of each vector, we improve upon existing results by obtaining O(r/n) error, where r is the number of bits used to represent a floating point value.