Space- and Computationally-Efficient Set Reconciliation via Parity Bitmap Sketch (PBS)

Set reconciliation is a fundamental algorithmic problem that arises in many networking, system, and database applications. In this problem, two large setsAandBof objects (bitcoins, files, records, etc.) are stored respectively at two different network-connected hosts, which we name Alice and Bob respectively. Alice and Bob communicate with each other to learnAΔB, the difference betweenAandB, and as a result the reconciled setA∪B.Current set reconciliation schemes are based on either invertible Bloom filters (IBF) or error-correction codes (ECC). The former has a low computational complexity ofO(d), wheredis the cardinality ofAΔB, but has a high communication overhead that is several times larger than the theoretical minimum. The latter has a low communication overhead close to the theoretical minimum, but has a much higher computational complexity ofO(d2). In this work, we propose Parity Bitmap Sketch (PBS), an ECC-based set reconciliation scheme that gets the better of both worlds: PBS has both a low computational complexity ofO(d)just like IBF-based solutions and a low communication overhead of roughly twice the theoretical minimum. A separate contribution of this work is a novel rigorous analytical framework that can be used for the precise calculation of various performance metrics and for the near-optimal parameter tuning of PBS.