Stefano Rini, Daniela Tuninetti, Natasha Devroye
The capacity of the two-user Gaussian cognitive interference channel, a variation of the classical interference channel where one of the transmitters has knowledge of both messages, is known in several parameter regimes but remains unknown in general. In this paper, we consider the following achievable scheme: the cognitive transmitter pre-codes its message against the interference created at its intended receiver by the primary user, and the cognitive receiver only decodes its intended message, similar to the optimal scheme for "weak interference"; the primary decoder decodes both messages, similar to the optimal scheme for "very strong interference". Although the cognitive message is pre-coded against the primary message, by decoding it, the primary receiver obtains information about its own message, thereby improving its rate. We show: (1) that this proposed scheme achieves capacity in what we term the "primary decodes cognitive" regime, i.e., a subset of the "strong interference" regime that is not included in the "very strong interference" regime for which capacity was known; (2) that this scheme is within one bit/s/Hz, or a factor two, of capacity for a much larger set of parameters, thus improving the best known constant gap result; (3) we provide insights into the trade-off between interference pre-coding at the cognitive encoder and interference decoding at the primary receiver based on the analysis of the approximate capacity results.
Stefano Rini, Daniela Tuninetti, Natasha Devroye, Andrea Goldsmith
The InterFerence Channel with a Cognitive Relay (IFC-CR) consists of the classical interference channel with two independent source-destination pairs whose communication is aided by an additional node, referred to as the cognitive relay, that has a priori knowledge of both sources' messages. This a priori message knowledge is termed cognition and idealizes the relay learning the messages of the two sources from their transmissions over a wireless channel. This paper presents new inner and outer bounds for the capacity region of the general memoryless IFC-CR that are shown to be tight for a certain class of channels. The new outer bound follows from arguments originally devised for broadcast channels among which Sato's observation that the capacity region of channels with non-cooperative receivers only depends on the channel output conditional marginal distributions. The new inner bound is shown to include all previously proposed coding schemes and it is thus the largest known achievable rate region to date. The new inner and outer bounds coincide for a subset of channel satisfying a strong interference condition. For these channels there is no loss in optimality if both destinations decode both messages. This result parallels analogous results for the classical IFC and for the cognitive IFC and is the first known capacity result for the general IFC-CR. Numerical evaluations of the proposed inner and outer bounds are presented for the Gaussian noise case.
Stefano Rini, Andrea Goldsmith
A unified approach to the derivation of rate regions for single-hop memoryless networks is presented. A general transmission scheme for any memoryless, single-hop, k-user channel with or without common information, is defined through two steps. The first step is user virtualization: each user is divided into multiple virtual sub-users according to a chosen rate-splitting strategy which preserves the rates of the original messages. This results in an enhanced channel with a possibly larger number of users for which more coding possibilities are available. Moreover, user virtualization provides a simple mechanism to encode common messages to any subset of users. Following user virtualization, the message of each user in the enhanced model is coded using a chosen combination of coded time-sharing, superposition coding and joint binning. A graph is used to represent the chosen coding strategies: nodes in the graph represent codewords while edges represent coding operations. This graph is used to construct a graphical Markov model which illustrates the statistical dependency among codewords that can be introduced by the superposition coding or joint binning. Using this statistical representation of the overall codebook distribution, the error probability of the code is shown to vanish via a unified analysis. The rate bounds that define the achievable rate region are obtained by linking the error analysis to the properties of the graphical Markov model. This proposed framework makes it possible to numerically obtain an achievable rate region by specifying a user virtualization strategy and describing a set of coding operations. The largest achievable rate region can be obtained by considering all the possible rate-splitting strategies and taking the union over all the possible ways to superimpose or bin codewords.
Stefano Rini, Daniela Tuninetti, Natasha Devroye
The InterFerence Channel with a Cognitive Relay (IFC-CR) consists of a classical two-user interference channel in which the two independent messages are also non-causally known at a cognitive relay node. In this work a special class of IFC-CRs in which the sources do not create interference at the non-intended destinations is analyzed. This special model results in a channel with two non-interfering point-to-point channels whose transmission is aided by an in-band cognitive relay, which is thus referred to as the Parallel Channel with a Cognitive Relay (PC-CR). We determine the capacity of the PC-CR channel to within 3 bits/s/Hz for all channel parameters. In particular, we present several new outer bounds which we achieve to within a constant gap by proper selection of Gaussian input distributions in a simple rate-splitting and superposition coding-based inner bound. The inner and outer bounds are numerically evaluated to show that the actual gap can be far less than 3 bits/s/Hz.
Ali Khajegili Mirabadi, Stefano Rini
Feature extraction and description is an important topic of computer vision, as it is the starting point of a number of tasks such as image reconstruction, stitching, registration, and recognition among many others. In this paper, two new image features are proposed: the Information Ratio (IR) and the Mutual Information Ratio (MIR). The IR is a feature of a single image, while the MIR describes features common across two or more images.We begin by introducing the IR and the MIR and motivate these features in an information theoretical context as the ratio of the self-information of an intensity level over the information contained over the pixels of the same intensity. Notably, the relationship of the IR and MIR with the image entropy and mutual information, classic information measures, are discussed. Finally, the effectiveness of these features is tested through feature extraction over INRIA Copydays datasets and feature matching over the Oxfords Affine Covariant Regions. These numerical evaluations validate the relevance of the IR and MIR in practical computer vision tasks
Stefano Rini, Shlomo Shamai
The "Writing onto Fast Fading Dirt" (WFFD) channel is investigated to study the effects of partial channel knowledge on the capacity of the "writing on dirty paper" channel. The WFFD channel is the Gel'fand-Pinsker channel in which the output is obtained as the sum of the input, white Gaussian noise and a fading-times-state term. The fading-times-state term is equal to the element-wise product of the channel state sequence, known only at the transmitter, and a fast fading process, known only at the receiver. We consider the case of Gaussian distributed channel states and derive an approximate characterization of capacity for different classes of fading distributions, both continuous and discrete. In particular, we prove that if the fading distribution concentrates in a sufficiently small interval, then capacity is approximately equal to the AWGN capacity times the probability of this interval. We also show that there exists a class of fading distributions for which having the transmitter treat the fading-times-state term as additional noise closely approaches capacity. Although a closed-form expression of the capacity of the general WFFD channel remains unknown, our results show that the presence of fading can severely reduce the usefulness of channel state knowledge at the transmitter.
Alon Kipnis, Stefano Rini, Andrea J. Goldsmith
Consider the problem of estimating a latent signal from a lossy compressed version of the data when the compressor is agnostic to the relation between the signal and the data. This situation arises in a host of modern applications when data is transmitted or stored prior to determining the downstream inference task. Given a bitrate constraint and a distortion measure between the data and its compressed version, let us consider the joint distribution achieving Shannon's rate-distortion (RD) function. Given an estimator and a loss function associated with the downstream inference task, define the rate-distortion risk as the expected loss under the RD-achieving distribution. We provide general conditions under which the operational risk in estimating from the compressed data is asymptotically equivalent to the RD risk. The main theoretical tools to prove this equivalence are transportation-cost inequalities in conjunction with properties of compression codes achieving Shannon's RD function. Whenever such equivalence holds, a recipe for designing estimators from datasets undergoing lossy compression without specifying the actual compression technique emerges: design the estimator to minimize the RD risk. Our conditions simplified in the special cases of discrete memoryless or multivariate normal data. For these scenarios, we derive explicit expressions for the RD risk of several estimators and compare them to the optimal source coding performance associated with full knowledge of the relation between the latent signal and the data.
Stefano Rini, Shlomo Shamai Shitz
Costa's "writing on dirty paper" capacity result establishes that full state pre-cancellation can be attained in Gelfand-Pinsker channel with additive state and additive Gaussian noise. The "carbon copy onto dirty paper" channel is the extension of Costa's model to the compound setting: M receivers each observe the sum of the channel input, Gaussian noise and one of M Gaussian state sequences and attempt to decode the same common message. The state sequences are all non-causally known at the transmitter which attempts to simultaneously pre-code its transmission against the channel state affecting each output. In this correspondence we derive the capacity to within 2.25 bits-per-channel-use of the carbon copying onto dirty paper channel in which the state sequences are statistically equivalent, having the same variance and the same pairwise correlation. For this channel capacity is approached by letting the channel input be the superposition of two codewords: a base codeword, simultaneously decoded at each user, and a top codeword which is pre-coded against the state realization at each user for a portion 1/M of the time. The outer bound relies on a recursive bounding in which incremental side information is provided at each receiver. This result represents a significant first step toward determining the capacity of the most general "carbon copy onto dirty paper" channel in which state sequences appearing in the different channel outputs have any jointly Gaussian distribution.
Stefano Rini, Shlomo Shamai
The classical writing on dirty paper capacity result establishes that full interference pre-cancellation can be attained in Gelfand-Pinsker problem with additive state and additive white Gaussian noise. This result holds under the idealized assumption that perfect channel knowledge is available at both transmitter and receiver. While channel knowledge at the receiver can be obtained through pilot tones, transmitter channel knowledge is harder to acquire. For this reason, we are interested in characterizing the capacity under the more realistic assumption that only partial channel knowledge is available at the transmitter. We study, more specifically, the dirty paper channel in which the interference sequence in multiplied by fading value unknown to the transmitter but known at the receiver. For this model, we establish an approximate characterization of capacity for the case in which fading values vary greatly in between channel realizations. In this regime, which we term the strong fading regime, the capacity pre-log factor is equal to the inverse of the number of possible fading realizations.
Stefano Rini, Daniela Tuninetti, Natasha Devroye
The cognitive interference channel (C-IFC) consists of a classical two-user interference channel in which the message of one user (the "primary" user) is non-causally available at the transmitter of the other user (the "cognitive" user). We obtain the capacity of the semi-deterministic C-IFC: a discrete memoryless C-IFC in which the cognitive receiver output is a noise-less deterministic function of the channel inputs. We then use the insights obtained from the capacity-achieving scheme for the semi-deterministic model to derive new, unified and tighter constant gap results for the complex-valued Gaussian C-IFC. We prove: (1) a constant additive gap (difference between inner and outer bounds) of half a bit/sec/Hz per real dimension, of relevance at high SNRs, and (b) a constant multiplicative gap (ratio between outer and inner bounds) of a factor two, of relevance at low SNRs
Alexios Balatsoukas-Stimming, Stefano Rini, Joerg Kliewer
The joint design of input constellation and low-density parity-check (LDPC) codes to approach the symmetric capacity of the two-user Gaussian multiple access channel is studied. More specifically, multilevel coding is employed at each user to construct a high-order input constellation and the constellations of the users are jointly designed so as to maximize the multiuser shaping gain. At the receiver, each layer of the multilevel coding is jointly decoded among users, while successive cancellation is employed across layers. The LDPC code employed by each user in each layer is designed using EXIT charts to support joint decoding among users for the prescribed per-layer rate and SNR. Numerical simulations are provided to validate the proposed constellation and LDPC code designs.
Milind Rao, Stefano Rini, Andrea Goldsmith
In this paper, a distributed convex optimization algorithm, termed \emph{distributed coordinate dual averaging} (DCDA) algorithm, is proposed. The DCDA algorithm addresses the scenario of a large distributed optimization problem with limited communication among nodes in the network. Currently known distributed subgradient methods, such as the distributed dual averaging or the distributed alternating direction method of multipliers algorithms, assume that nodes can exchange messages of large cardinality. Such network communication capabilities are not valid in many scenarios of practical relevance. In the DCDA algorithm, on the other hand, communication of each coordinate of the optimization variable is restricted over time. For the proposed algorithm, we bound the rate of convergence under different communication protocols and network architectures. We also consider the extensions to the case of imperfect gradient knowledge and the case in which transmitted messages are corrupted by additive noise or are quantized. Relevant numerical simulations are also provided.
Busra Tegin, Eduin E. Hernandez, Stefano Rini, Tolga M. Duman
Large-scale machine learning and data mining methods routinely distribute computations across multiple agents to parallelize processing. The time required for computation at the agents is affected by the availability of local resources giving rise to the "straggler problem" in which the computation results are held back by unresponsive agents. For this problem, linear coding of the matrix sub-blocks can be used to introduce resilience toward straggling. The Parameter Server (PS) utilizes a channel code and distributes the matrices to the workers for multiplication. It then produces an approximation to the desired matrix multiplication using the results of the computations received at a given deadline. In this paper, we propose to employ Unequal Error Protection (UEP) codes to alleviate the straggler problem. The resiliency level of each sub-block is chosen according to its norm as blocks with larger norms have higher effects on the result of the matrix multiplication. We validate the effectiveness of our scheme both theoretically and through numerical evaluations. We derive a theoretical characterization of the performance of UEP using random linear codes, and compare it the case of equal error protection. We also apply the proposed coding strategy to the computation of the back-propagation step in the training of a Deep Neural Network (DNN), for which we investigate the fundamental trade-off between precision and the time required for the computations.
Stefano Rini
Proving capacity for networks without feedback or cooperation usually involves two fundamental random coding techniques: superposition coding and binning. Although conceptually very different, these two techniques often achieve the same performance, suggesting an underlying similarity. In this correspondence we propose a new random coding technique that generalizes superposition coding and binning and provides new insight on relationship among the two With this new theoretical tool, we derive new achievable regions for three classical information theoretical models: multi-access channel, broadcast channel, the interference channel, and show that, unfortunately, it does not improve over the largest known achievable regions for these cases.
Stefano Rini, Andrea Goldsmith
A unified graphical approach to random coding for any memoryless, single-hop, K-user channel with or without common information is defined through two steps. The first step is user virtualization: each user is divided into multiple virtual sub-users according to a chosen rate-splitting strategy. This results in an enhanced channel with a possibly larger number of users for which more coding possibilities are available and for which common messages to any subset of users can be encoded. Following user virtualization, the message of each user in the enhanced model is coded using a chosen combination of coded time-sharing, superposition coding and joint binning. A graph is used to represent the chosen coding strategies: nodes in the graph represent codewords while edges represent coding operations. This graph is used to construct a graphical Markov model which illustrates the statistical dependency among codewords that can be introduced by the superposition coding or joint binning. Using this statistical representation of the overall codebook distribution, the error probability of the code is shown to vanish via a unified analysis. The rate bounds that define the achievable rate region are obtained by linking the error analysis to the properties of the graphical Markov model. This proposed framework makes it possible to numerically obtain an achievable rate region by specifying a user virtualization strategy and describing a set of coding operations. The union of these rate regions defines the maximum achievable rate region of our unified coding strategy.
Rajarshi Saha, Stefano Rini, Milind Rao, Andrea Goldsmith
In decentralized optimization, multiple nodes in a network collaborate to minimize the sum of their local loss functions. The information exchange between nodes required for this task, is often limited by network connectivity. We consider a setting in which communication between nodes is hindered by both (i) a finite rate-constraint on the signal transmitted by any node, and (ii) additive noise corrupting the signal received by any node. We propose a novel algorithm for this scenario: Decentralized Lazy Mirror Descent with Differential Exchanges (DLMD-DiffEx), which guarantees convergence of the local estimates to the optimal solution under the given communication constraints. A salient feature of DLMD-DiffEx is the introduction of additional proxy variables that are maintained by the nodes to account for the disagreement in their estimates due to channel noise and rate-constraints. Convergence to the optimal solution is attained by having nodes iteratively exchange these disagreement terms until consensus is achieved. In order to prevent noise accumulation during this exchange, DLMD-DiffEx relies on two sequences; one controlling the power of the transmitted signal, and the other determining the consensus rate. We provide clear insights on the design of these two sequences which highlights the interplay between consensus rate and noise amplification. We investigate the performance of DLMD-DiffEx both from a theoretical perspective as well as through numerical evaluations.
Stefano Rini, Carolin Huppert
In this paper the cognitive interference channel with a common message, a variation of the classical cognitive interference channel in which the cognitive message is decoded at both receivers, is studied. For this channel model new outer and inner bounds are developed as well as new capacity results for both the discrete memoryless and the Gaussian case. The outer bounds are derived using bounding techniques originally developed by Sato for the classical interference channel and Nair and El Gamal for the broadcast channel. A general inner bound is obtained combining rate-splitting, superposition coding and binning. Inner and outer bounds are shown to coincide in the "very strong interference" and the "primary decodes cognitive" regimes. The first regime consists of channels in which there is no loss of optimality in having both receivers decode both messages while in the latter regime interference pre-cancellation at the cognitive receiver achieves capacity. Capacity for the Gaussian channel is shown to within a constant additive gap and a constant multiplicative factor.
Andrea Grigorescu, Marek Rudnicki, Michael Isik, Werner Hemmert, Stefano Rini
Apr 23, 2012·q-bio.NC·PDF In this correspondence information theoretical tools are used to investigate the statistical properties of modeled cochlear nucleus globular bushy cell spike trains. The firing patterns are obtained from a simulation software that generates sample spike trains from any auditory input. Here we analyze for the first time the responses of globular bushy cells to voiced and unvoiced speech sounds. Classical entropy estimates, such as the direct method, are improved upon by considering a time-varying and time-dependent entropy estimate. With this method we investigated the relationship between the predictability of the neuronal response and the frequency content in the auditory signals. The analysis quantifies the temporal precision of the neuronal coding and the memory in the neuronal response.
Stefano Rini
The capacity of the multiple-access channel with any distribution of messages among the transmitting nodes was determined by Han in 1979 and the expression of the capacity region contains a number of rate bounds and that grows exponentially with the number of messages. We derive a more compact expression for the capacity region of this channel in which the number of rate bounds depends on the distribution of the messages at the encoders. Using this expression we prove capacity for a class of general cognitive network that we denote as "very strong interference" regime. In this regime there is no rate loss in having all the receivers decode all the messages and the capacity region reduces to the capacity of the compound multiple-access channel. This result generalizes the "very strong interference" capacity results for the interference channel, the cognitive interference channel, the interference channel with a cognitive relay and many others.
Stefano Rini, Carolin Huppert
In this paper the study of the cognitive interference channel with a common message, a variation of the classical cognitive interference channel in which the cognitive message is decoded at both receivers. We derive the capacity for the semideterministic channel in which the output at the cognitive decoder is a deterministic function of the channel inputs. We also show capacity to within a constant gap and a constant factor for the Gaussian channel in which the outputs are linear combinations of the channel inputs plus an additive Gaussian noise term. Most of these results are shown using an interesting transmission scheme in which the cognitive message, decoded at both receivers, is also pre-coded against the interference experienced at the cognitive decoder. The pre-coding of the cognitive message does not allow the primary decoder to reconstruct the interfering signal. The cognitive message acts instead as a side information at the primary receiver when decoding its intended message.