Optimal and Suboptimal Decoders under Finite-Alphabet Interference: A Mismatched Decoding Perspective

Interference widely exists in communication systems and is often not optimally treated at the receivers due to limited knowledge and/or computational burden. Evolutions of receivers have been proposed to balance complexity and spectral efficiency, for example, for 6G, while commonly used performance metrics, such as capacity and mutual information (MI), fail to capture the suboptimal treatment of interference, leading to potentially inaccurate performance evaluations. Mismatched decoding is an information-theoretic tool for analyzing communications with suboptimal decoders. In this work, we use mismatched decoding to analyze communications with decoders that treat interference suboptimally, aiming at more accurate performance metrics. Specifically, we consider a finite-alphabet input Gaussian channel under interference, representative of modern systems, where the decoder can be matched (optimal) or mismatched (suboptimal) to the channel. The matched capacity is derived using MI, while a lower bound on the mismatched capacity under various decoding metrics is derived using generalized mutual information (GMI). We show that the decoding metric in the proposed channel model is closely related to the behavior of the demodulator in bit-interleaved coded modulation (BICM) systems. Simulations illustrate that GMI/MI accurately predicts the throughput of BICM-type systems {with various demodulators}. Finally, we extend the channel model and the GMI to multiple antenna cases, with an example of multi-user multiple-input-single-output (MU-MISO) precoder optimization problem considering GMI under different decoding strategies. In short, this work discovers new insights about the impact of interference, proposes novel receivers, and introduces a new design and performance evaluation framework that more accurately captures the effect of interference.