Loss Resilient Real-Time Video via Neural Codec

In real-time video streaming, packet loss can severely impair user experience by causing video freezes or distortion. Although there are several existing solutions such as Forward Error Correction (FEC), and error concealment, these primarily enhance loss resiliency either at the sender or decoder end. GRACE introduces a novel approach by jointly optimizing both the encoder and decoder to handle various packet loss scenarios. This method uses the neural video codecs jointly trained under simulated packet losses to maintain decent quality in decoded frames despite packet losses. It enables direct decoding of loss-impacted frames without waiting for retransmissions while ensuring high video quality.