Firstly, the input sequence is downscaled using a non-normative down-scaler. Depending on the chosen encoder configuration, the downscaling can be applied up to two consecutive times. The video, now at a lower resolution than the input sequence, is fed into the base encoder, which may be compliant with the H.264/AVC, H.265/HEVC, H.266/VVC standards or even a proprietary solution: LCEVC is agnostic to the base layer compression technology employed. Accordingly, the base layer compression process is not further specified in LCEVC: any encoder that produces a decodable bitstream can be used.

The normative up-scaler specified in the standard (ISO/IEC DIS, 2022), is employed to produce an upscaled version of the base-layer which is then used as input to this encoding stage. The enhancement sub-layer 1 (L-1) residuals are created by subtracting, for each color plane, the downscaled input sequence and the reconstructed base-layer. The residuals obtained are typically sparse, that is containing samples with value close to zero, except where image edges or textured details are present in the original content. Residuals associated with each color plane are then partitioned over a non overlapping grid of coding blocks with size either 4 × 4 or 2 × 2, depending on the selected encoder’s configuration. Over each coding block, spatial transformation, quantization and entropy encoding are applied. Details of the different tools used for transform, quantization and entropy encoding will be provided after the three main encoding stages are described. After the sub-layer 1 residuals are encoded, the inverse quantization and inverse transform stage are applied. Additionally, a filter can be applied to the sub-layer 1 reconstructed residuals which functions as a simple deblocking filter. Worth mentioning here that the LCEVC standard supports the signaling of different values for the kernels used to upscale the base-layer to the sub-layer L-1, moreover a nonlinear correction denoted as Predicted Residuals (PR) may also be used to zero out and/or adjust the sample values with the ultimate goal of improving the coding efficiency of the different coefficient groups. The PR may be seen as a pre-processing step which increases the probability of having a sparse residual signal with long runs of zero values, so that the associated entropy is further reduced. The transform used in LCEVC has a simple structure and uses a small kernel of size 2 × 2 or 4 × 4. This allows to both efficiently code sparse information and parallelize the transforms, since individual blocks are not dependent on other blocks within a frame. A linear quantizer, which may include an adaptive dead-zone, is used to further process the transform coefficients. The entropy encoder, which consists of a run-length encoder (RLE) and an optional prefix encoder (Huffman encoder), processes the quantized transform coefficients and creates the coefficient groups for sub-layer 1.

The sub-layer 1 reconstructed residuals are upscaled to full resolution and subtracted from the original input sequence pixels. The resulting enhancement sub-layer 2 (L-2) residuals are fed into the temporal prediction module depicted in Figure 4. LCVEC uses a zero-motion vector temporal scheme which operates on a coding block basis. The residuals from the previous frame are stored in a temporal buffer and are added to the L-2 residuals in case the temporal prediction is activated. To reduce the signaling overhead in the case of a fast-moving sequence, where the zero-motion compensation scheme would likely not be beneficial, temporal prediction may be disabled by simply transmitting a binary flag for a group of pixels of size 32 × 32 or for the entire frame. As for the case of sub-layer 1 residuals, also the sub-layer 2 residuals are transformed, quantized and entropy encoded using the same tools from sub-layer 1.

Now that the high level workflow specified by the LCEVC standard has been described, the details associated with the coding tools for transform, quantization and entropy coding can be provided.

LCEVC uses a linear block-based transformation which is similar to 2 × 2 or 4 × 4 Hadamard kernel as shown by the following two matrix formulations where R x y are the residuals and d x y are the transform coefficients. { R 00 } { R 01 } { R 10 } { R 11 } = { 1,1,1,1 } { 1 , − 1,1 , − 1 } { 1,1 , − 1 , − 1 } { 1 , − 1 , − 1,1 } * { d 00 } { d 01 } { d 10 } { d 11 } , { R 00 } { R 01 } { R 02 } { R 03 } { R 10 } { R 11 } { R 12 } { R 13 } { R 20 } { R 21 } { R 22 } { R 23 } { R 30 } { R 31 } { R 32 } { R 33 } = { 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 } { 1 1 1 1 − 1 − 1 − 1 − 1 1 1 1 1 − 1 − 1 − 1 − 1 } { 1 − 1 1 − 1 1 − 1 1 − 1 1 − 1 1 − 1 1 − 1 1 − 1 } { 1 − 1 1 − 1 − 1 1 − 1 1 1 − 1 1 − 1 − 1 1 − 1 − 1 } { 1 1 1 1 1 1 1 1 − 1 − 1 − 1 − 1 − 1 − 1 − 1 − 1 } { 1 1 1 1 − 1 − 1 − 1 − 1 − 1 − 1 − 1 − 1 1 1 1 1 } { 1 − 1 1 − 1 1 − 1 1 − 1 − 1 1 − 1 1 − 1 1 − 1 1 } { 1 − 1 1 − 1 − 1 1 − 1 1 − 1 1 − 1 1 1 − 1 1 − 1 } { 1 1 − 1 − 1 1 1 − 1 − 1 1 1 1 1 − 1 − 1 1 1 } { 1 1 − 1 − 1 − 1 − 1 1 1 1 1 1 − 1 − 1 − 1 1 1 } { 1 − 1 − 1 1 1 − 1 − 1 1 1 − 1 − 1 1 1 − 1 − 1 1 } { 1 − 1 − 1 1 − 1 1 1 − 1 1 − 1 − 1 1 − 1 1 1 − 1 } { 1 1 − 1 − 1 1 1 − 1 − 1 − 1 − 1 1 1 − 1 − 1 1 1 } { 1 1 − 1 − 1 − 1 − 1 1 1 − 1 − 1 1 1 1 1 − 1 − 1 } { 1 − 1 − 1 1 1 − 1 − 1 1 − 1 1 1 − 1 − 1 1 1 − 1 } { 1 − 1 − 1 1 − 1 1 1 − 1 − 1 1 1 − 1 1 − 1 − 1 1 } * { d 00 } { d 01 } { d 02 } { d 03 } { d 10 } { d 11 } { d 12 } { d 13 } { d 20 } { d 21 } { d 22 } { d 23 } { d 30 } { d 31 } { d 32 } { d 33 }

The use of the Hadamard transform is due to its better energy compaction of sparse input data, which is indeed the case for the enhancement layers prediction residuals. Moreover, the block size is limited to 4 × 4 to allow for parallel encoding implementations, whereby each block can be transformed independently on a multi threaded CPU or GPU architecture. The size used by the encoder is transmitted in the bitstream as coding metadata.

LCEVC specifies a uniform scalar quantizer with dead-zone adjustment. The quantizer can use different quantization steps (denoted as quantization parameters in the standard document) for the two enhancement sub-layers allowing to balance the bit budget between the two sub-layers and decide where to add more details. Reconstruction of residuals using the inverse quantization process can be performed using asymmetric dequantization whereby the dequantized coefficient values can be offset to correct the edge of the bins depending on the magnitude of the quantization parameter used.

The quantized transform coefficients are grouped together with respect to their associated frequency band. Over each band, the quantized coefficients of each coding block are encoded using a simple run-length encoding. More precisely, ( r , v ) pairs are formed from the coefficients with r being the run length of a consecutive group of zero valued quantized coefficients and v is the value which interrupts the run. The values for r and v are entropy encoded using prefix coding, whereas a different table is used for r and v and transmitted in the bitstream.

As already mentioned, LCEVC uses a zero motion compensation temporal prediction scheme which operates over the prediction residuals. The standard’s syntax supports the possibility to signal the use of inter prediction on a coding block basis via a binary flag. The ensemble of these binary flags is entropy encoded to improve the whole coding efficiency.

For the target bitrates, the following Quantization Parameter (QP) values have been selected:

• For the full resolution anchors, we have used the overall target bitrate as specified in (Call for Proposal Phase 2, 2021) (i.e., TC1.1.4, TC1.2.4, TC11.4 and TC11.5 from the CfP), namely:

Although not necessarily optimal, the proportions specified above for the Base Layer coding rate are more aligned with those used in the MPEG Verification Test (Verification Test Report on the Compression Performance of Low Complexity Enhancement Video Coding, 2021), and are closer to the appropriate proportions used by LCEVC when encoding the enhancement layers. Because of the enhancement nature of the layers above the one, such a proportions results in good visual quality (see Section 4.4 for further details). It is worth noting that the spread of the total bitrate among the base and enhancement layers is central for rate control purposes in practical implementation of LCEVC. However, given that the TV 3.0 project target the assessment of the shear coding advantage brought by the LCEVC technology, a more naïve and fixed rate allocation across layers was used.

For the target bitrates, the following QP values have been selected:

• For the full resolution anchors, we have used the overall target bitrate as specified in (Call for Proposal Phase 2, 2021) (TC1.1.4, TC1.2.4, TC11.4 and TC11.5 from the CfP) namely:

Formal subjective evaluations have been conducted for the tests specified in Section 4.1. The tests were conducted by GBTech laboratories and VABTech laboratories under the following conditions:

• Test Method: DSIS (11 grades impairment scale) according to ITU-R BT.500

In Tables 1–4 we report the BD-rates computed using the MOS scores for both PQ And HLG material including the MOS scores and the confidence intervals (CI)

As it may be seen, LCEVC enhancing H.266/VVC provides an average bitrate saving of between 52% and 54% for both PQ and HLG sequences when Configuration A is used, and an average bitrate saving of almost 50% for HLG sequences and 36% for PQ sequences when Configuration B is used instead.

In Figure 6 we include a couple of exemplary charts - one for PQ (clip SBT_vc04-dre) and one for HLG (clip SBT_vc10_globo) - to illustrate the typical rate distortion curves for the subjective formal assessments.

In Tables 5, 6, we report the Bjøntegaard Delta (BD) on rate (BD-rate) savings for Configuration A computed based on a selection of the objective quality metrics used in (Call for Proposal Phase 2, 2021) (i.e. VMAF, MS-SSIM and PSNR).

It is important to note that objective metrics based on MSE, such as PSNR, are known to be an unreliable predictor of visual quality (see for example (Wang and Bovik, 2009), (Lin et al., 2003) and (On learning based video quality metrics, 2021)), and this is particularly true for multi-resolution schemes such as LCEVC (see for example (Lcevc, 2020b)). Therefore, it is expected that BD-rates based on PSNR are not favorable for LCEVC, especially in the range of non visually lossless coding0F ¹ .

Figure 7 shows an illustrative example of the phenomena at play when small quantization errors in mid-low frequencies (quite visible due to block correlation) are substituted with more accurate mid-low frequencies and larger quantization errors in fine details. Due to the quadratic nature of Mean Squared Error, in lossy range the left picture is scored better by PSNR despite obviously worse visual quality and a higher number of errors than the right picture.

Although perceptual-based objective metrics such MS-SSIM and VMAF are better predictors of visual quality - with VMAF being probably the best to date - they are still not fully correlated with visual quality, which for LCEVC tends to outperform even those metrics.

Accordingly, we provide the results of the formal subjective assessments to reliably evaluate the comparative quality of the encoded sequences and use the objective metrics purely for cross-checking the sequences.

In this section we have compared the results of Configuration A and Configuration B.

In both tests the overall target bitrate (i.e., the bitrate for the base layer plus the bitrate for the enhancement layer) is substantially the same. However, the percentage of the overall target bitrate allocated to the base layer (and, consequently, to the enhancement layer) is different.

Specifically, in Configuration A more bitrate is allocated to the base layer than to the enhancement layer (on average, between 82% and 85% to the base layer and between 15% and 18% to the enhancement), whilst in Configuration B more bitrate is allocated to the enhancement layer than to the base layer (on average, between 38% and 41% to the base layer and between 59% and 62% to the enhancement—more akin to what would be used for a traditional scalable scheme).

As it may be seen from the results, Configuration A provides a much better performance as it is better suited for how LCEVC works.

Contrary to typical scalable encoding schemes, LCEVC is very efficient at predicting from the base layer and at encoding the residuals in the enhancement layer, therefore requiring only a small percentage of an overall bitrate. The larger remaining portion of the overall bitrate can be used to encode a base layer at a much higher quality.

The overall effect is that with LCEVC used with appropriate bitrate allocation not only the quality of the full resolution is higher, but also the quality of the base layer is much higher than if a scalable-like bitrate allocation were used. This means that—using the same overall bitrate—it is possible to provide both a higher-quality higher resolution service (e.g., 4k/UHD) and a higher-quality lower resolution service (e.g., 1080p).

In Figures 8–10 we show a selection of graphs comparing Configuration A (LCEVC-like proportion—green tones) and Configuration B (scalable-like proportion—red tones) for the base layer (HD) and the LCEVC-enhanced full resolution (UHD).

As it may be seen, both base layers and LCEVC-enhanced full resolution in Configuration A are significantly better for all metrics and at all rate points.

In this scenario, we compared HM as an anchor and LCEVC-enhanced H.266/VVC (i.e., base layer and enhancement layer), both transmitted over a single 6MHz/5 Mbps channel. As such, the target bitrate for each encoded stream is 5 Mbps. The actual bitrate achieved is typically within 10% of the target bitrate, mainly due to the HM quantization and VVenC rate control.

As done with Test 1, in Tables 7, 8 are reporting the results of formal subjective assessments performed on these sequences and conducted at the same time and with the same settings as reported in Section 4.1.5 for Test 1.

In Figure 11 we also represent graphically the same data reported in the above tables.

As it can be seen LCEVC-enhancing H.266/VVC provides a significant improvement in visual quality over the HM anchor—over 1.3 MOS points of improvement on average—whilst staying very close to the 5 Mbps target bitrate.

In this scenario, we again compared HM as an anchor and LCEVC-enhanced H.266/VVC.

However, in this scenario H.266/VVC as a base layer is transmitted over one 6MHz/5 Mbps channel, and the LCEVC enhancement layer is transmitted over another, separate 6 MHz/5 Mbps channel. As for the HM anchor the target bitrate is 10 Mbps. Of course, this is a purely “notional” anchor, as in the case of two separate 6MHz/5 Mbps channels it would not be possible to transmit a 10 Mbps HM-encoded sequence.

The actual bitrate achieved for HM is typically within 5%–10% of the target bitrate, mainly due to the HM quantization. For the LCEVC-enhanced sequences, the actual bitrate is typically lower than the 10 Mbps, and on average 15%–18% less than the target bitrate. The reason for this is that the LCEVC-enhanced sequences, the base layer component is encoded with VVenC targeting 5 Mbps, whilst the enhancement layer component is encoded with LCEVC using significantly less than the target 5 Mbps due to the ability of LCEVC to efficiently compress the enhancement component without needing too much bitrate, as discussed and demonstrated in Section 4.1.6.

As done with Test 1, in Tables 9, 10 we are reporting the results of formal subjective assessments performed on these sequences and conducted at the same time and with the same settings as reported in Section 4.1.5 for Test 1.

As it may be seen from Figure 12, LCEVC-enhancing H.266/VVC provides a significant improvement in visual quality over the “notional” HM anchor—between 0.5 and almost 0.9 MOS points of improvement on average—whilst also requiring much less than the 10 Mbps target bitrate and, crucially, being able to leverage two separate 6MHz/5 Mbps channels.

In this section, we report some exemplary screenshots, mainly from the Test 2.2 as reported in Section 4.2.2, to provide some evidence of the improvements associated with the LCEVC-enhanced sequences when compared to their HM counterpart.

In general, it is possible to notice that LCEVC provides a generally improved sharpness and accuracy in reproducing details as demonstrated in Figure 13.