Opened 10 years ago

Closed 4 years ago

#1321 closed defect (fixed)

Overflow in cross component prediction

Reported by: peterderivaz Owned by: rajan_joshi
Priority: minor Milestone:
Component: SCC text Version:
Keywords: Cc: davidf, karlsharman, jct-vc@…


In TComTrQuant.cpp:crossComponentPrediction the residuals are combined with the lines:

pResiT[x] = pResiC[x] + (( alpha * rightShift( pResiL[x], diffBitDepth) ) >> 3);


pResiT[x] = pResiC[x] - (( alpha * rightShift(pResiL[x], diffBitDepth) ) >> 3);

However, for high values of the residual this calculation can overflow the size of a Pel and change from a large positive to a large negative.

This occurs even for Main 4:4:4 10 profiles (but only in artificial test streams).

A temporary work around is to compile in highbitdepth mode as this increases the size of Pel.

Change History (11)

comment:1 Changed 10 years ago by DefaultCC Plugin

  • Cc davidf karlsharman jct-vc@… added

comment:2 Changed 10 years ago by karlsharman

Thank you Peter.

This has highlighted an issue with the possible range of the residual due to cross-component-prediction, and a corresponding thread will be created on the main JCTVC email reflector.

With cross-component-prediction, the output of the inverse-transforms (inverse-DCT/DST, inverse-RDPCM, inverse-transform-skip, inverse-transquant-bypass etc) are combined: a fraction of the luma is added to the chroma.

The specification does not have a clipping stage at the output of the inverse transforms; however, in the software, a clipping stage has been applied at the end of the inverse-transform to protect the resulting data from overflowing the size of the 16-bit Pel residual array. This was previously valid for Ver.1 since the only processing-stages following were the addition of the residual to the prediction and then clipping to the internal bit depth (IBD).

With CCP present, the clipping has to be removed - it cannot be clipped at the end of the transform, because there is a possibility that a large luma value is combined with a large negative chroma value and turns into a small value.

It is likely that this only occurs for pathological / "designed-to-be-evil" bit-streams.

Here are some calculations (please check) showing the output bit depths of the various tools using non-extended-precision and various internal bit depths (IBD), and highlighting cases that require residuals to be more than 16-bit.

Tool IBD=IBD IBD=8 IBD=10 IBD=12
Inv quantiser* 16 16 16 16
Inv transform stage 1* 16 16 16 16
Inv transform stage 2 IBD+2+5 15 17 19
Inv transform skip IBD+1 9 11 13
Inv transquant-bypass 16 16 16 16
Inv RDPCM+transform skip IBD+1+5 14 16 18
Inv RDPCM+transquant-bypass 16+5 21 21 21

(* Not an input to cross-component-prediction.)

For now, the work-around to use the HIGH_BIT_DEPTH_SUPPORT=1 build (which uses 32-bit Pels) is valid, although this will double the run-time memory footprint.

comment:3 Changed 10 years ago by ksuehring

  • Milestone HM+RExt-8.0 deleted

comment:4 Changed 10 years ago by karlsharman

  • Component changed from HM RExt to HM
  • Version changed from RExt D7 (Q1005) v8 to HM-16.1

Modified active ticket for RExt component to HM component.

comment:5 Changed 9 years ago by karlsharman

At the 19th JCTVC meeting in Strasbourg, the JCTVC concluded that a
restriction should be placed on the bit-stream, as such overflow
conditions are not expected to be generated by a real encoder. This
restriction has been marked for inclusion into the HEVC Screen
Content Coding Draft Text.

The restriction would likely be placed in Section 8.6.6: Residual
modification process for transform blocks using cross-component
prediction, and would be something similar to:

The bitstream shall not cause the luma residual
values rY to be outside of the inclusive range
defined by (CoeffMinY to CoeffMaxY), nor the residual
samples r to be outside of the inclusive range
defined by (CoeffMinC to CoeffMaxC).

The restriction may be further refined during the development of
Version 3.

In HM (& SCM) the software function
TComTrQuant::crossComponentPrediction still needs to be slightly
modified: the pResiL[x] should be cast to an Intermediate_Int
and a clipping stage should be added to the software to prevent
the modified residual exceeding the range of a Pel.

For reference, the calculation is:

Let YCBitDiff be the difference between the luma and chroma bit-depths

For each sample in the block:

Let chromaResi be the (restricted) chroma residual value.
Let lumaResi be the (restricted) luma residual value.
Let shiftedLuma be lumaResi shifted left/right by YCBitDiff.
result = chromaResi + ((alpha * shiftedLuma)>>3)

The worst case of the CCP residual calculation
for extended-precision disabled is when luma's
bit depth is 8, and chroma's bit depth is 16. In
this case lumaResi (eg -32768) will be left shifted
8 places to be 24-bit (shiftedLuma=-8388608).
The result then gets multiplied by alpha, and
is potentially 28-bit (-8*-8388608=67108864).

The worst case of the CCP residual calculation
for extended-precision enabled is 23+8+4=35 bit.

Hence 32-bit (or 64-bit for extended-precision)
must be used for the calculation, and a clipping
stage is required to protect the Pel type.

comment:6 Changed 9 years ago by karlsharman

As indicated previously, the constraint might be further refined by
the JCT-VC.

For example, does the case of 16-bit with extended precision disabled
actually work?

Constraining the inputs to CCP to the range of -32768 to 32767 does
not seem like a good solution in this case (given the valid range
of lossless residuals at the output of CCP would be -65535 to 65535).

Perhaps restricting the values at the input to CCP to bitDepthY+4
and bitDepthC+4 signed-bits might be a more appropriate solution?

This would also prevent the size of shiftedLuma in the above
calculation becoming too large.

  • Note that I had an error in my previous update to this bug-report: I believe the worst case of the CCP residual calculation for extended-precision enabled is 16+8+4=28 bit, not 35 bit.

comment:7 Changed 9 years ago by karlsharman

  • Component changed from HM to SCC text
  • Version HM-16.1 deleted

Software fix applied in r4227, assuming that inputs for 12-bit and below will not exceed 16-bit signed and beyond 12-bit will not exceed 23-bit signed. Modifying ticket to become a SCC text ticket as a reminder to editors.

comment:8 Changed 9 years ago by ksuehring

  • Owner set to rajan_joshi
  • Status changed from new to assigned

comment:9 Changed 9 years ago by karlsharman

The meeting notes (JCTVC-T1000) indicated the following during the review of JCTVC-T0003:

This issue was discussed at the previous JCT-VC meeting in Strasbourg and it was agreed that a constraint could be added to the next version of the HEVC specification. (However, it was not yet incorporated into a draft text output document.)


Decision: As previously planned, for RExt purposes, specify a normative constraint that limits inputs to the inverse CCP to the range of the quantized coefficients CoeffMinY/C to CoeffMaxY/C. (Include in SCC draft output.)

comment:10 Changed 4 years ago by karlsharman

Constraint added to HEVC v4.

comment:11 Changed 4 years ago by crosewarne

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Subscriber, Participant)
  • karl.sharman@…(Always)
  • Karsten Suehring(Participant, Always)
  • Peter de Rivaz(Reporter)
  • Rajan Joshi(Owner)