Opened 8 years ago Closed 8 years ago #1320 closed defect (fixed)Range of values in residual modification process
Description
I wonder if there should be some limit on the values generated by the residual modification process?
At the moment there does not appear to be any constraint, so if cu_transquant_bypass_flag is equal to 1, equation 8267 allows r[x][y] to be as large as CoeffMax (2^{15} 1 for Main 4:4:4 10).
The residual modification process 865 then computes a cumulative sum of up to nTbS of these values using equation 8292.
The final value can therefore be nTbS*CoeffMax.
The reference code assumes that the values are at most 16bits unless compiled with a target of all_highbitdepth.
This means that we can make a legal Main 4:4:4 10 stream that decodes differently depending on whether the reference code is compiled in highbitdepth mode or not.
It seems to makes little sense for the residual values to be allowed to be this large as they will just be clipped during the reconstruction process. Attachments (1)Change History (7)comment:1 Changed 8 years ago by DefaultCC Plugin
comment:2 Changed 8 years ago by karlsharmancomment:3 Changed 8 years ago by peterderivaz
Thanks for the links to the meeting notes.
I am not sure I quite understand how an implementation can safely include clipping in the accumulator.
Suppose the inputs are 30000,30000,30000
If we clip the accumulator at 32767 then the final answer will be 2767, while if we compute with full precision won't the final answer be 30000? comment:4 Changed 8 years ago by karlsharman
The accumulator part cannot be clipped. But the final block of residuals can be. comment:5 Changed 8 years ago by karlsharman
comment:6 Changed 8 years ago by karlsharman
Fixed in r4228. No effect on CTCs (except perhaps runtime). Note: See
TracTickets for help on using
tickets.  This list contains all users that will be notified about changes made to this ticket. These roles will be notified: Reporter, Owner, Subscriber, Participant

No additional clipping has been explicitly added to the RDPCM path, as discussed during presentation of JCTVCO0066, with meeting notes found in the BoG report JCTVCO0352.
Once the residual is combined with the prediction it is clipped to the valid video output range; this permits implementations to include clipping to the residual path to limit intermediate values.
This tracreport is perhaps identifying a missing clip in the HM decoder implementation (with the current calculation overflowing in extreme cases) ?