Opened 10 years ago Closed 10 years ago #1320 closed defect (fixed)Range of values in residual modification process
Description
I wonder if there should be some limit on the values generated by the residual modification process?
At the moment there does not appear to be any constraint, so if cu_transquant_bypass_flag is equal to 1, equation 8-267 allows r[x][y] to be as large as CoeffMax (215 -1 for Main 4:4:4 10).
The residual modification process 8-6-5 then computes a cumulative sum of up to nTbS of these values using equation 8-292.
The final value can therefore be nTbS*CoeffMax.
The reference code assumes that the values are at most 16bits unless compiled with a target of all_highbitdepth.
This means that we can make a legal Main 4:4:4 10 stream that decodes differently depending on whether the reference code is compiled in highbitdepth mode or not.
It seems to makes little sense for the residual values to be allowed to be this large as they will just be clipped during the reconstruction process. Attachments (1)Change History (7)comment:1 Changed 10 years ago by DefaultCC Plugin
comment:2 Changed 10 years ago by karlsharmancomment:3 Changed 10 years ago by peterderivaz
Thanks for the links to the meeting notes.
I am not sure I quite understand how an implementation can safely include clipping in the accumulator.
Suppose the inputs are 30000,30000,-30000
If we clip the accumulator at 32767 then the final answer will be 2767, while if we compute with full precision won't the final answer be 30000? comment:4 Changed 10 years ago by karlsharman
The accumulator part cannot be clipped. But the final block of residuals can be. comment:5 Changed 10 years ago by karlsharman
comment:6 Changed 10 years ago by karlsharman
Fixed in r4228. No effect on CTCs (except perhaps run-time). Note: See
TracTickets for help on using
tickets. | This list contains all users that will be notified about changes made to this ticket. These roles will be notified: Reporter, Owner, Subscriber, Participant
|
No additional clipping has been explicitly added to the RDPCM path, as discussed during presentation of JCTVC-O0066, with meeting notes found in the BoG report JCTVC-O0352.
Once the residual is combined with the prediction it is clipped to the valid video output range; this permits implementations to include clipping to the residual path to limit intermediate values.
This trac-report is perhaps identifying a missing clip in the HM decoder implementation (with the current calculation overflowing in extreme cases) ?