Opened 12 years ago Closed 12 years ago #946 closed defect (fixed)32-bit Dequantization Overflow when QP=47
Description
The non-normative clipping described in H0541, and implemented in HM, attempts to avoid 32-bit overflow of intermediate results by clipping coefficients to a 15-bit range prior to dequantization for 4x4 blocks when scaling lists are used and QP>=48.
However, the intermediate results can still overflow when QP=47.
For example, using the largest weight and coefficient level:
(L * IQ * W) << (QP/6 - iShift) = (-32768 * 72 * 255) << 2 = -2406481920 (33 bit signed value)
In HM, the resulting dequantized coefficients overflow the 32 bit integers used to store these results. I believe the attached patch would fix the problem. Attachments (1)Change History (3)comment:1 Changed 12 years ago by DefaultCC Plugin
Changed 12 years ago by bhengcomment:2 Changed 12 years ago by fbossen
Note: See
TracTickets for help on using
tickets. | This list contains all users that will be notified about changes made to this ticket. These roles will be notified: Reporter, Owner, Subscriber, Participant
|
Fixed in r3205
Took a different approach with one more clipping operation that doesn't require calculation of the bitRange value