Opened 11 years ago

Closed 11 years ago

#946 closed defect (fixed)

32-bit Dequantization Overflow when QP=47

Reported by: bheng Owned by:
Priority: minor Milestone: HM-9.2
Component: HM Version: HM-9.1
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…


The non-normative clipping described in H0541, and implemented in HM, attempts to avoid 32-bit overflow of intermediate results by clipping coefficients to a 15-bit range prior to dequantization for 4x4 blocks when scaling lists are used and QP>=48.

However, the intermediate results can still overflow when QP=47.

For example, using the largest weight and coefficient level:

(L * IQ * W) << (QP/6 - iShift) = (-32768 * 72 * 255) << 2 = -2406481920 (33 bit signed value)

In HM, the resulting dequantized coefficients overflow the 32 bit integers used to store these results. I believe the attached patch would fix the problem.

Attachments (1)

dequant_overflow.patch (613 bytes) - added by bheng 11 years ago.

Download all attachments as: .zip

Change History (3)

comment:1 Changed 11 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

Changed 11 years ago by bheng

comment:2 Changed 11 years ago by fbossen

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in r3205
Took a different approach with one more clipping operation that doesn't require calculation of the bitRange value

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • Brian Heng(Reporter)
  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber, Participant)
  • jct-vc@…(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Always)