id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
946	32-bit Dequantization Overflow when QP=47	bheng		"
The non-normative clipping described in H0541, and implemented in HM, attempts to avoid 32-bit overflow of intermediate results by clipping coefficients to a 15-bit range prior to dequantization for 4x4 blocks when scaling lists are used and QP>=48.

However, the intermediate results can still overflow when QP=47.  

For example, using the largest weight and coefficient level:

(L * IQ * W) << (QP/6 - iShift) = (-32768 * 72 * 255) << 2 = -2406481920   (33 bit signed value)

In HM, the resulting dequantized coefficients overflow the 32 bit integers used to store these results.  I believe the attached patch would fix the problem.


"	defect	closed	minor	HM-9.2	HM	HM-9.1	fixed		fbossen ksuehring davidf jct-vc@…