id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
640	Fix of deltaU calculation when ADAPTIVE_QP_SELECTION is 0	wangj		"When the macro ADAPTIVE_QP_SELECTION is set to 0 (default is 1), the variable 'deltaU' in function TComTrQuant::xQuant() is given by

deltaU[uiBlockPos] = (Int)( ((Int64)abs(iLevel) * piQuantCoeff[uiBlockPos] - (iLevel<<iQBits) )>> qBits8 );

The calulation is incorrect because piQuantCoeff should be multiplied with piCoef, not iLevel. The correct derivation is as follows

deltaU[uiBlockPos] = (Int)( ((Int64)abs(piCoef[uiBlockPos]) * piQuantCoeff[uiBlockPos] - (iLevel<<iQBits) )>> qBits8 );

A patch based on HM-7.1-2574 is provided to fix the problem."	defect	closed	minor	HM-9.1	HM	HM-7.1	fixed		fbossen ksuehring davidf jct-vc@…