id summary reporter owner description type status priority milestone component version resolution keywords cc 597 encoder-only fix for rdFactor calculation for HE10 settings wangj "The variable ‘rdFactor’ for computing rd-cost in the RDOQ function is currently given by Int rdFactor = (Int)((Double)(g_invQuantScales[m_cQP.rem()]*g_invQuantScales[m_cQP.rem()]<<(2*m_cQP.m_iPer))/m_dLambda/16 + 0.5) ; The calculation is based on 8-bit coding (g_uiBitIncrement = 0). It should be computed as follows, so that it also works for the HE10 setting where g_uiBitIncrement = 2. Int rdFactor = (Int)((Double)(g_invQuantScales[m_cQP.rem()])*(Double)(g_invQuantScales[m_cQP.rem()])*(Double)(1<<(2*m_cQP.m_iPer))/m_dLambda/16/(Double)(1<<(2*g_uiBitIncrement)) + 0.5); A patch is provided to fix the rdFactor calculation. This fix only affects HE10 settings when RDOQ is enabled. With this fix, on average (-0.13%, -0.32%, -0.54%) BD-rate gain were observed for AI-HE10, RA-HE10 and LB-HE10 settings. " defect closed major HM HM-7.0 fixed fbossen ksuehring davidf jct-vc@…