In the RDOQ function TComTrQuant::xRateDistOptQuant the cost of coding the coded block flag is based on m_pcEstBitsSbac->blockCbpBits which is always 1 bit when RQT is enabled [QuadtreeTUFlag:1 in default config] (it is based on a context initialized to a 50/50 probability and which is not used when RQT is enabled and hence never updated). The cost of the coded block flag is thus not taken into account when deciding whether to encode any coefficient at all in a block (which is not necessarily a bad thing by itself).

When RQT is disabled the cost of the coded block flag is taken into account as m_pcEstBitsSbac->blockCbpBits is updated according to past history. This has an impact on the comparison of RQT on vs off. In some cases (QP=37) the chroma PSNR was seen to drop by about 1dB when RQT is disabled.

The simplest solution may simply be to always ignore the cost of the coded block flag in the RDOQ function. This would not impact the default setting defined in JCT-VC B300.

At the end of function TEncSearch::xMotionEstimation, we have the following line

ruiCost = (UInt)( floor( fWeight * ( (Double)ruiCost - (Double)m_pcRdCost->getCost( uiMvBits ) ) ) + (Double)m_pcRdCost->getCost( ruiBits ) );

For P and B frames, fWeight=1 and 0.5, respectively. When we calculate the distortion (SAD/HAD) for B frames in motion estimation, we already used the average prediction of two hypotheses, such as in TComRdCost::xGetSADs32(). Then fWeight should also be set to 1 for B frames. To keep the original design (fWeight=0.5 for B frames) unchanged, during the calculation of sub-pixel distortion such as TComRdCost::xCalcHADs8x8, a one-bit left shift is applied after the difference calculation, i.e.,

pred = xClip( (piCur[i*iStep] + piRef[i] + bRound) >> 1 ); diff[k+i] = (piOrg[i] - pred) << 1;

which compensates the setting of fWeight=0.5. Such an implementation (fWeight=0.5 + one-bit left shift) is correct. However, the code is confused and difficult to understand. We hope it can be improved to be more clear. For example, set fWeight=1 for B frames while removing the left shift in functions like TComRdCost::xCalcHADs8x8.

The two macros DCM_PBIC and HHI_MRG_PU were added in parallel and need to be harmonized with each other