In the RDOQ function TComTrQuant::xRateDistOptQuant the cost of coding the coded block flag is based on m_pcEstBitsSbac->blockCbpBits which is always 1 bit when RQT is enabled [QuadtreeTUFlag:1 in default config] (it is based on a context initialized to a 50/50 probability and which is not used when RQT is enabled and hence never updated). The cost of the coded block flag is thus not taken into account when deciding whether to encode any coefficient at all in a block (which is not necessarily a bad thing by itself).

When RQT is disabled the cost of the coded block flag is taken into account as m_pcEstBitsSbac->blockCbpBits is updated according to past history. This has an impact on the comparison of RQT on vs off. In some cases (QP=37) the chroma PSNR was seen to drop by about 1dB when RQT is disabled.

The simplest solution may simply be to always ignore the cost of the coded block flag in the RDOQ function. This would not impact the default setting defined in JCT-VC B300.