In TEncPreanalyzer::xPreanalyze(...) (TEncPreanalyzer.cpp)

The sum of luma sample values and squared values is accumulated for each quadrant of a 2Nx2N cu - so each accumulates over NxN values, but the average is then calculated by dividing through by "uiNumPixInAQPart" which counts values over the whole 2Nx2N.

i.e. - for a 16x16 Cu, the 4 sums are accumulated over 64 values each, but the average value is calculated by dividing by 256. This gives an incorrect value for the variance of the blocks.

Possible fix -

Delete ", uiNumPixInAQPart++" in each of the 4 inner loops and insert at line 121

uiNumPixInAQPart = uiCurrAQPartWidth*uiCurrAQPartHeight/4;