Clipping is missing in xRateDistOptQuant in HM & RExt.

This means that under specific situations, the encoder can produce coefficient values outside HM's legal range of -32768 to 32767.

Here's an example command line for HM, where the first value encountered outside the legal range has the value 37068.

./bin/TAppEncoderStatic -c cfg/encoder_intra_main10.cfg -c cfg/per-sequence/SlideShow.cfg --MaxPartitionDepth=1 --QuadtreeTUMaxDepthIntra=1 --QuadtreeTUMaxDepthInter=1 --RDOQ=1 --TransformSkip=1 --QP=-12 -b str.bin --ConformanceMode=1 -f 1 -fs 3

The problem occurs primarily at negative QPs when large blocks are forced, but if a scaling matrix is used, the problem can occur for more cases.

The problem is particularly noticeable in RExt when extended precision processing is not enabled whilst the QP is set to a low value.

If RDOQ is disabled, the problem goes away (the values are clipped in xQuant).

Possible solution:

Add a clip in xRateDistOptQuant in RExt/HM, at the point uiMaxAbsLevel is first calculated, clipping to 32767 (or similar in RExt, based upon extended precision processing)

(this will mean that -32768 will never be encountered, but it's quite a limited/rare case)