Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#1298 closed defect (fixed)

Output value clipping missing from HM RDOQ

Reported by: karlsharman Owned by:
Priority: minor Milestone: HM-16.1
Component: HM Version: HM-14.0
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

Clipping is missing in xRateDistOptQuant in HM & RExt.

This means that under specific situations, the encoder can produce coefficient values outside HM's legal range of -32768 to 32767.

Here's an example command line for HM, where the first value encountered outside the legal range has the value 37068.

./bin/TAppEncoderStatic -c cfg/encoder_intra_main10.cfg -c cfg/per-sequence/SlideShow.cfg --MaxPartitionDepth=1 --QuadtreeTUMaxDepthIntra=1 --QuadtreeTUMaxDepthInter=1 --RDOQ=1 --TransformSkip=1 --QP=-12 -b str.bin --ConformanceMode=1 -f 1 -fs 3

The problem occurs primarily at negative QPs when large blocks are forced, but if a scaling matrix is used, the problem can occur for more cases.

The problem is particularly noticeable in RExt when extended precision processing is not enabled whilst the QP is set to a low value.

If RDOQ is disabled, the problem goes away (the values are clipped in xQuant).

Possible solution:

Add a clip in xRateDistOptQuant in RExt/HM, at the point uiMaxAbsLevel is first calculated, clipping to 32767 (or similar in RExt, based upon extended precision processing)

(this will mean that -32768 will never be encountered, but it's quite a limited/rare case)

Change History (3)

comment:1 Changed 7 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

comment:2 Changed 7 years ago by karlsharman

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in merge with RExt.

comment:3 Changed 7 years ago by ksuehring

  • Milestone changed from HM-15.0 to HM-16.1
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Reporter, Participant)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Participant, Always)