Opened 10 years ago

Closed 10 years ago

#1217 closed defect (fixed)

HM lossless coding search tidy

Reported by: karlsharman Owned by:
Priority: minor Milestone: HM-13.0
Component: HM Version: HM-12.1
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…


HM has three command line parameters that control lossless coding, namely:


There is some confusion over how these are used, probably because LosslessCuEnabled appears to have once been an SPS setting, equivalent to "qpprime_y_zero_transquant_bypass_flag". (There is even a place in TEncSearch where the value of QP' is checked to see if it is 0)

The flag was removed from the SPS, replaced with the transquant-bypass-enable flag in the PPS (and a per CU flag).

In HM, LosslessCuEnabled causes the search algorithm to consider checking transquant-bypass coding. However, unless TransquantBypassEnableFlag is also set, no per-CU transquant flag is sent. In addition the value of the transquant bypass CU flag is fixed at the value of CUTransquantBypassFlagValue, and is therefore if this is 0, you do not have transquant bypass coding in the search.

In RExt, there are just two settings, namely


where TransquantBypassEnableFlag enables the tool in the PPS and in the search, including use when maxDeltaQP is non zero.
If CUTransquantBypassFlagForce is 1, the transquant-bypass flag is forced to 1 for all CUs, otherwise the best mode is selected by encoder search.
When testing transquant-bypass modes, the value of QP used is set to the lowest QP value being tested, not QP'=0, as this reduces delta QP values.

The supplied match can be applied to HM12.1 as tagged to make HM use the same lossless coding search algorithm as HM12.1_RExt5.1 (with RExtBACKWARDS_COMPATIBILITY_HM_TRANSQUANTBYPASS set to 0).

Attachments (1)

HM12.1_transquant_bypass.patch (31.3 KB) - added by karlsharman 10 years ago.

Download all attachments as: .zip

Change History (3)

comment:1 Changed 10 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

Changed 10 years ago by karlsharman

comment:2 Changed 10 years ago by fbossen

  • Resolution set to fixed
  • Status changed from new to closed

Patch applied in r3768

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber, Participant)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Reporter)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Always)