Opened 12 years ago

Closed 12 years ago

#597 closed defect (fixed)

encoder-only fix for rdFactor calculation for HE10 settings

Reported by: wangj Owned by:
Priority: major Milestone:
Component: HM Version: HM-7.0
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

The variable ‘rdFactor’ for computing rd-cost in the RDOQ function is currently given by

Int rdFactor = (Int)((Double)(g_invQuantScales[m_cQP.rem()]*g_invQuantScales[m_cQP.rem()]<<(2*m_cQP.m_iPer))/m_dLambda/16 + 0.5) ;

The calculation is based on 8-bit coding (g_uiBitIncrement = 0). It should be computed as follows, so that it also works for the HE10 setting where g_uiBitIncrement = 2.

Int rdFactor = (Int)((Double)(g_invQuantScales[m_cQP.rem()])*(Double)(g_invQuantScales[m_cQP.rem()])*(Double)(1<<(2*m_cQP.m_iPer))/m_dLambda/16/(Double)(1<<(2*g_uiBitIncrement)) + 0.5);

A patch is provided to fix the rdFactor calculation. This fix only affects HE10 settings when RDOQ is enabled. With this fix, on average (-0.13%, -0.32%, -0.54%) BD-rate gain were observed for AI-HE10, RA-HE10 and LB-HE10 settings.

Attachments (1)

rdFactorFix.patch​ (675 bytes) - added by wangj 12 years ago.

Download all attachments as: .zip

Change History (4)

comment:1 Changed 12 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

Changed 12 years ago by wangj

comment:2 Changed 12 years ago by pandrivon

Thank you for your fix!
Ticket #596 should also be taken into account for 14-bit as coding gains (non-normative technique) with the overflow fixes proposed in ticket #596 (patch over HM7.1rc1) are with class B SVT contents for AI-HE 14-bit (-5.0%, -7.2%, -8.2%) and for RA-HE 14-bit (-2.4%, -5.1%, -2.3%) in comparison with JCTVC-I0108 tests. Tests on 8, 10, 12, 14-bit now show that results are coherent for 14-bit coding.

comment:3 Changed 12 years ago by fbossen

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in r2481

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber, Participant)
  • jct-vc@…(Subscriber)
  • Jing Wang(Reporter)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Always)
  • Pierre Andrivon(Participant)