Opened 13 years ago

Closed 12 years ago

#640 closed defect (fixed)

Fix of deltaU calculation when ADAPTIVE_QP_SELECTION is 0

Reported by: wangj Owned by:
Priority: minor Milestone: HM-9.1
Component: HM Version: HM-7.1
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

When the macro ADAPTIVE_QP_SELECTION is set to 0 (default is 1), the variable 'deltaU' in function TComTrQuant::xQuant() is given by

deltaU[uiBlockPos] = (Int)( ((Int64)abs(iLevel) * piQuantCoeff[uiBlockPos] - (iLevel<<iQBits) )>> qBits8 );

The calulation is incorrect because piQuantCoeff should be multiplied with piCoef, not iLevel. The correct derivation is as follows

deltaU[uiBlockPos] = (Int)( ((Int64)abs(piCoef[uiBlockPos]) * piQuantCoeff[uiBlockPos] - (iLevel<<iQBits) )>> qBits8 );

A patch based on HM-7.1-2574 is provided to fix the problem.

Attachments (1)

fix640.patch​ (652 bytes) - added by wangj 13 years ago.

Download all attachments as: .zip

Change History (4)

comment:1 Changed 13 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

Changed 13 years ago by wangj

comment:2 Changed 12 years ago by davidf

Resolved in r3097

comment:3 Changed 12 years ago by davidf

  • Milestone set to HM-9.1
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber, Participant)
  • Frank Bossen(Subscriber)
  • jct-vc@…(Subscriber)
  • Jing Wang(Reporter)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Always)