Opened 11 years ago

Closed 10 years ago

#1177 closed defect (fixed)

Incorrect equation when using transform_skip_rotation_enabled_flag

Reported by: peterderivaz Owned by:
Priority: major Milestone: RExt D5
Component: RExt text Version: RExt D4 (N1005) v3
Keywords: Cc: davidf, joel, teruhiko, jct-vc@…

Description

In 8.6.2 Scaling and transformation process, equation 8-271 is used when transform_skip_rotation_enabled_flag is equal to 1 and nTbS is equal to 4:

r[ x ][ y ] = ( TransCoeffLevel[ nTbS − x −1 ][ nTbS − y − 1 ] << tsShift )

This corresponds to the code at line 1861 of TComTrQuant.cpp:

rpcResidual[(k * uiStride) + j] = Pel(pcCoeff[coefficientIndex]);

but in the code, no shift is applied.

I believe equation 8-271 should change to:

r[ x ][ y ] = TransCoeffLevel[ nTbS − x −1 ][ nTbS − y − 1 ]

Change History (2)

comment:1 Changed 11 years ago by DefaultCC Plugin

  • Cc davidf joel teruhiko jct-vc@… added

comment:2 Changed 10 years ago by davidf

  • Milestone set to RExt D5
  • Resolution set to fixed
  • Status changed from new to closed

Fixed in O1005v1

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber, Participant)
  • jct-vc@…(Subscriber)
  • Joel(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Always)
  • Peter de Rivaz(Reporter)
  • Teruhiko Suzuki(Subscriber)