Opened 12 years ago

Closed 12 years ago

#544 closed defect (fixed)

Non-normative aspects in scaling process (8.6.3) should be removed

Reported by: fbossen Owned by: bbross
Priority: minor Milestone: D7
Component: Text Version: D7 (I1003) d1
Keywords: Cc: bbross, wjhan, jct-vc@…


Equations (8-262), (8-264), and (8-270) should be removed as they relate to a specific implementation to avoid 32-bit overflow in some corner cases. Removing these equations has no normative impact.
levelLimit defined in equations (8-262) and (8-264) takes value 15 most of the time, and value 14 for a combination of log2TrSize=2 and large qP values such that BitDepth - qP/6 = 0.
The case where levelLimit is 15 is not relevant anymore following adoption of I0254.
In the case where levelLimit is 14, the clipping to the -16384..16383 range defined in (8-270) has no effect on the value computed in (8-271) since (8-271) scales the output of (8-270) by a factor of at least 40 and defines another clipping operation to the -32768..32767 range. The clipping operation of (8-270) should thus be removed from the specification as it has no normative effect.

For simplicity, it may also be preferable to define the scaling process using a single equation (without changing the normative behavior):
d_ij = Clip3( −32768, 32767,( ( c_ij * M[i][j] * levelScale[ qP%6 ] << ( qP/6 ) ) + ( 1 << ( shift + 3 ) ) ) >> (shift + 4) )
where M[i][j] = 16 if scaling_list_present_flag is equal to 0

It may be appropriate to add a note to clarify than for c_ij = -32768, M[i][j] = 255, qP = 51, and shift = 1, the correct value of d_ij is -32768 (and not 32767, which is the value one might get if not correctly implementing corner cases).

Change History (2)

comment:1 Changed 12 years ago by DefaultCC Plugin

  • Cc bbross wjhan jct-vc@… added

comment:2 Changed 12 years ago by bbross

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in D7 (I1003) d2

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • Benjamin Bross(Owner, Subscriber, Participant)
  • Frank Bossen(Reporter)
  • jct-vc@…(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Always)
  • Woo-Jin Han(Subscriber)