Opened 12 years ago

Closed 12 years ago

#1085 closed defect (invalid)

Insufficient bits for the result of luma quarter pel interpolation in HM code.

Reported by: ManojJames Owned by:
Priority: minor Milestone: HM-10.1
Component: HM Version: HM-10.0
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

The data type of the interpolated pixels in Short *dst is insufficient for a implementation of the filter conforming to JCTVC-L1003_v34
'short' can be 16 bits depending upon the platform and compiler used.

template<Int N, Bool isVertical, Bool isFirst, Bool isLast>
Void TComInterpolationFilter::filter(Int bitDepth, Short const *src, Int srcStride, Short *dst, Int dstStride, Int width, Int height, Short const *coeff)

Examples of case when 16 bits is insufficient is the quarter-sample location 'j' (8-209).
The maximum and minimum interpolated values of quarter-sample location 'j' depends on the range of 'b' (8-200).

Assume eight bit input. In equation (8-200)
The maximum of 'b' occurs when A(−3,0) = 0, A(−2,0) = 255, A(−1,0) = 0, A(0,0) = 255, A(1,0) = 255, A(2,0) =0, A(3,0) = 255, A(4,0) = 0. The maximum of 'b' is 88*255.
The minimum of 'b' occurs when A(−3,0) = 255, A(−2,0) = 0, A(−1,0) = 255, A(0,0) = 0, A(1,0) = 0, A(2,0) =255, A(3,0) = 0, A(4,0) = 255. The minimum of 'b' is -24*255.

For quarter-sample location 'j', in equation (8-209)
The maximum of 'j' occurs when b(0,-3) = -24*255, b(0,−2) = 88*255, b(0,−1) = -24*255, b(0,0) = 88*255, b(0,1) = 88*255, b(0,2) = -24*255, b(0,3) = 88*255, b(0,4) = -24*255. The maximum of 'j' is 33150
The minimum of 'j' occurs when b(0,-3) = 88*255, b(0,−2) = -24*255, b(0,−1) = 88*255, b(0,0) = -24*255, b(0,1) = -24*255, b(0,2) = 88*255, b(0,3) = -24*255, b(0,4) = 88*255. The minimum value is -16830.

The maximum of 'j' is greater than 32767(the maximum that can be represented by 16 bits). The current implementation of the filter does not confrom to the JCTVC-L1003_v34 draft of the standard when bi-prediction is used.

Change History (2)

comment:1 Changed 12 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

comment:2 Changed 12 years ago by fbossen

  • Resolution set to invalid
  • Status changed from new to closed

Values are stored with an offset of -8192 such as to shift the -16380..33150 range to -24572..24958 which nicely fits into the signed 16-bit range.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber, Participant)
  • jct-vc@…(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Always)
  • Manoj James(Reporter)