id,summary,reporter,owner,description,type,status,priority,milestone,component,version,resolution,keywords,cc
1085,Insufficient bits for the result of luma quarter pel interpolation in HM code.,ManojJames,,"
The data type of the interpolated pixels in Short *dst is insufficient for a implementation of the filter conforming to JCTVC-L1003_v34
'short' can be 16 bits depending upon the platform and compiler used.
template
Void TComInterpolationFilter::filter(Int bitDepth, Short const *src, Int srcStride, Short *dst, Int dstStride, Int width, Int height, Short const *coeff)
Examples of case when 16 bits is insufficient is the quarter-sample location 'j' (8-209).
The maximum and minimum interpolated values of quarter-sample location 'j' depends on the range of 'b' (8-200).
Assume eight bit input. In equation (8-200)
The maximum of 'b' occurs when A(−3,0) = 0, A(−2,0) = 255, A(−1,0) = 0, A(0,0) = 255, A(1,0) = 255, A(2,0) =0, A(3,0) = 255, A(4,0) = 0. The maximum of 'b' is 88*255.
The minimum of 'b' occurs when A(−3,0) = 255, A(−2,0) = 0, A(−1,0) = 255, A(0,0) = 0, A(1,0) = 0, A(2,0) =255, A(3,0) = 0, A(4,0) = 255. The minimum of 'b' is -24*255.
For quarter-sample location 'j', in equation (8-209)
The maximum of 'j' occurs when b(0,-3) = -24*255, b(0,−2) = 88*255, b(0,−1) = -24*255, b(0,0) = 88*255, b(0,1) = 88*255, b(0,2) = -24*255, b(0,3) = 88*255, b(0,4) = -24*255. The maximum of 'j' is 33150
The minimum of 'j' occurs when b(0,-3) = 88*255, b(0,−2) = -24*255, b(0,−1) = 88*255, b(0,0) = -24*255, b(0,1) = -24*255, b(0,2) = 88*255, b(0,3) = -24*255, b(0,4) = 88*255. The minimum value is -16830.
The maximum of 'j' is greater than 32767(the maximum that can be represented by 16 bits). The current implementation of the filter does not confrom to the JCTVC-L1003_v34 draft of the standard when bi-prediction is used.
",defect,closed,minor,HM-10.1,HM,HM-10.0,invalid,,fbossen ksuehring davidf jct-vc@…