Opened 12 years ago Closed 12 years ago #1085 closed defect (invalid)Insufficient bits for the result of luma quarter pel interpolation in HM code.
Description
The data type of the interpolated pixels in Short *dst is insufficient for a implementation of the filter conforming to JCTVC-L1003_v34
template<Int N, Bool isVertical, Bool isFirst, Bool isLast>
Examples of case when 16 bits is insufficient is the quarter-sample location 'j' (8-209).
Assume eight bit input. In equation (8-200)
For quarter-sample location 'j', in equation (8-209)
The maximum of 'j' is greater than 32767(the maximum that can be represented by 16 bits). The current implementation of the filter does not confrom to the JCTVC-L1003_v34 draft of the standard when bi-prediction is used. Change History (2)comment:1 Changed 12 years ago by DefaultCC Plugin
comment:2 Changed 12 years ago by fbossen
Note: See
TracTickets for help on using
tickets. | This list contains all users that will be notified about changes made to this ticket. These roles will be notified: Reporter, Owner, Subscriber, Participant
|
Values are stored with an offset of -8192 such as to shift the -16380..33150 range to -24572..24958 which nicely fits into the signed 16-bit range.