Opened 9 years ago

Closed 9 years ago

# Filtering process of neighbouring samples, Equation 8-30 and 8-32, operation precedence issue

Reported by: Owned by: conrad.ho bbross minor D9 Text D9 (K1003) v13 bbross, wjhan, jct-vc@…

### Description

Current spec for equation 8-30:
pF[ −1 ][ y ] = p[ −1 ][ −1 ] + ( ( y + 1 )*( p[ −1 ][ 63 ] − p[ −1 ][ −1 ] ) + 32 ) >> 6

Under the spec table 5-1 precedence, this reads as:
pF[ −1 ][ y ] = (p[ −1 ][ −1 ] + ( ( y + 1 )*( p[ −1 ][ 63 ] − p[ −1 ][ −1 ] ) + 32 )) >> 6

However, it should be:
pF[ −1 ][ y ] = p[ −1 ][ −1 ] + ( ( ( y + 1 )*( p[ −1 ][ 63 ] − p[ −1 ][ −1 ] ) + 32 ) >> 6 )

The equation 8-32 is with the same issue.

### comment:1 Changed 9 years ago by DefaultCC Plugin

• Cc bbross wjhan jct-vc@… added

### comment:2 Changed 9 years ago by conrad.ho

Alternative modifications (which are equivalent to the proposed corrections, and are actually used in HM):

pF[−1][y] = ((y+1)*p[−1][ 63] + (63-y)*p[−1][−1] + 32) >> 6 for y = 0..62 (8-30)

pF[x][-1] = ((63-x)*p[-1][-1] + (x+1)*p[ 63][-1] + 32) >> 6 for x = 0..62 (8-32)

As a note, HM codes are:

for (i = 1; i < uiCuHeight2; i++)
{

piFilterBufN[i] = ((uiCuHeight2-i)*bottomLeft + i*topLeft + uiCuHeight) >> shift;

}
for (i = 1; i < uiCuWidth2; i++)
{

piFilterBufN[uiCuHeight2 + i] = ((uiCuWidth2-i)*topLeft + i*topRight + uiCuWidth) >> shift;

}

### comment:3 Changed 9 years ago by bbross

• Resolution set to fixed
• Status changed from new to closed

Fixed in JCTVC-L1003_v1.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

• Benjamin Bross(Owner, Subscriber, Participant)