Opened 10 years ago

Closed 9 years ago

# Out of bounds index computation in Angular Intra prediction

Reported by: Owned by: pieterkapsenberg bbross minor WD6 Text D6 (H1003) dI/dJ/dK intra angular index bbross, wjhan, jct-vc@…

### Description

In the WD, subclause 8.4.3.1.6 "Specification of Intra_Angular (2..9, 11..25, 27..34) prediction mode" the following equation is shown (8-48):
refMain[ x ] = p[ −1, −1+( ( x*invAngle+128 )>>8 ) ], with x=( nS*intraPredAngle ) >>5..−1

For the case of an intra 4x4 block with an intra mode of 25, intraPredAngle is -2 and invAngle is -4096 (table 8-5 and 8-6). Then, the index range goes from (4*-2)>>1 .. -1, which is -1..-1. This implies that the equation is applied once, with x = -1. In that case we index array p with [-1,-1+((-1*-4096+128)>>8)] = [-1,15]. However, array P is specified to be sized from [-1..2*ns-1,-1..2*ns-1], [-1..7,-1..7] in this case.

HM seems to avoid this by just skipping the loop altogether, since it has the following code:

```      Int invAngleSum    = 128;       // rounding for (shift by 8)
for (k=-1; k>blkSize*intraPredAngle>>5; k--)
{
invAngleSum += invAngle;
refMain[k] = refSide[invAngleSum>>8];
}

```

In this case, the exit condition is met immediately, since k is not larger than -1 (it is equal to -1). If the HM model followed the WD exactly, the loop exit condition would have been a great-or-equal operator like so:

```      for (k=-1; k>=blkSize*intraPredAngle>>5; k--)

```

### comment:1 Changed 10 years ago by DefaultCC Plugin

• Cc bbross wjhan jct-vc@… added

### comment:2 Changed 9 years ago by bbross

Adding the following condition between "– If intraPredAngle is less than 0," and "refMain[ x ] = ..." for both equations would avoid the invalid indices and match draft text with HM software.

• When ( nS*intraPredAngle ) >>5 is less than −1,

### comment:3 Changed 9 years ago by bbross

• Resolution set to fixed
• Status changed from new to closed

Fixed in JCTVC-I1003_d9.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

• Benjamin Bross(Owner, Subscriber, Participant)
• jct-vc@…(Subscriber)
• karl.sharman@…(Always)
• Karsten Suehring(Always)
• Pieter Kapsenberg(Reporter)
• Woo-Jin Han(Subscriber)