Opened 9 years ago

Closed 9 years ago

# The difference of motion vector out of valid range of "short integer" type

Reported by: Owned by: jeunder minor HM-10.0 HM HM-9.2 fbossen, ksuehring, davidf, jct-vc@…

### Description

In the following code to calculate boundary strength for deblocking process (in TComLoopFilter.cpp, xGetBoundaryStrengthSingle function)

pcMvP0 -= pcMvQ0; pcMvP1 -= pcMvQ1;
uiBs = (pcMvP0.getAbsHor() >= 4) | (pcMvP0.getAbsVer() >= 4) |
(pcMvP1.getAbsHor() >= 4) | (pcMvP1.getAbsVer() >= 4);

The horizontal and vertical component of motion vector was stored as "short integer" type in class TComMv, this will cause problem in the extreme case if the absolute difference between the horizontal or vertical component of motion vectors is greater than 32767 (the maximum value of "short integer" type)

### comment:1 Changed 9 years ago by DefaultCC Plugin

• Cc fbossen ksuehring davidf jct-vc@… added

### comment:2 Changed 9 years ago by jeunder

In the case of MVP+MVD, in 8.5.3.1 Derivation process for motion vector components and reference indices, the following rule applied

uLX[ 0 ] = (mvpLX[ 0 ] + mvdLX[ 0 ] + 216) % 216
mvLX[ 0 ] = ( uLX[ 0 ] >= 215 ) ? ( uLX[ 0 ] − 216 ) : uLX[ 0 ]
uLX[ 1 ] = (mvpLX[ 1 ] + mvdLX[ 1 ] + 216) % 216
mvLX[ 1 ] = ( uLX[ 1 ] >= 215 ) ? ( uLX[ 1 ] − 216 ) : uLX[ 1 ]

This is equivalent to

mvLX[ 0 ] = (Short)(mvpLX[ 0 ] + mvdLX[ 0 ]);
mvLX[ 1 ] = (Short)(mvpLX[ 1 ] + mvdLX[ 1 ]);

It may need to add similar rule to subtraction of motion vectors.

Last edited 9 years ago by jeunder (previous) (diff)

### comment:3 Changed 9 years ago by fbossen

• Milestone set to HM-10.0
• Resolution set to fixed
• Status changed from new to closed

Fixed in r3305

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

• David Flynn(Subscriber)
• Frank Bossen(Subscriber, Participant)
• jct-vc@…(Subscriber)
• karl.sharman@…(Always)
• Karsten Suehring(Subscriber, Always)