Opened 10 years ago

Closed 10 years ago

# The difference of motion vector out of valid range of "short integer" type

Reported by: Owned by: jeunder minor HM-10.0 HM HM-9.2 fbossen, ksuehring, davidf, jct-vc@…

### Description

In the following code to calculate boundary strength for deblocking process (in TComLoopFilter.cpp, xGetBoundaryStrengthSingle function)

pcMvP0 -= pcMvQ0; pcMvP1 -= pcMvQ1;
uiBs = (pcMvP0.getAbsHor() >= 4) | (pcMvP0.getAbsVer() >= 4) |
(pcMvP1.getAbsHor() >= 4) | (pcMvP1.getAbsVer() >= 4);

The horizontal and vertical component of motion vector was stored as "short integer" type in class TComMv, this will cause problem in the extreme case if the absolute difference between the horizontal or vertical component of motion vectors is greater than 32767 (the maximum value of "short integer" type)

### comment:1 Changed 10 years ago by DefaultCC Plugin

• Cc fbossen ksuehring davidf jct-vc@… added

### comment:2 Changed 10 years ago by jeunder

In the case of MVP+MVD, in 8.5.3.1 Derivation process for motion vector components and reference indices, the following rule applied

uLX[ 0 ] = (mvpLX[ 0 ] + mvdLX[ 0 ] + 216) % 216
mvLX[ 0 ] = ( uLX[ 0 ] >= 215 ) ? ( uLX[ 0 ] − 216 ) : uLX[ 0 ]
uLX[ 1 ] = (mvpLX[ 1 ] + mvdLX[ 1 ] + 216) % 216
mvLX[ 1 ] = ( uLX[ 1 ] >= 215 ) ? ( uLX[ 1 ] − 216 ) : uLX[ 1 ]

This is equivalent to

mvLX[ 0 ] = (Short)(mvpLX[ 0 ] + mvdLX[ 0 ]);
mvLX[ 1 ] = (Short)(mvpLX[ 1 ] + mvdLX[ 1 ]);

It may add similar rule to subtraction of motion vectors.

Version 1, edited 10 years ago by jeunder (previous) (next) (diff)

### comment:3 Changed 10 years ago by fbossen

• Milestone set to HM-10.0
• Resolution set to fixed
• Status changed from new to closed

Fixed in r3305

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

• David Flynn(Subscriber)
• Frank Bossen(Subscriber, Participant)
• jct-vc@…(Subscriber)
• karl.sharman@…(Always)
• Karsten Suehring(Subscriber, Always)