Opened 11 years ago

Closed 11 years ago

Encoder generates a non-conforming bitstream by encoding a coefficient level of +32768 when sign hiding is enabled

Reported by: Owned by: livium minor HM-10.1 HM HM-10.0 fbossen, ksuehring, davidf, jct-vc@…

Description

I encountered a case where the encoder generated a bitstream containing a +32768 coefficient level when sign hiding is enabled.
After some debugging, the code responsible is located inside the function xRateDistOptQuant() of TComTrQuant.cpp.
Relevant code (lines 1988-2000):

```          if(piQCoef[minPos] == 32767 || piQCoef[minPos] == -32768)
{
finalChange = -1;
}

if(plSrcCoeff[minPos]>=0)
{
piDstCoeff[minPos] += finalChange ;
}
else
{
piDstCoeff[minPos] -= finalChange ;
}
```

Adding a printf after the first 'if' block:

printf("xRateDistOptQuant: minPos=%d plSrcCoeff[minPos]=%d piQCoef[minPos]=%d piDstCoeff[minPos]=%d finalChange=%d\n", minPos, plSrcCoeff[minPos], piQCoef[minPos], piDstCoeff[minPos], finalChange);

I got the following output (for a DC coefficient):

xRateDistOptQuant: minPos=0 plSrcCoeff[minPos]=20059 piQCoef[minPos]=124277 piDstCoeff[minPos]=32767 finalChange=1

As a result, the second 'if' adjusted the coefficient level to +32768 which got placed into the bitstream.

Two possible solutions:

1. Add saturation to [-32768, +32767] for piDstCoeff[minPos] after the second 'if' block.
2. It's very likely that the first 'if' is wrong and piDstCoeff should be checked instead of piQCoef. A similar piece of code can be found in lines 969-981 of the same file.

comment:1 Changed 11 years ago by DefaultCC Plugin

• Cc fbossen ksuehring davidf jct-vc@… added

comment:2 Changed 11 years ago by wangj

This issue was reported in ticket #1027. Please refer to the patch provided for ticket #1026, which solves both this issue and the problem in ticket #1026. The solution provided in the patch is the suggested solution 2.

comment:3 Changed 11 years ago by ksuehring

• Resolution set to duplicate
• Status changed from new to closed

Closed as duplicate of #1027.
Closing as duplicate

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

• David Flynn(Subscriber)
• Frank Bossen(Subscriber)
• jct-vc@…(Subscriber)
• Jing Wang(Participant)
• karl.sharman@…(Always)
• Karsten Suehring(Subscriber, Participant, Always)
• Liviu Morogan(Reporter)