Opened 6 years ago

Closed 6 years ago

#1481 closed defect (fixed)

computation overflow in HM reference C model

Reported by: swong10 Owned by: swong10
Priority: major Milestone:
Component: HM SCC Version:
Keywords: overflow, palette Cc: jct-vc@…

Description

We may have found a bug in the HM model. There is a mismatch between the written specification and HM reference C model.

Dec 2016 Spec: 8.4.4.2.7 Decoding process for palette mode

HM code version : HM-16.15+SCM-8.5
Function : TDecCu::xDecodePaletteTexture() in TDecCu.cpp
The problem equation happens in both clauses of an if-else statement.
The overflow happens in tmpVal, which is used to store the inverse quantized palette escape values in (8-77).
tmpVal is then clipped according to (8-78) to generate the reconstructed samples.
levelScale <= 72 (7-bit).

In SCM, tmpVal is of type “Pel”. So, once we changed Pixel from Short to Int the overflow was fixed.

It is a high bit depth problem, most likely only happen when pixel depth is > 8-bit.

Attachments (1)

overflow bug2.pdf​ (83.0 KB) - added by swong10 6 years ago.
Indicates where the HM code and spec overflow mismatches

Download all attachments as: .zip

Change History (2)

Changed 6 years ago by swong10

Indicates where the HM code and spec overflow mismatches

comment:1 Changed 6 years ago by libin

  • Resolution set to fixed
  • Status changed from new to closed

fixed in r4932.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • Bin Li(Participant)
  • jct-vc@…(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Always)
  • Samuel Wong(Reporter, Owner)