Opened 10 years ago

Closed 10 years ago

#1219 closed defect (fixed)

SAO code for different InternalBitDepth(C)

Reported by: o.nakagami Owned by:
Priority: minor Milestone: HM-14.0
Component: HM Version: HM-12.1
Keywords: Cc: davidf, karlsharman, jct-vc@…

Description

HM-12.1+RExt-5.1 encoder crashes with segmantation fault when InternalBitDepth is less than InternalBitDepthC. It seems that the current SAO code has an issue in the case. By disabling SAO, the encoder works without error.

Example of failure:

TAppEncoderStatic
-c ../cfg/encoder_randomaccess_main_rext.cfg -c ../cfg/per-sequence/BirdsInCage_444_10bit.cfg
--InternalBitDepth=8 --InternalBitDepthC=12

It seems the issue is related to #1192

Attachments (1)

patch_hm13.0_ticket_1219.txt​ (1.8 KB) - added by karlsharman 10 years ago.

Download all attachments as: .zip

Change History (4)

comment:1 Changed 10 years ago by DefaultCC Plugin

  • Cc davidf karlsharman jct-vc@… added

Changed 10 years ago by karlsharman

comment:2 Changed 10 years ago by karlsharman

  • Component changed from HM RExt to HM
  • Version changed from RExt-4.1 (HM-12.0) to HM-12.1

The problem also affects HM.

In the latest SAO code, a table has been used to replace the sgn function. The table is initialised to be the size of the luma [is it really necessary to replace the sgn function with a look-up table, especially given the size of the table for higher bit depths?]

eg
The following produces encoder/decoder mismatch and, if you are lucky, a stack-trace being output from the encoder on exit.

./bin/TAppEncoderStatic -c cfg/encoder_randomaccess_main10.cfg -c cfg/per-sequence/BasketballDrillText.cfg -q 32 -f 9 --SEIDecodedPictureHash=1 --InternalBitDepth=8 --InternalBitDepthC=10

Patch attached for HM13.0

comment:3 Changed 10 years ago by ksuehring

  • Milestone set to HM-14.0
  • Resolution set to fixed
  • Status changed from new to closed

The patch has been applied in r3810.

I'm still curious what the performance impact of using a sign function instead of the lookup table would be.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Subscriber, Participant)
  • karl.sharman@…(Always)
  • Karsten Suehring(Participant, Always)
  • Ohji Nakagami(Reporter)