Opened 11 years ago

Closed 11 years ago

#1169 closed defect (fixed)

HM-12.0-RExt-4.1 encoder bug when enabling large-TU transformSkip and RDPCM

Reported by: xipe Owned by:
Priority: major Milestone: HM+RExt-4.2
Component: HM RExt Version: RExt-4.1 (HM-12.0)
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…, karlsharman, jct-vc@…

Description

In HM-12.0-RExt-4.1 RCE3 software, I encounterd the following problem.

When enabling large-TU transformSkip and RDPCM by setting TransformSkipLog2MaxSize=5 and TransformSkipFast=0, I run the encoding several times but each time the encoding results are different. Three different results as below appear
1) Bitrate 28163.0400 Y-PSNR 39.0933 U-PSNR 38.6322 V-PSNR 38.9020
2) Bitrate 28185.1200 Y_PSNR 39.0793 U-PSNR 38.6377 V-PSNR 38.9982
3) Bitrate 28092.9600 Y-PSNR 39.1154 U-PSNR 38.6070 V-PSNR 39.0385
The test condition is sc_wordEditing_1280x720_60_8bit_444.yuv, only encode the first frame, QP 32, Release x64

However, when I disabling RDPCM by setting the macro "RExtNRCE2_RESIDUAL_DPCM" to 0 and still enable the large-TU transformSkip, the problem disappears.

Another point is that when I enable TS only on 4x4, this problem also doesn't occur.

There might be some memory leak bug in the encoder but I didn't find the place. Did anybody encounter the same problem in the test?

Attachments (2)

encoder_intra_main_rext.cfg​ (9.0 KB) - added by xipe 11 years ago.
The encoder configuration file when the problem occurs
WordEditing_444.cfg​ (662 bytes) - added by xipe 11 years ago.
The encoder configuration file when the problem occurs

Download all attachments as: .zip

Change History (6)

comment:1 Changed 11 years ago by DefaultCC Plugin

  • Cc karlsharman jct-vc@… added

Changed 11 years ago by xipe

The encoder configuration file when the problem occurs

Changed 11 years ago by xipe

The encoder configuration file when the problem occurs

comment:2 follow-up: Changed 11 years ago by karlsharman

I believe this is caused by a missing '-2':

TComTrQuant.cpp:3969 (function xQuantiseSample)

Int *quantiserScale = getQuantCoeff(scal..., log2TrSize);

should be:

Int *quantiserScale = getQuantCoeff(scal..., log2TrSize-2);

This is causing the wrong scaling lists to be used whenever inter RDPCM is used.

Fortunately, for the test conditions of AHG5 (=RCE1), RCE2 and RCE3, the transform skip size is 4 and the scaling lists are flat, so this shouldn't have caused errors in anchors.

Please could you confirm the above fix is OK?

comment:3 in reply to: ↑ 2 Changed 11 years ago by xipe

Replying to karlsharman:

Thanks! The problem is solved by your fix.

I believe this is caused by a missing '-2':

TComTrQuant.cpp:3969 (function xQuantiseSample)

Int *quantiserScale = getQuantCoeff(scal..., log2TrSize);

should be:

Int *quantiserScale = getQuantCoeff(scal..., log2TrSize-2);

This is causing the wrong scaling lists to be used whenever inter RDPCM is used.

Fortunately, for the test conditions of AHG5 (=RCE1), RCE2 and RCE3, the transform skip size is 4 and the scaling lists are flat, so this shouldn't have caused errors in anchors.

Please could you confirm the above fix is OK?

comment:4 Changed 11 years ago by karlsharman

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in #3666.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Subscriber, Participant)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Always)