Opened 9 years ago

Closed 7 years ago

Last modified 7 years ago

#1070 closed defect (fixed)

HM performs loop filtering after replacing SPS/PPS

Reported by: pieterkapsenberg Owned by:
Priority: major Milestone: HM-16.3
Component: HM Version: HM-10.0
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

In HM-10.0, loop filtering and SEI digest checking is performed after the first slice header of the next picture is decoded. By then, the SPS and PPS needed to deblock & SAO-filter the finished picture may have already been replaced by new ones with the same ID.

A patch is attached that fixed the issue. It changes the overal decoder flow a bit:

  • The slice decode updates a boolean that indicates if the decompressed slice was the last of the pic, as detected by last CU address being equal to numCUsInFrame-1.
  • If the boolean is true, perform loop filtering.
  • When SEI digest message arrives, perform the check then. Digest code was moved from TDecGop to TDecTop

Attachments (4)

lf_fix.patch (14.0 KB) - added by pieterkapsenberg 9 years ago.
lf_fix_2.patch (15.5 KB) - added by pieterkapsenberg 9 years ago.
modified to handle dependent slices and the case where SEI arrives before the last slice of the picture
TDecTop.cpp.patch (1.4 KB) - added by pieterkapsenberg 9 years ago.
comments out SEI parameter sets
lf_fix_hm10dev.patch (15.6 KB) - added by pieterkapsenberg 9 years ago.
loop filter patch updated for HM10.0-dev

Download all attachments as: .zip

Change History (25)

comment:1 Changed 9 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

Changed 9 years ago by pieterkapsenberg

Changed 9 years ago by pieterkapsenberg

modified to handle dependent slices and the case where SEI arrives before the last slice of the picture

comment:2 Changed 9 years ago by ksuehring

I think that creates an issue when a decoded picture hash message arrives between two VCL NAL units of a picture.

SPS and PPS are not directly overwritten, see ParametersetManagerDecoder. They are stored in a "prefetch"-buffer and only overwrite the old parameter sets when applyPrefetchedPS() is called. This is called only with the activation of a new parameter set, which should happen in decoding, not parsing.

There exist two functions for accessing either a new parameter set (getPrefetched{VSP}PS) or an old one (get{VSP}PS).

comment:3 Changed 9 years ago by pieterkapsenberg

The second patch should handle the case of the SEI showing up between two slices. Regarding the operation of the ParametersetManagerDecoder, I think current HM still ends up replacing the PPS/SPS before loop filtering when a parameter activation SEI message shows up.

Changed 9 years ago by pieterkapsenberg

comments out SEI parameter sets

comment:4 Changed 9 years ago by pieterkapsenberg

I suppose a simpler fix is to just have HM ignore parameter activation SEIs, since SEIs are ignorable. Patch that does this is attached, and also solves the issue I was seeing.

comment:5 Changed 9 years ago by ksuehring

Unfortunately for a stream that contains buffering and timing SEI the startup won't work when ignoring the SEI (i.e. an SPS need to be active before buffering and timing can be parsed). Maybe we can ignore the SEI in the middle of the bitstream.

comment:6 Changed 9 years ago by pieterkapsenberg

That would be ok, it is still simpler than the second patch that initiates loop filtering after the picture completes, although I do think that seems more "correct".

comment:7 Changed 9 years ago by ksuehring

I would like to test lf_fix_2. Filtering at the end of the picture seems to be helpful to also avoid reading the slice header twice. Unfortunately the patch does not apply to the current HM-10.0-dev branch. Could you please update it?

Changed 9 years ago by pieterkapsenberg

loop filter patch updated for HM10.0-dev

comment:8 Changed 9 years ago by pieterkapsenberg

Attached

comment:9 Changed 9 years ago by pieterkapsenberg

ok the patch needs a small tweak, otherwise SEIs of skipped leading pictures will try to get applied to the previously decoded IRAP.

In TDecTop::xDecodeSlice, add the m_digestCanBeChecked assignments:

  if (isRandomAccessSkipPicture(iSkipFrame, iPOCLastDisplay))
  {
    m_digestCanBeChecked = false;
    return false;
  }
  // Skip TFD pictures associated with BLA/BLANT pictures
  if (isSkipPictureForBLA(iPOCLastDisplay))
  {
    m_digestCanBeChecked = false;
    return false;
  }

comment:10 follow-up: Changed 9 years ago by conrad.ho

The activation SEI can activate VPS, SPS, but not PPS. VPS and PPS do not contain SAO and DBF related parameters. So I'm wondering why activation SEI will make it wrong?

comment:11 in reply to: ↑ 10 Changed 9 years ago by pieterkapsenberg

Replying to conrad.ho:

The activation SEI can activate VPS, SPS, but not PPS. VPS and PPS do not contain SAO and DBF related parameters. So I'm wondering why activation SEI will make it wrong?

The real problem is that loop filter code gets the pic height & width from the newly activated SPS, which isn't parsed yet (all the values are garbage).

comment:12 Changed 9 years ago by pieterkapsenberg

Sorry, my previous statement is incorrect. The slice's pointer to the old SPS refers to de-allocated memory by the time the loop filter runs.

comment:13 Changed 8 years ago by pieterkapsenberg

  • Milestone changed from HM-10.1 to HM-10.2

comment:14 Changed 8 years ago by ksuehring

  • Resolution set to fixed
  • Status changed from new to closed

The patch has been applied in r3497.

There are alternative ways to solve the issue, but they might require more work.

comment:15 Changed 8 years ago by ksuehring

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening. The patch was reverted in trunk r3508 because it breaks decoding of conformance streams SLIST_B and SLIST_D

comment:16 Changed 8 years ago by ksuehring

A severe effect is that the output picture is wrong from the beginning while the MD5 check fails only much later. We should make sure this does not happen.

comment:17 Changed 8 years ago by pieterkapsenberg

Perhaps each slice can simply allocate and keep a duplicate copy of its SPS/PPS, and deallocate in the destructor.

comment:18 Changed 8 years ago by pieterkapsenberg

  • Milestone changed from HM-11.0 to HM-11.1

comment:19 Changed 8 years ago by ksuehring

  • Milestone HM-11.1 deleted

comment:20 Changed 7 years ago by karlsharman

  • Resolution set to fixed
  • Status changed from reopened to closed

Fixed in r4219 by placing a constant instance of the SPS and PPS within
TComPicSym. The SPS/PPS pointers within a slice point to the instance
within the TComPicSym. The decoder will abort if the active PPS/SPS
changes whilst decoding a picture.

comment:21 Changed 7 years ago by karlsharman

  • Milestone set to HM-16.3

Batch modify of recently closed tickets to set milestone to HM-16.3.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • Conrad Ho(Participant)
  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Participant)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Participant, Always)
  • Pieter Kapsenberg(Reporter, Participant)