Opened 10 years ago

Closed 8 years ago

#419 closed defect (fixed)

transformSubdivFlag signaled when Log2CbSize=6 and Log2MaxTrafoSize=4 and implicit split

Reported by: bbross Owned by:
Priority: minor Milestone: HM-13.0
Component: HM Version: HM-6.0
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

In HM-6.0, when the transform tree of a CU with 64x64 luma coding block is processed and (not common conditions)
--QuadtreeTUMaxDepthInter=1 (implicit TU split for non-2Nx2N partitions)
--QuadtreeTULog2MaxSize=4

in Void TDecEntropy::xDecodeTransform, it occurs that for non-2Nx2N inter partitions, a transform block subdivision flag is signaled. Furthermore in HE10 configuration a encoder/decoder mismatch occurs.

This is reproducible with the following encoder calls:
Without mismatch (main):

1.a ../build/Release/TAppEncoder -c encoder_lowdelay_main.cfg -c per-sequence/BQSquare.cfg -b BQSquare_LD_main-2.bin -q 22 -f 8 --QuadtreeTULog2MaxSize=4 --QuadtreeTUMaxDepthInter=1
1.b ../build/Release/TAppEncoder -c ./encoder_lowdelay_main.cfg -c ./per-sequence/NebutaFestival_10bit.cfg -b str_main.bin -q 22 -f 8 --QuadtreeTULog2MaxSize=4 --MaxPartitionDepth=1 --QuadtreeTUMaxDepthInter=1 --QuadtreeTUMaxDepthIntra=1

With mismatch (he10):

2.a ../build/Release/TAppEncoder -c encoder_lowdelay_he10.cfg -c per-sequence/BQSquare.cfg -b BQSquare_LD_he10-2.bin -q 22 -f 8 --QuadtreeTULog2MaxSize=4 --QuadtreeTUMaxDepthInter=1
2.b ../build/Release/TAppEncoder -c ./encoder_lowdelay_he10.cfg -c ./per-sequence/NebutaFestival_10bit.cfg -b str_he10.bin -q 22 -f 10 --QuadtreeTULog2MaxSize=4 --MaxPartitionDepth=1 --QuadtreeTUMaxDepthInter=1 --QuadtreeTUMaxDepthIntra=1

and decoding the corresponding bitstreams with a breakpoint in l.633 in TDecEntropy.cpp

    m_pcEntropyDecoderIf->parseTransformSubdivFlag( uiSubdiv, uiDepth );

The corresponding bitstreams are attached.

This can easily be fixed by removing '- SplitFlag' in l.2055 in TComDataCU::getQuadtreeTULog2MinSizeInCU( UInt uiIdx ):

    uiLog2MinTUSizeInCU = m_pcSlice->getSPS()->getQuadtreeTULog2MaxSize();

After the fix is applied, the subdivision flag is not signaled anymore and the mismatch is gone and in config
2.a the case of a 64x64 CB with non-2Nx2N inter partitions is not encoded (chosen) anymore
2.b the case of a 64x64 CB with non-2Nx2N inter partitions is chosen but no subdivision flag signaled anymore.

After I decrypted what the function TComDataCU::getQuadtreeTULog2MinSizeInCU( UInt uiIdx ) is doing, especially w.r.t to draft text, I took the chance to clean the function up. The cleanup patch (bit exact) and the cleanup patch including the fix are attached. The fixed code matches with draft text.

Another observation in the code is that TComDataCU::getQuadtreeTULog2MinSizeInCU( UInt uiIdx ) only depends on the CU address AbsPartIdx/uiIdx and does not change during recursive call of TDecEntropy::xDecodeTransform/TDecEntropy::xDecodeTransformSubdiv. This function can be taken out of the recursion and passed, e.g. as a parameter uiLog2MinTUSizeInCU so that multiple calls of the function leading to the same result can be avoided.

Attachments (4)

BQSquare_LD_main-2.bin (51.1 KB) - added by bbross 10 years ago.
1.a bitstream
BQSquare_LD_he10-2.bin (50.5 KB) - added by bbross 10 years ago.
2.a bitstream
419cleanup.patch (3.4 KB) - added by bbross 10 years ago.
patch with bit-exact cleanup
419cleanup_fix.patch (4.0 KB) - added by bbross 10 years ago.
patch with bit-exact cleanup and fix

Download all attachments as: .zip

Change History (7)

comment:1 Changed 10 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

Changed 10 years ago by bbross

1.a bitstream

Changed 10 years ago by bbross

2.a bitstream

Changed 10 years ago by bbross

patch with bit-exact cleanup

Changed 10 years ago by bbross

patch with bit-exact cleanup and fix

comment:2 follow-up: Changed 9 years ago by ksuehring

Cleanup and patch have been committed in r2144 and r2145. Shall that ticket stay open for the suggested change to call getQuadtreeTULog2MinSizeInCU() only once?

comment:3 in reply to: ↑ 2 Changed 8 years ago by fbossen

  • Milestone set to HM-13.0
  • Resolution set to fixed
  • Status changed from new to closed

Replying to ksuehring:

Cleanup and patch have been committed in r2144 and r2145. Shall that ticket stay open for the suggested change to call getQuadtreeTULog2MinSizeInCU() only once?

Suggested change made in r3780

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • Benjamin Bross(Reporter)
  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber, Participant)
  • jct-vc@…(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Participant, Always)