Opened 10 years ago

Closed 10 years ago

#1259 closed defect (fixed)

Segmentation fault when using 4:2:2 subsampling

Reported by: jackh Owned by:
Priority: minor Milestone: HM+RExt-4.2
Component: HM RExt Version: RExt-4.1 (HM-12.0)
Keywords: Cc: davidf, karlsharman, jct-vc@…


When using 4:2:2, the following code in TComPrediction::initAdiPatternChType() (TComPattern.cpp):

  const Int  iBaseUnitSize    = g_uiMaxCUWidth >> g_uiMaxCUDepth;
  const Int  iUnitWidth       = iBaseUnitSize  >> pcCU->getPic()->getPicYuvRec()->getComponentScaleX(compID);
  const Int  iUnitHeight      = iBaseUnitSize  >> pcCU->getPic()->getPicYuvRec()->getComponentScaleY(compID);
  const Int  iTUWidthInUnits  = uiTuWidth  / iUnitWidth;
  const Int  iTUHeightInUnits = uiTuHeight / iUnitHeight;

can result in iTUHeightInUnits==0 when processing the two halves of a chroma TU, as the TU has been split one further than the height reflected in iUnitHeight. This then leads to a segmentation fault further down when iTUHeightInUnits-1 is used to index an array.

Change History (4)

comment:1 Changed 10 years ago by DefaultCC Plugin

  • Cc davidf karlsharman jct-vc@… added

comment:2 Changed 10 years ago by karlsharman

Fixed in r3869 by adding additional depth for parts for 4:2:2 when minimum luma TU size is not 4x4 - see #1262.

Example that I found to cause the above situation:

./bin/TAppEncoderStatic -c cfg/encoder_intra_main_rext.cfg

-c cfg/per-sequence/EBUWaterRocksClose_422_10bit.cfg
-f 1 -q 22

Please would the reporter check this solves their particular configurations.

comment:3 Changed 10 years ago by jackh

I can confirm that this is fixed.

comment:4 Changed 10 years ago by karlsharman

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • Jack Haughton(Reporter, Participant)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Subscriber, Participant)
  • karl.sharman@…(Always)
  • Karsten Suehring(Always)