Opened 11 years ago Closed 11 years ago #1259 closed defect (fixed)Segmentation fault when using 4:2:2 subsampling
Description
When using 4:2:2, the following code in TComPrediction::initAdiPatternChType() (TComPattern.cpp): const Int iBaseUnitSize = g_uiMaxCUWidth >> g_uiMaxCUDepth; const Int iUnitWidth = iBaseUnitSize >> pcCU->getPic()->getPicYuvRec()->getComponentScaleX(compID); const Int iUnitHeight = iBaseUnitSize >> pcCU->getPic()->getPicYuvRec()->getComponentScaleY(compID); const Int iTUWidthInUnits = uiTuWidth / iUnitWidth; const Int iTUHeightInUnits = uiTuHeight / iUnitHeight;
can result in iTUHeightInUnits==0 when processing the two halves of a chroma TU, as the TU has been split one further than the height reflected in iUnitHeight. This then leads to a segmentation fault further down when iTUHeightInUnits-1 is used to index an array. Change History (4)comment:1 Changed 11 years ago by DefaultCC Plugin
comment:2 Changed 11 years ago by karlsharmancomment:3 Changed 11 years ago by jackh
I can confirm that this is fixed. comment:4 Changed 11 years ago by karlsharman
Note: See
TracTickets for help on using
tickets. | This list contains all users that will be notified about changes made to this ticket. These roles will be notified: Reporter, Owner, Subscriber, Participant
|
Fixed in r3869 by adding additional depth for parts for 4:2:2 when minimum luma TU size is not 4x4 - see #1262.
Example that I found to cause the above situation:
./bin/TAppEncoderStatic -c cfg/encoder_intra_main_rext.cfg
Please would the reporter check this solves their particular configurations.