Opened 12 years ago

Closed 12 years ago

#670 closed defect (fixed)

HM-8.0 reads beyond end of arrays with transform skip on

Reported by: stworrall Owned by:
Priority: minor Milestone: HM-8.1
Component: HM Version: HM-8.0
Keywords: Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

Valgrind reports two issues that are only present when transform skip is on.

The relevant valgrind reports are below.

==6808== Invalid read of size 1
==6808== at 0x456A58: TComDataCU::getTransformSkip(unsigned int, TextType) (TComDataCU.h:336)
==6808== by 0x45FF1A: TEncSearch::xRecurIntraChromaCodingQT(TComDataCU*, unsigned int, unsigned int, TComYuv*, TComYuv*, TComYuv*, unsigned int&) (TEncSearch.c pp:2186)
==6808== by 0x460697: TEncSearch::xRecurIntraChromaCodingQT(TComDataCU*, unsigned int, unsigned int, TComYuv*, TComYuv*, TComYuv*, unsigned int&) (TEncSearch.c pp:2347)
==6808== by 0x460697: TEncSearch::xRecurIntraChromaCodingQT(TComDataCU*, unsigned int, unsigned int, TComYuv*, TComYuv*, TComYuv*, unsigned int&) (TEncSearch.c pp:2347)
==6808== by 0x462A2F: TEncSearch::estIntraPredChromaQT(TComDataCU*, TComYuv*, TComYuv*, TComYuv*, TComYuv*, unsigned int) (TEncSearch.cpp:2839)
==6808== by 0x43C9FB: TEncCu::xCheckRDCostIntra(TComDataCU*&, TComDataCU*&, PartSize) (TEncCu.cpp:1431)
==6808== by 0x43942D: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:707)
==6808== by 0x43A052: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:856)
==6808== by 0x43A052: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:856)
==6808== by 0x437C4E: TEncCu::compressCU(TComDataCU*&) (TEncCu.cpp:235)
==6808== by 0x47662C: TEncSlice::compressSlice(TComPic*&) (TEncSlice.cpp:908)
==6808== by 0x4441EC: TEncGOP::compressGOP(int, int, TComList<TComPic*>&, TComList<TComPicYuv*>&, std::list<AccessUnit, std::allocator<AccessUnit> >&) (TEncGOP .cpp:581)
==6808== Address 0x60479b0 is 0 bytes after a block of size 16 alloc'd
==6808== at 0x4C244E8: malloc (vg_replace_malloc.c:236)
==6808== by 0x485130: TComDataCU::create(unsigned int, unsigned int, unsigned int, bool, int, bool) (TComDataCU.cpp:166)
==6808== by 0x436DD2: TEncCu::create(unsigned char, unsigned int, unsigned int) (TEncCu.cpp:83)
==6808== by 0x41D380: TEncTop::create() (TEncTop.cpp:97)
==6808== by 0x4184DB: TAppEncTop::xCreateLib() (TAppEncTop.cpp:285)
==6808== by 0x418635: TAppEncTop::encode() (TAppEncTop.cpp:329)
==6808== by 0x406CC9: main (encmain.cpp:75)

==6808== Invalid read of size 1
==6808== at 0x4C26040: memcpy (mc_replace_strmem.c:497)
==6808== by 0x45E9C2: TEncSearch::xLoadIntraResultQT(TComDataCU*, unsigned int, unsigned int, bool) (TEncSearch.cpp:1909)
==6808== by 0x45C81E: TEncSearch::xRecurIntraCodingQT(TComDataCU*, unsigned int, unsigned int, bool, TComYuv*, TComYuv*, TComYuv*, unsigned int&, unsigned int& , bool, double&) (TEncSearch.cpp:1533)
==6808== by 0x4619DA: TEncSearch::estIntraPredQT(TComDataCU*, TComYuv*, TComYuv*, TComYuv*, TComYuv*, unsigned int&, bool) (TEncSearch.cpp:2605)
==6808== by 0x43C911: TEncCu::xCheckRDCostIntra(TComDataCU*&, TComDataCU*&, PartSize) (TEncCu.cpp:1427)
==6808== by 0x4394E5: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:713)
==6808== by 0x43A052: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:856)
==6808== by 0x43A052: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:856)
==6808== by 0x43A052: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:856)
==6808== by 0x437C4E: TEncCu::compressCU(TComDataCU*&) (TEncCu.cpp:235)
==6808== by 0x47662C: TEncSlice::compressSlice(TComPic*&) (TEncSlice.cpp:908)
==6808== by 0x4441EC: TEncGOP::compressGOP(int, int, TComList<TComPic*>&, TComList<TComPicYuv*>&, std::list<AccessUnit, std::allocator<AccessUnit> >&) (TEncGOP .cpp:581)
==6808== Address 0x664f140 is 0 bytes after a block of size 64 alloc'd
==6808== at 0x4C24A72: operator new[](unsigned long) (vg_replace_malloc.c:305)
==6808== by 0x45937F: TEncSearch::init(TEncCfg*, TComTrQuant*, int, int, int, int, TEncEntropy*, TComRdCost*, TEncSbac*, TEncSbac*) (TEncSearch.cpp:275)
==6808== by 0x41EBDB: TEncTop::init() (TEncTop.cpp:342)
==6808== by 0x418545: TAppEncTop::xInitLib() (TAppEncTop.cpp:300)
==6808== by 0x418644: TAppEncTop::encode() (TAppEncTop.cpp:330)
==6808== by 0x406CC9: main (encmain.cpp:75)

Attachments (2)

TEncSearch.cpp.patch​ (1.8 KB) - added by jzxu 12 years ago.
TEncSearch.cpp.2.patch​ (1.8 KB) - added by jzxu 12 years ago.

Download all attachments as: .zip

Change History (5)

comment:1 Changed 12 years ago by DefaultCC Plugin

  • Cc fbossen ksuehring davidf jct-vc@… added

Changed 12 years ago by jzxu

Changed 12 years ago by jzxu

comment:2 Changed 12 years ago by jzxu

The patch is attached. No change to the results under CTC.

comment:3 Changed 12 years ago by ksuehring

  • Milestone set to HM-8.1
  • Resolution set to fixed
  • Status changed from new to closed

The patch has been applied in r2726

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • Frank Bossen(Subscriber)
  • jct-vc@…(Subscriber)
  • jzxu(Participant)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Participant, Always)
  • Stewart Worrall(Reporter)