Opened 14 years ago

Closed 13 years ago

#136 closed defect (fixed)

Sparse images may cause the encoder to hang when ALF is on (observed only in release build)

Reported by: minoo Owned by:
Priority: minor Milestone:
Component: HM Version: HM-3.4
Keywords: ALF, Sparse data, Singularity Cc: fbossen, ksuehring, davidf, jct-vc@…

Description

The calculation of ALF coefficients is not robust for very sparse images.
Simplest solution:
Check the returned value by gnsSolveByChol().
If the return value indicates a singular case then assign filter
coefficients arbitrarily

#ifndef MOT_SPARSE_FIX

gnsSolveByChol(E, y, filterCoeff, sqrFiltLength);

#else

int singular = gnsSolveByChol(E, y, filterCoeff, sqrFiltLength);
if(singular == 0)
{

filterCoeff[0] = 1;
for (i=1; i<sqrFiltLength; i++)

filterCoeff[i] = 0;

}

#endif

Change History (5)

comment:1 Changed 14 years ago by davidf

Please could you provide a test case that shows the problem. I've attempted a quick test using an all-zero image and was unable to reproduce.

comment:2 Changed 13 years ago by fbossen

  • Resolution set to fixed
  • Status changed from new to closed
  • Version set to HM-3.4

Fixed in r1320

comment:3 Changed 13 years ago by davidf

  • Component set to HM

Updating component after adding WD (Text) tickets

comment:4 Changed 13 years ago by davidf

  • Cc fbossen ksuehring davidf added

comment:5 Changed 13 years ago by davidf

  • Cc jct-vc@… added
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber, Participant)
  • Frank Bossen(Subscriber, Participant)
  • jct-vc@…(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Subscriber, Always)
  • Koohyar Minoo(Reporter)