Vidyo4, 2nd Frame

DTRACE_CABAC_V( g_nSymbolCounter++ )
DTRACE_CABAC_T( "\twriteFilterCoeffs=" )
DTRACE_CABAC_T( "\t" )
DTRACE_CABAC_V( pAlfParam->filters_per_group_diff )
DTRACE_CABAC_T( "\t" )
DTRACE_CABAC_V( pAlfParam->num_coeff )
DTRACE_CABAC_N

* TraceDec.txt
15748678 ALF Flag=1
15748679 writeFilterCoeffs= 16 8
15748680 state=2
* TRACEENC.TXT
15748678 ALF Flag=1
15748679 writeFilterCoeffs= 1 8
15748680 state=2
*

This indeed seems to be an issue related to ALF. It looks like the pAlfParam structure used in the encoder is not valid and bad bitstreams are hence generated. The issue was also replicated on a single frame using low-complexity settings as a base:

bossen$ build/Release/TAppEncoder -c cfg/encoder_intra_loco.cfg -c cfg/per-sequence/Vidyo4.cfg -q 0 -f 1 --ALF=1 --BitIncrement=3

TMuC SW candidate: Encoder Version [0.9][Mac OS X][GCC 4.2.1][64 bit]

Input File : yuv/vidyo4_1280x720_60.yuv
Bitstream File : str.bin
Reconstruction File : rec.yuv
Real Format : 1280x720 60Hz
Internal Format : 1280x720 60Hz
Frame index : 0 - 0 (1 frames)
Number of Ref. frames (P) : 2
Number of Ref. frames (B_L0) : 1
Number of Ref. frames (B_L1) : 1
Number of Reference frames : 2
CU size / depth : 64 / 4
RQT trans. size (min / max) : 4 / 32
Max RQT depth inter : 2
Max RQT depth intra : 1
Motion search range : 64
Intra period : 1
QP : 0.00
GOP size : 1
Rate GOP size : 1
Bit increment : 3
Luma interpolation : TEN directional interpolation filter
Chroma interpolation : Bi-linear filter
Entropy coder : VLC

TOOL CFG: ALF:1 IBD:1 HAD:1 SRD:0 RDQ:1 SQP:0 ASR:0 PAD:0 LDC:1 NRF:0 BQP:0 QBO:0 GPB:1 FEN:1 RQT:1 CIP:0 ROT:0 AIS:0 MRG:1 IMP:0 SPF:1 AMP:0

POC 0 ( I-SLICE, QP 0 ) 4396520 bits [Y 65.0604 dB U 64.1718 dB V 64.2137 dB] [ET 11 ] [L0 ] [L1 ]

SUMMARY --------------------------------------------------------

Total Frames | Bitrate Y-PSNR U-PSNR V-PSNR

1 a 263791.2000 65.0604 64.1718 64.2137

I Slices--------------------------------------------------------

Total Frames | Bitrate Y-PSNR U-PSNR V-PSNR

1 i 263791.2000 65.0604 64.1718 64.2137

P Slices--------------------------------------------------------

Total Frames | Bitrate Y-PSNR U-PSNR V-PSNR

0 p nan nan nan nan

B Slices--------------------------------------------------------

Total Frames | Bitrate Y-PSNR U-PSNR V-PSNR

0 b nan nan nan nan

Total Time: 10.912 sec.

bossen$ build/Release/TAppDecoder -b str.bin -o dec.yuv
TMuC SW candidate: Decoder Version [0.9][Mac OS X][GCC 4.2.1][64 bit]
Assertion failed: (uiNumberOfBits <= 32), function read, file source/Lib/TLibCommon/TComBitStream.cpp, line 149.
Abort trap

Fixed in r431