The HM Encoder takes excessive memory, especially when considering increasing picture sizes and GOP structures.

For example, when encoding Traffic (2560x1600), with the 16-frame GOP structure used by JVET, memory requirements are 2482 MiB.

For 4K sequences, memory requirements are approximately 2.1 times higher; i.e. ~ 5361 MiB.

For derived branches, such as JEM, memory utilization is even higher.

The reason is because all memory is allocated for all the processing at the outset. This means that for a 16 frame GOP simulation, up to 32 TComPic objects will be present, all of which have memory allocated for encoding decisions.

15 of the frames will only actually contain source image data

• these are frames that have been buffered up prior to the complete GOP encoding.

1 of the frames will be being processed, and will therefore need

source data and encoding decision data.

Up to 16 frames will be kept as part of the reference picture

list structure. These technically only require the
reconstructed picture data and other information present in
the DPB.

The attached patch reduced the memory usage by ~60% for the above example, reducing Traffic, GOP-16 from 2482 MiB to just 1020 MiB.

It achieves this by allocating memory only when it is needed.
It also introduces a DPB structure into which side information
is placed for reference frame usage.

Note that there is not such a significant problem in the decoder, as it only allocates data for pictures when they are decoded, and destroys the data afterwards (as it also has to cope with video format changes).

The patch does not affect run-time or coding efficiency.

I have uploaded it as a ticket so that other branches may examine and comment.