Subjective Viewing

Preparing material for subjective viewing

The following describes the setup for CE8.f. The general principal can apply to other subjective viewing sessions and some notes are provided at the end as to changing the test configuration.

The subjective test to be performed is of the form "Reference-A-B-Vote". The reference is the uncompressed original; A and B are candidates under test. Scoring is performed by comparing against the Reference.

Video data

Please provide a directory per proposal or reference containing:

• decoded 8bit yuv files, using the naming convention of the original material on the uni-hannover ftp site. Eg, "BQMall_832x480_60.yuv".
  This will allow scripts to automatically determine the display parameters.

• if the same sequence at multiple QPs is to be tested, place video files in directories called qpNN.

• this data should be copied to Z:\XXX\SomeUsefulIdentifierEgCE11 on the playout computer.

• please avoid using spaces in file or directory names.

For example, assuming two proposals, "dut1" and "dut2", and the uncompressed references "orig" we should see the following:

$ ls -R dut1/
dut1/qp22/BQMall_832x480_60.yuv
dut1/qp22/ParkScene_1920x1080_24.yuv
...
dut1/qp27/BQMall_832x480_60.yuv
dut1/qp27/ParkScene_1920x1080_24.yuv
...

$ ls -R dut2/
dut2/qp22/BQMall_832x480_60.yuv
dut2/qp22/ParkScene_1920x1080_24.yuv
...
dut2/qp27/BQMall_832x480_60.yuv
dut2/qp27/ParkScene_1920x1080_24.yuv
...

$ ls -R orig/
orig/BQMall_832x480_60.yuv
orig/ParkScene_1920x1080_24.yuv
...

NB, the script is designed to look for the yuv file for the "Reference" set ("orig/" in the above example) without any extra directory parts.

Preparing a playlist

Given a number of directories and a text file that lists the test points, a script will generate a playlist in the correct form to do a randomized viewing.

Two test configurations are possible:

• ​gen-playlist-Ref-A-B.pl: Test points use the form "Reference-A-B-Vote".
• ​gen-playlist-A-B-A-B.pl: Test points use the form "A-B-A-B-Vote".

1. Create a textfile that names the order you wish to view the test points. Eg,

   $ cat order.txt
   qp22/BQMall_832x480_60.yuv
   qp27/BQMall_832x480_60.yuv
   ...
   qp22/ParkScene_1920x1080_24.yuv
   qp27/ParkScene_1920x1080_24.yuv
   ...
   

   Note the lack of any "dut1" or "dut2" prefix.
2. Create a similar text file that names three test points for use in training at the start of the session.

   $ cat training.txt
   qp27/Cactus_1920x1080_50.yuv
   qp37/RaceHorses_832x480_30.yuv
   qp32/BasketballDrill_832x480_50.yuv
   

3. Run ​gen-playlist-Ref-A-B.pl to generate a playlist:
   

   $ gen-playlist-Ref-A-B.pl /path/to/orig /path/to/dut1 /path/to/dut2 training.txt order.txt > out.plst
   

4. For a subjective test with multiple viewing sessions, it is advisable to generate multiple copies of the playlist, one for each viewing session. Each viewing session is then randomized differently.
   For example,

   $ for N in 1 2 3 4 5 6; do
        gen-playlist-Ref-A-B.pl /path/to/orig /path/to/dut1 /path/to/dut2 training.txt order.txt > session$N.plst
     done
   

Running the test

Launch the ​videoplayer using:

$ /path/to/videoplayer --vt.readahead=128 --vdu-fps=60 --quit --full --osdstate=2 $ANYOPTIONS_IVE_MISSED out.plst

The player will quit automatically when it reaches the end of the playlist.

DF: this could be made simpler if we create a small script and associate the .plst file extension with it. Then people only have to double click on the correct playlist

Notes on test configuration

• The ​gen-playlist-Ref-A-B.pl script will produce a playlist, with test points in the form "Reference-A-B-Vote".
  • A and B are randomized for each test point.
  • The ordering of test points is randomized.
  • Each test point is shown twice. Once in the first half of the test, once in the second.
• ​gen-playlist-A-B-A-B.pl script with two directories will produce a playlist, with test points in the form "A-B-A-B-Vote"
  • A and B are randomized for each test point.
  • The ordering of test points is randomized.
• ​gen-playlist-A-B-A-B.pl script with three or more directories will produce a playlist, with test points in the form "A-A-B-B-C-C-Vote"
  • A, B and C are randomized for each test point.
  • The ordering of test points is randomized.

The scripts provide a means for customizing the style of test generated. See %options.

Notes on black & white level

By default, the videoplayer outputs R'G'B' at 16-235, however on a computer monitor, this will look wrong. The following options will set the output range to 0-255: --out-black=0 --out-range=256

Alternatively, one can interactively toggle between the two to see the difference by using the 'n' and 'b' keys.

DF: what range has the monitor been configured for?