Version 8 (modified by davidf, 12 years ago) (diff)

add A-Vote-B-Vote, clarify randomization

Subjective Viewing

Preparing material for subjective viewing

The following describes the setup for CE8.f. The general principal can apply to other subjective viewing sessions and some notes are provided at the end as to changing the test configuration.

The subjective test to be performed is of the form "Reference-A-B-Vote". The reference is the uncompressed original; A and B are candidates under test. Scoring is performed by comparing against the Reference.

Video data

Please provide a directory per proposal or reference containing:

  • decoded 8bit yuv files, using the naming convention of the original material on the uni-hannover ftp site. Eg, "BQMall_832x480_60.yuv".
    This will allow scripts to automatically determine the display parameters.
  • if the same sequence at multiple QPs is to be tested, place video files in directories called qpNN.
  • this data should be copied to Z:\XXX\SomeUsefulIdentifierEgCE11 on the playout computer.
    TODO: update with correct location
  • please avoid using spaces in file or directory names.

For example, assuming two proposals, "dut1" and "dut2", and the uncompressed references "orig" we should see the following:

$ ls -R dut1/

$ ls -R dut2/

$ ls -R orig/

NB, the script is designed to look for the yuv file for the "Reference" set ("orig/" in the above example) without any extra directory parts.

Preparing a playlist

Given a number of directories and a text file that lists the test points, a script will generate a playlist in the correct form to do a randomized viewing.

Two test configurations are possible:

  1. Create a textfile that names the order you wish to view the test points. Eg,
    $ cat order.txt
    Note the lack of any "dut1" or "dut2" prefix.
  2. Create a similar text file that names three test points for use in training at the start of the session.
    $ cat training.txt
  3. Run to generate a playlist:
    $ /path/to/orig /path/to/dut1 /path/to/dut2 training.txt order.txt > out.plst
  4. For a subjective test with multiple viewing sessions, it is advisable to generate multiple copies of the playlist, one for each viewing session. Each viewing session is then randomized differently.
    For example,
    $ for N in 1 2 3 4 5 6; do /path/to/orig /path/to/dut1 /path/to/dut2 training.txt order.txt > session$N.plst

Running the test

Launch the videoplayer using:

$ /path/to/videoplayer --vt.readahead=128 --vdu-fps=60 --quit --full --osdvcentre --osdstate=2 --osdtextcolour=0 $ANYOPTIONS_IVE_MISSED out.plst

The player will quit automatically when it reaches the end of the playlist.

DF: this could be made simpler if we create a small script and associate the .plst file extension with it. Then people only have to double click on the correct playlist

Notes on test configuration

  • The script will produce a playlist, with test points in the form "Reference-A-B-Vote".
    • A and B are randomized for each test point.
    • Each test point is shown twice. Once in the first half of the test, once in the second.
  • The script with two directories will produce a playlist, with test points in the form "A-B-A-B-Vote"
    • A and B are randomized for each test point.
  • The script with two or more directories will produce a playlist, with test points in the form "A-Vote-B-Vote-C-Vote"
    • A, B and C are randomized for each test point.

The scripts provide a means for customizing the style of test generated. See %options.

Notes on black & white level

By default, the videoplayer outputs R'G'B' at 16-235, however on a computer monitor, this will look wrong. The following options will set the output range to 0-255: --out-black=0 --out-range=256

Alternatively, one can interactively toggle between the two to see the difference by using the 'n' and 'b' keys.

DF: what range has the monitor been configured for?