Changes between Version 1 and Version 2 of SanJoséSubjectiveViewing


Ignore:
Timestamp:
Jan 28, 2012, 7:15:05 PM (13 years ago)
Author:
davidf
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • SanJoséSubjectiveViewing

    v1 v2  
    22
    33== Preparing material for subjective viewing ==
     4The following describes the setup for CE8.f.  The general principal can apply to
     5other subjective viewing sessions and some notes are provided at the end as to
     6changing the test configuration.
     7
     8The subjective test to be performed is of the form "Reference-A-B-Vote".
     9The reference is the uncompressed original; A and B are candidates under test.
     10Scoring is performed by comparing against the Reference.
    411
    512=== Video data ===
    … …  
    1724 - please avoid using spaces in file or directory names.
    1825
    19 For example, assuming two proposals, "REF" and "DUT" we should see the following:
     26For example, assuming two proposals, "dut1" and "dut2", and the uncompressed
     27references "orig" we should see the following:
    2028{{{
    21 $ ls -R REF/
    22 REF/qp22/BQMall_832x480_60.yuv
    23 REF/qp22/ParkScene_1920x1080_24.yuv
     29$ ls -R dut1/
     30dut1/qp22/BQMall_832x480_60.yuv
     31dut1/qp22/ParkScene_1920x1080_24.yuv
    2432...
    25 REF/qp27/BQMall_832x480_60.yuv
    26 REF/qp27/ParkScene_1920x1080_24.yuv
     33dut1/qp27/BQMall_832x480_60.yuv
     34dut1/qp27/ParkScene_1920x1080_24.yuv
    2735...
    2836
    29 $ ls -R DUT/
    30 DUT/qp22/BQMall_832x480_60.yuv
    31 DUT/qp22/ParkScene_1920x1080_24.yuv
     37$ ls -R dut2/
     38dut2/qp22/BQMall_832x480_60.yuv
     39dut2/qp22/ParkScene_1920x1080_24.yuv
    3240...
    33 DUT/qp27/BQMall_832x480_60.yuv
    34 DUT/qp27/ParkScene_1920x1080_24.yuv
     41dut2/qp27/BQMall_832x480_60.yuv
     42dut2/qp27/ParkScene_1920x1080_24.yuv
     43...
     44
     45$ ls -R orig/
     46orig/BQMall_832x480_60.yuv
     47orig/ParkScene_1920x1080_24.yuv
    3548...
    3649}}}
    3750
     51NB, the script is designed to look for the yuv file for the "Reference" set ("orig/"
     52in the above example) without any extra directory parts.
     53
    3854=== Preparing a playlist ===
    39 We have a script that will create a playlist in the correct form to do a randomized
    40 A-B-A-B viewing.  It requires an input file to list the sequences that are of interest
    41 and the order in which to evaluate them:
     55Given a number of directories and a text file that lists the test points,
     56a script will generate a playlist in the correct form to do a randomized
     57viewing.
     58
     59Two test configurations are possible:
     60 - [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-Ref-A-B.pl|gen-playlist-Ref-A-B.pl]]:
     61   Test points use the form "Reference-A-B-Vote".
     62 - [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-A-B-A-B.pl|gen-playlist-A-B-A-B.pl]]:
     63   Test points use the form "A-B-A-B-Vote".
    4264
    4365 1. Create a textfile that names the order you wish to view the test points.  Eg,
    … …  
    5173...
    5274}}}
    53     Note the lack of any "DUT" or "REF" prefix.
    54  1. Run [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-shuffled.pl|gen-playlist-shuffled.pl]] to generate a playlist:
     75    Note the lack of any "dut1" or "dut2" prefix.
     76 1. Create a similar text file that names three test points for use in training at
     77    the start of the session.
     78{{{
     79$ cat training.txt
     80qp27/Cactus_1920x1080_50.yuv
     81qp37/RaceHorses_832x480_30.yuv
     82qp32/BasketballDrill_832x480_50.yuv
     83}}}
     84 1. Run [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-Ref-A-B.pl|gen-playlist-Ref-A-B.pl]] to generate a playlist:
    5585    [[BR]]
    5686{{{
    57 $ gen-playlist-shuffled.pl /path/to/REF /path/to/DUT order.txt > out.plst
     87$ gen-playlist-Ref-A-B.pl /path/to/orig /path/to/dut1 /path/to/dut2 training.txt > out.plst
     88$ gen-playlist-Ref-A-B.pl /path/to/orig /path/to/dut1 /path/to/dut2 order.txt >> out.plst
    5889}}}
    59     NB, the path given to REF and DUT should be fully qualified.
     90    NB, the directory paths should be fully qualified.
    6091 1. For a subjective test with multiple viewing sessions, it is advisable to generate
    6192    multiple copies of the playlist, one for each viewing session.  Each viewing session
    … …  
    6596{{{
    6697$ for N in 1 2 3 4 5 6; do
    67      gen-playlist-shuffled.pl /path/to/REF /path/to/DUT order.txt > session$N.plst
     98     gen-playlist-Ref-A-B.pl /path/to/orig /path/to/dut1 /path/to/dut2 training.txt > session$N.plst
     99     gen-playlist-Ref-A-B.pl /path/to/orig /path/to/dut1 /path/to/dut2 order.txt >> session$N.plst
    68100  done
    69101}}}
    … …  
    81113
    82114== Notes on test configuration ==
    83  - The [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-shuffled.pl|gen-playlist-shuffled.pl]] script
    84    will produce an A-B-A-B stimulus, with A and B randomized for each test point.
    85    [[BR]]
    86    eg, for BQMall @ qp22, it might be REF-DUT-REF-DUT, but for qp27 it might be DUT-REF-DUT-REF.
     115 - The [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-Ref-A-B.pl|gen-playlist-Ref-A-B.pl]] script
     116   will produce a playlist, with test points in the form "Reference-A-B-Vote".
     117    - A and B are randomized for each test point.
     118    - The ordering of test points is randomized.
     119    - Each test point is shown twice.  Once in the first half of the test, once in the second.
     120 - [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-A-B-A-B.pl|gen-playlist-A-B-A-B.pl]] script
     121   with two directories will produce a playlist, with test points in the form "A-B-A-B-Vote"
     122    - A and B are randomized for each test point.
     123    - The ordering of test points is randomized.
     124 - [[http://ftp.kw.bbc.co.uk/davidf/jctvc/gen-playlist-A-B-A-B.pl|gen-playlist-A-B-A-B.pl]] script
     125   with three or more directories will produce a playlist, with test points in the form "A-A-B-B-C-C-Vote"
     126    - A, B and C are randomized for each test point.
     127    - The ordering of test points is randomized.
    87128
    88  - If the test involves multiple proposals, an A-B-C-D-E-..Z-A-B-C-D-E-..Z stimulus can be generated
    89    by providing multiple directories.  The stimulus order for each test point is randomized.
    90    Eg, {{{ gen-playlist-shuffled.pl REF DUT1 DUT2 DUT3 DUT4 order.txt }}}
    91    [[BR]]
    92    //DF: this seems kind of pointless, surely it should be A-A-B-B-C-C-D-D-E-E-...Z-Z, which is what we've done before//
     129The scripts provide a means for customizing the style of test generated.  See {{{%options}}}.
    93130
    94131== Notes on black & white level ==