123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206 |
- \input texinfo @c -*- texinfo -*-
- @documentencoding UTF-8
- @settitle FFmpeg Automated Testing Environment
- @titlepage
- @center @titlefont{FFmpeg Automated Testing Environment}
- @end titlepage
- @node Top
- @top
- @contents
- @chapter Introduction
- FATE is an extended regression suite on the client-side and a means
- for results aggregation and presentation on the server-side.
- The first part of this document explains how you can use FATE from
- your FFmpeg source directory to test your ffmpeg binary. The second
- part describes how you can run FATE to submit the results to FFmpeg's
- FATE server.
- In any way you can have a look at the publicly viewable FATE results
- by visiting this website:
- @url{http://fate.ffmpeg.org/}
- This is especially recommended for all people contributing source
- code to FFmpeg, as it can be seen if some test on some platform broke
- with their recent contribution. This usually happens on the platforms
- the developers could not test on.
- The second part of this document describes how you can run FATE to
- submit your results to FFmpeg's FATE server. If you want to submit your
- results be sure to check that your combination of CPU, OS and compiler
- is not already listed on the above mentioned website.
- In the third part you can find a comprehensive listing of FATE makefile
- targets and variables.
- @chapter Using FATE from your FFmpeg source directory
- If you want to run FATE on your machine you need to have the samples
- in place. You can get the samples via the build target fate-rsync.
- Use this command from the top-level source directory:
- @example
- make fate-rsync SAMPLES=fate-suite/
- make fate SAMPLES=fate-suite/
- @end example
- The above commands set the samples location by passing a makefile
- variable via command line. It is also possible to set the samples
- location at source configuration time by invoking configure with
- @option{--samples=<path to the samples directory>}. Afterwards you can
- invoke the makefile targets without setting the @var{SAMPLES} makefile
- variable. This is illustrated by the following commands:
- @example
- ./configure --samples=fate-suite/
- make fate-rsync
- make fate
- @end example
- Yet another way to tell FATE about the location of the sample
- directory is by making sure the environment variable FATE_SAMPLES
- contains the path to your samples directory. This can be achieved
- by e.g. putting that variable in your shell profile or by setting
- it in your interactive session.
- @example
- FATE_SAMPLES=fate-suite/ make fate
- @end example
- @float NOTE
- Do not put a '~' character in the samples path to indicate a home
- directory. Because of shell nuances, this will cause FATE to fail.
- @end float
- To use a custom wrapper to run the test, pass @option{--target-exec} to
- @command{configure} or set the @var{TARGET_EXEC} Make variable.
- @chapter Submitting the results to the FFmpeg result aggregation server
- To submit your results to the server you should run fate through the
- shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs
- to be invoked with a configuration file as its first argument.
- @example
- tests/fate.sh /path/to/fate_config
- @end example
- A configuration file template with comments describing the individual
- configuration variables can be found at @file{doc/fate_config.sh.template}.
- @ifhtml
- The mentioned configuration template is also available here:
- @verbatiminclude fate_config.sh.template
- @end ifhtml
- Create a configuration that suits your needs, based on the configuration
- template. The @env{slot} configuration variable can be any string that is not
- yet used, but it is suggested that you name it adhering to the following
- pattern @samp{@var{arch}-@var{os}-@var{compiler}-@var{compiler version}}. The
- configuration file itself will be sourced in a shell script, therefore all
- shell features may be used. This enables you to setup the environment as you
- need it for your build.
- For your first test runs the @env{fate_recv} variable should be empty or
- commented out. This will run everything as normal except that it will omit
- the submission of the results to the server. The following files should be
- present in $workdir as specified in the configuration file:
- @itemize
- @item configure.log
- @item compile.log
- @item test.log
- @item report
- @item version
- @end itemize
- When you have everything working properly you can create an SSH key pair
- and send the public key to the FATE server administrator who can be contacted
- at the email address @email{fate-admin@@ffmpeg.org}.
- Configure your SSH client to use public key authentication with that key
- when connecting to the FATE server. Also do not forget to check the identity
- of the server and to accept its host key. This can usually be achieved by
- running your SSH client manually and killing it after you accepted the key.
- The FATE server's fingerprint is:
- @table @samp
- @item RSA
- d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51
- @item ECDSA
- 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86
- @end table
- If you have problems connecting to the FATE server, it may help to try out
- the @command{ssh} command with one or more @option{-v} options. You should
- get detailed output concerning your SSH configuration and the authentication
- process.
- The only thing left is to automate the execution of the fate.sh script and
- the synchronisation of the samples directory.
- @chapter FATE makefile targets and variables
- @section Makefile targets
- @table @option
- @item fate-rsync
- Download/synchronize sample files to the configured samples directory.
- @item fate-list
- Will list all fate/regression test targets.
- @item fate
- Run the FATE test suite (requires the fate-suite dataset).
- @end table
- @section Makefile variables
- @table @env
- @item V
- Verbosity level, can be set to 0, 1 or 2.
- @itemize
- @item 0: show just the test arguments
- @item 1: show just the command used in the test
- @item 2: show everything
- @end itemize
- @item SAMPLES
- Specify or override the path to the FATE samples at make time, it has a
- meaning only while running the regression tests.
- @item THREADS
- Specify how many threads to use while running regression tests, it is
- quite useful to detect thread-related regressions.
- @item THREAD_TYPE
- Specify which threading strategy test, either @samp{slice} or @samp{frame},
- by default @samp{slice+frame}
- @item CPUFLAGS
- Specify CPU flags.
- @item TARGET_EXEC
- Specify or override the wrapper used to run the tests.
- The @env{TARGET_EXEC} option provides a way to run FATE wrapped in
- @command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets
- through @command{ssh}.
- @item GEN
- Set to @samp{1} to generate the missing or mismatched references.
- @end table
- @section Examples
- @example
- make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
- @end example
|