123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139 |
- FATE Automated Testing Environment
- ==================================
- FATE is an extended regression suite on the client-side and a means
- for results aggregation and presentation on the server-side.
- The first part of this document explains how you can use FATE from
- your FFmpeg source directory to test your ffmpeg binary. The second
- part describes how you can run FATE to submit the results to FFmpeg's
- FATE server.
- In any way you can have a look at the publicly viewable FATE results
- by visiting this website:
- http://fate.ffmpeg.org/
- This is especially recommended for all people contributing source
- code to FFmpeg, as it can be seen if some test on some platform broke
- with there recent contribution. This usually happens on the platforms
- the developers could not test on.
- The second part of this document describes how you can run FATE to
- submit your results to FFmpeg's FATE server. If you want to submit your
- results be sure to check that your combination of CPU, OS and compiler
- is not already listed on the above mentioned website.
- In the third part you can find a comprehensive listing of FATE makefile
- targets and variables.
- 1. Using FATE from your FFmpeg source directory
- -----------------------------------------------
- If you want to run FATE on your machine you need to have the samples
- in place. You can get the samples via the build target fate-rsync.
- Use this command from the top-level source directory:
- # make fate-rsync SAMPLES=fate-suite/
- # make fate SAMPLES=fate-suite/
- The above commands set the samples location by passing a makefile
- variable via command line. It is also possible to set the samples
- location at source configuration time by invoking configure with
- `--samples=<path to the samples directory>'. Afterwards you can
- invoke the makefile targets without setting the SAMPLES makefile
- variable. This is illustrated by the following commands:
- # ./configure --samples=fate-suite/
- # make fate-rsync
- # make fate
- Yet another way to tell FATE about the location of the sample
- directory is by making sure the environment variable FATE_SAMPLES
- contains the path to your samples directory. This can be achieved
- by e.g. putting that variable in your shell profile or by setting
- it in your interactive session.
- # FATE_SAMPLES=fate-suite/ make fate
- NOTE:
- Do not put a '~' character in the samples path to indicate a home
- directory. Because of shell nuances, this will cause FATE to fail.
- 2. Submitting the results to the FFmpeg result aggregation server
- -----------------------------------------------------------------
- To submit your results to the server you should run fate through the
- shell script tests/fate.sh from the FFmpeg sources. This script needs
- to be invoked with a configuration file as its first argument.
- # tests/fate.sh /path/to/fate_config
- A configuration file template with comments describing the individual
- configuration variables can be found at tests/fate_config.sh.template .
- Create a configuration that suits your needs, based on the configuration
- template. The `slot' configuration variable can be any string that is not
- yet used, but it is suggested that you name it adhering to the following
- pattern <arch>-<os>-<compiler>-<compiler version>. The configuration file
- itself will be sourced in a shell script, therefore all shell features may
- be used. This enables you to setup the environment as you need it for your
- build.
- For your first test runs the `fate_recv' variable should be empty or
- commented out. This will run everything as normal except that it will omit
- the submission of the results to the server. The following files should be
- present in $workdir as specified in the configuration file:
- - configure.log
- - compile.log
- - test.log
- - report
- - version
- When you have everything working properly you can create an SSH key and
- send its public part to the FATE server administrator.
- Configure your SSH client to use public key authentication with that key
- when connecting to the FATE server. Also do not forget to check the identity
- of the server and to accept its host key. This can usually be achieved by
- running your SSH client manually and killing it after you accepted the key.
- The FATE server's fingerprint is:
- b1:31:c8:79:3f:04:1d:f8:f2:23:26:5a:fd:55:fa:92
- The only thing left is to automate the execution of the fate.sh script and
- the synchronisation of the samples directory.
- 3. FATE makefile targets and variables
- --------------------------------------
- FATE Makefile targets:
- fate-list
- Will list all fate/regression test targets.
- fate
- Run the FATE test suite (requires the fate-suite dataset).
- FATE Makefile variables:
- V
- Verbosity level, can be set to 0, 1 or 2.
- * 0: show just the test arguments
- * 1: show just the command used in the test
- * 2: show everything
- SAMPLES
- Specify or override the path to the FATE samples at make time, it has a
- meaning only while running the regression tests.
- THREADS
- Specify how many threads to use while running regression tests, it is
- quite useful to detect thread-related regressions.
- Example:
- make V=1 SAMPLES=/var/fate/samples THREADS=2 fate
|