fate.texi 7.9 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238
  1. \input texinfo @c -*- texinfo -*-
  2. @documentencoding UTF-8
  3. @settitle FFmpeg Automated Testing Environment
  4. @titlepage
  5. @center @titlefont{FFmpeg Automated Testing Environment}
  6. @end titlepage
  7. @node Top
  8. @top
  9. @contents
  10. @chapter Introduction
  11. FATE is an extended regression suite on the client-side and a means
  12. for results aggregation and presentation on the server-side.
  13. The first part of this document explains how you can use FATE from
  14. your FFmpeg source directory to test your ffmpeg binary. The second
  15. part describes how you can run FATE to submit the results to FFmpeg's
  16. FATE server.
  17. In any way you can have a look at the publicly viewable FATE results
  18. by visiting this website:
  19. @url{http://fate.ffmpeg.org/}
  20. This is especially recommended for all people contributing source
  21. code to FFmpeg, as it can be seen if some test on some platform broke
  22. with their recent contribution. This usually happens on the platforms
  23. the developers could not test on.
  24. The second part of this document describes how you can run FATE to
  25. submit your results to FFmpeg's FATE server. If you want to submit your
  26. results be sure to check that your combination of CPU, OS and compiler
  27. is not already listed on the above mentioned website.
  28. In the third part you can find a comprehensive listing of FATE makefile
  29. targets and variables.
  30. @chapter Using FATE from your FFmpeg source directory
  31. If you want to run FATE on your machine you need to have the samples
  32. in place. You can get the samples via the build target fate-rsync.
  33. Use this command from the top-level source directory:
  34. @example
  35. make fate-rsync SAMPLES=fate-suite/
  36. make fate SAMPLES=fate-suite/
  37. @end example
  38. The above commands set the samples location by passing a makefile
  39. variable via command line. It is also possible to set the samples
  40. location at source configuration time by invoking configure with
  41. @option{--samples=<path to the samples directory>}. Afterwards you can
  42. invoke the makefile targets without setting the @var{SAMPLES} makefile
  43. variable. This is illustrated by the following commands:
  44. @example
  45. ./configure --samples=fate-suite/
  46. make fate-rsync
  47. make fate
  48. @end example
  49. Yet another way to tell FATE about the location of the sample
  50. directory is by making sure the environment variable FATE_SAMPLES
  51. contains the path to your samples directory. This can be achieved
  52. by e.g. putting that variable in your shell profile or by setting
  53. it in your interactive session.
  54. @example
  55. FATE_SAMPLES=fate-suite/ make fate
  56. @end example
  57. @float NOTE
  58. Do not put a '~' character in the samples path to indicate a home
  59. directory. Because of shell nuances, this will cause FATE to fail.
  60. @end float
  61. To use a custom wrapper to run the test, pass @option{--target-exec} to
  62. @command{configure} or set the @var{TARGET_EXEC} Make variable.
  63. @chapter Submitting the results to the FFmpeg result aggregation server
  64. To submit your results to the server you should run fate through the
  65. shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs
  66. to be invoked with a configuration file as its first argument.
  67. @example
  68. tests/fate.sh /path/to/fate_config
  69. @end example
  70. A configuration file template with comments describing the individual
  71. configuration variables can be found at @file{doc/fate_config.sh.template}.
  72. @ifhtml
  73. The mentioned configuration template is also available here:
  74. @verbatiminclude fate_config.sh.template
  75. @end ifhtml
  76. Create a configuration that suits your needs, based on the configuration
  77. template. The @env{slot} configuration variable can be any string that is not
  78. yet used, but it is suggested that you name it adhering to the following
  79. pattern @samp{@var{arch}-@var{os}-@var{compiler}-@var{compiler version}}. The
  80. configuration file itself will be sourced in a shell script, therefore all
  81. shell features may be used. This enables you to setup the environment as you
  82. need it for your build.
  83. For your first test runs the @env{fate_recv} variable should be empty or
  84. commented out. This will run everything as normal except that it will omit
  85. the submission of the results to the server. The following files should be
  86. present in $workdir as specified in the configuration file:
  87. @itemize
  88. @item configure.log
  89. @item compile.log
  90. @item test.log
  91. @item report
  92. @item version
  93. @end itemize
  94. When you have everything working properly you can create an SSH key pair
  95. and send the public key to the FATE server administrator who can be contacted
  96. at the email address @email{fate-admin@@ffmpeg.org}.
  97. Configure your SSH client to use public key authentication with that key
  98. when connecting to the FATE server. Also do not forget to check the identity
  99. of the server and to accept its host key. This can usually be achieved by
  100. running your SSH client manually and killing it after you accepted the key.
  101. The FATE server's fingerprint is:
  102. @table @samp
  103. @item RSA
  104. d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51
  105. @item ECDSA
  106. 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86
  107. @end table
  108. If you have problems connecting to the FATE server, it may help to try out
  109. the @command{ssh} command with one or more @option{-v} options. You should
  110. get detailed output concerning your SSH configuration and the authentication
  111. process.
  112. The only thing left is to automate the execution of the fate.sh script and
  113. the synchronisation of the samples directory.
  114. @chapter Uploading new samples to the fate suite
  115. This is for developers who have an account on the fate suite server.
  116. If you upload new samples, please make sure they are as small as possible,
  117. space on each client, network bandwidth and so on benefit from smaller test cases.
  118. Also keep in mind older checkouts use existing sample files, that means in
  119. practice generally do not replace, remove or overwrite files as it likely would
  120. break older checkouts or releases.
  121. Also all needed samples for a commit should be uploaded, ideally 24
  122. hours, before the push.
  123. @example
  124. #First update your local samples copy:
  125. rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite
  126. #Then do a dry run checking what would be uploaded:
  127. rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite
  128. #Upload the files:
  129. rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite
  130. @end example
  131. @chapter FATE makefile targets and variables
  132. @section Makefile targets
  133. @table @option
  134. @item fate-rsync
  135. Download/synchronize sample files to the configured samples directory.
  136. @item fate-list
  137. Will list all fate/regression test targets.
  138. @item fate
  139. Run the FATE test suite (requires the fate-suite dataset).
  140. @end table
  141. @section Makefile variables
  142. @table @env
  143. @item V
  144. Verbosity level, can be set to 0, 1 or 2.
  145. @itemize
  146. @item 0: show just the test arguments
  147. @item 1: show just the command used in the test
  148. @item 2: show everything
  149. @end itemize
  150. @item SAMPLES
  151. Specify or override the path to the FATE samples at make time, it has a
  152. meaning only while running the regression tests.
  153. @item THREADS
  154. Specify how many threads to use while running regression tests, it is
  155. quite useful to detect thread-related regressions.
  156. @item THREAD_TYPE
  157. Specify which threading strategy test, either @samp{slice} or @samp{frame},
  158. by default @samp{slice+frame}
  159. @item CPUFLAGS
  160. Specify CPU flags.
  161. @item TARGET_EXEC
  162. Specify or override the wrapper used to run the tests.
  163. The @env{TARGET_EXEC} option provides a way to run FATE wrapped in
  164. @command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets
  165. through @command{ssh}.
  166. @item GEN
  167. Set to @samp{1} to generate the missing or mismatched references.
  168. @item HWACCEL
  169. Specify which hardware acceleration to use while running regression tests,
  170. by default @samp{none} is used.
  171. @item KEEP
  172. Set to @samp{1} to keep temp files generated by fate test(s) when test is successful.
  173. Default is @samp{0}, which removes these files. Files are always kept when a test
  174. fails.
  175. @end table
  176. @section Examples
  177. @example
  178. make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
  179. @end example