CHANGELOG 40 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721
  1. v1.5.2 (Jan, 2022)
  2. perf: Regain Minimal memset()-ing During Reuse of Compression Contexts (@Cyan4973, #2969)
  3. build: Build Zstd with `noexecstack` on All Architectures (@felixhandte, #2964)
  4. doc: Clarify Licensing (@terrelln, #2981)
  5. v1.5.1 (Dec, 2021)
  6. perf: rebalanced compression levels, to better match the intended speed/level curve, by @senhuang42
  7. perf: faster huffman decoder, using x64 assembly, by @terrelln
  8. perf: slightly faster high speed modes (strategies fast & dfast), by @felixhandte
  9. perf: improved binary size and faster compilation times, by @terrelln
  10. perf: new row64 mode, used notably in level 12, by @senhuang42
  11. perf: faster mid-level compression speed in presence of highly repetitive patterns, by @senhuang42
  12. perf: minor compression ratio improvements for small data at high levels, by @cyan4973
  13. perf: reduced stack usage (mostly useful for Linux Kernel), by @terrelln
  14. perf: faster compression speed on incompressible data, by @bindhvo
  15. perf: on-demand reduced ZSTD_DCtx state size, using build macro ZSTD_DECODER_INTERNAL_BUFFER, at a small cost of performance, by @bindhvo
  16. build: allows hiding static symbols in the dynamic library, using build macro, by @skitt
  17. build: support for m68k (Motorola 68000's), by @cyan4973
  18. build: improved AIX support, by @Helflym
  19. build: improved meson unofficial build, by @eli-schwartz
  20. cli : custom memory limit when training dictionary (#2925), by @embg
  21. cli : report advanced parameters information when compressing in very verbose mode (``-vv`), by @Svetlitski-FB
  22. v1.5.0 (May 11, 2021)
  23. api: Various functions promoted from experimental to stable API: (#2579-2581, @senhuang42)
  24. `ZSTD_defaultCLevel()`
  25. `ZSTD_getDictID_fromCDict()`
  26. api: Several experimental functions have been deprecated and will emit a compiler warning (#2582, @senhuang42)
  27. `ZSTD_compress_advanced()`
  28. `ZSTD_compress_usingCDict_advanced()`
  29. `ZSTD_compressBegin_advanced()`
  30. `ZSTD_compressBegin_usingCDict_advanced()`
  31. `ZSTD_initCStream_srcSize()`
  32. `ZSTD_initCStream_usingDict()`
  33. `ZSTD_initCStream_usingCDict()`
  34. `ZSTD_initCStream_advanced()`
  35. `ZSTD_initCStream_usingCDict_advanced()`
  36. `ZSTD_resetCStream()`
  37. api: ZSTDMT_NBWORKERS_MAX reduced to 64 for 32-bit environments (@Cyan4973)
  38. perf: Significant speed improvements for middle compression levels (#2494, @senhuang42 @terrelln)
  39. perf: Block splitter to improve compression ratio, enabled by default for high compression levels (#2447, @senhuang42)
  40. perf: Decompression loop refactor, speed improvements on `clang` and for `--long` modes (#2614 #2630, @Cyan4973)
  41. perf: Reduced stack usage during compression and decompression entropy stage (#2522 #2524, @terrelln)
  42. bug: Improve setting permissions of created files (#2525, @felixhandte)
  43. bug: Fix large dictionary non-determinism (#2607, @terrelln)
  44. bug: Fix non-determinism test failures on Linux i686 (#2606, @terrelln)
  45. bug: Fix various dedicated dictionary search bugs (#2540 #2586, @senhuang42 @felixhandte)
  46. bug: Ensure `ZSTD_estimateCCtxSize*() `monotonically increases with compression level (#2538, @senhuang42)
  47. bug: Fix --patch-from mode parameter bound bug with small files (#2637, @occivink)
  48. bug: Fix UBSAN error in decompression (#2625, @terrelln)
  49. bug: Fix superblock compression divide by zero bug (#2592, @senhuang42)
  50. bug: Make the number of physical CPU cores detection more robust (#2517, @PaulBone)
  51. doc: Improve `zdict.h` dictionary training API documentation (#2622, @terrelln)
  52. doc: Note that public `ZSTD_free*()` functions accept NULL pointers (#2521, @animalize)
  53. doc: Add style guide docs for open source contributors (#2626, @Cyan4973)
  54. tests: Better regression test coverage for different dictionary modes (#2559, @senhuang42)
  55. tests: Better test coverage of index reduction (#2603, @terrelln)
  56. tests: OSS-Fuzz coverage for seekable format (#2617, @senhuang42)
  57. tests: Test coverage for ZSTD threadpool API (#2604, @senhuang42)
  58. build: Dynamic library built multithreaded by default (#2584, @senhuang42)
  59. build: Move `zstd_errors.h` and `zdict.h` to `lib/` root (#2597, @terrelln)
  60. build: Allow `ZSTDMT_JOBSIZE_MIN` to be configured at compile-time, reduce default to 512KB (#2611, @Cyan4973)
  61. build: Single file library build script moved to `build/` directory (#2618, @felixhandte)
  62. build: `ZBUFF_*()` is no longer built by default (#2583, @senhuang42)
  63. build: Fixed Meson build (#2548, @SupervisedThinking @kloczek)
  64. build: Fix excessive compiler warnings with clang-cl and CMake (#2600, @nickhutchinson)
  65. build: Detect presence of `md5` on Darwin (#2609, @felixhandte)
  66. build: Avoid SIGBUS on armv6 (#2633, @bmwiedmann)
  67. cli: `--progress` flag added to always display progress bar (#2595, @senhuang42)
  68. cli: Allow reading from block devices with `--force` (#2613, @felixhandte)
  69. cli: Fix CLI filesize display bug (#2550, @Cyan4973)
  70. cli: Fix windows CLI `--filelist` end-of-line bug (#2620, @Cyan4973)
  71. contrib: Various fixes for linux kernel patch (#2539, @terrelln)
  72. contrib: Seekable format - Decompression hanging edge case fix (#2516, @senhuang42)
  73. contrib: Seekable format - New seek table-only API (#2113 #2518, @mdittmer @Cyan4973)
  74. contrib: Seekable format - Fix seek table descriptor check when loading (#2534, @foxeng)
  75. contrib: Seekable format - Decompression fix for large offsets, (#2594, @azat)
  76. misc: Automatically published release tarballs available on Github (#2535, @felixhandte)
  77. v1.4.9 (Mar 1, 2021)
  78. bug: Use `umask()` to Constrain Created File Permissions (#2495, @felixhandte)
  79. bug: Make Simple Single-Pass Functions Ignore Advanced Parameters (#2498, @terrelln)
  80. api: Add (De)Compression Tracing Functionality (#2482, @terrelln)
  81. api: Support References to Multiple DDicts (#2446, @senhuang42)
  82. api: Add Function to Generate Skippable Frame (#2439, @senhuang42)
  83. perf: New Algorithms for the Long Distance Matcher (#2483, @mpu)
  84. perf: Performance Improvements for Long Distance Matcher (#2464, @mpu)
  85. perf: Don't Shrink Window Log when Streaming with a Dictionary (#2451, @terrelln)
  86. cli: Fix `--output-dir-mirror`'s Rejection of `..`-Containing Paths (#2512, @felixhandte)
  87. cli: Allow Input From Console When `-f`/`--force` is Passed (#2466, @felixhandte)
  88. cli: Improve Help Message (#2500, @senhuang42)
  89. tests: Remove Flaky Tests (#2455, #2486, #2445, @Cyan4973)
  90. tests: Correctly Invoke md5 Utility on NetBSD (#2492, @niacat)
  91. tests: Avoid Using `stat -c` on NetBSD (#2513, @felixhandte)
  92. build: Zstd CLI Can Now be Linked to Dynamic `libzstd` (#2457, #2454 @Cyan4973)
  93. build: Hide and Avoid Using Static-Only Symbols (#2501, #2504, @skitt)
  94. build: CMake: Enable Only C for lib/ and programs/ Projects (#2498, @concatime)
  95. build: CMake: Use `configure_file()` to Create the `.pc` File (#2462, @lazka)
  96. build: Fix Fuzzer Compiler Detection & Update UBSAN Flags (#2503, @terrelln)
  97. build: Add Guards for `_LARGEFILE_SOURCE` and `_LARGEFILE64_SOURCE` (#2444, @indygreg)
  98. build: Improve `zlibwrapper` Makefile (#2437, @Cyan4973)
  99. contrib: Add `recover_directory` Program (#2473, @terrelln)
  100. doc: Change License Year to 2021 (#2452 & #2465, @terrelln & @senhuang42)
  101. doc: Fix Typos (#2459, @ThomasWaldmann)
  102. v1.4.8 (Dec 18, 2020)
  103. hotfix: wrong alignment of an internal buffer
  104. v1.4.7 (Dec 16, 2020)
  105. perf: stronger --long mode at high compression levels, by @senhuang42
  106. perf: stronger --patch-from at high compression levels, thanks to --long improvements
  107. perf: faster dictionary compression at medium compression levels, by @felixhandte
  108. perf: small speed & memory usage improvements for ZSTD_compress2(), by @terrelln
  109. perf: improved fast compression speeds with Visual Studio, by @animalize
  110. cli : Set nb of threads with environment variable ZSTD_NBTHREADS, by @senhuang42
  111. cli : accept decompressing files with *.zstd suffix
  112. cli : provide a condensed summary by default when processing multiple files
  113. cli : fix : stdin input no longer confused as user prompt
  114. cli : improve accuracy of several error messages
  115. api : new sequence ingestion API, by @senhuang42
  116. api : shared thread pool: control total nb of threads used by multiple compression jobs, by @marxin
  117. api : new ZSTD_getDictID_fromCDict(), by @LuAPi
  118. api : zlibWrapper only uses public API, and is compatible with dynamic library, by @terrelln
  119. api : fix : multithreaded compression has predictable output even in special cases (see #2327) (issue not accessible from cli)
  120. api : fix : dictionary compression correctly respects dictionary compression level (see #2303) (issue not accessible from cli)
  121. build: fix cmake script when using path with spaces, by @terrelln
  122. build: improved compile-time detection of aarch64/neon platforms, by @bsdimp
  123. build: Fix building on AIX 5.1, by @likema
  124. build: compile paramgrill with cmake on Windows, requested by @mirh
  125. doc : clarify repcode updates in format specification, by @felixhandte
  126. v1.4.6
  127. fix : Always return dstSize_tooSmall when that is the case
  128. fix : Fix ZSTD_initCStream_advanced() with static allocation and no dictionary
  129. perf: Improve small block decompression speed by 20%+, by @terrelln
  130. perf: Reduce compression stack usage by 1 KB, by @terrelln
  131. perf: Improve decompression speed by improving ZSTD_wildcopy, by @helloguo (#2252, #2256)
  132. perf: Improve histogram construction, by @cyan4973 (#2253)
  133. cli : Add --output-dir-mirror option, by @xxie24 (#2219)
  134. cli : Warn when (de)compressing multiple files into a single output, by @senhuang42 (#2279)
  135. cli : Improved progress bar and status summary when (de)compressing multiple files, by @senhuang42 (#2283)
  136. cli : Call stat less often, by @felixhandte (#2262)
  137. cli : Allow --patch-from XXX and --filelist XXX in addition to --patch-from=XXX and --filelist=XXX, by @cyan4973 (#2250)
  138. cli : Allow --patch-from to compress stdin with --stream-size, by @bimbashrestha (#2206)
  139. api : Do not install zbuff.h, since it has long been deprecated, by @cyan4973 (#2166).
  140. api : Fix ZSTD_CCtx_setParameter() with ZSTD_c_compressionLevel to make 0 mean default level, by @i-do-cpp (#2291)
  141. api : Rename ZSTDMT_NBTHREADS_MAX to ZSTDMT_NBWORKERS_MAX, by @marxin (#2228).
  142. build: Install pkg-config file with CMake and MinGW, by @tonytheodore (#2183)
  143. build: Install DLL with CMake on Windows, by @BioDataAnalysis (#2221)
  144. build: Fix DLL install location with CMake, by @xantares and @bimbashrestha (#2186)
  145. build: Add ZSTD_NO_UNUSED_FUNCTIONS macro to hide unused functions
  146. build: Add ZSTD_NO_INTRINSICS macro to avoid explicit intrinsics
  147. build: Add STATIC_BMI2 macro for compile time detection of BMI2 on MSVC, by @Niadb (#2258)
  148. build: Fix -Wcomma warnings, by @cwoffenden
  149. build: Remove distutils requirement for meson build, by @neheb (#2197)
  150. build: Fix cli compilation with uclibc
  151. build: Fix cli compilation without st_mtime, by @ffontaine (#2246)
  152. build: Fix shadowing warnings in library
  153. build: Fix single file library compilation with Enscripten, by @yoshihitoh (#2227)
  154. misc: Improve single file library and include dictBuilder, by @cwoffenden
  155. misc: Allow compression dictionaries with missing symbols
  156. misc: Add freestanding translation script in contrib/freestanding_lib
  157. misc: Collect all of zstd's libc dependencies into zstd_deps.h
  158. doc : Add ZSTD_versionString() to manual, by @animalize
  159. doc : Fix documentation for ZSTD_CCtxParams_setParameter(), by @felixhandte (#2270)
  160. v1.4.5 (May 22, 2020)
  161. fix : Compression ratio regression on huge files (> 3 GB) using high levels (--ultra) and multithreading, by @terrelln
  162. perf: Improved decompression speed: x64 : +10% (clang) / +5% (gcc); ARM : from +15% to +50%, depending on SoC, by @terrelln
  163. perf: Automatically downsizes ZSTD_DCtx when too large for too long (#2069, by @bimbashreshta)
  164. perf: Improved fast compression speed on aarch64 (#2040, ~+3%, by @caoyzh)
  165. perf: Small level 1 compression speed gains (depending on compiler)
  166. cli : New --patch-from command, create and apply patches from files, by @bimbashreshta
  167. cli : New --filelist= : Provide a list of files to operate upon from a file
  168. cli : -b -d command can now benchmark decompression on multiple files
  169. cli : New --no-content-size command
  170. cli : New --show-default-cparams information command
  171. api : ZDICT_finalizeDictionary() is promoted to stable (#2111)
  172. api : new experimental parameter ZSTD_d_stableOutBuffer (#2094)
  173. build: Generate a single-file libzstd library (#2065, by @cwoffenden)
  174. build: Relative includes no longer require -I compiler flags for zstd lib subdirs (#2103, by @felixhandte)
  175. build: zstd now compiles cleanly under -pedantic (#2099)
  176. build: zstd now compiles with make-4.3
  177. build: Support mingw cross-compilation from Linux, by @Ericson2314
  178. build: Meson multi-thread build fix on windows
  179. build: Some misc icc fixes backed by new ci test on travis
  180. misc: bitflip analyzer tool, by @felixhandte
  181. misc: Extend largeNbDicts benchmark to compression
  182. misc: Edit-distance match finder in contrib/
  183. doc : Improved beginner CONTRIBUTING.md docs
  184. doc : New issue templates for zstd
  185. v1.4.4 (Nov 6, 2019)
  186. perf: Improved decompression speed, by > 10%, by @terrelln
  187. perf: Better compression speed when re-using a context, by @felixhandte
  188. perf: Fix compression ratio when compressing large files with small dictionary, by @senhuang42
  189. perf: zstd reference encoder can generate RLE blocks, by @bimbashrestha
  190. perf: minor generic speed optimization, by @davidbolvansky
  191. api: new ability to extract sequences from the parser for analysis, by @bimbashrestha
  192. api: fixed decoding of magic-less frames, by @terrelln
  193. api: fixed ZSTD_initCStream_advanced() performance with fast modes, reported by @QrczakMK
  194. cli: Named pipes support, by @bimbashrestha
  195. cli: short tar's extension support, by @stokito
  196. cli: command --output-dir-flat= , generates target files into requested directory, by @senhuang42
  197. cli: commands --stream-size=# and --size-hint=#, by @nmagerko
  198. cli: command --exclude-compressed, by @shashank0791
  199. cli: faster `-t` test mode
  200. cli: improved some error messages, by @vangyzen
  201. cli: fix command `-D dictionary` on Windows, reported by @artyompetrov
  202. cli: fix rare deadlock condition within dictionary builder, by @terrelln
  203. build: single-file decoder with emscripten compilation script, by @cwoffenden
  204. build: fixed zlibWrapper compilation on Visual Studio, reported by @bluenlive
  205. build: fixed deprecation warning for certain gcc version, reported by @jasonma163
  206. build: fix compilation on old gcc versions, by @cemeyer
  207. build: improved installation directories for cmake script, by Dmitri Shubin
  208. pack: modified pkgconfig, for better integration into openwrt, requested by @neheb
  209. misc: Improved documentation : ZSTD_CLEVEL, DYNAMIC_BMI2, ZSTD_CDict, function deprecation, zstd format
  210. misc: fixed educational decoder : accept larger literals section, and removed UNALIGNED() macro
  211. v1.4.3 (Aug 20, 2019)
  212. bug: Fix Dictionary Compression Ratio Regression by @cyan4973 (#1709)
  213. bug: Fix Buffer Overflow in legacy v0.3 decompression by @felixhandte (#1722)
  214. build: Add support for IAR C/C++ Compiler for Arm by @joseph0918 (#1705)
  215. v1.4.2 (Jul 26, 2019)
  216. bug: Fix bug in zstd-0.5 decoder by @terrelln (#1696)
  217. bug: Fix seekable decompression in-memory API by @iburinoc (#1695)
  218. misc: Validate blocks are smaller than size limit by @vivekmg (#1685)
  219. misc: Restructure source files by @ephiepark (#1679)
  220. v1.4.1 (Jul 20, 2019)
  221. bug: Fix data corruption in niche use cases by @terrelln (#1659)
  222. bug: Fuzz legacy modes, fix uncovered bugs by @terrelln (#1593, #1594, #1595)
  223. bug: Fix out of bounds read by @terrelln (#1590)
  224. perf: Improve decode speed by ~7% @mgrice (#1668)
  225. perf: Slightly improved compression ratio of level 3 and 4 (ZSTD_dfast) by @cyan4973 (#1681)
  226. perf: Slightly faster compression speed when re-using a context by @cyan4973 (#1658)
  227. perf: Improve compression ratio for small windowLog by @cyan4973 (#1624)
  228. perf: Faster compression speed in high compression mode for repetitive data by @terrelln (#1635)
  229. api: Add parameter to generate smaller dictionaries by @tyler-tran (#1656)
  230. cli: Recognize symlinks when built in C99 mode by @felixhandte (#1640)
  231. cli: Expose cpu load indicator for each file on -vv mode by @ephiepark (#1631)
  232. cli: Restrict read permissions on destination files by @chungy (#1644)
  233. cli: zstdgrep: handle -f flag by @felixhandte (#1618)
  234. cli: zstdcat: follow symlinks by @vejnar (#1604)
  235. doc: Remove extra size limit on compressed blocks by @felixhandte (#1689)
  236. doc: Fix typo by @yk-tanigawa (#1633)
  237. doc: Improve documentation on streaming buffer sizes by @cyan4973 (#1629)
  238. build: CMake: support building with LZ4 @leeyoung624 (#1626)
  239. build: CMake: install zstdless and zstdgrep by @leeyoung624 (#1647)
  240. build: CMake: respect existing uninstall target by @j301scott (#1619)
  241. build: Make: skip multithread tests when built without support by @michaelforney (#1620)
  242. build: Make: Fix examples/ test target by @sjnam (#1603)
  243. build: Meson: rename options out of deprecated namespace by @lzutao (#1665)
  244. build: Meson: fix build by @lzutao (#1602)
  245. build: Visual Studio: don't export symbols in static lib by @scharan (#1650)
  246. build: Visual Studio: fix linking by @absotively (#1639)
  247. build: Fix MinGW-W64 build by @myzhang1029 (#1600)
  248. misc: Expand decodecorpus coverage by @ephiepark (#1664)
  249. v1.4.0 (Apr 17, 2019)
  250. perf: Improve level 1 compression speed in most scenarios by 6% by @gbtucker and @terrelln
  251. api: Move the advanced API, including all functions in the staging section, to the stable section
  252. api: Make ZSTD_e_flush and ZSTD_e_end block for maximum forward progress
  253. api: Rename ZSTD_CCtxParam_getParameter to ZSTD_CCtxParams_getParameter
  254. api: Rename ZSTD_CCtxParam_setParameter to ZSTD_CCtxParams_setParameter
  255. api: Don't export ZSTDMT functions from the shared library by default
  256. api: Require ZSTD_MULTITHREAD to be defined to use ZSTDMT
  257. api: Add ZSTD_decompressBound() to provide an upper bound on decompressed size by @shakeelrao
  258. api: Fix ZSTD_decompressDCtx() corner cases with a dictionary
  259. api: Move ZSTD_getDictID_*() functions to the stable section
  260. api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln
  261. api: Allow compression parameters to be set when a dictionary is used
  262. api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called
  263. api: Fix ZSTD_estimateCStreamSize_usingCCtxParams()
  264. api: Setting ZSTD_d_maxWindowLog to 0 means use the default
  265. cli: Ensure that a dictionary is not used to compress itself by @shakeelrao
  266. cli: Add --[no-]compress-literals flag to enable or disable literal compression
  267. doc: Update the examples to use the advanced API
  268. doc: Explain how to transition from old streaming functions to the advanced API in the header
  269. build: Improve the Windows release packages
  270. build: Improve CMake build by @hjmjohnson
  271. build: Build fixes for FreeBSD by @lwhsu
  272. build: Remove redundant warnings by @thatsafunnyname
  273. build: Fix tests on OpenBSD by @bket
  274. build: Extend fuzzer build system to work with the new clang engine
  275. build: CMake now creates the libzstd.so.1 symlink
  276. build: Improve Menson build by @lzutao
  277. misc: Fix symbolic link detection on FreeBSD
  278. misc: Use physical core count for -T0 on FreeBSD by @cemeyer
  279. misc: Fix zstd --list on truncated files by @kostmo
  280. misc: Improve logging in debug mode by @felixhandte
  281. misc: Add CirrusCI tests by @lwhsu
  282. misc: Optimize dictionary memory usage in corner cases
  283. misc: Improve the dictionary builder on small or homogeneous data
  284. misc: Fix spelling across the repo by @jsoref
  285. v1.3.8 (Dec 28, 2018)
  286. perf: better decompression speed on large files (+7%) and cold dictionaries (+15%)
  287. perf: slightly better compression ratio at high compression modes
  288. api : finalized advanced API, last stage before "stable" status
  289. api : new --rsyncable mode, by @terrelln
  290. api : support decompression of empty frames into NULL (used to be an error) (#1385)
  291. build: new set of macros to build a minimal size decoder, by @felixhandte
  292. build: fix compilation on MIPS32, reported by @clbr (#1441)
  293. build: fix compilation with multiple -arch flags, by @ryandesign
  294. build: highly upgraded meson build, by @lzutao
  295. build: improved buck support, by @obelisk
  296. build: fix cmake script : can create debug build, by @pitrou
  297. build: Makefile : grep works on both colored consoles and systems without color support
  298. build: fixed zstd-pgo, by @bmwiedemann
  299. cli : support ZSTD_CLEVEL environment variable, by @yijinfb (#1423)
  300. cli : --no-progress flag, preserving final summary (#1371), by @terrelln
  301. cli : ensure destination file is not source file (#1422)
  302. cli : clearer error messages, especially when input file not present
  303. doc : clarified zstd_compression_format.md, by @ulikunitz
  304. misc: fixed zstdgrep, returns 1 on failure, by @lzutao
  305. misc: NEWS renamed as CHANGELOG, in accordance with fboss
  306. v1.3.7 (Oct 20, 2018)
  307. perf: slightly better decompression speed on clang (depending on hardware target)
  308. fix : performance of dictionary compression for small input < 4 KB at levels 9 and 10
  309. build: no longer build backtrace by default in release mode; restrict further automatic mode
  310. build: control backtrace support through build macro BACKTRACE
  311. misc: added man pages for zstdless and zstdgrep, by @samrussell
  312. v1.3.6 (Oct 6, 2018)
  313. perf: much faster dictionary builder, by @jenniferliu
  314. perf: faster dictionary compression on small data when using multiple contexts, by @felixhandte
  315. perf: faster dictionary decompression when using a very large number of dictionaries simultaneously
  316. cli : fix : does no longer overwrite destination when source does not exist (#1082)
  317. cli : new command --adapt, for automatic compression level adaptation
  318. api : fix : block api can be streamed with > 4 GB, reported by @catid
  319. api : reduced ZSTD_DDict size by 2 KB
  320. api : minimum negative compression level is defined, and can be queried using ZSTD_minCLevel().
  321. build: support Haiku target, by @korli
  322. build: Read Legacy format is limited to v0.5+ by default. Can be changed at compile time with macro ZSTD_LEGACY_SUPPORT.
  323. doc : zstd_compression_format.md updated to match wording in IETF RFC 8478
  324. misc: tests/paramgrill, a parameter optimizer, by @GeorgeLu97
  325. v1.3.5 (Jun 29, 2018)
  326. perf: much faster dictionary compression, by @felixhandte
  327. perf: small quality improvement for dictionary generation, by @terrelln
  328. perf: slightly improved high compression levels (notably level 19)
  329. mem : automatic memory release for long duration contexts
  330. cli : fix : overlapLog can be manually set
  331. cli : fix : decoding invalid lz4 frames
  332. api : fix : performance degradation for dictionary compression when using advanced API, by @terrelln
  333. api : change : clarify ZSTD_CCtx_reset() vs ZSTD_CCtx_resetParameters(), by @terrelln
  334. build: select custom libzstd scope through control macros, by @GeorgeLu97
  335. build: OpenBSD patch, by @bket
  336. build: make and make all are compatible with -j
  337. doc : clarify zstd_compression_format.md, updated for IETF RFC process
  338. misc: pzstd compatible with reproducible compilation, by @lamby
  339. v1.3.4 (Mar 27, 2018)
  340. perf: faster speed (especially decoding speed) on recent cpus (haswell+)
  341. perf: much better performance associating --long with multi-threading, by @terrelln
  342. perf: better compression at levels 13-15
  343. cli : asynchronous compression by default, for faster experience (use --single-thread for former behavior)
  344. cli : smoother status report in multi-threading mode
  345. cli : added command --fast=#, for faster compression modes
  346. cli : fix crash when not overwriting existing files, by Pádraig Brady (@pixelb)
  347. api : `nbThreads` becomes `nbWorkers` : 1 triggers asynchronous mode
  348. api : compression levels can be negative, for even more speed
  349. api : ZSTD_getFrameProgression() : get precise progress status of ZSTDMT anytime
  350. api : ZSTDMT can accept new compression parameters during compression
  351. api : implemented all advanced dictionary decompression prototypes
  352. build: improved meson recipe, by Shawn Landden (@shawnl)
  353. build: VS2017 scripts, by @HaydnTrigg
  354. misc: all /contrib projects fixed
  355. misc: added /contrib/docker script by @gyscos
  356. v1.3.3 (Dec 21, 2017)
  357. perf: faster zstd_opt strategy (levels 16-19)
  358. fix : bug #944 : multithreading with shared ditionary and large data, reported by @gsliepen
  359. cli : fix : content size written in header by default
  360. cli : fix : improved LZ4 format support, by @felixhandte
  361. cli : new : hidden command `-S`, to benchmark multiple files while generating one result per file
  362. api : fix : support large skippable frames, by @terrelln
  363. api : fix : streaming interface was adding a useless 3-bytes null block to small frames
  364. api : change : when setting `pledgedSrcSize`, use `ZSTD_CONTENTSIZE_UNKNOWN` macro value to mean "unknown"
  365. build: fix : compilation under rhel6 and centos6, reported by @pixelb
  366. build: added `check` target
  367. v1.3.2 (Oct 10, 2017)
  368. new : long range mode, using --long command, by Stella Lau (@stellamplau)
  369. new : ability to generate and decode magicless frames (#591)
  370. changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode
  371. fix : multi-threading compression works with custom allocators
  372. fix : ZSTD_sizeof_CStream() was over-evaluating memory usage
  373. fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22)
  374. fix : 32-bits build can now decode large offsets (levels 21+)
  375. cli : added LZ4 frame support by default, by Felix Handte (@felixhandte)
  376. cli : improved --list output
  377. cli : new : can split input file for dictionary training, using command -B#
  378. cli : new : clean operation artefact on Ctrl-C interruption
  379. cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851)
  380. cli : fix : write file size in header in multiple-files mode
  381. api : added macro ZSTD_COMPRESSBOUND() for static allocation
  382. api : experimental : new advanced decompression API
  383. api : fix : sizeof_CCtx() used to over-estimate
  384. build: fix : no-multithread variant compiles without pool.c dependency, reported by Mitchell Blank Jr (@mitchblank) (#819)
  385. build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818)
  386. example : added streaming_memory_usage
  387. license : changed /examples license to BSD + GPLv2
  388. license : fix a few header files to reflect new license (#825)
  389. v1.3.1 (Aug 21, 2017)
  390. New license : BSD + GPLv2
  391. perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk)
  392. perf: Multi-threading supports up to 256 threads. Cap at 256 when more are requested (#760)
  393. cli : improved and fixed --list command, by @ib (#772)
  394. cli : command -vV to list supported formats, by @ib (#771)
  395. build : fixed binary variants, reported by @svenha (#788)
  396. build : fix Visual compilation for non x86/x64 targets, reported by Greg Slazinski (@GregSlazinski) (#718)
  397. API exp : breaking change : ZSTD_getframeHeader() provides more information
  398. API exp : breaking change : pinned down values of error codes
  399. doc : fixed huffman example, by Ulrich Kunitz (@ulikunitz)
  400. new : contrib/adaptive-compression, I/O driven compression strength, by Paul Cruz (@paulcruz74)
  401. new : contrib/long_distance_matching, statistics by Stella Lau (@stellamplau)
  402. updated : contrib/linux-kernel, by Nick Terrell (@terrelln)
  403. v1.3.0 (Jul 6, 2017)
  404. cli : new : `--list` command, by Paul Cruz
  405. cli : changed : xz/lzma support enabled by default
  406. cli : changed : `-t *` continue processing list after a decompression error
  407. API : added : ZSTD_versionString()
  408. API : promoted to stable status : ZSTD_getFrameContentSize(), by Sean Purcell
  409. API exp : new advanced API : ZSTD_compress_generic(), ZSTD_CCtx_setParameter()
  410. API exp : new : API for static or external allocation : ZSTD_initStatic?Ctx()
  411. API exp : added : ZSTD_decompressBegin_usingDDict(), requested by Guy Riddle (#700)
  412. API exp : clarified memory estimation / measurement functions.
  413. API exp : changed : strongest strategy renamed ZSTD_btultra, fastest strategy ZSTD_fast set to 1
  414. tools : decodecorpus can generate random dictionary-compressed samples, by Paul Cruz
  415. new : contrib/seekable_format, demo and API, by Sean Purcell
  416. changed : contrib/linux-kernel, updated version and license, by Nick Terrell
  417. v1.2.0 (May 5, 2017)
  418. cli : changed : Multithreading enabled by default (use target zstd-nomt or HAVE_THREAD=0 to disable)
  419. cli : new : command -T0 means "detect and use nb of cores", by Sean Purcell
  420. cli : new : zstdmt symlink hardwired to `zstd -T0`
  421. cli : new : command --threads=# (#671)
  422. cli : changed : cover dictionary builder by default, for improved quality, by Nick Terrell
  423. cli : new : commands --train-cover and --train-legacy, to select dictionary algorithm and parameters
  424. cli : experimental targets `zstd4` and `xzstd4`, with support for lz4 format, by Sean Purcell
  425. cli : fix : does not output compressed data on console
  426. cli : fix : ignore symbolic links unless --force specified,
  427. API : breaking change : ZSTD_createCDict_advanced(), only use compressionParameters as argument
  428. API : added : prototypes ZSTD_*_usingCDict_advanced(), for direct control over frameParameters.
  429. API : improved: ZSTDMT_compressCCtx() reduced memory usage
  430. API : fix : ZSTDMT_compressCCtx() now provides srcSize in header (#634)
  431. API : fix : src size stored in frame header is controlled at end of frame
  432. API : fix : enforced consistent rules for pledgedSrcSize==0 (#641)
  433. API : fix : error code "GENERIC" replaced by "dstSizeTooSmall" when appropriate
  434. build: improved cmake script, by @Majlen
  435. build: enabled Multi-threading support for *BSD, by Baptiste Daroussin
  436. tools: updated Paramgrill. Command -O# provides best parameters for sample and speed target.
  437. new : contrib/linux-kernel version, by Nick Terrell
  438. v1.1.4 (Mar 18, 2017)
  439. cli : new : can compress in *.gz format, using --format=gzip command, by Przemyslaw Skibinski
  440. cli : new : advanced benchmark command --priority=rt
  441. cli : fix : write on sparse-enabled file systems in 32-bits mode, by @ds77
  442. cli : fix : --rm remains silent when input is stdin
  443. cli : experimental : xzstd, with support for xz/lzma decoding, by Przemyslaw Skibinski
  444. speed : improved decompression speed in streaming mode for single shot scenarios (+5%)
  445. memory: DDict (decompression dictionary) memory usage down from 150 KB to 20 KB
  446. arch: 32-bits variant able to generate and decode very long matches (>32 MB), by Sean Purcell
  447. API : new : ZSTD_findFrameCompressedSize(), ZSTD_getFrameContentSize(), ZSTD_findDecompressedSize()
  448. API : changed : dropped support of legacy versions <= v0.3 (can be changed by modifying ZSTD_LEGACY_SUPPORT value)
  449. build : new: meson build system in contrib/meson, by Dima Krasner
  450. build : improved cmake script, by @Majlen
  451. build : added -Wformat-security flag, as recommended by Padraig Brady
  452. doc : new : educational decoder, by Sean Purcell
  453. v1.1.3 (Feb 7, 2017)
  454. cli : zstd can decompress .gz files (can be disabled with `make zstd-nogz` or `make HAVE_ZLIB=0`)
  455. cli : new : experimental target `make zstdmt`, with multi-threading support
  456. cli : new : improved dictionary builder "cover" (experimental), by Nick Terrell, based on prior work by Giuseppe Ottaviano.
  457. cli : new : advanced commands for detailed parameters, by Przemyslaw Skibinski
  458. cli : fix zstdless on Mac OS-X, by Andrew Janke
  459. cli : fix #232 "compress non-files"
  460. dictBuilder : improved dictionary generation quality, thanks to Nick Terrell
  461. API : new : lib/compress/ZSTDMT_compress.h multithreading API (experimental)
  462. API : new : ZSTD_create?Dict_byReference(), requested by Bartosz Taudul
  463. API : new : ZDICT_finalizeDictionary()
  464. API : fix : ZSTD_initCStream_usingCDict() properly writes dictID into frame header, by Gregory Szorc (#511)
  465. API : fix : all symbols properly exposed in libzstd, by Nick Terrell
  466. build : support for Solaris target, by Przemyslaw Skibinski
  467. doc : clarified specification, by Sean Purcell
  468. v1.1.2 (Dec 15, 2016)
  469. API : streaming : decompression : changed : automatic implicit reset when chain-decoding new frames without init
  470. API : experimental : added : dictID retrieval functions, and ZSTD_initCStream_srcSize()
  471. API : zbuff : changed : prototypes now generate deprecation warnings
  472. lib : improved : faster decompression speed at ultra compression settings and 32-bits mode
  473. lib : changed : only public ZSTD_ symbols are now exposed
  474. lib : changed : reduced usage of stack memory
  475. lib : fixed : several corner case bugs, by Nick Terrell
  476. cli : new : gzstd, experimental version able to decode .gz files, by Przemyslaw Skibinski
  477. cli : new : preserve file attributes
  478. cli : new : added zstdless and zstdgrep tools
  479. cli : fixed : status displays total amount decoded, even for file consisting of multiple frames (like pzstd)
  480. cli : fixed : zstdcat
  481. zlib_wrapper : added support for gz* functions, by Przemyslaw Skibinski
  482. install : better compatibility with FreeBSD, by Dimitry Andric
  483. source tree : changed : zbuff source files moved to lib/deprecated
  484. v1.1.1 (Nov 2, 2016)
  485. New : command -M#, --memory=, --memlimit=, --memlimit-decompress= to limit allowed memory consumption
  486. New : doc/zstd_manual.html, by Przemyslaw Skibinski
  487. Improved : slightly better compression ratio at --ultra levels (>= 20)
  488. Improved : better memory usage when using streaming compression API, thanks to @Rogier-5 report
  489. Added : API : ZSTD_initCStream_usingCDict(), ZSTD_initDStream_usingDDict() (experimental section)
  490. Added : example/multiple_streaming_compression.c
  491. Changed : zstd_errors.h is now installed within /include (and replaces errors_public.h)
  492. Updated man page
  493. Fixed : zstd-small, zstd-compress and zstd-decompress compilation targets
  494. v1.1.0 (Sep 28, 2016)
  495. New : contrib/pzstd, parallel version of zstd, by Nick Terrell
  496. added : NetBSD install target (#338)
  497. Improved : speed for batches of small files
  498. Improved : speed of zlib wrapper, by Przemyslaw Skibinski
  499. Changed : libzstd on Windows supports legacy formats, by Christophe Chevalier
  500. Fixed : CLI -d output to stdout by default when input is stdin (#322)
  501. Fixed : CLI correctly detects console on Mac OS-X
  502. Fixed : CLI supports recursive mode `-r` on Mac OS-X
  503. Fixed : Legacy decoders use unified error codes, reported by benrg (#341), fixed by Przemyslaw Skibinski
  504. Fixed : compatibility with OpenBSD, reported by Juan Francisco Cantero Hurtado (#319)
  505. Fixed : compatibility with Hurd, by Przemyslaw Skibinski (#365)
  506. Fixed : zstd-pgo, reported by octoploid (#329)
  507. v1.0.0 (Sep 1, 2016)
  508. Change Licensing, all project is now BSD, Copyright Facebook
  509. Small decompression speed improvement
  510. API : Streaming API supports legacy format
  511. API : ZDICT_getDictID(), ZSTD_sizeof_{CCtx, DCtx, CStream, DStream}(), ZSTD_setDStreamParameter()
  512. CLI supports legacy formats v0.4+
  513. Fixed : compression fails on certain huge files, reported by Jesse McGrew
  514. Enhanced documentation, by Przemyslaw Skibinski
  515. v0.8.1 (Aug 18, 2016)
  516. New streaming API
  517. Changed : --ultra now enables levels beyond 19
  518. Changed : -i# now selects benchmark time in second
  519. Fixed : ZSTD_compress* can now compress > 4 GB in a single pass, reported by Nick Terrell
  520. Fixed : speed regression on specific patterns (#272)
  521. Fixed : support for Z_SYNC_FLUSH, by Dmitry Krot (#291)
  522. Fixed : ICC compilation, by Przemyslaw Skibinski
  523. v0.8.0 (Aug 2, 2016)
  524. Improved : better speed on clang and gcc -O2, thanks to Eric Biggers
  525. New : Build on FreeBSD and DragonFly, thanks to JrMarino
  526. Changed : modified API : ZSTD_compressEnd()
  527. Fixed : legacy mode with ZSTD_HEAPMODE=0, by Christopher Bergqvist
  528. Fixed : premature end of frame when zero-sized raw block, reported by Eric Biggers
  529. Fixed : large dictionaries (> 384 KB), reported by Ilona Papava
  530. Fixed : checksum correctly checked in single-pass mode
  531. Fixed : combined --test amd --rm, reported by Andreas M. Nilsson
  532. Modified : minor compression level adaptations
  533. Updated : compression format specification to v0.2.0
  534. changed : zstd.h moved to /lib directory
  535. v0.7.5 (Aug 1, 2016)
  536. Transition version, supporting decoding of v0.8.x
  537. v0.7.4 (Jul 17, 2016)
  538. Added : homebrew for Mac, by Daniel Cade
  539. Added : more examples
  540. Fixed : segfault when using small dictionaries, reported by Felix Handte
  541. Modified : default compression level for CLI is now 3
  542. Updated : specification, to v0.1.1
  543. v0.7.3 (Jul 9, 2016)
  544. New : compression format specification
  545. New : `--` separator, stating that all following arguments are file names. Suggested by Chip Turner.
  546. New : `ZSTD_getDecompressedSize()`
  547. New : OpenBSD target, by Juan Francisco Cantero Hurtado
  548. New : `examples` directory
  549. fixed : dictBuilder using HC levels, reported by Bartosz Taudul
  550. fixed : legacy support from ZSTD_decompress_usingDDict(), reported by Felix Handte
  551. fixed : multi-blocks decoding with intermediate uncompressed blocks, reported by Greg Slazinski
  552. modified : removed "mem.h" and "error_public.h" dependencies from "zstd.h" (experimental section)
  553. modified : legacy functions no longer need magic number
  554. v0.7.2 (Jul 4, 2016)
  555. fixed : ZSTD_decompressBlock() using multiple consecutive blocks. Reported by Greg Slazinski.
  556. fixed : potential segfault on very large files (many gigabytes). Reported by Chip Turner.
  557. fixed : CLI displays system error message when destination file cannot be created (#231). Reported by Chip Turner.
  558. v0.7.1 (Jun 23, 2016)
  559. fixed : ZBUFF_compressEnd() called multiple times with too small `dst` buffer, reported by Christophe Chevalier
  560. fixed : dictBuilder fails if first sample is too small, reported by Руслан Ковалёв
  561. fixed : corruption issue, reported by cj
  562. modified : checksum enabled by default in command line mode
  563. v0.7.0 (Jun 17, 2016)
  564. New : Support for directory compression, using `-r`, thanks to Przemyslaw Skibinski
  565. New : Command `--rm`, to remove source file after successful de/compression
  566. New : Visual build scripts, by Christophe Chevalier
  567. New : Support for Sparse File-systems (do not use space for zero-filled sectors)
  568. New : Frame checksum support
  569. New : Support pass-through mode (when using `-df`)
  570. API : more efficient Dictionary API : `ZSTD_compress_usingCDict()`, `ZSTD_decompress_usingDDict()`
  571. API : create dictionary files from custom content, by Giuseppe Ottaviano
  572. API : support for custom malloc/free functions
  573. New : controllable Dictionary ID
  574. New : Support for skippable frames
  575. v0.6.1 (May 13, 2016)
  576. New : zlib wrapper API, thanks to Przemyslaw Skibinski
  577. New : Ability to compile compressor / decompressor separately
  578. Changed : new lib directory structure
  579. Fixed : Legacy codec v0.5 compatible with dictionary decompression
  580. Fixed : Decoder corruption error (#173)
  581. Fixed : null-string roundtrip (#176)
  582. New : benchmark mode can select directory as input
  583. Experimental : midipix support, VMS support
  584. v0.6.0 (Apr 13, 2016)
  585. Stronger high compression modes, thanks to Przemyslaw Skibinski
  586. API : ZSTD_getFrameParams() provides size of decompressed content
  587. New : highest compression modes require `--ultra` command to fully unleash their capacity
  588. Fixed : zstd cli return error code > 0 and removes dst file artifact when decompression fails, thanks to Chip Turner
  589. v0.5.1 (Feb 18, 2016)
  590. New : Optimal parsing => Very high compression modes, thanks to Przemyslaw Skibinski
  591. Changed : Dictionary builder integrated into libzstd and zstd cli
  592. Changed (!) : zstd cli now uses "multiple input files" as default mode. See `zstd -h`.
  593. Fix : high compression modes for big-endian platforms
  594. New : zstd cli : `-t` | `--test` command
  595. v0.5.0 (Feb 5, 2016)
  596. New : dictionary builder utility
  597. Changed : streaming & dictionary API
  598. Improved : better compression of small data
  599. v0.4.7 (Jan 22, 2016)
  600. Improved : small compression speed improvement in HC mode
  601. Changed : `zstd_decompress.c` has ZSTD_LEGACY_SUPPORT to 0 by default
  602. fix : bt search bug
  603. v0.4.6 (Jan 13, 2016)
  604. fix : fast compression mode on Windows
  605. New : cmake configuration file, thanks to Artyom Dymchenko
  606. Improved : high compression mode on repetitive data
  607. New : block-level API
  608. New : ZSTD_duplicateCCtx()
  609. v0.4.5 (Dec 18, 2015)
  610. new : -m/--multiple : compress/decompress multiple files
  611. v0.4.4 (Dec 14, 2015)
  612. Fixed : high compression modes for Windows 32 bits
  613. new : external dictionary API extended to buffered mode and accessible through command line
  614. new : windows DLL project, thanks to Christophe Chevalier
  615. v0.4.3 (Dec 7, 2015)
  616. new : external dictionary API
  617. new : zstd-frugal
  618. v0.4.2 (Dec 2, 2015)
  619. Generic minor improvements for small blocks
  620. Fixed : big-endian compatibility, by Peter Harris (#85)
  621. v0.4.1 (Dec 1, 2015)
  622. Fixed : ZSTD_LEGACY_SUPPORT=0 build mode (reported by Luben)
  623. removed `zstd.c`
  624. v0.4.0 (Nov 29, 2015)
  625. Command line utility compatible with high compression levels
  626. Removed zstdhc => merged into zstd
  627. Added : ZBUFF API (see zstd_buffered.h)
  628. Rolling buffer support
  629. v0.3.6 (Nov 10, 2015)
  630. small blocks params
  631. v0.3.5 (Nov 9, 2015)
  632. minor generic compression improvements
  633. v0.3.4 (Nov 6, 2015)
  634. Faster fast cLevels
  635. v0.3.3 (Nov 5, 2015)
  636. Small compression ratio improvement
  637. v0.3.2 (Nov 2, 2015)
  638. Fixed Visual Studio
  639. v0.3.1 (Nov 2, 2015)
  640. Small compression ratio improvement
  641. v0.3 (Oct 30, 2015)
  642. HC mode : compression levels 2-26
  643. v0.2.2 (Oct 28, 2015)
  644. Fix : Visual Studio 2013 & 2015 release compilation, by Christophe Chevalier
  645. v0.2.1 (Oct 24, 2015)
  646. Fix : Read errors, advanced fuzzer tests, by Hanno Böck
  647. v0.2.0 (Oct 22, 2015)
  648. **Breaking format change**
  649. Faster decompression speed
  650. Can still decode v0.1 format
  651. v0.1.3 (Oct 15, 2015)
  652. fix uninitialization warning, reported by Evan Nemerson
  653. v0.1.2 (Sep 11, 2015)
  654. frame concatenation support
  655. v0.1.1 (Aug 27, 2015)
  656. fix compression bug
  657. detects write-flush errors
  658. v0.1.0 (Aug 25, 2015)
  659. first release