This release highlights the deployment of Google Chrome 123, introducing zstd-encoding for Web traffic, introduced as a preferable option for compression of dynamic contents. With limited web server support for zstd-encoding due to its novelty, we are launching an updated Zstandard version to facilitate broader adoption.

New stable parameter ZSTD_c_targetCBlockSize

Using zstd compression for large documents over the Internet, data is segmented into smaller blocks of up to 128 KB, for incremental updates. This is crucial for applications like Chrome that process parts of documents as they arrive. However, on slow or congested networks, there can be some brief unresponsiveness in the middle of a block transmission, delaying update. To mitigate such scenarios, libzstd introduces the new parameter ZSTD_c_targetCBlockSize, enabling the division of blocks into even smaller segments to enhance initial byte delivery speed. Activating this feature incurs a cost, both runtime (equivalent to -2% speed at level 8) and a slight compression efficiency decrease (<0.1%), but offers some interesting latency reduction, notably beneficial in areas with less powerful network infrastructure.

Granular binary size selection

libzstd provides build customization, including options to compile only the compression or decompression modules, minimizing binary size. Enhanced in v1.5.6 (source), it now allows for even finer control by enabling selective inclusion or exclusion of specific components within these modules. This advancement aids applications needing precise binary size management.

Miscellaneous Enhancements

This release includes various minor enhancements and bug fixes to enhance user experience. Key updates include an expanded list of recognized compressed file suffixes for the --exclude-compressed flag, improving efficiency by skipping presumed incompressible content. Furthermore, compatibility has been broadened to include additional chipsets (sparc64, ARM64EC, risc-v) and operating systems (QNX, AIX, Solaris, HP-UX).

Change Log

api: Promote ZSTD_c_targetCBlockSize to Stable API by @felixhandte
api: new experimental ZSTD_d_maxBlockSize parameter, to reduce streaming decompression memory, by @terrelln
perf: improve performance of param ZSTD_c_targetCBlockSize, by @Cyan4973
perf: improved compression of arrays of integers at high compression, by @Cyan4973
lib: reduce binary size with selective built-time exclusion, by @felixhandte
lib: improved huffman speed on small data and linux kernel, by @terrelln
lib: accept dictionaries with partial literal tables, by @terrelln
lib: fix CCtx size estimation with external sequence producer, by @embg
lib: fix corner case decoder behaviors, by @Cyan4973 and @aimuz
lib: fix zdict prototype mismatch in static_only mode, by @ldv-alt
lib: fix several bugs in magicless-format decoding, by @embg
cli: add common compressed file types to --exclude-compressed by @daniellerozenblit (requested by @dcog989)
cli: fix mixing -c and -o commands with --rm, by @Cyan4973
cli: fix erroneous exclusion of hidden files with --output-dir-mirror by @felixhandte
cli: improved time accuracy on BSD, by @felixhandte
cli: better errors on argument parsing, by @KapJI
tests: better compatibility with older versions of grep, by @Cyan4973
tests: lorem ipsum generator as default content generator, by @Cyan4973
build: cmake improvements by @terrelln, @sighingnow, @gjasny, @JohanMabille, @Saverio976, @gruenich, @teo-tsirpanis
build: bazel support, by @jondo2010
build: fix cross-compiling for AArch64 with lld by @jcelerier
build: fix Apple platform compatibility, by @nidhijaju
build: fix Visual 2012 and lower compatibility, by @Cyan4973
build: improve win32 support, by @DimitriPapadopoulos
build: better C90 compliance for zlibWrapper, by @emaste
port: make: fat binaries on macos, by @mredig
port: ARM64EC compatibility for Windows, by @dunhor
port: QNX support by @klausholstjacobsen
port: MSYS2 and Cygwin makefile installation and test support, by @QBos07
port: risc-v support validation in CI, by @Cyan4973
port: sparc64 support validation in CI, by @Cyan4973
port: AIX compatibility, by @likema
port: HP-UX compatibility, by @likema
doc: Improved specification accuracy, by @elasota
bug: Fix and deprecate ZSTD_generateSequences (#3981), by @terrelln

Full change list (auto-generated)

• Add win32 to windows-artifacts.yml by @Kim-SSi in #3600
• Fix mmap-dict help output by @daniellerozenblit in #3601
• [oss-fuzz] Fix simple_round_trip fuzzer with overlapping decompression by @terrelln in #3612
• Reduce streaming decompression memory by (128KB - blockSizeMax) by @terrelln in #3616
• removed travis & appveyor scripts by @Cyan4973 in #3621
• Add ZSTD_d_maxBlockSize parameter by @terrelln in #3617
• [doc] add decoder errata paragraph by @Cyan4973 in #3620
• add makefile entry to build fat binary on macos by @mredig in #3614
• Disable unused variable warning in msan configurations by @danlark1 in #3624
  #3634
• Allow Build-Time Exclusion of Individual Compression Strategies by @felixhandte in #3623
• Get zstd working with ARM64EC on Windows by @dunhor in #3636
• minor : update streaming_compression example by @Cyan4973 in #3631
• Fix UBSAN issue (zero addition to NULL) by @terrelln in #3658
• Add options in Makefile to cmake by @sighingnow in #3657
• fix a minor inefficiency in compress_superblock by @Cyan4973 in #3668
• Fixed a bug in the educational decoder by @Cyan4973 in #3659
• changed LLU suffix into ULL for Visual 2012 and lower by @Cyan4973 in #3664
• fixed decoder behavior when nbSeqs==0 is encoded using 2 bytes by @Cyan4973 in #3669
• detect extraneous bytes in the Sequences section by @Cyan4973 in #3674
• Bitstream produces only zeroes after an overflow event by @Cyan4973 in #3676
• Update FreeBSD CI images to latest supported releases by @emaste in #3684
• Clean up a false error message in the LDM debug log by @embg in #3686
• Hide ASM symbols on Apple platforms by @nidhijaju in #3688
• Changed the decoding loop to detect more invalid cases of corruption sooner by @Cyan4973 in #3677
• Fix Intel Xcode builds with assembly by @gjasny in #3665
• Save one byte on the frame epilogue by @Coder-256 in #3700
• Update fileio.c: fix build failure with enabled LTO by @LocutusOfBorg in #3695
• fileio_asyncio: handle malloc fails in AIO_ReadPool_create by @void0red in #3704
• Fix typographical error in README.md by @nikohoffren in #3701
• Fixed typo by @alexsifivetw in #3712
• Improve dual license wording in README by @terrelln in #3718
• Unpoison Workspace Memory Before Custom-Free by @felixhandte in #3725
• added ZSTD_decompressDCtx() benchmark option to fullbench by @Cyan4973 in #3726
• No longer reject dictionaries with literals maxSymbolValue < 255 by @terrelln in #3731
• fix: ZSTD_BUILD_DECOMPRESSION message by @0o001 in #3728
• Updated Makefiles for full MSYS2 and Cygwin installation and testing … by @QBos07 in #3720
• Work around nullptr-with-nonzero-offset warning by @terrelln in #3738
• Fix & refactor Huffman repeat tables for dictionaries by @terrelln in #3737
• zdictlib: fix prototype mismatch by @ldv-alt in #3733
• Fixed zstd cmake shared build on windows by @JohanMabille in #3739
• Added qnx in the posix test section of platform.h by @klausholstjacobsen in #3745
• added some documentation on ZSTD_estimate*Size() variants by @Cyan4973 in #3755
• Improve macro guards for ZSTD_assertValidSequence by @terrelln in #3770
• Stop suppressing pointer-overflow UBSAN errors by @terrelln in #3776
• fix x32 tests on Github CI by @Cyan4973 in #3777
• Fix new typos found by codespell by @DimitriPapadopoulos in #3771
• Do not test WIN32, instead test _WIN32 by @DimitriPapadopoulos in #3772
• Fix a very small formatting typo in the lib/README.md file by @dloidolt in #3763
• Fix pzstd Makefile to allow setting DESTDIR and BINDIR separately by @paulmenzel in #3752
• Remove FlexArray pattern from ZSTDMT by @Cyan4973 in #3786
• solving flexArray issue #3785 in fse by @Cyan4973 in #3789
• Add doc on how to use it with cmake FetchContent by @Saverio976 in #3795
• Correct FSE probability bit consumption in specification by @elasota in #3806
• Add Bazel module instructions to README.md by @jondo2010 in #3812
• Clarify that a stream containing too many Huffman weights is invalid by @elasota in #3813
• [cmake] Require CMake version 3.5 or newer by @gruenich in #3807
• Three fixes for the Linux kernel by @terrelln in #3822
• [huf] Improve fast huffman decoding speed in linux kernel by @terrelln in #3826
• [huf] Improve fast C & ASM performance on small data by @terrelln in #3827
• update xxhash library to v0.8.2 by @Cyan4973 in #3820
• Modernize macros to use do { } while (0) by @terrelln in #3831
• Clarify that the presence of weight value 1 is required, and a lone implied 1 weight is invalid by @elasota in #3814
• Move offload API params into ZSTD_CCtx_params by @embg in #3839
• Update FreeBSD CI: drop 12.4 (nearly EOL) by @emaste in #3845
• Make offload API compatible with static CCtx by @embg in #3854
• zlibWrapper: convert to C89 / ANSI C by @emaste in #3846
• Fix a nullptr dereference in ZSTD_createCDict_advanced2() by @michoecho in #3847
• Cirrus-CI: Add FreeBSD 14 by @emaste in #3855
• CI: meson: use builtin handling for MSVC by @eli-schwartz in #3858
• cli: better errors on argument parsing by @KapJI in #3850
• Clarify that probability tables must not contain non-zero probabilities for invalid values by @elasota in #3817
• [x-compile] Fix cross-compiling for AArch64 with lld by @jcelerier in #3760
• playTests.sh does no longer needs grep -E by @Cyan4973 in #3865
• minor: playTests.sh more compatible with older versions of grep by @Cyan4973 in #3877
• disable Intel CET Compatibility tests by @Cyan4973 in #3884
• improve cmake test by @Cyan4973 in #3883
• add sparc64 compilation test by @Cyan4973 in #3886
• add a lorem ipsum generator by @Cyan4973 in #3890
• Update Dependency in Intel CET Test; Re-Enable Test by @felixhandte in #3893
• Improve compression of Arrays of Integers (High compression mode) by @Cyan4973 in #3895
• [Zstd] Less verbose log for patch mode. by @sandreenko in #3899
• fix 5921623844651008 by @Cyan4973 in #3900
• Fix fuzz issue 5131069967892480 by @Cyan4973 in #3902
• Advertise Availability of Security Vulnerability Notifications by @felixhandte in #3909
• updated setup-msys2 to v2.22.0 by @Cyan4973 in #3914
• Lorem Ipsum generator update by @Cyan4973 in #3913
• Reduce scope of variables by @gruenich in #3903
• Improve speed of ZSTD_c_targetCBlockSize by @Cyan4973 in #3915
• More regular block sizes with targetCBlockSize by @Cyan4973 in #3917
• removed sprintf usage from zstdcli.c by @Cyan4973 in #3916
• Export a zstd::libzstd CMake target if only static or dynamic linkage is specified. by @teo-tsirpanis in #3811
• fix version of actions/checkout by @Cyan4973 in #3926
• minor Makefile refactoring by @Cyan4973 in #3753
• lib/decompress: check for reserved bit corruption in zstd by @aimuz in #3840
• Fix state table formatting by @elasota in #3816
• Specify offset 0 as invalid and specify required fixup behavior by @elasota in #3824
• update -V documentation by @Cyan4973 in #3928
• fix LLU->ULL by @Cyan4973 in #3929
• Fix building xxhash on AIX 5.1 by @likema in #3860
• Fix building on HP-UX 11.11 PA-RISC by @likema in #3862
• Fix AsyncIO reading seed queueing by @yoniko in #3940
• Use ZSTD_LEGACY_SUPPORT=5 in "make test" by @embg in #3943
• Pin sanitizer CI jobs to ubuntu-20.04 by @embg in #3945
• chore: fix some typos by @acceptacross in #3949
• new method to deal with offset==0 erroneous edge case by @Cyan4973 in #3937
• add tests inspired from #2927 by @Cyan4973 in #3948
• cmake refactor: move HP-UX specific logic into its own function by @Cyan4973 in #3946
• Fix #3719 : mixing -c, -o and --rm by @Cyan4973 in #3942
• minor: fix incorrect debug level by @Cyan4973 in #3936
• add RISC-V emulation tests to Github CI by @Cyan4973 in #3934
• prevent XXH64 from being autovectorized by XXH512 by default by @Cyan4973 in #3933
• Stop Hardcoding the POSIX Version on BSDs by @felixhandte in #3952
• Convert the CircleCI workflow to a GitHub Actions workflow by @jk0 in #3901
• Add common compressed file types to --exclude-compressed by @daniellerozenblit in #3951
• Export ZSTD_LEGACY_SUPPORT in tests/Makefile by @embg in #3955
• Exercise ZSTD_findDecompressedSize() in the simple decompression fuzzer by @embg in #3959
• Update ZSTD_RowFindBestMatch comment by @yoniko in #3947
• Add the zeroSeq sample by @Cyan4973 in #3954
• [cpu] Backport fix for rbx clobbering on Windows with Clang by @terrelln in #3957
• Do not truncate file name in verbose mode by @Cyan4973 in #3956
• updated documentation by @Cyan4973 in #3958
• [asm][aarch64] Mark that BTI and PAC are supported by @terrelln in #3961
• Use utimensat() on FreeBSD by @felixhandte in #3960
• reduce the amount of #include in cover.h by @Cyan4973 in #3962
• Remove Erroneous Exclusion of Hidden Files and Folders in --output-dir-mirror by @felixhandte in #3963
• Promote ZSTD_c_targetCBlockSize Parameter to Stable API by @felixhandte in #3964
• [cmake] Always create libzstd target by @terrelln in #3965
• Remove incorrect docs regarding ZSTD_findFrameCompressedSize() by @embg in #3967
• add line number to debug traces by @Cyan4973 in #3966
• bump version number by @Cyan4973 in #3969
• Export zstd's public headers via BUILD_INTERFACE by @terrelln in #3968
• Fix bug with streaming decompression of magicless format by @embg in #3971
• pzstd: use c++14 without conditions by @kanavin in #3682
• Fix bugs in simple decompression fuzzer by @yoniko in #3978
• Fuzzing and bugfixes for magicless-format decoding by @embg in #3976
• Fix & fuzz ZSTD_generateSequences by @terrelln in #3981
• Fail on errors when building fuzzers by @yoniko in #3979
• [cmake] Emit warnings for contradictory build settings by @terrelln in #3975
• Document the process for adding a new fuzzer by @embg in #3982
• Fix -Werror=pointer-arith in fuzzers by @embg in #3983
• Doc update by @Cyan4973 in #3977
• v1.5.6 by @Cyan4973 in #3984

New Contributors

• @Kim-SSi made their first contribution in #3600
• @mredig made their first contribution in #3614
• @dunhor made their first contribution in #3636
• @sighingnow made their first contribution in #3657
• @nidhijaju made their first contribution in #3688
• @gjasny made their first contribution in #3665
• @Coder-256 made their first contribution in #3700
• @LocutusOfBorg made their first contribution in #3695
• @void0red made their first contribution in #3704
• @nikohoffren made their first contribution in #3701
• @alexsifivetw made their first contribution in #3712
• @0o001 made their first contribution in #3728
• @QBos07 made their first contribution in #3720
• @JohanMabille made their first contribution in #3739
• @klausholstjacobsen made their first contribution in #3745
• @Saverio976 made their first contribution in #3795
• @elasota made their first contribution in #3806
• @jondo2010 made their first contribution in #3812
• @gruenich made their first contribution in #3807
• @michoecho made their first contribution in #3847
• @KapJI made their first contribution in #3850
• @jcelerier made their first contribution in #3760
• @sandreenko made their first contribution in #3899
• @teo-tsirpanis made their first contribution in #3811
• @aimuz made their first contribution in #3840
• @acceptacross made their first contribution in #3949
• @jk0 made their first contribution in #3901

Full Changelog: v1.5.5...v1.5.6