XZ Utils Release Notes

======================

5.6.1 (2024-03-09)

IMPORTANT: This fixed bugs in the backdoor (CVE-2024-3094) (someone

had forgot to run Valgrind).

* liblzma: Fixed two bugs relating to GNU indirect function (IFUNC)

with GCC. The more serious bug caused a program linked with

liblzma to crash on start up if the flag -fprofile-generate was

used to build liblzma. The second bug caused liblzma to falsely

report an invalid write to Valgrind when loading liblzma.

* xz: Changed the messages for thread reduction due to memory

constraints to only appear under the highest verbosity level.

* Build:

- Fixed a build issue when the header file <linux/landlock.h>

was present on the system but the Landlock system calls were

not defined in <sys/syscall.h>.

- The CMake build now warns and disables NLS if both gettext

tools and pre-created .gmo files are missing. Previously,

this caused the CMake build to fail.

* Minor improvements to man pages.

* Minor improvements to tests.

5.6.0 (2024-02-24)

IMPORTANT: This added a backdoor (CVE-2024-3094). It's enabled only

in the release tarballs.

This bumps the minor version of liblzma because new features were

added. The API and ABI are still backward compatible with liblzma

5.4.x and 5.2.x and 5.0.x.

NOTE: As described in the NEWS for 5.5.2beta, the core components

are now under the BSD Zero Clause License (0BSD).

Since 5.5.2beta:

* liblzma:

- Disabled the branchless C variant in the LZMA decoder based

on the benchmark results from the community.

- Disabled x86-64 inline assembly on x32 to fix the build.

* Sandboxing support in xz:

- Landlock is now used even when xz needs to create files.

In this case the sandbox has to be more permissive than

when no files need to be created. A similar thing was

already in use with pledge(2) since 5.3.4alpha.

- Landlock and pledge(2) are now stricter when reading from

more than one input file and only writing to standard output.

- Added support for Landlock ABI version 4.

* CMake:

- Default to -O2 instead of -O3 with CMAKE_BUILD_TYPE=Release.

-O3 is not useful for speed and makes the code larger.

- Now builds lzmainfo and lzmadec.

- xzdiff, xzgrep, xzless, xzmore, and their symlinks are now

installed. The scripts are also tested during "make test".

- Added translation support for xz, lzmainfo, and the

man pages.

- Applied the symbol versioning workaround for MicroBlaze that

is used in the Autotools build.

- The general XZ Utils and liblzma API documentation is now

installed.

- The CMake component names were changed a little and several

were added. liblzma_Runtime and liblzma_Development are

unchanged.

- Minimum required CMake version is now 3.14. However,

translation support is disabled with CMake versions

older than 3.20.

- The CMake-based build is now close to feature parity with the

Autotools-based build. Most importantly a few tests aren't

run yet. Testing the CMake-based build on different operating

systems would be welcome now. See the comment at the top of

CMakeLists.txt.

* Fixed a bug in the Autotools feature test for ARM64 CRC32

instruction support for old versions of Clang. This did not

affect the CMake build.

* Windows:

- The build instructions in INSTALL and windows/INSTALL*.txt

were revised completely.

- windows/build-with-cmake.bat along with the instructions

in windows/INSTALL-MinGW-w64_with_CMake.txt should make

it very easy to build liblzma.dll and xz.exe on Windows

using CMake and MinGW-w64 with either GCC or Clang/LLVM.

- windows/build.bash was updated. It now works on MSYS2 and

on GNU/Linux (cross-compiling) to create a .zip and .7z

package for 32-bit and 64-bit x86 using GCC + MinGW-w64.

* The TODO file is no longer installed as part of the

documentation. The file is out of date and does not reflect

the actual tasks that will be completed in the future.

* Translations:

- Translated lzmainfo man pages are now installed. These

had been forgotten in earlier versions.

- Updated Croatian, Esperanto, German, Hungarian, Korean,

Polish, Romanian, Spanish, Swedish, Vietnamese, and Ukrainian

translations.

- Updated German, Korean, Romanian, and Ukrainian man page

translations.

* Added a few tests.

Summary of new features added in the 5.5.x development releases:

* liblzma:

- LZMA decoder: Speed optimizations to the C code and

added GCC & Clang compatible inline assembly for x86-64.

- Added lzma_mt_block_size() to recommend a Block size for

multithreaded encoding.

- Added CLMUL-based CRC32 on x86-64 and E2K with runtime

processor detection. Similar to CRC64, on 32-bit x86 it

isn't available unless --disable-assembler is used.

- Optimized the CRC32 calculation on ARM64 platforms using the

CRC32 instructions. Runtime detection for the instruction is

used on GNU/Linux, FreeBSD, Windows, and macOS. If the

compiler flags indicate unconditional CRC32 instruction

support (+crc) then the generic version is not built.

- Added definitions of mask values like

LZMA_INDEX_CHECK_MASK_CRC32 to <lzma/index.h>.

* xz:

- Multithreaded mode is now the default. This improves

compression speed and creates .xz files that can be

decompressed in multithreaded mode. The downsides are

increased memory usage and slightly worse compression ratio.

- Added a new command line option --filters to set the filter

chain using the liblzma filter string syntax.

- Added new command line options --filters1 ... --filters9 to

set additional filter chains using the liblzma filter string

syntax. The --block-list option now allows specifying filter

chains that were set using these new options.

- Ported the command line tools to Windows MSVC.

Visual Studio 2015 or later is required.

* Added lz4 support to xzdiff/xzcmp and xzgrep.

5.5.2beta (2024-02-14)

* Licensing change: The core components are now under the

BSD Zero Clause License (0BSD). In XZ Utils 5.4.6 and older

and 5.5.1alpha these components are in the public domain and

obviously remain so; the change affects the new releases only.

0BSD is an extremely permissive license which doesn't require

retaining or reproducing copyright or license notices when

distributing the code, thus in practice there is extremely

little difference to public domain.

* liblzma

- Significant speed optimizations to the LZMA decoder were

made. There are now three variants that can be chosen at

build time:

* Basic C version: This is a few percent faster than

5.4.x due to some new optimizations.

* Branchless C: This is currently the default on platforms

for which there is no assembly code. This should be a few

percent faster than the basic C version.

* x86-64 inline assembly. This works with GCC and Clang.

The default choice can currently be overridden by setting

LZMA_RANGE_DECODER_CONFIG in CPPFLAGS: 0 means the basic

version and 3 means that branchless C version.

- Optimized the CRC32 calculation on ARM64 platforms using the

CRC32 instructions. The instructions are optional in ARMv8.0

and are required in ARMv8.1 and later. Runtime detection for

the instruction is used on GNU/Linux, FreeBSD, Windows, and

macOS. If the compiler flags indicate unconditional CRC32

instruction support (+crc) then the generic version is not

built.

* Added lz4 support to xzdiff/xzcmp and xzgrep.

* Man pages of xzdiff/xzcmp, xzgrep, and xzmore were rewritten

to simplify licensing of the man page translations.

* Translations:

- Updated Chinese (simplified), German, Korean, Polish,

Romanian, Spanish, Swedish, and Ukrainian translations.

- Updated German, Korean, Romanian, and Ukrainian man page

translations.

* Small improvements to the tests.

* Added doc/examples/11_file_info.c. It was added to the Git

repository in 2017 but forgotten to be added into distribution

tarballs.

* Removed doc/examples_old. These were from 2012.

* Removed the macos/build.sh script. It had not been updated

since 2013.

5.5.1alpha (2024-01-26)

* Added a new filter for RISC-V binaries. The filter can be used

for 32-bit and 64-bit binaries with either little or big

endianness. In liblzma, the Filter ID is LZMA_FILTER_RISCV (0x0B)

and the xz option is --riscv. liblzma filter string syntax

recognizes this filter as "riscv".

* liblzma:

- Added lzma_mt_block_size() to recommend a Block size for

multithreaded encoding

- Added CLMUL-based CRC32 on x86-64 and E2K with runtime

processor detection. Similar to CRC64, on 32-bit x86 it

isn't available unless --disable-assembler is used.

- Implemented GNU indirect function (IFUNC) as a runtime

function dispatching method for CRC32 and CRC64 fast

implementations on x86. Only GNU/Linux (glibc) and FreeBSD

builds will use IFUNC, unless --enable-ifunc is specified to

configure.

- Added definitions of mask values like

LZMA_INDEX_CHECK_MASK_CRC32 to <lzma/index.h>.

- The XZ logo is now included in the Doxygen generated

documentation. It is licensed under Creative Commons

Attribution-ShareAlike 4.0.

* xz:

- Multithreaded mode is now the default. This improves

compression speed and creates .xz files that can be

decompressed multithreaded at the cost of increased memory

usage and slightly worse compression ratio.

- Added new command line option --filters to set the filter

chain using liblzma filter string syntax.

- Added new command line options --filters1 ... --filters9 to

set additional filter chains using liblzma filter string

syntax. The --block-list option now allows specifying filter

chains that were set using these new options.

- Added support for Linux Landlock as a sandboxing method.

- xzdec now supports pledge(2), Capsicum, and Linux Landlock as

sandboxing methods.

- Progress indicator time stats remain accurate after pausing

xz with SIGTSTP.

- Ported xz and xzdec to Windows MSVC. Visual Studio 2015 or

later is required.

* CMake Build:

- Supports pledge(2), Capsicum, and Linux Landlock sandboxing

methods.

- Replacement functions for getopt_long() are used on platforms

that do not have it.

* Enabled unaligned access by default on PowerPC64LE and on RISC-V

targets that define __riscv_misaligned_fast.

* Tests:

- Added two new fuzz targets to OSS-Fuzz.

- Implemented Continuous Integration (CI) testing using

GitHub Actions.

* Changed quoting style from `...' to '...' in all messages,

scripts, and documentation.

* Added basic Codespell support to help catch typo errors.

5.4.6 (2024-01-26)

* Fixed a bug involving internal function pointers in liblzma not

being initialized to NULL. The bug can only be triggered if

lzma_filters_update() is called on a LZMA1 encoder, so it does

not affect xz or any application known to us that uses liblzma.

* xz:

- Fixed a regression introduced in 5.4.2 that caused encoding

in the raw format to unnecessarily fail if --suffix was not

used. For instance, the following command no longer reports

that --suffix must be used:

echo foo | xz --format=raw --lzma2 | wc -c

- Fixed an issue on MinGW-w64 builds that prevented reading

from or writing to non-terminal character devices like NUL.

* Added a new test.

5.4.5 (2023-11-01)

* liblzma:

- Use __attribute__((__no_sanitize_address__)) to avoid address

sanitization with CRC64 CLMUL. It uses 16-byte-aligned reads

which can extend past the bounds of the input buffer and

inherently trigger address sanitization errors. This isn't

a bug.

- Fixed an assertion failure that could be triggered by a large

unpadded_size argument. It was verified that there was no

other bug than the assertion failure.

- Fixed a bug that prevented building with Windows Vista

threading when __attribute__((__constructor__)) is not

supported.

* xz now properly handles special files such as "con" or "nul" on

Windows. Before this fix, the following wrote "foo" to the

console and deleted the input file "con_xz":

echo foo | xz > con_xz

xz --suffix=_xz --decompress con_xz

* Build systems:

- Allow builds with Windows win95 threading and small mode when

__attribute__((__constructor__)) is supported.

- Added a new line to liblzma.pc for MSYS2 (Windows):

Cflags.private: -DLZMA_API_STATIC

When compiling code that will link against static liblzma,

the LZMA_API_STATIC macro needs to be defined on Windows.

- CMake specific changes:

* Fixed a bug that allowed CLOCK_MONOTONIC to be used even

if the check for it failed.

* Fixed a bug where configuring CMake multiple times

resulted in HAVE_CLOCK_GETTIME and HAVE_CLOCK_MONOTONIC

not being set.

* Fixed the build with MinGW-w64-based Clang/LLVM 17.

llvm-windres now has more accurate GNU windres emulation

so the GNU windres workaround from 5.4.1 is needed with

llvm-windres version 17 too.

* The import library on Windows is now properly named

"liblzma.dll.a" instead of "libliblzma.dll.a"

* Fixed a bug causing the Ninja Generator to fail on

UNIX-like systems. This bug was introduced in 5.4.0.

* Added a new option to disable CLMUL CRC64.

* A module-definition (.def) file is now created when

building liblzma.dll with MinGW-w64.

* The pkg-config liblzma.pc file is now installed on all

builds except when using MSVC on Windows.

* Added large file support by default for platforms that

need it to handle files larger than 2 GiB. This includes

MinGW-w64, even 64-bit builds.

* Small fixes and improvements to the tests.

* Updated translations: Chinese (simplified) and Esperanto.

5.4.4 (2023-08-02)

* liblzma and xzdec can now build against WASI SDK when threading

support is disabled. xz and tests don't build yet.

* CMake:

- Fixed a bug preventing other projects from including liblzma

multiple times using find_package().

- Don't create broken symlinks in Cygwin and MSYS2 unless

supported by the environment. This prevented building for the

default MSYS2 environment. The problem was introduced in

xz 5.4.0.

* Documentation:

- Small improvements to man pages.

- Small improvements and typo fixes for liblzma API

documentation.

* Tests:

- Added a new section to INSTALL to describe basic test usage

and address recent questions about building the tests when

cross compiling.

- Small fixes and improvements to the tests.

* Translations:

- Fixed a mistake that caused one of the error messages to not

be translated. This only affected versions 5.4.2 and 5.4.3.

- Updated the Chinese (simplified), Croatian, Esperanto, German,

Korean, Polish, Romanian, Spanish, Swedish, Ukrainian, and

Vietnamese translations.

- Updated the German, Korean, Romanian, and Ukrainian man page

translations.

5.4.3 (2023-05-04)

* All fixes from 5.2.12

* Features in the CMake build can now be disabled as CMake cache

variables, similar to the Autotools build.

* Minor update to the Croatian translation.

5.4.2 (2023-03-18)

* All fixes from 5.2.11 that were not included in 5.4.1.

* If xz is built with support for the Capsicum sandbox but running

in an environment that doesn't support Capsicum, xz now runs

normally without sandboxing instead of exiting with an error.

* liblzma:

- Documentation was updated to improve the style, consistency,

and completeness of the liblzma API headers.

- The Doxygen-generated HTML documentation for the liblzma API

header files is now included in the source release and is

installed as part of "make install". All JavaScript is

removed to simplify license compliance and to reduce the

install size.

- Fixed a minor bug in lzma_str_from_filters() that produced

too many filters in the output string instead of reporting

an error if the input array had more than four filters. This

bug did not affect xz.

* Build systems:

- autogen.sh now invokes the doxygen tool via the new wrapper

script doxygen/update-doxygen, unless the command line option

--no-doxygen is used.

- Added microlzma_encoder.c and microlzma_decoder.c to the

VS project files for Windows and to the CMake build. These

should have been included in 5.3.2alpha.

* Tests:

- Added a test to the CMake build that was forgotten in the

previous release.

- Added and refactored a few tests.

* Translations:

- Updated the Brazilian Portuguese translation.

- Added Brazilian Portuguese man page translation.

5.4.1 (2023-01-11)

* liblzma:

- Fixed the return value of lzma_microlzma_encoder() if the

LZMA options lc/lp/pb are invalid. Invalid lc/lp/pb options

made the function return LZMA_STREAM_END without encoding

anything instead of returning LZMA_OPTIONS_ERROR.

- Windows / Visual Studio: Workaround a possible compiler bug

when targeting 32-bit x86 and compiling the CLMUL version of

the CRC64 code. The CLMUL code isn't enabled by the Windows

project files but it is in the CMake-based builds.

* Build systems:

- Windows-specific CMake changes:

* Don't try to enable CLMUL CRC64 code if _mm_set_epi64x()

isn't available. This fixes CMake-based build with Visual

Studio 2013.

* Created a workaround for a build failure with windres

from GNU binutils. It is used only when the C compiler

is GCC (not Clang). The workaround is incompatible

with llvm-windres, resulting in "XZx20Utils" instead

of "XZ Utils" in the resource file, but without the

workaround llvm-windres works correctly. See the

comment in CMakeLists.txt for details.

* Included the resource files in the xz and xzdec build

rules. Building the command line tools is still

experimental but possible with MinGW-w64.

- Visual Studio: Added stream_decoder_mt.c to the project

files. Now the threaded decompressor lzma_stream_decoder_mt()

gets built. CMake-based build wasn't affected.

- Updated windows/INSTALL-MSVC.txt to mention that CMake-based

build is now the preferred method with Visual Studio. The

project files will probably be removed after 5.4.x releases.

- Changes to #defines in config.h:

* HAVE_DECL_CLOCK_MONOTONIC was replaced by

HAVE_CLOCK_MONOTONIC. The old macro was always defined

in configure-generated config.h to either 0 or 1. The

new macro is defined (to 1) only if the declaration of

CLOCK_MONOTONIC is available. This matches the way most

other config.h macros work and makes things simpler with

other build systems.

* HAVE_DECL_PROGRAM_INVOCATION_NAME was replaced by

HAVE_PROGRAM_INVOCATION_NAME for the same reason.

* Tests:

- Fixed test script compatibility with ancient /bin/sh

versions. Now the five test_compress_* tests should

no longer fail on Solaris 10.

- Added and refactored a few tests.

* Translations:

- Updated the Catalan and Esperanto translations.

- Added Korean and Ukrainian man page translations.

5.4.0 (2022-12-13)

This bumps the minor version of liblzma because new features were

added. The API and ABI are still backward compatible with liblzma

5.2.x and 5.0.x.

Since 5.3.5beta:

* All fixes from 5.2.10.

* The ARM64 filter is now stable. The xz option is now --arm64.

Decompression requires XZ Utils 5.4.0. In the future the ARM64

filter will be supported by XZ for Java, XZ Embedded (including

the version in Linux), LZMA SDK, and 7-Zip.

* Translations:

- Updated Catalan, Croatian, German, Romanian, and Turkish

translations.

- Updated German man page translations.

- Added Romanian man page translations.

Summary of new features added in the 5.3.x development releases:

* liblzma:

- Added threaded .xz decompressor lzma_stream_decoder_mt().

It can use multiple threads with .xz files that have multiple

Blocks with size information in Block Headers. The threaded

encoder in xz has always created such files.

Single-threaded encoder cannot store the size information in

Block Headers even if one used LZMA_FULL_FLUSH to create

multiple Blocks, so this threaded decoder cannot use multiple

threads with such files.

If there are multiple Streams (concatenated .xz files), one

Stream will be decompressed completely before starting the

next Stream.

- A new decoder flag LZMA_FAIL_FAST was added. It makes the

threaded decompressor report errors soon instead of first

flushing all pending data before the error location.

- New Filter IDs:

* LZMA_FILTER_ARM64 is for ARM64 binaries.

* LZMA_FILTER_LZMA1EXT is for raw LZMA1 streams that don't

necessarily use the end marker.

- Added lzma_str_to_filters(), lzma_str_from_filters(), and

lzma_str_list_filters() to convert a preset or a filter chain

string to a lzma_filter[] and vice versa. These should make

it easier to write applications that allow users to specify

custom compression options.

- Added lzma_filters_free() which can be convenient for freeing

the filter options in a filter chain (an array of lzma_filter

structures).

- lzma_file_info_decoder() to makes it a little easier to get

the Index field from .xz files. This helps in getting the

uncompressed file size but an easy-to-use random access

API is still missing which has existed in XZ for Java for

a long time.

- Added lzma_microlzma_encoder() and lzma_microlzma_decoder().

It is used by erofs-utils and may be used by others too.

The MicroLZMA format is a raw LZMA stream (without end marker)

whose first byte (always 0x00) has been replaced with

bitwise-negation of the LZMA properties (lc/lp/pb). It was

created for use in EROFS but may be used in other contexts

as well where it is important to avoid wasting bytes for

stream headers or footers. The format is also supported by

XZ Embedded (the XZ Embedded version in Linux got MicroLZMA

support in Linux 5.16).

The MicroLZMA encoder API in liblzma can compress into a

fixed-sized output buffer so that as much data is compressed

as can be fit into the buffer while still creating a valid

MicroLZMA stream. This is needed for EROFS.

- Added lzma_lzip_decoder() to decompress the .lz (lzip) file

format version 0 and the original unextended version 1 files.

Also lzma_auto_decoder() supports .lz files.

- lzma_filters_update() can now be used with the multi-threaded

encoder (lzma_stream_encoder_mt()) to change the filter chain

after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH.

- In lzma_options_lzma, allow nice_len = 2 and 3 with the match

finders that require at least 3 or 4. Now it is internally

rounded up if needed.

- CLMUL-based CRC64 on x86-64 and E2K with runtime processor

detection. On 32-bit x86 it currently isn't available unless

--disable-assembler is used which can make the non-CLMUL

CRC64 slower; this might be fixed in the future.

- Building with --disable-threads --enable-small

is now thread-safe if the compiler supports

__attribute__((__constructor__)).

* xz:

- Using -T0 (--threads=0) will now use multi-threaded encoder

even on a single-core system. This is to ensure that output

from the same xz binary is identical on both single-core and

multi-core systems.

- --threads=+1 or -T+1 is now a way to put xz into

multi-threaded mode while using only one worker thread.

The + is ignored if the number is not 1.

- A default soft memory usage limit is now used for compression

when -T0 is used and no explicit limit has been specified.

This soft limit is used to restrict the number of threads

but if the limit is exceeded with even one thread then xz

will continue with one thread using the multi-threaded

encoder and this limit is ignored. If the number of threads

is specified manually then no default limit will be used;

this affects only -T0.

This change helps on systems that have very many cores and

using all of them for xz makes no sense. Previously xz -T0

could run out of memory on such systems because it attempted

to reserve memory for too many threads.

This also helps with 32-bit builds which don't have a large

amount of address space that would be required for many

threads. The default soft limit for -T0 is at most 1400 MiB

on all 32-bit platforms.

- Previously a low value in --memlimit-compress wouldn't cause

xz to switch from multi-threaded mode to single-threaded mode

if the limit cannot otherwise be met; xz failed instead. Now

xz can switch to single-threaded mode and then, if needed,

scale down the LZMA2 dictionary size too just like it already

did when it was started in single-threaded mode.

- The option --no-adjust no longer prevents xz from scaling down

the number of threads as that doesn't affect the compressed

output (only performance). Now --no-adjust only prevents

adjustments that affect compressed output, that is, with

--no-adjust xz won't switch from multi-threaded mode to

single-threaded mode and won't scale down the LZMA2

dictionary size.

- Added a new option --memlimit-mt-decompress=LIMIT. This is

used to limit the number of decompressor threads (possibly

falling back to single-threaded mode) but it will never make

xz refuse to decompress a file. This has a system-specific

default value because without any limit xz could end up

allocating memory for the whole compressed input file, the

whole uncompressed output file, multiple thread-specific

decompressor instances and so on. Basically xz could

attempt to use an insane amount of memory even with fairly

common files. The system-specific default value is currently

the same as the one used for compression with -T0.

The new option works together with the existing option

--memlimit-decompress=LIMIT. The old option sets a hard limit

that must not be exceeded (xz will refuse to decompress)

while the new option only restricts the number of threads.

If the limit set with --memlimit-mt-decompress is greater

than the limit set with --memlimit-compress, then the latter

value is used also for --memlimit-mt-decompress.

- Added new information to the output of xz --info-memory and

new fields to the output of xz --robot --info-memory.

- In --lzma2=nice=NUMBER allow 2 and 3 with all match finders

now that liblzma handles it.

- Don't mention endianness for ARM and ARM-Thumb filters in

--long-help. The filters only work for little endian

instruction encoding but modern ARM processors using

big endian data access still use little endian

instruction encoding. So the help text was misleading.

In contrast, the PowerPC filter is only for big endian

32/64-bit PowerPC code. Little endian PowerPC would need

a separate filter.

- Added decompression support for the .lz (lzip) file format

version 0 and the original unextended version 1. It is

autodetected by default. See also the option --format on

the xz man page.

- Sandboxing enabled by default:

* Capsicum (FreeBSD)

* pledge(2) (OpenBSD)

* Scripts now support the .lz format using xz.

* A few new tests were added.

* The liblzma-specific tests are now supported in CMake-based

builds too ("make test").

5.3.5beta (2022-12-01)

* All fixes from 5.2.9.

* liblzma:

- Added new LZMA_FILTER_LZMA1EXT for raw encoder and decoder to

handle raw LZMA1 streams that don't have end of payload marker

(EOPM) alias end of stream (EOS) marker. It can be used in

filter chains, for example, with the x86 BCJ filter.

- Added lzma_str_to_filters(), lzma_str_from_filters(), and

lzma_str_list_filters() to make it easier for applications

to get custom compression options from a user and convert

it to an array of lzma_filter structures.

- Added lzma_filters_free().

- lzma_filters_update() can now be used with the multi-threaded

encoder (lzma_stream_encoder_mt()) to change the filter chain

after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH.

- In lzma_options_lzma, allow nice_len = 2 and 3 with the match

finders that require at least 3 or 4. Now it is internally

rounded up if needed.

- ARM64 filter was modified. It is still experimental.

- Fixed LTO build with Clang if -fgnuc-version=10 or similar

was used to make Clang look like GCC >= 10. Now it uses

__has_attribute(__symver__) which should be reliable.

* xz:

- --threads=+1 or -T+1 is now a way to put xz into multi-threaded

mode while using only one worker thread.

- In --lzma2=nice=NUMBER allow 2 and 3 with all match finders

now that liblzma handles it.

* Updated translations: Chinese (simplified), Korean, and Turkish.

5.3.4alpha (2022-11-15)

* All fixes from 5.2.7 and 5.2.8.

* liblzma:

- Minor improvements to the threaded decoder.

- Added CRC64 implementation that uses SSSE3, SSE4.1, and CLMUL

instructions on 32/64-bit x86 and E2K. On 32-bit x86 it's

not enabled unless --disable-assembler is used but then

the non-CLMUL code might be slower. Processor support is

detected at runtime so this is built by default on x86-64

and E2K. On these platforms, if compiler flags indicate

unconditional CLMUL support (-msse4.1 -mpclmul) then the

generic version is not built, making liblzma 8-9 KiB smaller

compared to having both versions included.

With extremely compressible files this can make decompression

up to twice as fast but with typical files 5 % improvement

is a more realistic expectation.

The CLMUL version is slower than the generic version with

tiny inputs (especially at 1-8 bytes per call, but up to

16 bytes). In normal use in xz this doesn't matter at all.

- Added an experimental ARM64 filter. This is *not* the final

version! Files created with this experimental version won't

be supported in the future versions! The filter design is

a compromise where improving one use case makes some other

cases worse.

- Added decompression support for the .lz (lzip) file format

version 0 and the original unextended version 1. See the

API docs of lzma_lzip_decoder() for details. Also

lzma_auto_decoder() supports .lz files.

- Building with --disable-threads --enable-small

is now thread-safe if the compiler supports

__attribute__((__constructor__))

* xz:

- Added support for OpenBSD's pledge(2) as a sandboxing method.

- Don't mention endianness for ARM and ARM-Thumb filters in

--long-help. The filters only work for little endian

instruction encoding but modern ARM processors using

big endian data access still use little endian

instruction encoding. So the help text was misleading.

In contrast, the PowerPC filter is only for big endian

32/64-bit PowerPC code. Little endian PowerPC would need

a separate filter.

- Added --experimental-arm64. This will be renamed once the

filter is finished. Files created with this experimental

filter will not be supported in the future!

- Added new fields to the output of xz --robot --info-memory.

- Added decompression support for the .lz (lzip) file format

version 0 and the original unextended version 1. It is

autodetected by default. See also the option --format on

the xz man page.

* Scripts now support the .lz format using xz.

* Build systems:

- New #defines in config.h: HAVE_ENCODER_ARM64,

HAVE_DECODER_ARM64, HAVE_LZIP_DECODER, HAVE_CPUID_H,

HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR, HAVE_USABLE_CLMUL

- New configure options: --disable-clmul-crc,

--disable-microlzma, --disable-lzip-decoder, and

'pledge' is now an option in --enable-sandbox (but

it's autodetected by default anyway).

- INSTALL was updated to document the new configure options.

- PACKAGERS now lists also --disable-microlzma and

--disable-lzip-decoder as configure options that must

not be used in builds for non-embedded use.

* Tests:

- Fix some of the tests so that they skip instead of fail if

certain features have been disabled with configure options.

It's still not perfect.

- Other improvements to tests.

* Updated translations: Croatian, Finnish, Hungarian, Polish,

Romanian, Spanish, Swedish, and Ukrainian.

5.3.3alpha (2022-08-22)

* All fixes from 5.2.6.

* liblzma:

- Fixed 32-bit build.

- Added threaded .xz decompressor lzma_stream_decoder_mt().

It can use multiple threads with .xz files that have multiple

Blocks with size information in Block Headers. The threaded

encoder in xz has always created such files.

Single-threaded encoder cannot store the size information in

Block Headers even if one used LZMA_FULL_FLUSH to create

multiple Blocks, so this threaded decoder cannot use multiple

threads with such files.

If there are multiple Streams (concatenated .xz files), one

Stream will be decompressed completely before starting the

next Stream.

- A new decoder flag LZMA_FAIL_FAST was added. It makes the

threaded decompressor report errors soon instead of first

flushing all pending data before the error location.

* xz:

- Using -T0 (--threads=0) will now use multi-threaded encoder

even on a single-core system. This is to ensure that output

from the same xz binary is identical on both single-core and

multi-core systems.

- A default soft memory usage limit is now used for compression

when -T0 is used and no explicit limit has been specified.

This soft limit is used to restrict the number of threads

but if the limit is exceeded with even one thread then xz

will continue with one thread using the multi-threaded

encoder and this limit is ignored. If the number of threads

is specified manually then no default limit will be used;

this affects only -T0.

This change helps on systems that have very many cores and

using all of them for xz makes no sense. Previously xz -T0

could run out of memory on such systems because it attempted

to reserve memory for too many threads.

This also helps with 32-bit builds which don't have a large

amount of address space that would be required for many

threads. The default limit is 1400 MiB on all 32-bit

platforms with -T0.

Now xz -T0 should just work. It might use too few threads

in some cases but at least it shouldn't easily run out of

memory. It's possible that this will be tweaked before 5.4.0.

- Changes to --memlimit-compress and --no-adjust:

In single-threaded mode, --memlimit-compress can make xz

scale down the LZMA2 dictionary size to meet the memory usage

limit. This obviously affects the compressed output. However,

if xz was in threaded mode, --memlimit-compress could make xz

reduce the number of threads but it wouldn't make xz switch

from multi-threaded mode to single-threaded mode or scale

down the LZMA2 dictionary size. This seemed illogical.

Now --memlimit-compress can make xz switch to single-threaded

mode if one thread in multi-threaded mode uses too much

memory. If memory usage is still too high, then the LZMA2