Compression is built into countless systems, either visible or internally. There are countless compression algorithms and file formats surrounding them, making the landscape confusing to navigate.
What we cover
This article will discuss only commonly used general purpose compression. This means that algorithms must be capable of compressing text, media files or arbitrary bytes, and reproduce the exact input byte for byte after decompression.
These requirements naturally exclude specialized compression such as those built into video codecs or image pixel formats, or outdated/replaced ones like lzw or older lzma predecessors.
The overview will cover practical points only, no implementation details or maths explanations.
Compression algorithms
Compression algorithms are the fundamental process turning input bytes into a smaller amount of output bytes that can be turned back into the original input when decompressed. It does not know or care what that input data may be, where it came from or how it is stored in any way.
Algorithms are compared on five metrics: compression ratio (how much smaller is the compression output compared to the input), compression/decompression cost (how much cpu/memory they consume) and compression/decompression speed (how fast data can be compressed/uncompressed). The values given are relative to the other algorithms in the list and results will vary widely based on used hardware, parameters and input data type. Treat them as a rough guideline, not absolute fact.
deflate
compression ratio: medium | compression: moderate cost / fast | decompression: low cost / very fast
This is easily the most used algorithm in this list. While it is quite old and offers neither the fastest speeds nor the best compression ratio, it is fairly simple to implement, works decently on weak hardware and has been around for decades. All these factors make it the most supported compression algorithm in existence, which is why it was even built into networking protocols like HTTP, since countless difference client devices needed to support a single compression algorithm, and deflate was the only realistic choice.
It is still widely used today, especially when compressed data needs to be transferred between devices.
bzip2
compression ratio: high | compression: high cost / slow | decompression: moderate cost / moderate speed
Initially designed as an alternative to deflate that trades higher CPU usage for better compression ratios, it is increasingly replaced by lzma2. It is still used around the UNIX ecosystem to compress larger files before distribution over the internet or physical media, mostly programs related files like manuals.
brotli
compression ratio: high | compression: high cost / moderate speed | decompression: moderate cost / very fast
The newest algorithm in this list, developed by Google specifically for better compression of website assets. The tradeoff was to use more resources for better compression ratio, since computer hardware had become much faster and the bottleneck increasingly shifted to network bandwidth. It was quickly adopted in protocols like HTTP and is increasingly becoming the default for web compression.
lz4
compression ratio: low | compression: very low cost / extremely fast | decompression: very low cost / extremely fast
This algorithm is built for lightweight speed. It is aimed to provide some form of compression with near zero impact on resources or latency, making it a great use case inside databases and caches that need fast reads and writes, but can still save significant storage space on repetitive data using lz4. It is mostly used internally by programs or accessed as a library, offering only low-hanging fruit space savings at near zero cost.
zstd
compression ratio: high | compression: moderate cost / very fast | decompression: moderate cost / extremely fast
A modern general purpose compression algorithm trading much better compression ratios for slightly higher resource usage than deflate, but at significantly faster compression speed and decompression speeds close to lz4. It is a good fit for modern general purpose compression with good compression ratios at low cost. It is also a sensible choice for modern read-heavy workloads like transactional databases, where space savings matter, decompression speed shines and the slightly slower compression speed is not a bottleneck.
lzma2
compression ratio: very high | compression: very high cost / very slow | decompression: high cost, moderate speed
Built for maximum compression ratio at the cost of resource usage and speed, the lzma2 algorithm is best suited for long-term storage or archival scenarios, or to compress large amounts of data for distribution or syncing across networks.
Its high resource usage makes it a bad choice for hot storage systems or databases, instead lending itself to cold-storage use cases.
Streaming formats
When using tools to compress files or serving compressed content over the network, a file format wrapper is added on top of the raw compressed data dump. Formats add metadata like algorithm parameters used, checksums to catch data corruption and sometimes total decompressed data size.
They are built as streaming formats, meaning a single sequence of formatted bytes may contain multiple compressed data streams. What each stream contains or represents is not standardized - an application could choose to compress multiple files in a single stream, or split a single input into multiple streams for parallel decompression.
Compression formats do not understand directory trees, file permissions or other filesystem abstractions - they still operate on and store arbitrary data.
When compressing single files, you typically add the compression format extension to the filename, for example when compressing readme.txt with gzip, the output would be named readme.txt.gz to indicate a gzip-compressed text file.
For file compression, these are the most common format extensions:
.gz
The classic output when compressing a file with gzip, using the deflate compression algorithm. It is still very popular especially on linux systems for backward-compatible compression needs. The gzip format includes a lot of optional metadata fields for filenames, timestamps, source operating system etc.
.bz2
Developed to store bzip2 compressed files, it is mostly aimed at unix environments trading better compression for higher resource usage, it includes only minimal checksumming and largely ignores optional metadata support. Use cases revolved around linux systems that want better compression ratios without the full cost of lzma2, for example when compressing large text based log files.
.br
A brotli compressed file. This is an outlier, as brotli produces only stream format output and there is no distinction between the output of the algorithm and its representation inside a file. However, the format was designed for serving over a web context, so it lacks a magic number sequence at the start of the file that would allow tools to detect the real file type even when the extension was lost.
.lz4
Although noted as a file extension for consistency, you will likely never see this in a real system. The lz4 algorithm does provide its own minimal streaming format, omitting all metadata and even making checksums optional to yield best in class decompression speeds, even on the fly. Some networked services may use it to transfer lz4 compressed data to storage replicas or sync data between cluster members. Exposing the format to end users happens almost never.
.zst
Contains a stream of zstandard blocks, called "frames". The format is optimized for streaming and on-the-fly decoding, with more flexibility around frame-handling. For example, the zstd command may decide that a block is not worth compressing and store it raw/unaltered in a frame within the stream, allowing the reader to skip decompressing the entire frame on read.
.xz
The modern xz compression tool uses this format to encode lzma2 compressed data. This format is the most feature-rich in the list, offering filter support to reshape data into more repetitive patterns before compression (storing sequences as diffs/deltas, rewriting memory address jump/call addresses in executables etc). Additionally, compressed content is split into blocks with per-block checksums, with support for multiple checksum hashes.
It doesn't have optional fields for source information like original filenames or timestamps.
Single file compression is most prominent in unix environments (linux. bsd, mac) and rare to see on Windows operating systems, which instead leans heavier on archives.
Raw Archive formats
In order to store multiple files in a single file, you need an archive file format. Their goal is to store one or more files, potentially entire directory trees and metadata, in a single output file that can be expanded back into the original structure (unpacked) as needed.
Since raw archives have no builtin support for compression in any way, you need to compress their output with another tool if compression is required, producing double file extensions like .tar.gz for a gzip-compressed tar archive or, .tar.xz for a tar archive compressed with the xz tool. While this design requires two separate tools to unpack the archive, it also makes tar compatible with any existing and future compression formats without changing the tar format or tooling.
For specialized archives, ar stands out as the only widely used option. If you have ever compiled an executable or library statically, or seen a .a file on a unix system, you have been in contact with this archive format. It essentially exists to bundle multiple compiled object files (.o) into a single file (.a), which ld can then link to an executable at runtime. Since only compiled function libraries are supposed to be archived by ar, it has no support for directory hierarchies, file metadata or other file types like devices or links.
To archive a complete directory tree, including subdirectories, files and their metadata like permissions and timestamps, tar is the most popular format on unix systems. It aims for the simplest possible way to preserve and entire filesystem hierarchy into a single file, even including edge cases like extended ACL permissions, device files and symbolic links.
Much less widely known is the cpio family, consisting of multiple different formats. In terms of features they are basically equal to tar, except with builtin kernel support and a focus on streaming files in and out. They are still used by modern machines, but in places that most users don't come in direct contact with, like the initramfs involved in every modern linux boot sequence, or as the packaging format for rpm packages on RHEL family linux systems.
Compressed archive formats
Compressed archive formats combine archival with a set of compression algorithms into a single format. The primary advantage is portability: A system that understands a compressed archive does not require another tool to handle the compression and has no external dependencies other than the archival tool itself to pack or unpack archived contents.
The primary advantage to compressing raw archives is that files can be read/accessed individually without decompressing the entire archive, yielding much better random file access.
The most common ones are:
.zip
One of the longest standing compressed archive formats in existence, and easily the most well-supported across operating systems. Nearly any environment has support for working with zip compressed archives, making it the obvious choice when sharing files between varying device and OS combinations. While it does support multiple algorithms, it typically sticks to DEFLATE since that is the only one reliably implemented by all clients, limiting its potential compression ratio. The original zip implementation was 32bit and had a hard limit of 4GiB file size, now replaced by zip64 implementations supporting up to 16EB (1024TB). Contained files are compressed individually, making individual file access trivial at the expensive of some compression ratio, which is why .zip files tend to be slightly larger than .tar.gz archives of the same content even when they use the same algorithm and parameters.
.rar
The only proprietary compression algorithm and archive format in this list. It was built as a competitor to zip offering better compression ratios, although those have since been overtaken by the lzma algorithm family. It was not widely supported and primarily used to share large files, especially because of its builtin recovery metadata, allowing some data corruption to be repaired locally. In an environment of growing file sizes and limited network bandwidth it flourished, now increasingly losing relevance as networks and disk speeds catch up. Files are combined into a single binary stream and chunked into blocks for better compression ratio efficiency, while allowing access to individual files without decompressing the entire archive (at worst one block decompression overhead).
.7z
The modern exchange format, primarily using the lzma2 and zstd algorithms, often defaulting to lzma2 for superb compression ratios at the cost of speed. While it may never catch up to the old zip archives in terms of compatibility, it has become more and more relevant for long-term storage or use cases that prefer compression ratios over speed like backups. Same as rar, it benefits from compressing multiple files as a single binary stream chunked into blocks while maintaining metadata to allow random file access without decompressing the entire archive.