Random-access compression and decompression. More...

Data Structures
struct	zxc_reader_t
	Storage-agnostic reader interface for seekable archives. More...

Typedefs
typedef struct zxc_seekable_s	zxc_seekable
	Opaque handle for a seekable ZXC archive.

Functions
zxc_seekable *	zxc_seekable_open (const void *src, const size_t src_size)
	Opens a seekable archive from a memory buffer.

zxc_seekable *	zxc_seekable_open_reader (const zxc_reader_t *r)
	Opens a seekable archive through a user-supplied reader.

uint64_t	zxc_seekable_get_num_blocks (const zxc_seekable *s)
	Returns the total number of blocks in the seekable archive.

uint64_t	zxc_seekable_get_decompressed_size (const zxc_seekable *s)
	Returns the total decompressed size of the seekable archive.

uint32_t	zxc_seekable_get_block_comp_size (const zxc_seekable *s, const uint64_t block_idx)
	Returns the compressed size of a specific block.

uint32_t	zxc_seekable_get_block_decomp_size (const zxc_seekable *s, const uint64_t block_idx)
	Returns the decompressed size of a specific block.

int64_t	zxc_seekable_decompress_range (zxc_seekable s, void dst, const size_t dst_capacity, const uint64_t offset, const size_t len)
	Decompresses an arbitrary byte range of the original data.

int64_t	zxc_seekable_decompress_range_mt (zxc_seekable s, void dst, const size_t dst_capacity, const uint64_t offset, const size_t len, int n_threads)
	Multi-threaded variant of zxc_seekable_decompress_range().

void	zxc_seekable_free (zxc_seekable *s)
	Frees a seekable handle and all associated resources.

int	zxc_seekable_set_checksum (zxc_seekable *s, int enabled)
	Turns per-block checksum verification on or off.

int	zxc_seekable_set_dict (zxc_seekable s, const void dict, size_t dict_size, const void *dict_huf)
	Attaches a pre-trained dictionary to a seekable handle.

int64_t	zxc_write_seek_table (uint8_t dst, const size_t dst_capacity, const uint32_t comp_sizes, const uint64_t num_blocks)
	Writes a seek table to the destination buffer.

size_t	zxc_seek_table_size (const uint64_t num_blocks)
	Returns the encoded size of a seek table for the given block count.

Detailed Description

Random-access compression and decompression.

Typedef Documentation

◆ zxc_seekable

typedef struct zxc_seekable_s zxc_seekable

Opaque handle for a seekable ZXC archive.

Created by zxc_seekable_open(), zxc_seekable_open_reader(), or zxc_seekable_open_file() (see zxc_stream.h). Free with zxc_seekable_free().

Definition at line 88 of file zxc_seekable.h.

Function Documentation

◆ zxc_seek_table_size()

size_t zxc_seek_table_size ( const uint64_t num_blocks )

Returns the encoded size of a seek table for the given block count.

Parameters

[in] num_blocks Number of blocks.

Returns: Total byte size of the seek table, or 0 when it does not fit size_t.

Returns the encoded size of a seek table for the given block count.

Public API; sizes the destination of zxc_write_seek_table. 0 when the table does not fit size_t (32-bit hosts) or num_blocks is past what one can describe.

Definition at line 731 of file zxc_common.c.

References ZXC_BLOCK_HEADER_SIZE, ZXC_SEEK_ANCHOR_SIZE, and zxc_seek_table_bytes().

Referenced by zxc_write_seek_table(), and zxc_write_seek_table_from_blocks().

◆ zxc_seekable_decompress_range()

int64_t zxc_seekable_decompress_range	(	zxc_seekable *	s,
		void *	dst,
		const size_t	dst_capacity,
		const uint64_t	offset,
		const size_t	len
	)

Decompresses an arbitrary byte range of the original data.

The core random-access primitive: only the blocks overlapping [offset, offset + len) are read and decompressed.

Checksums: Per-block checksums are not verified unless zxc_seekable_set_checksum was called with a non-zero argument; a mismatch then returns ZXC_ERROR_BAD_CHECKSUM. Without it, a forged seek table can return another block's bytes with no error.

Parameters

[in,out]	s	Seekable handle.
[out]	dst	Destination buffer.
[in]	dst_capacity	Capacity of `dst` (must be >= `len`).
[in]	offset	Byte offset into the original uncompressed data.
[in]	len	Number of bytes to decompress.

Returns: Number of bytes written to dst (== len on success), or a negative zxc_error_t code on failure.

Decompresses an arbitrary byte range of the original data.

Public API; full contract in zxc_seekable.h. Maps the range to its blocks (a frame lookup, then a division), decodes each through a reusable, lazily-initialised, dictionary-aware context, and copies out only the requested sub-range. Single-threaded; see zxc_seekable_decompress_range_mt for the parallel variant.

Definition at line 553 of file zxc_seekable.c.

References zxc_cctx_init(), zxc_decompress_chunk_wrapper(), ZXC_DECOMPRESS_TAIL_PAD, ZXC_ERROR_CORRUPT_DATA, ZXC_ERROR_DST_TOO_SMALL, ZXC_ERROR_MEMORY, ZXC_ERROR_NULL_INPUT, ZXC_ERROR_SRC_TOO_SMALL, ZXC_OK, ZXC_PAD_SIZE, ZXC_REALLOC, zxc_seek_block_of_offset(), zxc_seek_decomp_size(), zxc_seek_frame_of_block(), ZXC_SEEK_GROUP, ZXC_SEEK_GROUP_BYTES, zxc_seek_load_spans(), zxc_seek_read_block(), ZXC_SEEK_TAIL_HDRS, and zxc_seek_use_dict().

Referenced by zxc_seekable_decompress_range_mt().

◆ zxc_seekable_decompress_range_mt()

int64_t zxc_seekable_decompress_range_mt	(	zxc_seekable *	s,
		void *	dst,
		const size_t	dst_capacity,
		const uint64_t	offset,
		const size_t	len,
		int	n_threads
	)

Multi-threaded variant of zxc_seekable_decompress_range().

Decompresses blocks in parallel over a fork-join pool. Each worker owns its decompression context and reads through pread() (POSIX) or ReadFile() (Windows), so the I/O stays lock-free.

Falls back to the single-threaded path when n_threads <= 1, the range fits in one block, or it spans more than 2^32 blocks (16 TiB of dst). Same checksum rule as zxc_seekable_decompress_range.

Parameters

[in,out]	s	Seekable handle.
[out]	dst	Destination buffer.
[in]	dst_capacity	Capacity of `dst` (must be >= `len`).
[in]	offset	Byte offset into the original uncompressed data.
[in]	len	Number of bytes to decompress.
[in]	n_threads	Number of worker threads (0 = auto-detect CPU cores).

Returns: Number of bytes written to dst (== len on success), or a negative zxc_error_t code on failure.

Multi-threaded variant of zxc_seekable_decompress_range().

Public API; full contract in zxc_seekable.h. Plans one job per covered block (each with its own thread-local context and read buffer) and runs them fork-join in waves of up to n_threads. Falls back to the single-threaded path for trivial spans. n_threads == 0 auto-detects the core count.

Definition at line 807 of file zxc_seekable.c.

References ZXC_CALLOC, ZXC_ERROR_DST_TOO_SMALL, ZXC_ERROR_MEMORY, ZXC_ERROR_NULL_INPUT, ZXC_ERROR_SRC_TOO_SMALL, ZXC_FREE, ZXC_MALLOC, ZXC_MAX_THREADS, zxc_num_procs(), ZXC_OK, zxc_seek_block_of_offset(), zxc_seek_decomp_size(), zxc_seek_frame_of_block(), zxc_seek_load_spans(), zxc_seek_mt_worker(), zxc_seek_spans_raw_max(), and zxc_seekable_decompress_range().

◆ zxc_seekable_free()

void zxc_seekable_free ( zxc_seekable * s )

Frees a seekable handle and all associated resources.

Safe to call with NULL.

Parameters

[in] s Handle to free.

Frees a seekable handle and all associated resources.

Public API; see zxc_seekable.h. Tears down the reusable context, the owned dictionary copy and any attached reader context. NULL-safe.

Definition at line 947 of file zxc_seekable.c.

References zxc_cctx_free(), and ZXC_FREE.

◆ zxc_seekable_get_block_comp_size()

uint32_t zxc_seekable_get_block_comp_size	(	const zxc_seekable *	s,
		const uint64_t	block_idx
	)

Returns the compressed size of a specific block.

The on-disk size: block header + payload + optional per-block checksum, from the block's seek table group: one read per call. Checked against bounds only: a forged size within them comes back as is.

Parameters

[in]	s	Seekable handle.
[in]	block_idx	Zero-based block index.

Returns: Compressed block size, or 0 if block_idx is out of range or its group is unreadable or invalid.

Returns the compressed size of a specific block.

Definition at line 447 of file zxc_seekable.c.

References ZXC_OK, zxc_seek_frame_of_block(), ZXC_SEEK_GROUP_BYTES, zxc_seek_load_spans(), and ZXC_SEEK_TAIL_HDRS.

◆ zxc_seekable_get_block_decomp_size()

uint32_t zxc_seekable_get_block_decomp_size	(	const zxc_seekable *	s,
		const uint64_t	block_idx
	)

Returns the decompressed size of a specific block.

Parameters

[in]	s	Seekable handle.
[in]	block_idx	Zero-based block index.

Returns: Decompressed block size, or 0 if block_idx is out of range.

Returns the decompressed size of a specific block.

Definition at line 478 of file zxc_seekable.c.

References zxc_seek_decomp_size(), and zxc_seek_frame_of_block().

◆ zxc_seekable_get_decompressed_size()

uint64_t zxc_seekable_get_decompressed_size ( const zxc_seekable * s )

Returns the total decompressed size of the seekable archive.

Parameters

[in] s Seekable handle.

Returns: Total decompressed size in bytes.

Returns the total decompressed size of the seekable archive.

Definition at line 440 of file zxc_seekable.c.

◆ zxc_seekable_get_num_blocks()

uint64_t zxc_seekable_get_num_blocks ( const zxc_seekable * s )

Returns the total number of blocks in the seekable archive.

The count is derived from the footer, not stored, so it is bounded by the archive size and not by any field: 64 bits here.

Parameters

[in] s Seekable handle.

Returns: Number of data blocks (excluding EOF).

Returns the total number of blocks in the seekable archive.

Definition at line 435 of file zxc_seekable.c.

◆ zxc_seekable_open()

zxc_seekable * zxc_seekable_open	(	const void *	src,
		const size_t	src_size
	)

Opens a seekable archive from a memory buffer.

Locates and validates the seek table at the end of the buffer; entries are read from src on access. src must stay valid for the lifetime of the handle. An empty seekable archive opens with 0 blocks: any non-empty range is then ZXC_ERROR_SRC_TOO_SMALL.

Parameters

[in]	src	Pointer to the compressed data.
[in]	src_size	Size of the compressed data in bytes.

Returns: Handle on success, or NULL if the buffer does not contain a valid seekable archive (e.g. missing seek block, bad block type).

Opens a seekable archive from a memory buffer.

Public API; see zxc_seekable.h. Thin guard around zxc_seekable_parse, which detects and validates the trailing seek table.

Definition at line 409 of file zxc_seekable.c.

References zxc_seekable_parse().

◆ zxc_seekable_open_reader()

zxc_seekable * zxc_seekable_open_reader ( const zxc_reader_t * r )

Opens a seekable archive through a user-supplied reader.

Two reads per frame at open (at most 2^20 frames), plus the last frame's EOF/SEK headers; per range, its table groups and every block decoded. This is the entry point for backing the seekable API with any storage that does positional reads (mmap, HTTP, S3, a kernel file descriptor).

Parameters

[in] r Reader interface (must remain valid for the handle lifetime).

Returns: Handle (0 blocks if the archive is empty), or NULL on error.

Opens a seekable archive through a user-supplied reader.

Public API; see zxc_seekable.h. Reads the file header, footer and the EOF/SEK block headers through r->read_at (the FILE* variant wraps pread this way). The archive is never mapped whole; entries are read as blocks are accessed.

Definition at line 426 of file zxc_seekable.c.

References zxc_reader_t::read_at, zxc_reader_t::size, and zxc_seekable_parse().

Referenced by zxc_seekable_open_file().

◆ zxc_seekable_set_checksum()

int zxc_seekable_set_checksum	(	zxc_seekable *	s,
		const int	enabled
	)

Turns per-block checksum verification on or off.

Off by default, as in the frame API. No effect without checksums in the archive. Applies from the next call, on both paths. Checksums are seeded with each block's position, so this also binds blocks to their index, which the seek table cannot.

Parameters

[in,out]	s	Seekable handle.
[in]	enabled	Non-zero to verify, 0 to skip.

Returns: ZXC_OK, or ZXC_ERROR_NULL_INPUT if s is NULL.

Public API; see zxc_seekable.h. Only records the wish; the decode path pushes it into the context on every call, like dict_size.

Definition at line 964 of file zxc_seekable.c.

References ZXC_ERROR_NULL_INPUT, and ZXC_OK.

◆ zxc_seekable_set_dict()

int zxc_seekable_set_dict	(	zxc_seekable *	s,
		const void *	dict,
		const size_t	dict_size,
		const void *	dict_huf
	)

Attaches a pre-trained dictionary to a seekable handle.

Content and table are copied, so the caller may free them once this returns. Call it once per dictionary the frames use; a range through a frame whose dictionary is missing returns ZXC_ERROR_DICT_REQUIRED.

Parameters

[in]	s	Seekable handle.
[in]	dict	Dictionary content.
[in]	dict_size	Size in bytes (max ZXC_DICT_SIZE_MAX).
[in]	dict_huf	Shared literal Huffman table (128 bytes, see zxc_dict_huf()), or NULL if the archive was compressed without one. Must be the compression-time table: the archive's dict_id binds the (dict, table) pair. A pair no frame uses: ZXC_ERROR_DICT_MISMATCH, on this call, unless no frame needs a dictionary: then it is ignored.

Returns: ZXC_OK on success, or a negative zxc_error_t code.

Attaches a pre-trained dictionary to a seekable handle.

Public API; full contract in zxc_seekable.h. Validates the dict_id against the file header, then takes an owned copy of dict (and the optional shared literal Huffman table dict_huf). Drops any context already built so the [dict | decode] bounce buffer is re-carved on the next decompress.

Definition at line 978 of file zxc_seekable.c.

References zxc_cctx_free(), zxc_dict_id(), ZXC_DICT_SIZE_MAX, ZXC_ERROR_DICT_MISMATCH, ZXC_ERROR_DICT_TOO_LARGE, ZXC_ERROR_MEMORY, ZXC_ERROR_NULL_INPUT, ZXC_FREE, ZXC_HUF_TABLE_SIZE, ZXC_MALLOC, ZXC_OK, and ZXC_REALLOC.

◆ zxc_write_seek_table()

int64_t zxc_write_seek_table	(	uint8_t *	dst,
		const size_t	dst_capacity,
		const uint32_t *	comp_sizes,
		const uint64_t	num_blocks
	)

Writes a seek table to the destination buffer.

Low-level helper used by the seekable compression paths. Layout is block_header(8) + groups of 64 blocks, each a u64 anchor (its first block's offset) then one u32 on-disk size per block; the header's size field holds the groups' byte size folded to 32 bits, exact below 4 GiB.

Parameters

[out]	dst	Destination buffer.
[in]	dst_capacity	Capacity of `dst` in bytes.
[in]	comp_sizes	Array of compressed block sizes.
[in]	num_blocks	Number of blocks.

Returns: Number of bytes written, or a negative zxc_error_t on failure.

Writes a seek table to the destination buffer.

Public API; contract in zxc_seekable.h. Block header, then the groups.

Definition at line 786 of file zxc_common.c.

References ZXC_ERROR_DST_TOO_SMALL, ZXC_ERROR_NULL_INPUT, ZXC_ERROR_OVERFLOW, ZXC_FILE_HEADER_SIZE, ZXC_SEEK_GROUP, zxc_seek_group_len(), zxc_seek_table_header(), zxc_seek_table_size(), and zxc_seek_write_group().

Referenced by zxc_compress().

Data Structures

Typedefs

Functions

Detailed Description

Typedef Documentation

◆ zxc_seekable

Function Documentation

◆ zxc_seek_table_size()

◆ zxc_seekable_decompress_range()

◆ zxc_seekable_decompress_range_mt()

◆ zxc_seekable_free()

◆ zxc_seekable_get_block_comp_size()

◆ zxc_seekable_get_block_decomp_size()

◆ zxc_seekable_get_decompressed_size()

◆ zxc_seekable_get_num_blocks()

◆ zxc_seekable_open()

◆ zxc_seekable_open_reader()

◆ zxc_seekable_set_checksum()

◆ zxc_seekable_set_dict()

◆ zxc_write_seek_table()