zxc_seekable.c File Reference

Seekable archive reader (random-access decompression) and seek table writer. More...

#include "../../include/zxc_seekable.h"
#include "../../include/zxc_dict.h"
#include "../../include/zxc_error.h"
#include "zxc_internal.h"
#include "zxc_threads.h"

Go to the source code of this file.

Macros
#define	ZXC_SEEK_MAX_FRAMES   (1U << 20)
	Most frames a handle indexes: bounds the reads and memory of an open.
#define	ZXC_SEEK_TAIL_HDRS   (2 * ZXC_BLOCK_HEADER_SIZE)
	EOF and SEK block headers, ahead of a frame's table: checked when it is read.

Functions
static int	zxc_seek_source_read (const zxc_seek_source_t *src, void *dst, const size_t len, const uint64_t off)
	Reads a byte range from the archive, whatever backs it: the one bounds check every parsed offset goes through, written so it cannot wrap.
static zxc_seek_source_t	zxc_seek_source_of (const zxc_seekable *s)
	The handle's archive as a read source, for the per-block readers.
static int	zxc_seek_scan_read (const zxc_scan_src_t *scan, const uint64_t off, void *dst, const size_t len)
	zxc_scan_src_t over a zxc_seek_source_t.
static int	zxc_seek_parse_frame (zxc_seek_source_t *src, const uint64_t end, zxc_seek_frame_t *f)
	Validates the seekable frame that ends at end, in two reads: header and footer as zxc_get_last_frame_info() does.
static zxc_seekable *	zxc_seekable_parse (const zxc_seek_source_t *source)
	Walks the frames back from the end through zxc_seek_parse_frame, then numbers their bytes and blocks.
static const zxc_seek_frame_t *	zxc_seek_frame_of_block (const zxc_seekable *s, const uint64_t idx)
	The frame holding archive-wide block idx, below the block count.
static uint64_t	zxc_seek_block_of_offset (const zxc_seekable *s, const uint64_t offset)
	Archive-wide index of the block holding decompressed offset, below the size.
static size_t	zxc_seek_spans_raw_max (const uint64_t first, const uint32_t n)
	Scratch bound for zxc_seek_load_spans(): the groups blocks [first, first + n) touch.
static int	zxc_seek_load_spans (const zxc_seekable *s, const zxc_seek_frame_t *f, const uint64_t first, const uint32_t n, uint64_t *starts, uint32_t *sizes, uint8_t *raw)
	Loads where each block of [first, first + n) starts and its on-disk size, both taken from the block's own group.
zxc_seekable *	zxc_seekable_open (const void *src, const size_t src_size)
	Opens a seekable archive held entirely in a memory buffer.
zxc_seekable *	zxc_seekable_open_reader (const zxc_reader_t *r)
	Opens a seekable archive over a caller-supplied random-access reader.
uint64_t	zxc_seekable_get_num_blocks (const zxc_seekable *s)
	Number of blocks in the archive.
uint64_t	zxc_seekable_get_decompressed_size (const zxc_seekable *s)
	Total decompressed size of the archive.
uint32_t	zxc_seekable_get_block_comp_size (const zxc_seekable *s, const uint64_t block_idx)
	Compressed byte size of a given block, from its seek table entry.
static uint32_t	zxc_seek_decomp_size (const uint32_t block_size, const uint64_t total_decomp, const uint64_t idx)
	Decompressed size of block idx (O(1)).
uint32_t	zxc_seekable_get_block_decomp_size (const zxc_seekable *s, const uint64_t block_idx)
	Decompressed byte size of a given block.
static int	zxc_seek_use_dict (const zxc_seekable *s, zxc_cctx_t *ctx, const zxc_seek_dict_t **held, const zxc_seek_frame_t *f)
	Gives ctx frame f's dictionary, copied in unless *held, the one it holds, is already that one.
static int	zxc_seek_read_block (const zxc_seekable *s, const uint64_t off, const uint32_t csz, const int has_cs, uint8_t *buf, const size_t buf_cap)
	Reads a compressed block into buf from the memory buffer or reader.
int64_t	zxc_seekable_decompress_range (zxc_seekable *s, void *dst, const size_t dst_capacity, const uint64_t offset, const size_t len)
	Decompresses the byte range [offset, offset + len) into dst.
static void	zxc_seek_mt_fail_stripe (const zxc_seek_mt_shared_t *sh, const uint32_t first, const int code)
	Marks every job of stripe first with code (setup-failure path).
static void *	zxc_seek_mt_worker (void *arg)
	Worker thread entry point for multi-threaded seekable decompression.
int64_t	zxc_seekable_decompress_range_mt (zxc_seekable *s, void *dst, const size_t dst_capacity, const uint64_t offset, const size_t len, int n_threads)
	Multi-threaded variant of zxc_seekable_decompress_range.
void	zxc_seekable_free (zxc_seekable *s)
	Releases a seekable handle and every resource it owns.
int	zxc_seekable_set_checksum (zxc_seekable *s, const int enabled)
	Turns per-block checksum verification on or off.
int	zxc_seekable_set_dict (zxc_seekable *s, const void *dict, const size_t dict_size, const void *dict_huf)
	Installs the dictionary needed to decode a dict-compressed archive.
void	zxc_seekable_attach_owned_ctx (zxc_seekable *s, void *ctx)
	Transfers ownership of a heap reader context to the handle.

Detailed Description

Seekable archive reader (random-access decompression) and seek table writer.

The seek table is a standard ZXC block (type = ZXC_BLOCK_SEK) appended between the EOF block and the file footer. It records where every block starts, so a byte range costs one small table read plus the blocks it covers; no table stays resident and opening costs two reads per frame, plus one for the last frame's EOF and SEK headers.

On-disk layout of a SEK block, for N blocks in G = ceil(N / 64) groups:

[Block Header (8B)] block_type=SEK, block_flags=0, comp_size=(u32)(T ^ T >> 32), T = G*8 + N*4 the table's byte size G x group: [Anchor (8B)] byte offset (u64 LE) of the group's first block [<= 64 x Size (4B)] on-disk size (u32 LE) of each of its blocks

Anchor + sizes lands on the next anchor (the EOF block for the last group); the reader checks each group alone. N comes from the footer: the header's 32-bit field does not bound it.

Detection from the end, frame by frame:

1. Parse the footer back from its last byte => total_decompressed_size and the frame's compressed size, so where its header starts
2. The header: block_size, and HAS_SEEK_TABLE, without which the archive is not seekable
3. Derive num_blocks = ceil(total_decomp / block_size)
4. Groups are read and checked on access, with the EOF and SEK block headers ahead of them

Definition in file zxc_seekable.c.

Macro Definition Documentation

ZXC_SEEK_MAX_FRAMES

#define ZXC_SEEK_MAX_FRAMES   (1U << 20)

Most frames a handle indexes: bounds the reads and memory of an open.

Definition at line 70 of file zxc_seekable.c.

ZXC_SEEK_TAIL_HDRS

#define ZXC_SEEK_TAIL_HDRS   (2 * ZXC_BLOCK_HEADER_SIZE)

EOF and SEK block headers, ahead of a frame's table: checked when it is read.

Definition at line 73 of file zxc_seekable.c.

Function Documentation

zxc_seek_block_of_offset()

static uint64_t zxc_seek_block_of_offset ( const zxc_seekable *  s, const uint64_t  offset  )

static

Archive-wide index of the block holding decompressed offset, below the size.

Definition at line 307 of file zxc_seekable.c.

References zxc_ctz32().

Referenced by zxc_seekable_decompress_range(), and zxc_seekable_decompress_range_mt().

zxc_seek_decomp_size()

static uint32_t zxc_seek_decomp_size ( const uint32_t  block_size, const uint64_t  total_decomp, const uint64_t  idx  )

static

Decompressed size of block idx (O(1)).

Returns block_size for every block except the last, which holds the remainder of total_decomp.

Parameters

[in]	block_size	Fixed decompressed block size.
[in]	total_decomp	Total decompressed archive size.
[in]	idx	Zero-based block index.

Returns

Decompressed byte size of block idx.

Definition at line 470 of file zxc_seekable.c.

Referenced by zxc_seekable_decompress_range(), zxc_seekable_decompress_range_mt(), and zxc_seekable_get_block_decomp_size().

zxc_seek_frame_of_block()

static const zxc_seek_frame_t * zxc_seek_frame_of_block ( const zxc_seekable *  s, const uint64_t  idx  )

static

The frame holding archive-wide block idx, below the block count.

Definition at line 293 of file zxc_seekable.c.

Referenced by zxc_seekable_decompress_range(), zxc_seekable_decompress_range_mt(), zxc_seekable_get_block_comp_size(), and zxc_seekable_get_block_decomp_size().

zxc_seek_load_spans()

static int zxc_seek_load_spans ( const zxc_seekable *  s, const zxc_seek_frame_t *  f, const uint64_t  first, const uint32_t  n, uint64_t *  starts, uint32_t *  sizes, uint8_t *  raw  )

static

Loads where each block of [first, first + n) starts and its on-disk size, both taken from the block's own group.

One read of the groups the range touches, plus the frame's EOF/SEK headers: in the same read from group 0, in a 16-byte read of their own past it. Each group is checked alone: anchor in the data area, sizes in [header, entry_max], end within the data area and, for the last group, on the EOF block. The next anchor is not read, so a bad one costs only its own group. A block's size is its own entry, never the gap to the next anchor, so a range spanning groups gets the verdict each group gives alone.

Parameters

[in]	s	Handle.
[in]	f	Frame of the blocks; first + n must not exceed its block count.
[in]	first	First block index within f.
[in]	n	Block count, at least 1.
[out]	starts	Room for n archive offsets.
[out]	sizes	Room for n sizes, each in [header, entry_max].
[out]	raw	Scratch of zxc_seek_spans_raw_max bytes.

Returns

ZXC_OK, ZXC_ERROR_IO (short read), ZXC_ERROR_CORRUPT_DATA.

Definition at line 349 of file zxc_seekable.c.

References ZXC_BLOCK_HEADER_SIZE, zxc_check_eof_header(), zxc_check_seek_header(), ZXC_ERROR_CORRUPT_DATA, ZXC_FILE_HEADER_SIZE, zxc_le32(), zxc_le64(), ZXC_OK, ZXC_SEEK_ANCHOR_SIZE, ZXC_SEEK_GROUP, ZXC_SEEK_GROUP_BYTES, zxc_seek_group_count(), zxc_seek_group_len(), ZXC_SEEK_SIZE_ENTRY, zxc_seek_source_of(), zxc_seek_source_read(), and ZXC_SEEK_TAIL_HDRS.

Referenced by zxc_seekable_decompress_range(), zxc_seekable_decompress_range_mt(), and zxc_seekable_get_block_comp_size().

zxc_seek_mt_fail_stripe()

static void zxc_seek_mt_fail_stripe ( const zxc_seek_mt_shared_t *  sh, const uint32_t  first, const int  code  )

static

Marks every job of stripe first with code (setup-failure path).

Parameters

[in,out]	sh	Shared state.
[in]	first	Stripe index.
[in]	code	Negative zxc_error_t value.

Definition at line 704 of file zxc_seekable.c.

Referenced by zxc_seek_mt_worker().

zxc_seek_mt_worker()

static void * zxc_seek_mt_worker ( void *  arg )

static

Worker thread entry point for multi-threaded seekable decompression.

Sets up its context, dictionary copy and read buffer once, then for each block of its stripe: read (thread-safe pread), decompress, copy the requested sub-range into the caller's output. The dict prefix survives across blocks because the decoder never writes below its dst.

Each job's outcome goes into its result (read by the main thread after join); on error the worker abandons the rest of its stripe.

Parameters

[in,out]

arg

Pointer to the read's zxc_seek_mt_shared_t.

Returns

Always NULL (result codes are reported via the jobs).

Definition at line 723 of file zxc_seekable.c.

References zxc_cctx_t::checksum_enabled, zxc_cctx_t::chunk_size, zxc_cctx_t::dict_buffer, zxc_cctx_t::dict_size, zxc_cctx_t::work_buf, zxc_cctx_free(), zxc_cctx_init(), zxc_decompress_chunk_wrapper(), ZXC_DECOMPRESS_TAIL_PAD, ZXC_ERROR_CORRUPT_DATA, ZXC_ERROR_MEMORY, ZXC_FREE, ZXC_MALLOC, ZXC_OK, ZXC_PAD_SIZE, zxc_seek_mt_fail_stripe(), zxc_seek_read_block(), and zxc_seek_use_dict().

Referenced by zxc_seekable_decompress_range_mt().

zxc_seek_parse_frame()

static int zxc_seek_parse_frame ( zxc_seek_source_t *  src, const uint64_t  end, zxc_seek_frame_t *  f  )

static

Validates the seekable frame that ends at end, in two reads: header and footer as zxc_get_last_frame_info() does.

Its EOF and SEK headers are checked with its table, on access (zxc_seek_load_spans).

Parameters

[in]	src	The archive.
[in]	end	Offset just past the frame.
[out]	f	The frame; its decompressed and block bases are left to the caller.

Returns

1 for a seekable frame, 0 otherwise.

Definition at line 178 of file zxc_seekable.c.

References zxc_frame_info_t::block_size, zxc_frame_info_t::decompressed_size, zxc_frame_info_t::dict_id, zxc_frame_info_t::has_checksum, zxc_frame_info_t::has_seek_table, ZXC_BLOCK_CHECKSUM_SIZE, ZXC_BLOCK_HEADER_SIZE, ZXC_FILE_HEADER_SIZE, ZXC_OK, zxc_scan_frame(), zxc_seek_block_count(), zxc_seek_scan_read(), and zxc_seek_table_bytes().

Referenced by zxc_seekable_parse().

zxc_seek_read_block()

static int zxc_seek_read_block ( const zxc_seekable *  s, const uint64_t  off, const uint32_t  csz, const int  has_cs, uint8_t *  buf, const size_t  buf_cap  )

static

Reads a compressed block into buf from the memory buffer or reader.

off and csz come from zxc_seek_load_spans. What off points at must parse as a block header, checksum included, and agree with csz. An entry pointing into a block is refused; one moved onto a block of the same size is not, and only the position-seeded checksum catches that.

Parameters

[in]	s	Seekable handle.
[in]	off	Byte offset of the block in the archive.
[in]	csz	On-disk size of the block, at least a header.
[in]	has_cs	Its frame carries block checksums.
[out]	buf	Destination buffer.
[in]	buf_cap	Capacity of buf in bytes.

Returns

csz, or a negative zxc_error_t (ZXC_ERROR_DST_TOO_SMALL, ZXC_ERROR_SRC_TOO_SMALL, ZXC_ERROR_IO, ZXC_ERROR_BAD_HEADER, ZXC_ERROR_CORRUPT_DATA).

Definition at line 528 of file zxc_seekable.c.

References zxc_block_header_t::comp_size, ZXC_BLOCK_CHECKSUM_SIZE, ZXC_BLOCK_HEADER_SIZE, ZXC_ERROR_CORRUPT_DATA, ZXC_ERROR_DST_TOO_SMALL, ZXC_OK, zxc_read_block_header(), zxc_seek_source_of(), and zxc_seek_source_read().

Referenced by zxc_seek_mt_worker(), and zxc_seekable_decompress_range().

zxc_seek_scan_read()

static int zxc_seek_scan_read ( const zxc_scan_src_t *  scan, const uint64_t  off, void *  dst, const size_t  len  )

static

zxc_scan_src_t over a zxc_seek_source_t.

Definition at line 163 of file zxc_seekable.c.

References zxc_seek_source_read().

Referenced by zxc_seek_parse_frame().

zxc_seek_source_of()

static zxc_seek_source_t zxc_seek_source_of ( const zxc_seekable *  s )

static

The handle's archive as a read source, for the per-block readers.

Definition at line 157 of file zxc_seekable.c.

Referenced by zxc_seek_load_spans(), and zxc_seek_read_block().

zxc_seek_source_read()

static int zxc_seek_source_read ( const zxc_seek_source_t *  src, void *  dst, const size_t  len, const uint64_t  off  )

static

Reads a byte range from the archive, whatever backs it: the one bounds check every parsed offset goes through, written so it cannot wrap.

Returns

ZXC_OK; ZXC_ERROR_SRC_TOO_SMALL outside the archive; the reader's code, or ZXC_ERROR_IO, on a short read.

Definition at line 140 of file zxc_seekable.c.

References ZXC_ERROR_IO, ZXC_ERROR_SRC_TOO_SMALL, and ZXC_OK.

Referenced by zxc_seek_load_spans(), zxc_seek_read_block(), zxc_seek_scan_read(), and zxc_seekable_parse().

zxc_seek_spans_raw_max()

static size_t zxc_seek_spans_raw_max ( const uint64_t  first, const uint32_t  n  )

static

Scratch bound for zxc_seek_load_spans(): the groups blocks [first, first + n) touch.

Definition at line 323 of file zxc_seekable.c.

References ZXC_SEEK_GROUP, ZXC_SEEK_GROUP_BYTES, and ZXC_SEEK_TAIL_HDRS.

Referenced by zxc_seekable_decompress_range_mt().

zxc_seek_use_dict()

static int zxc_seek_use_dict ( const zxc_seekable *  s, zxc_cctx_t *  ctx, const zxc_seek_dict_t **  held, const zxc_seek_frame_t *  f  )

static

Gives ctx frame f's dictionary, copied in unless *held, the one it holds, is already that one.

Returns

ZXC_OK, or ZXC_ERROR_DICT_REQUIRED when it is not installed.

Definition at line 494 of file zxc_seekable.c.

References zxc_cctx_t::dict_buffer, zxc_cctx_t::dict_size, zxc_cctx_attach_dict_huf(), ZXC_ERROR_CORRUPT_DATA, ZXC_ERROR_DICT_REQUIRED, and ZXC_OK.

Referenced by zxc_seek_mt_worker(), and zxc_seekable_decompress_range().

zxc_seekable_attach_owned_ctx()

void zxc_seekable_attach_owned_ctx	(	zxc_seekable *	s,
		void *	ctx
	)

Transfers ownership of a heap reader context to the handle.

Hands ownership of a heap-allocated reader context to a seekable handle.

Cross-TU hook (declared in zxc_internal.h): ctx is released via ZXC_FREE when zxc_seekable_free runs. Used by zxc_seekable_open_file so its allocated reader state outlives the open call. NULL-safe on s.

Definition at line 1033 of file zxc_seekable.c.

Referenced by zxc_seekable_open_file().

zxc_seekable_parse()

static zxc_seekable * zxc_seekable_parse ( const zxc_seek_source_t *  source )

static

Walks the frames back from the end through zxc_seek_parse_frame, then numbers their bytes and blocks.

NULL unless every frame is seekable.

Definition at line 224 of file zxc_seekable.c.

References ZXC_BLOCK_HEADER_SIZE, ZXC_CALLOC, zxc_check_eof_header(), zxc_check_seek_header(), ZXC_FREE, ZXC_OK, ZXC_REALLOC, ZXC_SEEK_MAX_FRAMES, zxc_seek_parse_frame(), zxc_seek_source_read(), and ZXC_SEEK_TAIL_HDRS.

Referenced by zxc_seekable_open(), and zxc_seekable_open_reader().