Dictionary

Pre-trained dictionary training, serialization, and identification. More...

Functions
uint32_t	zxc_dict_id (const void *dict, size_t dict_size, const void *huf_lengths)
	Computes the dictionary ID for the given content and optional table.
int	zxc_dict_load (const void *buf, size_t buf_size, const void **content_out, size_t *content_size_out, const void **huf_out, uint32_t *dict_id_out)
	Loads and validates a .zxd dictionary file from a memory buffer.
int64_t	zxc_dict_save (const void *content, size_t content_size, const void *huf_lengths, void *buf, size_t buf_capacity)
	Serializes dictionary content and its shared Huffman table to .zxd.
size_t	zxc_dict_save_bound (size_t content_size)
	Returns the .zxd file size for a given content size.
uint32_t	zxc_dict_get_id (const void *buf, size_t buf_size)
	Returns the dictionary ID stored in a .zxd file buffer.
int64_t	zxc_train_dict (const void *const *samples, const size_t *sample_sizes, size_t n_samples, void *dict_buf, size_t dict_capacity)
	Trains a dictionary from a corpus of samples.
int	zxc_train_dict_huf (const void *const *samples, const size_t *sample_sizes, size_t n_samples, const void *dict, size_t dict_size, uint8_t *huf_lengths_out)
	Trains the shared literal Huffman table for an already-trained dictionary.
int64_t	zxc_dict_train (const void *const *samples, const size_t *sample_sizes, size_t n_samples, void *zxd_buf, size_t zxd_capacity)
	One-call dictionary creation: content + shared table, serialized to ready-to-write .zxd bytes.
const void *	zxc_dict_huf (const void *buf, size_t buf_size)
	Returns a pointer to the shared Huffman table inside a .zxd buffer.

Detailed Description

Pre-trained dictionary training, serialization, and identification.

Function Documentation

zxc_dict_get_id()

uint32_t zxc_dict_get_id	(	const void *	buf,
		const size_t	buf_size
	)

Returns the dictionary ID stored in a .zxd file buffer.

Reads the header's dict_id field only; the rest of the file is not validated.

Parameters

[in]	buf	Buffer containing the .zxd file.
[in]	buf_size	Size of buf in bytes.

Returns

Dictionary ID, or 0 if buf is too small or the magic word does not match.

Returns the dictionary ID stored in a .zxd file buffer.

Public API; see zxc_dict.h. Reads the header field without recomputing it.

Definition at line 57 of file zxc_dict.c.

References ZXC_DICT_HEADER_SIZE, ZXC_DICT_MAGIC, and zxc_le32().

zxc_dict_huf()

const void * zxc_dict_huf	(	const void *	buf,
		const size_t	buf_size
	)

Returns a pointer to the shared Huffman table inside a .zxd buffer.

Zero-copy: the returned pointer aims into buf and lives as long as it does.

Parameters

[in]	buf	Buffer containing the .zxd file.
[in]	buf_size	Size of buf in bytes.

Returns

Pointer to the 128-byte packed code-lengths table, or NULL if buf is not a valid .zxd file or carries no table.

Returns a pointer to the shared Huffman table inside a .zxd buffer.

Public API; see zxc_dict.h. Locates the table on magic/version/size checks alone, skipping zxc_dict_load's full validation. Aliases buf.

Definition at line 156 of file zxc_dict.c.

References ZXC_DICT_HEADER_SIZE, ZXC_DICT_MAGIC, ZXC_DICT_VERSION, ZXC_HUF_TABLE_SIZE, zxc_le16(), and zxc_le32().

zxc_dict_id()

uint32_t zxc_dict_id	(	const void *	dict,
		const size_t	dict_size,
		const void *	huf_lengths
	)

Computes the dictionary ID for the given content and optional table.

A deterministic 32-bit hash, stored in the ZXC file header so the decoder can check it was handed the right dictionary. With huf_lengths NULL it hashes the content alone (the content-only dictionary of the buffer API); with a table it binds the pair as hash(table, seed = hash(content)), which is what .zxd files and archives compressed with a shared table carry.

Parameters

[in]	dict	Pointer to dictionary content.
[in]	dict_size	Size in bytes.
[in]	huf_lengths	Shared literal Huffman table (ZXC_HUF_TABLE_SIZE bytes), or NULL for a content-only ID.

Returns

32-bit dictionary ID. Returns 0 if dict is NULL or dict_size is 0.

Computes the dictionary ID for the given content and optional table.

Public API; see zxc_dict.h. A checksum over the content, then a second one over the packed Huffman lengths seeded by the first, so a single id covers both. Stored in the archive header and re-checked on decode.

Definition at line 29 of file zxc_dict.c.

References zxc_checksum(), ZXC_CHECKSUM_RAPIDHASH, and ZXC_HUF_TABLE_SIZE.

Referenced by zxc_compress(), zxc_compress_cctx(), zxc_dctx_decode_frame(), zxc_dict_load(), zxc_dict_save(), zxc_seekable_set_dict(), zxc_stream_compress(), and zxc_stream_decompress().

zxc_dict_load()

int zxc_dict_load	(	const void *	buf,
		const size_t	buf_size,
		const void **	content_out,
		size_t *	content_size_out,
		const void **	huf_out,
		uint32_t *	dict_id_out
	)

Loads and validates a .zxd dictionary file from a memory buffer.

Zero-copy: on success content_out and huf_out point into buf, which must stay alive as long as they are in use. One call gives everything needed to (de)compress with the dictionary, feed content_out / huf_out straight to the dict / dict_huf option fields.

Parameters

[in]	buf	Buffer containing the .zxd file.
[in]	buf_size	Size of buf in bytes.
[out]	content_out	Receives a pointer to the dictionary content.
[out]	content_size_out	Receives the content size in bytes.
[out]	huf_out	Receives a pointer to the 128-byte shared Huffman table (may be NULL if not needed).
[out]	dict_id_out	Receives the dictionary ID (may be NULL).

Returns

ZXC_OK on success, or a negative zxc_error_t code.

Loads and validates a .zxd dictionary file from a memory buffer.

Public API; full contract in zxc_dict.h. Validates magic, version, header checksum and dict_id, then points the out-params into buf - no copy, so they stay valid only while buf does.

Definition at line 112 of file zxc_dict.c.

References ZXC_DICT_HEADER_SIZE, zxc_dict_id(), ZXC_DICT_MAGIC, ZXC_DICT_SIZE_MAX, ZXC_DICT_VERSION, ZXC_ERROR_BAD_CHECKSUM, ZXC_ERROR_BAD_HEADER, ZXC_ERROR_BAD_MAGIC, ZXC_ERROR_BAD_VERSION, ZXC_ERROR_CORRUPT_DATA, ZXC_ERROR_DICT_TOO_LARGE, ZXC_ERROR_NULL_INPUT, ZXC_ERROR_SRC_TOO_SMALL, zxc_hash16(), ZXC_HUF_TABLE_SIZE, zxc_le16(), zxc_le32(), ZXC_OK, and zxc_store_le32().

zxc_dict_save()

int64_t zxc_dict_save	(	const void *	content,
		const size_t	content_size,
		const void *	huf_lengths,
		void *	buf,
		const size_t	buf_capacity
	)

Serializes dictionary content and its shared Huffman table to .zxd.

The 128-byte packed code-lengths table (from zxc_train_dict_huf()) is mandatory and follows the content. The stored dict_id covers both, so archives compressed with this dictionary are bound to the exact pair.

Parameters

[in]	content	Raw dictionary content.
[in]	content_size	Size of content in bytes (max ZXC_DICT_SIZE_MAX).
[in]	huf_lengths	128-byte packed Huffman code lengths (required).
[out]	buf	Output buffer for the .zxd file.
[in]	buf_capacity	Capacity of buf (see zxc_dict_save_bound()).

Returns

Number of bytes written on success, or a negative zxc_error_t code.

Serializes dictionary content and its shared Huffman table to .zxd.

Public API; full contract in zxc_dict.h. Header, then content, then packed Huffman lengths - layout above.

Definition at line 79 of file zxc_dict.c.

References ZXC_DICT_HEADER_SIZE, zxc_dict_id(), ZXC_DICT_MAGIC, ZXC_DICT_SIZE_MAX, ZXC_DICT_VERSION, ZXC_ERROR_DICT_TOO_LARGE, ZXC_ERROR_DST_TOO_SMALL, ZXC_ERROR_NULL_INPUT, zxc_hash16(), ZXC_HUF_TABLE_SIZE, zxc_store_le16(), and zxc_store_le32().

Referenced by zxc_dict_train().

zxc_dict_save_bound()

size_t zxc_dict_save_bound

(

const size_t

content_size

)

Returns the .zxd file size for a given content size.

Parameters

[in]

content_size

Size of the dictionary content.

Returns

Total .zxd file size (header + content).

Returns the .zxd file size for a given content size.

Public API; see zxc_dict.h. Sizes the buffer for zxc_dict_save.

Definition at line 69 of file zxc_dict.c.

References ZXC_DICT_HEADER_SIZE, and ZXC_HUF_TABLE_SIZE.

zxc_dict_train()

int64_t zxc_dict_train	(	const void *const *	samples,
		const size_t *	sample_sizes,
		const size_t	n_samples,
		void *	zxd_buf,
		const size_t	zxd_capacity
	)

One-call dictionary creation: content + shared table, serialized to ready-to-write .zxd bytes.

Runs zxc_train_dict(), then zxc_train_dict_huf() on the trained content, then zxc_dict_save() into zxd_buf. Size zxd_capacity from the dictionary you expect, or use zxc_dict_save_bound(ZXC_DICT_SIZE_MAX) for a safe upper bound. The three primitives stay available for the cases this shortcut does not cover: content-only dictionaries, retraining just the table, externally sourced content.

Parameters

[in]	samples	Array of pointers to sample buffers.
[in]	sample_sizes	Array of sample sizes in bytes.
[in]	n_samples	Number of samples.
[out]	zxd_buf	Output buffer for the .zxd file.
[in]	zxd_capacity	Capacity of zxd_buf.

Returns

Number of .zxd bytes written on success, or a negative zxc_error_t code.

One-call dictionary creation: content + shared table, serialized to ready-to-write .zxd bytes.

Public API; see zxc_dict.h. Trains the content (zxc_train_dict), then the shared table from that content (zxc_train_dict_huf), then serializes both (zxc_dict_save). The two phases are a real data dependency.

Definition at line 540 of file zxc_dict.c.

References zxc_dict_save(), ZXC_DICT_SIZE_MAX, ZXC_ERROR_MEMORY, ZXC_ERROR_NULL_INPUT, ZXC_ERROR_SRC_TOO_SMALL, ZXC_FREE, ZXC_HUF_TABLE_SIZE, ZXC_MALLOC, ZXC_OK, zxc_train_dict(), and zxc_train_dict_huf().

zxc_train_dict()

int64_t zxc_train_dict	(	const void *const *	samples,
		const size_t *	sample_sizes,
		const size_t	n_samples,
		void *	dict_buf,
		const size_t	dict_capacity
	)

Trains a dictionary from a corpus of samples.

Picks the byte sequences that maximize LZ77 match coverage across the samples. The content can go straight into zxc_compress_opts_t::dict, or through zxc_dict_save().

Parameters

[in]	samples	Array of pointers to sample buffers.
[in]	sample_sizes	Array of sample sizes in bytes.
[in]	n_samples	Number of samples.
[out]	dict_buf	Output buffer for trained dictionary content.
[in]	dict_capacity	Capacity of dict_buf (max ZXC_DICT_SIZE_MAX).

Returns

Size of the trained dictionary on success, or a negative zxc_error_t code.

Trains a dictionary from a corpus of samples.

Public API; full contract in zxc_dict.h, algorithm in the note above. Picks are emitted in reverse so the highest-coverage bytes land at the dict's end, where match offsets are smallest.

Definition at line 263 of file zxc_dict.c.

References ZXC_CALLOC, zxc_dict_hash(), ZXC_DICT_HASH_SIZE, ZXC_DICT_KGRAM_LEN, ZXC_DICT_MAX_SEGMENTS, ZXC_DICT_SAMPLE_TARGET, ZXC_DICT_SIZE_MAX, zxc_dict_sort_segs_desc(), ZXC_ERROR_DICT_TOO_LARGE, ZXC_ERROR_MEMORY, ZXC_ERROR_NULL_INPUT, ZXC_ERROR_SRC_TOO_SMALL, ZXC_FREE, and ZXC_MALLOC.

Referenced by zxc_dict_train().

zxc_train_dict_huf()

int zxc_train_dict_huf	(	const void *const *	samples,
		const size_t *	sample_sizes,
		const size_t	n_samples,
		const void *	dict,
		const size_t	dict_size,
		uint8_t *	huf_lengths_out
	)

Trains the shared literal Huffman table for an already-trained dictionary.

Compresses the samples with dict and derives canonical Huffman code lengths from the real post-LZ literal distribution. Embed the resulting 128-byte packed table in a .zxd via zxc_dict_save(), or pass it through the dict_huf option field. Blocks whose literals compress better with the shared table drop their own 128-byte table header, which is decisive at small block sizes.

Used at levels 6 and 7 only. Pass the same table to decompress: dict_id covers content and table, so the content alone is a ZXC_ERROR_DICT_MISMATCH.

Parameters

[in]	samples	Array of pointers to sample buffers (typically the same corpus used for zxc_train_dict()).
[in]	sample_sizes	Array of sample sizes in bytes.
[in]	n_samples	Number of samples.
[in]	dict	Trained dictionary content.
[in]	dict_size	Dictionary content size in bytes.
[out]	huf_lengths_out	Receives the 128-byte packed code-lengths table.

Returns

ZXC_OK on success, or a negative zxc_error_t code.

Trains the shared literal Huffman table for an already-trained dictionary.

Public API; see zxc_dict.h and the algorithm note above. Slices are sub-sampled to a fixed budget.

Definition at line 437 of file zxc_dict.c.

References zxc_cctx_t::dict_buffer, zxc_cctx_t::lit_freq_acc, zxc_block_size_ceil(), zxc_cctx_free(), zxc_cctx_init(), zxc_compress_bound(), zxc_compress_chunk_wrapper(), ZXC_DICT_HUF_SAMPLE_BUDGET, ZXC_DICT_HUF_TRAIN_BLOCK, ZXC_DICT_SIZE_MAX, ZXC_ERROR_DICT_TOO_LARGE, ZXC_ERROR_MEMORY, ZXC_ERROR_NULL_INPUT, ZXC_FREE, zxc_huf_build_code_lengths(), ZXC_HUF_MAX_CODE_LEN_DENSITY, zxc_huf_nudge_code_lengths(), ZXC_HUF_NUM_SYMBOLS, zxc_huf_pack_lengths(), ZXC_HUF_TABLE_SIZE, ZXC_LEVEL_DENSITY, ZXC_MALLOC, and ZXC_OK.

Referenced by zxc_dict_train().