zxc_dict.c File Reference

Pre-trained dictionary: ID computation, .zxd serialization, and training. More...

#include "../../include/zxc_dict.h"
#include "../../include/zxc_buffer.h"
#include "zxc_internal.h"

Go to the source code of this file.

Functions
uint32_t	zxc_dict_id (const void *dict, const size_t dict_size, const void *huf_lengths)
	Computes the dictionary identifier for dict (and optional table).
uint32_t	zxc_dict_get_id (const void *buf, const size_t buf_size)
	Extracts the stored dict_id from a serialized .zxd buffer.
size_t	zxc_dict_save_bound (const size_t content_size)
	Worst-case .zxd byte size for a dictionary of content_size bytes.
int64_t	zxc_dict_save (const void *content, const size_t content_size, const void *huf_lengths, void *buf, const size_t buf_capacity)
	Serializes dictionary content + Huffman table into the .zxd format.
int	zxc_dict_load (const void *buf, const size_t buf_size, const void **content_out, size_t *content_size_out, const void **huf_out, uint32_t *dict_id_out)
	Parses a .zxd buffer, returning in-buffer views of content and table.
const void *	zxc_dict_huf (const void *buf, const size_t buf_size)
	Returns a pointer to the packed Huffman table inside a .zxd buffer.
static uint32_t	zxc_dict_hash (const uint8_t *p)
	Hashes the k-gram at p into a frequency-table bucket index.
static void	zxc_dict_sift_down (zxc_dict_seg_t *a, size_t root, const size_t n)
	Restore the min-heap property at root over the range a[0..n).
static void	zxc_dict_sort_segs_desc (zxc_dict_seg_t *a, const size_t n)
	Sort a[0..n) by zxc_dict_seg_t::score in descending order.
int64_t	zxc_train_dict (const void *const *samples, const size_t *sample_sizes, const size_t n_samples, void *dict_buf, const size_t dict_capacity)
	Trains a raw dictionary from sample buffers by k-gram coverage.
int	zxc_train_dict_huf (const void *const *samples, const size_t *sample_sizes, const size_t n_samples, const void *dict, const size_t dict_size, uint8_t *huf_lengths_out)
	Trains the shared literal Huffman table for a dictionary (Tier-2).
int64_t	zxc_dict_train (const void *const *samples, const size_t *sample_sizes, const size_t n_samples, void *zxd_buf, const size_t zxd_capacity)
	One-shot training: sample buffers in, ready-to-write .zxd bytes out.

Detailed Description

Pre-trained dictionary: ID computation, .zxd serialization, and training.

Definition in file zxc_dict.c.

Function Documentation

zxc_dict_hash()

static uint32_t zxc_dict_hash ( const uint8_t *  p )

static

Hashes the k-gram at p into a frequency-table bucket index.

Multiplicative hash of the ZXC_DICT_KGRAM_LEN-byte k-gram, folded down to ZXC_DICT_HASH_BITS.

Parameters

[in]

p

Pointer to at least ZXC_DICT_KGRAM_LEN readable bytes.

Returns

Bucket index in [0, ZXC_DICT_HASH_SIZE).

Definition at line 192 of file zxc_dict.c.

References ZXC_DICT_HASH_BITS, ZXC_HASH_MULT32, and zxc_le32().

Referenced by zxc_train_dict().

zxc_dict_sift_down()

static void zxc_dict_sift_down ( zxc_dict_seg_t *  a, size_t  root, const size_t  n  )

static

Restore the min-heap property at root over the range a[0..n).

Sinks a[root] until both children are >= it, comparing on zxc_dict_seg_t::score. Iterative, so the stack stays O(1).

Parameters

[in,out]	a	Heap-ordered array; a[0..n) is treated as the heap.
[in]	root	Index of the element to sift down. Must be < n.
[in]	n	Number of valid elements in the heap.

Definition at line 217 of file zxc_dict.c.

Referenced by zxc_dict_sort_segs_desc().

zxc_dict_sort_segs_desc()

static void zxc_dict_sort_segs_desc ( zxc_dict_seg_t *  a, const size_t  n  )

static

Sort a[0..n) by zxc_dict_seg_t::score in descending order.

In-place heapsort: a min-heap pushes the smallest scores to the tail, so the array ends up descending, which is what the fill step wants. Not qsort: no libc dependency, no indirect comparator, and the order is fixed here rather than by the platform - dictionaries stay reproducible.

Equal scores keep an unspecified-but-deterministic relative order, matching the previous comparator that returned 0 on ties (heapsort is not stable).

Parameters

[in,out]	a	Array of n segments, sorted in place.
[in]	n	Number of segments. n < 2 is a no-op.

Definition at line 244 of file zxc_dict.c.

References zxc_dict_sift_down().

Referenced by zxc_train_dict().