ZXC

Asymmetric lossless compression, built for ultra-fast decode

Next: , Previous: , Up: Documentation

ZXC Compressed File Format (Technical Specification)

ZXC 0.15.0. Generated from docs/FORMAT.md; corrections are welcome as pull requests.

Table of Contents

Date: October 2026 Format Version: 9

This document describes the on-disk binary format of a ZXC compressed file. It formalizes the current reference implementation of format version 9.

1. Conventions#

2. Full File Layout#

+----------------------+ 16 bytes
| File Header          |
+----------------------+
| Block #0             |
|  - 8B Block Header   |
|  - Block Payload     |
|  - Optional 4B Cksum |
+----------------------+
| Block #1             |
|  ...                 |
+----------------------+
| EOF Block            | 8 bytes (type=255, comp_size=0)
+----------------------+
| SEK Block           | iff HAS_SEEK_TABLE: table of contents for random access
+----------------------+
| File Footer          | 3 to 17 bytes, plus 8 with a digest
+----------------------+

This is one frame. Sections 3 to 8 describe it; a file holds one or more.

2.1 Concatenated frames#

A file holds one or more frames back to back, each the layout above starting with the ZXC magic word 0x9CB02EF5. Frames are independent: each has its own header, block size, flags, block indices (the checksum seed of § 7.2 restarts at 0) and footer. Their outputs are concatenated in order, so cat a.zxc b.zxc decodes to cat a b.

After each footer, the decoder reads the next 4 bytes:

Next bytes Action
end of input done
0x9CB02EF5 decode the next frame
anything else, or 1-3 bytes reject as corrupt data

An empty file is truncated; a bad magic word at offset 0 is a bad magic word.

Each footer stores its frame's compressed_frame_size (§ 8), so a reader can also walk the frames back from the end of the input without decoding them, as the size queries and the seekable reader (§ 5.5) do. Seekable frames concatenated stay seekable when every frame carries a table.

3. File Header (16 bytes)#

Offset  Size  Field
0x00    4     Magic Word
0x04    1     Format Version
0x05    1     Chunk Size Code
0x06    1     Flags
0x07    7     Reserved (must be 0)
0x0E    2     Header Checksum

3.1 Field definitions#

4. Generic Block Container#

Each block starts with a fixed 8-byte block header.

Offset  Size  Field
0x00    1     Block Type
0x01    1     Block Flags
0x02    1     Reserved
0x03    4     Compressed Payload Size (comp_size)
0x07    1     Header Checksum

4.1 Header semantics#

4.2 Block physical layout#

[8B Block Header] + [comp_size bytes payload] + [optional 4B checksum]

When checksums are enabled at file level, each data block carries one trailing 4-byte checksum of its decompressed bytes (§7.2).

5. Block Types and Payload Formats#

5.1 RAW block (type=0)#

Payload is uncompressed data.

RAW payload layout#

+-------------------------------+
| Literal bytes (uncompressed)  |
+-------------------------------+

No internal sub-header: unlike GLO and GHI, the payload starts at the first byte. The decoded size is therefore the Compressed Payload Size, and a RAW block is the only one whose payload reaches that bound exactly.

5.2 GLO block (type=1)#

"General LZ, LOw throughput": LZ-style format with separated streams, trading decode throughput for ratio.

GLO payload layout#

+-------------------------------+
| GLO Header (12 bytes)         |
+-------------------------------+
| Section descriptors (0/4/8B)  |
+-------------------------------+
| Literals section              |
+-------------------------------+
| Tokens section                |
+-------------------------------+
| Offsets section               |
+-------------------------------+
| Extras section (+ slack pad)  |
+-------------------------------+

GLO Header (12 bytes)#

Offset  Size  Field
0x00    4     Sequence Count (n_sequences)
0x04    4     Literal Count (n_literals)
0x08    1     Literal Encoding (enc_lit)
0x09    1     Token Encoding (enc_tok)
0x0A    1     Match Length Encoding (enc_mlen)
0x0B    1     Offset Encoding (enc_off)

GLO header semantics#

Literal Encoding, Token Encoding and Offset Encoding are closed value sets: a GLO decoder MUST reject any value outside the ones listed above rather than fall back to a default. This is not a memory-safety requirement — a GLO block sizes its offset section and reads it from the same width, so the two cannot disagree — but it keeps the undefined byte values genuinely free for a later format version instead of aliasing them onto an existing meaning.

GLO section descriptors (0, 4 or 8 bytes)#

Only the two sizes the header cannot imply are stored, each a u32 and only when present, in this order:

Descriptor Present when Meaning
Literal Section Size (lit_comp) Literal Encoding != 0 compressed size of the literal section
Token Section Size (tok_comp) Token Encoding == 2 compressed size of the token section

Everything else is derived, so it cannot be forged inconsistently:

RAW literals and RAW tokens carry no descriptor at all. RLE or Huffman literals add 4 bytes (RLE at any GLO level, Huffman from level 6) and Huffman tokens (level 7) 4 more: 0 or 4 bytes at levels 3 to 6, up to 8 at level 7.

Because the extras section is sized as the residue and its values are read on demand, an end pointer sitting past the last real extra is harmless. That is what lets the slack padding below hide there at no cost in the header.

Literal section slack (normative)#

At least 32 bytes of payload MUST follow the literal section:

comp_size - header - descriptors - lit_comp >= 32

Decoders copy literals with an over-reading wild copy, and a RAW literal section points straight into the caller's buffer, so those bytes are what keeps the last copy of a block inside the payload. A decoder MUST reject a block that does not satisfy the inequality rather than read past the literals.

The tokens, offsets and extras sections already cover it on any real block -- 32 bytes is reached from 16 sequences with 1-byte offsets, 11 with 2-byte ones. When they do not (a block of very few sequences), the encoder appends zero padding until they do. The padding falls inside the extras residue, so it needs no field of its own; its contents are unconstrained and MUST NOT be validated.

The same rule and the same reasoning apply to GHI (§ 5.3), where the sequences and extras sections play the role of tokens/offsets/extras.

GLO stream content#

Overflow rules:

5.2.1 Huffman literal section#

Literal Encoding 2 carries a length-limited canonical Huffman code over the literal bytes. The bits are placed on the wire with the PivCo layout (level-ordered Huffman, after Żukowski 2026): the encoding is ordinary Huffman — same code lengths, same code bits, same size — and PivCo only reorders those bits, grouping them by TREE LEVEL rather than by symbol so decoding runs data-parallel list merges instead of a serial bit chain.

Offset  Size  Field
0x00    128   256 x 4-bit code lengths, packed two-per-byte (low nibble first).
              code_len[i] in [0, 11] (0 means symbol absent).
0x80    var   One run per EMITTING node of the canonical code tree, in BFS
              order (parents before children, left before right). Runs are
              LSB-first within bytes and each run is padded to a byte
              boundary. Emitting nodes and run contents are defined below.

Canonical code. Codes are length-limited at L = 11 bits (levels ≤ 6 never emit codes longer than 8; level 7 / ULTRA may emit up to 11). Symbols are ordered by (code_len, symbol) and assigned consecutive code values in that order, starting at 0 and left-shifting when the length increases — the standard canonical construction. The code tree is the binary trie of these codewords read MSB-first: at depth d, a codeword's bit code_len-1-d selects the left (0) or right (1) child. The Kraft equality (validated below) makes this trie complete: every internal node has exactly two children.

Emitting nodes and flat subtrees. Both sides derive, from the code lengths alone, the set of flat roots: an internal node is a flat root iff

  1. it is not itself inside another flat subtree (BFS order resolves this: parents are classified first), and
  2. every leaf below it sits at the same relative depth D, and
  3. D >= 2.

(Completeness of the trie makes condition 2 imply a perfect binary subtree of 2^D leaves. Every maximal complete subtree of depth D >= 2 is a flat root — a fixed format rule. The reference decoder unpacks the packed codes directly, with SIMD kernels for D in 2..6 and a scalar table lookup for D >= 7, instead of running the level-merge cascade.)

Every internal node that is neither a flat root nor a descendant of one is a bitmap node: its run holds one branch bit per symbol routed through it, in symbol-sequence order (0 = left child, 1 = right child). A flat root's run instead holds D packed bits per symbol routed through it, in symbol sequence order: bit j (bit 0 first) is the branch taken at relative depth j below the flat root. Strict descendants of a flat root emit no run at all — they are skipped in the BFS enumeration.

Derived sizes. There is no stored size field of any kind: the root handles n_literals symbols, the popcount of a bitmap node's bits equals its right child's symbol count, and a flat root consuming c symbols occupies exactly ceil(c*D/8) bytes, so every run length is derived while walking the BFS order once.

Selection is an encoder policy, not a format rule: the reference encoder requires level ≥ 6 and at least 139 literals for a per-block table (any non-empty section for the dictionary's shared table; level-7 tokens need 139 sequences), prices every candidate (RAW / RLE / Huffman / shared-table Huffman) as J = size + premium(level) * n_decoded_bytes, and picks the minimum — a space-speed Lagrangian with a per-level decode-time premium. Before that pricing, the reference encoder also applies a joint flat/length nudge to the fitted code lengths at levels 6-7 (literals; level 7 also nudges the token table): it trades a few bytes of code-length optimality for a flatter tree whose PivCo runs decode faster. The nudged lengths remain an ordinary canonical, Kraft-exact code — decoders see nothing special.

A second encoder-side policy governs match distances. At levels 1 to 5 the reference encoder refuses to emit a match closer than 32 bytes, so the decoder's match copy keeps to its single widest arm. It applies per block and only where a sampled scan of the block finds few repeats inside that distance: data built on short repeats — periodic runs, tightly structured records — keeps them, because banning them there costs size and decode time. Levels 6 and 7 keep every distance. Like the nudge, this changes which sequences an encoder emits, never how they are encoded, so the archive stays an ordinary GLO/GHI block and any conforming decoder reads it.

A third encoder-side policy re-emits long matches inline. A GLO token holds the match length in its 4-bit ML field and escapes any match longer than 19 bytes (ML == 15) to a varint. At levels 3 to 5 the reference encoder instead re-emits such a match, up to a per-level cap, as a chain of inline matches of at most 19 bytes at the same offset (a 24-byte match as 19 + 5, a 38-byte match as 19 + 19), so the decoder never takes its ML escape. It does this per block and only when that escape looks poorly predicted: on regular data the branch is already predictable and splitting would only add sequences for no decode gain. Levels 1 and 2 (GHI) and levels 6 and 7 keep every escape. Like the distance floor, this changes which sequences an encoder emits, never how they are encoded, so the archive stays an ordinary GLO block and any conforming decoder reads it.

Decoder validation requirements:

5.2.2 Shared-table Huffman literal section#

Literal Encoding 3 is only valid in archives compressed with a dictionary (HAS_DICTIONARY set): the payload is the same Huffman/PivCo section as § 5.2.1 with the 128-byte lengths header omitted — the code lengths come from the shared literal table carried by the .zxd dictionary (see § 12.4), validated once when the dictionary is attached (same rules as § 5.2.1). Decoders MUST reject Literal Encoding 3 sections when no dictionary table is attached. The archive's dict_id binds the (content, table) pair, so a matching table is guaranteed present whenever the dictionary check passed.

The shared table is trained on the corpus' post-LZ literal distribution and covers only the symbols seen in training; the encoder falls back to a per-block table (Literal Encoding 2) or RAW/RLE for any block containing a literal byte without a code.

The level-7 token section reuses the § 5.2.1 layout (Token Encoding 2, with the inline lengths header) over the token byte alphabet.

5.2.3 RLE literal section#

Literal Encoding 1. The section is a sequence of runs, each starting with a control byte t:

The runs must produce exactly n_literals bytes and use the whole section: a run that overshoots n_literals, a section that ends before reaching it, or bytes left over once it is reached are corrupt data. An RLE section for n_literals = 0 is empty.

5.3 GHI block (type=2)#

"General LZ, HIgh throughput": LZ format with packed 32-bit sequences, trading ratio for decode throughput.

GHI payload layout#

+-------------------------------+
| GHI Header (12 bytes)         |
+-------------------------------+
| Literals section              |
+-------------------------------+
| Sequences stream (N * 4B)     |
+-------------------------------+
| Extras section (+ slack pad)  |
+-------------------------------+

GHI Header (12 bytes)#

Same binary layout as the GLO header: Sequence Count, Literal Count, Literal Encoding, Token Encoding, Match Length Encoding, Offset Encoding.

In practice for GHI:

GHI section descriptors: none#

Every size follows from the header, so GHI writes no descriptor at all:

The § 5.2 literal-section slack rule applies here too.

GHI sequence word format (32 bits)#

Bits 31..24 : LL (literal length, 8 bits)
Bits 23..16 : ML (match length minus 5, 8 bits)
Bits 15..0  : Offset - 1 (16 bits, biased; decode: stored + 1)

Memory order (little-endian word):

byte0 = offset low
byte1 = offset high
byte2 = ML
byte3 = LL

Overflow rules:

5.4 EOF block (type=255)#

EOF marks end of block stream.

Constraints:

Immediately after EOF block header comes the SEK block when HAS_SEEK_TABLE=1, then the file footer (§ 8).

5.5 SEK block (type=254)#

The Seek Table block sits between the EOF block and the File Footer when, and only when, the file header sets HAS_SEEK_TABLE. An empty source still gets one: a bare SEK header, payload size 0. It provides O(1) random-access capabilities by recording, in groups, where blocks start and how long each is on disk. Decompressed sizes and block indices are derived from the file header's block_size (all blocks are block_size except the last, which may be smaller).

Layout of a SEK Block:

  Offset             Size    Field
  0x00               8       Block Header (Block Type 254, Compressed Payload Size = table bytes, folded)
  0x08               8       Group 0 anchor: offset of block 0 (u64 LE), always 16
  0x10               4       Block 0 on-disk size (u32 LE)
  0x14               4       Block 1 on-disk size
  ...                4       ... one size per block, 64 per group
  0x110              8       Group 1 anchor: offset of block 64 (u64 LE)
  0x118              4       Block 64 on-disk size
  ...                ...     ...

The table is a sequence of groups of K = 64 blocks. A group is its anchor - the byte offset of its first block's header from the start of the archive, as a u64 - followed by one u32 per block: the block's on-disk size (header + payload + optional checksum). Group j starts j × 264 bytes into the table; only the last group may hold fewer than 64 sizes. Block i starts at anchor(i / 64) plus the sizes of the blocks before it in its group, so locating any block costs one bounded read. The table is T = ⌈N / 64⌉ × 8 + N × 4 bytes. The header's 32-bit Compressed Payload Size holds T folded, (T XOR (T >> 32)) mod 2^32: T itself below 4 GiB, like every other block's payload size, with all 64 bits taking part above. Decoders never read the length from it: they derive N from the footer and compare the fold of T, so the field does not bound the table.

The table is not authenticated. Bounds checks (step 7) reject damaged entries, but a table rewritten consistently can point a block at another well-formed block of the same on-disk size, whose bytes the read returns with no error; any seek index checked only against itself shares this. Only the per-block checksum, seeded with the block's position (§ 7.2), binds a block to its index.

Backward Reading, from the end of the input, one frame at a time (§ 2.1):

  1. Parse the File Footer back from the end (§ 8) -> total_decompressed_size and compressed_frame_size, which gives where the frame's header starts.
  2. Read that File Header -> extract block_size; with HAS_SEEK_TABLE clear, the input is not seekable.
  3. Derive num_blocks = ceil(total_decompressed_size / block_size), in 64 bits: no field holds N, so nothing caps it but the footer's 64-bit size.
  4. Calculate seek_block_size = 8 + ⌈N / 64⌉ × 8 + N × 4, in 64 bits.
  5. Seek backward by seek_block_size bytes from the start of the footer (its sizes, and the 8-byte digest before them with HAS_CHECKSUM=1) to read the Block Header.
  6. Require the 8-byte Block Header to be the one SEK header N admits: type 254, Block Flags and Reserved 0, Compressed Payload Size the fold of ⌈N / 64⌉ × 8 + N × 4, and its header checksum; require an EOF block header 8 bytes before it.
  7. Nothing else is read at open. In a well-formed archive anchor + sizes lands exactly on the next anchor, or on the EOF block for the last group. A decoder validates a group alone when it accesses one of its blocks: anchor 0 is 16, every anchor lies in [16, EOF block], every size in [8, 8 + block_size + checksum_size], the group ends at or before the EOF block, the last one exactly on it. It rejects a block whose header disagrees with its entry's size, which catches an entry pointing into a block but not one moved onto another block of the same size. A block's size is always its own entry, never the gap to the next anchor. Checking that anchor is optional, and refuses an intact group when it is damaged.
  8. Anchors are offsets from the frame's own header, so a frame's table reads the same once concatenated. The frame before ends where this one starts: repeat from step 1 until the start of the input. Blocks and decompressed offsets then run across the frames in order; a block's checksum seed is still its index within its frame (§ 7.2).

Sequential Reading: the file header says what follows the EOF block, so a decoder never guesses from those bytes, which is unreliable: the footer opens with the digest or the source size, and one value in about 65536 parses as a valid SEK header. With HAS_SEEK_TABLE=1, it reads the SEK block header and rejects it unless it is, byte for byte, the one header the frame admits: type 254, Block Flags and Reserved 0, a Compressed Payload Size equal to the fold of T = ⌈N / 64⌉ × 8 + N × 4, N derived from the bytes it produced, and its header checksum. It then skips T bytes, counted in 64 bits, and reads the footer. With the flag clear, the footer comes next.

That header being fully determined, a decoder can check its bytes as they arrive: a differing byte is corruption, a matching but short prefix a truncation. With zero reserved bytes, a short footer behind a lying flag never matches its start, so it reads as corrupt, not truncated.

6. Prefix Varint (Extras section)#

ZXC extras use a prefix-length varint.

The length is encoded in unary form in the high bits of the first byte: the number of leading 1 bits, followed by a terminating 0, indicates how many additional payload bytes follow. The total length is therefore known from the first byte alone, and every following byte carries eight payload bits. This differs from LEB128, which spends a continuation bit in each byte and reveals the length only as the value is scanned.

The scheme generalizes to N bytes (11110xxx = 5, 111110xx = 6, ...), but the current ZXC spec caps the encoding at 3 bytes because no legitimate value exceeds 21 bits (see below).

Encodings used:

The first byte's payload bits — those below its unary prefix — are the least significant bits of the value, and each following byte contributes the next eight bits up. Writing b0, b1, b2 for the bytes in stream order, the accepted forms decode as:

1 byte:   value = b0
2 bytes:  value = (b0 AND 0x3F) OR (b1 << 6)
3 bytes:  value = (b0 AND 0x1F) OR (b1 << 5) OR (b2 << 13)

For example, the two bytes AC 04 decode as the 2-byte form: (0xAC AND 0x3F) OR (0x04 << 6) = 44 + 256 = 300. The three bytes C3 35 0C decode as the 3-byte form: (0xC3 AND 0x1F) OR (0x35 << 5) OR (0x0C << 13) = 3 + 1696 + 98304 = 100003.

Used by GLO/GHI to carry LL/ML overflows beyond token/sequence inline limits.

Value bound: a varint encodes (LL - MASK) or (ML - MASK). Since LL/ML are bounded by the largest block size a Chunk Size Code can select (2 MiB at code 21, see 3), every legitimate varint value is strictly less than 2^21 and therefore fits in at most 3 bytes.

Any prefix indicating a length >= 4 bytes (first byte >= 0xE0) is out of spec for this format version: encoders must never emit such a varint, and conforming decoders reject it as corrupt input. This caps the varint surface to the format-defined block size limit and neutralizes integer-overflow attacks in downstream bounds arithmetic. A future version of the format that raises the per-block size limit would also extend the accepted prefix lengths.

7. Checksums and Integrity#

7.1 Header checksums#

The file header carries a 16-bit checksum at 0x0E..0x0F; every block header carries an 8-bit checksum at 0x07. They are computed differently, because they are checked at different rates: the block checksum runs once per block, on the decode path, and must cost almost nothing; the file checksum runs once per archive, and can afford a guarantee.

All arithmetic below is on unsigned 64-bit integers modulo 2^64, and all shifts are logical.

Block header (8 bits)#

The 8-bit block header checksum takes the 8 header bytes as a single little-endian 64-bit integer v, with the checksum byte at 0x07 treated as zero, multiplies, and keeps the top byte:

h = (v XOR 0x9E3779B97F4A7C15) * 0x9E3779B97F4A7C15
checksum8 = h >> 56

Two properties are normative intent rather than implementation detail:

The XOR with the constant before multiplying keeps an all-zero header from hashing to zero. The cost is three instructions.

File header (16 bits)#

The 16-bit file header checksum takes the 16 header bytes as two little-endian 64-bit integers, v1 at 0x00 and v2 at 0x08, with the two checksum bytes at 0x0E..0x0F treated as zero. The halves are chained, not summed: the first is mixed, the second is added to the result and mixed again.

h = (v1 XOR 0xD2D84A61D2D84A61) * 0xD2D84A61D2D84A61
h = (h + v2 + 0x9E3779B97F4A7C15) * 0x9E3779B97F4A7C15
checksum16 = h >> 48

Why a chain and not a sum: with two independent products summed, a bit in each half can shift the two products by amounts whose top halfwords cancel, and that cancellation does not depend on the header content — a structural blind spot for two-bit errors. In the chain every flip still shifts the result by an exact amount (±2^k times the constants, the sign set by the bit's value), but the two halves no longer meet as equals, and the constants are chosen so that no single-bit or two-bit error can leave the top halfword unchanged, whatever the header holds. The order matters: 0xD2D84A61D2D84A61 must be the inner constant and 0x9E3779B97F4A7C15 the outer one; swapped, two-bit cancellations reappear. The reference decoder's test suite re-derives this from the constants. Beyond two bits, misses sit at the 1/65536 of a random function.

The same function checks the dictionary file header (§ 12), over the same bytes.

These protect metadata/navigation fields.

7.2 Per-block checksum (optional)#

When file header has HAS_CHECKSUM=1:

The seed binds a block to its position: a block moved elsewhere in the frame fails its own check, also under a range read through the seek table, which sees only the blocks it touches. The archive digest of § 7.3 is the ordered fold of these checksums, a whole-archive identity that a full decode verifies.

A decoder therefore verifies a block after decoding it, and a corrupted block reports whatever the decoder tripped on first. In exchange the checksum covers the whole pipeline: it catches a wrong dictionary accepted through a dict_id collision, an encoder or decoder defect, and a divergence between SIMD variants -- none of which touch the compressed bytes.

7.3 Archive digest (optional)#

When the file header sets HAS_CHECKSUM=1, an 8-byte digest opens the footer, before the sizes. It is an ordered 64-bit fold of every data block's checksum, in stream order:

digest = 0
for each data block, checksum c (the 4-byte value stored after the block):
    x      = ((u64)c + 1) * 0x9E3779B97F4A7C15
    digest = mix(digest XOR x, 0x2545F4914F6CDD1D)
mix(a, b): p = (u128)a * b; result = (u64)p XOR (u64)(p >> 64)

The digest identifies the archive as a whole: two archives of the same content at the same block size share it, and any block reordered, dropped or altered changes it. It is verified on a full decode with checksum verification on (and by zxc -t), never on a seekable range read, which cannot see every block. It is written into the footer and read back from it; it is not folded from the block payloads on the wire beyond their checksums.

Footer is mandatory, immediately after the EOF block header (or after the SEK block, on the seekable layout of § 5.5). It ends the frame.

Size      Field
8         archive_digest (u64)              -- only when HAS_CHECKSUM=1
nd        original_source_size (LE, nd = 1..8 bytes)
nf        compressed_frame_size (LE, nf = 1..8 bytes)
1         L = (nd - 1) | (nf - 1) << 4      -- bits 3 and 7 reserved, zero

Each size is written on the fewest bytes that hold it (0 takes one byte), and a decoder rejects any other length: a frame has exactly one valid footer. compressed_frame_size counts its own bytes; nf is the smallest length for which the total it yields fits in nf bytes, which is unique.

Reading back from the end (seekable reader, size queries): L gives nd and nf, so the two sizes are found without the file header; compressed_frame_size then gives the offset of the frame's header, and with HAS_CHECKSUM=1 the digest sits right before the sizes.

Reading forward (sequential decoders): a decoder knows the bytes it produced and the offset the footer starts at, so it computes the one footer the frame implies and compares it byte for byte, the digest aside; it reads no byte past L.

Example: 10 000 000 source bytes compressed to a 3 000 000-byte prefix, no checksums: 80 96 98 | C7 C6 2D | 22 (size 0x989680 on 3 bytes, compressed frame size 3 000 007 on 3, L = 0x22).

9. Decoder Validation Checklist (Practical)#

  1. Validate file header magic/version/checksum.
  2. Parse blocks sequentially:
    • validate block header checksum,
    • check block bounds using the Compressed Payload Size.
  3. Decode payload according to block type. For GLO/GHI:
    • reject Literal Encoding, Token Encoding and (GLO) Offset Encoding values outside § 5.2 (GHI: Literal or Token Encoding other than 0, § 5.3),
    • reject a block leaving fewer than 32 bytes behind its literal section,
    • check the derived section sizes still fit the payload.
  4. If enabled, verify the trailing block checksum over the decoded bytes, seeded with the block's position (§ 7.2).
  5. On EOF:
    • require the EOF header's one form § 5.4 (Block Flags, Reserved and comp_size all 0),
    • if HAS_SEEK_TABLE=1, read the SEK block header, require it to be the one § 5.5 derives from the output, Block Flags and Reserved 0, and skip the table,
    • compute the footer § 8 implies from the produced output size and the bytes read so far, read exactly that many bytes, and require them to match, the digest aside,
    • if verifying, compare the archive digest (§ 7.3) with the fold of the block checksums.
  6. After the footer, read the next 4 bytes and follow § 2.1: end of input, another frame (back to step 1), or reject.

10. Versioning Policy#

10.1 Format version field#

The format version is a single byte at offset 0x04 of the file header. A conforming decoder MUST reject any file whose version it does not support.

10.2 Version bump criteria#

ZXC has no forward compatibility: the set of block types and the meaning of every field are fixed per format version. Any change a decoder must understand — adding a block type, assigning meaning to a reserved field/flag bit, changing an encoding, layout, or the checksum algorithm — requires a version bump.

Change class Version action Example
New block type added Version bump (decoders reject unknown types) Adding a hypothetical GLR block type
Reserved field/flag bit assigned meaning Version bump Defining a reserved flag bit
Existing block encoding changed Version bump Changing GLO token layout
Header/footer layout changed Version bump Resizing the file header
Checksum algorithm changed Version bump Replacing RapidHash with Komihash

10.3 Compatibility rules#

10.4 Minimum conforming decoder#

A minimal conforming decoder for version 9 MUST support:

Support for checksum verification is RECOMMENDED but not strictly required for a minimal implementation.

11. Error Handling#

11.1 Error classes#

Decoders MUST detect and handle the following error conditions. The recommended behavior for each class is specified below.

Error Detection point Required behavior
Bad magic File header, offset 0x00 Reject immediately. Not a ZXC file.
Unsupported version File header, offset 0x04 Reject immediately. Version not supported.
File header checksum mismatch File header, offset 0x0E Reject. Header is corrupt or truncated.
Invalid chunk size code File header, offset 0x05 Reject. Code outside the valid range [12..21].
Block header checksum mismatch Block header, offset 0x07 Reject block. Stream is corrupt.
Unknown block type Block header, offset 0x00 Reject. The block-type set is fixed per format version (see §10.3); a decoder must not skip past unrecognised data.
Block payload truncated While reading the Compressed Payload Size bytes Reject. Unexpected end of stream.
Block checksum mismatch Trailing 4-byte checksum, after decoding the block Reject block. The decoded bytes are wrong: corrupt payload, wrong dictionary, or a block out of place.
EOF header not in its one form EOF block header Reject. Non-zero comp_size, Block Flags or Reserved (§5.4).
Data block comp_size above block size Block header, offset 0x03 Reject. A data block never compresses past its own content (§4.1).
Block walk ends without an EOF block End of the block walk Reject. A forged Compressed Payload Size can span the EOF marker; the resulting short decode must not be reported as success.
Seek-table flag disagrees with the tail Between the EOF block and the footer Reject. HAS_SEEK_TABLE=1 without the SEK header §5.5 derives from the output (Block Flags and Reserved 0), or any byte there with HAS_SEEK_TABLE=0.
Seek table group inconsistent SEK payload Reject. A size outside [8, one block], an anchor outside the data area, a group running past the EOF block, or a last group not ending on it (§5.5).
Block disagrees with its seek entry Block header, when a seekable reader accesses the block Reject. The entry's size is not header + payload + checksum of the block found there (§5.5).
Footer mismatch File footer, after the digest Reject. The sizes or L are not the ones the frame implies: wrong output size, wrong compressed frame size, a non-minimal length or a reserved bit set (§ 8).
Archive digest mismatch File footer, first 8 bytes (when HAS_CHECKSUM=1) Reject (if verifying). Blocks were reordered, dropped or altered (§7.3).
Decompressed output exceeds chunk size During LZ decode Reject. Corrupt or malicious payload.
Match offset out of bounds During LZ copy Reject. Offset references data before output start.
Varint exceeds maximum length Extras section Reject. Overflow or corrupt extras data.

11.2 Severity levels#

11.3 Partial output#

When a fatal error occurs mid-stream, the decoder SHOULD:

  1. Stop producing output immediately.
  2. Report the specific error condition (see zxc_error_name in the reference implementation).
  3. Not return partially decompressed data as a valid result.

Buffer-mode decoders MUST return a negative error code. Stream-mode decoders MUST signal the error and cease writing to the output.

11.4 Decoder hardening recommendations#

For decoders processing untrusted input (e.g. network data, user uploads):

12. Pre-Trained Dictionary Support#

12.1 Overview#

A pre-trained dictionary improves compression ratio on small, similar payloads (e.g. JSON API responses, game assets, structured logs) by prefilling the LZ77 sliding window at the start of each block. The dictionary is an external file (.zxd format) referenced by a 32-bit ID in the ZXC file header.

12.2 Mechanism#

The dictionary contains raw byte content (max 64 KB, bounded by the 64 KB LZ sliding window). At compression time, the dictionary is logically prepended to each block's input, seeding the hash tables so the match finder can reference dictionary content immediately. At decompression time, the dictionary is prepended to the output buffer so match copies that reference dictionary bytes resolve naturally via pointer arithmetic.

Since each block is independent, the dictionary prefill happens per-block. This preserves O(1) seekable random-access: load the dictionary once, then decompress any block independently.

12.3 File header encoding#

When HAS_DICTIONARY (flag bit 6) is set, the reserved bytes at offsets 0x07..0x0A contain the dict_id (u32 LE). A decoder MUST:

  1. Verify that a dictionary is provided; reject the archive if not.
  2. Verify that the dictionary id matches header.dict_id ; reject on mismatch. For a raw in-memory dictionary without a shared table, the id is fold32(rapidhash(content)) (zxc_dict_id(dict, dict_size, NULL)). When a shared literal table is attached, the id also binds the table: id = fold32(rapidhash(table_128_bytes, seed = fold32(rapidhash(content)))) (zxc_dict_id(content, size, table)). The seed is the folded 32-bit content hash, zero-extended to rapidhash's 64-bit seed. rapidhash and fold32 are defined in 7.2.

Blocks compressed with a dictionary contain match offsets that reference dictionary content; decoding without the dictionary produces corrupt output. Per-block checksums (when enabled) detect it, because they cover the decompressed bytes (§7.2). This also covers the residual risk of a 32-bit dict_id collision, where the header check passes on the wrong dictionary.

12.4 Dictionary file format (.zxd)#

Dictionaries are stored as standalone .zxd files with the following layout:

Offset  Size  Field
0x00    4     Magic Word (0x9CB0D1C7 LE)
0x04    1     Dictionary format version (currently 2)
0x05    1     Flags (bits 0..3: checksum algorithm id; bits 4..7 reserved)
0x06    2     Content size (u16 LE, max 65535)
0x08    4     dict_id (u32 LE, binds content AND shared table, see below)
0x0C    2     Reserved (0)
0x0E    2     Header Checksum (see 7.1, computed with 0x0C-0x0F zeroed)
0x10    N     Dictionary content (raw bytes)
0x10+N  128   Shared literal Huffman table (256 × 4-bit packed code lengths,
              same layout as the § 5.2.1 code-length header; always present)

12.5 Dictionary training#

The zxc_train_dict() function analyzes a corpus of representative samples to select byte segments that maximize LZ77 match coverage. The most frequently matched segments are placed at the end of the dictionary so they produce the shortest offsets (closest to the block start in the virtual window).

12.6 Naming convention#

The .zxd extension is cosmetic — files are identified by the magic word at offset 0x00, never by extension. This is a tooling convention, not a format requirement; it does not affect bytes on the wire. The reference CLI applies it as follows:

13. Summary of Useful Fixed Sizes#

Magic words — both are little-endian u32 at offset 0x00 and deliberately share the 0x9CB0... family prefix, so check the full value (or the file extension) to tell them apart:

File Magic (value) On-disk bytes (LE)
ZXC archive (.zxc) 0x9CB02EF5 F5 2E B0 9C
ZXC dictionary (.zxd) 0x9CB0D1C7 C7 D1 B0 9C

14. Worked Example (Real Hexdump)#

This example was produced with the CLI from a 10-byte input (Hello ZXC\n) using:

zxc -z -C -1 sample.txt

Generated archive size: 57 bytes.

14.1 Full hexdump#

00000000: F5 2E B0 9C 09 13 80 00 00 00 00 00 00 00 6D 86
00000010: 00 00 00 0A 00 00 00 A0 48 65 6C 6C 6F 20 5A 58
00000020: 43 0A 90 BB A1 75 FF 00 00 00 00 00 00 83 BD 8A
00000030: 9E 74 2A A2 9A B6 0A 39 00

14.2 Byte-level decoding#

A) File Header (offset 0x00, 16 bytes)#

F5 2E B0 9C | 09 | 13 | 80 | 00 00 00 00 00 00 00 | 6D 86

B) Data Block #0 (RAW)#

Block header at offset 0x10:

00 | 00 | 00 | 0A 00 00 00 | A0

Payload at 0x18..0x21 (10 bytes):

48 65 6C 6C 6F 20 5A 58 43 0A

ASCII: Hello ZXC\n.

Trailing block checksum at 0x22..0x25:

90 BB A1 75

LE value: 0x75A1BB90. This is a RAW block, so its payload already is the decompressed data and §7.2's input is those same ten bytes, seeded with the block's position, 0.

C) EOF Block (offset 0x26, 8 bytes)#

FF | 00 | 00 | 00 00 00 00 | 83
BD 8A 9E 74 2A A2 9A B6 | 0A | 39 | 00

14.3 Structural view with absolute offsets#

0x00..0x0F  File Header (16)
0x10..0x17  RAW Block Header (8)
0x18..0x21  RAW Payload (10)
0x22..0x25  RAW Block Checksum (4)
0x26..0x2D  EOF Block Header (8)
0x2E..0x35  Archive Digest (8)
0x36        File Source Size (1)
0x37        Compressed Frame Size (1)
0x38        Lengths byte L (1)

14.4 Seekable Variant (with Seek Table)#

Same 10-byte input (Hello ZXC\n), compressed with seekable mode enabled:

zxc -z -C -1 -S sample.txt

Generated archive size: 77 bytes (20 bytes larger than the non-seekable variant).

Full hexdump#

00000000: F5 2E B0 9C 09 13 A0 00 00 00 00 00 00 00 0D 45
00000010: 00 00 00 0A 00 00 00 A0 48 65 6C 6C 6F 20 5A 58
00000020: 43 0A 90 BB A1 75 FF 00 00 00 00 00 00 83 FE 00
00000030: 00 0C 00 00 00 6F 10 00 00 00 00 00 00 00 16 00
00000040: 00 00 BD 8A 9E 74 2A A2 9A B6 0A 4D 00

Byte-level decoding#

A) File Header (offset 0x00, 16 bytes) - as non-seekable, but for two fields:

B) Data Block #0 (RAW) (offset 0x10, 22 bytes) - identical to non-seekable.

C) EOF Block (offset 0x26, 8 bytes) - identical to non-seekable.

D) SEK Block (offset 0x2E, 20 bytes)

Block header at 0x2E:

FE | 00 | 00 | 0C 00 00 00 | 6F

Group 0 at 0x36:

10 00 00 00 00 00 00 00 | 16 00 00 00

E) File Footer (offset 0x42, 11 bytes)

BD 8A 9E 74 2A A2 9A B6 | 0A | 4D | 00

Structural view with absolute offsets#

0x00..0x0F  File Header (16)
0x10..0x17  RAW Block Header (8)
0x18..0x21  RAW Payload (10)
0x22..0x25  RAW Block Checksum (4)
0x26..0x2D  EOF Block Header (8)
0x2E..0x35  SEK Block Header (8)    <- seek table
0x36..0x3D  Group 0 anchor (8)      <- offset of block #0
0x3E..0x41  Block #0 size (4)
0x42..0x49  Archive Digest (8)
0x4A        File Source Size (1)
0x4B        Compressed Frame Size (1)
0x4C        Lengths byte L (1)

Compatibility note: The SEK block is inserted between the EOF block and the file footer, so the footer stays at the very end of the file (§ 8). Decoders that parse it from the end, starting with its last byte L, work unchanged with seekable archives. Streaming decoders, which reach the footer sequentially, learn from HAS_SEEK_TABLE whether a SEK block comes first.

15. Worked Example: Dictionary File (.zxd Hexdump)#

A minimal dictionary whose content is the 5 ASCII bytes hello. Total file size: 149 bytes (16-byte header + 5-byte content + 128-byte shared Huffman table). This is the on-disk form produced by zxc_dict_save() (see §12.4); the table is always present.

15.1 Full hexdump#

00000000: C7 D1 B0 9C 02 00 05 00 34 07 FC 0C 00 00 C6 51
00000010: 68 65 6C 6C 6F 00 00 00 00 00 00 00 00 00 00 00
00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000040: 00 00 00 00 00 00 00 20 00 02 00 02 20 00 00 00
00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000090: 00 00 00 00 00

15.2 Byte-level decoding#

A) Dictionary Header (offset 0x00, 16 bytes)#

C7 D1 B0 9C | 02 | 00 | 05 00 | 34 07 FC 0C | 00 00 | C6 51

B) Dictionary Content (offset 0x10, 5 bytes)#

68 65 6C 6C 6F

ASCII: hello. Raw bytes that prefill the LZ77 window — not compressed.

C) Shared Huffman Table (offset 0x15, 128 bytes)#

... 20 00 02 00 02 20 ...   (remaining bytes 0x00)

256 × 4-bit code lengths, packed two-per-byte (low nibble first), for the shared literal table (§5.2.2). Symbols absent from the training distribution have length 0; here only the four bytes of hello carry codes (e.g. the nibble at table index 'e'=0x65 gives length 2), so all other entries are zero.

15.3 Structural view with absolute offsets#

0x00..0x0F  Dictionary Header (16)
0x10..0x14  Dictionary Content (5)
0x15..0x94  Shared Huffman Table (128)
Next: , Previous: , Up: Documentation