Pick an output format
ndjson and csv write one line per event and use O(1) memory regardless of result size. json accumulates the entire result in memory before writing, and json-stream writes section by section but still isn’t built for very large results. table is for a human at a terminal, not for machine consumption or large runs.
Rule of thumb: above roughly 500k rows, don’t use
json or table.
Pick an index backend
auto switches to disk once the right-side file exceeds auto_max_right_file_mb, which defaults to 2048 MB when unset or zero. spill_dir is where disk, auto, and partitioned write temporary files; if omitted, Reconify uses the system temp directory.
Size the machine
The right-side index dominates peak memory. As a directly actionable reference, without needing the underlying formula:
This assumes typical string lengths for IDs, currency codes, and names. Long UUID-style identifiers or verbose merchant names push actual usage toward the higher end of a given row count. If your right side doesn’t fit comfortably at these sizes, use
disk, auto, or partitioned instead of memory.
Set a resource budget
max_memory_mb and max_temp_disk_mb are optional safety budgets; 0 leaves that budget uncapped. With a budget set, auto evaluates memory, then disk, then partitioned indexing in order, checks actual free space in spill_dir, and records the selected backend and the rejection reasons for candidates it passed over in structured output (or on stderr for csv/table).
Without a budget, auto keeps its plain file-size threshold behavior.
Say the honest thing here: these are resource safeguards, not throughput guarantees. If no candidate backend can meet its estimate, the run fails explicitly rather than degrading silently or running out of memory partway through.
Partitioning
partitioned hashes the configured matching key (reference, name, or group key) into temporary partition files for both inputs, then externally sorts and merge-reads one partition at a time.
partition_count: 0 selects a count adaptively, targeting roughly one million rows per active partition; explicit values must be 2 or higher. More partitions means less memory per partition but more partition-file overhead and disk passes.
--partition-workers defaults to 0, which is serial processing. Start small and compare row-level output against a serial run before increasing it; workers write to private disk-backed chunks and a single writer replays them in partition order, so output ordering is preserved either way.
Constraints worth knowing before you reach for this backend: it applies to CSV pairs only, and needs a consistent reference, name, or group-key selector across all passes. Duplicate groups must co-locate with the partition key; if they don’t, the CLI rejects partitioning outright and recommends memory or disk instead of silently producing wrong duplicate counts.
Watch a run
--progress logs to stderr, so it never corrupts NDJSON output or interferes with piped commands. For unattended jobs, write telemetry to a separate file instead:
progress.ndjson receives lifecycle events while results.ndjson keeps only reconciliation data. Telemetry carries rows/sec, elapsed time, stage, and source; totals and ETA are absent when computing them would cost a full extra scan. A telemetry sink failure never interrupts reconciliation itself: Reconify disables that one sink after reporting the failure once.
Reduce per-row cost
raw. Set skip_raw: true per source when you don’t need those fields in output; it cuts allocation pressure meaningfully at scale.
If name_mode: tokens is set, unmatched rows are buffered after reference matching for a second token-similarity pass. That buffer can become substantial on a large unmatched set:
--max-token-buffer 0 means unlimited, but only set that when you have memory to spare for the full unmatched set. If you don’t need name-token matching, set name_mode: none and skip the buffer entirely.
If a run still exhausts memory after switching to disk, the token-match buffer is the usual remaining culprit: reduce --max-token-buffer or disable token mode before reaching for partitioned.