> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reconifyhq.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Reconify's API reference is read-only for customer data. Do not invent endpoints or authentication behavior beyond the OpenAPI contract.
> The public OpenAPI document contains only the documented external /v2 contract.

# Handle large files

> Keep a reconciliation inside available memory, with visibility while it runs.

Reconify indexes the right source, then streams the left source row by row. Peak memory is set by the **right-side index**, not the combined size of both files.

That one fact drives every decision on this page: if a run is slow, exhausts memory, or produces an output file too large to open, the right-side index and the output format are almost always where the fix lives.

## Pick an output format

`ndjson` and `csv` write one line per event and use O(1) memory regardless of result size. `json` accumulates the entire result in memory before writing, and `json-stream` writes section by section but still isn't built for very large results. `table` is for a human at a terminal, not for machine consumption or large runs.

| Format        | Best for                                                     |
| ------------- | ------------------------------------------------------------ |
| `ndjson`      | Piping into `jq`, streaming to downstream tools.             |
| `csv`         | Loading into spreadsheets or databases row by row.           |
| `json-stream` | Section-by-section JSON without one massive buffered object. |
| `json`        | Small results, or when a consumer needs one JSON object.     |
| `table`       | Reading in a terminal. Never for large runs.                 |

Rule of thumb: above roughly 500k rows, don't use `json` or `table`.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
reconify reconcile --config reconify.yaml --pair left_source_vs_right_source --format ndjson --out results.ndjson
```

## Pick an index backend

| Backend       | Use when                                                                                                                      |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `memory`      | The right-side index fits comfortably in RAM. Fastest.                                                                        |
| `disk`        | You need lower RAM usage and can accept slower point lookups; spills to a SQLite-backed file.                                 |
| `auto`        | Threshold-based selection by default, or resource-aware memory/disk/partitioned fallback when you set budgets. Start here.    |
| `partitioned` | Bounded memory for a large single-counterpart CSV run, including grouped passes, at the cost of extra sequential disk passes. |

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
index:
  backend: auto
  spill_dir: "/tmp/reconify"
  auto_max_right_file_mb: 2048
```

`auto` switches to `disk` once the right-side file exceeds `auto_max_right_file_mb`, which defaults to `2048` MB when unset or zero. `spill_dir` is where `disk`, `auto`, and `partitioned` write temporary files; if omitted, Reconify uses the system temp directory.

## Size the machine

The right-side index dominates peak memory. As a directly actionable reference, without needing the underlying formula:

| Right rows | Left rows | Recommended RAM |
| ---------- | --------- | --------------- |
| 1M         | 1M        | 4 GB            |
| 10M        | 10M       | 8 GB            |
| 20M        | 20M       | 16 GB           |
| 50M        | 50M       | 32 GB           |
| 100M       | 100M      | 64 GB           |

This assumes typical string lengths for IDs, currency codes, and names. Long UUID-style identifiers or verbose merchant names push actual usage toward the higher end of a given row count. If your right side doesn't fit comfortably at these sizes, use `disk`, `auto`, or `partitioned` instead of `memory`.

## Set a resource budget

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
index:
  backend: auto
  spill_dir: "/var/tmp/reconify"
  max_memory_mb: 8192
  max_temp_disk_mb: 16384
```

`max_memory_mb` and `max_temp_disk_mb` are optional safety budgets; `0` leaves that budget uncapped. With a budget set, `auto` evaluates memory, then disk, then partitioned indexing in order, checks actual free space in `spill_dir`, and records the selected backend and the rejection reasons for candidates it passed over in structured output (or on stderr for `csv`/`table`).

Without a budget, `auto` keeps its plain file-size threshold behavior.

Say the honest thing here: **these are resource safeguards, not throughput guarantees.** If no candidate backend can meet its estimate, the run fails explicitly rather than degrading silently or running out of memory partway through.

## Partitioning

`partitioned` hashes the configured matching key (reference, name, or group key) into temporary partition files for both inputs, then externally sorts and merge-reads one partition at a time.

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
index:
  backend: partitioned
  partition_count: 32
```

`partition_count: 0` selects a count adaptively, targeting roughly one million rows per active partition; explicit values must be `2` or higher. More partitions means less memory per partition but more partition-file overhead and disk passes.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
reconify reconcile \
  --config reconify.yaml \
  --pair left_source_vs_right_source \
  --format ndjson \
  --partition-workers 4 \
  --out results.ndjson
```

`--partition-workers` defaults to `0`, which is serial processing. Start small and compare row-level output against a serial run before increasing it; workers write to private disk-backed chunks and a single writer replays them in partition order, so output ordering is preserved either way.

Constraints worth knowing before you reach for this backend: it applies to CSV pairs only, and needs a consistent reference, name, or group-key selector across all passes. Duplicate groups must co-locate with the partition key; if they don't, the CLI rejects partitioning outright and recommends `memory` or `disk` instead of silently producing wrong duplicate counts.

## Watch a run

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
reconify reconcile \
  --config reconify.yaml \
  --pair left_source_vs_right_source \
  --format ndjson \
  --progress \
  --out results.ndjson
```

`--progress` logs to stderr, so it never corrupts NDJSON output or interferes with piped commands. For unattended jobs, write telemetry to a separate file instead:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
reconify reconcile \
  --config reconify.yaml \
  --pair left_source_vs_right_source \
  --format ndjson \
  --progress-out progress.ndjson \
  --heartbeat-every 30s \
  --out results.ndjson
```

`progress.ndjson` receives lifecycle events while `results.ndjson` keeps only reconciliation data. Telemetry carries rows/sec, elapsed time, stage, and source; totals and ETA are absent when computing them would cost a full extra scan. A telemetry sink failure never interrupts reconciliation itself: Reconify disables that one sink after reporting the failure once.

## Reduce per-row cost

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
parser:
  skip_raw: true
```

By default every parsed transaction carries its original row fields in `raw`. Set `skip_raw: true` per source when you don't need those fields in output; it cuts allocation pressure meaningfully at scale.

If `name_mode: tokens` is set, unmatched rows are buffered after reference matching for a second token-similarity pass. That buffer can become substantial on a large unmatched set:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
reconify reconcile --config reconify.yaml --pair left_source_vs_right_source --max-token-buffer 100000
```

`--max-token-buffer 0` means unlimited, but only set that when you have memory to spare for the full unmatched set. If you don't need name-token matching, set `name_mode: none` and skip the buffer entirely.

If a run still exhausts memory after switching to `disk`, the token-match buffer is the usual remaining culprit: reduce `--max-token-buffer` or disable token mode before reaching for `partitioned`.