# Sizing guide

> How much worker disk, worker memory, PostgreSQL and vector storage self-hosted MergeHarbor needs per 1,000 repositories, which embedder to run, when to turn on Qdrant or add a cell, and how to size from your own indexing numbers.

This guide turns the number of repositories you connect into worker disk, worker memory, PostgreSQL size and vector storage. Start with the `profile` on [Self-hosting](https://mergeharbor.com/docs/self-hosting#sizing), then use this page to check the parts that grow with your repositories.

Two kinds of numbers appear below:

- **Measured** numbers come from MergeHarbor's benchmarks on synthetic repositories, run on a 15-core laptop with PostgreSQL on the same machine. Expect a cloud vCPU to be about half as fast.
- **Estimates** show their arithmetic, so you can redo it with your own inputs. Once you have indexed some of your repositories, replace them with your own numbers: see [Size from your own numbers](#size-from-your-own-numbers).

## Inputs

| Input                                                    | Value                    | Kind                                                  |
| -------------------------------------------------------- | ------------------------ | ----------------------------------------------------- |
| PostgreSQL per indexed file (files, symbols, call graph) | about 4.6 KB             | Measured 4.3 KB, plus about 0.3 KB for the call graph |
| Vectors per indexed file                                 | about 5.9                | Measured on synthetic Go, Python and TypeScript code  |
| Share of chunks embedded                                 | about 60%                | Measured 50–62%: files unchanged for a year are found by name and text instead |
| Bytes per stored vector, with its indexes (pgvector)      | about 2.0 KB at 384 dimensions | Measured: vectors are stored as 16-bit floats |
| Code search index on the worker                          | about 2.7 × indexed code | Measured                                              |
| Indexed files per repository                             | 2,000                    | Assumption: use your own average                      |
| Mirror size of a typical repository                      | 20–200 MB                | Rule of thumb for a full mirror; MergeHarbor's mirrors hold only the file contents it reads, usually far less |

Files that are not indexed don't count: files over 1 MB, binaries, generated and vendored files, and anything in `.mergeharborignore`. Identical files are stored and embedded once.

## PostgreSQL per 1,000 repositories

At 2,000 indexed files each, 1,000 repositories are 2 million indexed files.

| Part                                    | Arithmetic                             | Estimate        |
| --------------------------------------- | -------------------------------------- | --------------- |
| Code graph                              | 2M files × 4.6 KB                      | about 9 GB      |
| Vectors, 384 dimensions                 | 2M × 5.9 × 60% ≈ 7.1M vectors × 2.0 KB | about 14.5 GB   |
| Vectors, 512 dimensions                 | 7.1M vectors × 2.0 KB × 4/3            | about 19 GB     |
| Total, vectors in PostgreSQL (pgvector) | 9 GB + 14.5–19 GB                      | about 24–29 GB  |

Vector sizes include their indexes. Repositories with no activity for 180 days give back their vectors (rebuilt on their next push or pull request), which lowers this further. Reviews, findings and the audit log grow with review activity, not with the number of repositories, so add room for them too.

## Worker disk

Each worker keeps a mirror of its repositories and the code search index on its volume. Repositories are spread evenly over the workers of a cell, so each worker holds about _repositories ÷ workers_ of them.

- **Mirrors and the code search index** share one quota: up to 80% of the volume (`cells[].mirrors.maxPercent`). When they reach it, the least recently used repository's mirror and search index are removed together (never while in use) and rebuilt on its next job, so a full quota slows the first job on a cold repository but loses nothing; until the index is rebuilt, code search on that repository uses a slower plain search.

| Profile  | Default volume | Quota (mirrors and search index) | Mirrors kept warm at 20–200 MB each, before the search index (estimate) |
| -------- | -------------: | -------------------------------: | ----------------------------------------------------------------------: |
| `small`  |         50 GiB |                           40 GiB |                                                         about 200–2,000 |
| `medium` |        200 GiB |                          160 GiB |                                                         about 800–8,000 |
| `large`  |        500 GiB |                          400 GiB |                                                      about 2,000–20,000 |

The search index comes out of the same quota. Example: a repository with 2,000 files of 1.25 KB has 2.5 MB of code and about 6.75 MB of index (estimate). If removals keep happening (`mergeharbor_mirror_evictions_total`, `mergeharbor_zoekt_evictions_total`), grow the volume (`cells[].mirrors.size`).

Give a large monorepo a worker of its own with `cells[].pins` (see [Install with Helm](https://mergeharbor.com/docs/self-hosting-helm)).

With Docker Compose, the mirrors, the search index and PostgreSQL share one disk, and the 80% is a share of that whole disk. Set `MIRROR_MAX_BYTES` in `.env` to leave room for PostgreSQL, for example a quarter of the disk.

## Worker memory

| Profile  | Worker request / limit | Index jobs at once | Reviews at once | Content per index batch |
| -------- | ---------------------- | -----------------: | --------------: | ----------------------: |
| `small`  | 1.5 GiB / 2 GiB        |                  2 |               4 |                   32 MB |
| `medium` | 6 GiB / 8 GiB          |                  8 |              16 |                  128 MB |
| `large`  | 24 GiB / 32 GiB        |                 24 |              64 |                  256 MB |

Measured:

| Work                                        | Memory                     |
| ------------------------------------------- | -------------------------- |
| Indexing a 1-million-file repository        | 200–230 MB                 |
| One index batch of 900 KB files (32 MB cap) | about 280 MB               |
| Building the code search index              | 460 MB (`small`) to 990 MB |
| 16 reviews at once                          | 110–135 MB                 |

Memory follows the batch, not the repository: indexing 300,000 files on the `small` profile used 43 MB more than at rest. Before a heavy job starts, the worker checks its memory and waits if it is above 85% of its limit. If workers still restart for lack of memory while indexing very large files, lower `INDEX_BATCH_BYTES` or `INDEX_WORKERS`.

## Indexing time

| Work                                | Measured                             |
| ----------------------------------- | ------------------------------------ |
| Parse and store, without embeddings | about 1,700 files/s on `small`       |
| Plan a full repository              | 46,000–53,500 files/s                |
| Onboard many small repositories     | 45–62 repositories/s (20 files each) |

Estimate for 1,000 repositories of 2,000 files without embeddings: 2M files ÷ 1,700 files/s ≈ 20 minutes, about 40 minutes per worker on cloud CPUs. Embeddings usually take longer than that; see the next section.

## Embedder

MergeHarbor calls any OpenAI-compatible `/v1/embeddings` endpoint you configure (`EMBED_URL`, `EMBED_MODEL`, `EMBED_DIM`).

|                                | MergeHarbor's embedder (`BAAI/bge-small-en-v1.5`, CPU) or your own server | Hosted embeddings API, e.g. `text-embedding-3-small` at 512 dimensions |
| ------------------------------ | ------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| Cost                           | No per-token charge; you run the CPU (or your own GPU server)             | Per token, at your provider's price; first indexes and re-embeds use the provider's batch API at half price where it has one |
| Vectors per 1,000 repositories | 384 dimensions: about 14.5 GB                                             | 512 dimensions: about 19 GB                                            |
| Throughput                     | Measured: about 2.8 chunks/s per CPU                                      | Your provider's rate limits                                            |
| Code leaves your network       | No                                                                        | Yes, to the provider                                                   |

Estimate for embedding 1,000 repositories (about 7.1M chunks) with the CPU embedder: 7.1M ÷ 2.8 chunks/s ≈ 700 CPU-hours, so run several embedder replicas (`embedder.replicas`) or point `EMBED_URL` at a GPU server. Repositories are indexed by activity (recently active first), so the ones in use are searchable long before the rest. For a hosted API, the cost is _indexed files × tokens per file × price per token_; read tokens per file from your own numbers below.

Measure your own endpoint before you size for it. `mergeharbor bench embed -url <endpoint>` reports chunks per second, tokens per second and the best batch size and concurrency for it.

## When to turn on Qdrant

Vectors are stored in PostgreSQL with pgvector by default. A repository with up to a million vectors is searched exactly (in parallel above 50,000); larger ones use a vector index.

Switch the cell to Qdrant (`VECTOR_BACKEND: qdrant`, `cells[].qdrantUrl`) when it passes about **10 million vectors**. At 2,000 files per repository that is about 1,400 repositories (10M ÷ 7,100). Qdrant keeps vectors in memory compressed to 8 bits per dimension, about a quarter of their raw size, keeps the originals on disk and re-scores results with them.

| Per 1,000 repositories (estimate) | 384 dimensions             | 512 dimensions             |
| --------------------------------- | -------------------------- | -------------------------- |
| Qdrant memory                     | 7.1M × 384 bytes ≈ 2.7 GB  | 7.1M × 512 bytes ≈ 3.6 GB  |
| Qdrant disk (originals)           | about 11 GB                | about 15 GB                |

## When to add a cell

A cell is one PostgreSQL database, one vector store and the workers that serve its repositories. Add a cell when one of these is close:

- **Repositories:** one cell up to about 50,000; 2–8 cells for 50,000–500,000.
- **Database size:** 2–4 TB in one cell's PostgreSQL.
- **Workers:** more workers in a cell than `shardCount` (256 by default).
- **Vectors:** one Qdrant cluster can no longer keep the cell's compressed vectors in memory.

Shard ranges then move to the new cell; vectors and search indexes rebuild from your code hosts. Contact support before your first cell split.

## Size from your own numbers

MergeHarbor records what indexing does, per repository and per UTC day with indexing activity, in the `repo_index_cost_daily` table of your main PostgreSQL database. It keeps 180 days.

| Column                               | What it tells you                                                     |
| ------------------------------------ | --------------------------------------------------------------------- |
| `embed_tokens`, `embed_batch_tokens` | Embedding tokens; on days with `full_scan` true, the first full index |
| `embed_cost_usd`, `summary_cost_usd` | Dollars for embeddings and file summaries, at the prices you set      |
| `index_seconds`, `index_runs`        | Worker time spent indexing, and completed index runs                  |
| `vectors_stored`, `symbols_stored`   | Vectors and symbols the repository holds                              |
| `mirror_bytes`, `zoekt_bytes`        | Worker disk for the mirror and the code search index                  |

Average per repository, from each repository's latest row:

```sql
SELECT count(*)                         AS repositories,
       avg(vectors_stored)::bigint      AS vectors,
       avg(mirror_bytes)::bigint        AS mirror_bytes,
       avg(zoekt_bytes)::bigint         AS search_index_bytes
FROM (
  SELECT DISTINCT ON (repo_id) repo_id, vectors_stored, mirror_bytes, zoekt_bytes
  FROM repo_index_cost_daily
  WHERE vectors_stored IS NOT NULL
  ORDER BY repo_id, day DESC
) latest;
```

Embedding tokens for a first full index, per repository:

```sql
SELECT avg(tokens)::bigint AS tokens_per_repository
FROM (
  SELECT repo_id, sum(embed_tokens + embed_batch_tokens) AS tokens
  FROM repo_index_cost_daily
  WHERE full_scan
  GROUP BY repo_id
) t;
```

Multiply by 1,000 and put these in place of the inputs above. To have dollars filled in, set your prices (all default to 0, unpriced):

| Setting                            | Price of                                      |
| ---------------------------------- | --------------------------------------------- |
| `EMBED_PRICE_USD_PER_MTOK`         | 1M embedding tokens                           |
| `EMBED_BATCH_PRICE_USD_PER_MTOK`   | 1M batch embedding tokens (0 = same as above) |
| `INDEX_COST_DISK_USD_PER_GB_MONTH` | Worker disk (mirrors and search index)        |
| `INDEX_COST_DB_USD_PER_GB_MONTH`   | Database storage for vectors                  |
| `INDEX_COST_COMPUTE_USD_PER_HOUR`  | One hour of one indexing job                  |

### Cost alerts

An hourly check flags an organization when its indexing cost for a day (embeddings, summaries and compute) is above all three of: `INDEX_COST_ALERT_FLOOR_USD` (20), `INDEX_COST_ALERT_BASELINE_MULTIPLE` (5) times its median of the previous `INDEX_COST_ALERT_BASELINE_DAYS` (14) days, and what its activity explains: repositories indexed in full that day × `INDEX_COST_ONBOARDING_PER_REPO_USD` (5.00), plus other active repositories × `INDEX_COST_ALERT_PER_REPO_USD` (0.50). Flagged days are in the `index_cost_outliers` table, and the Prometheus gauge `mergeharbor_index_cost_outlier_orgs` counts the organizations flagged for today or yesterday; the Helm chart's monitoring rules alert on it.