Sizing guide

This guide turns the number of repositories you connect into worker disk, worker memory, PostgreSQL size and vector storage. Start with the profile on Self-hosting, then use this page to check the parts that grow with your repositories.

Two kinds of numbers appear below:

• Measured numbers come from MergeHarbor’s benchmarks on synthetic repositories, run on a 15-core laptop with PostgreSQL on the same machine. Expect a cloud vCPU to be about half as fast.
• Estimates show their arithmetic, so you can redo it with your own inputs. Once you have indexed some of your repositories, replace them with your own numbers: see Size from your own numbers.

Inputs

Input	Value	Kind
PostgreSQL per indexed file (files, symbols, call graph)	about 4.6 KB	Measured 4.3 KB, plus about 0.3 KB for the call graph
Vectors per indexed file	about 5.9	Measured on synthetic Go, Python and TypeScript code
Share of chunks embedded	about 60%	Measured 50–62%: files unchanged for a year are found by name and text instead
Bytes per stored vector, with its indexes (pgvector)	about 2.0 KB at 384 dimensions	Measured: vectors are stored as 16-bit floats
Code search index on the worker	about 2.7 × indexed code	Measured
Indexed files per repository	2,000	Assumption: use your own average
Mirror size of a typical repository	20–200 MB	Rule of thumb for a full mirror; MergeHarbor’s mirrors hold only the file contents it reads, usually far less

Files that are not indexed don’t count: files over 1 MB, binaries, generated and vendored files, and anything in .mergeharborignore. Identical files are stored and embedded once.

PostgreSQL per 1,000 repositories

At 2,000 indexed files each, 1,000 repositories are 2 million indexed files.

Part	Arithmetic	Estimate
Code graph	2M files × 4.6 KB	about 9 GB
Vectors, 384 dimensions	2M × 5.9 × 60% ≈ 7.1M vectors × 2.0 KB	about 14.5 GB
Vectors, 512 dimensions	7.1M vectors × 2.0 KB × 4/3	about 19 GB
Total, vectors in PostgreSQL (pgvector)	9 GB + 14.5–19 GB	about 24–29 GB

Vector sizes include their indexes. Repositories with no activity for 180 days give back their vectors (rebuilt on their next push or pull request), which lowers this further. Reviews, findings and the audit log grow with review activity, not with the number of repositories, so add room for them too.

Worker disk

Each worker keeps a mirror of its repositories and the code search index on its volume. Repositories are spread evenly over the workers of a cell, so each worker holds about repositories ÷ workers of them.

• Mirrors and the code search index share one quota: up to 80% of the volume (cells[].mirrors.maxPercent). When they reach it, the least recently used repository’s mirror and search index are removed together (never while in use) and rebuilt on its next job, so a full quota slows the first job on a cold repository but loses nothing; until the index is rebuilt, code search on that repository uses a slower plain search.

Profile	Default volume	Quota (mirrors and search index)	Mirrors kept warm at 20–200 MB each, before the search index (estimate)
small	50 GiB	40 GiB	about 200–2,000
medium	200 GiB	160 GiB	about 800–8,000
large	500 GiB	400 GiB	about 2,000–20,000

The search index comes out of the same quota. Example: a repository with 2,000 files of 1.25 KB has 2.5 MB of code and about 6.75 MB of index (estimate). If removals keep happening (mergeharbor_mirror_evictions_total, mergeharbor_zoekt_evictions_total), grow the volume (cells[].mirrors.size).

Give a large monorepo a worker of its own with cells[].pins (see Install with Helm).

With Docker Compose, the mirrors, the search index and PostgreSQL share one disk, and the 80% is a share of that whole disk. Set MIRROR_MAX_BYTES in .env to leave room for PostgreSQL, for example a quarter of the disk.

Worker memory

Profile	Worker request / limit	Index jobs at once	Reviews at once	Content per index batch
small	1.5 GiB / 2 GiB	2	4	32 MB
medium	6 GiB / 8 GiB	8	16	128 MB
large	24 GiB / 32 GiB	24	64	256 MB

Measured:

Work	Memory
Indexing a 1-million-file repository	200–230 MB
One index batch of 900 KB files (32 MB cap)	about 280 MB
Building the code search index	460 MB (small) to 990 MB
16 reviews at once	110–135 MB

Memory follows the batch, not the repository: indexing 300,000 files on the small profile used 43 MB more than at rest. Before a heavy job starts, the worker checks its memory and waits if it is above 85% of its limit. If workers still restart for lack of memory while indexing very large files, lower INDEX_BATCH_BYTES or INDEX_WORKERS.

Indexing time

Work	Measured
Parse and store, without embeddings	about 1,700 files/s on small
Plan a full repository	46,000–53,500 files/s
Onboard many small repositories	45–62 repositories/s (20 files each)

Estimate for 1,000 repositories of 2,000 files without embeddings: 2M files ÷ 1,700 files/s ≈ 20 minutes, about 40 minutes per worker on cloud CPUs. Embeddings usually take longer than that; see the next section.

Embedder

MergeHarbor calls any OpenAI-compatible /v1/embeddings endpoint you configure (EMBED_URL, EMBED_MODEL, EMBED_DIM).

	MergeHarbor’s embedder (BAAI/bge-small-en-v1.5, CPU) or your own server	Hosted embeddings API, e.g. text-embedding-3-small at 512 dimensions
Cost	No per-token charge; you run the CPU (or your own GPU server)	Per token, at your provider’s price; first indexes and re-embeds use the provider’s batch API at half price where it has one
Vectors per 1,000 repositories	384 dimensions: about 14.5 GB	512 dimensions: about 19 GB
Throughput	Measured: about 2.8 chunks/s per CPU	Your provider’s rate limits
Code leaves your network	No	Yes, to the provider

Estimate for embedding 1,000 repositories (about 7.1M chunks) with the CPU embedder: 7.1M ÷ 2.8 chunks/s ≈ 700 CPU-hours, so run several embedder replicas (embedder.replicas) or point EMBED_URL at a GPU server. Repositories are indexed by activity (recently active first), so the ones in use are searchable long before the rest. For a hosted API, the cost is indexed files × tokens per file × price per token; read tokens per file from your own numbers below.

Measure your own endpoint before you size for it. mergeharbor bench embed -url <endpoint> reports chunks per second, tokens per second and the best batch size and concurrency for it.

When to turn on Qdrant

Vectors are stored in PostgreSQL with pgvector by default. A repository with up to a million vectors is searched exactly (in parallel above 50,000); larger ones use a vector index.

Switch the cell to Qdrant (VECTOR_BACKEND: qdrant, cells[].qdrantUrl) when it passes about 10 million vectors. At 2,000 files per repository that is about 1,400 repositories (10M ÷ 7,100). Qdrant keeps vectors in memory compressed to 8 bits per dimension, about a quarter of their raw size, keeps the originals on disk and re-scores results with them.

Per 1,000 repositories (estimate)	384 dimensions	512 dimensions
Qdrant memory	7.1M × 384 bytes ≈ 2.7 GB	7.1M × 512 bytes ≈ 3.6 GB
Qdrant disk (originals)	about 11 GB	about 15 GB

When to add a cell

A cell is one PostgreSQL database, one vector store and the workers that serve its repositories. Add a cell when one of these is close:

• Repositories: one cell up to about 50,000; 2–8 cells for 50,000–500,000.
• Database size: 2–4 TB in one cell’s PostgreSQL.
• Workers: more workers in a cell than shardCount (256 by default).
• Vectors: one Qdrant cluster can no longer keep the cell’s compressed vectors in memory.

Shard ranges then move to the new cell; vectors and search indexes rebuild from your code hosts. Contact support before your first cell split.

Size from your own numbers

MergeHarbor records what indexing does, per repository and per UTC day with indexing activity, in the repo_index_cost_daily table of your main PostgreSQL database. It keeps 180 days.

Column	What it tells you
embed_tokens, embed_batch_tokens	Embedding tokens; on days with full_scan true, the first full index
embed_cost_usd, summary_cost_usd	Dollars for embeddings and file summaries, at the prices you set
index_seconds, index_runs	Worker time spent indexing, and completed index runs
vectors_stored, symbols_stored	Vectors and symbols the repository holds
mirror_bytes, zoekt_bytes	Worker disk for the mirror and the code search index

Average per repository, from each repository’s latest row:

SELECT count(*)                         AS repositories,
       avg(vectors_stored)::bigint      AS vectors,
       avg(mirror_bytes)::bigint        AS mirror_bytes,
       avg(zoekt_bytes)::bigint         AS search_index_bytes
FROM (
  SELECT DISTINCT ON (repo_id) repo_id, vectors_stored, mirror_bytes, zoekt_bytes
  FROM repo_index_cost_daily
  WHERE vectors_stored IS NOT NULL
  ORDER BY repo_id, day DESC
) latest;

Embedding tokens for a first full index, per repository:

SELECT avg(tokens)::bigint AS tokens_per_repository
FROM (
  SELECT repo_id, sum(embed_tokens + embed_batch_tokens) AS tokens
  FROM repo_index_cost_daily
  WHERE full_scan
  GROUP BY repo_id
) t;

Multiply by 1,000 and put these in place of the inputs above. To have dollars filled in, set your prices (all default to 0, unpriced):

Setting	Price of
EMBED_PRICE_USD_PER_MTOK	1M embedding tokens
EMBED_BATCH_PRICE_USD_PER_MTOK	1M batch embedding tokens (0 = same as above)
INDEX_COST_DISK_USD_PER_GB_MONTH	Worker disk (mirrors and search index)
INDEX_COST_DB_USD_PER_GB_MONTH	Database storage for vectors
INDEX_COST_COMPUTE_USD_PER_HOUR	One hour of one indexing job

Cost alerts

An hourly check flags an organization when its indexing cost for a day (embeddings, summaries and compute) is above all three of: INDEX_COST_ALERT_FLOOR_USD (20), INDEX_COST_ALERT_BASELINE_MULTIPLE (5) times its median of the previous INDEX_COST_ALERT_BASELINE_DAYS (14) days, and what its activity explains: repositories indexed in full that day × INDEX_COST_ONBOARDING_PER_REPO_USD (5.00), plus other active repositories × INDEX_COST_ALERT_PER_REPO_USD (0.50). Flagged days are in the index_cost_outliers table, and the Prometheus gauge mergeharbor_index_cost_outlier_orgs counts the organizations flagged for today or yesterday; the Helm chart’s monitoring rules alert on it.