Build your knowledge base

Index the code graph

Turn your mirrored repos into a queryable code graph: incremental indexing, and the full node and edge model across 14 languages, Terraform, SQL, and web topology.

Indexing turns your mirrored repos into a queryable knowledge graph. contextlake kb index --workspace ~/work walks every git repo under a folder and builds the graph. Runs are incremental by default; --force rebuilds from scratch.

flowchart TD
  D(["a directory you point kb index at"]) --> B{"--bundle passed?"}
  B -->|yes| ONE["indexed as one repository"]
  B -->|no| G{"is it itself a git repo?"}
  G -->|yes| ONE
  G -->|no| S{"does it hold git repos, with
nothing of yours outside them?"}
  S -->|no| ONE
  S -->|yes| R(["refused, naming the command that fits"])
  R -.->|"kb index --workspace"| EACH["each repo indexed under its own id"]
  ONE --> GR[("the graph")]
  EACH --> GR

The shape of the directory is measured before any of it is indexed, and the next section walks each outcome.

Which command for which directory#

kb index <dir> indexes that directory as one repository. kb index --workspace <dir> walks it and indexes each git repository under it separately, under its own identity. Getting those two the wrong way round is the most expensive mistake available here, so a directory that is not itself a git repository but holds some is refused, with the command that fits what was found:

$ contextlake kb index ~/work
✗ /home/you/work isn't itself a git repo, but contains 20 git working tree(s) at depths 1-2
  (alpha, beta, billing-core, frontend, gateway, …); no indexable file lies outside them.
  That is a workspace mirroring several repositories. Indexing it this way files all of them
  under ONE id, so every symbol they hold is in the graph twice and the copies cannot be told apart.
  → Run this instead, which indexes each repository separately under its own identity:
        contextlake kb index --workspace ~/work
  → Or pass --bundle to index this directory as one repository anyway. …

It refuses rather than quietly switching to --workspace for you, because switching can lose data: --workspace indexes the nested repositories and nothing outside them, so on a tree of your own loose sources that happens to carry a dependency with its own .git it would index the dependency and drop your files. So the shape is measured first, from how much indexable content lies outside the nested repositories, and only then does it decide:

--bundle opts back in. It indexes the directory as one repository regardless of shape, and it is read before anything is measured, so it always works. Reach for it when you genuinely want one bundled repository -- and note the cost you are accepting: the nested repositories' files are filed under the directory's name, so if you later index them properly they are in the graph twice. kb forget <repo-id> removes a bundle you did not mean to create.

Incremental and time-travel#

index --workspace is incremental, it re-indexes only repos whose git HEAD moved since their last index, so a scheduled (cron) run stays cheap; pass --force to rebuild everything, or --watch [--interval N] to keep re-indexing in a loop (the same --watch / --interval flags also drive connect and embed). Every indexed snapshot is kept, so query "<text>" --repo R --as-of <commit> does time-travel, it searches repo R as it was at a previously-indexed commit.

Re-indexing one repository by its id#

A repository's id is derived from its origin remote (so it survives being moved or re-cloned) and has no relation to where the clone sits on disk. That is the id kb lint and the dashboard report, so --source accepts it directly:

contextlake kb index --source example.com/team/widgets   # the id, exactly as reported
contextlake kb index --source widgets                    # the tail, when only one repo ends that way

The repository is re-indexed under the id it was already filed as, never a second time under its directory name. If the id is unknown, the error names near-miss ids from the store; if it is known but its recorded checkout is gone, the error names the path it was indexed from.

For a checkout that has no usable origin, or one you want filed under a different name, --repo sets the id explicitly: contextlake kb index --source ./widgets --repo team/widgets. Without it the id falls back to the directory name, which is exactly the fragile, path-dependent id this section exists to avoid, so name it once rather than living with a repo filed as widgets because that is what the folder happened to be called.

Parallelism and noise-pruning#

Repositories are parsed across worker processes (CPU-bound work) while the SQLite store is written serially from the parent. The spawn start method is used on every platform, so behaviour is identical on Linux, macOS, and Windows, with an automatic serial fallback if a worker pool can't start. It defaults to cpu_count - 1 workers (capped at 8); set [kb] index_workers to tune it (1 forces serial).

The parser also skips machine-generated and derived files: names like *.designer.cs, *.min.js and AssemblyInfo.cs, and any of four case-insensitive header markers in the first 2048 bytes, <auto-generated, @generated, do not edit and code generated by. That last pair catches more than it looks: a file whose banner just says "DO NOT EDIT" is skipped too. Also skipped are code files larger than [kb] max_file_bytes (5 MB). That's derived noise, not real source, and every skip is reported (no silent gaps). Set [kb] skip_generated = false or raise max_file_bytes to index them anyway. Discovery also skips vendored nested repos: an upstream clone carried inside the mirror with its own .git under a module-federation path segment, which is not your source and would flood the global graph with upstream-demo nodes. Each such skip is logged. (node_modules trees are already pruned before discovery descends into them.)

To exclude your own paths, drop a .contextlakeignore at a repo's root: one glob per line (# comments and blank lines ignored), matched against each file's path relative to the repo and its name, so *.lock ignores by name anywhere and vendor/ ignores a directory and everything under it. It's a small, dependency-free subset of gitignore syntax (no negation, **, or anchoring), enough to drop vendored trees and lockfiles from the graph.

Health and maintenance#

contextlake doctor checks the environment (FTS5, git / glab on PATH, the store, the embedder, and the ANN index) and exits non-zero if anything is wrong, so it doubles as a CI health gate. It also flags (advisory, doesn't affect the exit code) any shard indexed with an older parser version than the one currently installed -- contextlake kb index rebuilds those on its next run, so they pick up parser-correctness fixes. contextlake kb lint audits the graph itself, reporting stale repos (HEAD moved since they were indexed), dangling edges (an edge whose endpoint node is missing), and the same older-parser repos doctor reports, so the two commands never disagree about one store. Both exit non-zero on problems -- for lint that means dangling edges, HEAD-stale repos, or repos it cannot read; the older-parser count is reported but deliberately kept out of its exit code, so upgrading to a build with a new parser can't turn a green CI gate red on its own.

Two states are reported apart from stale, because re-indexing clears a stale repository and cannot clear either of them:

• empty -- the repository has no commits at all, so there is no HEAD and nothing to index. Reported, and not counted against the exit code: there is nothing for a reader to do about it.
• shard-imported -- indexed from a graph-shard JSON rather than a checkout, so it has no history to be behind. Also advisory, for the same reason.
• unreadable -- the recorded path no longer exists, or git will not answer for a repository there. Re-clone it or drop it from the store. This one does fail the run, because nothing can be cited from it.

What the graph captures#

Indexing builds a typed graph of your source. tree-sitter extracts files, classes, functions/methods, interfaces, imports, an intra-repo call graph, and an inheritance graph (inherits edges for extends / implements / base classes), so "what extends BaseController?" is one hop, and changing a base class shows its subclasses in blast_radius. The diagram below imports its colors from the same module contextlake kb graph and the dashboard render with, so node kinds match everywhere. Edge relations mostly do: 10 of them carry a dedicated hue (calls, imports, contains, depends_on, publishes, tracked_by, documented_by, flow, exposes, calls_http), while inherits and references ride the neutral default edge color, and reads, writes and transitions_to, which this page documents further down, are real edges the diagram does not show at all:

Languages#

tree-sitter covers 14 languages across 13 grammars (TypeScript and TSX share one grammar), and the parser registry is pluggable:

Frameworks are indexed through their base language: React / Next.js / Node.js are JS/TS(X), Angular is TS (its templates are HTML), and .NET is C#.

Missing yours? "Pluggable" is meant literally: a language is a grammar package plus a few table entries. Adding a language is the ordered recipe, with the verification commands that prove a new grammar works.

C++ completeness#

A method defined outside its class (ReturnType Class::method(...) { ... }, at any :: qualification depth) resolves to a method contained by its class, repo-wide -- not just the file it's declared in -- so an out-of-line definition in one file still shows up under its class even when the class itself is declared in a different header. A namespace { ... } block is its own containing node, the same way a class or file is. Header files (.h) are parsed as C++, so a class declared in a header and defined in a matching .cpp is visible as one unit rather than the class half going missing. This parsing choice is kept transparent to languages filtering: .h files are indexed whenever either "c" or "cpp" is enabled, since C and C++ headers are shared infrastructure -- restricting kb.toml's languages to just ["c"] still indexes .h files, no need to list both. An #ifdef/#else (or #ifndef/#else) pair -- two definitions of the same method in different branches of the same conditional -- collapses into one node instead of appearing as duplicate, ambiguous call targets. A bare #ifndef header guard with no #else does not trigger this: both copies would sit in the same (only) branch, so nothing collapses, and genuinely distinct overloads anywhere are kept as separate definitions, never merged into one. See Health and maintenance above: doctor flags any C/C++ shard indexed before these fixes landed, as a signal to re-index.

Infrastructure: Terraform / HCL#

.tf files build an infrastructure dependency graph: resource / data / variable / output / module / local definitions with depends_on edges resolving var. / module. / data. / resource references across files in a repo. resource nodes are semantically searchable.

Resolution is repo-wide, so a block address defined identically in separate root-module directories (for example environments/prod and environments/staging) surfaces as an AMBIGUOUS edge; directory-scoped resolution is a future refinement. Render it with contextlake kb graph --repo <repo> --format deploymentdiagram (a Mermaid flowchart grouped by inferred category: network/compute/storage/ database/security/other/module, where other catches any resource type none of the keyword lists claims), see Visualize.

Databases: SQL DDL#

.sql files build a referential graph: table / view / procedure definitions with references edges from foreign-key REFERENCES clauses, resolved across files in a repo. table and view nodes are semantically searchable.

It uses a regex DDL extractor (the fleet's T-SQL/PL-SQL defeats a tree-sitter AST), so it targets the high-value defs and FK references and is a deliberate undercount. Render it with contextlake kb graph --repo <repo> --format erdiagram (a Mermaid ER diagram), see Visualize.

Measured, not just asserted: tests/kb/fixtures/sql/ is a small, synthetic, hand-labelled orders/customers/inventory corpus with a checked ground truth of every FK a human reading the DDL would call real (expected_edges.json, 13 edges); tests/kb/test_sql_fixture_corpus.py scores the parser's emitted references edges against it on every CI run, and asserts this page still quotes what it measures. Current numbers on that corpus: precision 1.00 (9 true positives / 0 false positives), recall 0.69 (9 / 13 ground-truth edges found), a small, hand-built corpus, not a claim about the whole fleet, but real and reproducible. Two documented gap classes account for the four missed edges, both by design, not bugs: a self-referencing FK (referred_by/parent_category_id-style hierarchies) is dropped because the extractor excludes target == name, and an FK attached via a separate ALTER TABLE ... ADD CONSTRAINT statement is never captured because the scope tracker only scans REFERENCES inside a CREATE TABLE's own text span. Precision was 0.90 on this corpus until the extractor learned to blank out -- and /* */ comments before matching: its one false positive was a commented-out REFERENCES line, dead DDL history that resolved into a real-looking edge because the table it named still existed elsewhere in the repo. That case is now pinned as a negative in the corpus test. These are the numbers to distrust a graph INFERRED SQL edge by, and the floors in the corpus test are meant to be ratcheted up as the extractor improves, not treated as a target already met.

Architecture decisions (ADRs)#

A repo's own decision records become first-class adr nodes in that repo's shard. The match is any .md file with a directory segment named adr, adrs or decisions anywhere in its path, case insensitively, so docs/adr/, docs/decisions/, decisions/, adrs/ and architecture/adr/ all qualify, one file per decision. Each node takes its title from the file's first # heading (or the filename otherwise). Unlike connector-sourced content (enrich/connect, external systems reached over the network), an ADR is authored, checked into the repo's own git history: a recorded fact, not something to attribute or hedge on. adr nodes are semantically searchable, and their content feeds into wiki generation as a grounded "Recorded decisions" section, cited alongside the repo's other extracted facts. No column data, no edges to other graph nodes: an ADR mentioning a class by name isn't a verified reference the way an import or call site is, so nothing is inferred from that mention.

Entity state machines#

Guarded assignments to a status/state/stage field (if order.status == Created: order.status = Paid) become transitions_to edges between state nodes, labeled with the method that makes the transition. Only guarded transitions are emitted: the source state must be established by a preceding comparison on the same field, so a diagram never claims a transition the code doesn't actually establish. Python, JS/TS, and C# (regex, every edge INFERRED). Render with contextlake kb graph --repo <repo> --format statediagram (a Mermaid entity state machine), see Visualize.

transitions_to is deliberately not in impact's default relation set: unlike a table schema a query depends on, a state value is rarely a thing other code breaks against when it changes (renaming an enum member is a language-level rename, not a graph-discoverable break); pass --relation transitions_to explicitly if you do want that walk.

Intra-repo dataflow: reads and writes#

Application code querying a table or view it never explicitly imports still shows up in the graph: a literal SELECT ... FROM / INSERT INTO / UPDATE ... SET / DELETE FROM in a string (any language, SQL text looks the same embedded anywhere) becomes a reads or writes edge from the file to the table/view node the SQL DDL extractor already found, resolved by name across the whole repo the same way an FK references edge is. A query against a table this repo never defines is an honest miss, not a guessed link. reads/writes are in impact's default relation set, so contextlake kb impact <table> answers "what code touches this table" out of the box.

Web topology: endpoints and routes#

Two web-topology layers sit on top of the definitions.

HTTP endpoints a repo exposes or calls become shared endpoint nodes that join across repos into flow edges, from the caller repo to the exposer repo. Detection is regex-based and framework-targeted, so read the list as what is actually matched:

• Python: FastAPI and Flask decorator routing (@app.get("..."), @router.route("...")).
• JavaScript and TypeScript: the Express-shaped call form, app/router/api/server .get|.post|.put|.delete|.patch("..."). Fastify code that happens to use one of those variable names is caught incidentally. NestJS is not supported: its @Controller / @Get() decorator routing matches nothing here.
• C#: minimal-API MapGet/MapPost/… and attribute routing, [HttpGet("...")] and [Route("...")], so a conventional MVC or Web API controller is picked up too.
• Next.js App Router route.ts handlers, via the file convention.

Every one of these edges is INFERRED by construction, a likely undercount rather than an assertion.

Frontend routes become repo-scoped, embeddable route nodes from three frameworks:

The object-literal forms use a tree-sitter AST walk anchored on the route-table container (a Routes-typed declarator, or the array argument to RouterModule.forRoot / RouterModule.forChild / provideRouter / create*Router), so nested children compose into full paths and bare {path:...} config objects are never mis-read as routes.

Manifests and cross-repo dependencies#

Indexing also reads manifests (pyproject.toml, package.json, *.csproj, pom.xml) to build a cross-repo dependency graph through shared package nodes. Agents traverse all of this over MCP, from finding a definition to cross-repo blast_radius ("what could break if I change this"); see the full tool list under Serve.

The same change-impact walk is a one-liner from the shell: contextlake kb impact <symbol> [--hops N] lists what calls / depends on a node, no editor needed. When a symbol name (e.g. Node, Catalog) is defined in more than one repo, impact lists the candidates and you narrow it with --repo <repo> rather than getting a silent best-guess.

See also#

• Connect and enrich
• Semantic search
• Serve it to your editor