Graphify-Labs/graphify

Chunking. Non-code files (docs, PDFs, images) are processed by the LLM-based semantic pipeline in llm.py. Files are first split at _FILE_CHAR_CAP (20,000 characters) into FileSlice units via expand_oversized_files (llm.py:2638). These units are then packed into chunks by _pack_chunks_by_tokens() (llm.py:2160–2203), using a greedy algorithm that groups by parent directory (so related files share one LLM call) and closes a chunk when a configurable token budget (default 60,000 tokens) would be exceeded, with a hard cap of 20 images per chunk.

Entity and relation extraction. Code files are handled entirely deterministically via tree-sitter AST extractors in graphify/extractors/ — one per language (Python, JS/TS, Rust, Go, Java, C++, etc.). These produce nodes representing classes, functions, imports, and variables, and edges for calls, imports, inherits, references, etc. The LLM semantic pass uses a detailed system prompt (_EXTRACTION_SYSTEM in llm.py:481–511) that defines a JSON schema with nodes, edges, and hyperedges. Every edge is tagged with a confidence tier — EXTRACTED, INFERRED, or AMBIGUOUS — and each node carries a file_type of code, document, paper, image, rationale, or concept. Injection sentinels in source text are neutralised (_neutralise_injection_sentinels, llm.py:575–582). Fabrication is mitigated via _bind_node_evidence (llm.py:703–767): code-typed nodes whose symbol name has no substring match in the file bytes the model actually read are flagged with verification = "unverified".

Entity resolution / deduplication. The dedup.py module implements a pipeline: deduplicate_entities() (dedup.py:557–657) runs exact-ID dedup first (one survivor per node ID, deterministically ranked by _collision_rank), then fuzzy label-based dedup via a cascade: exact normalization → entropy gate (_entropy ≥ 2.5, dedup.py:29–38) → MinHash/LSH blocking at threshold 0.7 (dedup.py:48–53) → Jaro-Winkler verification at 92.0 (dedup.py:247) → same-community boost (+5.0) → union-find merge. An optional LLM pass (dedup_llm_backend) resolves ambiguous pairs in the 75–92 Jaro-Winkler zone.

Schema / ontology. There is no formal ontology. The type vocabulary is limited to file_type on nodes and relation on edges. The ID scheme follows a {path}_{entity} pattern where the path is the repo-relative file path with extension dropped, all segments joined by underscores (llm.py:500).

In-memory NetworkX. The graph lives in memory as a networkx.Graph (or nx.DiGraph when directed=True) throughout building, clustering, analysis, and query serving. build_from_json() (build.py:877–1038) constructs it: nodes are added with all attributes except id unpacked as kwargs, and edges are added with their metadata.

Serialised to JSON on disk. The canonical serialisation is graph.json, written by to_json() in export.py (line 272). It uses NetworkX's node_link_data format: a JSON object with top-level nodes (array of dicts) and links (array of edge dicts). Community IDs are stamped onto each node at write time. A safety check (export.py:278–331) refuses to overwrite an existing graph with a shrinking node count unless force=True. A backup is snapshotted before overwrite when the graph has semantic or curated content (backup_if_protected, export.py:42–104).

Optional graph database export. The exporters/graphdb.py module provides push_to_neo4j() (graphdb.py:22–98) which uses MERGE statements to upsert nodes and edges into a running Neo4j instance. Node labels are derived from file_type (capitalised). A similar push_to_falkordb exists. These are pure exports — there is no database-backed storage layer.

No vector store or embeddings. The README (line 38) states explicitly: "No embeddings, no vector store: a real graph you traverse." Node attributes include norm_label (diacritic-stripped lowercased label) for textual matching, but no embedding vectors are stored.

Global graph. A ~/.graphify/global-graph.json accumulates graphs from multiple repos via global_add() (global_graph.py:79). It is a flat union with repo-tagged nodes, not a federated index.

Community detection — Leiden with Louvain fallback. The cluster() function in cluster.py (lines 232–362) runs Leiden community detection via the native Rust graspologic_native.leiden() when available, falling back to the Python networkx.algorithms.community.louvain_communities implementation. Directed graphs are converted to undirected first. The resolution parameter (default 1.0) controls community granularity: >1.0 yields more, smaller communities.

Hub exclusion. An exclude_hubs_percentile option (cluster.py:248) removes nodes whose degree exceeds a given percentile of the degree distribution before partitioning, then reattaches them by majority-vote neighbour community. This prevents super-hubs (utility modules, entry points) from pulling unrelated subsystems into the same community.

Oversized community splitting. Communities larger than 25% of graph nodes (min 10) are split by running a second Leiden pass on the subgraph (cluster.py:336–342). A subsequent cohesion check (cluster.py:344–353) re-splits communities whose cohesion_score (internal edge density relative to outgoing edges) falls below 0.3, catching remaining doc-hub bridges.

Community IDs are stable. Communities are indexed by descending size, with a tiebreak on sorted node tuples, so identical groupings get identical IDs run-to-run (cluster.py:355–362).

No hierarchical summaries. Communities are flat — there are no hierarchical summaries, no community-summary generation via LLM, and no community-level text aggregation. The community ID is simply stamped onto each node when writing graph.json (export.py:343–347). The get_community MCP tool (serve.py:2008–2016) returns all nodes belonging to a community ID, with no accompanying summary text.

When computed. Communities are computed once per build via cluster() after the graph is assembled. They are not incrementally updated — a full re-cluster runs on each build. The MCP server reconstructs the community dict from stored node attributes on load (_communities_from_graph, serve.py:94–101).

MCP server tools. The query surface is a set of MCP tools defined in serve.py (lines 1959–2114): query_graph (the primary search+traverse tool), get_node, get_neighbors, get_community, shortest_path, god_nodes, and graph_stats. There is no distinct local/global/hybrid mode — the single query_graph tool supports BFS and DFS traversal modes.

Scoring and seed selection. A query in _query_graph_text() (serve.py:1361–1420) first splits the question into search terms via _query_terms() (serve.py:293), which handles Chinese segmentation via jieba when available. These terms are scored against every node in _score_query() (serve.py:554–680), which computes a combined TF-IDF–style score using per-token frequency (via _compute_idf) with a full-query exact-match bonus. Seeds are selected by _pick_seeds() via a gap-based heuristic over the ranked list, guaranteeing at least one seed per distinct query term. Relational-intent verbs ("calls", "uses", etc.) are demoted from the per-term guarantee to prevent an incidental match on a verb string from seating a decoy root (serve.py:1386–1391).

Trigram prefilter for speed. Before scoring each node, a trigram index (_get_trigram_index, serve.py:451–472) prunes the candidate set to nodes whose normalised text trigrams overlap with query trigrams, reducing passes over the full graph. On sparse queries the fallback is a full scan.

Traversal. Starting from selected seeds, the graph is traversed via BFS (breadth search — the default) or DFS (depth-first trace). Depth is capped (default 3, max 6). Edge relations can be filtered via explicit context_filter or auto-inferred from query intent words (_infer_context_filters, serve.py:940). The traversal runs on a filtered subgraph view.

Context assembly. The visited subgraph is rendered to plain text by _subgraph_to_text() — nodes as labelled entries with their source file, edges as directional relation statements — and capped at a token_budget (default 2000 tokens) so the output fits in an LLM context window. Seeds are rendered first so they survive truncation.

No vector search. There is no vector/hybrid search layer. All retrieval is text-based (trigram prefilter → token/scored exact & prefix matching → graph traversal). The output is returned as text directly to the calling agent (the MCP client), not fed into any secondary LLM query pipeline within graphify.

Per-file AST cache. The AST extraction cache (cache.py) is content-addressed: each file's extraction result is keyed by a hash of its bytes and namespaced by package version (cache/ast/v{version}-s{schema}/). On re-extraction, load_cached() returns the cached result for unchanged files; only new or modified files are re-parsed. Stale entries from older versions are cleaned up eagerly (_cleanup_stale_ast_entries, cache.py:45–69).

Per-file semantic cache. LLM extraction results are cached individually under cache/semantic/ (or cache/semantic-{mode}/ for deep mode), keyed by file-content hash. The cache also fingerprints the extraction prompt text (_PROMPT_FP_LEN = 12, cache.py:82), so entries invalidate when the prompt changes without re-billing unchanged files on every patch release. The MCP server checkpoints each chunk's results to the semantic cache as it completes, so an interrupted run loses only in-flight chunks (_checkpoint_chunk, llm.py:2677–2700). Corrupt entries are detected and reported without silently serving stale data.

File-system watch mode. graphify watch (watch.py) monitors the filesystem for changes and writes changed paths to a .pending_changes file. The pending drain mechanism (_drain_pending, watch.py:49–78) reads and deduplicates these paths. A co-operative lock (_rebuild_lock) prevents concurrent rebuilds. Post-commit hooks can append to the pending file without owning the lock.

Incremental merge. The build_merge() function (build.py:1953) takes an existing graph plus new extraction results and produces a merged graph without a full rebuild. Node deduplication and edge rewiring handle ID collisions deterministically. The resolution_context_nodes and resolution_context_edges parameters (extract.py:7417–7457) let incremental re-extraction pass unchanged file metadata as read-only context, so cross-file resolvers can still bind calls to unchanged callees without re-parsing them.

Deletion. The --update CLI flag deletes graph nodes for files that no longer exist on disk before merging new extractions. There is no explicit per-entity deletion API.

Global graph. global_graph.py tracks per-repo commits and allows adding/updating a repo's graph in a cross-repo global graph via global_add().

Token-budget chunk packing. The semantic extraction pipeline (extract_corpus_parallel, llm.py:2567) packs files into chunks using a configurable token_budget (default 60,000 tokens). Each file's token cost is estimated (characters/4 for code, 1,600 fixed per image), and chunks are closed when adding the next file would exceed the budget (llm.py:2160–2203). This prevents single over-large LLM requests that waste money on truncated retries.

Adaptive retry with bisection. When an LLM response hits finish_reason="length" (truncation), _extract_with_adaptive_retry() (llm.py:2319) bisects the chunk and re-extracts each half recursively, up to a configurable max_retry_depth (default 3, controlled by GRAPHIFY_MAX_RETRY_DEPTH). This bounds worst-case per-chunk cost at 2^depth calls. Setting depth to 0 disables all retries — one call per chunk, full stop.

Hollow-response backoff, not bisection. A hollow response (HTTP 200 with empty/unparseable content) is retried with backoff (2s, 8s) rather than bisected, because bisecting a backend issue cannot converge and would cost far more (llm.py:2406–2421). After 3 attempts the chunk fails loudly.

Model choice per stage. The BACKENDS dict (llm.py:104–224) defines model defaults per provider (claude, openai, gemini, kimi, ollama, deepseek, azure, bedrock, claude-cli) with configurable pricing per megatoken. Users can shift cost by selecting cheaper models (e.g., Ollama/local models cost $0). Model overrides are settable per provider via env vars (ANTHROPIC_MODEL, GRAPHIFY_OPENAI_MODEL, etc.).

Caching avoids re-extraction. Both the AST cache (versioned content-addressed) and semantic cache (content-hash + prompt-fingerprinted) skip re-extraction of unchanged files, eliminating LLM calls entirely for files that haven't changed since the last run.

Concurrency control. The thread pool defaults to 4 concurrent chunks (max_concurrency, llm.py:2594). Ollama defaults to serial (1) because concurrent requests cause VRAM pressure and hollow responses (llm.py:2671–2672). The GRAPHIFY_MAX_RETRIES env var controls SDK-level retries on rate limits, defaulting to 6 (llm.py:430).

Timeout guard. GRAPHIFY_API_TIMEOUT (default 600s) caps total request wall-clock time per chunk (llm.py:410–420). A timed-out chunk triggers adaptive bisection rather than silent failure.