semantica-agi/semantica

Graph construction flows through the GraphBuilder.build() method in semantica/kg/graph_builder.py. It processes sources that can be raw text, pre-extracted Entity/Relation objects, or dicts with entities/relationships keys. For raw text, the builder calls _extract_from_text which runs three extractors in sequence: NERExtractor (default method "ml", using spaCy; alternatives: "pattern", "regex", "huggingface", "llm"), RelationExtractor (default "pattern", alternatives up to "llm"), and TripletExtractor (default "pattern"). The NERExtractor (semantica/semantic_extract/ner_extractor.py) supports fallback chains — if the primary method returns nothing, it falls through pattern matching to a capitalized-word heuristic as a last resort. The GraphChunker module (semantica/split/kg_chunkers.py) provides EntityAwareChunker, RelationAwareChunker, GraphBasedChunker, OntologyAwareChunker, and HierarchicalChunker — which can preserve entity boundaries or triplet integrity during chunking. LLM enhancement is additive: LLMExtraction.enhance_entities() and enhance_relations() call an LLM provider with a structured prompt, merge results back by matching entity text or exact triples, and always preserve originals on failure (semantica/semantic_extract/llm_extraction.py:192-285). Entity resolution is done by EntityResolver (if merge_entities=True) using fuzzy/exact/ML strategies, and EntityMerger in semantica/deduplication/entity_merger.py detects duplicate groups via DuplicateDetector, then applies configurable strategies (keep_first, keep_most_complete, merge_all, etc.) with provenance tracking. After resolution, _remap_relationship_endpoints() rewrites relation source/target IDs to the surviving canonical entity ID (graph_builder.py:370-454). Conflict detection via ConflictDetector (optional, resolve_conflicts=True) runs after the graph is built and can auto-resolve conflicts. The output is a dict with entities, relationships, and metadata. Temporal edges are supported via add_temporal_edge(). No fixed schema/ontology is enforced at build time; entity types and relation predicates are free-form strings from the extractors, although the ontology/ module provides schema mapping post-hoc.

The graph is stored via a pluggable backend architecture. The core GraphStore class in semantica/graph_store/graph_store.py defines the interface (NodeManager, RelationshipManager, QueryEngine), and concrete implementations exist for Neo4j (Neo4jStore in neo4j_store.py, using Cypher queries via the neo4j driver), Apache AGE (ApacheAgeStore in age_store.py, wrapping openCypher through PostgreSQL psycopg2), FalkorDB (FalkorDBStore in falkordb_store.py), and Amazon Neptune. The builder's build() method optionally persists the graph to a store via self.graph_store.add_nodes(resolved_entities) and self.graph_store.add_edges(formatted_edges) (graph_builder.py:1012-1044). When no database backend is configured, the graph is returned as an in-memory Python dict {"entities": [...], "relationships": [...]} and held by the caller. Nodes are stored as property-graph entities with an id, optional labels (e.g. ["Person"]), and arbitrary properties. Edges carry source_id, target_id, type, and properties. The Neo4j driver (neo4j_store.py:96-100) normalizes Node/Relationship objects to plain dicts with underscored identity keys (_labels, _element_id, _type) so user properties are never shadowed. Embeddings sit alongside the graph in a separate vector store (semantica/vector_store/), supporting FAISS, Qdrant, Weaviate, Pinecone, Milvus, PgVector, and SQLiteVec. The ContextRetriever orchestrates hybrid retrieval: it queries the vector store via cosine similarity and the graph via semantic entity matching / BFS traversal, then merges results with configurable hybrid_alpha weighting — 0 = vector only, 1 = graph only, 0.5 = balanced (context_retriever.py:178). Embeddings for entities are generated via DecisionEmbeddingPipeline which uses the vector store's own embed method.

Semantica has a full community detection and hierarchical summarization pipeline. The CommunityDetector class (semantica/kg/community_detector.py) supports Louvain (via NetworkX greedy_modularity_communities with resolution parameter, and a basic fallback), Leiden (via optional igraph+leidenalg or cdlib, falling back to a native Python implementation with local-moving and refinement phases at lines 894-1034), overlapping k-clique communities, and label propagation (supports chunked processing for large graphs). The CommunityHierarchyBuilder (semantica/kg/community_hierarchy.py:553-1419) builds multi-level partitions by iteratively coarsening the graph via Louvain or Leiden at increasing resolution, then constructs a CommunityHierarchy — a tree of HierarchicalCommunity nodes keyed by SHA-256 content hashes with metrics (internal/external edges, density, conductance) and deterministic parent-child resolution. The CommunitySummarizer (semantica/kg/community_summarizer.py:547-2297) generates structured CommunityReport objects bottom-up via summarize_hierarchy(): for each community it extracts a subgraph, computes centrality scores (degree/betweenness/closeness/eigenvector/PageRank), packs context within a token budget (prioritizing child reports, edges, anchor entities, and source text), and calls an LLM with a Pydantic-typed schema (CommunityReportLLMSchema) to produce a title, summary, findings list, and impact rating. It has a 5-tier LLM unwrap strategy (_call_llm) and falls back to an extractive baseline when no LLM is configured. Reports are cached via SHA-256 content hashing with atomic disk persistence. The hierarchy is computed on demand when the user invokes community detection; it is not automatically triggered during ingestion or graph building.

Query-time retrieval supports three modes — local, global, and hybrid — selected via the mode parameter of ContextRetriever.retrieve() (context_retriever.py:210-295). Local mode runs three retrievers in parallel: (1) vector search via _retrieve_from_vector() which queries the vector store (FAISS/Qdrant/etc.) for semantically similar content; (2) graph retrieval via _retrieve_from_graph() which first extracts query intent (relationship types, entity types, keywords), then computes cosine similarity between the query embedding and every entity/relationship embedding — boosting scores when entity types or relation types match the intent — and finally performs BFS-based graph expansion (max_expansion_hops, default 2) to pull in related entities. All results are merged via _rank_and_merge() which normalizes scores per source, applies contextual boosting (graph results with more neighbors get up to 20% boost; results found in both vector and graph get 20% cross-source boost), and re-ranks using 70% original + 30% query-content similarity. Global mode delegates to GlobalGraphRetriever.search() (global_retriever.py:329+), which implements a Map-Reduce pattern over community reports: it dynamically selects a hierarchy level (promoting to coarser levels if token budget is exceeded), runs Map LLM calls concurrently via ThreadPoolExecutor (each extracting key points from a community report into a structured MapPointSchema with relevance scores), then synthesizes the Reduce response from top-scored points within a token budget, with automatic evidence/citation validation. Hybrid mode runs both local and global retrievers, concatenates results, and re-ranks. DRIFT mode (drift_search.py) combines global thematic framing with directed facet reasoning and local entity-hop graph traversal. The assembled context is returned as RetrievedContext objects with content, score, source, related_entities, and related_relationships — ready for LLM consumption.

Incremental updates are handled at multiple levels. The EntityMerger.incremental_merge() method (semantica/deduplication/entity_merger.py:388-473) merges new entities into an existing set by running DuplicateDetector.incremental_detect() between new and existing entities, then merging each candidate pair with configurable strategies — avoiding duplicate merges via processed-ID tracking. The GraphBuilder.build() method is designed to be re-runnable: it merges sources and can be called repeatedly on new documents, with merge_entities=True enabling deduplication against previously built entity sets. The builder's entity resolver (EntityResolver) can operate on the full accumulated set each time, and _remap_relationship_endpoints() rewrites relation endpoints after entity merges change canonical IDs (graph_builder.py:370-454). Ingestion is source-agnostic: FileIngestor, StreamIngestor (Kafka, RabbitMQ, Pulsar, Kinesis), WebIngestor, DBIngestor, and dozens of other ingestors in semantica/ingest/ each produce documents that feed into extraction. The Pipeline module (pipeline_builder.py, execution_engine.py) provides full orchestration with retry, parallelism, and resource scheduling. Version management via TemporalVersionManager (change_management/managers.py) supports snapshot creation, diffs between versions, and persistent storage (SQLite or in-memory). Deletion is not explicitly handled as a first-class operation — there is no built-in mechanism to remove a document's extracted entities/relations after they are added. The system is additive: incremental merges consolidate but do not remove stale data. Extraction caching (semantica/semantic_extract/cache.py) provides TTL-based caching for entity, relation, and triplet extraction results, with pluggable backends (in-memory LRU or persistent SQLite). The cache is keyed by text hash and method parameters, avoiding re-extraction from unchanged source texts across rebuilds.

Semantica controls LLM cost and latency through several mechanisms. Extraction-level caching (semantica/semantic_extract/cache.py) stores entity, relation, and triplet extraction results keyed by text hash and method, with configurable TTL (default 3600s) and dual backends: in-memory LRU or persistent SQLite. The cache sits behind all extractor calls so repeated texts never reach the LLM. Batching is configurable in the default optimization config: batch_size: 10, max_tokens_per_batch: 2000, enable_batching: True (semantica/semantic_extract/config.py:58-71). The max_workers setting (default 8, auto-clamped to CPU count and capped at 32) controls parallelism in pipelines. Model choice per stage is fully configurable: the GraphBuilder allows separate selection of ner_method, relation_method, and triplet_method — each can be "ml" (spaCy, free), "pattern" (regex, free), or "llm" (paid). The default stack uses free local methods ("ml" for NER, "pattern" for relations/triplets), requiring no API keys (graph_builder.py:519-521). LLM enhancement is opt-in. The LLM provider layer (semantica/llms/ and semantica/semantic_extract/providers.py) supports 10+ providers including local/cheap options: Ollama for local open-source models, Groq for fast cheap inference, and LiteLLM for routing to the cheapest available model. Each provider exposes generate_typed() for structured output via Pydantic schemas, reducing the need for multiple retries. Community summarization has a max_tokens budget (default 4000) that limits prompt size, and centrality-based token budgeting (_pack_context) prioritizes the most central entities and highest-impact child reports within that budget (community_summarizer.py:1169-1728). The _extractive_fallback method produces deterministic reports with no LLM call when no model is configured. Global query uses similar token budgeting (max_context_tokens: 4000, response_token_budget: 600) with dynamic level promotion — if level-0 community reports exceed budget, it promotes to coarser levels where fewer but denser reports exist (global_retriever.py:428-534). The Map phase runs concurrently across community reports with a configurable max_workers (default 4).