Topic Clustering with Graphs

I have a list of 15000 topics along with their description and usecases, way i want to cluster them into topic groups, domain and then industries Hierarchy is: Industry>Domain>Topic Group>Topic The topics are very technical in nature, I have already tried embeddings and then hierarchical clustering and BerTopic but the clustering isn't very accurate. Please suggest any approaches. ---if the following is an answer expand on it : Have you tried building a knowledge graph? Supplementing FAISS or ChromaDB with a graph database helps add context and accuracy.

Yes — the suggestion about **combining embeddings with a knowledge graph** is actually a very strong approach for your problem. The reason is that **pure vector clustering struggles with technical taxonomies**, especially when you want a **strict hierarchy like:** **Industry → Domain → Topic Group → Topic** Embeddings capture **semantic similarity**, but they don't understand **structural relationships or ontologies**. A **knowledge graph adds structure**, which dramatically improves grouping accuracy. Let me expand on how this approach works. --- # 1. Why embeddings alone struggle Methods like: - hierarchical clustering - BERTopic - k-means - HDBSCAN work well for **flat topic discovery**, but your task is actually **taxonomy construction**. Technical problems you hit: ### 1. Polysemy Example: ``` Transformer ``` Could belong to: ``` Electrical Engineering Machine Learning Power Systems ``` Embeddings alone may cluster incorrectly. --- ### 2. Multi-domain relationships Example topic: ``` Vector Databases ``` Possible placements: ``` AI Infrastructure Search Systems Data Engineering ``` Embeddings cannot easily choose the correct **industry hierarchy**. --- ### 3. Hierarchical structure is missing Embeddings tell you: ``` Topic A ≈ Topic B ``` But not: ``` Topic A belongs to Topic Group X Topic Group X belongs to Domain Y Domain Y belongs to Industry Z ``` This is where a **knowledge graph helps**. --- # 2. Architecture: Vector DB + Knowledge Graph The idea is to combine: ``` Vector similarity + Graph relationships + LLM reasoning ``` Example architecture: ``` Topics ↓ Embedding model ↓ Vector DB (FAISS / ChromaDB) ↓ Knowledge Graph (Neo4j / ArangoDB) ↓ Graph clustering + reasoning ``` --- # 3. Step 1 — Generate embeddings Use a strong embedding model. Good options: ``` bge-large e5-large Instructor-xl OpenAI text-embedding-3-large ``` Each topic embedding should include: ``` Topic Description Use cases ``` Example prompt structure: ``` Topic: Reinforcement Learning Description: Machine learning paradigm where agents learn via reward signals. Use Cases: robotics, game AI, optimization ``` Embedding this richer context improves clustering. --- # 4. Step 2 — Build a similarity graph Instead of clustering directly, build a **graph of relationships**. Example: ``` Topic A ---similar_to---> Topic B Topic B ---similar_to---> Topic C ``` You create edges based on similarity threshold. Example: ``` if cosine_similarity > 0.82 create edge ``` Graph representation: ``` Node: Topic Edge: semantic_similarity Weight: cosine score ``` Now you have a **topic network**. --- # 5. Step 3 — Graph community detection Instead of hierarchical clustering, run **graph clustering algorithms**. Best ones: ``` Louvain Leiden Infomap Label propagation ``` These work extremely well on **semantic networks**. Example result: ``` Community 1 Deep Learning CNN Transformers Vision Transformers Community 2 SQL Optimization Query Planning Indexing ``` These communities become **Topic Groups**. --- # 6. Step 4 — Build higher-level nodes Now you introduce **Domain and Industry nodes**. Example: ``` Industry └── Artificial Intelligence └── Machine Learning └── Deep Learning └── CNN └── Transformers ``` Edges: ``` Topic -> Topic Group Topic Group -> Domain Domain -> Industry ``` This creates a **true ontology**. --- # 7. Step 5 — Use LLMs to refine hierarchy LLMs are very good at **taxonomy reasoning**. Example prompt: ``` You are building a technical taxonomy. Topics: - CNN - Transformers - GANs - Diffusion Models Determine the best Topic Group. ``` Output: ``` Topic Group: Deep Learning Architectures Domain: Machine Learning Industry: Artificial Intelligence ``` You can do this **per cluster** instead of per topic. This dramatically reduces LLM cost. --- # 8. Step 6 — Use graph + vector search together When assigning a new topic: 1. search similar topics in **vector DB** 2. retrieve neighbors from **graph** 3. use LLM to choose hierarchy Example: ``` New topic: Retrieval Augmented Generation ``` Vector search returns: ``` RAG Vector Databases Semantic Search LLM Agents ``` Graph context: ``` Topic Group: LLM Infrastructure Domain: Natural Language Processing Industry: Artificial Intelligence ``` Assignment becomes accurate. --- # 9. Tools for this architecture Vector DB: ``` FAISS ChromaDB Weaviate Qdrant ``` Graph DB: ``` Neo4j (best) ArangoDB TigerGraph NetworkX (for prototype) ``` Graph algorithms: ``` networkx igraph graph-tool Neo4j GDS library ``` --- # 10. Example pipeline Pseudo workflow: ``` topics → embeddings embeddings → similarity search similarity edges → graph graph → community detection communities → topic groups LLM → label topic groups LLM → assign domain + industry store hierarchy in graph DB ``` --- # 11. Why this works better Instead of forcing clustering, you get: ``` semantic similarity + network structure + ontology reasoning ``` This combination is **far more stable**. --- # 12. Bonus: automatic taxonomy growth Once the graph exists you can also do: ### centrality detection Find core topics in each domain. ### missing taxonomy Detect clusters without parents. ### topic evolution Track how topics emerge over time. --- # 13. If I were solving your problem My stack would be: ``` Embedding: bge-large Vector DB: FAISS Graph: Neo4j Clustering: Leiden LLM labeling: small model (Mixtral / GPT / Claude) ``` This handles **15k topics easily**. --- # 14. One extremely powerful improvement (rarely used) Instead of embedding topics individually, embed: ``` Topic + nearest 10 topics ``` This creates **contextual embeddings**, which improves clustering dramatically. --- # 15. If you'd like, I can also show you something even more powerful: A **3-stage taxonomy generation pipeline used in research labs** that clusters **100k+ topics with extremely high accuracy**. It combines: ``` embedding similarity + graph clustering + LLM ontology generation + recursive taxonomy building ``` It's surprisingly elegant and would work extremely well for your **Industry → Domain → Topic Group → Topic** problem.

give a short answer

Yes — a **knowledge graph combined with a vector database** can significantly improve clustering accuracy. Embeddings (FAISS, ChromaDB, etc.) are good at finding **semantic similarity**, but they don’t capture **hierarchical relationships** like *Industry → Domain → Topic Group → Topic*. A **knowledge graph adds structure and context**. **Approach:** 1. **Generate embeddings** for each topic using the topic + description + use cases. 2. Store them in a **vector database** (FAISS or ChromaDB) to find similar topics. 3. Build a **knowledge graph** where: - Nodes = topics - Edges = similarity or conceptual relationships 4. Run **graph community detection algorithms** (Louvain/Leiden) to form **topic groups**. 5. Use an LLM or rules to label clusters as **domains and industries**. 6. Store the hierarchy in the graph: ``` Industry → Domain → Topic Group → Topic ``` The advantage is that the **vector DB handles semantic similarity**, while the **graph captures relationships and hierarchy**, which usually produces much cleaner taxonomy structures for large technical datasets like your 15k topics.