Program for md file analysis

I want to write a program which takes a folder of .md files and creates a database from it which I can use local ollama inference to analyze and create a way to go through the contents of the folder of .md files

Great — here’s a clean and modular outline for building a program that: 1. **Ingests a folder of `.md` files** 2. **Stores them in a local vector database (like ChromaDB)** 3. **Uses a local LLM via Ollama for analysis** 4. **Creates an interface (CLI or web) for querying/navigating the content** --- ## 🧠 Overview **Tech Stack:** - Python - [ChromaDB](https://www.trychroma.com/) for vector storage - [Ollama](https://ollama.com/) for local LLM inference (e.g., `llama3`, `mistral`) - Markdown parsing: `frontmatter` or `markdown-it` - (Optional) FastAPI or Gradio for an interface --- ## ✅ Step-by-Step Plan ### 1. **Set Up Project Structure** ``` md-analyzer/ │ ├── main.py ├── md_ingest.py ├── ollama_analysis.py ├── chroma_manager.py ├── prompts/ │ └── summarize.md ├── data/ │ └── your_markdown_files.md └── chroma_db/ ``` --- ### 2. **Markdown Ingestion (md_ingest.py)** ```python import os import frontmatter def load_markdown_files(folder_path): documents = [] for filename in os.listdir(folder_path): if filename.endswith(".md"): full_path = os.path.join(folder_path, filename) post = frontmatter.load(full_path) documents.append({ "path": full_path, "title": post.get("title", filename), "content": post.content }) return documents ``` --- ### 3. **Chroma Vector Store Setup (chroma_manager.py)** ```python import chromadb from chromadb.config import Settings def get_chroma_client(db_path="./chroma_db"): return chromadb.PersistentClient(path=db_path) def create_collection(client, name="markdown_docs"): try: return client.get_collection(name) except: return client.create_collection(name) def add_documents_to_chroma(collection, docs): for i, doc in enumerate(docs): collection.add( documents=[doc["content"]], metadatas=[{"title": doc["title"], "path": doc["path"]}], ids=[f"doc_{i}"] ) ``` --- ### 4. **Ollama Analysis (ollama_analysis.py)** ```python import requests def query_ollama(prompt, model="llama3", base_url="http://localhost:11434"): response = requests.post( f"{base_url}/api/generate", json={ "model": model, "prompt": prompt, "stream": False } ) return response.json()["response"] ``` --- ### 5. **Putting It All Together (main.py)** ```python from md_ingest import load_markdown_files from chroma_manager import get_chroma_client, create_collection, add_documents_to_chroma from ollama_analysis import query_ollama def main(): folder = "data/" docs = load_markdown_files(folder) client = get_chroma_client() collection = create_collection(client) add_documents_to_chroma(collection, docs) for doc in docs: prompt = f"Summarize the following content:\n\n{doc['content'][:2000]}" summary = query_ollama(prompt) print(f"📄 {doc['title']}\n---\n{summary}\n\n") if __name__ == "__main__": main() ``` --- ### 🧪 Optional: Add Search If you want to **query by topic**, e.g. "Give me all documents about 'resilience'" using embeddings: ```python def search(collection, query_text, n_results=3): return collection.query( query_texts=[query_text], n_results=n_results ) ``` --- ### 🌐 Optional: Web Interface with Gradio or FastAPI If you want to search and summarize interactively. Let me know if you want the **CLI**, **TUI**, or **Gradio** version next — or a local-first way to annotate results. --- Would you like me to zip this into a working repo template for you, or expand it with custom prompt chaining?

Got it! You want to build a program that: - Reads a folder of Markdown (`.md`) files, - Creates a local searchable database from their contents, - Uses local Ollama inference to analyze or query the data, - And provides a way to interactively go through that content. Here's a high-level approach and example code to get you started: --- ## Plan 1. **Parse Markdown files:** Read all `.md` files in a folder, extract content. 2. **Preprocess content:** Optionally split content into chunks (e.g., paragraphs or sections). 3. **Embed text:** Use an embedding model (e.g., Ollama embeddings) to convert chunks into vectors. 4. **Store vectors:** Save embeddings + metadata in a vector database (e.g., ChromaDB, FAISS, or SQLite with embeddings). 5. **Query interface:** Build a function to query your vector DB and run inference on top of the retrieved chunks via Ollama. 6. **Interactive exploration:** Optionally, build a CLI or web UI to browse and query the data. --- ## Minimal Python Example (using ChromaDB + Ollama embeddings) ```python import os from pathlib import Path import chromadb from chromadb.config import Settings from langchain_community.embeddings.ollama import OllamaEmbeddings # 1. Initialize Chroma client and collection client = chromadb.Client(Settings(chroma_db_impl="duckdb+parquet", persist_directory="./chroma_db")) collection = client.get_or_create_collection(name="markdown_docs") # 2. Initialize Ollama embedding model (replace with your model name) embedding_model = OllamaEmbeddings(model="nomic-embed-text") def read_md_files(folder_path): md_files = list(Path(folder_path).glob("*.md")) documents = [] for md_file in md_files: text = md_file.read_text(encoding="utf-8") documents.append({ "id": md_file.stem, "text": text, "metadata": {"filename": md_file.name} }) return documents def create_embeddings_and_store(documents): texts = [doc["text"] for doc in documents] ids = [doc["id"] for doc in documents] metadatas = [doc["metadata"] for doc in documents] # Compute embeddings embeddings = embedding_model.embed_documents(texts) # Add to vector DB collection.add( ids=ids, documents=texts, embeddings=embeddings, metadatas=metadatas, ) def main(): folder_path = "./md_folder" docs = read_md_files(folder_path) create_embeddings_and_store(docs) client.persist() if __name__ == "__main__": main() ``` --- ## Next steps: Querying and Analyzing - To search, use similarity search from ChromaDB, e.g.: ```python query = "Your question here" query_embedding = embedding_model.embed_query(query) results = collection.query( query_embeddings=[query_embedding], n_results=3 ) print(results) ``` - Then feed the retrieved documents to your local Ollama model for inference or summarization. --- If you want, I can help you build the query interface or extend it to a CLI or web app. Let me know!