App for content generation

instead I want to create an application which takes .md folders, generates a chromadb from them and is able to generate a .json person from it which is then outputted as a detailed system prompt which can be used for the generation of new content through using the recalled embeddings along with the .json persona to generate the new content returned

You're describing a pipeline-based app that does the following: --- ## ✅ App Summary > An application that ingests a folder of `.md` files, creates embeddings stored in a **local ChromaDB**, analyzes those markdown files to generate a **detailed persona in `.json` format**, and enables **content generation** using both: - Retrieved contextual embeddings - The structured persona profile You could host this **locally**, or eventually build a **static frontend + backend** using frameworks like: - **Backend**: Python (FastAPI, LangChain) - **Frontend**: React (or Svelte), optionally deployed via Cloudflare Pages - **Embedding & Vector Store**: `Ollama` for embedding + `ChromaDB` --- ## 🔧 App Architecture ### 1. **Ingest .md Files into ChromaDB** - Use a `loader.py` to read `.md` files into text chunks - Embed chunks using a local model (`Ollama`, `Instructor`, etc.) - Store embeddings in **ChromaDB**, persisted locally (`persist_directory="./chromadb"`) ```python from langchain.document_loaders import DirectoryLoader from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.vectorstores import Chroma from langchain.embeddings import OllamaEmbeddings loader = DirectoryLoader('./markdowns', glob='**/*.md') docs = loader.load() splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50) chunks = splitter.split_documents(docs) embedding = OllamaEmbeddings(model="nomic-embed-text") db = Chroma.from_documents(chunks, embedding=embedding, persist_directory="./chromadb") db.persist() ``` --- ### 2. **Generate `.json` Persona File From Content** You could: - Summarize themes, tone, word choice, POV, and recurrent ideas - Create a persona schema with fields like: ```json { "name": "TechEssayist01", "style": "Analytical, precise, explanatory", "domain": "AI, programming, research", "tone": "Neutral, thoughtful, methodical", "frequent_words": ["embedding", "pipeline", "token", "latency"], "persona_traits": { "assertiveness": 4, "emotionality": 2, "curiosity": 8, "formality": 7 } } ``` Write a Python function that analyzes your docs (frequency, style, etc.) to output this `.json`. --- ### 3. **System Prompt Construction** - Combine the `.json` persona and retrieved content to generate a system prompt. - Example: ```txt You are TechEssayist01, a writer focused on AI and code. Your style is analytical, precise, and explanatory. You often write about embedding pipelines and model latency. Using the following context extracted from previous writings, generate a new piece of writing in your characteristic style. Context: 1. Chunk A... 2. Chunk B... ``` --- ### 4. **Content Generation** You now pass this system prompt + user task to a local model: ```python from langchain.llms import Ollama llm = Ollama(model="llama3") response = llm.invoke(prompt) print(response) ``` --- ## 🧠 Optional Enhancements - Use `FastAPI` to expose endpoints: - `/ingest` to upload `.md` folders - `/persona` to generate `.json` file - `/generate` to pass a query + persona + recalled chunks - Use `LangGraph` or `SmolAgents` to orchestrate steps - Add agent memory via ChromaDB retrieval - Visualize persona and embeddings in the UI --- ## 📁 Project Structure ``` my_persona_gen_app/ │ ├── markdowns/ # .md files ├── chromadb/ # persisted vector db ├── personas/ │ └── techessayist01.json │ ├── scripts/ │ ├── ingest.py │ ├── generate_persona.py │ └── generate_content.py │ ├── app.py # FastAPI backend ├── ui/ # Optional frontend └── requirements.txt ``` --- ## ✅ CLI Prompt Example ``` python generate_persona.py --folder=./markdowns --output=./personas/techessayist01.json python generate_content.py --persona=techessayist01.json --query="Write a blog post about agentic workflows" ``` --- Would you like the code for each script in this pipeline (`ingest.py`, `generate_persona.py`, `generate_content.py`) next?

yes

Great — below are the full implementations of each script in your pipeline: --- ## ✅ 1. `ingest.py` > Ingests `.md` files, splits them, embeds them using `Ollama`, and stores in **ChromaDB** ```python # scripts/ingest.py import os from langchain.document_loaders import DirectoryLoader from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.vectorstores import Chroma from langchain.embeddings import OllamaEmbeddings def ingest_markdown(markdown_dir="./markdowns", persist_dir="./chromadb"): loader = DirectoryLoader(markdown_dir, glob="**/*.md") docs = loader.load() splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50) chunks = splitter.split_documents(docs) embedding = OllamaEmbeddings(model="nomic-embed-text") db = Chroma.from_documents(chunks, embedding=embedding, persist_directory=persist_dir) db.persist() print(f"Ingested {len(chunks)} chunks into ChromaDB at {persist_dir}") if __name__ == "__main__": ingest_markdown() ``` --- ## ✅ 2. `generate_persona.py` > Analyzes markdown content to output a `.json` persona ```python # scripts/generate_persona.py import os import json import sys import glob from collections import Counter from langchain.document_loaders import DirectoryLoader def extract_persona(markdown_dir): loader = DirectoryLoader(markdown_dir, glob="**/*.md") docs = loader.load() full_text = " ".join([doc.page_content for doc in docs]) words = full_text.lower().split() word_freq = Counter(words) common_words = [w for w in word_freq.most_common(50) if len(w[0]) > 4] persona = { "name": os.path.basename(markdown_dir), "style": "Auto-generated from writing samples.", "domain": "Inferred from content", "tone": "Inferred", "frequent_words": [w[0] for w in common_words[:15]], "persona_traits": { "assertiveness": 5, "emotionality": 3, "curiosity": 7, "formality": 6 } } return persona def save_persona(markdown_dir, output_path): persona = extract_persona(markdown_dir) with open(output_path, "w") as f: json.dump(persona, f, indent=4) print(f"Saved persona to {output_path}") if __name__ == "__main__": folder = sys.argv[1] if len(sys.argv) > 1 else "./markdowns" name = os.path.basename(folder.rstrip("/")) output_path = f"./personas/{name}.json" os.makedirs("personas", exist_ok=True) save_persona(folder, output_path) ``` --- ## ✅ 3. `generate_content.py` > Uses ChromaDB + `.json` persona to generate new writing ```python # scripts/generate_content.py import json import sys from langchain.vectorstores import Chroma from langchain.embeddings import OllamaEmbeddings from langchain.llms import Ollama from langchain.chains import RetrievalQA def load_persona(persona_path): with open(persona_path, "r") as f: return json.load(f) def build_system_prompt(persona, context): prompt = f"""You are {persona['name']}, a writer who is: Style: {persona['style']} Domain: {persona['domain']} Tone: {persona['tone']} Frequent Words: {', '.join(persona['frequent_words'])} Your task is to write in your characteristic style. Context: {context} Respond in a way that reflects your persona. """ return prompt def generate_response(query, persona_path, chroma_path="./chromadb"): persona = load_persona(persona_path) embedding = OllamaEmbeddings(model="nomic-embed-text") db = Chroma(persist_directory=chroma_path, embedding_function=embedding) retriever = db.as_retriever(search_kwargs={"k": 3}) relevant_docs = retriever.get_relevant_documents(query) context = "\n".join([doc.page_content for doc in relevant_docs]) prompt = build_system_prompt(persona, context + "\n\nTask: " + query) llm = Ollama(model="llama3") response = llm.invoke(prompt) return response if __name__ == "__main__": persona_path = sys.argv[1] if len(sys.argv) > 1 else "./personas/default.json" query = sys.argv[2] if len(sys.argv) > 2 else "Write a tutorial on how to use agentic workflows with LLMs" result = generate_response(query, persona_path) print("\nGenerated Content:\n") print(result) ``` --- ## 📦 `requirements.txt` ```txt langchain chromadb ollama tiktoken ``` --- ## 🧪 Example Workflow ### 1. Ingest `.md` files: ```bash python scripts/ingest.py ``` ### 2. Generate persona: ```bash python scripts/generate_persona.py ./markdowns ``` ### 3. Generate new content: ```bash python scripts/generate_content.py personas/yourfoldername.json "How to set up a local LLM search engine" ``` --- Would you like a `FastAPI` backend version of this next? Or a web UI?