Programming content analysis

I want to just make single file scripts one at a time and run them get the results and the create the next script, the first script will take the "content" folder and ingest the contents first by running an llm call which analyzes the text and takes all of the information and extracts the educational content regarding programming and extracts the key information and things learned from the process. What I want is a running ledger of the things that were learned from the content and then periodically that ledger can be summarized into smaller and smaller ledgers until all the files have been gone through and analyzed in this way- the end result would be a very detailed output consisting of a high level outline of every single thing learned as well as several files which are composed of smaller blog post sized posts containing the essence of each thing learned. ----so what it will do is transform the texts from the folder into something that makes cohesive sense and eliminates duplicate or extraneous information that is not relevant to learning programming

Got it! You want to build this process step-by-step as a series of simple, single-file scripts. The first script will: - Read all files in a folder called `"content"` (presumably markdown or text files). - For each file, run an LLM call that: - Analyzes the text. - Extracts **educational programming content** and key learnings. - Keep a **running ledger** (accumulated list) of everything learned so far. - Periodically summarize the ledger to condense and remove duplicates or irrelevant info. - Eventually output: - A high-level outline summarizing all learnings. - Multiple smaller blog-post-sized files, each capturing a core learning or concept. --- ### Let's start with the **first single-file script** that: 1. Reads all `.md` (or `.txt`) files from `"content"` folder. 2. For each file: - Sends its content to an LLM prompt asking for educational programming insights and key points learned. 3. Appends the results (the "things learned") to a local ledger file (`ledger.json` or `.txt`). 4. Prints or saves this ledger. --- ### Here’s a minimal Python script outline that does this — you can run it and iterate from here. ```python import os import json # You will replace this stub with actual LLM call to Ollama or your preferred model def llm_extract_educational_content(text): # This is a mock response for demonstration. # Replace this with your real LLM call. return { "key_points": [ "Learned about variables and data types.", "Understanding of functions and scope.", "Basics of loops and conditionals." ], "summary": "This text explains fundamental programming concepts including variables, functions, loops, and conditionals." } def read_files_from_folder(folder_path): files_content = {} for filename in os.listdir(folder_path): if filename.endswith(".md") or filename.endswith(".txt"): with open(os.path.join(folder_path, filename), "r", encoding="utf-8") as f: files_content[filename] = f.read() return files_content def main(): content_folder = "content" ledger_file = "ledger.json" ledger = [] # Load existing ledger if exists if os.path.exists(ledger_file): with open(ledger_file, "r", encoding="utf-8") as f: ledger = json.load(f) files_content = read_files_from_folder(content_folder) for filename, text in files_content.items(): print(f"Processing {filename}...") result = llm_extract_educational_content(text) ledger.append({ "filename": filename, "key_points": result["key_points"], "summary": result["summary"] }) # Save updated ledger with open(ledger_file, "w", encoding="utf-8") as f: json.dump(ledger, f, indent=2, ensure_ascii=False) print(f"Updated ledger saved to {ledger_file}") if __name__ == "__main__": main() ``` --- ### What to do next after you run this: - **Replace** the `llm_extract_educational_content()` stub with an actual API call to your LLM (like Ollama with GPT-OSS 20B). - The function should feed the text into an LLM prompt designed to: - Extract educational programming content and key learnings. - Run this script repeatedly as you add files. - After this, the next script can read the `ledger.json` and create condensed summaries or blog-post-sized content blocks. --- ### Would you like me to help write the actual Ollama LLM API call in Python next? Or should I help craft the exact LLM prompt text you should use for extracting the educational content? Let me know how you'd like to proceed!

replace teh stub

Absolutely! I'll replace the stub function `llm_extract_educational_content` with a real example of how to call Ollama’s local API (assuming you are running Ollama locally and using the `gpt-oss:20b` model). --- ### Here's an example using Python’s `requests` library to call the Ollama local API: ```python import requests import os import json OLLAMA_MODEL = "gpt-oss:20b" OLLAMA_API_URL = f"http://localhost:11434/api/chat" # Ollama default local API endpoint def llm_extract_educational_content(text): prompt = f""" You are an assistant that reads programming educational content. Analyze the following text and extract: 1. The key programming concepts or lessons taught. 2. Important takeaways or things someone would learn. 3. A concise summary focusing only on educational programming content. Text: \"\"\" {text} \"\"\" Respond in JSON format with two keys: "key_points" (a list of strings) and "summary" (a concise string). """ # Build the request payload for Ollama chat completion payload = { "model": OLLAMA_MODEL, "messages": [ {"role": "user", "content": prompt} ] } response = requests.post(OLLAMA_API_URL, json=payload) response.raise_for_status() data = response.json() # Ollama usually returns an array of choices, get the first one answer_text = data["choices"][0]["message"]["content"] # Parse the JSON response from the LLM output safely try: # The model is expected to respond with a JSON string exactly as asked extracted = json.loads(answer_text) except json.JSONDecodeError: # If parsing fails, return fallback info extracted = { "key_points": [answer_text], "summary": answer_text } return extracted def read_files_from_folder(folder_path): files_content = {} for filename in os.listdir(folder_path): if filename.endswith(".md") or filename.endswith(".txt"): with open(os.path.join(folder_path, filename), "r", encoding="utf-8") as f: files_content[filename] = f.read() return files_content def main(): content_folder = "content" ledger_file = "ledger.json" ledger = [] # Load existing ledger if exists if os.path.exists(ledger_file): with open(ledger_file, "r", encoding="utf-8") as f: ledger = json.load(f) files_content = read_files_from_folder(content_folder) for filename, text in files_content.items(): print(f"Processing {filename}...") result = llm_extract_educational_content(text) ledger.append({ "filename": filename, "key_points": result.get("key_points", []), "summary": result.get("summary", "") }) # Save updated ledger with open(ledger_file, "w", encoding="utf-8") as f: json.dump(ledger, f, indent=2, ensure_ascii=False) print(f"Updated ledger saved to {ledger_file}") if __name__ == "__main__": main() ``` --- ### What this does: - Sends a prompt to your local Ollama instance running `gpt-oss:20b` that asks it to extract key programming concepts and a summary from each file. - Expects the model to respond with JSON containing `key_points` and `summary`. - Parses the response JSON and adds it to the ledger. - Saves the ledger as `ledger.json`. --- ### Important: - Make sure Ollama is running locally and its API is reachable at `http://localhost:11434/api/chat`. - If your Ollama API is on a different port or address, update `OLLAMA_API_URL` accordingly. - You need to have `requests` installed (`pip install requests`). - You might want to add error handling for network issues or rate limits. --- If you want, I can help you write the exact prompt to put inside the code to maximize the quality of educational content extraction or help with the next step of summarizing the ledger!