Remaking News13 Project

I want to remake this, please help me do so and improve it, https://github.com/kliewerdaniel/news13

Your old repo already had the right instinct: local AI + RSS + automated generation + a clean frontend. The problem is that the architecture is still “2024 AI app.” If you remake it today, you should turn it into something closer to a self-hosted autonomous media system. The biggest improvement is separating the system into independent services instead of one giant script. Your future version should look more like this: ```text RSS / APIs / Reddit / YouTube ↓ Ingestion Layer ↓ Queue / Event Bus ↓ AI Processing Pipeline ├── summarizer ├── classifier ├── embedding service ├── rewrite engine ├── image generator └── ranking engine ↓ PostgreSQL + Vector DB ↓ API Layer ↓ Next.js Frontend + Admin Panel ``` Your original project already touched some of these ideas. citeturn0search0 The modern version should become: # Core Stack ## Frontend Use: - urlNext.jshttps://nextjs.org - urlTailwind CSShttps://tailwindcss.com - urlshadcn/uihttps://ui.shadcn.com - urlTanStack Queryhttps://tanstack.com/query/latest Why: - streaming SSR - edge caching - partial hydration - modern app router - easy deployment - beautiful dashboards You already like Next.js from your earlier blog systems, so lean into it fully. --- # Backend Instead of one Python script, split it into services. ## Suggested Services ### 1. Feed Service Handles: - RSS - Reddit - YouTube transcripts - HN scraping - Twitter/X mirrors - News APIs Tech: - FastAPI - APScheduler - feedparser --- ### 2. Embedding Service Handles: - embeddings - semantic search - deduplication - clustering Tech: - sentence-transformers - Qdrant or ChromaDB I would switch from ChromaDB to: - urlQdranthttps://qdrant.tech Qdrant is much better now for production workloads. --- ### 3. AI Generation Service Handles: - summarization - rewriting - title generation - style transfer - category prediction Tech: - Ollama - vLLM - llama.cpp - LiteLLM You should abstract providers so you can swap: - local models - OpenAI - Anthropic - Groq - DeepSeek - whatever appears next year --- ### 4. Ranking Engine This is the most important upgrade. Your old system generates content. The new system should decide: - what matters - what trends - what becomes viral - what aligns with user interests This is where your persona/reddit-style ideas become powerful. You could score articles based on: - semantic novelty - emotional intensity - Reddit traction - cluster velocity - ideological spread - engagement prediction This becomes your moat. --- # Database Since you already use PostgreSQL: Use: - urlPostgreSQLhttps://www.postgresql.org - pgvector extension Honestly this is probably cleaner than maintaining separate vector infra unless scale explodes. Tables: ```text articles sources embeddings topics clusters generated_posts personas summaries trends users bookmarks ``` --- # New Features You Should Add ## 1. Multi-Agent Editorial Pipeline Your old repo hinted at agents. citeturn0search0 Now make it real: ```text Research Agent ↓ Fact Checker ↓ Bias Detector ↓ Style Writer ↓ Headline Optimizer ↓ SEO Agent ``` Each agent produces structured JSON. --- # 2. Time-Based Story Evolution One of the coolest possible features: Track how stories evolve over time. Example: ```text Day 1: "Minor AI model release" Day 5: "Open source forks explode" Day 12: "Company valuation jumps" Day 20: "Regulatory scrutiny" ``` Visualize narrative drift. That is genuinely interesting. --- # 3. Semantic News Graph This aligns perfectly with your graph/agent interests. Represent: - topics - companies - people - narratives - emotions as nodes. Then: - discover emerging themes - auto-generate explainers - visualize ideological ecosystems Use: - networkx - Neo4j - Cytoscape --- # 4. “Living Articles” Instead of static posts: An article updates automatically as new information arrives. Like: ```text AI Chip War Timeline Updated every hour ``` That is much more valuable than static AI sludge blogs. --- # 5. Local-First AI This is extremely aligned with your philosophy. Support: - Ollama - llama.cpp - vLLM - local embeddings - local reranking - offline mode Your audience from /r/LocalLLaMA would love this. --- # Frontend Ideas You should stop thinking “blog.” Think: - Bloomberg Terminal - hacker dashboard - cyberpunk Reuters - semantic newspaper Possible sections: ```text /stream /trends /clusters /world-model /timelines /contradictions /emerging-signals ``` --- # What I Would Personally Build If I were rebuilding your repo from scratch today: ## Version 2 Architecture ```text Next.js Frontend ↓ FastAPI Gateway ↓ Event Queue (Redis / RabbitMQ) ↓ Microservices ├── ingestion ├── embeddings ├── clustering ├── generation ├── ranking ├── media └── analytics ↓ Postgres + pgvector ↓ Ollama/vLLM ``` --- # Biggest Conceptual Upgrade Your original system was: ```text news → summarize → blog post ``` The new system should become: ```text global information stream ↓ semantic analysis ↓ narrative detection ↓ knowledge synthesis ↓ living intelligence system ``` That difference matters. --- A few repos/projects worth studying for inspiration: - urlQdranthttps://github.com/qdrant/qdrant - urlLangGraphhttps://github.com/langchain-ai/langgraph - urlOpenWebUIhttps://github.com/open-webui/open-webui - urlHaystackhttps://github.com/deepset-ai/haystack - urlFlowisehttps://github.com/FlowiseAI/Flowise - urln8nhttps://n8n.io And your original inspiration repo: - urlautoblog01turn0search0 You are actually in a good position to build this because your interests already overlap: - local LLMs - graphs - personas - automated generation - semantic systems - content pipelines - agent orchestration This project is basically the convergence point of everything you’ve been experimenting with for the last two years.

write a coding agent which will generate the full spec to assemble this

```python # spec_agent.py # # Autonomous specification-generation agent for rebuilding News13 # into a modern AI-native semantic news platform. # # Purpose: # - Generates full architecture specs # - Generates implementation roadmaps # - Generates API schemas # - Generates DB schemas # - Generates agent workflows # - Generates frontend plans # - Generates infra/deployment docs # - Generates task trees for coding agents # # Designed for: # - Local LLMs # - Ollama # - OpenAI-compatible APIs # - Multi-agent orchestration # # Author Philosophy: # "Turn vague ideas into executable systems." # from dataclasses import dataclass, field from typing import Dict, List, Any import json import uuid import os from datetime import datetime ########################################################### # CONFIG ########################################################### MODEL = "qwen2.5-coder:14b" OUTPUT_DIR = "./generated_specs" SYSTEM_PROMPT = """ You are a senior systems architect and autonomous coding planner. Your task is to transform project ideas into: - complete technical specifications - architecture documents - implementation plans - task trees - database schemas - API contracts - infrastructure requirements - frontend structures - backend services - AI orchestration systems You think like: - a principal engineer - a startup CTO - a distributed systems architect - an AI infrastructure engineer Output MUST be: - highly detailed - modular - implementation-ready - realistic - scalable Prefer: - PostgreSQL - FastAPI - Next.js - Docker - Redis - pgvector - Ollama - local-first AI Always output valid markdown. """ ########################################################### # LLM PROVIDER ########################################################### class OllamaProvider: def __init__(self, model: str = MODEL): self.model = model def generate(self, prompt: str) -> str: import requests response = requests.post( "http://localhost:11434/api/generate", json={ "model": self.model, "prompt": f"{SYSTEM_PROMPT}\n\n{prompt}", "stream": False } ) data = response.json() return data["response"] ########################################################### # DATA MODELS ########################################################### @dataclass class SpecSection: name: str prompt: str filename: str @dataclass class ProjectSpec: project_name: str description: str sections: List[SpecSection] = field(default_factory=list) ########################################################### # SPEC AGENT ########################################################### class SpecAgent: def __init__(self, provider): self.provider = provider def generate_section( self, project: ProjectSpec, section: SpecSection ) -> str: full_prompt = f""" PROJECT NAME: {project.project_name} PROJECT DESCRIPTION: {project.description} SECTION TO GENERATE: {section.name} REQUIREMENTS: {section.prompt} Generate a complete specification document. """ print(f"[+] Generating: {section.name}") result = self.provider.generate(full_prompt) return result def save_output( self, filename: str, content: str ): os.makedirs(OUTPUT_DIR, exist_ok=True) path = os.path.join(OUTPUT_DIR, filename) with open(path, "w") as f: f.write(content) print(f"[+] Saved: {path}") def run(self, project: ProjectSpec): manifest = { "project": project.project_name, "generated_at": datetime.utcnow().isoformat(), "files": [] } for section in project.sections: content = self.generate_section( project, section ) self.save_output( section.filename, content ) manifest["files"].append(section.filename) self.save_output( "manifest.json", json.dumps(manifest, indent=2) ) ########################################################### # NEWS13 V2 SPEC ########################################################### news13_spec = ProjectSpec( project_name="News13 V2", description=""" An autonomous semantic news intelligence platform. The system: - ingests global information streams - performs semantic analysis - clusters narratives - generates evolving living articles - tracks story evolution - uses local LLMs - supports agentic workflows - visualizes narrative ecosystems Architecture goals: - local-first - scalable - AI-native - event-driven - multi-agent - modular """, sections=[ ################################################### # SYSTEM ARCHITECTURE ################################################### SpecSection( name="System Architecture", filename="01_system_architecture.md", prompt=""" Generate: - full distributed system architecture - service boundaries - event flow diagrams - communication patterns - queue architecture - scaling strategy - fault tolerance - service orchestration - request lifecycle - security model """ ), ################################################### # DATABASE ################################################### SpecSection( name="Database Design", filename="02_database_design.md", prompt=""" Generate: - PostgreSQL schema - pgvector usage - indexing strategy - partitioning strategy - full SQL tables - relationships - migrations - optimization recommendations - semantic search structures """ ), ################################################### # INGESTION ################################################### SpecSection( name="Ingestion Service", filename="03_ingestion_service.md", prompt=""" Generate: - RSS ingestion architecture - Reddit ingestion - YouTube transcript ingestion - web scraping architecture - deduplication systems - queue ingestion - retry handling - content normalization - source reliability scoring """ ), ################################################### # AI PIPELINE ################################################### SpecSection( name="AI Processing Pipeline", filename="04_ai_pipeline.md", prompt=""" Generate: - summarization agents - embedding generation - clustering algorithms - topic extraction - sentiment analysis - narrative detection - contradiction detection - timeline generation - hallucination mitigation - reranking systems """ ), ################################################### # AGENT SYSTEM ################################################### SpecSection( name="Multi-Agent System", filename="05_multi_agent_system.md", prompt=""" Generate: - autonomous agent architecture - planner agents - researcher agents - fact-checker agents - editor agents - SEO agents - memory systems - tool use systems - orchestration logic - graph execution systems """ ), ################################################### # FRONTEND ################################################### SpecSection( name="Frontend Architecture", filename="06_frontend_architecture.md", prompt=""" Generate: - Next.js architecture - app router structure - frontend state management - streaming UI - semantic dashboards - narrative graph visualizations - timeline visualizations - search UX - mobile responsiveness - accessibility strategy """ ), ################################################### # API ################################################### SpecSection( name="API Design", filename="07_api_design.md", prompt=""" Generate: - REST API - websocket API - streaming endpoints - auth system - RBAC - OpenAPI schemas - pagination - semantic search endpoints - article generation endpoints - timeline APIs """ ), ################################################### # INFRASTRUCTURE ################################################### SpecSection( name="Infrastructure", filename="08_infrastructure.md", prompt=""" Generate: - Docker architecture - Kubernetes strategy - CI/CD - observability - metrics - logging - Redis usage - CDN strategy - edge deployment - backup strategy - disaster recovery """ ), ################################################### # LOCAL AI ################################################### SpecSection( name="Local AI Stack", filename="09_local_ai_stack.md", prompt=""" Generate: - Ollama architecture - llama.cpp integration - vLLM integration - model routing - quantization strategy - GPU planning - embedding model recommendations - inference batching - token optimization - local-first fallback systems """ ), ################################################### # TASK TREE ################################################### SpecSection( name="Implementation Task Tree", filename="10_task_tree.md", prompt=""" Generate: - complete implementation roadmap - milestone breakdown - sprint structure - dependency graph - coding priorities - MVP definition - production rollout strategy - testing plan - deployment phases """ ), ################################################### # AUTONOMOUS CODING ################################################### SpecSection( name="Coding Agent Instructions", filename="11_coding_agents.md", prompt=""" Generate: - prompts for coding agents - autonomous implementation instructions - repo structure instructions - PR generation rules - commit message conventions - test generation strategy - self-healing agent logic - code review agent systems """ ), ] ) ########################################################### # MAIN ########################################################### if __name__ == "__main__": provider = OllamaProvider() agent = SpecAgent(provider) agent.run(news13_spec) print("\n[✓] Full specification generated.") ``` The important idea here is that this is not merely a “prompt wrapper.” It is the beginning of a recursive architecture engine. The next evolution after this is: ```text spec generation ↓ task tree generation ↓ repo generation ↓ code generation ↓ test generation ↓ self-review ↓ repair loop ↓ deployment ``` Which means eventually: ```text idea → production system ``` with humans mostly supervising instead of hand-writing every file. The real unlock for you specifically is combining: - graph orchestration - local LLMs - coding agents - semantic memory - autonomous planning into one recursive development ecosystem. At that point News13 stops being a news app and becomes an autonomous information operating system.