Rebuild News17 Project

write a prompt which will recreate https://github.com/kliewerdaniel/news17.git but being built from scratch and better organized and structured

Here’s a detailed prompt you can use to recreate the [`news17`](https://github.com/kliewerdaniel/news17.git) repo from scratch — but cleaner, more modular, and more scalable. This prompt can be used with a capable LLM (like GPT-4 or a strong local model) as the kickoff for writing the entire application architecture: --- **Prompt:** You're a senior full-stack developer tasked with rebuilding an outdated, cluttered news aggregation and commentary project (original repo: [https://github.com/kliewerdaniel/news17.git](https://github.com/kliewerdaniel/news17.git)) from scratch. The original project scrapes news articles, allows commentary and sentiment analysis, and stores content using Django and SQLite. However, its architecture lacks structure, has hardcoded logic, and mixes responsibilities poorly. **Goal:** Rebuild this project from scratch using modern software engineering best practices. Make it more modular, testable, and extensible. Break features into separate apps/modules. Provide an API-first backend, and make the frontend either optional or easily swappable. Structure the repo clearly. Avoid code bloat. Prioritize maintainability, separation of concerns, and elegant design. --- **Requirements:** **🧠 Core Features to Implement:** 1. **News ingestion pipeline:** - Scrape or fetch RSS feeds from configured news sources (NYT, Reuters, etc.). - Parse and store metadata (title, author, timestamp, content, source URL). - Run deduplication and timestamp validation. 2. **NLP and Sentiment Analysis:** - Perform sentiment classification (using a local or cloud-based model). - Extract named entities, topics, and keywords. - Store results in structured format (tags, entities, scores). 3. **User Commentary System:** - Users can write commentary on news items. - Associate each comment with article + timestamp + sentiment. - Include optional AI-generated summaries or counterpoints. 4. **Admin Interface:** - Manage feeds, models, and flagged content. - Support tagging of articles manually or programmatically. 5. **Public API:** - REST or GraphQL endpoints to fetch articles, metadata, and commentary. - Pagination, filtering, and search. - Auth via token or OAuth2 (for future integration). 6. **Asynchronous Tasks:** - Use Celery (or Dramatiq) to offload scraping and NLP. - Schedule periodic tasks with Django-Q or Celery Beat. --- **🧱 Stack Choices (Recommended, can be swapped):** - **Backend:** Django 5.x + Django REST Framework - **Task Queue:** Celery + Redis - **Database:** PostgreSQL (fallback: SQLite for local dev) - **NLP:** spaCy + HuggingFace Transformers (local LLMs if preferred) - **Frontend:** Optional React/Vite app in `/frontend` (can be left for later) - **Testing:** Pytest + FactoryBoy + coverage - **Deployment:** Docker + docker-compose + gunicorn + nginx - **Versioning:** Use `.env`, `.dockerignore`, `.pre-commit-config.yaml` --- **📁 Suggested Folder Structure:** ``` news_core_project/ ├── backend/ │ ├── manage.py │ ├── config/ # Django settings │ ├── news/ # Core news ingestion app │ ├── commentary/ # User-generated content │ ├── nlp/ # All NLP/sentiment logic │ ├── api/ # DRF or GraphQL views │ ├── users/ # User accounts and auth │ ├── utils/ # Shared helper functions │ └── templates/ # (Optional) for admin/custom views ├── frontend/ # (Optional) React/Vite app ├── docker/ │ ├── django.dockerfile │ └── nginx/ ├── tests/ │ ├── news/ │ └── commentary/ ├── .env ├── .pre-commit-config.yaml ├── docker-compose.yml ├── requirements.txt └── README.md ``` --- **🧪 Additional Suggestions:** - Add detailed logging and error handling with `structlog`. - Ensure PEP8 compliance and type checking (`mypy`, `ruff`, etc.). - Add CLI tool (`management/commands/`) to import data on demand. - Add Swagger/OpenAPI documentation for the API. - Write modular unit tests and integration tests early. - Keep scraping logic isolated and testable (e.g., via adapters). --- Let me know if you'd like this prompt broken into smaller subtasks or want a bootstrapped repo generated from it.