LLM Development Process

write a program with this: Update: State of Software Development with LLMs - v2 Discussion update: I put some thinking into how to adhere the UI to DDD, which from user POV is not always useful (e.g. multiple domains in one screen), see below. I also integrated your feedback and comments from various threads. Prologue I’ve compiled insights from my experience and various channels over the past year to share a practical, evolving approach to developing sophisticated applications with LLMs. This is a work in progress, so feel free to contribute and critique! Introduction We’ve all witnessed relevant LLM advancements in the past year: Decreasing hallucinations Improved consistency Expanded context lengths Yet, they still struggle with generating complex, high-quality software solutions exceeding a few files without lots of manual intervention. What do humans do when tasks get complex? We model the work, break it into manageable pieces, and execute step-by-step. This principle drives this approach for AI as well: building a separated front/backend application using React (TS), Python, and any RDBMS. I chose these technologies due to their compatibility and relatively high-quality LLM-outputs (despite my limited prior experience in them). I won’t dive into well-known optimization techniques like CoT, ToT, or Mixture of Experts. For a good overview of those methods, see this excellent post. Approach Breakdown 1. Ideation Phase Goal: Have ALL high-level requirements for your applications. How: Use a prompt to enhances context, purpose, and business area and group requirements into meaningful sorted sub-domains. Tool: Utilize a custom UI interacting with your favorite LLM to manually review, refine, and trigger LLM rethinking for better outputs. As LLMs get better, we might not need this anymore. 2. Requirements Phase Goal: Have a full list of detailed requirements for your application How: Use a prompt to expand the high-level requirements into a comprehensive list of detailed requirements (e.g. user stories with acceptance criteria) for each sub-domain. Tool: A similar custom tool like above 3. Structuring Phase Goal: Have a consistent Domain-Driven Design (DDD) model. How: Use a prompt to output a specific JSON-based schema reflecting a DDD model for every domain based on the user stories. Use a ddd_schematon.md. Tool: The custom tool from above 4. Development Phase 1 Goal: Have consistent and high quality code for both backend and frontend components. Steps: Start with TDD: Define structure, then create the database (tables, schema). Develop DB-tables and backend code with APIs adhering to DDD interfaces. Generate frontend components based on mock-ups and backend specifications. Package the frontend components into a library to be used below Best Practices: Use templates to ensure consistency Use architecture and coding patterns (e.g., SOLID, OOP, PURE) (architecture.md) Consider using prompt templates (see Cursor Examples) First prompt LLMs for an implementation plan, then let it execute it. automatically feed errors back into the LLM, only GIT commit and push without compiler warnings u/IMYoric suggested proofs as a way to eliminate LLM faults, also using BDD during the requirements phase could help. Tool: Any IDE with an integrated LLM which is git-enabled (e.g., for branch creation, git diffs). Avoid using LLMs for code diffs—git is better suited for this task. 5. UX Design Phase Goal: Generate mock ups and the screen design from the list of HL requirements using above front-end components How: Use prompts informed by your DDD model and a predefined style guide (style-guide.md). Best Practices: Use tools like ComfyUI for asset creation Validate your UIs with simple code-created from paper-scribbles (I use chatgpt to create flutter and flutlabs.io to send me the APK) Tool: UX LLM-enabled tool like figma for the UI, I am not aware of any tool which can adhere to specific component definition though. 6. Development Phase 2 Goal: Have high-quality, maintainable front code How: Use a prompt to create code from above mock-ups and component definition for each UI. Best Practices see Dev Phase 1 Tool: see Dev Phase 1 7. Deployment Phase Goal: Have your application deployed How: Use a prompt to deploy your backend services and your front-end code with e.g. Kubernetes. u/snoosquirrels6702 created a PoC for AWS: https://www.reddit.com/r/ChatGPTCoding/comments/1i3ovtv/ai_agents_to_do_devops_work_can_be_used_by/ 8. Validation Phase Goal: Automate functional end-to-end and NFR testing. How: Prompt the LLM to generate test scripts (e.g., Selenium) based on your mock-ups and user stories. Use a prompt library to improve on non-functional requirements (NFRs) for maintainability, security, usability, and performance. AI can also help with that Integrations with profiling tools to automate aspects of NFR validation, would be valuable. Errors during E2E testing trigger the restart of the process from Dev Phase 1. My Tooling So Far I’ve successfully applied steps 1, 2, 3, and 5a (minus mock-ups). Using LLMs, I also created a custom UI with a state machine and DB to manage these processes and store the output. Output Code is manually pushed to GitHub. Shout outs Thanks to u/alexanderisora, u/bongsfordingdongs, u/LorestForest, u/RonaldTheRight for their inspiring prior work! See also https://www.reddit.com/r/ChatGPTPro/comments/1i00wmh/this_is_the_right_way_to_build_ios_app_with_ai/ for a similar approach. About Me 7 years as a professional developer (C#, Java, LAMP mostly web apps in enterprise settings). I also shorty worked as Product Owner and Tester shortly in my career. 8 years in architecture (business and application), working with startups and large enterprises. Recently led a product organization of ~200 people. Upvote 1 Downvote 1 Go to comments

Here's a Python program that organizes the **"State of Software Development with LLMs - v2"** approach into actionable steps, creating a CLI-based tool. This tool helps developers and teams walk through each phase while maintaining outputs for the next step. It integrates DDD principles, prompts, and manual validation. ### Code ```python import os import json from typing import List, Dict # Helper functions for file management and input def save_to_file(filename: str, data: Dict): with open(filename, 'w') as f: json.dump(data, f, indent=4) def load_from_file(filename: str) -> Dict: if os.path.exists(filename): with open(filename, 'r') as f: return json.load(f) return {} def user_input(prompt: str) -> str: return input(f"{prompt}: ") # Phases Implementation class LLMDevelopmentProcess: def __init__(self): self.data_store = load_from_file("llm_process_data.json") def save_progress(self): save_to_file("llm_process_data.json", self.data_store) def ideation_phase(self): print("\n=== Ideation Phase ===") high_level_requirements = user_input("Enter high-level requirements") domains = user_input("Enter sub-domains (comma-separated)") self.data_store['ideation'] = { "high_level_requirements": high_level_requirements, "domains": domains.split(',') } self.save_progress() print("Ideation phase completed and saved.") def requirements_phase(self): print("\n=== Requirements Phase ===") if 'ideation' not in self.data_store: print("Please complete the Ideation Phase first.") return detailed_requirements = {} for domain in self.data_store['ideation']['domains']: req = user_input(f"Enter user stories for domain '{domain}'") detailed_requirements[domain] = req.split(';') self.data_store['requirements'] = detailed_requirements self.save_progress() print("Requirements phase completed and saved.") def structuring_phase(self): print("\n=== Structuring Phase ===") if 'requirements' not in self.data_store: print("Please complete the Requirements Phase first.") return ddd_model = {} for domain, stories in self.data_store['requirements'].items(): schema = user_input(f"Define DDD schema for domain '{domain}'") ddd_model[domain] = schema self.data_store['structuring'] = ddd_model self.save_progress() print("Structuring phase completed and saved.") def development_phase_1(self): print("\n=== Development Phase 1 ===") if 'structuring' not in self.data_store: print("Please complete the Structuring Phase first.") return for domain, schema in self.data_store['structuring'].items(): db_tables = user_input(f"Define DB tables for domain '{domain}'") backend_code = user_input(f"Describe backend API implementation for domain '{domain}'") self.data_store.setdefault('development_phase_1', {})[domain] = { "db_tables": db_tables, "backend_code": backend_code } self.save_progress() print("Development Phase 1 completed and saved.") def ux_design_phase(self): print("\n=== UX Design Phase ===") if 'ideation' not in self.data_store: print("Please complete the Ideation Phase first.") return mockups = {} for domain in self.data_store['ideation']['domains']: mockup = user_input(f"Define mockup details for domain '{domain}'") mockups[domain] = mockup self.data_store['ux_design'] = mockups self.save_progress() print("UX Design phase completed and saved.") def deployment_phase(self): print("\n=== Deployment Phase ===") deployment_steps = user_input("Enter deployment steps (e.g., Kubernetes configuration)") self.data_store['deployment'] = {"steps": deployment_steps} self.save_progress() print("Deployment phase completed and saved.") def validation_phase(self): print("\n=== Validation Phase ===") test_scripts = user_input("Enter details for test scripts (e.g., Selenium scripts)") self.data_store['validation'] = {"test_scripts": test_scripts} self.save_progress() print("Validation phase completed and saved.") def show_progress(self): print("\n=== Current Progress ===") print(json.dumps(self.data_store, indent=4)) # CLI Interface def main(): process = LLMDevelopmentProcess() actions = { "1": process.ideation_phase, "2": process.requirements_phase, "3": process.structuring_phase, "4": process.development_phase_1, "5": process.ux_design_phase, "6": process.deployment_phase, "7": process.validation_phase, "8": process.show_progress } while True: print("\nChoose an action:") print("1. Ideation Phase") print("2. Requirements Phase") print("3. Structuring Phase") print("4. Development Phase 1") print("5. UX Design Phase") print("6. Deployment Phase") print("7. Validation Phase") print("8. Show Progress") print("9. Exit") choice = input("Enter your choice: ") if choice == "9": print("Exiting...") break action = actions.get(choice) if action: action() else: print("Invalid choice, please try again.") if __name__ == "__main__": main() ``` ### How It Works 1. **Phases**: Each phase has its own function to capture user inputs, such as requirements, schemas, or deployment steps. 2. **Progress Saving**: All data is saved to a JSON file (`llm_process_data.json`) for persistence across sessions. 3. **CLI Interface**: Users can navigate phases sequentially or jump to specific ones, viewing progress at any time. ### Next Steps 1. Integrate LLM APIs to automate schema generation, code suggestions, or testing. 2. Expand with templates for prompt engineering or DDD modeling. 3. Add validation rules for inputs to ensure consistency. Let me know if you'd like enhancements!

now rewrite as an agentic architecture using ollama

To rewrite this as an **agentic architecture** using **Ollama** for task orchestration, we can structure the system into distinct agents representing each phase of development. Each agent operates independently but collaborates through shared data and orchestrated tasks. The revised system will: 1. Use **Ollama** for generating responses to prompts in each phase. 2. Implement agents as Python classes. 3. Use a shared database (JSON or another lightweight store) for inter-agent communication. Here's the updated implementation: --- ### Code: Agentic Architecture ```python import os import json from typing import Dict, Any from ollama import Ollama # Helper functions for shared data def load_data(filename: str) -> Dict: if os.path.exists(filename): with open(filename, 'r') as f: return json.load(f) return {} def save_data(filename: str, data: Dict): with open(filename, 'w') as f: json.dump(data, f, indent=4) # Base Agent Class class BaseAgent: def __init__(self, data_file: str): self.data_file = data_file self.data = load_data(data_file) self.ollama = Ollama() # Initialize Ollama API def save_progress(self): save_data(self.data_file, self.data) def interact_with_llm(self, prompt: str) -> str: """Send a prompt to Ollama and retrieve the response.""" response = self.ollama.chat(prompt) return response.get("response", "No response received") # Agents for Each Phase class IdeationAgent(BaseAgent): def run(self): print("\n=== Ideation Phase ===") prompt = "Generate high-level requirements and group them into sub-domains for a software project." response = self.interact_with_llm(prompt) self.data['ideation'] = { "requirements": response, } self.save_progress() print("Ideation phase completed and saved.") class RequirementsAgent(BaseAgent): def run(self): if 'ideation' not in self.data: print("Please complete the Ideation Phase first.") return print("\n=== Requirements Phase ===") requirements = self.data['ideation']['requirements'] prompt = f"Expand the following high-level requirements into detailed user stories:\n{requirements}" response = self.interact_with_llm(prompt) self.data['requirements'] = { "user_stories": response, } self.save_progress() print("Requirements phase completed and saved.") class StructuringAgent(BaseAgent): def run(self): if 'requirements' not in self.data: print("Please complete the Requirements Phase first.") return print("\n=== Structuring Phase ===") user_stories = self.data['requirements']['user_stories'] prompt = f"Create a DDD JSON schema for the following user stories:\n{user_stories}" response = self.interact_with_llm(prompt) self.data['structuring'] = { "ddd_schema": response, } self.save_progress() print("Structuring phase completed and saved.") class DevelopmentPhase1Agent(BaseAgent): def run(self): if 'structuring' not in self.data: print("Please complete the Structuring Phase first.") return print("\n=== Development Phase 1 ===") ddd_schema = self.data['structuring']['ddd_schema'] prompt = f"Generate database tables and backend code for the following DDD schema:\n{ddd_schema}" response = self.interact_with_llm(prompt) self.data['development_phase_1'] = { "backend_code": response, } self.save_progress() print("Development Phase 1 completed and saved.") class UXDesignAgent(BaseAgent): def run(self): if 'ideation' not in self.data: print("Please complete the Ideation Phase first.") return print("\n=== UX Design Phase ===") requirements = self.data['ideation']['requirements'] prompt = f"Create mockups and screen designs for the following requirements:\n{requirements}" response = self.interact_with_llm(prompt) self.data['ux_design'] = { "mockups": response, } self.save_progress() print("UX Design phase completed and saved.") class DeploymentAgent(BaseAgent): def run(self): print("\n=== Deployment Phase ===") prompt = "Generate deployment steps (e.g., Kubernetes YAMLs) for a full-stack application." response = self.interact_with_llm(prompt) self.data['deployment'] = { "deployment_steps": response, } self.save_progress() print("Deployment phase completed and saved.") class ValidationAgent(BaseAgent): def run(self): if 'requirements' not in self.data: print("Please complete the Requirements Phase first.") return print("\n=== Validation Phase ===") user_stories = self.data['requirements']['user_stories'] prompt = f"Generate end-to-end test scripts and non-functional validation for the following user stories:\n{user_stories}" response = self.interact_with_llm(prompt) self.data['validation'] = { "test_scripts": response, } self.save_progress() print("Validation phase completed and saved.") # Orchestrator class Orchestrator: def __init__(self, data_file: str = "agentic_data.json"): self.data_file = data_file self.agents = { "1": IdeationAgent(data_file), "2": RequirementsAgent(data_file), "3": StructuringAgent(data_file), "4": DevelopmentPhase1Agent(data_file), "5": UXDesignAgent(data_file), "6": DeploymentAgent(data_file), "7": ValidationAgent(data_file), } def run(self): while True: print("\nChoose an action:") print("1. Ideation Phase") print("2. Requirements Phase") print("3. Structuring Phase") print("4. Development Phase 1") print("5. UX Design Phase") print("6. Deployment Phase") print("7. Validation Phase") print("8. Exit") choice = input("Enter your choice: ") if choice == "8": print("Exiting...") break agent = self.agents.get(choice) if agent: agent.run() else: print("Invalid choice, please try again.") # Main Execution if __name__ == "__main__": orchestrator = Orchestrator() orchestrator.run() ``` --- ### Features 1. **Agent-Based Modularity**: Each phase is managed by its own agent for separation of concerns and easy extensibility. 2. **Ollama Integration**: Uses `Ollama` for generating phase-specific outputs via prompts. 3. **Persistent Data**: Saves progress to `agentic_data.json`, ensuring data is carried across sessions. 4. **Orchestrator**: Allows users to navigate phases in a non-linear way while maintaining dependencies. ### Next Steps 1. **Error Handling**: Improve `interact_with_llm` to retry or refine prompts on failure. 2. **UI/UX**: Build a frontend or CLI-enhanced version for better interactivity. 3. **Automation**: Add hooks to execute generated code, test scripts, or deployment steps automatically. Let me know if you'd like further customization!