MCP Browser Automation Integration

using https://github.com/kliewerdaniel/orchestrator-ollama.git integrate a locally run MCP browser client and browser server. I want to create tools for each of the agents to use and to have access to in order to accomplish a task. The functions they have access to will allow them to edit the ai_guidelines.md which is composed of the guidelines from software development principles that will be used. What I want you to do is to create a series of iterative prompts I can begin with this repo and build this functionality using Ollama and local models only you can use the following to help you set up the MCP: # AI Guidelines for MCP Browser Automation Project ## Overview This document provides the architecture, construction prompts, and instructions for maintaining an AI-generated ledger (`ai_output.md`) to track project context. The project leverages MCP (Model Context Protocol) with Selenium for browser automation and a locally hosted Ollama model to facilitate AI-driven interactions. Additionally, it includes instructions for modifying `ai_guidelines.md` to adapt and improve the program over time. --- ## Architecture ### **Components** 1. **MCP Server (`mcp_browser_server.py`)** - Hosts tools for browser navigation. - Uses Selenium with a Chrome WebDriver. - Handles page interactions (e.g., navigation, form filling, button clicks). 2. **MCP Client (`mcp_browser_client.py`)** - Connects to the MCP server. - Issues commands to automate browser tasks. - Uses Ollama for AI-driven decision-making. 3. **Ollama Integration** - Runs a local LLM (e.g., Gemma3 or Mistral). - Processes natural language queries to automate tasks. - Augments browser interactions by making context-aware decisions. 4. **AI Output Ledger (`ai_output.md`)** - Stores AI-generated responses, metadata, and execution logs. - Tracks command history and context for reproducibility. 5. **Modifiable AI Guidelines (`ai_guidelines.md`)** - This document itself can be updated to refine the workflow and enhance AI capabilities. - Instructions for modifying are included below. --- ## **Installation Requirements** Create a `requirements.txt` file with the following dependencies: ``` mcp selenium webdriver-manager ollama beautifulsoup4 requests undetected-chromedriver ``` Run the following command to install dependencies: ```bash pip install -r requirements.txt ``` To pull a model for Ollama, use: ```bash ollama pull mistral ``` --- ## **Prompt Series for Constructing the Project** ### **1. Setup and Install Dependencies** > "Ensure MCP, Selenium, WebDriver, and Ollama are installed. Configure a virtual environment if necessary. Verify that Chrome WebDriver is functional." ### **2. Implement the MCP Server for Browser Automation** > "Generate a Python script (`mcp_browser_server.py`) that defines an MCP server with a `navigate` tool using Selenium. The tool should accept a URL, load the page, return the title, and handle exceptions gracefully." Example `mcp_browser_server.py`: import time from selenium import webdriver from selenium.webdriver.chrome.service import Service as ChromeService from webdriver_manager.chrome import ChromeDriverManager from mcp.server.fastmcp import FastMCP # Initialize the MCP server with a descriptive name. mcp = FastMCP("Browser Agent Server") @mcp.tool() def navigate(url: str) -> str: """ Navigates to a given URL using a headless Chrome browser and returns the page title. """ # Set up headless Chrome options. options = webdriver.ChromeOptions() options.add_argument("--headless") # Initialize the Chrome driver. driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options) try: driver.get(url) time.sleep(2) # Give the page time to load. title = driver.title return f"Page title: {title}" except Exception as e: return f"Error: {str(e)}" finally: driver.quit() if __name__ == "__main__": print("Starting MCP Browser Agent Server...") mcp.run(transport="stdio") ### **3. Implement the MCP Client** > "Generate a Python script (`mcp_browser_client.py`) that connects to the MCP server, lists available tools, and calls `navigate` with a sample URL, printing the page title." Example `mcp_browser_client.py`: import asyncio from mcp.client.stdio import stdio_client from mcp import ClientSession, StdioServerParameters # Define the parameters for the MCP server; adjust the command/args as needed. server_params = StdioServerParameters( command="python", args=["./mcp_browser_server.py"] ) async def run_client(): # Connect to the MCP server. async with stdio_client(server_params) as session: # List the available tools (for debugging or dynamic agent behavior). tools = await session.list_tools() print("Available tools:", [tool.name for tool in tools.tools]) # Use the 'navigate' tool to visit a URL. response = await session.run_tool("navigate", {"url": "https://example.com"}) print("Response:", response) if __name__ == "__main__": asyncio.run(run_client()) ### **4. Integrate Ollama for AI-Driven Navigation** > "Modify the MCP client to use a locally hosted Ollama model to process user input and determine browser commands dynamically. Include a system prompt instructing the AI to act as an autonomous browsing assistant." ### **5. Maintain AI Context in `ai_output.md`** > "Generate a format for recording AI-generated responses in `ai_output.md`, including metadata such as timestamps, executed commands, and extracted webpage content." ### **6. Expand Browser Capabilities** > "Provide additional tools in the MCP server for filling forms, clicking buttons, and extracting page elements. Modify the client to support these interactions dynamically based on AI instructions." ### **7. Test the Workflow** > "Outline the steps to test the full pipeline: running the MCP server, executing the client with AI-driven commands, and verifying interactions via `ai_output.md`." --- ## **Instructions for Using `ai_output.md` as a Ledger** 1. **Log Every AI Interaction** - Each action executed by the AI should be recorded with: - **Timestamp** (UTC format) - **Command issued** (by AI or user) - **Execution result** (success/failure message) - **Extracted information** (page titles, form values, etc.) 2. **Include Metadata for Context Tracking** - Every entry should contain: - `session_id`: Unique identifier for an AI session. - `user_query`: Original prompt given to the AI. - `ai_response`: AI-generated command output. - `browser_state`: Captured data from Selenium (page titles, element values). 3. **Format for AI Output Entries** ``` ## Entry: YYYY-MM-DD HH:MM:SS UTC - **Session ID**: abc123xyz - **User Query**: "Navigate to example.com and extract the main heading." - **AI Response**: "Navigating to example.com... Extracting H1 tag..." - **Executed Command**: `navigate("https://example.com")` - **Browser State**: `{"page_title": "Example Domain", "h1_text": "Example Domain"}` - **Execution Status**: Success ``` 4. **Ensure Reproducibility** - Future executions should refer to previous entries to maintain context. - If an error occurs, logs should include failure reasons and retry suggestions. --- ## **Modifying `ai_guidelines.md` to Improve the Project** ### **Updating Architecture & Prompts** - If a new feature is added, update the **Architecture** section with new components. - Modify the **Prompt Series** to reflect new development goals. ### **Enhancing AI Context Tracking** - Adjust the **AI Output Ledger** format if new metadata fields are needed. - Introduce additional tracking mechanisms like screenshots or browser session IDs. ### **Expanding Browser Capabilities** - Define new tools for advanced automation (e.g., JavaScript execution, mouse movements). - Modify existing tools to improve performance or error handling. ### **Submitting Updates** 1. Open `ai_guidelines.md` in an editor. 2. Update relevant sections with new information. By maintaining and refining this document, we ensure the MCP browser automation project evolves effectively and adapts to new challenges.

To integrate a locally run MCP (Model Context Protocol) browser client and server using the [orchestrator-ollama](https://github.com/kliewerdaniel/orchestrator-ollama.git) repository, and to develop tools enabling agents to edit the `ai_guidelines.md` based on software development principles, follow this series of iterative prompts: **1. Clone and Explore the Repository** Begin by cloning the `orchestrator-ollama` repository and examining its structure to understand the existing components. ```bash git clone https://github.com/kliewerdaniel/orchestrator-ollama.git cd orchestrator-ollama ```  **2. Set Up the Development Environment** Ensure your development environment has Python installed, and create a virtual environment to manage dependencies. ```bash python -m venv env source env/bin/activate # On Windows, use 'env\Scripts\activate' ```  **3. Install Necessary Dependencies** Create a `requirements.txt` file with the following dependencies: ``` mcp selenium webdriver-manager ollama beautifulsoup4 requests undetected-chromedriver ```  Install the dependencies using pip: ```bash pip install -r requirements.txt ```  **4. Implement the MCP Server for Browser Automation** Develop an MCP server (`mcp_browser_server.py`) that utilizes Selenium to automate browser tasks. This server will expose tools such as `navigate`, `fill_form`, and `click_button`. ```python import time from selenium import webdriver from selenium.webdriver.chrome.service import Service as ChromeService from webdriver_manager.chrome import ChromeDriverManager from mcp.server.fastmcp import FastMCP mcp = FastMCP("Browser Automation Server") @mcp.tool() def navigate(url: str) -> str: options = webdriver.ChromeOptions() options.add_argument("--headless") driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options) try: driver.get(url) time.sleep(2) return f"Page title: {driver.title}" except Exception as e: return f"Error: {str(e)}" finally: driver.quit() if __name__ == "__main__": mcp.run(transport="stdio") ```  **5. Develop the MCP Client** Create an MCP client (`mcp_browser_client.py`) that connects to the MCP server and utilizes the available tools. ```python import asyncio from mcp.client.stdio import stdio_client from mcp import ClientSession, StdioServerParameters server_params = StdioServerParameters( command="python", args=["./mcp_browser_server.py"] ) async def run_client(): async with stdio_client(server_params) as session: tools = await session.list_tools() print("Available tools:", [tool.name for tool in tools.tools]) response = await session.run_tool("navigate", {"url": "https://example.com"}) print("Response:", response) if __name__ == "__main__": asyncio.run(run_client()) ```  **6. Integrate Ollama for AI-Driven Decision Making** Incorporate a locally hosted Ollama model to process natural language instructions and determine appropriate browser commands. ```python import asyncio import ollama from mcp.client.stdio import stdio_client from mcp import ClientSession, StdioServerParameters server_params = StdioServerParameters( command="python", args=["./mcp_browser_server.py"] ) async def run_client(): async with stdio_client(server_params) as session: tools = await session.list_tools() print("Available tools:", [tool.name for tool in tools.tools]) user_input = input("Enter your instruction: ") ollama_response = ollama.complete(model="mistral", prompt=user_input) command = parse_ollama_response(ollama_response) response = await session.run_tool(command['tool'], command['params']) print("Response:", response) def parse_ollama_response(response): # Implement parsing logic based on Ollama's output return {"tool": "navigate", "params": {"url": "https://example.com"}} if __name__ == "__main__": asyncio.run(run_client()) ```  **7. Implement AI Guidelines Editing Tool** Develop a tool that allows agents to edit the `ai_guidelines.md` file based on AI-generated suggestions. ```python def edit_ai_guidelines(suggestions: str): with open('ai_guidelines.md', 'a') as file: file.write("\n" + suggestions) # Example usage ollama_response = ollama.complete(model="mistral", prompt="Suggest improvements for ai_guidelines.md") edit_ai_guidelines(ollama_response) ```  **8. Maintain an AI Output Ledger** Create an `ai_output.md` file to log AI interactions, including timestamps, commands issued, and execution results. ```markdown ## Entry: 2025-03-17 23:54:29 UTC - **Session ID**: abc123xyz - **User Query**: "Navigate to example.com and extract the main heading." - **AI Response**: "Navigating to example.com... Extracting H1 tag..." - **Executed Command**: `navigate("https://example.com")` - **Browser State**: `{"page_title": "Example Domain", "h1_text": "Example Domain"}` - **Execution Status**: Success ```  **9. Test and Iterate** Continuously test the integrated system, refine the tools, and update the `ai_guidelines.md` to enhance the agents' capabilities and ensure alignment with software development principles. By following these steps, you will build a functional system where agents can autonomously perform browser automation tasks and collaboratively edit guidelines to improve their performance over time.

Below is a series of iterative prompts you can follow to build out your MCP browser automation project using the provided repo and local Ollama models. Each prompt builds on the previous one and includes both implementation guidance and instructions for modifying your AI guidelines file (`ai_guidelines.md`). You can run these one after another to incrementally integrate the MCP server/client, Ollama, and ledger functionality. --- ### **Prompt 1: Setup and Install Dependencies** > **Objective:** > - Clone the [orchestrator-ollama](https://github.com/kliewerdaniel/orchestrator-ollama.git) repository. > - Create a Python virtual environment. > - Create a `requirements.txt` (if not already present) and install the following packages: > ``` > mcp > selenium > webdriver-manager > ollama > beautifulsoup4 > requests > undetected-chromedriver > ``` > - Pull the desired model for Ollama, for example: > ```bash > ollama pull mistral > ``` > - Verify that Chrome WebDriver is correctly installed and accessible. --- ### **Prompt 2: Implement the MCP Server for Browser Automation** > **Objective:** > - Create a file `mcp_browser_server.py` that starts an MCP server using Selenium. > - Define a tool called `navigate` that accepts a URL, opens it in a headless Chrome instance, and returns the page title. > - Ensure error handling and proper cleanup (closing the browser). **Example Code:** ```python import time from selenium import webdriver from selenium.webdriver.chrome.service import Service as ChromeService from webdriver_manager.chrome import ChromeDriverManager from mcp.server.fastmcp import FastMCP # Initialize the MCP server with a descriptive name. mcp = FastMCP("Browser Agent Server") @mcp.tool() def navigate(url: str) -> str: """ Navigates to a given URL using a headless Chrome browser and returns the page title. """ options = webdriver.ChromeOptions() options.add_argument("--headless") driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options) try: driver.get(url) time.sleep(2) # Allow the page to load. title = driver.title return f"Page title: {title}" except Exception as e: return f"Error: {str(e)}" finally: driver.quit() if __name__ == "__main__": print("Starting MCP Browser Agent Server...") mcp.run(transport="stdio") ``` --- ### **Prompt 3: Implement the MCP Client** > **Objective:** > - Create a file `mcp_browser_client.py` that connects to your MCP server. > - The client should list available tools (for confirmation) and then use the `navigate` tool to visit a sample URL (e.g., "https://example.com") and print the result. **Example Code:** ```python import asyncio from mcp.client.stdio import stdio_client from mcp import ClientSession, StdioServerParameters # Define parameters for the MCP server. server_params = StdioServerParameters( command="python", args=["./mcp_browser_server.py"] ) async def run_client(): async with stdio_client(server_params) as session: # List available tools. tools = await session.list_tools() print("Available tools:", [tool.name for tool in tools.tools]) # Use the 'navigate' tool to visit a sample URL. response = await session.run_tool("navigate", {"url": "https://example.com"}) print("Response:", response) if __name__ == "__main__": asyncio.run(run_client()) ``` --- ### **Prompt 4: Integrate Ollama for AI-Driven Navigation** > **Objective:** > - Update the MCP client to integrate a locally hosted Ollama model. > - Process natural language queries to generate browser commands dynamically. > - Include a system prompt that instructs the AI to act as an autonomous browsing assistant. > - You might need to create an interface that accepts user input, sends it to Ollama for interpretation, and maps the output to one or more MCP tool commands. **Guidance:** > - Investigate the Ollama Python integration (or its API) to send user prompts and receive responses. > - Extend your client code to include a function such as `process_query(query: str) -> dict` that returns the necessary tool command and parameters. > - Use the processed command to call the appropriate MCP tool. --- ### **Prompt 5: Create and Maintain an AI Output Ledger** > **Objective:** > - Design and implement a logging mechanism for all AI-driven interactions. > - Create a file `ai_output.md` that acts as a ledger, logging each step with: > - **Timestamp (UTC)** > - **Session ID** > - **User Query** > - **AI Response/Generated Command** > - **Executed MCP Command** > - **Execution Status (Success/Failure)** > - **Extracted data (e.g., page title or other elements)** > - Ensure that every action taken by the AI is appended to this ledger. **Example Entry Format:** ``` ## Entry: 2025-03-18 14:23:00 UTC - **Session ID**: abc123xyz - **User Query**: "Navigate to example.com and get the title." - **AI Response**: "Executing navigation to example.com." - **Executed Command**: `navigate("https://example.com")` - **Browser State**: `{"page_title": "Example Domain"}` - **Execution Status**: Success ``` --- ### **Prompt 6: Expand Browser Capabilities** > **Objective:** > - Add more tools to the MCP server to handle additional browser interactions: > - **Filling forms** > - **Clicking buttons** > - **Extracting specific page elements (using BeautifulSoup or Selenium methods)** > - Modify the client to accept dynamic commands based on AI instructions. > - Update `ai_guidelines.md` to document any new tools and behaviors introduced. **Example Addition for Clicking a Button:** ```python @mcp.tool() def click_button(selector: str) -> str: """ Finds a button on the page using a CSS selector and clicks it. """ options = webdriver.ChromeOptions() options.add_argument("--headless") driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options) try: driver.get("current_page_url_placeholder") # Replace with session context management. button = driver.find_element_by_css_selector(selector) button.click() return "Button clicked successfully." except Exception as e: return f"Error clicking button: {str(e)}" finally: driver.quit() ``` --- ### **Prompt 7: Test the Full Pipeline** > **Objective:** > - Run the complete system: start the MCP server, execute the client, and process a user query through Ollama. > - Verify that: > - AI instructions correctly translate into MCP commands. > - Each interaction is logged to `ai_output.md`. > - The browser interactions (navigation, clicks, etc.) behave as expected. > - Simulate error scenarios and ensure that error messages and logs capture all necessary metadata. **Steps to Test:** > 1. Start `mcp_browser_server.py` in one terminal. > 2. Run `mcp_browser_client.py` in another terminal. > 3. Input a natural language query that the Ollama model processes (e.g., "Go to example.com and click the login button"). > 4. Check the output in the terminal and verify that an entry is appended in `ai_output.md`. --- ### **Prompt 8: Iterative Improvements and Updating `ai_guidelines.md`** > **Objective:** > - As you add new features or modify behavior, keep the `ai_guidelines.md` document updated. > - Document any changes in architecture, tool functions, and ledger format. > - Review and refine the AI’s system prompts so that future AI decisions are based on the most up-to-date guidelines. **Instructions for Updating:** > 1. Open `ai_guidelines.md` in your preferred editor. > 2. Update sections under “Architecture”, “Prompt Series”, and “AI Output Ledger” with the latest details. > 3. Commit your changes to the repository to keep a version history of guidelines and project evolution. --- By following these iterative prompts, you'll progressively build a locally run MCP browser client and server integrated with Ollama for AI-driven browser automation. Each step is designed to ensure modular development and continuous logging, providing both functionality and a robust record of AI interactions.