Ollama Continue.dev Setup

Write a guide on setting up continue.dev to work with ollama and pulled models, part of it is editing the config.json, which requires you to add the models by adding under the models: { "title": "Qwen 2.5 Coder 3b", "model": "qwen2.5-coder:3b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" }, something like that, which is what it expects, you can get the names of the models by typesing ollama ls to list all the models with their name, if you are using ollama with continue.dev you have to add the apiBase/v1 key with the value of the ollama endpoint: localhost:11434/api/generate, this is for vscode for the installation of continue.dev in vscode, also tell me how to load any model I want for ollama from the gguf file or from pulling it from anywhere

Below is a detailed guide on how to integrate Continue.dev with Ollama within VSCode, configure your `config.json` to recognize Ollama models, and load new models from local gguf files or by pulling them from external sources. ### Prerequisites 1. **VSCode + Continue.dev**: Ensure you have Visual Studio Code installed and the [Continue.dev](https://marketplace.visualstudio.com/items?itemName=Continue.continue) extension installed. 2. **Ollama**: Install [Ollama](https://github.com/jmorganca/ollama), a local LLM runner that can host various models. Make sure Ollama is running and that you know the port it's listening on (default: `11434`). ### Step-by-Step Instructions #### 1. Start Ollama - Run `ollama server` or ensure Ollama is already running in the background. By default, Ollama exposes its API at `http://localhost:11434`. - You can verify this by navigating to `http://localhost:11434/version` in your browser or using `curl http://localhost:11434/version`. #### 2. List Available Models in Ollama To know which models Ollama currently manages, run: ```bash ollama ls ``` This will output something like: ``` qwen2.5-coder:3b llama-2-7b mistral-7b ... ``` Each line shows a model identifier you can use in the Continue.dev configuration. Models managed by Ollama often follow the format: `modelName:variantOrSize`, for example `qwen2.5-coder:3b`. #### 3. Configuring Continue.dev’s `config.json` Continue.dev reads its model configuration from a JSON file which you can typically find in your VSCode settings directory for Continue. The configuration might look like this (adjust the path as necessary): - On Linux/MacOS, a common location might be `~/.continue/config.json`. - On Windows, it might be in your user directory under a `.continue` folder. If you’re unsure, refer to the Continue.dev documentation or run the `Continue: Open Config` command from the VSCode command palette. Inside the `config.json`, you’ll have a `models` array. To integrate an Ollama model, you need to add an entry for it. A minimal example looks like this: ```json { "models": [ { "title": "Qwen 2.5 Coder 3b", "model": "qwen2.5-coder:3b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" } ] } ``` **Key Points:** - **`title`**: A human-friendly name for your model as it will appear in Continue’s model selection. - **`model`**: The exact name of the model as listed by `ollama ls`. This includes any tags like `:3b` or `:7b`. - **`provider`**: Set this to `"ollama"` so Continue knows to route prompts to the Ollama backend. - **`apiBase/v1`**: This must point to Ollama’s API endpoint for generating responses. By default, Ollama listens on `http://localhost:11434/api/generate`. Make sure this is included exactly as shown. You can add as many models as you like by including multiple objects in the `models` array, for example: ```json { "models": [ { "title": "Qwen 2.5 Coder 3b", "model": "qwen2.5-coder:3b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" }, { "title": "Llama 2 7B", "model": "llama-2-7b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" } ] } ``` #### 4. Loading Models into Ollama **Option A: Pulling Models from a Remote Source** If a model is hosted in a repository or by Ollama itself, you can pull it directly: ```bash ollama pull qwen2.5-coder:3b ``` This downloads the model files into Ollama’s directory. Once pulled, you can list it with `ollama ls` and add it to `config.json`. **Option B: Loading a Local GGUF Model** If you have a GGUF model file on your local machine (for example, `my-model.gguf`), you can integrate it with Ollama by creating a custom model YAML file that tells Ollama how to load it. Ollama’s documentation details this process, but it typically looks like: 1. Create a model YAML file (e.g. `my-model.yaml`) in your Ollama models directory (commonly `~/.ollama/models/`): ```yaml name: my-local-model model: /path/to/my-model.gguf ``` 2. Once you have the YAML file in place, run: ```bash ollama import my-local-model.yaml ``` This makes Ollama aware of the model. 3. After importing, you can verify it’s recognized: ```bash ollama ls ``` You should see `my-local-model` listed. 4. Add the model to Continue’s `config.json`: ```json { "models": [ { "title": "My Local Model", "model": "my-local-model", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" } ] } ``` #### 5. Using the Models in VSCode with Continue.dev - After editing `config.json`, restart Visual Studio Code or run `Continue: Reload` command from the command palette if available. - Open the Continue.dev panel (usually on the sidebar or by using the `Continue: Open` command). - Select the desired model from the model dropdown at the top of the Continue panel. - Start interacting with the model. Your queries and code completions should now route through Ollama’s locally hosted model. #### 6. Troubleshooting - **Connection Issues**: If Continue can’t reach Ollama, verify the `apiBase/v1` URL and port. The default should be `http://localhost:11434/api/generate` unless you changed Ollama’s default port. - **Missing Models**: If a model doesn’t show up, verify it’s listed by `ollama ls` and that you spelled it correctly in `config.json`. - **File Permissions**: On some systems, ensure you have the correct file permissions for the `.continue` directory and the Ollama model directories. --- **Summary:** To integrate Ollama with Continue.dev in VSCode, you need to edit your `config.json` to include a model entry pointing to Ollama’s `apiBase/v1` endpoint and referencing the model’s name exactly as Ollama recognizes it. You can load models by pulling them with `ollama pull` or importing a local GGUF file via a model YAML. After configuration, you can switch between any models you’ve added directly from Continue.dev’s interface in VSCode.

write a title for this post

**"Seamless Integration: Setting Up Continue.dev with Ollama for Local LLMs in VSCode"**

rewrite: https://danielkliewer.com/2024/12/19/continue.dev-ollama This set up was not outlined anywhere in a straightforward way so I wrote a guide on how to do this since I can see a lot of people using this method to be able to use their local machine rather than OpenAI or Anthropic or an internet connection required for running a LLM. In the guide I also show how you can load models you have downloaded. So you can fine-tune your own models and integrate them into whatever use case you have. This basically allows you to use your local compute instead of pay for an IDE integrated with an LLM with the ability to write and edit the files, tab autocompletions etc. There are paid versions, such as Cursor, Windsurf, and then Cline is an extension which I do not know if you can load local models as "easily" as with Continue.dev as I did in this example use case of a locally hosted LLM with Ollama. You can use paid API keys that you have yourself which would most likely return better results but it would cost money. This is why I am considering getting a better computer. If I can get a machine with enough VRAM to fit a large enough parameter model that can load models that compete with what Cursor or Windsurf provide, then I can avoid paying a monthly fee or per API call. In the long run that is what the investment would be. With how much I have spent on Cline or just API from testing, it would offset some of the expense for the hardware.

### Setting Up Continue.dev with Ollama for Local LLM Integration I couldn’t find a clear guide for this setup, so I decided to write one myself. This method is perfect for anyone who wants to use their local machine to run an LLM without relying on OpenAI, Anthropic, or even an internet connection. In this guide, I also explain how to load models you’ve downloaded, enabling you to fine-tune and integrate them into your workflows. The result? A cost-effective, fully local setup for tasks like writing, editing, and tab autocompletion—all integrated into your IDE. ### Why Use Continue.dev and Ollama? Paid tools like Cursor and Windsurf offer similar features, but they require subscriptions. Cline is another option, but it’s unclear whether it supports local model loading as easily as Continue.dev. With Continue.dev and Ollama, you can: - Use local compute power to run LLMs. - Avoid recurring costs associated with API calls or subscriptions. - Load and fine-tune your own models for custom use cases. While API keys for services like OpenAI or Anthropic may yield higher-quality results, they also come at a cost. This local-first approach provides a viable alternative for those who want to control expenses and rely on their own hardware. ### Future Plans: Investing in Better Hardware To make this setup even more powerful, I’m considering investing in a machine with enough VRAM to handle larger parameter models. This would allow me to run models comparable to those used by Cursor or Windsurf, avoiding monthly fees or per-API call charges. In the long run, the upfront investment in hardware could offset the costs I’ve incurred through subscriptions and API usage for testing. For anyone serious about local-first AI workflows, this could be a game-changer.

rewrite without titles bullet points or lists and as paragraphs

I couldn’t find a clear guide for setting up Continue.dev with Ollama, so I decided to write one myself. This approach is ideal for anyone who wants to use their local machine to run an LLM without relying on services like OpenAI, Anthropic, or even an internet connection. In the process, I also explain how to load models you’ve downloaded, making it possible to fine-tune and integrate them into whatever use cases you have. The result is a cost-effective, fully local setup that provides the ability to write, edit, and use tab autocompletions seamlessly within your IDE. While there are paid tools like Cursor and Windsurf that offer similar features, they often require subscriptions. Cline is another option, though it’s unclear whether it supports local model loading as easily as Continue.dev. By using Continue.dev and Ollama, you can leverage your own computing power to run LLMs locally, sidestepping recurring costs tied to API calls or subscriptions. This setup also allows you to load and fine-tune your own models for custom workflows. Although using paid API keys from services like OpenAI or Anthropic might deliver higher-quality results, they come with an ongoing cost, making this local-first approach an attractive alternative for those wanting more control over their expenses. Looking ahead, I’m considering investing in a machine with enough VRAM to handle larger parameter models. With better hardware, I could run models comparable to those used by Cursor or Windsurf, eliminating the need for monthly fees or per-API call charges. The upfront expense of such a system could be offset over time by reducing the money I’ve spent on subscriptions and API testing. For anyone exploring local-first AI workflows, investing in robust hardware could open up significant possibilities and long-term savings.

I couldn’t find a clear guide for setting up Continue.dev with Ollama, so I decided to create one myself. This method is ideal for anyone who wants to run a large language model (LLM) on their local machine without relying on services like OpenAI or Anthropic, or needing an internet connection. In my guide, I also explain how to load models you’ve downloaded, allowing you to fine-tune and integrate them into your specific use cases. Essentially, this setup lets you use your local computing power instead of paying for an IDE integrated with an LLM that offers features like writing, editing files, and tab autocompletions. There are paid alternatives available, such as Cursor and Windsurf, and an extension called Cline. However, I’m not sure if Cline allows for loading local models as easily as Continue.dev does in my example of a locally hosted LLM with Ollama. While you can use paid API keys from services like OpenAI or Anthropic, which likely provide better results, they come with ongoing costs. This is why I’m considering upgrading my computer. If I can acquire a machine with enough VRAM to handle a large parameter model that can compete with what Cursor or Windsurf offer, I could eliminate the need for monthly fees or per-API call charges. In the long run, this investment in better hardware would offset the expenses I’ve already incurred from using Cline or paying for API access during testing. For anyone serious about maintaining a local-first AI workflow, investing in robust hardware could be a worthwhile decision that provides both cost savings and greater control over your AI tools.

write a reddit post for this content: Daniel Kliewer About Setting Up Continue.dev with Ollama for Local LLMs in VSCode Prerequisites VSCode + Continue.dev: Ensure you have Visual Studio Code installed and the Continue.dev extension installed. Ollama: Install Ollama, a local LLM runner that can host various models. Make sure Ollama is running and that you know the port it’s listening on (default: 11434). Step-by-Step Instructions 1. Start Ollama Run ollama server or ensure Ollama is already running in the background. By default, Ollama exposes its API at http://localhost:11434. You can verify this by navigating to http://localhost:11434/version in your browser or using curl http://localhost:11434/version. 2. List Available Models in Ollama To know which models Ollama currently manages, run: ollama ls This will output something like: qwen2.5-coder:3b llama-2-7b mistral-7b ... Each line shows a model identifier you can use in the Continue.dev configuration. Models managed by Ollama often follow the format: modelName:variantOrSize, for example qwen2.5-coder:3b. 3. Configuring Continue.dev’s config.json Continue.dev reads its model configuration from a JSON file which you can typically find in your VSCode settings directory for Continue. The configuration might look like this (adjust the path as necessary): On Linux/MacOS, a common location might be ~/.continue/config.json. On Windows, it might be in your user directory under a .continue folder. If you’re unsure, refer to the Continue.dev documentation or run the Continue: Open Config command from the VSCode command palette. Inside the config.json, you’ll have a models array. To integrate an Ollama model, you need to add an entry for it. A minimal example looks like this: { "models": [ { "title": "Qwen 2.5 Coder 3b", "model": "qwen2.5-coder:3b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" } ] } Key Points: title: A human-friendly name for your model as it will appear in Continue’s model selection. model: The exact name of the model as listed by ollama ls. This includes any tags like :3b or :7b. provider: Set this to "ollama" so Continue knows to route prompts to the Ollama backend. apiBase/v1: This must point to Ollama’s API endpoint for generating responses. By default, Ollama listens on http://localhost:11434/api/generate. Make sure this is included exactly as shown. You can add as many models as you like by including multiple objects in the models array, for example: { "models": [ { "title": "Qwen 2.5 Coder 3b", "model": "qwen2.5-coder:3b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" }, { "title": "Llama 2 7B", "model": "llama-2-7b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" } ] } 4. Loading Models into Ollama Option A: Pulling Models from a Remote Source If a model is hosted in a repository or by Ollama itself, you can pull it directly: ollama pull qwen2.5-coder:3b This downloads the model files into Ollama’s directory. Once pulled, you can list it with ollama ls and add it to config.json. Option B: Loading a Local GGUF Model If you have a GGUF model file on your local machine (for example, my-model.gguf), you can integrate it with Ollama by creating a custom model YAML file that tells Ollama how to load it. Ollama’s documentation details this process, but it typically looks like: Create a model YAML file (e.g. my-model.yaml) in your Ollama models directory (commonly ~/.ollama/models/): name: my-local-model model: /path/to/my-model.gguf Once you have the YAML file in place, run: ollama import my-local-model.yaml This makes Ollama aware of the model. After importing, you can verify it’s recognized: ollama ls You should see my-local-model listed. Add the model to Continue’s config.json: { "models": [ { "title": "My Local Model", "model": "my-local-model", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" } ] } 5. Using the Models in VSCode with Continue.dev After editing config.json, restart Visual Studio Code or run Continue: Reload command from the command palette if available. Open the Continue.dev panel (usually on the sidebar or by using the Continue: Open command). Select the desired model from the model dropdown at the top of the Continue panel. Start interacting with the model. Your queries and code completions should now route through Ollama’s locally hosted model. 6. Troubleshooting Connection Issues: If Continue can’t reach Ollama, verify the apiBase/v1 URL and port. The default should be http://localhost:11434/api/generate unless you changed Ollama’s default port. Missing Models: If a model doesn’t show up, verify it’s listed by ollama ls and that you spelled it correctly in config.json. File Permissions: On some systems, ensure you have the correct file permissions for the .continue directory and the Ollama model directories. Summary: To integrate Ollama with Continue.dev in VSCode, you need to edit your config.json to include a model entry pointing to Ollama’s apiBase/v1 endpoint and referencing the model’s name exactly as Ollama recognizes it. You can load models by pulling them with ollama pull or importing a local GGUF file via a model YAML. After configuration, you can switch between any models you’ve added directly from Continue.dev’s interface in VSCode. © 2024 Daniel Kliewer. All rights reserved.

**Title:** Using Continue.dev with Ollama for Local LLMs in VSCode Hey everyone, I recently figured out how to set up Continue.dev with Ollama to run local LLMs right inside VSCode, and I thought I’d share a quick guide since I couldn’t find a straightforward tutorial out there. **The Basics:** - **Requirements:** You’ll need VSCode with the Continue.dev extension, and Ollama installed and running on your machine. Ollama uses a local server (usually at `http://localhost:11434`), so no external API calls are needed. - **Why Do This?:** If you want to rely on local compute rather than paying for OpenAI or Anthropic tokens, this setup lets you integrate local models directly into your coding workflow. You can load models you’ve pulled from a repo or integrate models stored locally as GGUF files, giving you full flexibility over what LLM you’re running. **Step-by-Step Setup:** 1. **Run Ollama:** Make sure Ollama is up and running. You can verify by hitting `http://localhost:11434/version` or by running `ollama ls` to see which models are available. 2. **Check Your Models:** Use `ollama ls` to see what models are currently managed by Ollama. Models often look like `modelName:variant`, e.g., `qwen2.5-coder:3b`. 3. **Edit Continue.dev Config:** - Find your `config.json` for Continue.dev (often in `~/.continue/config.json`). - Add an entry for your Ollama model. For example: ```json { "models": [ { "title": "Qwen 2.5 Coder 3b", "model": "qwen2.5-coder:3b", "provider": "ollama", "apiBase/v1": "http://localhost:11434/api/generate" } ] } ``` - This tells Continue.dev to send requests to Ollama’s local endpoint. You can add multiple models if you like. 4. **Loading Models into Ollama:** - **Pulling a remote model:** `ollama pull qwen2.5-coder:3b` (adjust model name as needed) - **Local GGUF files:** Create a `.yaml` file in your Ollama models directory that points to your `.gguf` file, then run `ollama import` on it. After that, just reference it by name in `config.json`. 5. **Use It in VSCode:** - Restart VSCode or run `Continue: Reload` from the command palette. - Open the Continue.dev panel, select your newly added model, and start coding with it. Your completions and prompts now run on your local LLM through Ollama. **Troubleshooting:** - If you can’t connect, double-check the `apiBase/v1` URL and port in `config.json`. - Make sure the model name in `config.json` matches exactly what `ollama ls` shows. - Check file permissions on your `.continue` directory or Ollama’s model directories if you run into issues. **Why This Matters:** This setup lets you save money on API calls and keeps your workflow offline if you want. While you can still use paid keys for better results, the local approach is a great way to avoid recurring costs—especially if you’re considering investing in better hardware to handle larger models. Hope this helps anyone looking to run their own local LLM inside VSCode using Continue.dev and Ollama. If you run into any issues or have suggestions, drop a comment!