Qwen3.5 Claude Code Integration

I have qwen3.5 running on http://0.0.0.0:8080 with llama.cpp how do I get this to work with claude code if I am using it in VSCode

If you have **Qwen 3.5 running via llama.cpp at `http://0.0.0.0:8080`**, the key question is: 👉 **Does your llama.cpp server expose an OpenAI-compatible API?** Because **entity["company","Anthropic","ai company"] Claude Code** expects Anthropic’s API — and won’t natively talk to a raw llama.cpp endpoint unless you bridge it. So you have two realistic paths. --- # ✅ Option 1 (Recommended): Run llama.cpp in OpenAI-compatible mode If you started llama.cpp like this: ```bash ./server -m qwen.gguf --port 8080 ``` You need to make sure it’s running the **OpenAI-compatible server**. Start it like: ```bash ./server \ -m qwen.gguf \ --port 8080 \ --host 0.0.0.0 \ --api-key sk-local \ --chat-template chatml ``` Then test: ```bash curl http://localhost:8080/v1/models ``` If you see JSON output, you're good. --- # ✅ Option 2: Point VSCode / Claude Code to your local endpoint Now the problem: **Claude Code is hardwired for Anthropic’s API.** It does NOT natively support custom endpoints unless you spoof the API. So what you do instead: ## 🔥 Use OpenAI-Compatible Mode in VSCode In VSCode settings (or `.env`): ```bash OPENAI_API_BASE=http://localhost:8080/v1 OPENAI_API_KEY=sk-local OPENAI_MODEL=qwen ``` If you're using: - Continue.dev - Cline - CodeGPT - Cursor (OpenAI mode) - Roo Code - Any OpenAI-compatible extension It will work immediately. --- # 🚫 Why Claude Code Won’t Directly Work Claude Code is built specifically for: ``` https://api.anthropic.com/v1/messages ``` Your llama.cpp server is: ``` http://localhost:8080/v1/chat/completions ``` Different format. Different auth. Different protocol. --- # 🧠 The Clean Architecture (What I Would Do If I Were You) Given: - You already use Ollama - You prefer local models - You dislike proprietary lock-in - You want long context I would: ### Option A — Use Cline in VSCode Cline lets you configure: ```json { "apiBaseUrl": "http://localhost:8080/v1", "apiKey": "sk-local", "modelId": "qwen" } ``` Done. --- ### Option B — Use OpenWebUI as Proxy Run: ```bash docker run -p 3000:8080 ghcr.io/open-webui/open-webui:main ``` Point it to llama.cpp backend. Then use OpenAI-compatible endpoint from there. --- # 🧬 If You REALLY Want Claude Code to Work You need a proxy layer that: 1. Accepts Anthropic format 2. Translates to OpenAI format 3. Sends to llama.cpp 4. Translates back There are projects like: - litellm proxy - custom FastAPI wrapper - LangServe bridge But it’s overkill unless you specifically want Claude Code UI. --- # 🎯 My Direct Advice For You You’re building: - local-first infra - agent orchestration - VSCode automation - long context coding Claude Code is not aligned with that philosophy. Use: - Cline - Continue.dev - or your own OpenAI wrapper You already prefer local control. This keeps you sovereign. --- If you want, tell me: - Are you using plain llama.cpp? - Or is this through Ollama? - What VSCode extension specifically? And I’ll give you exact config JSON tailored to your stack.

{"message":"[object Object]","modelId":"Qwen3.5-27B-Q4_K_M","providerId":"ollama","details":{"message":"File Not Found","type":"not_found_error","code":404}}

That error is coming from **Ollama**, not VSCode. ``` "type": "not_found_error", "code": 404, "message": "File Not Found" ``` When Ollama returns 404 for a model, it almost always means: > 🔴 The model name you passed does not exist in Ollama’s registry. --- # 🔎 What’s Wrong You passed: ``` Qwen3.5-27B-Q4_K_M ``` But Ollama does **not** use GGUF filenames as model IDs. Ollama model names look like: ``` qwen3.5:27b qwen3.5:14b qwen3.5:7b ``` Not: ``` Qwen3.5-27B-Q4_K_M ``` That’s a GGUF file name, not an Ollama model tag. --- # ✅ Step 1 — Check What Ollama Actually Has Run: ```bash ollama list ``` You’ll see something like: ``` qwen3.5:27b llama3:8b codellama:13b ``` Use exactly what appears there. Case-sensitive. --- # ✅ Step 2 — Fix Your VSCode Config If you're using a JSON config like: ```json { "modelId": "Qwen3.5-27B-Q4_K_M", "providerId": "ollama" } ``` Change it to something like: ```json { "modelId": "qwen3.5:27b", "providerId": "ollama" } ``` --- # ⚠️ If `ollama list` Shows Nothing Then you never pulled the model. Run: ```bash ollama pull qwen3.5:27b ``` Or if it’s custom: ```bash ollama create qwen-custom -f Modelfile ``` Then use: ``` qwen-custom ``` --- # 🧠 Important Distinction (Your Current Setup) Earlier you said: > “I have qwen3.5 running on http://0.0.0.0:8080 with llama.cpp” But now your error says: ``` "providerId": "ollama" ``` So you're mixing two systems: | System | Model Naming | Port | |--------|--------------|------| | llama.cpp | GGUF filename | 8080 | | Ollama | `model:tag` | 11434 | If you’re using Ollama in VSCode, it talks to: ``` http://localhost:11434 ``` Not your llama.cpp server on 8080. --- # 🚀 Clean Fix Options ### Option A — Use Ollama Properly - Stop llama.cpp - Run: ```bash ollama run qwen3.5:27b ``` - Use that model name in VSCode ### Option B — Use llama.cpp Instead Change provider to OpenAI-compatible instead of `ollama`: ```json { "providerId": "openai", "apiBaseUrl": "http://localhost:8080/v1", "apiKey": "sk-local", "modelId": "qwen" } ``` --- # 🎯 What I Think Is Happening You: - Installed llama.cpp manually - Also installed Ollama - VSCode extension is configured for Ollama - But the model only exists in llama.cpp So Ollama says: > “I don’t know that model.” → 404 --- Tell me: - What does `ollama list` output? - Are you trying to use Cline? Continue? Claude Code? - Do you want to run llama.cpp OR Ollama? And I’ll give you the exact config that will work.