Using GGUF with Ollama

I want to use https://huggingface.co/unsloth/Qwen3.5-27B-GGUF?show_file_info=Qwen3.5-27B-Q4_K_M.gguf with https://github.com/ollama/ollama/tree/v0.17.1-rc2

Here’s how you can **use the Hugging Face GGUF model `Qwen3.5-27B-GGUF` (especially the `Q4_K_M` file) with Ollama v0.17.1-rc2** — both the easy “run directly from Hugging Face” method and the import method: citeturn0search0turn0search2 --- ## ✅ 1. Run the model *directly* from Hugging Face (easiest) Ollama now supports running **GGUF models hosted on the Hugging Face Hub directly** without manually downloading or creating a modelfile. citeturn0search0 ### Steps 1. Make sure you have **Ollama v0.17.1-rc2 installed** and updated — this release includes support for running arbitrary GGUF models from HF. 2. In your terminal, run: ```bash ollama run hf.co/unsloth/Qwen3.5-27B-GGUF ``` ✔ That will automatically download the GGUF files and start the model. ✔ If there are multiple quant files, Ollama will pick the default quant (usually **Q4_K_M**). ✔ If you want to explicitly choose the `Q4_K_M` quant, you can specify it: ```bash ollama run hf.co/unsloth/Qwen3.5-27B-GGUF:Q4_K_M ``` You can now interact with the model, e.g.: ```bash ollama run hf.co/unsloth/Qwen3.5-27B-GGUF "Hello, what’s up?" ``` --- ## 👍 2. Import the GGUF locally (alternative) If you want the model **locally on disk** and registered under a custom name: ### a) Download the GGUF file manually You can either: - Download it from the Hugging Face UI, or - Use `git lfs` or a tool like `huggingface-cli` to fetch `Qwen3.5-27B-Q4_K_M.gguf` locally. ### b) Create a local Ollama model Put the GGUF file in a folder with a `Modelfile` like this: ``` my-qwen3.5/ ├── Modelfile └── Qwen3.5-27B-Q4_K_M.gguf ``` And **Modelfile** contents: ```dockerfile FROM ./Qwen3.5-27B-Q4_K_M.gguf ``` Then run: ```bash ollama create qwen3.5-local -f ./my-qwen3.5/Modelfile ``` Now test it: ```bash ollama run qwen3.5-local "Hello!" ``` This way, the model stays local and has a friendly name you can reference. citeturn0search2 📌 **Note:** Once imported, Ollama stores the GGUF weights in its blobs directory, so you *can* delete the original `.gguf` if you don’t need the separate copy anymore (but keep the Ollama model entry). citeturn0reddit27 --- ## 💡 Tips & Notes - **No conversion needed:** With recent Ollama versions, you usually *don’t need to convert anything* — just `ollama run` directly on the HF repo works. citeturn0search0 - **SSH keys for private repos:** If your model repo is private, add your local Ollama SSH key to your Hugging Face account first. citeturn0reddit24 - **Model architecture support:** Make sure your Ollama build supports Qwen-style architectures — update if necessary (newer builds have broader GGUF support). citeturn0reddit31 --- ## 🧪 Example usage **Interactive run:** ```bash ollama run hf.co/unsloth/Qwen3.5-27B-GGUF "Explain quantum computing in simple terms." ``` **API call via Ollama server:** ```bash curl http://localhost:11434/api/generate \ -H "Content-Type: application/json" \ -d '{ "model": "hf.co/unsloth/Qwen3.5-27B-GGUF", "prompt": "Write a short story about space travel.", "stream": false }' ``` --- If you want, I can also show a **sample config for using this model with a specific prompt format or toolchain (Python/JS)**—just tell me which environment you’re using!

first I need to install https://github.com/ollama/ollama/tree/v0.17.1-rc2

To install **entity["organization","Ollama","llm runtime"]** at the specific tag **entity["software","Ollama v0.17.1-rc2","github release tag"]**, you have three clean options. Since you're on macOS (Apple Silicon), I’ll prioritize that path. --- # ✅ Option 1 — Install from Prebuilt Release (Easiest) ### 1️⃣ Download the release binary Go to the GitHub releases page for: **Ollama → v0.17.1-rc2** Download the macOS ARM64 `.zip` or `.pkg`. If you prefer terminal: ```bash curl -L https://github.com/ollama/ollama/releases/download/v0.17.1-rc2/ollama-darwin-arm64.zip -o ollama.zip unzip ollama.zip ``` Then move it into your PATH: ```bash sudo mv ollama /usr/local/bin/ ``` Verify: ```bash ollama --version ``` You should see: ``` ollama version 0.17.1-rc2 ``` --- # ✅ Option 2 — Build From Source (More Control) Since you like working close to the metal and local models, this may be your preferred route. ### 1️⃣ Install dependencies ```bash brew install go cmake ``` Check Go version: ```bash go version ``` (Needs Go ≥ 1.22 typically.) --- ### 2️⃣ Clone and checkout the exact tag ```bash git clone https://github.com/ollama/ollama.git cd ollama git checkout v0.17.1-rc2 ``` --- ### 3️⃣ Build ```bash go build . ``` After it builds: ```bash ./ollama --version ``` If good: ```bash sudo mv ollama /usr/local/bin/ ``` --- # ✅ Option 3 — If You Already Have Ollama Installed If you previously installed Ollama via Homebrew: ```bash brew uninstall ollama ``` Then install the release manually (Option 1). Homebrew will not install release candidates. --- # 🔎 After Installation Start the server: ```bash ollama serve ``` In another terminal: ```bash ollama run hf.co/unsloth/Qwen3.5-27B-GGUF:Q4_K_M ``` Which references: - Model repo: **entity["organization","Hugging Face","ai model hosting platform"]** - Model: **entity["ai_model","Qwen3.5-27B-GGUF","unsloth quantized model"]** - Quant: `Q4_K_M` (~4-bit quantized, good balance of RAM vs quality) --- # ⚠️ Important for Your Machine 27B Q4_K_M needs roughly: - ~16–20GB RAM available - Apple Silicon unified memory preferred If you’re on 16GB total RAM, it will run but may swap. If you're on 32GB+, you’re good. --- If you'd like, tell me: - Your RAM amount - Whether you want GPU acceleration (Metal) - Whether you plan to use this with VSCode + Cline And I’ll tune the setup exactly for your workflow.

= gpt2 llama_model_loader: - kv 32: tokenizer.ggml.pre str = qwen35 llama_model_loader: - kv 33: tokenizer.ggml.tokens arr[str,248320] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 34: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 35: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 36: tokenizer.ggml.eos_token_id u32 = 248046 llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 248044 llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {%- set image_count = namespace(value... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 353 tensors llama_model_loader: - type q8_0: 96 tensors llama_model_loader: - type q4_K: 263 tensors llama_model_loader: - type q5_K: 96 tensors llama_model_loader: - type q6_K: 43 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 15.58 GiB (4.98 BPW) llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35' llama_model_load_from_file_impl: failed to load model time=2026-02-26T12:43:17.170-06:00 level=INFO source=sched.go:473 msg="NewLlamaServer failed" model=/Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17 error="unable to load model: /Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17" [GIN] 2026/02/26 - 12:43:17 | 500 | 199.671875ms | 127.0.0.1 | POST "/api/chat" llama_model_load_from_file_impl: using device Metal (Apple M4 Pro) (unknown id) - 38338 MiB free llama_model_loader: loaded meta data with 42 key-value pairs and 851 tensors from /Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen35 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.sampling.top_k i32 = 20 llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000 llama_model_loader: - kv 4: general.sampling.temp f32 = 0.600000 llama_model_loader: - kv 5: general.name str = Qwen3.5-27B llama_model_loader: - kv 6: general.basename str = Qwen3.5-27B llama_model_loader: - kv 7: general.quantized_by str = Unsloth llama_model_loader: - kv 8: general.size_label str = 27B llama_model_loader: - kv 9: general.license str = apache-2.0 llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-2... llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth llama_model_loader: - kv 12: general.tags arr[str,1] = ["image-text-to-text"] llama_model_loader: - kv 13: qwen35.block_count u32 = 64 llama_model_loader: - kv 14: qwen35.context_length u32 = 262144 llama_model_loader: - kv 15: qwen35.embedding_length u32 = 5120 llama_model_loader: - kv 16: qwen35.feed_forward_length u32 = 17408 llama_model_loader: - kv 17: qwen35.attention.head_count u32 = 24 llama_model_loader: - kv 18: qwen35.attention.head_count_kv u32 = 4 llama_model_loader: - kv 19: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0] llama_model_loader: - kv 20: qwen35.rope.freq_base f32 = 10000000.000000 llama_model_loader: - kv 21: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 22: qwen35.attention.key_length u32 = 256 llama_model_loader: - kv 23: qwen35.attention.value_length u32 = 256 llama_model_loader: - kv 24: qwen35.ssm.conv_kernel u32 = 4 llama_model_loader: - kv 25: qwen35.ssm.state_size u32 = 128 llama_model_loader: - kv 26: qwen35.ssm.group_count u32 = 16 llama_model_loader: - kv 27: qwen35.ssm.time_step_rank u32 = 48 llama_model_loader: - kv 28: qwen35.ssm.inner_size u32 = 6144 llama_model_loader: - kv 29: qwen35.full_attention_interval u32 = 4 llama_model_loader: - kv 30: qwen35.rope.dimension_count u32 = 64 llama_model_loader: - kv 31: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 32: tokenizer.ggml.pre str = qwen35 llama_model_loader: - kv 33: tokenizer.ggml.tokens arr[str,248320] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 34: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 35: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 36: tokenizer.ggml.eos_token_id u32 = 248046 llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 248044 llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {%- set image_count = namespace(value... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 353 tensors llama_model_loader: - type q8_0: 96 tensors llama_model_loader: - type q4_K: 263 tensors llama_model_loader: - type q5_K: 96 tensors llama_model_loader: - type q6_K: 43 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 15.58 GiB (4.98 BPW) llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35' llama_model_load_from_file_impl: failed to load model time=2026-02-26T12:43:18.379-06:00 level=INFO source=sched.go:473 msg="NewLlamaServer failed" model=/Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17 error="unable to load model: /Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17" [GIN] 2026/02/26 - 12:43:18 | 500 | 206.0015ms | 127.0.0.1 | POST "/api/chat" llama_model_load_from_file_impl: using device Metal (Apple M4 Pro) (unknown id) - 38338 MiB free llama_model_loader: loaded meta data with 42 key-value pairs and 851 tensors from /Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen35 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.sampling.top_k i32 = 20 llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000 llama_model_loader: - kv 4: general.sampling.temp f32 = 0.600000 llama_model_loader: - kv 5: general.name str = Qwen3.5-27B llama_model_loader: - kv 6: general.basename str = Qwen3.5-27B llama_model_loader: - kv 7: general.quantized_by str = Unsloth llama_model_loader: - kv 8: general.size_label str = 27B llama_model_loader: - kv 9: general.license str = apache-2.0 llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-2... llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth llama_model_loader: - kv 12: general.tags arr[str,1] = ["image-text-to-text"] llama_model_loader: - kv 13: qwen35.block_count u32 = 64 llama_model_loader: - kv 14: qwen35.context_length u32 = 262144 llama_model_loader: - kv 15: qwen35.embedding_length u32 = 5120 llama_model_loader: - kv 16: qwen35.feed_forward_length u32 = 17408 llama_model_loader: - kv 17: qwen35.attention.head_count u32 = 24 llama_model_loader: - kv 18: qwen35.attention.head_count_kv u32 = 4 llama_model_loader: - kv 19: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0] llama_model_loader: - kv 20: qwen35.rope.freq_base f32 = 10000000.000000 llama_model_loader: - kv 21: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 22: qwen35.attention.key_length u32 = 256 llama_model_loader: - kv 23: qwen35.attention.value_length u32 = 256 llama_model_loader: - kv 24: qwen35.ssm.conv_kernel u32 = 4 llama_model_loader: - kv 25: qwen35.ssm.state_size u32 = 128 llama_model_loader: - kv 26: qwen35.ssm.group_count u32 = 16 llama_model_loader: - kv 27: qwen35.ssm.time_step_rank u32 = 48 llama_model_loader: - kv 28: qwen35.ssm.inner_size u32 = 6144 llama_model_loader: - kv 29: qwen35.full_attention_interval u32 = 4 llama_model_loader: - kv 30: qwen35.rope.dimension_count u32 = 64 llama_model_loader: - kv 31: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 32: tokenizer.ggml.pre str = qwen35 llama_model_loader: - kv 33: tokenizer.ggml.tokens arr[str,248320] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 34: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 35: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 36: tokenizer.ggml.eos_token_id u32 = 248046 llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 248044 llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {%- set image_count = namespace(value... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 353 tensors llama_model_loader: - type q8_0: 96 tensors llama_model_loader: - type q4_K: 263 tensors llama_model_loader: - type q5_K: 96 tensors llama_model_loader: - type q6_K: 43 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 15.58 GiB (4.98 BPW) llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35' llama_model_load_from_file_impl: failed to load model time=2026-02-26T12:43:20.596-06:00 level=INFO source=sched.go:473 msg="NewLlamaServer failed" model=/Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17 error="unable to load model: /Users/danielkliewer/.ollama/models/blobs/sha256-728960e4dda52d4f2af5bee09b2cbe86addfa93220fe9324bfac9dc727605c17" [GIN] 2026/02/26 - 12:43:20 | 500 | 211.804625ms | 127.0.0.1 | POST "/api/chat"

Good — this log tells us exactly what’s wrong. The key line is: ``` error loading model architecture: unknown model architecture: 'qwen35' ``` Your GGUF file: **entity["ai_model","Qwen3.5-27B","large language model"]** has: ``` general.architecture = qwen35 ``` But your installed **entity["organization","Ollama","llm runtime"]** build (v0.17.1-rc2) does not yet support the `qwen35` architecture. So the model loads, parses metadata, then fails because the backend (llama.cpp inside Ollama) doesn’t recognize the new architecture tag. --- # 🚨 Why This Is Happening `Qwen3.5` is newer than: - Qwen 2 - Qwen 2.5 It introduced: - Hybrid SSM + attention layers - New metadata namespace `qwen35.*` Your Ollama build includes a llama.cpp version **older than the commit that added qwen35 support**. So the model file is valid. Your runtime is just too old. --- # ✅ The Fix You need a **newer Ollama build** — not v0.17.1-rc2. That release candidate predates Qwen3.5 support. --- # 🔥 Solution 1 (Recommended): Install Latest Ollama Instead of pinning to v0.17.1-rc2: ```bash brew uninstall ollama brew install ollama ``` Or download the latest release from GitHub. Then verify: ```bash ollama --version ``` You want something newer than the RC build — ideally a version that mentions Qwen3 or recent llama.cpp sync. Then run: ```bash ollama run hf.co/unsloth/Qwen3.5-27B-GGUF:Q4_K_M ``` --- # 🧠 Why This Matters Your log shows: ``` qwen35.context_length = 262144 ``` This is a **262k context model**. Older llama.cpp builds: - Don’t support Qwen3.5 - Don’t support its SSM layers - Don’t recognize `general.architecture = qwen35` So even though Metal initializes fine on your M4 Pro (38GB free — you're totally fine on RAM), the architecture mismatch kills it. --- # 🛠 If You *Must* Use v0.17.1-rc2 Then you cannot use Qwen3.5. You would need: - Qwen2.5 GGUF - Or a model whose `general.architecture` is `qwen2` or `llama` For example: - Qwen2.5-32B GGUF - Llama 3 - Mistral variants --- # 🎯 What I Recommend For You Specifically Given: - You're on Apple Silicon - You like long context - You run local tooling + VSCode I would: 1. Install latest Ollama 2. Run Qwen3.5-27B Q4_K_M 3. Set context manually lower (like 32k–64k) unless you really need 262k Because 262k context will eat VRAM + slow generation significantly. --- # 🔍 Quick Sanity Check Run: ```bash ollama --version ``` Tell me what it outputs. If it's still 0.17.1-rc2, that’s the problem. --- If you'd like, I can also explain: - Why Qwen3.5 uses SSM layers - Why that breaks older llama.cpp builds - Or how to compile Ollama against the newest llama.cpp manually You're very close — this is just a version mismatch.

I want to completely uninstall ollama

Got it — let’s fully uninstall **entity["organization","Ollama","llm runtime"]** from your Mac so no traces are left. Here’s a complete cleanup plan: --- ## 1️⃣ Stop Ollama services If Ollama is running: ```bash ollama stop ``` Or check for running processes: ```bash ps aux | grep ollama ``` Kill any processes: ```bash kill -9 <PID> ``` --- ## 2️⃣ Remove the main executable Depending on how you installed it: ### If installed via Homebrew: ```bash brew uninstall ollama ``` Check with: ```bash which ollama ``` If it still shows a path (e.g., `/usr/local/bin/ollama`), remove manually: ```bash sudo rm -f /usr/local/bin/ollama ``` ### If installed from GitHub ZIP / pkg: Find where you put `ollama` and delete it: ```bash sudo rm -f /usr/local/bin/ollama sudo rm -f /usr/bin/ollama # unlikely, just in case ``` --- ## 3️⃣ Remove Ollama model blobs & configs Ollama keeps local models, cache, and settings in: ``` ~/.ollama ``` Remove the entire directory: ```bash rm -rf ~/.ollama ``` This deletes: - All downloaded GGUF models - Ollama database - Logs - Custom Modelfiles --- ## 4️⃣ Optional: Remove shell completions or config files Check if you added Ollama to shell profiles: ```bash grep -i ollama ~/.zshrc ~/.bash_profile ~/.bashrc ~/.zprofile ``` If anything shows up, remove those lines. Reload your shell: ```bash source ~/.zshrc ``` --- ## 5️⃣ Verify ```bash ollama --version ``` You should see: ``` command not found: ollama ``` --- After this, your system will be completely free of Ollama. If you want, I can give you a **one-line command to fully uninstall Ollama including models, configs, and binaries**, so it’s totally clean in one shot. Do you want me to do that?