Key Takeaways
- Saluki 27B is an Apache 2.0 licensed 2-bit GGUF quantization model released by Conway Research’s Underdog team, compressing the Qwen3.8-27B [Vendor Designation] BF16 original from 54GB down to 7.89GB. Measured on the same harness across 120 Underdog Bench tasks, Saluki 27B scored 88 while the full model scored 84, surpassing the original. On BFCL v4 parallel tool calling (100 tasks), Saluki 27B scored 42 versus 35 for the full model, achieving 120% retention. On SWE-bench Verified (50 issues), Saluki 27B solved 30 versus 33 for the full model, showing a slight deficit. The 9-benchmark average retention is 96%, with the largest loss concentrated in mathematical/competitive reasoning—AIME 2025 dropped from 96.7 to 79.2 [Vendor Self-Reported / Unverified]. The model uses a 64-layer architecture mixing Gated DeltaNet linear attention with gated attention, and its native context is 262,144 tokens. The quantization stack is a 3-layer structure: ISTA-DASLab’s Qwen3.8-27B [Vendor Designation]-GSQ-RCO-GGUF (IQ2_XS, 8.4GB, ~2.34 bpw) as the base, with Underdog’s proprietary pass (IQ2-mix, imatrix tags) layered on top. Optional 629MB or 928MB vision adapters are also distributed. A same-class alternative, PrismML Bonsai 2, achieved 98.2% retention on 14 reasoning-mode benchmarks and scored 95.00 on AIME25, giving it an edge in math, but it requires its own llama.cpp fork and does not run on stock llama.cpp. Saluki 27B runs with full GPU offload on stock llama.cpp, Ollama, and LM Studio.
A 27B-class LLM has been compressed into a file under 8GB, reaching the point where it can run on an ordinary laptop. Quantization loss appears asymmetrically across task domains—it beats the original on tool calling and parallel function calling but drops 12–18 points on competitive math and multi-step reasoning. Therefore, when building on-device agents, it is necessary to separate quantization-friendly workloads (function calling, format compliance, short reasoning) from quantization-unfriendly workloads (long-form multi-step math, complex chains of thought) and route them accordingly.
Table of Contents
Introduction: The On-Device Pressure of the Agent Era
Since the start of 2026, demand for local LLM agents has exploded, and concrete attempts to fit 27B-class models into the 8GB VRAM of an ordinary laptop have accelerated. Saluki 27B is the most striking example of this trend: the original Qwen3.8-27B [Vendor Designation] BF16 at 54GB has been compressed into a single 7.89GB file through a 2-bit quantization stack. With 27B inference now possible via full GPU offload on a single RTX 4060 Laptop (8GB VRAM), precisely understanding the trade-offs of 2-bit quantization has become the starting point for on-device agent design.
Saluki 27B is a model released under the Apache 2.0 license by the Underdog team at Conway Research, and it runs on stock runtimes such as llama.cpp, Ollama, and LM Studio without any additional forks. This stands in clear contrast to same-class models that demand a separate inference engine. The artifact composition and imatrix logs published on the Underdog launch page make reproducing the same pipeline possible.
Saluki 27B Core Specifications
The backbone of Saluki 27B is identical to Qwen3.8-27B [Vendor Designation]. The central structure is a 64-layer hybrid attention. Gated DeltaNet linear attention and standard gated attention are interleaved at the layer level, combining the memory efficiency of linear attention with the accuracy of standard attention. The native context is 262,144 tokens, comfortably accommodating long codebases or multi-document RAG.
The quantization artifact is a single 7.89GB GGUF file. Converted to bits per weight, it sits at approximately 2.50 bpw, placing it in the IQ2_XS family—the lowest tier of the extreme-compression spectrum. Optional 629MB or 928MB vision adapters are distributed alongside, enabling multimodal workloads. The Apache 2.0 license permits commercial fine-tuning and redistribution alike, minimizing legal friction when adopting it as an agent backend.
Anatomy of the 3-Layer Quantization Stack
The 7.89GB artifact from Saluki 27B is not the product of a single algorithm. It passes through a 3-layer pipeline: Base → GSQ-RCO → IQ2-mix. The role of each layer is cleanly separated, making it possible to reapply the same pipeline to a custom fine-tuned model.
Layer 1 is ISTA-DASLab’s Qwen3.8-27B [Vendor Designation]-GSQ-RCO-GGUF. GSQ (Grid Scalar Quantization) learns per-tensor low-bit scalar grids to build non-uniform codes matched to weight distributions, and RCO (Rate-Constrained Optimization) determines per-tensor bit allocation under a fixed overall model-size budget. The base artifact is IQ2_XS at 8.4GB, 2.50 bpw.
Layer 2 is Underdog’s proprietary IQ2-mix pass. An importance matrix (imatrix) is computed, and a mixed-precision strategy is applied: loss-sensitive tensors are assigned higher bits, while insensitive ones are compressed more aggressively. At an effective ~2.34 bpw (down ~0.5GB from the 8.4GB IQ2_XS base), per-tensor bit allocation is rebalanced to pull the average loss down, and the resulting file is 0.5GB smaller than the base, at 7.89GB.
Layer 3 is the step that measures—via benchmarks—the asymmetry between quantization-friendly workloads (function calling, format compliance) and quantization-unfriendly workloads (long-form multi-step reasoning, math competitions), then delivers routing guidance to users. This is not a technical quantization step itself, but the Underdog Bench of 120 tasks itself plays the role of Layer 3 metadata.
Cross-Benchmark Analysis: A Map of Asymmetric Loss
The most interesting property of Saluki 27B is that loss appears asymmetrically across task domains. The 9-benchmark average retention measured on the same harness is 96%, but the variance by domain is large. On tool calling and function calling domains, Saluki 27B surpasses the original.
On Underdog Bench (120 tasks), Saluki 27B scored 88 while the full model scored 84. On BFCL v4 parallel tool calling (100 tasks), Saluki 27B scored 42 versus 35 for the full model, achieving 120% retention. Function calling and format compliance are task types that are robust to quantization, because they only require reproducing outputs already patterned during training.
In contrast, math and competitive reasoning domains show a clear deficit. On AIME 2025, Saluki 27B scored 79.2 versus 96.7 for the full model—a 17.5-point gap. On SWE-bench Verified (50 issues), Saluki 27B solved 30 versus 33 for the full model, trailing slightly. The pattern of the largest loss among the 9 benchmarks being concentrated in math and multi-step reasoning is consistent with the hypothesis that weight-precision loss under extreme compression propagates directly into chain-of-thought tasks that are sensitive to accumulated error.
| Benchmark [Vendor Self-Reported / Unverified] | Task Type | Saluki 27B | Qwen3.8-27B [Vendor Designation] BF16 | Retention |
|---|---|---|---|---|
| Underdog Bench (120) | General Agent | 88 | 84 | 104.8% |
| BFCL v4 (100) | Parallel Tool Calling | 42 | 35 | 120.0% |
| SWE-bench Verified (50) | Coding Agent | 30 | 33 | 90.9% |
| AIME 2025 | Math Competition | 79.2 | 96.7 | 81.9% |
| 9-Benchmark Average | Overall | – | – | 96.0% |
Comparison with Bonsai 2: Retention Advantage vs. Runtime Dependency
PrismML’s Bonsai 2 is frequently mentioned as a same-class alternative in the IQ2 family. Bonsai 2 achieved 98.2% retention on 14 reasoning-mode benchmarks, and on AIME25 it scored 95.00, giving it a higher math-domain retention than Saluki 27B. However, Bonsai 2 requires its own llama.cpp fork and does not run on stock llama.cpp. Adoption in production introduces a new variable: runtime dependency.
Saluki 27B runs with full GPU offload on stock llama.cpp, Ollama, and LM Studio without any conversion. This is because llama.cpp’s GGUF loader natively supports the IQ2 family. Combined with the ability to run a 27B model even on an 8GB VRAM card, standard runtime compatibility, and the Apache 2.0 license, Saluki 27B has the lowest barrier to entry among on-device agent backend candidates. Looking at retention and math performance alone, Bonsai 2 leads—but when integration cost is factored in, the picture changes.
Practical Setup Procedure: llama.cpp and Ollama
There are two fastest paths to running Saluki 27B locally. First, direct execution with stock llama.cpp. Second, one-stop serving via Ollama. Both paths end without any additional build or fork. Use the following commands to fetch the 7.89GB file and full-offload it onto an 8GB VRAM card.
# Build llama.cpp (CUDA backend example)
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j
# Download model (Hugging Face)
huggingface-cli download ConwayResearch/Underdog-Saluki-27B-1.0 \
--include "Underdog-Saluki-27B-1.0-IQ2-mix.gguf" --local-dir ./models
# 8GB VRAM full offload (-ngl 999: load all layers onto GPU)
./build/bin/llama-cli -m ./models/Underdog-Saluki-27B-1.0-IQ2-mix.gguf \
-c 8192 -ngl 999 --temp 0.7 --top-p 0.9 -n 512 \
-p "You are a tool-calling agent. Use the following tools: ..."
The Ollama path is even shorter. Write a Modelfile, register the GGUF file, then serve. If the function-calling schema is specified in the SYSTEM message, it can be used immediately as a BFCL v4 schema-compatible agent. Because it uses the same llama.cpp backend as-is, no extra flag is required for IQ2-mix recognition.
# Ollama Modelfile example
FROM ./models/Underdog-Saluki-27B-1.0-IQ2-mix.gguf
PARAMETER num_ctx 8192
PARAMETER num_gpu 999
PARAMETER temperature 0.7
SYSTEM """You are a function-calling agent. When the user requests an action, output a JSON object with 'name', 'arguments' fields following the BFCL v4 schema. Do not include explanations outside the JSON block when calling a tool."""
[Hardware Feasibility Note] On a single 8GB VRAM card (e.g., RTX 4060 Laptop), the 7.89GB weight footprint leaves virtually zero headroom. To run 8K context without OOM, users must apply KV cache quantization (--cache-type-k q8_0 --cache-type-v q8_0) or split initial layers across CPU. For stable full GPU offloading with 8K–32K contexts, 12GB–16GB VRAM (or Apple Silicon 16GB+ Unified Memory) is practically required.
Routing Patterns: Separating Quantization-Friendly and Unfriendly Workloads
Deploying Saluki 27B as a single backend for all tasks is not recommended. Because the asymmetric loss pattern is clear, a router that branches by task characteristic is needed. This pattern is already adopted by many enterprise agent frameworks.
Quantization-friendly workloads (Saluki 27B first) include: ① function calling and tool schema compliance, ② short reasoning, ③ strict format compliance, ④ classification and labeling, and ⑤ medium-length code auto-completion. Quantization-unfriendly workloads (route to full BF16 or higher-bit quantization) include: ① AIME/Olympiad-style math, ② multi-step chains of thought, ③ SWE-bench Verified-level agentic coding, and ④ long-form multi-document reasoning.
Placing Saluki 27B in the router’s low-cost, high-throughput slot and falling back to Qwen3.8-27B [Vendor Designation] BF16 or another high-bit model on math and complex-reasoning triggers is the cost-versus-performance sweet spot. Teams that have already entered the local LLM serving pipeline can review the 5-step on-device AI adoption guide alongside this and consider Saluki 27B as the default slot.
Design Principles for LLM Agents in the 2-Bit Era
The implication Saluki 27B presents is simple. In an era where a 27B-class model fits into an 8GB file, the center of gravity in agent design shifts from a single all-purpose model to a workload-branched router. 2-bit quantization does not produce uniform loss across all domains, and the fact that it can beat the full model on tool calling and format compliance points to a new direction for quantization research.
At the same time, stock llama.cpp compatibility and the Apache 2.0 license have substantially lowered the practical barrier to on-device deployment. The Saluki 27B model card publicly exposes imatrix calculation logs and quantization parameters, so re-quantization after custom fine-tuning can also be reproduced through the same pipeline. In an era where a single ordinary laptop runs a 27B agent, understanding the asymmetry of 2-bit quantization is itself a design competitive advantage.
What to Try Right Now
- Secure hardware such as an RTX 4060 Laptop (8GB) or an M2 Pro with 16GB+ Unified Memory, then build stock llama.cpp with the CUDA or Metal backend.
- Download the
Underdog-Saluki-27B-1.0-IQ2-mix.gguffile (7.89GB) forConwayResearch/Underdog-Saluki-27B-1.0from Hugging Face and verify full GPU offload with the-ngl 999 -c 8192options. - Specify the BFCL v4 function-calling schema as a SYSTEM prompt in the Ollama Modelfile, then measure accuracy against 10+ tool-calling scenarios.
- Feed the same prompt simultaneously to Saluki 27B and Qwen3.8-27B [Vendor Designation] BF16, measure the loss margin directly against 5 AIME 2025-style math problems, and set the routing trigger baseline.
- Add quantization-friendly/unfriendly branching logic to the router of the existing agent framework, so that function calls go to Saluki 27B and multi-step math falls back to a BF16 model.
Frequently Asked Questions
Q. Saluki 27B is 7.89GB (~2.34 bpw), so why is it 0.5GB smaller than the IQ2_XS base?
Underdog’s IQ2-mix pass uses the imatrix to compress loss-insensitive tensors more aggressively while assigning only slightly higher bits to sensitive ones, in a mixed-precision strategy. The average bpw is identical at 2.50, but the per-tensor distribution is rebalanced, shrinking the resulting file by 0.5GB.
Q. What is the recommended Unified Memory capacity for running Saluki 27B on an Apple Silicon MacBook?
At an 8K context, 16GB Unified Memory is stable. If the context is raised to 32K or more, 24GB or higher is recommended due to KV cache footprint. There are confirmed cases of the -ngl 999 -c 8192 combination working on M2 Pro 16GB models.
Q. How noticeable is the 17.5-point AIME 2025 loss in production agents?
It is not noticeable at the level of general coding assistance, document summarization, or API-calling agents. However, in workloads that require solving math olympiad-style problems or multi-step proof chains correctly, a clear accuracy drop occurs, so BF16 fallback routing is essential for such tasks.
Reference Source
This article was prepared after reviewing the following source: MarkTechPost — Meet the Underdog Saluki 27B: A 2-bit Qwen3.8-27B [Vendor Designation] That Beats the Original at Tool Calling

Leave a Reply