Instructions to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF",
	filename="Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-Q4_K_S.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S
# Run inference directly in the terminal:
llama cli -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S
# Run inference directly in the terminal:
llama cli -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S
# Run inference directly in the terminal:
./llama-cli -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Use Docker

docker model run hf.co/andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

LM Studio
Jan

vLLM

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Ollama
How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with Ollama:
```
ollama run hf.co/andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S
```

Unsloth Studio

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF to start chatting

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with Docker Model Runner:
```
docker model run hf.co/andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S
```

Lemonade

How to use andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF:Q4_K_S

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-Hebrus-GGUF-Q4_K_S

List all available models

lemonade list

Hebrus Studio — Metal first. SSD when memory ends.

Qwen3.6-35B-A3B — Hebrus GGUF releases

This repository keeps the versioned Qwen3.6-35B-A3B releases for the Hebrus inference engine. Hebrus began as a fork of Salvatore Sanfilippo's antirez/ds4. Historical DS4 names remain only in filenames and serialized compatibility fields. Its routed MoE weights are embedded in the GGUF as a self-describing ds4.expert_major.v2 store; no sidecar, runtime repack, or second copy of the routed bank is required.

Compatibility: existing filenames, the dsbox.json manifest, and the ds4.expert_major.v2 marker remain unchanged serialized contracts.

Versions

File	Status	Runtime
`Qwen3.6-35B-A3B-DS4-ExpertMajor-v2-MLX-Affine4-G64.gguf`	Current	Hebrus `main` at `73a332f` or newer
`Qwen3.6-35B-A3B-DS4-ExpertMajor-v2-Q4_K_S.gguf`	Retired v2/Q4 evidence	Rejected by the current affine-only runtime
`Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-Q4_K_S.gguf`	Legacy migration evidence	Retired Hebrus v1 runtime only
`Qwen3.6-35B-A3B-ds4-Q4_K_S.gguf`	Legacy canonical converter input	Generic GGUF tooling; rejected by current Hebrus inference

Hebrus Studio and the root dsbox.json select only the affine4/group-64 ExpertMajor v2 artifact. The older files are kept so existing links and reproducibility records remain valid; they are not fallbacks and must not be auto-selected by current Hebrus.

This is a Hebrus-native GGUF extension, not a portable GGUF layout. Generic GGUF loaders cannot reconstruct or execute its routed weights. Canonical Qwen GGUFs, ExpertMajor v1 files, sidecar layouts, and non-Metal inference are not supported by the qualified Hebrus runtime for this model. Use the original model or a canonical quantization with another runtime when generic-loader compatibility is required.

Artifact

Property	Value
Base model	`Qwen/Qwen3.6-35B-A3B`
File	`Qwen3.6-35B-A3B-DS4-ExpertMajor-v2-MLX-Affine4-G64.gguf`
Size	20,808,566,880 bytes (19.38 GiB)
SHA-256	`dd17266185833a9f05531ce366fd7284ddca1ed64aa3dcf06e321e8c72c9ea3d`
Container	GGUF v3 with one embedded `ds4.expert_major.v2` store
Routed geometry	40 routed layers, 256 experts per layer
Routed payload	18,119,393,280 bytes
Routed storage	MLX-compatible affine 4-bit groups: 32 packed bytes + BF16 scale + BF16 bias per 64 weights
Qualified hardware	Apple Silicon Metal, at least 16 GiB unified memory

The affine conversion changes the physical representation consumed by Hebrus' resident and SSD-streamed Metal kernels without introducing a sidecar or a new container version. The same GGUF runs in AUTO resident or SSD mode. It is not a portable llama.cpp, MLX, Ollama, or generic GGUF artifact.

Run with Hebrus

Use andreaborio/hebrus main at commit 73a332fef82a0bcdd567d17e0de17aa004cad85d or a compatible descendant:

git clone https://github.com/andreaborio/hebrus.git
cd hebrus
make -j

./hebrus \
  -m /absolute/path/to/Qwen3.6-35B-A3B-DS4-ExpertMajor-v2-MLX-Affine4-G64.gguf \
  --ctx 8192

The embedded ExpertMajor v2 store, Metal backend, and residency policy are selected automatically. No Qwen enable flag, sidecar path, pack path, or hash environment variable is needed. AUTO uses SSD streaming when the resident budget does not fit; the admission floor is 16 GiB unified memory.

Correctness and measured performance

Final qualification on an Apple M5 Pro 64 GiB measured:

Prompt frontier	Resident prefill	Forced-SSD prefill
128	523.72 t/s	210.70 t/s
2,048	1,661.18 t/s	551.57 t/s
8,192	1,421.90 t/s	269.10 t/s
32,768	877.34 t/s	83.69 t/s

The 128+16 lane measured 57.43 t/s resident decode and 23.94 t/s with a cold 321-expert SSD cache. Both modes produced the same 16 greedy token IDs. At 32K, adaptive macro-prefill reduced expert reads from 60.776 GiB to 16.875 GiB (-72.24%) with a -0.63% throughput delta. Actual graph allocation also passed the model-declared 262,144-token maximum. These are bounded local measurements; context, output length, thermals, storage, and memory pressure affect results.

Current limitations

Apple Silicon Metal is the only qualified inference backend.
At least 16 GiB unified memory is required; constrained hosts use SSD streaming.
The file requires Hebrus commit 73a332fef82a0bcdd567d17e0de17aa004cad85d or a compatible descendant implementing ds4.expert_major.v2 for qwen35moe.
Canonical Qwen GGUF inference, the retired ExpertMajor v2 Q4 payload, ExpertMajor v1, sidecars, and non-Metal inference are deliberately rejected.
A generic GGUF loader can inspect the outer container but cannot execute the opaque routed store.

License and attribution

The base model is released by the Qwen team under the Apache 2.0 license. See the upstream model card for model architecture, training, safety, license, and attribution details.

Downloads last month: 720

GGUF

Model size

21B params

Architecture

qwen35moe

Hardware compatibility

4-bit

View +1 variant

Model tree for andreaborio/Qwen3.6-35B-A3B-Hebrus-GGUF

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(657)

this model