Filter by type:
Filter by tags:
qwen3.8-27b-imatrix-nvfp4-mtp
Qwen3.8-27B in cdiamond's iMatrix-calibrated mixed-precision NVFP4
build. Selected attention and feed-forward tensors retain higher
precision. Includes the F16 vision projector and enables the embedded
MTP draft head with one draft token per step.
Repository: localaiLicense: apache-2.0
underdog-saluki-27b-1.0-abliterated-mtp
Kujira's abliterated adaptation of Underdog Saluki 27B 1.0 uses mixed
IQ2 weights with an embedded MTP head. This text-only llama.cpp build
enables multi-token prediction with up to two draft tokens.
Repository: localaiLicense: apache-2.0
underdog-saluki-27b-1.0-mtp
Underdog Saluki 27B 1.0 with its mixed IQ2 main weights preserved and
a Qwen3.8-27B MTP head added. This llama.cpp build enables multi-token
prediction with up to two draft tokens and includes the F16 vision
projector. The weights and projector total approximately 9.28 GB.
Repository: localaiLicense: apache-2.0
apodex-1.1-mini-q4-mtp
Apodex-1.1-mini with MTP speculative decoding enabled on the recommended
Q4_K_M GGUF. The model carries its native MTP head, so it needs no separate
draft model. The F16 vision projector supports multimodal prompts.
Repository: localaiLicense: apache-2.0
qwen3.8-27b-uncensored-q4
Qwen3.8-27B-Uncensored reduces refusal behavior while retaining the base
model's text, vision, reasoning, and tool-use capabilities. Its integrated
MTP head supports speculative decoding without a separate draft model.
This default entry uses the Q4_K_M GGUF and F16 vision projector. A
higher-quality Q8_0 model is available as a variant.
Repository: localaiLicense: apache-2.0
qwen3.8-27b-q4-mtp
Qwen3.8-27B with the official Q4_K_M model and Q4_0 MTP draft model.
MTP speculative decoding can increase generation speed by proposing
multiple tokens for the target model to verify.
Repository: localaiLicense: apache-2.0
qwen3.8-27b-nvfp4-mtp
Qwen3.8-27B in a compact NVFP4 GGUF format with its MTP draft head
embedded in the model file. This entry uses the medium tier, which keeps
the NVFP4 backbone while using higher-precision output and embedding
tensors. MTP speculative decoding proposes multiple tokens for the target
model to verify.
Repository: localaiLicense: apache-2.0
muse-glimmer-30b-dflash
Muse Glimmer's higher-quality dynamic K-quant GGUF with the official
quantized perception encoder and DFlash drafter. DFlash proposes blocks of
up to 16 tokens for the target to verify in parallel, accelerating output
without changing model quality. Flash attention is enabled for this path.
Repository: localaiLicense: apache-2.0
muse-glimmer-30b-17gb-dflash
Muse Glimmer's smaller 17 GB K-quant GGUF with the official quantized
perception encoder and DFlash drafter. This is the lowest-memory published
build that retains image understanding and block-speculative decoding.
Flash attention is enabled for the DFlash path.
Repository: localaiLicense: apache-2.0
tess-4-27b-mtp
Tess-4-27B with its Q4_K_M multi-token prediction draft enabled for
speculative decoding. The main model verifies every proposed token, and
the entry also includes the shared F16 vision projector.
Repository: localaiLicense: apache-2.0
gemmable-4-12b-mtp
## Gemmable 4 12B
Gemmable 4 12B is a GGUF export of Gemma 4 12B fine-tuned on Fable-5 style
reasoning and assistant traces.
## Highlights
- Base model: `google/gemma-4-12B`
- Format: GGUF
- Training style: Fable-5 style reasoning and assistant traces
- Distribution: fp16 GGUF plus matching assistant GGUFs for each quant
- Intended use: local inference, coding, reasoning, and assistant workflows
## How to use
### llama.cpp
Standard load:
```bash
llama-server -m "gemmable-4-12b-fp16.gguf"
```
Speculative / draft-MTP load:
```bash
llama-server -m "gemmable-4-12b-Q4_K_M.gguf" \
--spec-draft-model "gemmable-4-12b-Q4_K_M-mtp.gguf" \
--spec-type draft-mtp \
--spec-draft-n-max 4
```
Use the matching fp16 or quantized main file with its `-mtp` companion.
### LM Studio
1. Search this repo, download target + mtp file.
2. Load target.
3. Load settings → Speculative Decoding → select mtp file file.
(Requires a llama.cpp runtime with Gemma 4 MTP support from ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp backend already carries it, so this entry runs draft-mtp out of the box.)
## GGUF / local inference notes
...
Repository: localai
laguna-s-2.1-dflash
Laguna S 2.1's 96 GB Q4_K_M target paired with Poolside's 2.2 GB BF16 DFlash drafter for speculative decoding. DFlash drafts a block of tokens per forward pass and accelerates generation without changing the target model's outputs. Flash attention is enabled because the DFlash path requires it.
License: OpenMDW 1.1.
Repository: localaiLicense: other
lfm2.5-2.6b-dspark
LFM2.5-2.6B with LiquidAI's DSpark speculative drafter. This build pairs
the Q4_K_M target with the compact Q4_K_M draft sidecar for lower-memory
hosts. DSpark proposes blocks of tokens that the target model verifies,
which preserves the target model's output while accelerating generation.
Repository: localaiLicense: other
lfm2.5-2.6b-q8-dspark
LFM2.5-2.6B with LiquidAI's DSpark speculative drafter. This build pairs
the higher-quality Q8_0 target with the recommended F16 draft sidecar for
the best acceptance length. DSpark proposes blocks of tokens that the
target model verifies, which preserves the target model's output while
accelerating generation.
Repository: localaiLicense: other
qwen3-4b-dflash
Qwen3-4B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. This is the canonical DFlash pairing documented upstream (`z-lab/Qwen3-4B-DFlash` + `Qwen/Qwen3-4B`).
DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3-4B target; the ~0.5 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here.
Flash attention is required for DFlash and is enabled in this config. A GPU is recommended.
License: Apache 2.0 (Qwen3-4B target) / MIT (z-lab DFlash drafter).
Repository: localaiLicense: apache-2.0
qwen3.5-4b-dflash
Qwen3.5-4B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend.
DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3.5-4B target; the ~0.6 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here.
Flash attention is required for DFlash and is enabled in this config. A GPU is recommended.
License: Apache 2.0 (Qwen3.5-4B target) / MIT (z-lab DFlash drafter).
Repository: localaiLicense: apache-2.0
qwen3.5-9b-dflash
Qwen3.5-9B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend.
DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3.5-9B target; the ~1 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here.
Flash attention is required for DFlash and is enabled in this config. A GPU is recommended.
License: Apache 2.0 (Qwen3.5-9B target) / MIT (z-lab DFlash drafter).
Repository: localaiLicense: apache-2.0
qwen3.6-27b-dflash
Qwen3.6-27B (dense) paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. DFlash gives its largest speedups on dense targets like this one.
DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3.6-27B target; the ~1.8 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here.
Flash attention is required for DFlash and is enabled in this config. A GPU is recommended.
License: Apache 2.0.
Repository: localaiLicense: apache-2.0
qwen3.6-35b-a3b-dflash
Qwen3.6-35B-A3B (Mixture-of-Experts, ~3B active per token) paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. DFlash speedups on MoE targets are smaller than on dense models, but still useful.
DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The UD-Q4_K_M file carries the full Qwen3.6-35B-A3B target; the ~0.4 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here.
Flash attention is required for DFlash and is enabled in this config. A GPU is recommended.
License: Apache 2.0.
Repository: localaiLicense: apache-2.0
gemma-4-12b-qat-hauhaucs-balanced-mtp
HauhauCS Balanced is an uncensored Gemma 4 12B model built from
quantization-aware-trained weights for chat, coding, and creative writing.
This Q4_K_M GGUF build includes the vision projector for image input.
The MTP variant adds the publisher-provided draft head and enables
llama.cpp multi-token-prediction speculative decoding.
Repository: localaiLicense: gemma
gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2-mtp
Gemma 4 12B Agentic v2 is an Apache-2.0 fine-tune for coding, terminal
work, multi-step tool use, and reasoning. This variant pairs the Q4_K_M
target with the upstream Q8_0 Gemma 4 MTP drafter for faster lossless
generation through llama.cpp speculative decoding.
Repository: localaiLicense: apache-2.0
Page 1