Filter by type:
Filter by tags:
hy4-preview
# Hy4-preview GGUF
Three GGUF builds of Hy4-Preview: https://huggingface.co/tencent/Hy4-preview
**Language / 语言:** English · 中文
**Neither file runs on stock llama.cpp.** The `hyv4` architecture is not upstream. Apply the
patches in `hy4-preview-patch/`
## English
### 1. What these are
**`Hy4-preview-Q4_K_M.gguf`** — a conventional Q4_K_M. Most tensors are Q4_K; `ffn_down_exps`
gets Q6_K on 37 layers via llama.cpp's own logic. Use this unless you are memory-constrained.
**`Hy4-preview-UD-IQ1_M.gguf`** - mixed precision with UD-IQ1_M strategy at ~2.44 bpw, roughly **half the size** for the
same model. The routed-expert `gate`/`up` projections run at 1.75 bpw (IQ1_M) and 2.0625 bpw (IQ2_XXS).
**`Hy4-preview-STQ1_0.gguf`** — mixed precision with MIX-STQ1_0 strategy at ~2.38 bpw, roughly **half the size** for the
same model. The routed-expert `gate`/`up` projections run at 1.3125 bpw (STQ1_0) on 29 layers and
2.0625 bpw (IQ2_XXS) on the other 48. See section 3.
### 2. Running them
Build a patched llama.cpp
```bash
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 0cea36222
...
Repository: localai
gemmable-4-12b-mtp
## Gemmable 4 12B
Gemmable 4 12B is a GGUF export of Gemma 4 12B fine-tuned on Fable-5 style
reasoning and assistant traces.
## Highlights
- Base model: `google/gemma-4-12B`
- Format: GGUF
- Training style: Fable-5 style reasoning and assistant traces
- Distribution: fp16 GGUF plus matching assistant GGUFs for each quant
- Intended use: local inference, coding, reasoning, and assistant workflows
## How to use
### llama.cpp
Standard load:
```bash
llama-server -m "gemmable-4-12b-fp16.gguf"
```
Speculative / draft-MTP load:
```bash
llama-server -m "gemmable-4-12b-Q4_K_M.gguf" \
--spec-draft-model "gemmable-4-12b-Q4_K_M-mtp.gguf" \
--spec-type draft-mtp \
--spec-draft-n-max 4
```
Use the matching fp16 or quantized main file with its `-mtp` companion.
### LM Studio
1. Search this repo, download target + mtp file.
2. Load target.
3. Load settings → Speculative Decoding → select mtp file file.
(Requires a llama.cpp runtime with Gemma 4 MTP support from ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp backend already carries it, so this entry runs draft-mtp out of the box.)
## GGUF / local inference notes
...
Repository: localai
ced-base-f16
CED (Consistent Ensemble Distillation, Xiaomi) is a sound-event classifier that tags everyday sounds (baby cry, footsteps, glass breaking, alarms, dog bark, ...) into the 527-class AudioSet ontology. This is the f16 GGUF for the ced backend (a standalone C++/ggml port). Recommended default: fastest on CPU and near-lossless. Use POST /v1/audio/classification, or the realtime websocket API for live recognition.
Repository: localaiLicense: apache-2.0
minimax-h3-fl2va-unpruned-q4-ggml
MiniMax-H3 served by stable-diffusion.cpp. It generates video AND stereo
audio jointly from a text prompt, so a clip comes back as an MP4 with a
soundtrack. Muxing the finished frames needs ffmpeg on the host.
Runtime: stable-diffusion.cpp (backend stablediffusion-ggml). Partition:
fl2va. DiT layout: unpruned. Format: Q4_K_M.
This entry is the FL2VA partition: text-to-video, first-frame (I2VA) and first/last-frame (FL2VA) conditioning. It refuses reference conditioning (reference image or audio): that needs minimax-h3-ref2va-pruned-q4-ggml, which is a separate checkpoint.
Render status: verified render (the stable-diffusion.cpp page shows a text-to-video and
an image-to-video render with this file).
Weights are the Q4_K_M quantisation of the DiT (~18.8 GB), the
Qwen3-VL-32B MiniMax-H3 text encoder (~18 GB), the fp16 video VAE (~5 GB)
and the fp32 audio VAE (~0.6 GB). MiniMax-H3 runs at 24 fps and the width
and height are rounded up to a multiple of 32. The frame count is
rounded up to the 17k + 5 grid.
Repository: localaiLicense: other
minimax-h3-fl2va-pruned-q4-ggml
MiniMax-H3 served by stable-diffusion.cpp. It generates video AND stereo
audio jointly from a text prompt, so a clip comes back as an MP4 with a
soundtrack. Muxing the finished frames needs ffmpeg on the host.
Runtime: stable-diffusion.cpp (backend stablediffusion-ggml). Partition:
fl2va. DiT layout: pruned. Format: Q4_K_M.
This entry is the FL2VA partition: text-to-video, first-frame (I2VA) and first/last-frame (FL2VA) conditioning. It refuses reference conditioning (reference image or audio): that needs minimax-h3-ref2va-pruned-q4-ggml, which is a separate checkpoint. It is the smaller, pruned counterpart of minimax-h3-fl2va-unpruned-q4-ggml.
Render status: rendered with stable-diffusion.cpp on CUDA (text-to-video, 512x288, 22 frames, with audio). Not yet rendered through LocalAI.
Weights are the Q4_K_M quantisation of the DiT (~11.4 GB), the
Qwen3-VL-32B MiniMax-H3 text encoder (~18 GB), the fp16 video VAE (~5 GB)
and the fp32 audio VAE (~0.6 GB). MiniMax-H3 runs at 24 fps and the width
and height are rounded up to a multiple of 32. The frame count is
rounded up to the 17k + 5 grid.
Repository: localaiLicense: other
minimax-h3-ref2va-pruned-q4-ggml
MiniMax-H3 served by stable-diffusion.cpp. It generates video AND stereo
audio jointly from a text prompt, so a clip comes back as an MP4 with a
soundtrack. Muxing the finished frames needs ffmpeg on the host.
Runtime: stable-diffusion.cpp (backend stablediffusion-ggml). Partition:
ref2va. DiT layout: pruned. Format: Q4_K_M.
This entry is the REF2VA partition, the one that takes REFERENCE conditioning: reference images, reference videos (a directory of frames, with an optional paired soundtrack) and reference audio steer the subject and the voice of the clip. It does not serve first/last-frame keyframes, and keyframes cannot be mixed with references. For plain text-to-video or keyframes use minimax-h3-fl2va-unpruned-q4-ggml.
Render status: verified render (the stable-diffusion.cpp page shows a reference-to-video
render with this file).
Weights are the Q4_K_M quantisation of the DiT (~11.4 GB), the
Qwen3-VL-32B MiniMax-H3 text encoder (~18 GB), the fp16 video VAE (~5 GB)
and the fp32 audio VAE (~0.6 GB). MiniMax-H3 runs at 24 fps and the width
and height are rounded up to a multiple of 32. The frame count is
rounded up to the 17k + 5 grid.
Repository: localaiLicense: other
vibevoice-cpp
VibeVoice Realtime 0.5B (C++ / GGML, Q8_0) - native C++ port of Microsoft VibeVoice
via the vibevoice-cpp backend. 24kHz mono TTS with a selectable precomputed
voice prompt. Default voice prompt: en-Carter_man. This realtime variant does
not accept raw Voice Library reference WAVs.
Repository: localaiLicense: mit
vibevoice-cpp-asr
VibeVoice ASR 7B (C++ / GGML, Q4_K) - long-form speech-to-text with speaker
diarization. Returns per-speaker JSON segments with start/end timestamps.
English-only. ~10 GB download.
Repository: localaiLicense: mit
vits-streaming-assistant
English voice assistant preset: Silero VAD, Parakeet TDT 0.6B v3, Qwen3
0.6B chat, and VITS-LJS English single-speaker TTS through sherpa-onnx.
Buffers streamed LLM output into clauses and emits audio progressively.
Each clause starts a separate synthesis; this is not native incremental
text input. This installs only the pipeline configuration. First install
silero-vad-ggml, parakeet-cpp-tdt-0.6b-v3, qwen3-0.6b, and vits-ljs-sherpa.
WebRTC also needs the opus backend. No browser or hardware performance
is established by this preset.
Repository: localaiLicense: mit
piper-streaming-assistant
English voice assistant preset: Silero VAD, Parakeet TDT 0.6B v3, Qwen3
0.6B chat, and Piper Amy US English TTS through piper.
Buffers streamed LLM output into clauses and emits audio progressively.
Each clause starts a separate synthesis; this is not native incremental
text input. This installs only the pipeline configuration. First install
silero-vad-ggml, parakeet-cpp-tdt-0.6b-v3, qwen3-0.6b, and voice-en-us-amy-low.
WebRTC also needs the opus backend. No browser or hardware performance
is established by this preset.
Repository: localaiLicense: mit
qwen3-streaming-assistant
English voice assistant preset: Silero VAD, Parakeet TDT 0.6B v3, Qwen3
0.6B chat, and Qwen3-TTS 0.6B CustomVoice with Ryan's voice. Streams LLM
text into one persistent TTS session while audio is emitted.
This installs only the pipeline configuration. First install the separate
gallery models silero-vad-ggml, parakeet-cpp-tdt-0.6b-v3, qwen3-0.6b, and
qwen3-tts-cpp-customvoice. Pin its variant to qwen3-tts-cpp-customvoice
for Q8. WebRTC also needs the opus backend.
Requires a LocalAI build with incremental Qwen3-TTS input support.
CPU synthesis may be slower than playback; GPU and browser performance
are not established by this preset.
Repository: localaiLicense: mit
moss-tts-cpp-v1_5-q8_0
MOSS-TTS-Local v1.5 (C++/ggml, moss-tts.cpp), Q8_0 (~6 GB), near-lossless. Native
C++ text-to-speech for the OpenMOSS MOSS-TTS-Local Transformer v1.5 (a GPT-J local
transformer decoded through the MOSS-Audio-Tokenizer-v2 neural codec), 48kHz stereo
output with reference-audio voice cloning (set `voice` to a reference .wav). Verified
per component against the reference PyTorch and about 2x faster per frame on CPU.
Repository: localaiLicense: apache-2.0
moss-tts-cpp-v1_5-f16
MOSS-TTS-Local v1.5 (C++/ggml, moss-tts.cpp), F16 (~9.4 GB), code-exact versus the
reference. 48kHz stereo text-to-speech with reference-audio voice cloning.
Repository: localaiLicense: apache-2.0
magpie-tts-cpp-357m-q8_0
Magpie TTS Multilingual 357M (C++/ggml, magpie-tts.cpp), Q8_0 (~624 MB), near-lossless
and the fastest decode. Native C++ text-to-speech for NVIDIA's Magpie TTS Multilingual
(encoder + autoregressive decoder over NanoCodec tokens) from one self-contained GGUF
(model, codec, tokenizer, G2P dictionaries). 22.05kHz mono, 5 baked voices (set `voice`
to Aria, Jason, John, Leo, Sofia or 0-4), 9+ languages (en, es, de, fr, it, pt-BR, hi,
vi, ko, plus Arabic variants). Parity-gated per component against the NeMo reference.
Repository: localaiLicense: other
magpie-tts-cpp-357m-f16
Magpie TTS Multilingual 357M (C++/ggml, magpie-tts.cpp), F16 (~784 MB). 22.05kHz mono
text-to-speech with 5 baked voices and 9+ languages.
Repository: localaiLicense: other
realesrgan-x4plus-anime-6b
RealESRGAN x4plus anime 6B is a six-block ESRGAN model for anime images,
with a native 4x scale. Uses the official PyTorch .pth checkpoint through
stable-diffusion.cpp, without conversion. Upscale-only: not a text-to-image model.
Repository: localaiLicense: BSD-3-Clause
face-detect-buffalo-l
Face recognition with insightface's `buffalo_l` pack (SCRFD-10GF
detector + ResNet50 ArcFace 512-d embedder), ported to C++/ggml and
shipped as a single GGUF for the `face-detect` backend. Highest
accuracy of the buffalo line.
No Python / onnxruntime / torch runtime: face-detect.cpp reads the
detector and embedder architecture (`facedetect.arch`) directly from
the GGUF metadata, so installing this entry is all that is needed to
select buffalo_l. Drives the Embedding / Detect / FaceVerify /
FaceAnalyze gRPC rpcs and the /v1/face/{verify,analyze,embed,detect}
REST endpoints. This GGUF also embeds the MiniFASNet anti-spoof
ensemble, available via the FaceVerify `anti_spoof` request flag.
NON-COMMERCIAL RESEARCH USE ONLY: for commercial use see
`face-detect-yunet-sface`.
Repository: localaiLicense: insightface-non-commercial
face-detect-buffalo-m
Face recognition with insightface's `buffalo_m` pack (SCRFD-2.5GF
detector + ResNet50 ArcFace embedder), converted to a C++/ggml GGUF
for the `face-detect` backend. Same recognition accuracy as
`buffalo_l` with a cheaper detector: a good balance on mid-range
hardware.
The architecture (`facedetect.arch`) is read from the GGUF metadata,
so this entry alone selects the buffalo_m engine. This GGUF also
embeds the MiniFASNet anti-spoof ensemble, available via the
FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE
ONLY.
Repository: localaiLicense: insightface-non-commercial
face-detect-buffalo-s
Face recognition with insightface's `buffalo_s` pack (SCRFD-500MF
detector + MBF 512-d embedder), converted to a C++/ggml GGUF for the
`face-detect` backend. Small and CPU-friendly: a good fit for
mid-range and edge deployments.
The architecture (`facedetect.arch`) is read from the GGUF metadata,
so this entry alone selects the buffalo_s engine. This GGUF also
embeds the MiniFASNet anti-spoof ensemble, available via the
FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE
ONLY.
Repository: localaiLicense: insightface-non-commercial
face-detect-buffalo-sc
Face recognition with insightface's `buffalo_sc` pack (SCRFD-500M
detector + a small ArcFace embedder), converted to a C++/ggml GGUF
for the `face-detect` backend. This is the smallest insightface
pack: the lightest option for low-resource and edge deployments.
The architecture (`facedetect.arch`) is read from the GGUF metadata,
so this entry alone selects the buffalo_sc engine. If this GGUF
embeds the MiniFASNet anti-spoof ensemble, it is available via the
FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE
ONLY.
Repository: localaiLicense: insightface-non-commercial
face-detect-antelopev2
Face recognition with insightface's `antelopev2` pack (SCRFD-10G
detector + ArcFace glint360k R100, 512-d embedder), converted to a
C++/ggml GGUF for the `face-detect` backend. The higher-accuracy
insightface pack: heavier, but the best fit when recognition
quality matters more than speed.
The architecture (`facedetect.arch`) is read from the GGUF metadata,
so this entry alone selects the antelopev2 engine. If this GGUF
embeds the MiniFASNet anti-spoof ensemble, it is available via the
FaceVerify `anti_spoof` request flag. NON-COMMERCIAL RESEARCH USE
ONLY.
Repository: localaiLicense: insightface-non-commercial
Page 1