ZAYA1-8B: New MoE Model Trained for Reasoning with 700M Active Parameters
Zyphra has introduced ZAYA1-8B, a Mixture-of-Experts (MoE) model with 700 million active and 8 billion total parameters, optimised for reasoning tasks.
What happened?
ZAYA1-8B is a new MoE model from Zyphra, based on their MoE++ architecture. The model features 700 million active parameters and a total of 8 billion parameters. The development of ZAYA1-8B, from pre-training to Supervised Fine-Tuning (SFT), was carried out entirely on an AMD compute, networking, and software platform. The model demonstrates strong performance on mathematics and coding benchmarks.
Key facts
| Modellnamn | ZAYA1-8B |
|---|---|
| Arkitektur | Mixture-of-Experts (MoE) på MoE++ |
| Aktiva parametrar | 700 miljoner |
| Totala parametrar | 8 miljarder |
| Träningsplattform | Full-stack AMD |
| Fokusområden | Resonemang, matematik, kodning |
”We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture.”
”With under 1B active parameters, ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks, and remains competitive with substantially larger open-weight reasoning models.”
”ZAYA1-8B was trained from scratch for reasoning, with reasoning data included from pretraining onward using an answer-preserving trimming scheme.”
Why it matters
The development of ZAYA1-8B signals a trend towards more specialised and efficient language models. By focusing on reasoning from the start of training, including a unique method for preserving answers during data pruning and a four-stage RL cascade, Zyphra aims to create capable models with fewer active parameters. This could potentially lower computational costs and increase the accessibility of advanced AI models for specific tasks.
Who is affected?
AI developers and researchers, particularly those working with small and medium-sized language models (SLMs) or focus areas such as mathematics and programming, are affected by this launch. Companies seeking cost-effective solutions for complex reasoning tasks may benefit from models like ZAYA1-8B. The model is primarily aimed at those who benefit from high performance with relatively low resource requirements.
What else you should know
The training of ZAYA1-8B included a four-stage reinforcement learning (RL) cascade, with stages focusing on mathematics and puzzle tasks, a 400-task RLVE-Gym curriculum, as well as mathematical and code-based RL in synthetic code environments.
Quick answers about this story
Vad har hänt?
När hände det?
Varför spelar det roll?
Vilka områden har modellen specialiserats inom?
The link opens in a new window and leads to the publisher's own site.
Källan har spårats automatiskt från utgivaren via Aheadlines signalkedja.
AI-verktyg i artikeln
Topics
Get similar news straight to your inbox
The reader's room
Send in a question or an addition. The newsroom reads everything before it's published and replies when relevant. No AI-generated text – just people.
Sign in to submit a comment or question.
Read the article through your role
- Decide whether this affects strategy over 6–12 months or is just noise.
- Discuss with leadership: do we own the right question or does ownership need to move?
- Ask: what risk are we taking by NOT acting on this this quarter?
Generated angle — not editorial analysis of "ZAYA1-8B: New MoE Model Trained for Reasoning with 700M Acti"