AlexWortega/moe-600m-qwen3-upcycle

A ~600M-parameter Mixture-of-Experts language model created by sparse upcycling of Qwen/Qwen3-0.6B-Base, then continually pretrained on Ultra-FineWeb.

Checkpoint: step 174999, 5.73B tokens seen.

How it was built

The donor is a dense 596M model. Rather than duplicating its FFN into experts (which would overshoot the 600M budget to ~850M), each layer's 3072 MLP neurons are partitioned into one shared expert and several routed experts, so the total parameter count is preserved exactly.

Neurons are assigned by measured importance (E[a^2] * ||down[:, j]||^2): the shared expert, which fires on every token, takes the top-ranked neurons — empirically ~47% of the importance mass sits in the top 25% of neurons. The remainder is dealt out so every routed expert carries near-equal importance, which keeps load balancing from fighting the router.

The weight transfer itself is exact: logits match Qwen3ForCausalLM bit-for-bit before the experts are carved.

Architecture

total / active params	596M / 469M
layers	28
d_model	1024
heads	16 q / 8 kv, head_dim 128
experts	1 shared x 768 + 6 routed x 384, top-2
RoPE	full, theta 1000000
vocab	151936 (Qwen3 tokenizer), tied embeddings

Files

step_174999/model.safetensors — weights. model.py at the repo root defines MoEModel; the architecture is not a transformers class, so load it with that module rather than AutoModel.

import torch, json
from model import MoEModel, MoEModelConfig
from safetensors.torch import load_file

cfg = MoEModelConfig(**json.load(open("config.json"))["model_config"])
m = MoEModel(cfg)
m.load_state_dict(load_file("step_174999/model.safetensors"))

Training

fp16 with a manual loss scaler on 4x V100 (no bf16 on Volta), Muon for matrix parameters plus AdamW for the rest, WSD schedule. Muon runs at a much lower LR than a from-scratch run would use: its update has fixed spectral norm regardless of gradient size, so a pretrained weight keeps drifting even once it is correct.