Aether-7B-5Attn: A Fully Open Sovereign Foundation Model with Heterogeneous Attention
AIOpen SourceAether-7B-5Attn: A Fully Open Sovereign Foundation Model
On 19 July 2026, VIDRAFT, a single Korean AI startup, released Aether-7B-5Attn — a foundation model that meets the most stringent definition of open source: not just the weights, but the data recipe, the training code, the full logs, intermediate checkpoints, and the complete architecture source, all under Apache 2.0.
What "Fully Open" Means
Most models labelled "open" release weights under a permissive license but keep the training pipeline, data composition, and intermediate states proprietary. Aether follows the standard that Allen Institute's OLMo established: every artifact required to rebuild the model from scratch is published alongside the weights.
| Artifact | Available |
|---|---|
| Model weights | Apache 2.0 |
| Training data recipe and mixing weights | Published |
| Tokeniser | Open |
| FSDP launcher and training loop | Open |
| All hyperparameters | Documented |
| Step-by-step training logs | Published |
| Intermediate checkpoints (110k, 115k, 162k) | Published |
| Architecture source | Apache 2.0 |
The claim is narrow and verifiable: among the world's fully-open sovereign foundation models, Aether is the only one built by a single AI startup rather than a national lab, a government consortium, or a university coalition.
The Latin Square Attention Experiment
Aether's 49 layers place five distinct attention mechanisms on a 7×7 Latin square — a design that converts a vague architectural intuition into a controlled experiment. Each row and column of the square contains each attention type exactly once, isolating the effect of heterogeneous placement from the effect of which layer receives which operator.
| Attention Type | Layers Applied |
|---|---|
| Full (standard softmax) | Position (1,1), (2,5), (3,2), (4,6), (5,3), (6,7), (7,4) |
| Sliding window (local) | Position (1,2), (2,6), (3,3), (4,7), (5,4), (6,1), (7,5) |
| Dilated (strided) | Position (1,3), (2,7), (3,4), (4,1), (5,5), (6,2), (7,6) |
| Differential | Position (1,4), (2,1), (3,5), (4,2), (5,6), (6,3), (7,7) |
| Sparse (top-k) | Position (1,5), (2,2), (3,6), (4,3), (5,7), (6,4), (7,1) |
The five operators are realised as five distinct computational mechanisms: standard softmax attention, gated SDPA, native sparse attention, sliding-window attention, and differential attention. To our knowledge, no released model has arranged heterogeneous attention on a Latin square — making Aether a genuine research instrument as well as a deployable model.
Architecture and Training
Aether-7B-5Attn is a 6.59B-parameter MoE model with approximately 2.98B active parameters per token. It uses 25 routed experts (top-7) plus one shared expert, with standard load-balancing loss.
| Component | Specification |
|---|---|
| Architecture | Decoder-only Transformer, 49 layers |
| Total / active parameters | 6.59B / ~2.98B |
| Attention heads | 16 query / 4 key-value (GQA), head dim 128 |
| FFN intermediate | 6144 (dense) |
| Experts | 25 routed (top-7) + 1 shared |
| Vocabulary | 151,936 (BPE) |
| Context length | 4,096 (RoPE θ = 10,000) |
| Pretraining | 144.2B tokens, 162k steps, 16× B200, ~46 days |
| Precision | bfloat16 |
| License | Apache 2.0 |
Training on 144.2B tokens across 162,000 steps on 16 NVIDIA B200 GPUs in a 2-node FSDP configuration completed in roughly 46 days. The released checkpoints span 110k, 115k, and 162k (annealed) steps, enabling training-dynamics research on a fully-open model.
A Methodological Lesson in Post-Training
The instruct variant, Aether-7B-5Attn-it, was selected not by training loss but by held-out benchmark accuracy. This is a deliberate methodological stance: for small models, training loss rewards format-fitting, not competence. Sweeping three learning rates (2e-6, 6e-6, 2e-5) and selecting on held-out accuracy produced the instruct variant that scores 34.9% average across the K-AI 4-benchmark suite — an 8.2 percentage point gain over the base model. The lowest-loss checkpoint (2e-5) collapsed on one subject entirely.
The Sovereignty Thesis
For enterprise architecture teams, Aether's primary contribution is not its benchmark score. It is an existence proof that a mid-size organisation can hold the entire means of production of a foundation model — and therefore rebuild, audit, and evolve it independently.
"Real sovereignty is the capacity to rebuild," the VIDRAFT team writes. "It requires the recipe, not just the dish." Aether's recipe is verifiably complete: download the code, run the pipeline, and the training binaries reproduce byte-for-byte.
For enterprises evaluating sovereign AI infrastructure, Aether-7B-5Attn represents a new baseline for what "open" must mean.