Mistral-7B-v0.1-NCA-ARC

A two-stage preference-tuned fine-tune of mistralai/Mistral-7B-v0.1 (via in-house Reference SFT golfoscar/mistral-7b-sft-arc) reaching 0.6826 ± 0.0136 acc_norm on ARC-Challenge 25-shot — +6.83 pp over the pretrained baseline (0.6143). Submitted for the Upstage AIRE LLM/Deep-Learning coding test.

A LoRA-adapter version of the same checkpoint is at golfoscar/mistral-7b-v0.1-nca-arc-lora (~167 MB) for users who already have the base.

Recipe summary

1. Reference SFT on a curated 4-source science-MCQ corpus → golfoscar/mistral-7b-sft-arc (0.6323).
2. NCA preference fine-tune with TRL's nca_pair (β=0.10) on 8,619 preference pairs → 0.6775.
3. Hard-Margin NCA continuation (β=0.05, lr=1e-6, 2 epochs) on the bottom-25% pairs by the prior model's implicit-reward margin (2,155 pairs, 70 % OpenBookQA-skewed) → 0.6826. load_best_model_at_end=True makes best and final adapters byte-identical.

The dominant lever is NCA's explicit suppression of rejected log-probabilities (8 nats over 269 steps while chosen log-prob stays stable). The hard-margin curriculum adds a smaller but consistent +0.51 pp (0.38σ, single-seed PARTIAL pass).

Performance — ARC-Challenge 25-shot (acc_norm)

Run	acc_norm	Δ baseline
Mistral-7B-v0.1 (pretrained)	0.6143	—
Reference SFT	0.6323	+1.80 pp
DPO (sigmoid, β=0.05)	0.6544 ± 0.0139	+4.01 pp
NCA (nca_pair, β=0.10)	0.6775 ± 0.0137	+6.32 pp
This model (NCA + hard-margin)	0.6826 ± 0.0136	+6.83 pp
Likra paper (independent pipeline, reference only)	0.6630	—

Reproducing 0.6826

pip install lm-eval==0.4.5     # GitHub repo: lm-evaluation-harness
python -m lm_eval \
  --model hf \
  --model_args "pretrained=golfoscar/mistral-7b-v0.1-nca-arc,dtype=bfloat16,tokenizer=golfoscar/mistral-7b-v0.1-nca-arc" \
  --tasks arc_challenge \
  --num_fewshot 25 \
  --batch_size auto:4

Tested on A100 40 GB, torch==2.5.1+cu124, transformers==4.49.0, lm-eval==0.4.5, fresh Python process. If bf16 cublasGemmEx fails for long sequences on your environment, use dtype=float32 (slower but always works).

Model details

• Architecture: MistralForCausalLM, 7 B parameters (4096 hidden / 32 layers / 32 heads / 8 KV heads, GQA, head_dim 128, context 32 768, sliding-window 4 096).
• Weight precision on disk: fp16, 3 safetensors shards (~14 GB total). No quantization_config — load with plain transformers, no PEFT/bnb required.
• Tokenizer: identical to mistralai/Mistral-7B-v0.1 (slow-tokenizer files bundled for self-containment).

Training data

Single in-house preference corpus, 8,619 (prompt, chosen, rejected) triples:

Source	Pairs	License
ARC-Challenge (train+val)	3,378	CC-BY-SA 4.0
OpenBookQA	2,802	Apache 2.0
ARC-Easy (train+val)	1,878	CC-BY-SA 4.0
QASC	561	CC-BY 4.0

Pairs are filtered for: empty fields, chosen == rejected, exact-match with ARC-Challenge test prompts (zero leakage), word counts in [2, 50], length balance max/min ≤ 3.0 (Smaug-style length-classifier failure), exact triple-level dedup. SciQ (CC-BY-NC + empirical regression) and MedMCQA (out-of-distribution) are excluded.

Hard-margin subset (for stage 3): bottom 25 % by m = log π(chosen|x) − log π(rejected|x) under the stage-2 model (2,155 pairs; 413 with non-positive margin; 70.1 % OpenBookQA-skewed). Zero ARC-test leakage at both prompt-level and 13-gram level.

Training procedure

QLoRA on golfoscar/mistral-7b-sft-arc (4-bit NF4 base, compute_dtype=fp16, double-quant), LoRA r=16 / α=32 / dropout 0.05, target_modules="all-linear" (224 modules), paged_adamw_8bit optimizer, batch 2 × grad-accum 16 (effective 32), cosine schedule with warmup 0.03, max_length=512, seed=42.

Stage	Loss	β	LR	Data	Epochs	Steps
NCA (stage 2)	nca_pair	0.10	5e-6	full 8,619	1	269
Hard-margin NCA (stage 3, this model)	nca_pair	0.05	1e-6	bottom-25% 2,155	2	122

Stage-3 training: 78 min on a single A100 40 GB, peak 5.51 GB VRAM, zero NaN/Inf, zero abort triggers, eval loss monotonically decreases from 1.3088 to 1.3008 across 6 evaluations.

Bias, Risks, and Limitations

• Domain-specialized: trained only on grade-school science MCQ. Math, code, chat, instruction-following, summarization are not characterized and likely worse than general-purpose Mistral fine-tunes.
• No instruction or safety alignment beyond science-MCQ preference. Do not use for safety-critical applications.
• Prompt format dependence: trained for the bare Question: {q}\nAnswer: template used by lm-evaluation-harness arc_challenge.yaml; other formats may degrade performance.
• Single seed (42): the +0.51 pp stage-3 gain over stage-2 is ~0.38σ on a single seed (PARTIAL pass, not significant at 1σ). Multi-seed replication is the documented next step.
• English only.

License

Released under Apache 2.0 (inherited from mistralai/Mistral-7B-v0.1). Training data: Apache 2.0 / CC-BY 4.0 / CC-BY-SA 4.0 mixture (all commercial-safe). No CC-BY-NC data was used.

Citation

@inproceedings{chen2024nca,
  title={Noise Contrastive Alignment of Language Models with Explicit Rewards},
  author={Chen, Huayu and others},
  booktitle={NeurIPS 2024 / arXiv:2402.05369},
  year={2024}
}
@article{dettmers2023qlora,
  title={QLoRA: Efficient Finetuning of Quantized LLMs},
  author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  journal={arXiv:2305.14314},
  year={2023}
}
@article{clark2018arc,
  title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},
  author={Clark, Peter and others},
  journal={arXiv:1803.05457},
  year={2018}
}

Author

• Hyeonseop Shin — hysp.shin@gmail.com / hyeonseop.shin@snu.ac.kr
• Submission for the Upstage AIRE LLM/Deep-Learning coding test, May 2026.