التفاصيل البيبلوغرافية
| العنوان: |
Atla Selene Mini: A General Purpose Evaluation Model |
| المؤلفون: |
Alexandru, Andrei, Calvi, Antonia, Broomfield, Henry, Golden, Jackson, Dai, Kyle, Leys, Mathias, Burger, Maurice, Bartolo, Max, Engeler, Roman, Pisupati, Sashank, Drane, Toby, Park, Young Sun |
| سنة النشر: |
2025 |
| المجموعة: |
Computer Science |
| مصطلحات موضوعية: |
Computer Science - Computation and Language, Computer Science - Artificial Intelligence |
| الوصف: |
We introduce Atla Selene Mini, a state-of-the-art small language model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that outperforms the best SLMJs and GPT-4o-mini on overall performance across 11 out-of-distribution benchmarks, spanning absolute scoring, classification, and pairwise preference tasks. It is the highest-scoring 8B generative model on RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To achieve this, we develop a principled data curation strategy that augments public datasets with synthetically generated critiques and ensures high quality through filtering and dataset ablations. We train our model on a combined direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and produce a highly promptable evaluator that excels in real-world scenarios. Selene Mini shows dramatically improved zero-shot agreement with human expert evaluations on financial and medical industry datasets. It is also robust to variations in prompt format. Preliminary results indicate that Selene Mini is the top-ranking evaluator in a live, community-driven Judge Arena. We release the model weights on HuggingFace (https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage widespread community adoption. |
| نوع الوثيقة: |
Working Paper |
| URL الوصول: |
http://arxiv.org/abs/2501.17195 |
| رقم الانضمام: |
edsarx.2501.17195 |
| قاعدة البيانات: |
arXiv |