Report
KidLM: Advancing Language Models for Children -- Early Insights and Future Directions
| العنوان: | KidLM: Advancing Language Models for Children -- Early Insights and Future Directions |
|---|---|
| المؤلفون: | Nayeem, Mir Tafseer, Rafiei, Davood |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computer Science - Computers and Society, Computer Science - Human-Computer Interaction |
| الوصف: | Recent studies highlight the potential of large language models in creating educational tools for children, yet significant challenges remain in maintaining key child-specific properties such as linguistic nuances, cognitive needs, and safety standards. In this paper, we explore foundational steps toward the development of child-specific language models, emphasizing the necessity of high-quality pre-training data. We introduce a novel user-centric data collection pipeline that involves gathering and validating a corpus specifically written for and sometimes by children. Additionally, we propose a new training objective, Stratified Masking, which dynamically adjusts masking probabilities based on our domain-specific child language data, enabling models to prioritize vocabulary and concepts more suitable for children. Experimental evaluations demonstrate that our model excels in understanding lower grade-level text, maintains safety by avoiding stereotypes, and captures children's unique preferences. Furthermore, we provide actionable insights for future research and development in child-specific language modeling. Comment: Accepted to EMNLP 2024 (long, main) |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2410.03884 |
| رقم الانضمام: | edsarx.2410.03884 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |