HuggingFaceFW/fineweb-2
FineWeb-2 is a dataset of over 15 trillion tokens of cleaned and deduplicated English web data from CommonCrawl. This is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages.The 🥂 FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments.In particular, on the set of 9 diverse languages we used to guide our processing decisions, 🥂 FineWeb2 outperforms other popular pretraining datasets covering multiple languages (such as CC-100, mC4, CulturaX or HPLT, while being substantially larger) and, in some cases, even performs better than some datasets specifically curated for a single one of these languages, in our diverse set of carefully selected evaluation tasks: FineTasks.
Detaljert introduksjon
FineWeb-2 is a large-scale dataset designed to provide high-quality web data for training large language models. This is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages.The 🥂 FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments.In particular, on the set of 9 diverse languages we used to guide our processing decisions, 🥂 FineWeb2 outperforms other popular pretraining datasets covering multiple languages (such as CC-100, mC4, CulturaX or HPLT, while being substantially larger) and, in some cases, even performs better than some datasets specifically curated for a single one of these languages, in our diverse set of carefully selected evaluation tasks: FineTasks.
MerDatasett
SimpleToM Dataset
The SimpleToM dataset is designed to evaluate models' ability to reason about beliefs and actions in various scenarios. It includes a variety of situations with multiple choice questions and answers, covering topics such as food items, personal belongings, and service industries.
Psych-101: Human Psychological Experiment Transcripts Dataset
Psych-101 is a dataset of natural language transcripts from human psychological experiments, comprising trial-by-trial data from 160 experiments and 60,092 participants, making 10,681,650 choices. It provides valuable insights into human decision-making processes and is available under the Apache License 2.0.
Psychology Wiki Dataset
Psychology Wiki Datasetpsychology_wiki数据集的构建基于心理学领域的英文维基百科内容，通过系统化的数据采集与整理，确保了信息的广泛覆盖与深度挖掘。数据集中的每一篇文章均经过严格的筛选与标注，涵盖了标题、正文、相关性、受欢迎程度及排名等多个维度，为心理学研究提供了丰富的文本资源。