File size: 1,817 Bytes
25a1cef | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
datasets: HuggingFaceFW/fineweb-edu
widget:
- example_title: Example interaction
text: During photosynthesis in green plants
inference:
parameters:
repetition_penalty: 1.3
language:
- en
library_name: transformers
license: mit
---
# Model Card for gpt2-124M-edu-fineweb-10B
A 124M parameter GPT2 model trained with the 10B fineweb-edu dataset using [https://github.com/karpathy/llm.c](https://github.com/karpathy/llm.c)
Training took 20 hours on a single 4090 GPU (limited to 350W), giving the following graphs:
## Training
The training parameters where:
```
./train_gpt2cu \
-i "dev/data/edu_fineweb10B/edu_fineweb_train_*.bin" \
-j "dev/data/edu_fineweb10B/edu_fineweb_val_*.bin" \
-o log124M \
-e "d12" \
-b 56 -t 1024 \
-d 458752 \
-r 1 \
-z 1 \
-c 0.1 \
-l 0.002 \
-q 0.0 \
-u 700 \
-n 5000 \
-v 250 -s 20000 \
-h 1
```
The model has had no further finetuning.
## Evaluation
Evals using [Eleuther AI Harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463)
as described in the [open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
and comparing with those published for [openai-community/gpt2](https://huggingface.co/openai-community/gpt2)
| Eval Test | Score |
| --------- | ----- |
| arc_challenge (25 shot) | 24.49 |
| gsm8k (5 shot) | 0.08 |
| hellaswag (10 shot) | 32.64 |
| mmlu (5 shot) | 26.06 |
| truthfulqa (0 shot) | 42.45 |
| winogrande (5 shot) | 52.17 |
| **Overall Score** | **29.65** | |