CodeSoft/slm-arena-data
1.91 MB
CodeSoft PRO
CodeSoft
AI & ML interests
Working on small language models
Recent Activity
updated a bucket about 12 hours ago
CodeSoft/slm-arena-data updated a bucket about 22 hours ago
CodeSoft/voting new activity 2 days ago
slmconsortium/README:vice president suggestionOrganizations
Posts 6
Post
2645
Sorbet Mini Experimental is out!
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
Post
142
Over the past week or so, I've been working on some models, those released being CodeSoft/sorbet-25m and CodeSoft/sorbet-v2-25m. In general, I'm a little confused because no matter what hyperparameters I change or datasets I add/remove, the benchmarks never move up.
In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.
Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.
Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
models 7
CodeSoft/sorbet-mini-experimental
Text Generation • 4.92M • Updated • 1.16k • 1
CodeSoft/sorbet-v2-25m
Text Generation • 25.2M • Updated • 1.45k
CodeSoft/sorbet-25m-gguf
Text Generation • 25.2M • Updated • 108
CodeSoft/sorbet-25m
Text Generation • 25.2M • Updated • 1.34k • 4
CodeSoft/MetaDiffusion-150M-ChatBase
Text Generation • 0.2B • Updated • 1.01k • 2
CodeSoft/MetaDiffusion-600M-ChatBase
Text Generation • 0.8B • Updated • 612 • 1
CodeSoft/MetaDiffusion-150M-exp
Text Generation • 0.2B • Updated • 226 • 2
datasets 0
None public yet