basically AIOpen language models / Released
Pebble.
Start small.
A family of 10M and 25M parameter language models. Openly released, with plenty to learn from.
The idea
Small models.
Real questions.
Pebble is our small language model family: base models for further training, and instruction-tuned chat variants built on top.
Small models give us a manageable place to explore training and language model behavior. We share the weights so others can inspect the work and learn alongside us.
These are limited models. The chat variants use basic instruction fine-tuning, and the family has a 512-token context window. Treat Pebble as an open research release, with its capabilities and constraints in view.
The lineup
Meet the family.
Read each model card for usage details before getting started.
| Model | Parameters | Context tokens | Type | Hardware | Explore |
|---|---|---|---|---|---|
| Pebble 25M | 25M | 512 | Base | CUDA required | Model card ↗ |
| Pebble 10M | 10M | 512 | Base | CUDA required | Model card ↗ |
| Pebble 25M Chat | 25M | 512 | Basic IFT | CUDA required | Model card ↗ |
| Pebble 10M Chat | 10M | 512 | Basic IFT | CUDA required | Model card ↗ |
Hardware note: these releases require a CUDA-capable NVIDIA GPU. They do not currently run on CPU, AMD, or Intel hardware.
Got a good
weird idea?
Questions, collaborations, or a simple hello. We're listening.