falcon 40b
tiiuae/falcon-40b
Open Source · chat · open-weights
Context
2K
Max output
—
Pricing
Open weights
Modalities
text
Released
24 May 2023
License: apache-2.0 · tiiuae/falcon-40b
AI summary
● machine-written
Falcon 40B: 40B-parameter open-weights LLM trained on 1 trillion tokens
Falcon-40B is a 40 billion parameter causal decoder-only model developed by the Technology Innovation Institute, trained on 1 trillion tokens of RefinedWeb-enhanced data. The model implements rotary positional embeddings, multiquery attention, and FlashAttention for improved efficiency. It is available under the Apache 2.0 license for both commercial and research use.
What's new
- Trained on 1 trillion tokens of RefinedWeb-enhanced corpora
- Implements rotary embeddings and multiquery attention with FlashAttention
- Supports English plus German, Spanish, French, and other languages
- Parallel attention/MLP decoder design with dual-layer normalization
Best for
Open-source text generation and language understanding tasksResearch and commercial applications requiring permissive licensingDeployment environments with GPU infrastructure (tested on A4000x2)Applications needing multilingual capabilities across 10+ languages