TOPIC AI
Pre-training training on massive unlabeled data
IN 10 SECONDS
The first phase of LLM training where a model learns language structures, grammar, and facts by predicting hidden tokens across web-scale text collections.
GOTCHA Pre-trained models are raw text-completers. If you ask: 'What is a loop?', they might reply with a list of other questions instead of an answer.
HOW PRE-TRAINING FLOWS
01 Data crawl collects raw text archives (Wikipedia, books, codebases).
02 Token mask masks words in sequences, forcing the model to guess hidden tokens.
03 Feedback loop adjusts parameter weights to minimize token guess errors.
POKE IT YOURSELF
python -c "from transformers import AutoModel; ..." — load a raw pre-trained checkpoint
Drill this topic →
~65 sec read