Language & perception / Knowledge book
Natural Language Processing
Before generation, there was representation.
From text and tokens to meaning, search, and language models.
Words are what we write. Meaning is what the model must discover.
Meaning here is represented through learned relationships, evaluated on specific language tasks.
Published in this topic
01Before generation, there was representation.
From term counts to embeddings: the choices that determine what a language system can compare.
This collection grows section by section. Concepts describe the intended coverage; related reading may introduce only part of a section.
Start Here
Follow the numbered sections in order, or choose the subject you need. Published articles are linked where they support the section.
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- What is NLP?Guide planned
- Language as dataBefore generation, there was representation.
- NLP applicationsGuide planned
Text Processing
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- CleaningGuide planned
- TokenizationBefore generation, there was representation.
- StemmingGuide planned
- LemmatizationGuide planned
- Stop wordsGuide planned
Classical Representation
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Bag of WordsBefore generation, there was representation.
- N-gramsGuide planned
- TF-IDFBefore generation, there was representation.
Word Embeddings
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Word2VecGuide planned
- GloVeGuide planned
- FastTextGuide planned
- Vector similarityBefore generation, there was representation.
Sequence Models
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- RNNGuide planned
- LSTMGuide planned
- GRUGuide planned
- Seq2SeqGuide planned
Related reading
No published reading is assigned to this section yet. Its place in the learning sequence is reserved as the collection grows.
Attention
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Attention mechanismAttention changed how information moves.
- Self-attentionAttention changed how information moves.
- Contextual representationGuide planned
Transformers
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Transformer architectureAttention changed how information moves.
- EncoderAttention changed how information moves.
- DecoderAttention changed how information moves.
- Encoder-decoderAttention changed how information moves.
BERT & Encoder Models
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- BERTGuide planned
- Masked language modelingGuide planned
- ClassificationGuide planned
- NERGuide planned
- EmbeddingsGuide planned
Related reading
No published reading is assigned to this section yet. Its place in the learning sequence is reserved as the collection grows.
Language Models
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Autoregressive modelsGuide planned
- Next-token predictionGuide planned
- Decoder-only modelsGuide planned
- LLMsGuide planned
Related reading
No published reading is assigned to this section yet. Its place in the learning sequence is reserved as the collection grows.
NLP Tasks
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Text classificationGuide planned
- SentimentGuide planned
- NERGuide planned
- SummarizationGuide planned
- TranslationGuide planned
- Question answeringGuide planned
Related reading
No published reading is assigned to this section yet. Its place in the learning sequence is reserved as the collection grows.
Search & Retrieval
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Lexical searchBuild the baseline before the pipeline.
- Semantic searchGuide planned
- EmbeddingsGuide planned
- Hybrid retrievalGuide planned
- RerankingGuide planned
Evaluation
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- Accuracy / F1Guide planned
- BLEUGuide planned
- ROUGEGuide planned
- Semantic metricsGuide planned
- Human evaluationGuide planned
Related reading
No published reading is assigned to this section yet. Its place in the learning sequence is reserved as the collection grows.
Modern NLP
Concepts in this section
Follow the related reading where available. Dedicated guides for the remaining concepts are planned.
- LLMsGuide planned
- RAGRAG gives a model evidence. Not certainty.
- Multilingual NLPGuide planned
- Multimodal language systemsGuide planned