Data Preprocessing for Large Language Models

🚀 Announcing our new research on data efficiency for language model pre-training, it reduces data required to train Llama-1B by 22x: 🌟 CLIMB (Clustering-based Iterative Data Mixture Bootstrapping) 🌟Fresh on arXiv: https://lnkd.in/gfTXqd-n 📌 Challenge: Constructing optimal data mixtures for pre-training large language models (LLMs) is hard, given the enormous unlabeled web corpora with no domain indication. 📌 Our Solution: CLIMB introduces a scalable, iterative approach leveraging semantic clustering to identify the most impactful subsets of data: ➤ Embeds massive web-scale datasets. ➤ Uses k-means clustering to semantically partition the data (we analyze data on the web page). ➤ Trains a set of 100 proxy models (300M) with different cluster weighting. ➤ Iteratively refines data mixtures guided by a lightweight performance predictor. First it fits a model to predict proxy model performance, then samples mixures that maximize predictor output. 📈 Results: ➤ A 1B parameter model trained on our optimized 400B-token dataset (ClimbMix) surpasses LLaMA-3.2-1B accuracy by +2.0%, this is 22x reduction! ➤ Significant domain-specific boosts—training on our social-science optimized subset yields a +5% gain over random sampling. ➤ We introduce ClimbLab, a rich 1.2T-token, semantically clustered corpus across 20 distinct domains, available publicly. Both with CC license! 🛠 Practical Impact: ➤ Reduces unnecessary computation by focusing training on the highest-quality data. We observed that noisy data, and domains such as “advertisements” confuse the model.  ➤ Enables domain-specific fine-tuning with fewer resources and higher accuracy. This is helpful if you know the domain. ➤ ClimbMix (400B tokens) is a balanced dataset for ablation studies that results in high benchmarks numbers. 🔗 Read our paper: https://lnkd.in/gfTXqd-n 📂 Datasets available on Hugging Face with free license: https://lnkd.in/garzY6VF 🌐 Project page: https://lnkd.in/gx4p_BtK (check cluster visualizations) 🗨️ Discussion: https://lnkd.in/gY2A3dn5 👏 Huge thanks to the talented NVIDIA Research team behind this work: SHIZHE DIAO, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu (Danny) Y., Mostofa Patwary Yingyan (Celine) Lin, Jan Kautz, and Pavlo Molchanov. NVIDIA AI , NVIDIA

Before any prompt reaches a transformer, it’s broken down into tokens and that process is far more technical than it seems. Here’s a deep dive into how tokenization actually works. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐓𝐨𝐤𝐞𝐧𝐢𝐳𝐚𝐭𝐢𝐨𝐧? Tokenization is the process of converting text into discrete units called tokens. These tokens are then mapped to numeric IDs so the model can process them as input. But it’s not just splitting on spaces or punctuation. Tokenization is a compression and encoding strategy designed for efficiency, performance, and language adaptability. 𝐂𝐨𝐦𝐦𝐨𝐧 𝐓𝐨𝐤𝐞𝐧𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐀𝐥𝐠𝐨𝐫𝐢𝐭𝐡𝐦𝐬: 𝟏. 𝐁𝐲𝐭𝐞 𝐏𝐚𝐢𝐫 𝐄𝐧𝐜𝐨𝐝𝐢𝐧𝐠 (𝐁𝐏𝐄): Used by GPT models and many others. - Starts with individual characters - Iteratively merges the most frequent pair of tokens in the data - Continues until a predefined vocabulary size is reached - Efficient at compressing frequent patterns (like suffixes or word roots) 𝑬𝒙𝒂𝒎𝒑𝒍𝒆: "unbelievable" → ["un", "believ", "able"] "running" → ["run", "ning"] 𝟐. 𝐖𝐨𝐫𝐝𝐏𝐢𝐞𝐜𝐞: Used by BERT. - Similar to BPE, but uses likelihood based scoring - Helps handle out-of-vocabulary words with sub-word decomposition - Better suited for multilingual or morphologically rich languages 𝟑. 𝐒𝐞𝐧𝐭𝐞𝐧𝐜𝐞𝐏𝐢𝐞𝐜𝐞: Used by models like T5 and PaLM. - Treats input as a raw stream of Unicode characters - Doesn’t require whitespace for segmentation - Uses either BPE or Unigram LM under the hood - Can tokenize even languages without white space (like Chinese or Japanese) 𝐇𝐨𝐰 𝐭𝐡𝐞 𝐏𝐫𝐨𝐜𝐞𝐬𝐬 𝐖𝐨𝐫𝐤𝐬 (𝐒𝐭𝐞𝐩 𝐛𝐲 𝐒𝐭𝐞𝐩): 1. Input Text "AI is transforming industries." 2. Preprocessing Normalize white space, lowercase, remove/control special characters 3. Apply Tokenizer Algorithm Based on BPE, WordPiece, or SentencePiece logic Tokenize to: ["AI", " is", " transform", "ing", " industries", "."] 4. Map to Token IDs Each token is associated with an integer (e.g., ["AI"] → 3781), nothing but the Vocabulary ID. This numeric ID is what gets passed to the model 𝐖𝐡𝐲 𝐓𝐨𝐤𝐞𝐧𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐌𝐚𝐭𝐭𝐞𝐫𝐬? - Impacts how much context a model can handle (e.g., 4K, 8K, 100K tokens) - Affects model accuracy, especially with rare or domain-specific terms - Determines cost, speed, and performance in production 𝐏𝐫𝐨 𝐓𝐢𝐩𝐬: - Always count tokens, not characters, when designing prompts - Use tools from libraries like Hugging Face or OpenAI’s Tokenizer API to validate inputs - Optimize prompts and training data based on how your model segments and understands tokens Tokenization isn’t a preprocessing detail, it’s foundational. If you’re working with LLMs, understanding the underlying algorithm gives you control over performance, accuracy, and cost. #LLM #Tokenization #GenAI

A new video in my “Build A Large Language Model From Scratch” series is now live. In this tutorial, I cover text data handling, which is an important component for both understanding and training LLMs. In particular, the video walks through the following topics: - The process of tokenizing raw text and converting tokens into token IDs - Incorporating special context tokens and applying byte pair encoding - Techniques for data sampling using a sliding window - Setting up data loaders in PyTorch for efficient training Whether you are learning about how LLMs work for research, education, or production, I hope this look into text data handling provides useful insights! You can find the full video here: https://lnkd.in/gVi2Mxbh For reference, the table of contents is below: 00:00 Tokenizing text 14:02 Converting tokens into token IDs 23:56 Adding special context tokens 30:26 Byte pair encoding 44:00 Data sampling with a sliding window 1:07:10 Creating token embeddings 1:15:45 Encoding word positions

Training a Large Language Model (LLM) involves more than just scaling up data and compute. It requires a disciplined approach across multiple layers of the ML lifecycle to ensure performance, efficiency, safety, and adaptability. This visual framework outlines eight critical pillars necessary for successful LLM training, each with a defined workflow to guide implementation: 𝟭. 𝗛𝗶𝗴𝗵-𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗮𝘁𝗮 𝗖𝘂𝗿𝗮𝘁𝗶𝗼𝗻: Use diverse, clean, and domain-relevant datasets. Deduplicate, normalize, filter low-quality samples, and tokenize effectively before formatting for training. 𝟮. 𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮 𝗣𝗿𝗲𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Design efficient preprocessing pipelines—tokenization consistency, padding, caching, and batch streaming to GPU must be optimized for scale. 𝟯. 𝗠𝗼𝗱𝗲𝗹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗗𝗲𝘀𝗶𝗴𝗻: Select architectures based on task requirements. Configure embeddings, attention heads, and regularization, and then conduct mock tests to validate the architectural choices. 𝟰. 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 and 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Ensure convergence using techniques such as FP16 precision, gradient clipping, batch size tuning, and adaptive learning rate scheduling. Loss monitoring and checkpointing are crucial for long-running processes. 𝟱. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 & 𝗠𝗲𝗺𝗼𝗿𝘆 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Leverage distributed training, efficient attention mechanisms, and pipeline parallelism. Profile usage, compress checkpoints, and enable auto-resume for robustness. 𝟲. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 & 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻: Regularly evaluate using defined metrics and baseline comparisons. Test with few-shot prompts, review model outputs, and track performance metrics to prevent drift and overfitting. 𝟳. 𝗘𝘁𝗵𝗶𝗰𝗮𝗹 𝗮𝗻𝗱 𝗦𝗮𝗳𝗲𝘁𝘆 𝗖𝗵𝗲𝗰𝗸𝘀: Mitigate model risks by applying adversarial testing, output filtering, decoding constraints, and incorporating user feedback. Audit results to ensure responsible outputs. 🔸 𝟴. 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 & 𝗗𝗼𝗺𝗮𝗶𝗻 𝗔𝗱𝗮𝗽𝘁𝗮𝘁𝗶𝗼𝗻: Adapt models for specific domains using techniques like LoRA/PEFT and controlled learning rates. Monitor overfitting, evaluate continuously, and deploy with confidence. These principles form a unified blueprint for building robust, efficient, and production-ready LLMs—whether training from scratch or adapting pre-trained models.

🚀 Game-changer for LLM data prep: Microsoft's MarkItDown just went open source, and there's already a web version! Data scientists, this is the preprocessing swiss army knife you've been waiting for. Convert ANY file (PDFs, PPTs, Excel, images, audio) into clean, structured markdown - perfect for LLM training data preparation. The hidden complexity in LLM training isn't model architecture - it's data preparation. This tool solves that with: - Zero-config document structure preservation - Native LLM integration for image understanding - Built-in OCR and speech transcription - Batch processing capabilities - Clean, consistent markdown output The architecture uses format-specific handlers in a modular pipeline, allowing teams to process diverse data sources without writing custom parsers. The output is consistently structured markdown, eliminating those pesky preprocessing inconsistencies that plague LLM training datasets. Teams report cutting data preparation time by up to 70%. One data scientist mentioned reducing their preprocessing pipeline from 200+ lines to just 10. Best part? Choose your workflow: - Direct Python library integration for pipelines - Someone has already built a web interface for quick conversions (link in comments) - Full control over LLM integration For the ML architects: How would you handle edge cases in document structure preservation while maintaining conversion speed at scale? #MachineLearning #DataScience #LLM #AITools #OpenSource

Everyone interacts with ChatGPT. But can you build it from scratch? I received a PhD in Machine Learning from MIT in 2022. Then discovered my passion in teaching machine learning from scratch. 3 months back, I started a project to teach “How to Build Large Language Models from scratch” without any libraries! The goal is to empower students and industry professionals to master the building blocks of large language models and ChatGPT. The result is a mega-project with 15 videos covering everything about large language models. I have uploaded all videos on Youtube. Lecture 1: Building LLMs from scratch: Series introduction https://lnkd.in/dFJgqxxf Lecture 2: Large Language Models (LLM) Basics https://lnkd.in/dfACgdPX Lecture 3: Pretraining LLMs vs Finetuning LLMs https://lnkd.in/dga-NhmN Lecture 4: What are transformers? https://lnkd.in/dicb7rEk Lecture 5: How does GPT-3 really work? https://lnkd.in/dxt4GSkS Lecture 6: Stages of building an LLM from Scratch https://lnkd.in/dwfU8R5d Lecture 7: Code an LLM Tokenizer from Scratch in Python https://lnkd.in/d_6XC9PE Lecture 8: The GPT Tokenizer: Byte Pair Encoding https://lnkd.in/d4yZyNsT Lecture 9: Creating Input-Target data pairs using Python DataLoader https://lnkd.in/dbKVPCgt Lecture 10: What are token embeddings? https://lnkd.in/duVKJKvz Lecture 11: The importance of Positional Embeddings https://lnkd.in/dsP7vGJ5 Lecture 12: The entire Data Preprocessing Pipeline of Large Language Models (LLMs) https://lnkd.in/dFHfKPtc Lecture 13: Introduction to the Attention Mechanism in Large Language Models (LLMs) https://lnkd.in/d_kVY-Q2 Lecture 14: Simplified Attention Mechanism - Coded from scratch in Python | No trainable weights https://lnkd.in/dk7U7C5s Lecture 15: Coding the self attention mechanism with key, query and value matrices https://lnkd.in/dYR8u_Fp I have spent a lot of time and effort in making these lectures. I show everything on a whiteboard and then show it through Python code. Nothing is assumed. Everything is spelled out. P.S: Want to learn about all of this live - with me as your course instructor? Join our live bootcamp starting from January 3rd week 2025: https://vizuara.ai/spit/ 100+ students have already registered and we are closing registrations very soon!