Megatron-LM-FL Documentation

• Megatron-LM-FL User Guide
  • Overview
    • Key Features
    • Megatron Core
    • Project Structure
  • Getting Started
    • Quick Start
    • Simple Training Example
    • LLama-3 Training Example
  • Installation
    • Docker (Recommended)
    • Pip Installation
    • Source Installation (Megatron-LM-FL)
    • System Requirements
  • Platform Support
    • Supported Platforms
    • Platform Selection
    • Plugin System
  • Training
    • Data Preparation
  • Parallelism Strategies
    • Data Parallelism (DP)
    • Tensor Parallelism (TP)
    • Pipeline Parallelism (PP)
    • Context Parallelism (CP)
    • Expert Parallelism (EP)
    • Parallelism Selection Guide
  • Performance Optimizations
    • Mixed Precision Training
    • Activation Checkpointing
    • Communication Overlap
    • Distributed Optimizer