Operator Dispatch User Guide

The dispatch system provides three layers of operator replacement. You can control each layer independently and flexibly.

The dispatch system supports both YAML configuration and environment variables for fine-grained control. Environment variables take precedence over YAML config.

The priority chain is as follows:

SGLANG_FL_* env vars > YAML config (SGLANG_FL_CONFIG) > Platform auto-detect YAML > Code defaults

• Dispatch through YAML config file
  • Config Fields
  • Common Recipes
• Dispatch through environment variables
  • Layer 2 — Fused Op Dispatch
  • Layer 1 — ATen Replacement (FlagGems)
  • Layer 3 — Distributed Communication
  • System / Debug
  • Examples
• Debugging and diagnostics
  • Dispatch log
  • ATen replacement log
  • Troubleshoot numerical precision issues through Precision Bisection
  • Common Issues
• Vendor integration
  • 1. Backend class (my_chip.py)
  • 2. Registration (register_ops.py)
  • 3. Operator implementations (impl/)
  • Vendor’s backend auto-discovery