Getting Started

This section covers the requirements for installing vllm-plugin-FL and guides you through installing vllm, vllm-plugin-FL on different hardware platforms, and running a inference task.

• Requirements
  • Software Requirements
  • Supported hardware platforms
  • Supported models
• Install software for running an inference task
  • Install from source
  • Install from docker image
  • (Optional) Dispatch operators
• Run an inference task
  • Run an offline batched inference
  • Run a serving inference task
• Qwen2.5-1.5B inference on Iluvatar BI-V150
  • Contents
  • 1. Model overview
  • 2. Environment requirements
  • 3. Installation
  • 4. Model download
  • 5. Inference scripts
  • 6. Running instructions
  • 7. Parameter tuning
  • 8. Benchmark performance
  • 9. FAQ