Getting Started

This section covers the requirements for installing vllm-plugin-FL and guides you through installing vllm, vllm-plugin-FL on different hardware platforms, and running a inference task.

• Requirements
  • Software Requirements
  • Supported hardware platforms
  • Model Compatibility
• Install software for running an inference task
  • Additional setup steps for running an inference task on Huawei Ascend
  • Additional setup steps for running an inference task with CUDA
  • Dispatch operators
• Run an offline batched inference