Datasets

KernelGenBench provides five dataset variants for different evaluation scenarios.

Dataset Overview

Dataset	Operators	Sources	Platforms
KernelGenBench	210	ATen + vLLM + cuBLAS	NVIDIA only
KernelGenBench-nocublas	160	ATen + vLLM	NVIDIA only
KernelGenBench-aten	110	ATen only	All platforms
KernelGenBench-vllm	50	vLLM only	NVIDIA only
KernelGenBench-cublas	50	cuBLAS only	NVIDIA only

KernelGenBench (Full)

The complete benchmark for NVIDIA platforms.

Composition

Source	Count	Description
ATen	110	PyTorch operators
vLLM	50	Inference kernels
cuBLAS	50	BLAS routines

Use Case

• Comprehensive evaluation

• NVIDIA hardware only

• Full capability assessment

KernelGenBench-aten

Cross-platform operator set.

Composition

• 110 PyTorch ATen operators

• No external dependencies

Use Case

• Multi-chip evaluation

• Cross-platform testing

• Portable benchmark

KernelGenBench-vllm

LLM inference kernel benchmark.

Composition

• 50 vLLM operators

• Attention mechanisms (PagedAttention)

• KV cache management

• Quantization kernels

Use Case

• Inference optimization

• vLLM replacement testing

• Complex kernel evaluation

KernelGenBench-cublas

Linear algebra benchmark.

Composition

• 50 cuBLAS operators

• GEMM variants

• Multiple precisions (S/D/C/Z/H)

• Batched and strided versions

Use Case

• Performance ceiling testing

• BLAS replacement

• High-precision kernels

Dataset Selection

# Specify dataset
python scripts/generate_kernel_and_verify.py \
    --dataset KernelGenBench-aten \
    --server-type openai

On non-NVIDIA platforms, KernelGenBench-aten is automatically selected.