tokenlane

An OpenAI-compatible gateway for vLLM with traffic isolation, admission control, and first-class observability.

Tokenlane is a high-performance inference gateway designed to sit between applications and one or more vLLM servers. It exposes an OpenAI-compatible API while protecting the underlying GPU infrastructure from overload and unpredictable latency.

I built Tokenlane to solve a common problem in self-hosted AI infrastructure: interactive requests and long-running batch workloads often compete for the same finite GPU capacity. Without explicit isolation, a large batch job can consume available concurrency and significantly degrade time-to-first-token for real-time users.

Tokenlane separates these workloads into independent real-time and batch traffic lanes. Each lane has its own concurrency limits, token-bucket rate limits, and bounded queue. When the system reaches capacity, excess work is rejected quickly rather than accumulating in an unbounded queue and causing latency to spiral.

What it provides

Tokenlane implements the familiar OpenAI API surface, including streaming and non-streaming chat completions, model discovery, file management, and asynchronous batch processing. Existing OpenAI-compatible clients can therefore use it with minimal configuration changes.

The gateway also provides:

• Independent admission control for real-time and batch traffic
• Predictable latency through bounded queues and fast overload rejection
• HMAC-SHA256 API key authentication without storing raw keys in configuration
• PostgreSQL-backed batch and job persistence
• Redis-backed distributed rate limiting
• S3-compatible storage for batch input and output files
• OpenTelemetry traces, metrics, and structured logs
• Prometheus metrics, health checks, OpenAPI documentation, and Swagger UI
• TOML configuration with environment-based secret injection

Architecture

Tokenlane is implemented as a modular Rust workspace. The server binary composes focused crates for the HTTP API, authentication, concurrency management, rate limiting, persistence, batch processing, object storage, telemetry, configuration, and the upstream vLLM client.

Requests enter through an Axum-based OpenAI-compatible API and are classified by workload type. Tokenlane then applies authentication, rate limiting, queue bounds, and concurrency admission before forwarding accepted inference requests to vLLM.

Batch jobs follow a separate asynchronous path, with their state persisted in PostgreSQL and their files stored through an S3-compatible API such as MinIO. This keeps long-running workloads durable without allowing them to monopolise the capacity required by interactive traffic.

Why I built it

Running an inference server is relatively straightforward. Running one reliably under mixed production workloads is much harder.

The difficult part is not merely forwarding HTTP requests. It is controlling how work enters the system, preserving capacity for latency-sensitive requests, rejecting overload cleanly, and exposing enough telemetry to understand how the system behaves.

Tokenlane treats those operational concerns as core product features rather than infrastructure that must be assembled separately around vLLM.

Status

Tokenlane is on hold — I’m not actively developing it while I don’t have a second GPU on hand to run and test against multiple vLLM instances locally. I’d like to pick it back up if that changes.

Technology

Tokenlane is built with Rust, Tokio, Axum, PostgreSQL, SeaORM, Redis, S3-compatible object storage, OpenTelemetry, and Prometheus.

The project is dual-licensed under Apache 2.0 and MIT.