LLM Gateway

Forge LLM Gateway provides a single governed endpoint for model traffic. It centralizes authentication, identity attribution, model access, routing, policy enforcement, budgets, and telemetry while preserving the request formats applications already use.

Forge LLM Gateway showing endpoint setup, providers, access profiles, gateway keys, usage, and spend — LLM Gateway

How it works

Object	Purpose
Provider	Stores the upstream connection and credential mode.
Access profile	Defines the models, providers, data classes, policy checkpoints, and enforcement mode available to a caller.
Route	Maps a requested model and API surface to an upstream provider and model.
Gateway key	Authenticates one identity and assigns it to one access profile.
Service account	Represents a non-human app, agent, bot, or integration.
Budget	Applies spend and token limits to an access profile, key, or subject.

A request authenticates with a gateway key. Forge resolves the key’s identity and access profile, validates the requested model and API surface, applies the relevant policy and budget controls, selects an eligible route, and records the result.

With a supported network integration or Forge for devices, Forge can automatically route supported model traffic through the LLM Gateway. Teams get one governed path without reconfiguring every application.

Traffic entry

Traffic reaches the LLM Gateway through three supported paths:

Path	Best for	Experience
Network integration	Agentless coverage for traffic crossing the corporate network	Forge automatically routes supported model traffic through the LLM Gateway.
Forge for devices	Managed devices, including users working off-network	Forge automatically routes supported model traffic without per-application configuration.
Direct configuration	Applications and services with an explicit gateway connection	The application uses the Forge base URL and a gateway key.

Network routing gives organizations agentless coverage from an existing network control point. Forge for devices extends the same experience to managed endpoints wherever they work. Direct configuration remains available for teams that want to connect an application or service explicitly.

Providers

Forge maintains a catalog of supported providers and the API surfaces available for each one. Forge supports OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Vertex AI, Mistral, Cohere, xAI, Databricks, OpenRouter, and custom OpenAI-compatible and Anthropic-compatible providers. Provider credentials can use a Forge-managed secret reference, a customer-managed credential, or no credential when the upstream connection does not require one. Plaintext credentials are accepted only when a provider is configured and are never returned by the API. Forge reports each provider’s configuration state and health separately. Configuration states include configured, disabled, error, and deleted; health can be healthy, degraded, unhealthy, down, or unknown.

API surfaces

Routes are scoped to an API surface so a caller cannot use an upstream capability merely because it can reach the provider.

Family	Supported surfaces
OpenAI-compatible	Chat Completions, Responses, Completions, Embeddings, Images, Audio, Models, Files, Batches, Fine-tuning, Moderations, and Realtime
Anthropic-compatible	Messages
Cloud providers	Bedrock Converse and Gemini Generate Content
Other	Rerank, provider passthrough, and custom surfaces

The provider catalog is the source of truth for which surfaces a specific provider supports.

When policy enforcement is enabled, OpenAI Responses requests must include the complete provider-visible context in input. Forge rejects previous_response_id continuations because the referenced provider-side history cannot be evaluated or transformed by the gateway. Monitor and simulate modes may observe these continuations without claiming full-context enforcement.

Access profiles

An access profile is the reusable control plane for a class of gateway traffic. It defines what a caller assigned to the profile may request and what controls Forge applies. Access profiles configure:

Model selectors and provider selectors.
Data classes used by policy evaluation.
Prompt, pre-tool-use, and post-tool-use policy checkpoints.
Privacy mode for retained gateway data.
Enforcement mode and lifecycle state.

Setting	Values
State	`draft`, `active`, `disabled`, `archived`
Enforcement	`monitor`, `simulate`, `enforce`, `break_glass`
Checkpoint	`prompt`, `pre_tool_use`, `post_tool_use`
Privacy	`metadata_only`, `redacted_summary`, `hash_only`

Profiles are versioned so changes to model access and enforcement can be tracked. Identity assignment belongs to gateway keys: each key binds its subject to one access profile.

Routing

A route matches an access profile, requested model pattern, and API surface, then directs the request to a provider. A route can preserve the requested model name or rewrite it to a different upstream model.

Setting	Purpose
Provider	Upstream destination for the request.
Model pattern	Requested model names eligible for the route.
Upstream model	Optional model name sent to the provider.
API surface	Protocol and operation exposed by the route.
Strategy	How Forge selects among eligible destinations in the same tier.
Priority and weight	Fallback-tier ordering and distribution controls.
Rollout state	Whether the route is being evaluated, enforced, paused, or retired.
Enforcement mode	Policy behavior applied on the route.
Policy checkpoints	Prompt and tool-use evaluation points.
Tool-deny behavior	Hard-block the call or return a rewritten refusal.

Tiers and strategies

Routes with the same priority form a tier. Forge tries lower priority numbers first and moves to later tiers only when the earlier tiers have no eligible destination or their attempts fail. Later tiers are therefore the fallback path; fallback is not a strategy you select. The strategy controls selection among eligible destinations within one tier:

Strategy	Behavior
`fixed`	Use the tier’s single matching destination.
`weighted`	Distribute traffic among destinations according to their configured share.
`policy`	Use deterministic policy ordering among eligible destinations.
`cost`	Prefer the eligible destination with the lowest configured estimated cost.
`latency`	Prefer the eligible destination with the best observed response latency.

Forge excludes degraded or unhealthy destinations during their cooldown and uses eligible destinations in the same tier before moving to the next tier. A successful later probe restores a destination to healthy. Buffered request surfaces can fail over before a response is returned; Forge does not replay a stream after its response has begun. Routes can also set same-destination retries, cooldown periods, and provider-specific request overrides. Provider-owned and secret-bearing fields cannot be supplied through request overrides. Route rollout states are draft, monitor, simulate, enforce, paused, and archived. Tool denials can use hard_block or rewrite_refusal.

Gateway keys

Gateway keys authenticate callers and connect each caller to an access profile. A key binds one subject of type user, group, app, agent, service_account, or customer_tenant. Token classes include personal, group, service_account, app, agent, customer_tenant, and exchanged. Keys can expire and can be rotated, transferred, revoked, disabled, re-enabled, or deleted. Their runtime state is active, revoked, or expired.

Forge displays a new gateway key’s plaintext secret once. Store it in the application’s secret manager before leaving the creation flow.

Service accounts

Service accounts give non-human workloads a durable identity rather than sharing a person’s key. They can represent applications, agents, bots, and integrations. A service account records its owner, environment, rotation policy, and state. The owner can be a directory user, directory group, or app integration. States are active, disabled, and archived.

Budgets

Budgets can be attached to an access profile, gateway key, or subject. A budget can enforce any combination of:

Spend in USD.
Input tokens.
Output tokens.
Total tokens.

Supported windows are daily, weekly, monthly, rolling_24h, rolling_7d, and rolling_30d. Calendar windows reset in UTC; rolling windows move continuously. When either a configured spend or token limit is reached, Forge blocks subsequent requests in that budget window.

Switch an app

Point the existing SDK at the Forge endpoint and replace the provider credential with a scoped gateway key.

import OpenAI from 'openai'

const client = new OpenAI({
  apiKey: process.env.FORGE_LLM_GATEWAY_KEY,
  baseURL: 'https://gateway.forge.example/llm-gateway/v1',
})

import Anthropic from '@anthropic-ai/sdk'

const anthropic = new Anthropic({
  apiKey: process.env.FORGE_LLM_GATEWAY_KEY,
  baseURL: 'https://gateway.forge.example/llm-gateway/v1',
})

import { createOpenAI } from '@ai-sdk/openai'

export const forgeOpenAI = createOpenAI({
  apiKey: process.env.FORGE_LLM_GATEWAY_KEY,
  baseURL: 'https://gateway.forge.example/llm-gateway/v1',
})

const model = new ChatOpenAI({
  apiKey: process.env.FORGE_LLM_GATEWAY_KEY,
  configuration: {
    baseURL: 'https://gateway.forge.example/llm-gateway/v1',
  },
})

import { Settings } from 'llamaindex'
import { OpenAI } from '@llamaindex/openai'

Settings.llm = new OpenAI({
  apiKey: process.env.FORGE_LLM_GATEWAY_KEY,
  additionalSessionOptions: {
    baseURL: 'https://gateway.forge.example/llm-gateway/v1',
  },
})

curl https://gateway.forge.example/llm-gateway/v1/chat/completions \
  -H "Authorization: Bearer $FORGE_LLM_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}]}'

Use the endpoint displayed in your Forge workspace in place of the example hostname above.

Session continuity

To group multiple model requests from one logical agent interaction, generate a high-entropy session hint and send the same value unchanged on every request in that interaction:

-H "X-Forge-Session-Id: $YOUR_STABLE_SESSION_HINT"

Forge scopes the hint to the authenticated organization, access profile, key, and subject before using it for telemetry. The X-Forge-Session-Id response header contains Forge’s canonical session ID for observability. Treat that response value as read-only: continue sending your original hint rather than using the returned canonical ID as the next request’s hint.

User attribution

The gateway key authenticates the application or service account. When the application acts for an end user, it can also pass a verified identity token:

-H "X-Forge-Identity-JWT: $CUSTOMER_IDP_JWT"

Forge validates the token’s issuer, audience, signature, expiration, and mapped claims before using it to attribute the request to a user.

Policy checkpoints

Profiles and routes can invoke policies at three points:

Checkpoint	Evaluation point
`prompt`	Before prompt content is sent upstream.
`pre_tool_use`	Before a requested tool executes.
`post_tool_use`	After tool execution, when its result is available.

The profile and route enforcement modes determine whether a matching policy is observed, simulated, enforced, or processed under break-glass behavior. Tool denials can stop execution or return a rewritten refusal when that behavior is configured on the route.

Observability

Forge records the requested and served model, selected provider and route, identity, policy outcomes, tokens, cost, latency, retries, fallbacks, errors, cache hits, and denied tool calls for routed traffic. Gateway analytics include:

Request and session volume.
Input, output, and total tokens.
Provider-reported or Forge-estimated cost.
Policy hits and denied tool calls.
Route health, errors, retries, fallback rate, and cache hits.
Bypass findings for observed traffic that did not use the gateway.

Get started

Observe

Secure

Enable

Integrations

Resources

How it works

Traffic entry

Providers

API surfaces

Access profiles

Routing

Tiers and strategies

Gateway keys

Service accounts

Budgets

Switch an app

Session continuity

User attribution

Policy checkpoints

Observability

​How it works

​Traffic entry

​Providers

​API surfaces

​Access profiles

​Routing

​Tiers and strategies

​Gateway keys

​Service accounts

​Budgets

​Switch an app

​Session continuity

​User attribution

​Policy checkpoints

​Observability

How it works

Traffic entry

Providers

API surfaces

Access profiles

Routing

Tiers and strategies

Gateway keys

Service accounts

Budgets

Switch an app

Session continuity

User attribution

Policy checkpoints

Observability