Providers

A provider is an endpoint heph can send requests to, and how to authenticate. Configure them under [providers.<id>].

[providers.anthropic]
api_key_env = "ANTHROPIC_API_KEY"

[providers.bedrock]
region = "us-west-2"
profile = "work"

[providers.bedrock-openai]
region = "us-east-1"
profile = "work"

[providers.openai]
api_key_env = "OPENAI_API_KEY"

[providers.gateway]
kind = "messages"
base_url = "https://llm.internal.example/anthropic/"
api_key_command = "pass show llm-gateway"
headers = { "X-Team" = "platform" }

[providers.gateway.models."claude-sonnet-4-6"]
context_window = 200000
default_max_tokens = 8192
cost_per_1m_in = 3.0
cost_per_1m_out = 15.0
cost_per_1m_in_cached = 3.75
cost_per_1m_out_cached = 0.3
can_reason = true
supports_attachments = true

Catalog providers and custom providers

Heph ships a catalog of providers and models, with endpoints, rates, and capabilities; heph models prints it. A provider whose id is in the catalog (anthropic, bedrock, bedrock-europe, bedrock-openai, openai, and others) needs only authentication in config; everything else comes from the catalog, and any key you give overrides the catalog's value.

A provider whose id is not in the catalog is a custom one: a gateway, a proxy, or a local server. A Messages or Responses endpoint must give base_url and at least one model under models, with rates. Native Ollama endpoints instead discover their installed models; they need no manual catalog.

The [providers] section replaces the environment's choice entirely: once you name any provider, heph no longer looks at ANTHROPIC_API_KEY or AWS credentials to decide which provider to use. It still reads them for authentication where config says so.

Keys

project, for vertex, is accepted with the section but unused until Vertex lands.

Three protocols

The messages and bedrock kinds speak Anthropic's Messages API and carry Claude. The responses and bedrock-responses kinds speak OpenAI's Responses API and carry GPT: openai in the catalog is the first, with a bearer key; bedrock-openai is the second, signed with the same AWS credential chain as bedrock and reaching the us. and global. inference profiles (global.openai.gpt-5.6-sol and the like).

A conversation is stored in one form whichever model wrote it, and heph translates it for a GPT provider on every request, so you can point a tier at GPT and another at Claude in the same project, and switch a conversation between them across turns. What does not carry across is the model's private reasoning: each API reads only its own, so heph stores GPT's reasoning items as the API returns them and sends them back the same way, as it does Claude's thinking, and a tier's providers must all speak one protocol. Naming a Claude provider and a GPT provider in one tier is a config error that says which protocols it found.

GPT reasoning is billed as output whether or not the endpoint returns a summary. When a summary is returned, heph displays it in the folded thinking line. Cache reads on Bedrock show from the second request onward and can lag the first by a moment.

Maintained Bedrock metadata

The embedded catalog includes AWS-verified US/EU Claude metadata and US/Global GPT Runtime profiles, including Claude Opus 5.5, Sonnet 5.5, Haiku 5.5, US Fable 5.1, and GPT-6 Astra, Sol, and Luna. Existing provider defaults are unchanged. Every included GPT family uses Standard pricing with the exclusive 272000-input-token tier and the US CRIS premium. Haiku 5.5 reprices the whole request at five times its short-context rates above 100000 input tokens, including cached input. Opus 5.5 and Haiku 5.5 default to medium effort in their API metadata, while Sonnet 5.5 defaults to high. An explicit tier effort still wins.

The US/EU Claude providers are replaced by a maintained file after loading the Catwalk snapshot, and GPT is added by a separate supplement. make catalog does not overwrite either file. See the AWS model cards and the verification notes in crates/heph-catalog/data/README.md for pricing sources and scope. Pricing is frozen per request, so a catalog refresh does not reprice history.

The old EU Opus 4.1 and US/Global GPT-5.4/5.5 Runtime IDs are not supported and have been removed. AWS lists GPT-5.4 and GPT-5.5 as Mantle-only, not Runtime profiles. Opus 5.5 and Fable require thinking; set thinking = "adaptive" on their tiers, rather than "off". The catalog does not select Ultrafast service or include access-restricted cybersecurity model families.

Explicit model pricing in user/project config continues to replace catalog pricing, including the basis and tiers. Remove or update stale overrides if you want the refreshed catalog prices; declaring only capabilities still replaces pricing with the config's default zero rates.

The openai-compat providers the catalog lists (heph models --all) are not loaded on their own, since most speak only Chat Completions, which heph does not. One that speaks the Responses API is enabled by naming the kind:

[providers.xai]
kind = "responses"

The endpoint, key variable, and models then come from the catalog as for any catalog provider.

Declaring models

[providers.<id>.models."<model-id>"] declares a model, or adds to a catalog model's description:

When a declared model is also in the catalog, the rates always come from config, because declaring a model is how you say what it costs you (a gateway that bills differently, a local model that costs nothing); everything else comes from config where given and the catalog otherwise. A model's reasoning_levels, which decide whether it takes an effort level or a thinking budget, come from the catalog only.

Context-dependent prices

For an endpoint that changes prices above a token-count threshold, supply all four rates in each tier (USD per million). context_tier_basis selects the threshold count:

• "input" (default): uncached input plus cache reads and writes.

• "output": all generated tokens, including reasoning.

• "total": input plus output, including cached input and reasoning.

The following example matches AWS Standard Global CRIS pricing for GPT-6.1 Sol, verified in October 2026. These prices are already in the bundled catalog; this shows how to override pricing explicitly:

[providers.bedrock-openai.models."global.openai.gpt-6.1-sol"]
cost_per_1m_in = 2.0
cost_per_1m_out = 10.0
cost_per_1m_in_cached = 2.5  # cache write
cost_per_1m_out_cached = 0.1 # cache read
context_tier_basis = "input"

[[providers.bedrock-openai.models."global.openai.gpt-6.1-sol".context_tiers]]
above = 272000
input = 4.0
output = 15.0
cache_write = 5.0
cache_read = 0.2

above is exclusive: exactly 272000 input tokens use base rates; 272001 input tokens reprice all input, cached input, and output at long-context rates. The largest matching threshold wins, regardless of order. Rates must be finite and nonnegative (zero-cost models are allowed), thresholds must be distinct, and every tier must specify all four rates. A matched tier prices the whole request, not just excess tokens; one-hour cache writes remain twice the selected input rate. Graduated per-excess-token pricing is not supported.

For an output-length threshold, set context_tier_basis = "output" in the model table, before its [[context_tiers]] entries. Use "total" when the threshold is based on combined input and output length. Estimates use estimated input and zero output. Final provider usage can cross a threshold the estimate did not.

Request rows freeze base rates, the basis, and tiers before sending, and final usage chooses the applicable tier. Later config/catalog updates do not reprice historical requests. Older stored pricing without a basis keeps input-based selection. Declaring a catalog model replaces its entire pricing definition, including the basis and tiers. User/project provider tables merge by provider ID, with the entire project provider definition winning for the same ID.

The bundled us.openai.gpt-6.1-sol entry includes AWS's 10% US CRIS premium on both base and long-context prices. The bundled prices are for Standard requests; heph does not select Ultrafast or automatically match rates to a service tier. Declaring a provider in config still requires its normal authentication and endpoint settings.

Native Ollama (experimental)

With a running Ollama server and an already installed tool-capable model:

heph --model ollama/qwen3-coder:latest
heph models --provider ollama

No key or configuration is needed. Heph does not install Ollama, download weights, or choose among models. Discovery lists exact names, digests, capabilities, parameter sizes, quantizations, and context metadata; unknown values are not guessed. The name after the first / stays intact, including namespaces and quantization tags. An omitted tag resolves to the installed :latest name only.

The reserved ollama provider uses http://127.0.0.1:11434, overridden by OLLAMA_HOST or, with highest priority, an explicit base_url. Host/port forms are accepted; wildcard bind addresses and API paths are refused.

[providers.ollama]
kind = "ollama"
# Optional runtime allocation and residency policy:
num_ctx = 32768
keep_alive = "5m"

[tiers.local]
models = ["ollama/qwen3-coder:latest"]
roles = ["coding", "review", "design", "research", "summarize", "quick", "commit-message", "title", "progress-check"]
max_tokens = 4096

A custom server ID uses kind = "ollama" and its server-root base_url. Optional api_key_env, api_key_command, and headers can authenticate a reverse proxy. Loopback bypasses environment proxies; redirects are refused. Remote plaintext HTTP is warned about. keep_alive accepts a duration string or integer seconds; omit it to keep server residency policy. num_ctx and keep_alive are rejected on other kinds.

The runtime context is the model's explicit context_window, else provider num_ctx, else the model-configured num_ctx, else 32768. Explicit settings above a known architecture ceiling fail; inherited values are clamped. The architectural ceiling is shown separately and stays unknown if not advertised. Preflight reserves the requested output (including reasoning), 2048 template tokens, and 4096 per image, with an approximate text estimate of bytes divided by three rounded up. Requests that do not fit fail before inference; allocation errors suggest lowering context explicitly, never automatic resizing.

Unknown tools require supports_tools = true in the model table. Optional supports_images and thinking_controls = { values = [false, true] } fill missing metadata or restrict features; they cannot enable known unsupported features. No tier thinking setting means the server's default; explicit off or named effort must match supported controls. Thinking is stored and replayed for the same native model; other protocols and models omit it.

The progress-check role uses the same native route and tier controls. It offers only report_progress and validates the exact report because Ollama cannot force tool selection. Output is capped at 2048 tokens and the existing classifier deadline remains ten seconds; invalid, incomplete, or late reports require human approval rather than automatic renewal or cloud substitution.

Text-file attachments remain text and need no vision. Images require vision, including replayed history. Inline base64 PNG, JPEG, GIF, and WebP are accepted, with limits of 10 MiB, 32 million pixels each, 16 images across history, and 40 MiB per serialized request. Decoding is limited to 128 MiB allocation. PDFs, remote image URLs, and unsupported content fail rather than disappear.

There is one active inference per endpoint in a process. Foreground requests precede queued background naming; cancellation releases the client permit, not a promise of instant GPU reclamation. Connect, discovery, initial-response, and stream-inactivity timeouts are 10, 30, 300, and 120 seconds respectively. Tools run only after a valid terminal response. Zero default rates mean zero API charges, not zero hardware cost. Native timings, queue wait, endpoint, and digest are recorded; missing usage is explicitly incomplete.

A loopback server may route to cloud models. Chat discovery refuses known cloud-backed models and warns about unknown locality. A separate decide tier accepts any configured model without discovery or locality checks; see decision tiers. Set OLLAMA_NO_CLOUD=1 on the server and restart it for a stronger boundary. Network tools retain their ordinary permissions. Route every role locally to avoid cloud inference.

Tool replay and PNG/JPEG/GIF/WebP smoke tests passed on Ollama 0.34.2 with installed qwen3.5:9b; this does not establish a minimum supported version. Older-version and manual interactive release validation remain outstanding.

A compatibility-endpoint local model

[providers.local]
kind = "messages"
base_url = "http://localhost:8080/v1"
api_key_env = "LOCAL_KEY"

[providers.local.models."qwen3-coder"]
context_window = 128000
default_max_tokens = 8192

[tiers.offline]
models = ["local/qwen3-coder"]
roles = ["quick"]

Rates default to zero, so the status bar's cost stays at $0.00 for that tier. A messages provider needs a key source even if the server ignores the key: without api_key_env or api_key_command, the first turn fails with no api_key_env or api_key_command configured. Point it at any set variable.

Keeping Bedrock credentials fresh

Heph asks the AWS credential chain for credentials when the ones it holds are a minute from expiring, so any source the chain can renew keeps a long session signed in: aws-vault's credential server (aws-vault exec --ecs-server work -- heph), an SSO profile, or a credential_process profile. What cannot be renewed is a set of fixed credentials exported into the environment, which is what plain aws-vault exec work -- heph provides; when they expire, heph says so and names the alternatives. Bedrock Credentials walks through setting up each one.

Bedrock regions

The catalog has two Bedrock entries, because the inference profile ids differ: bedrock for US regions (us.anthropic.claude-...) and bedrock-europe for EU regions (eu.anthropic.claude-...). On the first run heph picks by AWS_REGION: an eu-* region gets bedrock-europe. If you configure [providers.bedrock] with an EU region, use the eu. model ids in your tiers, or configure [providers.bedrock-europe] instead.