Skip to content

halogen environment flags

Everything is configured by environment variable; there is no config file. All flags are read once at startup.

This lists the flags used to deploy and operate halogen. The engine carries additional kernel-tuning levers that are not documented here: they select internal implementation variants, they are not needed to run the server, and the shipped defaults are the measured winners. Changing what is listed below is supported; anything else is not.

class meaning
BITWISE output is byte-identical. Safe to change.
NUMERIC output can change.

Every flag below is BITWISE, deployment policy rather than arithmetic, except HALOGEN_CK_OVERLAY, which selects which weights run, HALOGEN_MATMUL_TUNING_FILE, which changes the order a sum is accumulated in, HALOGEN_INDEXER_BUDGET, which changes how much of the context the model attends, and HALOGEN_COMPOSABLE_CONTEXT, whose reused answers are close to but not byte-identical to fresh ones. Each is marked, and each was measured before being offered.

Model and tokenizer

flag default meaning
HALOGEN_CHECKPOINT (unset: /models/qwen38-flash-next-v2.hgn) Path to the .hgn checkpoint the engine loads. Unset (0.15 on), the server serves qwen38-flash-next-v2.hgn, one file with no sidecar and its lookup table (qwen38-flash-next-ngram.hgn) beside it, except on a volume that holds 0.14's qwen38-flash-next-w4b.hgn and not v2: there it keeps serving w4b, downloads nothing, and says at startup how to switch. Set, it is used as given. w4b is two files: the checkpoint and the quality sidecar <checkpoint>.overlay.hgn beside it, which the engine picks up on its own; the server says at startup which precision it is about to run, and warns if w4b's sidecar is absent. Since 0.7.0 it may name a llama.cpp GGUF of the model instead (any shard of a split; the siblings are found by name): the engine repacks it into RAM at startup, losslessly, reads its lookup table in place and takes the draft head from HALOGEN_MTP_HEAD. The README's Bring your own GGUF has which files are read and what it costs; since 0.12.1 the image's convert mode writes that repack out as a complete .hgn this variable can name instead (no GGUF beside it, no sidecar looked for).
HALOGEN_DOWNLOAD (unset = off) A Hugging Face repo id to fetch the weights from on first start (peonist-ai/halogen-qwen3.8-flash-next). Off by default, which is what keeps the container free of all outbound connections. It fetches the chosen checkpoint and what it needs, not the whole repo: v2's lookup table (or w4b's quality sidecar), the vision tower and the tokenizer; the free space is checked against their sizes before the first byte. It fires only when one of those is missing (a start with it set and the volume writable also refreshes w4b's 2.4 GiB quality sidecar when the one on disk predates the image; a checkpoint on disk is never re-fetched); partial transfers resume. Needs the models volume mounted read-write. The checkpoint's presence is verified after the download rather than trusted, so a truncated tree fails here rather than later and more confusingly in the engine. BITWISE: it moves bytes onto disk, it does not change arithmetic.
HALOGEN_MTP_DEPTH unset How many tokens the draft head proposes in a row for a greedy request (the head drafts, the model verifies every token, so the output is the same at any depth; only the speed changes). Unset, the server chooses the depth for each conversation, 2 or 3, which is the best all-round setting. A number fixes that depth for every request; 3 is slower than 2 on prose and agent traffic. It is capped at HALOGEN_SPEC_N minus one.
HALOGEN_MTP_HEAD qwen38-flash-next-mtp.hgn beside the checkpoint The engine's draft head for a GGUF trunk (1.4 GiB, on the weights repo; fetched by HALOGEN_DOWNLOAD when the volume is writable). A GGUF carries no draft head this engine can run, so without this file a GGUF does not start, and the entrypoint says so before anything is loaded. Not read for the engine's own checkpoint, which carries its head.
HALOGEN_GGUF_CACHE (unset) 1 writes the repacked weights out once beside the GGUF as a file the engine can load directly on later starts (about 70 GiB for an 8-bit trunk; =<dir> puts it elsewhere; the volume must be writable and have the room, or the write is skipped with a line and the server starts anyway). It is checked against the shards' sizes and modification times on every start and is never used stale: rebuilt with this set, ignored without, both said in the log. Measured on the reference machine: a restart is 1.4 s from a warm file and 16 s from a cold one, against the repack's 9 s warm and 18 s cold, so it buys seconds, not minutes; it is for a host that restarts often from a warm cache.
HALOGEN_GGUF_THREADS 8 Workers for the repack of a GGUF at startup. The output does not depend on the count. 8 and 16 read the reference machine's NVMe drive at the same rate, so more only helps a host whose CPU is slower than its drive.
HALOGEN_CK_OVERLAY (unset = the sidecar) w4b only: which precision sidecar to read (v2 carries its own precision and takes none). Unset means the quality sidecar beside the checkpoint, which is the default and needs no action. Point it at …overlay-speed.hgn to trade the o_proj calibration for about 2% of decode, or none to run the bare 4-bit checkpoint, which is the measurement control rather than a serving configuration. NUMERIC: it selects which weights run.
HALOGEN_TOKENIZER /tokenizer Flat tokenizer directory. You usually do not need to set this: the weights repo ships tokenizer/ inside it, and when nothing is mounted at /tokenizer the server falls back to tokenizer/ beside the checkpoint, so one -v of the models directory is enough. Set it only when your tokenizer lives elsewhere. It must be FLAT: HuggingFace cache snapshots are symlinks into a sibling blobs/ and dangle inside a container, so materialize with cp -L.
HALOGEN_MODEL_ID halogen-qwen3.8-flash-next The model id listed at /v1/models, reported at /health, and echoed in every completion. A label. Requests are never rejected for naming a different one. Override it to run two stacks on one host without the ids colliding.
HALOGEN_EOS_GUARD (unset: on); 0 = off The end-of-turn guard (0.13.4, issue #84). The tokenizer has one token for <|im_end|>, so a model that quotes a ChatML chat template in its reasoning writes the real end-of-turn token, and through 0.13.3 the reply ended there: finish_reason: "stop", no content, no tool call. With the guard on, that token is kept as text inside the thinking block (generation goes on, bounded by the thinking budget and the answer room) and inside an open tool call up to four times a reply; a complete tool call the model wrote inside its thinking block and then ended its turn on is made (finish_reason: "tool_calls", its text also left in reasoning_content). A kept <|im_end|> and a quoted <|im_start|> appear in the text literally. usage.completion_tokens_details carries end_of_turn_kept and tool_call_from_reasoning when either happened, and /health.end_of_turn_guard says whether it is on. An end-of-turn token in the answer itself still ends the reply; /v1/completions is never affected. 0 is the 0.13.3 behaviour. NUMERIC only for the replies it changes: every other reply is byte-identical.
HALOGEN_TEMPLATE_UNCHECKED (unset): the chat template is probed and a failure refuses to start Whether to serve a chat template whose thinking control does not work (0.12.2). At startup the server renders a one-message conversation with enable_thinking false and true and requires the first to carry the empty think block and the second not to close it; every thinking control (enable_thinking, reasoning_effort: "none", HALOGEN_ENABLE_THINKING=0, the budgets) depends on that branch, and a template from another repository without it would accept them all and render thinking anyway. The weights repo's own tokenizer/ passes. On a failure one sentence names the file and the container does not start; 1 prints the same sentence and serves it. The startup line and /health.chat_template name the template (path, sha256 of its text) and the probe's result either way. It changes no request.

Networking

flag default meaning
HALOGEN_API_PORT 8731 Port for the OpenAI-compatible front-end. The only port that should be published.
HALOGEN_API_KEY (unset): no key Require an API key on the front-end port. Clients send it as Authorization: Bearer <key> (OpenAI SDKs, Codex) or x-api-key: <key> (Anthropic SDK, Claude Code), the way they already send one; separate several keys with commas. /health stays open for health checks; every other route answers 401 without the key. Set but empty refuses to start, so a key that failed to expand cannot leave the port open. It does not cover the engine port.
HALOGEN_API_KEY_FILE (unset) The same, read from a file such as a mounted secret, so the key stays out of the container's environment. Set this or HALOGEN_API_KEY, not both.
HALOGEN_PORT 8730 Engine port. The engine protocol has no authentication, so keep it unpublished.
HALOGEN_BIND 127.0.0.1 Engine bind address. Loopback when engine and front-end share a container; 0.0.0.0 for the two-container topology, where it stays unpublished to the host.
HALOGEN_ENGINE 127.0.0.1:$HALOGEN_PORT Where the front-end reaches the engine. Compose gives the services separate network namespaces, so it needs engine:8730 there.

Request policy

flag default meaning
HALOGEN_MAX_TOKENS_CAP 65536 Largest max_tokens a request may ask for. Exceeding it is a 400, never a silent truncation: a truncated response and a model that stopped on its own both end with finish_reason: "length", so a client cannot tell them apart. The one exception is /v1/messages (since 0.18.1): Claude Code asks for its model's whole output limit on every request, so there a request above the cap runs with the cap, and the log says so once. Coupled to HALOGEN_QUEUE_TIMEOUT, see the README.
HALOGEN_KEEPALIVE_TIMEOUT 300 Seconds an idle keep-alive connection is held open before the server closes it. Agent clients idle between turns for as long as a tool call or a person reading takes, and a POST on a socket the server has just closed is not retried transparently by most HTTP clients, so a short value turns a rare race into a constant one. Releases before 0.5.7 never set this and got the framework default of 5 seconds. An idle connection costs a file descriptor and holds no conversation slot, so raising it is cheap; lower it only if you are fronting many short-lived clients.
HALOGEN_SSE_KEEPALIVE_S 10 On a streaming response, an SSE comment line (: keepalive) is sent whenever nothing else has been for this many seconds: during a long prefill before the first token, and anywhere else the model is quiet. SSE parsers ignore comment lines, and every HTTP client's inactivity timer resets on one. Node's undici (Pi, Qwen Code) closes a body that is silent for 300 s with no client-side setting, which is what a long tool call looked like before 0.6.1. 0 disables it.
HALOGEN_RETRY_GRACE_S 20 Seconds a reply is paused when its client disconnects in the middle of it, waiting for the same request to come back, as a client's automatic retry does (0.18.1). A retry inside that time continues the reply, and what was already generated is sent first. A paused reply generates nothing and gives up its slot at once when another request needs it. At most 120. 0 stops the reply at once, as before 0.18.1.
HALOGEN_QUEUE_TIMEOUT 3600 Seconds a queued request waits before 503 engine_busy. Must exceed the time a full-length request takes, or a long request 503s everyone behind it.
HALOGEN_MAX_TOKENS_DEFAULT (unset: 8192 on chat and responses, 128 on completions) The token budget a request gets when it sends none. One variable moves every route's default. Must not exceed HALOGEN_MAX_TOKENS_CAP or the server refuses to start. A request that sends a budget always wins. Reported by /health as max_tokens_default. The budget bounds reasoning AND content, so a small default removes the answer rather than shortening it.
HALOGEN_TEMPERATURE (unset: greedy) The sampling temperature for a request that sends none, 0 to 2. Qwen's recommended settings for this model are sampled (generation_config.json: temperature 1.0, top_p 0.95, top_k 20; the card's instruct point is 0.7 / 0.8 / 20 with presence_penalty 1.5), and most agent clients send no sampling fields, so this is how you run those settings without every client cooperating. The rule, for this and the five below: a field the request sends always wins; a default fills only a field the request omits; a request that sends temperature: 0 decodes greedy and takes none of the sampling defaults. Setting this moves every request that sends no temperature off the greedy path, which is the only path the byte-identical guarantees are made on. /health reports what is set under server_defaults and says which path a temperature-less request takes. An out-of-range value refuses to start, before the model loads, naming the variable.
HALOGEN_TOP_P (unset: no nucleus filter) top_p for a request that sends none, 0 to 1. Applies only when the request samples (its own temperature or the server's). A request that sends top_p wins.
HALOGEN_TOP_K (unset: no top-k filter) top_k for a request that sends none; 0 disables. A request that sends top_k, including 0, wins.
HALOGEN_MIN_P (unset: off) min_p for a request that sends none, 0 to 1. Same rule.
HALOGEN_PRESENCE_PENALTY (unset: 0) presence_penalty for a request that sends none, -2 to 2. Penalties are sampler-only, so it applies only when the request samples; on a greedy request it is simply not applied. A request that sends a penalty without sampling is still a 400, and a request that sends presence_penalty: 0 disables the default.
HALOGEN_FREQUENCY_PENALTY (unset: 0) frequency_penalty for a request that sends none, -2 to 2. Exactly as HALOGEN_PRESENCE_PENALTY.
HALOGEN_REPETITION_PENALTY (unset: 1, off) The repetition penalty for a sampled request that sends none (0.15.1): vLLM's rule, which scales down every token that already appears in the prompt or the reply (a positive logit is divided by the factor, a negative one multiplied). Qwen's card lists 1.0. A request's repetition_penalty wins. Like the other penalties it needs temperature above 0: a request that sends a value other than 1 with temperature 0 is refused with a 400.
HALOGEN_REASONING_EFFORT (unset: the chat template's own xhigh) The reasoning effort for a request that names none: minimal, low, medium, high, xhigh or max (the template has three levels: minimal maps to low, medium maps to medium, high and max map to xhigh). This is the largest budget lever there is: xhigh thinks for hundreds to thousands of tokens on an agentic prompt, and that spend comes out of max_tokens. A request that sends reasoning_effort, chat_template_kwargs.reasoning_effort, or reasoning.effort on /v1/responses wins. Reported by /health as reasoning_effort_default.
HALOGEN_ENABLE_THINKING (unset: the chat template's own default, thinking on) Whether a request that does not say renders the thinking block. 0 renders what enable_thinking: false renders (no reasoning, the model answers directly, the model card's instruct settings then apply); 1 is the template's default made explicit. A request that sends enable_thinking (top level or in chat_template_kwargs) wins. /v1/responses has no such field, so this is the only way to run that route without thinking. Reported by /health under server_defaults. Any other value refuses to start.
HALOGEN_MAX_THINKING_TOKENS (unset: no budget) The thinking budget for a request that sends none (0.8.1, issue #56): at most this many generated tokens inside the think block; if the model has not closed it by then the server does (the model's own budget sentence, then </think>) and the answer follows in the same stream. A request's max_thinking_tokens (or any of its aliases: thinking_budget_tokens, thinking_budget, thinking_token_budget, reasoning.max_tokens, thinking.budget_tokens) wins. max_tokens still covers thinking and the answer together.
HALOGEN_THINKING_BUDGET_MESSAGE (unset: the model's own budget sentence); set and empty = </think> alone The text the server writes before </think> when it closes the think block itself: at the thinking budget, at the answer room, and when a reply keeps writing end-of-turn markers inside its reasoning (0.15.1, issue #114; llama-server's --reasoning-budget-message). Plain text only: the server refuses to start if the message contains a chat marker such as <|im_end|> or </think>, or if it is longer than about 500 tokens. It counts against max_tokens like any generated text.
HALOGEN_THINKING_ANSWER_ROOM (unset: max(1024, 15% of the request's max_tokens)); 0 = off The answer room (0.11.0). When a request sends no thinking budget, the think block is closed with this many tokens of max_tokens left for the answer (the same close as max_thinking_tokens: the model's budget sentence, </think>, then the answer), so a capped request ends with content instead of finish_reason: length and an empty content. Agent harnesses send no thinking control to a custom server unless configured to, so the model's xhigh ran under whatever cap the harness set for the answer; a compaction capped at 13,000 tokens that the model thought past was a failed compaction. A request's own budget wins when it is smaller. Only a request whose thinking would have run past the line is affected. /health reports thinking_answer_room.
HALOGEN_DRAFTER_DEFAULT 1 Which drafter a request that names none gets: 1 the model's own MTP head (speculative, and byte-identical to serial greedy, since it only proposes), 0 batched serial. A speculating request speculates while it is the only one generating, and since 0.17.0 also beside exactly one other. The two then draft together and one forward pass checks both, while that pays. With three or more generating it joins the batch, so the setting costs no concurrency. Overridable per request with "drafter": "serial" | "mtp".
HALOGEN_PLD 3,3 Prompt lookup beside the MTP head, as N,K: when the last N tokens the request has produced also occur earlier in its own context (the prompt or what it has generated), the K tokens that followed that earlier occurrence are proposed as one chain and verified in a single step, and the head's own draft must open the chain. It drafts from the request's text, never from a model, so it pays where an answer repeats its context: tool-call arguments, file paths, code that quotes the file being edited. Byte-identical to serial greedy, like the head: every committed token is the model's own choice. Applies to greedy requests only, and only while the request is the only one generating. A sampled request uses the head alone. 0 turns it off. K above 3 needs more verify rows than the image reserves and the server says so at startup. With HALOGEN_MTP=0 there is no head to run beside, so it is off and the server says so at startup.
HALOGEN_SPEC_ADAPT 32,0.35,64 The MTP head's adaptive policy, as W,floor,R: acceptance is read over windows of W drafting rounds; when a window falls under floor the head is switched off for the next R tokens (the request decodes serially, which is faster than drafting that keeps being rejected, as on some prose), then a window is tried again. 0 never adapts (the head drafts every round). Byte-identical to serial greedy either way: the head only proposes, so every committed token is the model's own choice; the policy moves speed only. A sampled request runs the same policy with the acceptance coin in place of the argmax match.
HALOGEN_GRAMMAR 1 Structured output: whether a response_format of json_schema / json_object (text.format on /v1/responses) is enforced while decoding. 1: the schema is compiled once per distinct schema (a millisecond), every state's token mask lives on the GPU (up to 31 MB a schema, eight schemas cached), and each token is chosen among the tokens the schema allows next (the greedy choice at temperature 0; at a temperature above 0 a draw from the model's distribution over those tokens, with top_p, top_k and min_p acting on that distribution), so the reply parses and validates by construction at any temperature; a request without a schema is byte-identical to what it was, and a greedy constrained request is identical serial, with the head, with prompt lookup and beside other requests. The README lists which schema keywords are enforced, which are accepted without being enforced, and which are refused by name; /health lists the same under structured_output. A schema with an image is a 400, and so is a schema with logprobs. 0: every schema is a 400, as before 0.8.0.
HALOGEN_STRICT_TOOLS 1 Strict tool schemas (since 0.18.2): whether a tool declared strict: true (function.strict on /v1/chat/completions, strict on a /v1/responses or /v1/messages tool) is enforced while decoding. 1: when any tool on a request is strict, a tool call names one of the request's tools, and a strict tool's arguments match its parameter schema (enum and const values, each parameter's type, every required parameter, no parameter the schema does not declare), at any temperature; parallel_tool_calls: false (disable_parallel_tool_use on /v1/messages) then allows one call. A request with no strict tool is byte-identical to what it was. A string argument cannot contain the line </parameter>. A strict tool whose schema uses a keyword the server does not enforce, and every strict tool on a request with images, logprobs or more than 128 tools, runs as it would without strict, named in the request's log line and counted in /metrics; /health lists the rules under structured_output.strict_tools. 0: strict is accepted and not enforced, as before 0.18.2.
HALOGEN_TEMPLATE_CHECK warn At start, checks that the loaded chat template re-renders a recorded conversation (with and without the reasoning sent back, and with thinking off) as the same prompt the next turn extends, which is what lets the prompt cache reuse earlier turns. Prints one startup line, passed or FAILED with the first place the two differ, and never stops the server. off skips it. Since 0.18.3.
HALOGEN_CLIENT_MARKERS text How a chat-template marker written inside your own text is read (since 0.18.2). A message, a tool result, a tool description or a template argument that contains <|im_end|>, <|im_start|> or a similar spelling is read as the characters that were typed, so it cannot be mistaken for one of the template's own markers. Assistant history keeps the markers the model itself wrote, so a client that echoes back a previous reply still matches it. A request whose text holds no such spelling is unchanged, and its token ids are what they were. special reads those spellings as markers, as before 0.18.2. Front end only.

Images

flag default meaning
HALOGEN_VISION_TOWER (unset: images are refused) Path to the vision sidecar, or 1 to look for it beside the checkpoint. Unset, the server takes text only and an image is a 400 naming this flag, which is the shipped default: with no tower the image path is not merely disabled but absent, so text behaviour is byte-identical to a build without it. Set, /health reports vision.enabled and both /v1/chat/completions and /v1/responses accept an image content part as a data: URL or bare base64. Fetching an http(s) URL is refused by design, since that would make the server issue outbound requests on a client's say-so.
HALOGEN_VISION_MAX_PIXELS 3686400 (2560x1440) The most pixels an image is given before it is downscaled to fit, preserving aspect ratio. Nothing is refused for being large until four times this, which is a guard on decode cost rather than a quality limit. Measured end to end, one image costs about 5.5 / 11.8 / 25.3 / 105.8 seconds at 1280x800 / 1920x1080 / 2560x1440 / 3840x2160, and 4K reads no better than 1440p, which is why the default sits there. Images below 256x256 are scaled UP, which helps rather than hurts: a downscaled frame reads small text at least as well as a native one at the same final glyph size. Text at 12 pt and above is read exactly at every supported resolution; below that it degrades gradually rather than failing, and a densely filled page is harder than a sparse one at the same point size.

Context and concurrency

flag default meaning
HALOGEN_KV_SLOTS 4 Conversations generating at once. A slot costs about 115 MB; the pool below decides what fits. Each stream is byte-identical to running alone. Four streams together produce about 76 tokens per second in total, each at 11 to 19; eight produce about 85, each at about 10, so raising it trades per-stream speed for admitting more clients at once rather than queueing them. Past eight a step takes two forwards and total throughput stops growing. Capped at 64.
HALOGEN_KV_POOL_POSITIONS 2 x HALOGEN_CTX, capped at 1,048,576 Attention positions resident across all conversations; HALOGEN_CTX is the most one request may use. A request reserves its prompt plus max_tokens positions when admitted and waits in arrival order when they are not free. The default 524,288 holds two full-length conversations, about 27 GB on the device; 786,432 holds three at about 34 GB and 1,048,576 needs HALOGEN_MAX_TOK=16384 (about 37 GB with the default checkpoint, 40 with w4b). The server measures the device budget at startup and lowers the pool itself if the configured one will not fit, saying what it chose. Generation speed follows each conversation's own length, not the pool. The pool takes RAM the page cache would otherwise keep for the model's n-gram table, so a prompt whose rows are not cached reads them from disk first; a smaller pool leaves more cache.
HALOGEN_KV_POOL_FIT 1 Whether the server measures the device budget at startup and lowers HALOGEN_KV_POOL_POSITIONS when the configured pool will not fit, printing the pool it settled on. 0 allocates exactly what was asked for and lets the allocation fail, which is what every release before 0.3.1 did. It can only lower the pool, never raise it, and it never lowers it below HALOGEN_CTX; a shorter pool changes how many conversations stay resident and changes nothing about what any one of them computes.
HALOGEN_HOST_RESERVE_GIB 20 How much system RAM the server leaves free, on top of the weights, when it sizes the KV pool at startup. The model keeps a large lookup table on disk and reads it through the file cache rather than holding it in memory, so RAM the pool takes is RAM that table loses. Too little and the server starts, runs, and then crawls on long prompts with the disk busy. Raise it if you see that; a smaller pool is the trade.
HALOGEN_CTX 262144 The most context ONE request may use, the model's full native context, by default. A prompt over the limit is a hard error naming it, never a truncation. The published quality numbers cover 1k-32k; beyond that the context is served but unscored.
HALOGEN_ADMIT_CHUNK 0 (the engine's own prefill chunk) How a prompt is read in when other conversations are already generating: in pieces, with a generation step for the others between pieces. The default piece is the prefill call size (HALOGEN_MAX_TOK, 32,768), which is exactly how the same prompt is split when it runs alone, so the answer stays byte-identical to running alone; a 131k prompt pauses the others three times for about 28 s each instead of once for 105 s. HALOGEN_MAX_TOK=16384 halves the pause for everyone at about 8% slower prefill and keeps identity. A smaller value such as 8192 (measured: a 32k prompt pauses the others about 8 s rather than 28, and pays about 5 s on its own first token) trades that identity for the admitted prompt, whose answer then depends on the load when it arrived. -1 reads every prompt in one call. With HALOGEN_PROMPT_CACHE=1 a value is rounded down to whole prefill chunks (at least one), so a value below the chunk has no effect there. The engine reads this setting, so in the two-container setup it goes on the engine container. Since 0.18.3 its startup line shows both the value it read and the piece it uses.
HALOGEN_ADMIT_TICKS 1 How many generation steps the conversations already generating get between two pieces of a prompt being read in beside them (HALOGEN_ADMIT_CHUNK sets the piece). A larger value keeps them generating more smoothly through a long prompt, and that prompt's first token comes later by the extra steps. Answers are unchanged, the admitted prompt's included. One piece still runs to its end once started.
HALOGEN_CACHE_ENTRIES 24 How many prompt-cache entries the server keeps, least recently used out. A conversation holds six: one at the end of its system prompt, its four most recently used history entries (two a turn for each of two branches, see HALOGEN_CACHE_BRANCHES; 0.11.0, issue #61: a harness that sends the history plus a side question and then drops that turn continues from where the side turn branched, and with one history entry that point was gone; a third since the start of the last user message became a saved place, see HALOGEN_CACHE_SNAP3), and since 0.11.3 one at the end of its last request, which serves an exact repeat of that request without reading anything again. Twenty-four entries cover four conversations, and requests that share a system prompt and ask different things resume from it. Each entry is about 115 MB of RAM plus the pool positions its content covers, which stay reserved while it exists. When the pool is full the server forgets the least recently used conversation's region, never the entry the request is about to resume from, and prints a kv pool: no room ... line for each one.
HALOGEN_CACHE_BRANCHES 2 How many branches of one conversation the prompt cache keeps resume points for. Two is a conversation plus one side turn. Three is a parent forked into two children, which is what a harness fanning out subagents runs: on two, the parent's resume point is the one the children's stores replace, so the parent re-reads its history on its next turn. Raising it costs host memory in whole entries (about 115 MiB each) and raises the HALOGEN_CACHE_ENTRIES default with it, since that default is derived from this: 3 gives 32 entries and about 3.5 GiB against 2's 24 and 2.7. It changes only what is cached, never what an answer is.
HALOGEN_CACHE_SNAP3 1; 0 = off Whether the prompt cache also saves its place at the START of the last user message. A client that keeps a document in one message and asks each new question in the next ([doc][question 1], then [doc][question 2]) matched neither of the other two places (the end of the previous question, the end of the system prompt), so every new question re-read the document and only an exact repeat was served from the cache; with this on the second question resumes from the end of the document. Costs nothing measurable on the first request (since 0.12.1 the state is recorded as the pass goes by; the fixed cost of a save is about 0.1 s) and nothing on a hit. Harnesses that extend their history every turn were served by the other places already and are not changed by this. A one-message request (with or without a system prompt) has nothing behind its question and gets no third place.
HALOGEN_CACHE_FULL 1; 0 = off Whether the server keeps the end-of-request entry (0.11.3): the state at the end of every prompt plus its last logits, one per conversation, so an exact repeat of a request answers byte for byte its first answer and reads nothing again. 0 returns to three entries per conversation and about 115 MB less RAM per conversation; a repeated request then resumes from the nearest history entry, as before 0.11.3.
HALOGEN_MAX_TOK 32768 Largest single prefill call; longer prompts are prefilled in pieces. It sizes the prefill scratch (about 9 GB at 32768 with the default checkpoint), so do not raise it to the native context. That allocation does not fit and the server will not start.
HALOGEN_CACHE_INPLACE 1 The prompt cache keeps the conversation's attention state where it already is and saves only the small position-free state (about 110 MB at any context), so saving and resuming cost well under a second even at 1M. 0 copies the whole state instead (up to 26.6 GB at 1M); it is the older form, kept for comparison, and gives the same answers byte for byte.
HALOGEN_CACHE_FILE (unset) Keep the prompt cache's snapshot in a file at this path instead of host RAM: the same bytes, at the drive's speed. Only needed with HALOGEN_CACHE_INPLACE=0 past 262,144, where that form's snapshot is up to 26.6 GB; the container then sets it to /var/tmp/halogen-cache.snapshot. The file is created at startup and removed at exit.
HALOGEN_ROPE_YARN (unset) Contexts past the native 262,144, up to 1,048,576. Set it to the model card's static YaRN factor (4 for 1M, 2 for 524,288) together with a larger HALOGEN_CTX; the server refuses a context past 262,144 without it, and past 262,144 x factor with it. It is a different model configuration, not a cache setting: every position's RoPE is rescaled, short prompts included, and the model card advises it only when the context needs it. Past the native context the server caps HALOGEN_MAX_TOK at 16384 and prints it, because a 1M KV cache leaves no room for the larger prefill arena on a 128 GB machine. The README's 1M context section has what it costs. Unset, nothing changes.
HALOGEN_INDEXER_BUDGET (unset) = 2048 The model's sparse-attention budget: how many tokens of the context each query attends (the top budget / 4 blocks by the model's own indexer, plus the tail). The checkpoint's value is 2048 and is the default. Any value from 2048 to 8192, rounded down to a multiple of 16, is accepted; the effective value is printed at startup and reported by /health as indexer_budget. Set higher, it is a different model configuration, not a cache setting: every query past the budget attends a superset of what the checkpoint was trained on, so long-prompt answers are not byte-identical. Static for the server's life. The README's Attention budget section has what 4096 and 8192 bought and cost when measured: two more of 96 planted facts retrieved at 16k for about 7% of prefill speed at 32k and 2% of decode at 4096; every fact at 8192, at a consistent perplexity cost and a fifth of prefill. Unset, or set to 2048, nothing changes.

Speed levers

flag default meaning
HALOGEN_PROMPT_CACHE 2 (resume anywhere) Reuse a session's shared prefix instead of re-reading it. 2 (default) saves its place at the end of every request, so a follow-up turn at 100,000 tokens of context costs about 2 s against 88 s cold, at any prompt length. 1 saves only on a prefill-chunk boundary, which makes a resumed answer byte-identical to a cold one, at the price of caching nothing below one chunk. 0 is off. The README's Choosing a cache mode has the numbers and the cases. NUMERIC in mode 2.
HALOGEN_PREFILL_CHUNK 32768 How many tokens the engine prefills per call. Under the default cache mode this is purely a speed setting and 32,768 is the fastest value measured; 1,024 costs about half the throughput at long context. Under HALOGEN_PROMPT_CACHE=1 it is also the resume granularity, so lower it to 8,192 if you need byte-identical repeat answers on conversations shorter than 32,768 tokens, at about 13% less prefill throughput.
HALOGEN_CACHE_DIR (unset) = the prompt cache lives in memory only Persist the prompt cache to this directory so a conversation survives a server restart (0.10.0; issue #40). Each turn's new state is written behind the request, nothing on the request path waits, and a conversation no longer in memory is restored from disk on its next turn; the restore is byte-identical to the in-memory resume. About 27 KiB per token. Every build, weights file and setting keeps its own files and never restores another's. Needs a filesystem that accepts direct I/O (a tmpfs or overlay is refused and the cache stays in memory). See the README's Prompt cache on disk section.
HALOGEN_CACHE_DISK_GIB 64; 0 = unbounded The directory's size bound for HALOGEN_CACHE_DIR: least recently used conversations are removed when a new one would pass it.
HALOGEN_CACHE_PRUNE_OLD 0 (keep) 1 removes, at startup, the other builds' and configurations' subtrees under HALOGEN_CACHE_DIR (0.11.10; issue #78). Each engine build, weights file, context size or numeric setting keeps its own subtree there and HALOGEN_CACHE_DISK_GIB bounds only the current one, so an upgrade leaves the old build's files behind until something removes them. The default keeps them (a rollback finds its cache warm); with 1 the startup log names each subtree removed and its size. Only directories the cache itself wrote are candidates; anything else in the directory is left alone.
HALOGEN_COMPOSABLE_CONTEXT 0 (off) NUMERIC, opt-in preview. 1 retains each message at or above HALOGEN_COMPOSABLE_CONTEXT_FLOOR as it is first read and reuses that work when a later request repeats the message at any position behind the same system prompt: the shape of an agent harness's compaction. A reused answer is very close to, not byte-identical to, the answer from reading the text fresh. Needs HALOGEN_PROMPT_CACHE=2 and HALOGEN_KV_POOL=1 (the defaults); image requests are not composed. /health reports it under composable_context. See the README's Composable context section.
HALOGEN_COMPOSABLE_CONTEXT_FLOOR 2048 (tokens) The smallest message composable context retains and reuses. Messages below it are read fresh every time.
HALOGEN_COMPOSABLE_CONTEXT_BYTES 4294967296 (4 GiB) The host-memory budget of the retained messages, least recently used out; each retained message costs about 230 MB whatever its length, so the default holds about 18. In the server's memory, not on disk, and gone at restart.
HALOGEN_MATMUL_TUNING_FILE /opt/halogen/flash-tune.plan (baked in) A tuned GEMM plan: which kernel the matrix library should use for each shape, decided once and shipped in the image. On by default at no measurable quality cost, and the published prefill numbers include it. Every process reads the same decisions, so the same prompt keeps giving the same answer. The plan is tuned on one machine; a different Strix Halo SKU loads it with a warning. To retune for your own workload, point this at a path that does not exist yet, run with HALOGEN_MATMUL_ALGOS=8, and stop the container cleanly. Do not set HALOGEN_MATMUL_ALGOS=8 without a file: selection then happens per process by timing, and identical prompts stop answering identically across restarts.
HALOGEN_NGRAM_GATHER_THREADS 64 How many rows of the model's lookup table are read at once when a prompt needs rows that are not in the file cache. The table (47.7 GiB) is read through the page cache and never held in RAM, and on a 128 GB machine running this server there is never room to cache all of it, so the first long prompt after a restart, or any long prompt on a host with little RAM to spare, reads its rows from disk: one 4 KB random read per row. Before 0.6.3 those reads were issued one at a time; a 32,768-token prompt paid 46 to 52 s on top of its usual 25 s with the table evicted, 13.5 s on an 8,192-token one. At 64 in flight the same prompts pay about 1.3 s and under half a second on an NVMe drive (8 threads 7.3 s, 16 and 32 about 3 s, 128 no better than 64). Same bytes, same output. 1 restores the old loop. The log line lookup table: ... took N s on N threads reports any read of 2 s or more, and the health check keeps being answered through it. Decode reads 16 rows a token and is unchanged.
HALOGEN_PREFILL_KEEP_TRUNK 0 (off) 1 keeps the default checkpoint's trunk weights unpacked in memory for the server's life, so prefill does not unpack them again on every pass. Prompts past a few hundred tokens are read faster. It costs about 5.5 GiB of memory. The server counts that when it sizes the KV pool at startup, and the startup memory: line names it. On a 128 GB machine at the shipped settings the pool is unchanged. If the memory is not there the server refuses to start and says so, rather than failing a request later. Short prompts already take a path that needs no extra memory. With this on, prompts of a few hundred to about 1,300 tokens use the unpacked weights instead, so their answers can differ from the default in the last digits, as with any change of kernel. No effect on the w4b checkpoint.

Request records

flag default meaning
HALOGEN_RECORD_DIR (unset) = off Keep a record of the requests this server answers, in this directory (0.17.3). One JSON line per request: the API and the client family that sent it, the request's sizes and settings, how it finished, the tokens generated, its tool calls, and the same timings block a response carries; with HALOGEN_RECORD_TEXT, the text of the messages and the reply too. It shows what your clients and agents actually send and get back. Nothing on a request's path waits for the store, and a full or failing store drops records, never a request. Mount a writable directory: -v ~/halogen-records:/records -e HALOGEN_RECORD_DIR=/records. Without HALOGEN_API_KEY the API port has no authentication: anyone who can reach it has their requests written here, text included unless HALOGEN_RECORD_TEXT=none. No route of the server reads, deletes or serves the records. Unset, nothing is written and nothing changes. See the README's Request records section.
HALOGEN_RECORD_TEXT all (when HALOGEN_RECORD_DIR is set) Which text the records keep: all, none (the per-request lines only, no text), or a comma list of prompt (system, developer and user messages), reply (the answer), reasoning (the thinking), args (tool-call arguments) and results (tool results). A client's prompt carries the tool results and earlier tool calls it sends back, so leave out results and args to keep those out too. A field left out keeps only its length. Each message is stored once however many requests repeat it. Images are never stored, only a hash and their pixel size. An unknown field name refuses to start.
HALOGEN_RECORD_GIB 2 The size cap of the records directory, in GiB, over everything in it. Past it, new records are dropped and counted (/health, request_records.dropped_cap), and requests are answered as usual.
HALOGEN_RECORD_DAYS 90 How many days of records to keep. Older days (UTC) are removed at start and when the date changes, together with any stored text no remaining record refers to. 0 keeps everything.

Sessions: keep, fork and join

flag default meaning
HALOGEN_FORK (unset) = off Let a client keep a conversation's state under a name and start later requests from it. Off by default. On, /v1/chat/completions, /v1/messages and /v1/responses read two request headers. X-Halogen-Keep: <name> keeps the state where this reply ends. X-Halogen-From: <name> starts a request from a kept state when the request repeats that conversation, either with more messages after the kept reply (tool calls and their results included) or with part of the reply as the last message, to branch from inside it (continue_final_message on Chat Completions, a final assistant message on the other two). Several requests can start from one name at once (a fork), and one request can start from a name and keep another (a join). The response header X-Halogen-From says used, unmatched or unknown. The last two are answered as ordinary requests, so a stale name costs time, never a wrong answer. GET /halogen/handles lists the live names and DELETE /halogen/handles/<name> releases one. With HALOGEN_CACHE_DIR set, kept states survive a restart with the same settings. Every client of the server shares one set of names. Any client that can reach the port (any key holder, with HALOGEN_API_KEY) can use or release any name, so use names other clients cannot guess and keep the port private.
HALOGEN_FORK_HANDLES 8 The most names kept at once. A request that asks to keep one more is answered as usual and nothing is kept. Release a name to make room.
HALOGEN_FORK_TTL_S 600 A name not used for this many seconds is released.
HALOGEN_FORK_GIB 2 The most host memory kept states may hold, in GiB (each name holds about 0.2 GiB). Kept states also take at most half of the KV pool. A request that would pass either limit is answered as usual, nothing is kept, and the log says which limit. Changing HALOGEN_FORK, HALOGEN_FORK_GIB or HALOGEN_FORK_TTL_S starts the disk cache (HALOGEN_CACHE_DIR) cold.
HALOGEN_FORK_MAP (unset) = off claude-code: with HALOGEN_FORK=1, keep each Claude Code thread's state (the main conversation and each subagent) and start that thread's next request from it, with nothing to set in Claude Code (0.18.1). Each name is made from the session, the thread and the API key, so it cannot be guessed and two keys never share one. A subagent's first request starts from its parent's state when it repeats it, and a side request on a session (a title, a summary) never replaces the conversation's state. auto (since 0.18.3, opt-in): keep each conversation's state at the end of its reply, so its next turn starts from there instead of reading the reply back in; claude-code,auto turns on both. Any other value, or a map without HALOGEN_FORK=1, stops the server at start with one line.
HALOGEN_FORK_AUTO_KEEPS 4 With HALOGEN_FORK_MAP naming auto, how many reply-end states are kept at once; the least recently used goes first, and a named or Claude Code state is never taken. Each state costs about 0.22 GiB of host memory (4 is about 0.9 GiB) plus the KV pool it holds. At most half of HALOGEN_FORK_HANDLES. Since 0.18.3.

Troubleshooting

flag default meaning
HALOGEN_PREFILL_CANCEL 1 (on) A request cancelled while its prompt is still being read (its client hung up, or the front end cancelled it) stops at the next layer or chunk of that prompt, within about a second, and adds nothing to the prompt cache (0.16.0, issue #122). 0 restores the earlier behaviour, where a prompt being read alone was read to its end before the cancel was seen, and the startup log says so. Answers are unchanged either way. An image is still encoded to its end before a cancel is seen.
HALOGEN_QSA_FORCE_SEL (unset: follows the GPU firmware); 1 = on, 0 = off For a host whose GPU firmware predates 2026 (MEC firmware below 0x22: for example Debian's 20250410 firmware-amd-graphics package, or SteamOS). That firmware refuses one of the engine's default settings on prompts of about 70 to 2,000 tokens and on image prompts, and the engine stops with INVALID_DISPATCH_PARAMETERS (error 701, issues #6, #128, #166). Since 0.18.2, unset, the engine reads the firmware version at startup and on such a host runs as with 1, and says so in one NOTE line; on #128's host that cost about nothing (decode 53.3 against 53.5 tok/s, the same prefill). 1 forces it on any host; 0 keeps the default even on old firmware. The lasting fix is current linux-firmware (MEC 0x24 fixed #128), after which nothing needs setting. Check the version with cat /sys/class/drm/card*/device/fw_version/mec_fw_version. NUMERIC: a different order of the same arithmetic.
HALOGEN_GPU_DEVICE (unset: the Strix Halo GPU); auto = unset For a host with more than one AMD GPU, such as a Strix Halo with an AMD eGPU. Since 0.18.3, unset, the server runs on the Strix Halo GPU (gfx1151) whatever number ROCm gives it; before, it took device 0, so an eGPU listed first made it fail at startup. A number picks that device as ROCm numbers them (0, 1, ...); first picks device 0, the old behaviour. When more than one GPU is visible, or this is set, the log names the chosen card in one line. With no Strix Halo GPU visible at all the server refuses to start and lists what ROCm offers.
HALOGEN_VERBOSE 0 in this image Whether the server narrates how it is armed at startup, on top of what it is serving. Off in the published image, where the extra lines are internal detail rather than anything a deployer acts on. Set it to 1 when you are diagnosing a start that goes wrong and want the fullest account the server can give.
HALOGEN_STARTUP_PROGRESS 1 (on) Whether the server says which startup step it is on and how long it has taken: pinning weights, preparing them layer by layer, reserving working memory, reserving the KV pool, ready. A first start reads about 68 GB off disk, so on a slow or busy machine the wait is minutes, and these lines are what tell you it is progressing rather than stuck. Leave it on. If you report a slow start, the log with these lines in it is the single most useful thing to attach.
HALOGEN_ENGINE_WATCHDOG_S 180 (seconds); 0 = off How long the engine may go without answering the container's liveness probe before the container is taken down so a restart policy can recover it. It is for an engine that is alive and answering nothing, which nothing else can see: the process is up, the port accepts, the health check stays green. Silence inside the kernel does not count: before a silent probe counts, the container reads the engine's thread states and the kernel's memory-compaction counter, and an engine in uninterruptible sleep or a kernel compacting memory for it is a host short of memory, not a wedge. That silence is logged (not counted as a wedge) and the clock restarts when it ends. Killing an engine in that state is what left the driver holding its GPU memory after the process was gone on the host in #79, the end state three other hosts reached by other unclean exits (#34), after which every later start hangs until the host reboots. On a host that shares its memory with other work, raise it or turn it off before relying on a restart policy.
HALOGEN_ENGINE_WATCHDOG_DEFER_S 900 (seconds); 0 = no cap How long the watchdog keeps deferring a silent engine because the kernel's memory-compaction counter is climbing, before it counts the silence as a wedge and takes the container down anyway. The counter is host-wide, so on a machine that compacts memory without pause it says nothing about this engine, and before 0.12.3 the deferral had no end: one host sat with a wedged engine at 100% of a core and the container up for 30 minutes (issue #85). Past the cap, an engine whose threads are running and not inside the kernel is taken down with a line that says why. An engine with a thread in uninterruptible sleep is still never killed: the signal would not reach it, and that kill is what leaves the driver holding GPU memory (issue #79). Raise it if your host compacts memory constantly and your stalls really do clear in more than fifteen minutes; 0 restores the unbounded deferral.
HALOGEN_NEXT_TOKEN_S derived: 990 at the defaults; 300 with the watchdog off; 1800 with no deferral cap How long a stream waits between two tokens before it answers 504. By default it follows the engine watchdog's settings (the deferral cap plus two probe cycles), so a request is not failed while the watchdog still judges the engine healthy during a host memory stall (issue #85). Set it to bound a stream's silence yourself.
HALOGEN_ENGINE_WAIT_S 0 (wait as long as the load takes) How long the container waits for the engine to finish loading before giving up. The default waits indefinitely and watches the engine process, so a machine that needs twenty minutes to load gets twenty minutes, and a crash is still reported immediately. Set a number of seconds if you would rather the container fail than wait, for example under an orchestrator with its own startup probe; on expiry the container exits with a message saying the engine was still loading.
HALOGEN_FLASH_PIN_TRUNK 1 (on) Whether the weights are locked into memory. On, they are, and that is what the published decode speeds are measured with. This is the setting to reach for if you run other large workloads on the same machine. Loaded, this server holds most of the host, and what is left is mostly not in the large contiguous pieces a big allocation needs. If your other processes then take the remainder, both they and this server can stall for minutes at a time at full CPU with no disk activity and no output. That is not a crash and needs no restart. Setting 0 gives a great deal of memory back and costs several times the decode speed (6 tokens/s against 37 on the reference machine; every prompt of 64 tokens or more first copies each layer's experts onto the GPU, about 7 s at any length when the weights are in the file cache and 25 to 50 s when the kernel has let them go to disk, so a 32k prompt reads within a few percent of pinned at best and a short one several times slower), so it is a last resort rather than a tuning option; a machine of its own is the better answer. Through 0.12.1 the setting did not work at the shipped settings (the first request ended the server with internal sizing error in the per-forward arena, issue #83); since 0.12.2 it does. The startup line reports how much room is left. It applies to w4b only: v2 refuses 0 at startup.
HALOGEN_FRAG_WARN_BLOCKS 512 When to warn that the host's free memory is in the wrong SHAPE. Free memory can be plentiful and still have almost none of it in large contiguous pieces, which is the normal state for a few minutes after a large process exits. Every big allocation then stops to compact memory, most of those attempts fail, and startup can take tens of minutes at 100% of one core with no disk activity and no output, which looks exactly like a hang. Before allocating anything the server counts the free 2 MiB contiguous blocks and warns below this many. An idle machine has thousands. If the warning appears, stop other large workloads and, as root, echo 1 > /proc/sys/vm/compact_memory before starting again. Raise the number to see the warning on a healthy host; 0 turns it off.
HALOGEN_FRAG_WARN_STALLS 1000 The same problem measured directly instead of predicted: how many times a startup step had to stop and compact memory before it could get the pages it asked for. The server reports the count for each step and warns above this many. A healthy machine does almost none. Set 0 to make it report on every step, which is how to see what the line looks like; a negative number turns it off.
HALOGEN_DMALLOC_LOG 0 1 prints every device allocation of 64 MB or more as it happens, with its size, a running total and a count. It is this configuration's memory budget measured rather than estimated, and it is the right thing to attach to a report about a server that will not start. A number of 2 or more is a size in bytes instead, so a small value such as 4096 reports every allocation however small: that is the setting for asking what happens while requests are being served, as opposed to during startup. It prints sizes and totals only.

The NPU

flag default meaning
HALOGEN_NPU_MODELS (unset: no NPU) Small models to serve on the Ryzen AI NPU beside the Flash model, comma-separated, on the same port; a request's model picks one (0.16.0). A bare name is one of ours, docs/NPU.md lists them (decider-0.8b, qwen3-embedding-0.6b, qwen3-reranker-0.6b and the rest; since 0.18.3 also embeddinggemma-2, which embeds text, images and audio, and qwen3guard-stream-0.6b, the streaming guard a chat request turns on with its guard field), its files under /models/npu/<name>/ and fetched there under HALOGEN_DOWNLOAD. A path (/models/<dir>) is your own fine-tune of one of them, a Hugging Face checkpoint (config.json, model.safetensors, tokenizer.json): converted for the NPU at the first start (kept as <dir>/halogen-npu.hnpw) and served under the directory's name; anything else is refused at start with a line that says what runs. Unset, the server starts exactly as without it. Needs the NPU driver, the host's XRT mounted, --device /dev/accel/accel0, and the GPU's fabric clock held (HALOGEN_NPU_WITH_GPU). docs/NPU.md has the setup and the request shapes.
HALOGEN_NPU_WITH_GPU 0 With HALOGEN_NPU_MODELS, the start requires the GPU's fabric clock held at its top level, because GPU work and NPU work at the same time on Strix Halo can hang the machine and corrupt the NPU's results while that clock changes speed. Not held, the server holds it itself when it may (run as root with -v /sys:/host/sys) and gives it back at stop; otherwise it refuses before the Flash model loads and prints the commands, and deploy/host/halogen-fabric-clock.service holds it at every boot. 1 starts anyway, which is not supported.
HALOGEN_NPU_VERIFY 1 With HALOGEN_NPU_MODELS: 1 checks the size and sha256 of every file of our models against the image's record before anything starts, and refuses a name the record does not list. 0 runs files the record does not list; the weight file and the tokenizer are still checked against each other.
HALOGEN_NPU_EMB_BATCH 1 /v1/embeddings and /v1/rerank: short inputs in one request run together in one pass. Each result is exactly what it would be alone. 0 runs every input on its own.
HALOGEN_GUARD_STREAMS 4 With qwen3guard-stream-0.6b in HALOGEN_NPU_MODELS: how many chat replies the streaming guard watches at once (0.18.3). A request past it is served without the guard, and its stream says so in a halogen_guard event. The reply itself is never refused or slowed for it.
HALOGEN_GUARD_FLAG 0.8 The streaming guard's Unsafe probability at which a watched chunk of a reply is flagged (flagged: true in its event; the summary counts them). A request can send flag in its guard object to use another value. It only labels events: the reply is never stopped or changed.
HALOGEN_GUARD_LAG 4096 How many characters of a watched reply may wait for the streaming guard. Past it the oldest text not yet checked is skipped, and a skipped event says how much.
HALOGEN_GUARD_FINAL_S 3 How long, in seconds, the end of a watched streamed reply waits for the streaming guard's last verdicts before its summary goes out. The reply's own text has already been sent by then.
HALOGEN_NPU_QUEUE 64 With HALOGEN_NPU_MODELS: how many NPU passes may wait on the NPU at once. A request's inputs take one pass each, or one for a batch of short inputs. More requests than that wait in the server, up to HALOGEN_QUEUE_TIMEOUT, and then get 503 engine_busy with Retry-After. Raising it changes no result.

Benchmark tooling

flag default meaning
HALOGEN_API http://127.0.0.1:8731 Endpoint the bundled benchmarks target.
HALOGEN_API_LOG (unset) Path to a teed front-end log, so the serving benchmark can read throughput counters from inside a container.