Troubleshooting
A symptom-first runbook for operators. Every entry names the signal to look at (an audit event, an accounts list --details availability value, or an HTTP status and error.code), the cause, and the remedy. The field reference for every knob mentioned here is CONFIGURATION.md; the in-stream error codes are specified in STREAM_ERROR_CONTRACT.md.
Where the signals live
Audit log. One JSON object per line on stderr.
log_level(defaultinfo) is the minimum level emitted;debugenables the opt-inrequest_shapediagnostic. Fields hold counts, booleans, enumerated labels, and 16-hex-characterauditHashvalues (account_hash,conversation_hash,detail_hash); the log never carries prompt text, tool arguments, tokens, or signatures.kiro-provider accounts list --details(or--json). TheAVAILABILITYcolumn is the selection view of each account:Value Meaning availableHealthy, not rate-limited, quota not exhausted. rate-limitedA 429/backoff window is active untilRECHECK_AT.refresh-failedAuthentication refresh failed; inspect AUTH_ERRORandAUTH_RETRY_AT. An elapsed retry time does not prove recovery.quota-exhaustedKiro reported the quota as used up; a due probe re-admits it. overage-blockedHealthy and within quota, but stop_on_overageexcludes it because its paid overage count exceedsoverage_threshold.unhealthyMarked unhealthy for a transient reason ( HEALTHisunhealthy).needs-reloginThe refresh token or OIDC client is permanently dead; only accounts reloginrecovers it.With many accounts,
--sort availabilityputs the rows in that same order, worst last, and--sort usage --order descbrings the accounts closest to their quota to the top.HTTP status and
error.code. OpenAI-shaped routes return{ "error": { "type", "code", "message" } };/v1/messagesreturns the Anthropic envelope, where quota402becomes429 rate_limit_errorand the provider code is preserved in the message text.
journalctl recipes for the systemd user service
The audit log is the service's stderr, so the journal is the log. -o cat prints only the message, which keeps every line a parsable JSON object.
# Last 200 lines, human-readable
journalctl --user -u kiro-provider.service -n 200 --no-pager
# Warnings and errors in the last hour
journalctl --user -u kiro-provider.service --since -1h -o cat --no-pager \
| grep -E '"level":"(warn|error)"'
# Follow one event type live
journalctl --user -u kiro-provider.service -f -o cat \
| grep --line-buffered -F '"event":"sdk_stream_terminal"'
# Token-refresh failures per account hash (needs jq)
journalctl --user -u kiro-provider.service --since -1d -o cat --no-pager \
| grep -F '"event":"account_token_refresh_failed"' \
| jq -r '[.timestamp, .account_hash, .error_code, .refresh_token_dead] | @tsv'
# Count terminal provenance values for the day
journalctl --user -u kiro-provider.service --since today -o cat --no-pager \
| grep -F '"event":"sdk_stream_terminal"' \
| jq -r '.terminal_provenance' | sort | uniq -c
# Everything about one account (hash from `accounts list --json` is not the
# audit hash; copy `account_hash` from any event that names the account)
journalctl --user -u kiro-provider.service --since -1d -o cat --no-pager \
| grep -F '"account_hash":"0123456789abcdef"'Without systemd, redirect stderr to a file when starting kiro-provider serve and apply the same grep/jq filters to it.
Accounts and quota
An account shows needs-relogin; the log has account_token_refresh_failed
- Look at:
accounts list --details→AVAILABILITYneeds-relogin; auditaccount_token_refresh_failed(warn) withrefresh_token_dead: trueand anerror_codesuch asinvalid_grant,InvalidGrantException,ExpiredTokenException, orInvalidTokenException; the background maintenance pass logs the same condition asaccount_maintenance_token_refresh_failed. - Cause: Kiro's token service rejected the refresh token or the OIDC client registration itself. Access-token errors (
bearer token ... is invalid) are not permanent and are handled by a forced refresh; only the refresh-dead markers park the account. - Remedy:
kiro-provider accounts relogin <id|email>. The account keeps its internal ID and session-affinity rows. A transienterror_codesuch asNETWORK_ERRORor a bareHTTP_<status>(proxy or WAF HTML, empty body) never marks the account dead; it records a separate authentication backoff, then the pipeline switches to an available account and maintenance retries when that backoff expires.refresh_token_dead: falsemeans the failure is not proven permanent; it can also be an OIDCinvalid_request, not just a network or proxy failure.
Authentication refresh fails although quota remains
The background maintenance loop runs every minute by default and refreshes access tokens five minutes before expiry. A successful generation immediately after login does not establish that the next refresh will succeed. Token candidates rotate fairly across passes. The token phase uses at most half the existing maintenance deadline, leaving the remainder for quota and usage; cancelling that waiter does not cancel a shared refresh or classify it as an authentication failure.
- Refresh failures have separate, persistent authentication state. The first retry is delayed by 30 seconds; repeated failures back off exponentially to 15 minutes.
accounts list --detailsexposesAUTH_ERROR,AUTH_RETRY_AT,CLIENT_EXPIRES, andLAST_REFRESH;--jsonexposes the corresponding lifecycle timestamps, bounded failure count, and classification enum, never credentials. The account remainsrefresh-faileduntil a successful refresh or a new login clears that state. Reaching the retry time permits a probe; it does not mark the credentials healthy automatically. - When every otherwise usable account fails authentication refresh, the result is
503 upstream_token_refresh_failed, including in a mixed pool containing unrelated quota-exhausted rows. Messages projects this asoverloaded_error, not a quota429 rate_limit_error. A genuine upstream quota rejection keeps its existing quota contract. invalid_requestalone is not enough to mark an account permanently dead. Known revoked/expired grant codes, OAuthexpired_token, and a known-expired OIDC client registration require re-login. New direct logins retain the registration issue/expiry times returned by Kiro; legacy rows with no such metadata remain unknown, not guessed expired.- Refresh success, failure, and cooldown updates are bound to the credential tuple that started the operation. A late result cannot overwrite or disable a newer login; ordinary usage updates do not invalidate that protection.
- Imported OpenCode credentials are a one-time snapshot, not a live credential database. Do not reintroduce shared database access or repeatedly overwrite a newer local login from an older import. Refreshing an access token cannot revive a revoked grant or extend an expired client registration indefinitely.
Use kiro-provider accounts relogin <id|email> for confirmed permanent expiry. For invalid_request, inspect the safe error classification and issuer region; if the authorization is no longer usable, re-login rather than repeatedly clearing a rate-limit timer. Never print refresh tokens or client secrets while diagnosing this condition.
Builder ID approval succeeds, then profile discovery returns HTTP 403
- Cause:
v3.5.2throughv3.8.6sent every new device login through KiroList-Available-Profiles. AWS Builder ID authentication succeeds, but Kiro deliberately rejects that organization-profile operation for personal accounts, so the provider discarded the valid token before opening its local account database. Changing the OIDC region or supplying--profile-arndoes not make that operation valid. - Remedy: Upgrade to a build containing the Builder ID profile fix and run
kiro-provider loginagain. Fixed builds skip the unsupported catalog only when the login is provably Builder ID and use the same fixed request-level service profile as the shipped Kiro CLI. IAM Identity Center logins still use strict profile discovery and never receive this fallback. If direct login is temporarily unavailable, import an already authenticatedopencode-kiro-authdatabase, or useaccounts import --format kiro-client --from <file|->with a supported legacy BuilderId export on a fixed build. Do not copy tokens or editaccounts.dbby hand.
profileArn is required for this request after an IAM Identity Center login
- Cause: The account row has no
profile_arn. Older direct-login builds could store the OIDC token and start URL without discovering the Kiro profile, so runtime requests reached Kiro without the required profile binding. - Remedy: Run
kiro-provider accounts relogin <id|email>. Current builds call KiroList-Available-Profilesdirectly and persist the selected ARN before usage or inference. No Kiro CLI installation is involved. If the identity has multiple profiles, pass--profile-arn <arn>; an unavailable ARN or ambiguous profile list fails before credentials are written. The provider keeps the token's OIDC region separate from the runtime region encoded by that ARN.
A row shows the placeholder email builder-id@aws.amazon.com
- Look at:
accounts list→EMAILcolumn; thelogincommand printedWarning: Kiro usage did not include an account email; storing the placeholder .... - Cause: The IAM Identity Center / Builder ID device-code flow does not return an email; the provider fills it from Kiro's
getUsageLimitsresponse. If that lookup failed or the response carried noemail, the placeholder is stored. - Remedy:
kiro-provider accounts refresh <id>once the usage endpoint is reachable; the email is updated on the next successful usage sync. The placeholder is functional (selection uses the account ID), butaccounts reloginagainst a placeholder row accepts any Kiro identity, so verify the row is the account you meant before re-authenticating. Two placeholder rows cannot be told apart by email; use theIDcolumn.
quota-exhausted vs overage-blocked, and 402 quota_exhausted vs 402 paid_overage_blocked
- Look at:
accounts list --details→AVAILABILITY,USAGE, andOVERAGEcolumns; auditquota_exhausted_account_persisted(warn,account_hash,recheck_after),quota_exhausted_accounts_excluded(info,account_count), andquota_exhausted_account_recovered(info); HTTP402witherror.codequota_exhaustedorpaid_overage_blocked(on/v1/messages:429 rate_limit_errorwith the code in the message). - Cause:
quota-exhausted: Kiro reported the account's quota as used up (an upstream402, or a usage snapshot at the limit). The account stays healthy and is re-admitted only after a due authoritative usage probe confirms a new quota window (quota_recheck_interval_ms, or the reset time Kiro reports).overage-blocked: the account still has quota or is in paid overage, butstop_on_overage(defaulttrue) excludes any account whose overage count exceedsoverage_threshold(default0). This is a selection gate, not a health signal; the next usage sync re-evaluates it.402 quota_exhausted: every otherwise eligible account is exhausted.402 paid_overage_blocked: every otherwise eligible account is blocked only by the overage gate.
- Remedy: For quota exhaustion, wait for the reset (the
Retry-Afterheader andrate_limit_wait_for_resetshow the wait) or add accounts. For overage blocking, decide explicitly: setstop_on_overage: falseto knowingly spend paid overage, or raiseoverage_threshold. Do not mark the account unhealthy; it is not.
503 no_healthy_accounts or 503 upstream_token_refresh_failed
- Look at: HTTP
503witherror.codeno_healthy_accounts("All accounts are unhealthy or rate-limited") orupstream_token_refresh_failed("Token refresh failed for every usable Kiro account"); auditaccount_token_refresh_failed/account_token_refresh_retry,rate_limit_wait_for_reset(wait_ms,remaining_ms), andaccounts list --detailsfor the per-account reason. - Cause:
no_healthy_accountsmeans no account passed selection withinrequest_timeout_ms: all arerate-limited,unhealthy,needs-relogin,quota-exhausted, oroverage-blocked, or the model is unavailable to the remaining ones (account_model_unavailable). When the earliest rate-limit reset fits inside the request deadline the pipeline waits for it instead of failing.upstream_token_refresh_failedmeans every candidate failed its access-token refresh during this request; a network or proxy outage that affects all accounts at once produces exactly this. - Remedy: Read the availability column first.
needs-relogin→accounts relogin;rate-limited→ wait forRECHECK_AT;overage-blocked→ see the previous entry. Forupstream_token_refresh_failed, checkproxy_urland outbound connectivity, thenkiro-provider accounts refresh --allto confirm the token endpoint is reachable again.GET /readyreturningno_active_accountsis the same condition seen from the readiness probe.
Streams
502 upstream_stream_incomplete / upstream_stream_error, and accepted streams
Look at: Non-stream requests return HTTP
502witherror.typeupstream_errorand the code; streams end with the same code in the terminal frame (response.failedfor Responses, anerrorframe for Chat,overloaded_errorfor Anthropic). In the audit log the incident itself issdk_stream_upstream_error(warn, witherror_code,error_disposition,error_type, hashed messages,raw_event_count,last_event_type,event_type_counts) orsdk_stream_idle_timeout(warn,idle_timeout_ms). Around it, the stream-resilience layer emits:Event Level Fields Meaning sdk_stream_attempt_retrywarnattempt,max_attempts,error_code,same_account,account_hashA recoverable attempt failed before publication; this includes an omitted-thinking stream that received zero raw frames, so a bounded replacement is allowed. sdk_stream_attempts_exhaustedwarnattempt,max_attempts,error_code,account_hashPre-publication recovery exhausted its attempt budget; the last failure becomes HTTP 502.sdk_stream_empty_completion_retrywarnattempt,max_attempts,account_hashNon-stream collection obtained an empty witnessed completion; one same-account replacement is allowed within the budget. sdk_stream_transport_error_after_completionwarnerror_code,account_hash,completion_witnessedThe transport failed after an authoritative completion witness (token usage or a valid metering event). The completed turn is delivered; the error is recorded, not surfaced. Cause:
upstream_stream_erroris a reader, decoder, transport, or embedded upstream failure;upstream_stream_incompleteis a clean EOF without a completion witness. CheckX-Request-ID,attempt_id, phase and the first/last failure evidence. Since v3.1.1 an accepted stream is not replayed, even if it produced only lifecycle or partial argument events. For omitted-thinking prefix inspection, a transport failure withraw_event_count: 0has no upstream acceptance evidence and may use the bounded in-request retry; any raw frame keeps the no-replay boundary. Absence of a retry event does not establish which upstream component failed.Remedy: Downstream retries as a replacement attempt with the same session key (see the contract), within its existing total budget and with no replay of side effects. Correlate actual upstream status/request IDs before attributing an account, region or network outage. Raw activity refreshes idle but never the total deadline. Raising
stream_max_attemptsdoes not change the accepted-stream boundary.
Reading sdk_stream_terminal: "the assistant announced a next step and stopped"
Every stream ends with exactly one sdk_stream_terminal event (info). Fields: terminal_provenance, completion_witnessed, witness_kind, reasoning_chars, visible_chars, tool_count, tool_intent_open, finish_reason_synthesized, plus account_hash and conversation_hash.
terminal_provenance | What happened | Who to blame |
|---|---|---|
normal_complete | Kiro sent a completion witness and the stream closed cleanly. | Nobody: this is Kiro ending the turn. |
idle_timeout | No upstream event for stream_idle_timeout_ms. | Transport / upstream stall. |
upstream_error | The SDK reader or Kiro reported an error mid-stream. | Transport / upstream. |
consumer_cancel | The client closed the response before the stream ended. | The client (its own timeout or user cancel). |
external_abort | The provider aborted the upstream: request_timeout_ms deadline, shutdown, or lock compromise. | Provider configuration or lifecycle. |
When a user reports "the model said Next, I'll run the tests and then stopped":
- Find the turn's
sdk_stream_terminal. terminal_provenance: normal_complete,completion_witnessed: true, andtool_count: 0means Kiro ended the turn without emitting a tool call. The provider delivered everything it received; the "stop" is model behaviour, not a truncated stream.tool_intent_open: truerecords that the visible text ended on an announced action with no tool call behind it, which is the pattern to count when comparing prompts or projection modes.finish_reason_synthesized: trueis normal here: Kiro exposes no stop reason, so the provider derivesend_turn/tool_usefromtool_count.upstream_errororidle_timeoutis a transport incident: the client received the typed in-stream error and should retry per the contract; look for the matchingsdk_stream_upstream_error/sdk_stream_idle_timeoutand any retry events.consumer_cancelmeans the client left first. Check the client's read/idle timeout before suspecting the gateway; the provider then aborted the upstream request so the account lease was released.- Compare
visible_charsandreasoning_charswith what the client shows. A largereasoning_charswith smallvisible_charson anormal_completeis a model that thought and answered briefly, not a lost stream.
Reasoning replay
400 unsupported_reasoning_display
Look at: Claude Code reports
capability_rejected:thinking.display: summarized cannot be represented by Kiro; the provider audit identifiesunsupported_reasoning_displaywithparam: thinking.display.Cause: A shared
showThinkingSummaries: truesetting or explicit--thinking-display summarizedflag asks for summaries on a model whose summary display the gateway cannot represent.Action: Upgrade the provider for Opus 5.5: it now preserves an explicit
summarizedrequest, including its effort and signed tool history, as it already does for Fable 5.1. Other models still reject summaries. Use the updated repositorykiroclaudelauncher, which setsshowThinkingSummaries: falsein its Kiro overlay. For an explicit display override, launch with--thinking-display omitted:bashPATH="$PWD/scripts:$PATH" kiroclaude --thinking-display omittedThis changes display only; Ultra/xhigh or max effort and opaque signed reasoning replay remain enabled. It does not edit shared Claude settings. Other API clients must request a supported display explicitly; unsupported summaries on other models still receive the typed rejection.
400 invalid_reasoning_signature
- Look at: HTTP
400,error.codeinvalid_reasoning_signature(Anthropic envelope:invalid_request_errorwith the same message). The upstream message names the offending block, e.g.messages.1.content.0: Invalid signature in thinking block. - Cause: Kiro validates replayed
thinkingsignatures server-side. The client echoed athinkingblock whosesignaturewas altered, truncated, re-encoded, or produced by a different model/provider. Changing the signed system/tool/history prefix can also invalidate it on the same account. Cross-account portability is limited to the verified model/region/runtime/ profile cells; it does not authorize arbitrary prefix changes. - Remedy: Replay the original signed context exactly. If it cannot be reconstructed, preserve the old transcript and transfer visible task state into a fresh conversation, explicitly acknowledging the loss of hidden reasoning. Outside a migrated portable replay, the provider does not retry, switch accounts, or degrade silently on this error, and does not mark the account unhealthy. When this
400lands on an attempt thatreasoning_replay_account_failover: "verified"had just migrated to another account, it is handled as a rejected migration instead (see400 reasoning_replay_migration_rejectedbelow): one bounded fallback runs on the origin account and conversation, and if the origin is unavailable the client receivesreasoning_replay_migration_rejectedrather than this code.
400 ... is not a valid single assistant reasoning block (Claude Code)
- Look at: HTTP
400from/v1/messages, audit codeinvalid_reasoning_replay, and a path such asmessages.11.content.1. - Cause: Some earlier gateway responses could leave two different signature-only
thinkingblocks in one Claude Code assistant tool turn. Kiro history has only one reasoning slot, so replaying either signature would be a guess and replaying both is impossible. - Remedy: Current builds preserve the visible assistant/tool history, omit all conflicting empty direct-reasoning envelopes, emit the response header
x-kiro-reasoning-replay-mode: conflict-omitted, and write the sanitized warn eventanthropic_reasoning_replay_conflict_omittedwith message/block counts. This repair is deliberately narrow: non-empty reasoning, providerkr1_/kr2_tokens, redacted blocks, or mixed conflict types still fail closed.
Fable conflicting upstream reasoning signatures
Enabled Fable thinking with effective display: omitted can safely omit the entire conflicting signature-only prefix before any output is published. The prefix is bounded to 128 events and 1 MiB and must contain no reasoning text or redacted payload. The provider continues the same upstream request, emits x-kiro-reasoning-replay-mode: conflict-omitted and a count-only anthropic_output_reasoning_conflict_omitted audit, and does not capture, store or mint a token for that reasoning. Non-empty, mixed, late or oversized conflicts remain fatal. After HTTP headers are committed, failure is an SSE error rather than a replacement HTTP 502.
Claude upgrade withdraws TaskOutput
Older Claude histories can contain completed TaskOutput calls even when a newer client no longer declares that tool. Messages now validates those call/result pairs independently from current tool declarations. New output must still match the current tools and schemas. Repeated continue requests cannot repair an older provider's missing_tool_declaration rejection.
This local fix does not rewrite signed history. If a changed tool prefix causes invalid_reasoning_signature, use the signed-context guidance above; do not synthesize a current declaration for an unavailable tool.
Memory-pressure background stops and ConnectionRefused
Check the provider listener, systemctl --user status kiro-provider.service and its journal before assuming a firewall failure. An OOM-killed provider cannot accept loopback connections. Claude's background-command pressure reaper can independently stop an idle command; that notice alone does not identify which process exhausted memory.
The provider's global request/body admission budgets reject excess work with 503 and Retry-After: 1 before upstream dispatch. They complement per-account concurrency; they do not guarantee a heap ceiling or cure unrelated host memory pressure. Inspect parent cgroups and the user manager when automatic restart is delayed. Do not disable pressure protection, restart stopped heavy gates automatically, or delete replay data as an OOM workaround.
400 unsupported_reasoning_plaintext_replay (Responses)
- Look at: HTTP
400,error.codeunsupported_reasoning_plaintext_replay,parampointing at theinput[i]reasoning item;invalid_reasoning_replayis the sibling code for a malformed or non-provider (kr1_/kr2_)encrypted_content. - Cause: The client replayed a
reasoningitem that carries only plaintextsummary/contentand noencrypted_contentanywhere in that turn. The provider never converts plaintext reasoning into a prompt, so it cannot be projected. - Recovery: With
responses_fidelity_mode: "compatible", stateless requests may omit a valid summary-only item whose assistant output group has no replay token. The response reportsX-Kiro-Compatibility: reasoning_summary_replay_omittedand a count-onlyresponses_reasoning_summary_omittedaudit. Visible assistant messages and tool history remain intact; lost opaque reasoning is not recovered or converted into model input. Strict mode, non-emptycontent, orphan items, unknown fields, and any present but invalid token still fail closed. - Prevention: Request
include: ["reasoning.encrypted_content"]and echo the returnedencrypted_content(kr2_...by default; legacykr1_...remains accepted) on the reasoning item when replaying the turn. Exactly one reasoning item per turn carries the token; other reasoning items in the same turn may keep plaintext summaries. A client that cannot store the token should omit reasoning items from history instead of sending summaries alone.
400 reasoning_replay_expired
- Cause: A current
kr2_token reached its authenticated absolute expiry, a database-backedkr1_row reached its idle expiry, or a pre-releasekr2_envelope reached the persisted compatibility cutoff. Restarting the provider does not extend any of these deadlines. - Remedy: Continue from a newer assistant turn that carries a fresh token, or start/fork from history that omits the expired reasoning item. Do not keep an old replay key solely to bypass expiry; key retention cannot override the authenticated token lifetime.
reasoning_replay_locked: false in upstream_affinity_selected
- Look at: The
infoeventupstream_affinity_selected(per attempt) withaffinity_kind,affinity_bound,account_hash,conversation_hash, andreasoning_replay_locked. - Meaning:
reasoning_replay_locked: truemeans the request replays encrypted reasoning that remains bound to the account that minted it, so account failover is disabled for this request (a failure returnsreasoning_replay_*rather than switching).falsemeans either no replay is present, the replay kind carries no local binding (a native Anthropicthinkingsignature), or an authenticated provider token matched an exact verified migration cell. An actual migration additionally emitsreasoning_replay_account_migratedwith hashes only.falseis normal and is not an error.
400 reasoning_replay_migration_rejected
Look at: HTTP
400,error.codereasoning_replay_migration_rejected("Kiro rejected the request after its signed reasoning history was migrated to another account, and the original account is unavailable"), and the audit sequence for thatrequest_id:reasoning_replay_account_migrated(info): a verified portable replay is being dispatched away from the cell it is bound to. Emitted once per migration the request makes; a later attempt that extends an already committed cell (an empty-completion replacement, the fallback of a rejected second hop) is not a migration and emits nothing. Fields:request_id,protocol,model,from_account_hash,to_account_hash,replay_count,legacy_replay_count,binding.binding: "deferred"means an explicit session-affinity binding will be committed only after Kiro accepts the attempt;binding: "none"means the request has no explicit session affinity store, so there is no stored binding to commit.reasoning_replay_migration_committed(info): Kiro accepted the migrated attempt (first streamed event, or a completed non-streaming collection) and the stored binding, when there is one, now names the new account and conversation. Fields:request_id,protocol,model,from_account_hash,to_account_hash,conversation_hash,replay_count;replay_mode: "omitted"identifies a binding committed after the explicit visible-only recovery.reasoning_replay_migration_commit_failed(warn): Kiro accepted the migrated attempt but writing the new binding to the accounts database failed (disk full, I/O error). The accepted answer is still served, the stored binding keeps naming the origin, and the next request re-resolves it. Fields: thecommittedfields pluserror_type(the error class name) anderror_code(the driver code, for exampleSQLITE_FULL, when present).reasoning_replay_migration_rejected(warn): Kiro rejected the migrated attempt before any output. Fields:request_id,protocol,model,from_account_hash,to_account_hash,replay_count,upstream_status(the HTTP status),upstream_code(the upstream reason enum,REQUEST_BODY_INVALIDorTHINKING_SIGNATURE_INVALID; omitted when the signature rejection arrived as a message-only400without a reason),fallback, andfallback_blocked_reason.fallback: "origin"means one fallback attempt ran on the origin account and conversation;fallback: "reasoning-omitted"means the origin could not serve and one same-target, fresh-conversation retry omitted the authenticated opaque reasoning;fallback: "none"means every recovery was blocked.fallback_blocked_reasonrecords why the origin could not serve:fallback_spent,origin_missing,origin_ineligible,origin_quarantined,origin_unselectable, ordispatch_budget.reasoning_replay_migration_omitted(warn): the bounded same-target recovery is being dispatched without opaque reasoning. It contains only the migration hashes andreplay_count; visible/tool history remains on the request.reasoning_replay_migration_omission_failed(warn): the visible-only retry was also rejected before output. The original typedreasoning_replay_migration_rejectedresponse is returned and the binding stays on the origin.
These events carry only hashes, counts, enums, and error class names or driver codes; never message text, prompts, signatures, or account identifiers.
Cause: The request replayed verified portable signed reasoning but could not stay on its origin account (quota exhausted, rate limited, unhealthy, model-ineligible, quarantined, or at
account_inference_concurrency), soreasoning_replay_account_failover: "verified"migrated it to another account with a fresh Kiro conversation. Kiro validated the migrated body server-side and rejected it with400 ValidationException/REQUEST_BODY_INVALID(Improperly formed request.) or an invalid reasoning signature before producing output. The provider never rewrites the stored binding before Kiro accepts a migrated attempt, so the session still points at the origin. When the origin is selectable the provider retries exactly once there, in the original conversation. Otherwise, with dispatch budget left, it retries exactly once on the same target and a fresh conversation after removing only authenticated opaque reasoning. The400surfaces only when neither bounded recovery succeeds or no dispatch budget remains.Remedy: A successful recovery carries
x-kiro-reasoning-replay-mode: migration-rejected-omitted; accept that the hidden reasoning was lost while visible assistant/tool history was preserved. If400still surfaces, wait for the origin account to recover (quota reset, health recovery, orkiro-provider loginre-login), or fork a fresh conversation without the signed reasoning. To disable migration and this fallback entirely, setreasoning_replay_account_failover: "strict"; owner failures then return the replay-bound codes below. Do not loop client retries or edit tool history: the provider already bounded both recovery paths.
Replay-bound account error codes
When owner-bound signed reasoning cannot proceed, the provider keeps the cause without switching accounts: reasoning_replay_account_quota_exhausted (402), reasoning_replay_account_rate_limited (429), reasoning_replay_account_reauthentication_required (403), reasoning_replay_account_refresh_failed (503), reasoning_replay_model_unavailable (503), or an owner unavailable/unhealthy 503. A failed forced refresh after upstream 401/invalid bearer returns the re-authentication code for dead credentials and the refresh code for transport/temporary refresh failures; it no longer degrades to generic reasoning_replay_account_unavailable. Re-login for the 403. For refresh 503, check proxy/egress and retry without changing the replay token or account.
Process and configuration
Startup fails with service_instance_already_running; single_instance_lock_busy or single_instance_lock_compromised in the log
- Look at: Startup error
Another kiro-provider instance already holds the service lock at <path> (gave up after N attempt(s) ...; a lock left behind by a dead process becomes stale after 15000 ms)with codeservice_instance_already_running; auditsingle_instance_lock_busy(warn, first retry:retry_attempts,retry_delay_ms,stale_ms),single_instance_lock_acquired(info,attempts), andsingle_instance_lock_compromised(error,error_code,stale_ms,update_ms,handler_count). - Cause:
enforce_single_instance: true(default) takes a lock file under the platform config root (instance_lock_pathoverrides). The lock is refreshed every 5 s and becomes stale 15 s after the owner stops refreshing it; acquisition retries 20 × 1 s so a restart afterSIGKILLsucceeds once the stale window passes.single_instance_lock_compromisedmeans the lock was lost while running (lock directory deleted, or the mtime refresh missed the stale window because the process was frozen). The provider fails closed: it stops accepting requests, drains for up to 10 s, and exits1so the service manager restarts it. - Remedy: For
already_running, confirm withsystemctl --user is-active kiro-provider.servicethat only one service is defined and no foregroundserveis running as the same user; a second copy for another OS user selects a different config root and lock. For a lock compromise, keep the config directory intact and investigate host suspend/resume or I/O stalls longer than 15 s. Disabling the lock (enforce_single_instance: false, logged assingle_instance_protection_disabled) is not a fix: two processes rotating the same refresh tokens invalidate each other.
config_file_permissions_loose
- Look at: Audit
config_file_permissions_loose(warn) withpath,mode(e.g.0644),recommended_mode: "0600", and ahint. - Cause: On POSIX the config file is group- or world-readable and it contains
api_keys. - Remedy:
chmod 600 <path>. The systemd unit in the README setsUMask=0077so files the service creates are private; the config file itself is created by you.
Startup fails with unknown key "..." (did you mean "...")
- Look at: The
ConfigLoadErrorprinted before the port is bound; no audit event because the process never reachesserve. - Cause: The config file is validated strictly. Typos and keys from other versions are rejected with a nearest-match suggestion; environment variables use the
KIRO_PROVIDER_*names listed in CONFIGURATION.md and are reported as(from KIRO_PROVIDER_...). - Remedy: Rename or delete the key. Legacy
opencode_auth_db_pathis still accepted but ignored and logsconfig_opencode_auth_db_path_deprecated.
Startup fails with auth_source "opencode-shared" was removed in kiro-provider 0.7.0
- Look at: The startup message itself:
Copy the OpenCode accounts once with "kiro-provider accounts import [--from <path>]", then set auth_source to "local" or delete the key. - Cause: The live-reader compatibility mode was removed in 0.7.0 because it reintroduced cross-process credential ownership and could block the event loop on the shared SQLite lock.
- Remedy: Run the one-time import as the same OS user that runs the service, then set
auth_sourceto"local"(or remove it;localis the default). The OpenCode database is not read afterwards.
Claude Code sessions are titled with the first prompt (e.g. /model)
- Look at: The session list shows the literal first input (
/model, a slash command, or the opening prompt) instead of an AI-generated title, and the transcript has noai-titlerecord. On providers older than this version the log showsprotocol_projection_rejectedwithcodeunsupported_parameterandparamoutput_config.formaton/v1/messages, usually followed by a second rejection withunsupported_message_fieldandparammessages.1.output_configas Claude Code retries once with the field moved onto a message. On current builds, a remainingunsupported_structured_outputrejection withparamoutput_config.formatmeans the supplied schema is outside the recognized title and Goal/prompt-hook profiles. Check the request shape rather than assuming the rejection came from title generation. - Cause: Claude Code 2.1.280 generates titles with a side request that carries
output_config.format(ajson_schemawhose root object has exactly one required string propertytitle). Older providers accepted onlyoutput_config.effort, so the title request failed and the client fell back to the first prompt. - Remedy: Upgrade the provider. Current builds accept that shape as the bounded local
single-string-object-v1profile, buffer the upstream text, validate it locally, and answer with one text block containing{"title":"..."}plusx-kiro-structured-output: single-string-object-v1. Each recognized request writes the info eventanthropic_structured_output_enforced(request id, model, stream flag, profile name, and hashes of the schema and property name only); a failed publication writes the warn eventanthropic_structured_output_failedwith the code (structured_output_validation_failed,structured_output_buffer_exceeded,structured_output_unexpected_tool_call,structured_output_unexpected_reasoning) and never the text. Two cases remain client-side and no provider can fix them: a session whose only inputs are slash commands, and one whose prompts are all under 10 characters, because Claude Code never requests a title for either. Existing transcripts are not rewritten; use/renamefor those.
Claude Goal stays active but the agent stops after promising more work
- Look at: The main request ends with
sdk_stream_terminalreportingnormal_completeand zero tools. The Claude transcript then records aHook evaluator API errorand no continuation. Older providers reject the evaluator withunsupported_structured_output,param: output_config.format. - Cause: The Goal Stop evaluator needs a JSON decision with
ok,reason, and optionalimpossible; the old title-only profile rejected its schema. A model completing its turn does not prove that the Goal is complete. - Remedy: Use a build supporting
hook-evaluation-v1, described in the Claude Code guide. Its accepted requests emitanthropic_structured_output_enforcedwith that profile; JSON and SSE responses carry the corresponding header. The client continues onok: falseand may end onok: trueorok: false, impossible: true. Evaluate actual follow-up tool execution as well as HTTP status. An invalid evaluator output remains a typedstructured_output_validation_failederror, not a fabricated completion decision. Claude Code owns how it handles evaluator failures; API compatibility does not guarantee every model judgment.
Request body too large vs context length exceeded
- Look at: HTTP
413.error.coderequest_too_large(messageRequest body exceeds the N byte limit) is the ingress body limit. A413witherror.typeupstream_erroranderror.codecontext_length_exceededon Responses is Kiro rejecting the prompt as exceeding the model context (structured reasonsCONTENT_LENGTH_EXCEEDS_THRESHOLD/PROMPT_TOO_LONG, or theinput is too longmessage on older responses); the provider remaps that upstream400to its shared pipeline413contract. Messages translates that exactcontext_length_exceededclassification to HTTP400invalid_request_error, with a stablePrompt is too longmessage and the provider code in the message. This lets Claude Code recognize context overflow and attempt compaction, instead of showing its generic 32 MB image/attachment error for HTTP 413. Sanitized upstream failure details remain available inerror.details. A true request-body overflow remains HTTP 413. - Cause: The first is a body larger than
max_request_body_bytes(default 32 MiB; Bun answers even larger bodies with a plain413before the JSON envelope). The second is conversation history, tool results, or attached documents exceeding the model's context window. - Remedy: For the body limit, raise
max_request_body_bytesonly if the payload is legitimate (large inline documents). For context overflow, the client must compact history; the provider does not truncate, summarize, or drop messages on the model's behalf. Therequest_shapediagnostic (input_text_chars,document_count,tool_result_count) shows which part of the request grew. - Already streaming: Once a Messages SSE response has committed HTTP 200, an upstream failure terminates with an SSE error rather than a new HTTP status. The accepted generation is not replayed. The HTTP 400 translation above applies to errors received before the response is committed.
Proxy problems (proxy_url)
- Look at: Startup
ConfigLoadErrorproxy_url must be a valid URL/proxy_url must be http(s); at runtimeaccount_token_refresh_retrythenaccount_token_refresh_failedwitherror_code: NETWORK_ERRORor a bareHTTP_<status>,sdk_stream_upstream_errorwith transport codes (ECONNREFUSED,ECONNRESET,ETIMEDOUT,ENOTFOUND,EAI_AGAIN),model_catalog_refresh_failed, and finally503 upstream_token_refresh_failedor503 no_healthy_accounts.sdk_connection_pool_selected(info) showshttp_keep_aliveand pool hits per account but does not include the proxy address. - Cause:
proxy_urlroutes all egress (model calls, token refresh, usage probes, device-code login) through one HTTP(S) proxy. SOCKS is not supported. A proxy that returns an HTML block page yieldsHTTP_<status>refresh errors that are deliberately treated as transient, so accounts stayrate-limitedrather thanneeds-relogin. - Remedy: Verify the proxy from the service user's shell (
curl -x "$PROXY" https://oidc.us-east-1.amazonaws.com/); setproxy_urlin the config file orKIRO_PROVIDER_PROXY_URL(theserve --proxyflag wins over both); restart the service.kiro-provider accounts refresh --allis the quickest end-to-end check because it exercises the token endpoint and the usage endpoint through the same proxy resolution.
Request-shape diagnostics (request_shape, debug)
Set log_level: "debug" (or KIRO_PROVIDER_LOG_LEVEL=debug) and every Responses, Messages, and Chat request emits one request_shape event after its canonical request is built and before account selection. It contains only counts, booleans, one hash, and two labels:
| Field | Meaning |
|---|---|
request_id | Random per-public-request correlation id shared with projection, dispatch, and terminal events. |
protocol | responses, anthropic-messages, or chat-completions. |
projection_mode | v3-auto, safe, native-context-safe, or legacy-user-prefix. |
model | The requested public model name. |
message_count | Canonical messages after adaptation. |
user_message_count, assistant_message_count, tool_message_count, instruction_message_count | Role counts (instruction_message_count is system plus developer). |
tool_declaration_count | Declared tools. |
tool_call_count | Tool calls in history (assistant toolCalls plus tool_use parts, unique by id per message). |
tool_result_count | Tool results in history. |
orphan_tool_result_count | Results whose call id matches no call in an earlier message. |
image_count, document_count | Inline attachments. |
has_reasoning_replay, reasoning_replay_count | Whether and how many encrypted reasoning replays the request carries. |
system_instruction_present | Top-level instructions/system or a system/developer message exists. |
input_text_chars | Sum of text lengths over messages, tool-result text, and instructions. A size, never the content. |
tool_set_hash | auditHash of the sorted tool names, so identical tool sets correlate across requests without logging the names. |
Use it to answer "did the client send the whole history?", "is a tool result arriving without its call?", or "how big is this conversation?" without ever enabling request-body logging, which the provider does not offer.
At debug, the same request_id also appears on request_projection_completed and request_history_built. At info, each real SDK send emits sdk_dispatch_started with a one-based attempt; the same pair is copied to completion-witness and sdk_stream_terminal events. request_transform_rejected records only the stage, code, and source path. These events contain counts, enum labels, lengths, and hashes only.
Conflicting summarized reasoning signatures after upgrading
If an existing Claude session reports HTTP 502 with conflicting Kiro reasoning signatures, inspect its effective thinking display. Paseo can pass an explicit --thinking-display summarized, which overrides the launcher's preference. An operator may set anthropic_thinking_display_mode: "omitted" in the gateway configuration and restart it, or use KIRO_PROVIDER_ANTHROPIC_THINKING_DISPLAY_MODE=omitted. Existing requests then use omitted current output without lowering effort or editing signed history. Converted responses report x-kiro-thinking-display-mode: forced-omitted.
This is an explicit recovery policy. Summarized support remains strict when the policy is preserve, and nonempty or malformed conflicts still fail closed in recovery mode. sdk_reasoning_signature_observed records signature positions, lengths, relation enums and reasoning character counts; it never logs signature values, hashes or thinking text. These metrics distinguish real failures from short-request successes, but cannot alone prove that distinct signatures are fragments of one replayable value. Do not concatenate them without executable upstream replay evidence.
Replay context 400 after omitted-thinking recovery
A replay-context 400 is local binding validation, before inference. It is separate from Kiro's upstream signature-conflict 502. If a Claude conversation contains Edit default insertion or Bash directory-prefix removal, the bounded authenticated tool-input recovery handles those evidenced shapes without changing the model or effort. A rejected token still requires the complete original output, tenant, model and normalization scope. Do not disable binding validation or delete nonempty thinking merely to bypass the error.