1.42.1 — Streamed answers and tool use on Bedrock
Editorial identity incomplete
2026-11-09
The guarded LLM helper can now stream a Bedrock answer as it is written (llm.stream), and can let a Bedrock model ask for tools the caller declares, in both plain and streamed calls. Every streamed and tool call goes through the same safety policy, budgets, circuit breaker and ledger as llm.call. Existing callers see no change until a route opts in.
Added
llm.stream()streams a Bedrock route's answer:text_deltaevents as it is written, then onefinalevent. Itsresponseis exactly whatllm.call()returns, and it is sent only after usage has been settled.- A refused stream raises
LLMExecutionErrorfromllm.stream()itself, before anything is sent. A failure partway through raises from iteration. LLMStream.close()cancels a stream. Called from an event loop, or while another thread is reading, it only shuts the connection, and the iterating thread records the cancel.- Bedrock routes may declare
"tools"alongside"text". Callers declare tools withname,descriptionandinput_schema, receivetool_useblocks, and answer withtool_resultblocks. The helper never runs a tool itself. llm.call()andllm.stream()acceptmax_tool_rounds(1–32). It defaults to 8 on Bedrock; Anthropic callers are unaffected unless they pass it.- A cancelled stream's ledger row finishes as
failedwith the codecancelled.
Changed
- A Bedrock request is now checked for a valid shape before any budget is reserved. A malformed request is refused with
provider_rejectedand leaves no ledger row. - Bedrock responses no longer include text blocks that are only whitespace in
content. - A Bedrock conversation that contains
tool_useortool_resultblocks withouttools=is refused withcapability_unsupported.
Upgrade notes
- Adding
"tools"to a route changes the policy hash: deploy, then activate the deployed policy again. - Bedrock tool calls must pass
context["operation_id"]and reuse it for every call in one loop. A loop of N tool rounds makes N+1 calls, so the feature'smax_loop_callsmust be at least N+1. - Streaming needs the IAM action
bedrock:InvokeModelWithResponseStreamon the same inference profile and model resources asbedrock:InvokeModel. - A stream's deadline is the route's
timeout_seconds, and it includes time the caller spends between reads. llm.stream()is Bedrock-only; on an Anthropic route it is refused withcapability_unsupported.