Detection is useless without enforcement. The kill switch is the half of Phronis that turns “this agent is misbehaving” into “this agent cannot make another call.” From first bad call to enforced stop is about 600ms, and the stop survives a container restart.
// 01 — THE EXECUTOR
AlertExecutor is a background thread inside your process consuming the phronis.alerts topic. When an alert arrives, it calls registry.trip(agent_id, reason), flipping the agent to halted in an in-memory CircuitBreakerRegistry. One important config detail: enable.auto.commit = False, with manual commits after each alert, so an unclean restart can’t lose or replay offsets (that distinction caused a real bug, linked below).
// 02 — ENFORCEMENT INSIDE THE LOOP
The @tool decorator checks registry.is_halted(agent_id) at the start of every tool call. If the agent is tripped, it raises CircuitBreakerTripped immediately, before any external API call happens:
class CircuitBreakerTripped(Exception):
# carries .agent_id and .reasonThe enforcement point is inside the agent’s own call path, which is why the stop is immediate rather than advisory.
// 03 — DURABLE STATE
A circuit breaker that forgets on restart is a liability. The Kill Switch API (FastAPI) mirrors every trip and reset to PostgreSQL. On startup it runs init_db() and load_all(), restoring any previously-halted agents back into the registry, so a crashed or redeployed API comes back still enforcing the breakers that were open. If PostgreSQL is unreachable, it degrades to pure in-memory rather than failing. Mutating endpoints (/halt, /reset) are guarded by an X-API-Key; read-only ones stay open.
TAKEAWAYS
- The enforcement point must be inside the agent’s call loop, checking halted-state before each tool call, for the stop to be immediate rather than advisory.
- Disable auto-commit on the alert consumer and commit manually; lazy offset handling replays stale alerts after an unclean restart.
- Circuit-breaker state is safety state. Persist it. A breaker that resets on restart silently re-opens the door it was closing.
NEXT
- Build log 05: cold storage that time-travels: Iceberg on MinIO.
