Track background jobs
A tool that starts work and returns before the work is done reports ok for
the start, not for the work. A build handed to a thread, a render, a simulator
that dies minutes after the call that launched it: the call row cannot say how
those ended. Events can, and one join then answers the question that matters,
“which calls said ok while the work they started failed?”
The pattern on this page comes from mcardupilot, an MCP server for ArduPilot simulation (docs), whose builds, video renders and simulator sessions all outlive the call that starts them. It has four parts:
- a uuid per job, used as the event key;
job.started, recorded inside the start call, so it carries that call’s id;job.finished, recorded by the worker thread when the work ends;- an enricher that writes the uuid into the call row’s
extra.job.
Key each job by a uuid
Section titled “Key each job by a uuid”The key joins events to each other and to the call, so it must be unique for as long as you keep records, across restarts. The ids a server already hands out rarely are. mcardupilot’s build id names the build output, so building the same target again gives the same id; a per-boot counter starts over at every restart; a hash of the job’s inputs repeats whenever the inputs do. Make a uuid per job and keep it next to the id the tool returns:
import threadingimport timeimport uuidfrom dataclasses import dataclass, field
@dataclassclass Job: uid: str # the event key: new for every job build_id: str # what the tool returns; repeats for the same target report_delay: float = 0.0 # tests only: how long the finish takes to report cancelled: threading.Event = field(default_factory=threading.Event) done: threading.Event = field(default_factory=threading.Event)
jobs: dict[str, Job] = {} # build_id -> its latest jobstarted: list[Job] = [] # every job, for waiting on themRecord the start inside the call, the finish from the thread
Section titled “Record the start inside the call, the finish from the thread”job.started is recorded inside the tool, so it gets the call’s id as
call_id, plus its session and caller. job.finished comes from the worker
thread. record_event is safe from any thread and never blocks or raises, so
the thread does not need to care whether instrumentation is on.
from fastmcp import FastMCPfrom fastmcp_feedback.instrumentation import DatabaseSink, instrument
def run_build(recorder, job: Job, fail: bool) -> None: """The worker thread: do the work, then report how it ended.""" ok, error = True, None try: for _ in range(20): # the real work, in steps that notice a cancel if job.cancelled.is_set(): break time.sleep(0.01) if fail and not job.cancelled.is_set(): raise RuntimeError("waf: compile failed in libraries/AP_Motors") except Exception as exc: ok, error = False, str(exc) finally: status = "cancelled" if job.cancelled.is_set() else ("ok" if ok else "failed") time.sleep(job.report_delay) # Report first, then mark done: shutdown waits on done, then flushes. recorder.record_event( "job.finished", key=job.uid, attrs={"ok": ok and status == "ok", "status": status, "error": error}, ) job.done.set()
def enrich(tool, args, result, context): """The job behind a result, so the call row joins to its events.""" if isinstance(result, dict) and (job := jobs.get(result.get("build_id"))): return {"job": job.uid} return {}
app = FastMCP("Builder")mw = instrument(app, [DatabaseSink("sqlite+aiosqlite:///jobs.db", create_tables=True)], enricher=enrich)
@app.toolasync def build_sitl(target: str, fail: bool = False) -> dict: """Start a build and return at once.""" job = Job(uid=str(uuid.uuid4()), build_id=f"{target}-build") jobs[job.build_id] = job started.append(job) mw.record_event("job.started", key=job.uid, attrs={"build_id": job.build_id}) threading.Thread(target=run_build, args=(mw, job, fail), daemon=True).start() return {"build_id": job.build_id, "status": "started"}The enricher runs after the tool returns, so the job it looks up is the one
this call just started, even when build_id repeats.
Find the ok calls whose job failed
Section titled “Find the ok calls whose job failed”Start two builds of the same target, the second of which fails. Both calls
return ok and the same build_id; only the uuid tells the jobs apart:
import asyncio
from fastmcp import Clientfrom sqlalchemy import text
FAILED_AFTER_OK = """SELECT c.tool, c.id, json_extract(e.attrs, '$.error') AS errorFROM ffb_tool_calls cJOIN ffb_events e ON json_extract(c.extra, '$.job') = e.keyWHERE c.ok AND e.kind = 'job.finished' AND json_extract(e.attrs, '$.ok') = 0"""
async def demo(): async with Client(app) as client: first = await client.call_tool("build_sitl", {"target": "copter"}) second = await client.call_tool("build_sitl", {"target": "copter", "fail": True}) assert first.data["build_id"] == second.data["build_id"] # the id repeats await asyncio.to_thread(lambda: [job.done.wait(5) for job in started]) await mw.flush() async with mw.database_sink.engine.connect() as conn: rows = (await conn.execute(text(FAILED_AFTER_OK))).all() starts = (await conn.execute(text( "SELECT e.call_id = c.id FROM ffb_events e JOIN ffb_tool_calls c" " ON json_extract(c.extra, '$.job') = e.key WHERE e.kind = 'job.started'" ))).scalars().all() return rows, starts
rows, starts = asyncio.run(demo())[(tool, call_id, error)] = rowsassert tool == "build_sitl" and error == "waf: compile failed in libraries/AP_Motors"assert starts == [1, 1] # each job.started carries the id of the call that made itOn PostgreSQL the same join reads
c.extra->>'job' = e.key ... AND (e.attrs->>'ok')::boolean = false.
Keep error text small
Section titled “Keep error text small”Event attrs are stored as given, up to the middleware’s
max_payload_bytes (64 KiB by
default); a larger value becomes a marker with its size and the start of its
text. A failed build’s error can carry a page of compiler output, so if you
want rows that stay readable in a query, clip it yourself before recording,
for example "error": error and error[:500]. Before 2026.10.04 attrs had no
limit at all, and clipping was the only bound.
Shut down without losing the last finishes
Section titled “Shut down without losing the last finishes”At shutdown the server cancels running jobs, and their threads report
job.finished with status: "cancelled" on the way out. Two rules keep
those events:
- Record
job.finishedbefore marking the job done, asrun_builddoes. Shutdown waits ondone; ifdoneis set first, the wait can end and the middleware close before the event is recorded. - Wait for cancelled jobs, bounded, before
aclose().aclose()flushes what has been recorded and closes the sinks. From then on the middleware is closed: since 2026.10.05 a laterecord_event, from the loop or a thread, returns False, and the event is dropped and counted inmw.dispatcher.drop_reasons["dispatcher closed"], never written. Dropped is still lost, so wait first, and bound the wait so a stuck thread cannot hold shutdown up.
from fastmcp_feedback.instrumentation import InstrumentationMiddleware, MemorySink
JOB_EXIT_S = 5.0 # the longest shutdown waits for cancelled jobs
def wait_all(running: list[Job], budget_s: float) -> None: end = time.monotonic() + budget_s for job in running: job.done.wait(max(0.0, end - time.monotonic()))
async def shutdown(recorder, running: list[Job], wait: bool = True) -> None: for job in running: job.cancelled.set() if wait: await asyncio.to_thread(wait_all, running, JOB_EXIT_S) await recorder.aclose() # last: flush what the threads just recordedIn a real server this is the end of the lifespan, after the sessions close and the jobs are cancelled.
This is a race, so a test can pass by luck: a cancelled thread that reports
quickly enough lands its event before aclose() even without the wait.
mcardupilot found that, and its shutdown test slows the report by half a
second so that only the waiting version keeps the event. Without the wait the
finish arrives after the close and is counted as dropped:
async def stop_while_running(wait: bool): sink = MemorySink() recorder = InstrumentationMiddleware([sink]) job = Job(uid=str(uuid.uuid4()), build_id="plane-build", report_delay=0.5) recorder.record_event("job.started", key=job.uid) threading.Thread(target=run_build, args=(recorder, job, False), daemon=True).start() await shutdown(recorder, [job], wait=wait) await asyncio.to_thread(job.done.wait, 2) # let the thread end either way events = [(e.kind, e.attrs.get("status")) for e in sink.events] return events, recorder.dispatcher.drop_reasons
assert asyncio.run(stop_while_running(wait=True)) == ( [("job.started", None), ("job.finished", "cancelled")], {},)assert asyncio.run(stop_while_running(wait=False)) == ( [("job.started", None)], {"dispatcher closed": 1},)Before 2026.10.05 a record that arrived after aclose() while the loop was
still running restarted delivery and was written to the sinks that had just
been closed; once the loop stopped, it was dropped without being counted.
See also
Section titled “See also”- Record background work with events: keys, threads, and
reading events back with
events_for. - Add identity and enrichment: the enricher.
- Records:
ToolCallRecord,EventRecordand the truncation marker.