Skip to content

Track background jobs

A tool that starts work and returns before the work is done reports ok for the start, not for the work. A build handed to a thread, a render, a simulator that dies minutes after the call that launched it: the call row cannot say how those ended. Events can, and one join then answers the question that matters, “which calls said ok while the work they started failed?”

The pattern on this page comes from mcardupilot, an MCP server for ArduPilot simulation (docs), whose builds, video renders and simulator sessions all outlive the call that starts them. It has four parts:

  1. a uuid per job, used as the event key;
  2. job.started, recorded inside the start call, so it carries that call’s id;
  3. job.finished, recorded by the worker thread when the work ends;
  4. an enricher that writes the uuid into the call row’s extra.job.

The key joins events to each other and to the call, so it must be unique for as long as you keep records, across restarts. The ids a server already hands out rarely are. mcardupilot’s build id names the build output, so building the same target again gives the same id; a per-boot counter starts over at every restart; a hash of the job’s inputs repeats whenever the inputs do. Make a uuid per job and keep it next to the id the tool returns:

import threading
import time
import uuid
from dataclasses import dataclass, field
@dataclass
class Job:
uid: str # the event key: new for every job
build_id: str # what the tool returns; repeats for the same target
report_delay: float = 0.0 # tests only: how long the finish takes to report
cancelled: threading.Event = field(default_factory=threading.Event)
done: threading.Event = field(default_factory=threading.Event)
jobs: dict[str, Job] = {} # build_id -> its latest job
started: list[Job] = [] # every job, for waiting on them

Record the start inside the call, the finish from the thread

Section titled “Record the start inside the call, the finish from the thread”

job.started is recorded inside the tool, so it gets the call’s id as call_id, plus its session and caller. job.finished comes from the worker thread. record_event is safe from any thread and never blocks or raises, so the thread does not need to care whether instrumentation is on.

from fastmcp import FastMCP
from fastmcp_feedback.instrumentation import DatabaseSink, instrument
def run_build(recorder, job: Job, fail: bool) -> None:
"""The worker thread: do the work, then report how it ended."""
ok, error = True, None
try:
for _ in range(20): # the real work, in steps that notice a cancel
if job.cancelled.is_set():
break
time.sleep(0.01)
if fail and not job.cancelled.is_set():
raise RuntimeError("waf: compile failed in libraries/AP_Motors")
except Exception as exc:
ok, error = False, str(exc)
finally:
status = "cancelled" if job.cancelled.is_set() else ("ok" if ok else "failed")
time.sleep(job.report_delay)
# Report first, then mark done: shutdown waits on done, then flushes.
recorder.record_event(
"job.finished",
key=job.uid,
attrs={"ok": ok and status == "ok", "status": status, "error": error},
)
job.done.set()
def enrich(tool, args, result, context):
"""The job behind a result, so the call row joins to its events."""
if isinstance(result, dict) and (job := jobs.get(result.get("build_id"))):
return {"job": job.uid}
return {}
app = FastMCP("Builder")
mw = instrument(app, [DatabaseSink("sqlite+aiosqlite:///jobs.db", create_tables=True)], enricher=enrich)
@app.tool
async def build_sitl(target: str, fail: bool = False) -> dict:
"""Start a build and return at once."""
job = Job(uid=str(uuid.uuid4()), build_id=f"{target}-build")
jobs[job.build_id] = job
started.append(job)
mw.record_event("job.started", key=job.uid, attrs={"build_id": job.build_id})
threading.Thread(target=run_build, args=(mw, job, fail), daemon=True).start()
return {"build_id": job.build_id, "status": "started"}

The enricher runs after the tool returns, so the job it looks up is the one this call just started, even when build_id repeats.

Start two builds of the same target, the second of which fails. Both calls return ok and the same build_id; only the uuid tells the jobs apart:

import asyncio
from fastmcp import Client
from sqlalchemy import text
FAILED_AFTER_OK = """
SELECT c.tool, c.id, json_extract(e.attrs, '$.error') AS error
FROM ffb_tool_calls c
JOIN ffb_events e ON json_extract(c.extra, '$.job') = e.key
WHERE c.ok AND e.kind = 'job.finished' AND json_extract(e.attrs, '$.ok') = 0
"""
async def demo():
async with Client(app) as client:
first = await client.call_tool("build_sitl", {"target": "copter"})
second = await client.call_tool("build_sitl", {"target": "copter", "fail": True})
assert first.data["build_id"] == second.data["build_id"] # the id repeats
await asyncio.to_thread(lambda: [job.done.wait(5) for job in started])
await mw.flush()
async with mw.database_sink.engine.connect() as conn:
rows = (await conn.execute(text(FAILED_AFTER_OK))).all()
starts = (await conn.execute(text(
"SELECT e.call_id = c.id FROM ffb_events e JOIN ffb_tool_calls c"
" ON json_extract(c.extra, '$.job') = e.key WHERE e.kind = 'job.started'"
))).scalars().all()
return rows, starts
rows, starts = asyncio.run(demo())
[(tool, call_id, error)] = rows
assert tool == "build_sitl" and error == "waf: compile failed in libraries/AP_Motors"
assert starts == [1, 1] # each job.started carries the id of the call that made it

On PostgreSQL the same join reads c.extra->>'job' = e.key ... AND (e.attrs->>'ok')::boolean = false.

Event attrs are stored as given, up to the middleware’s max_payload_bytes (64 KiB by default); a larger value becomes a marker with its size and the start of its text. A failed build’s error can carry a page of compiler output, so if you want rows that stay readable in a query, clip it yourself before recording, for example "error": error and error[:500]. Before 2026.10.04 attrs had no limit at all, and clipping was the only bound.

Shut down without losing the last finishes

Section titled “Shut down without losing the last finishes”

At shutdown the server cancels running jobs, and their threads report job.finished with status: "cancelled" on the way out. Two rules keep those events:

  • Record job.finished before marking the job done, as run_build does. Shutdown waits on done; if done is set first, the wait can end and the middleware close before the event is recorded.
  • Wait for cancelled jobs, bounded, before aclose(). aclose() flushes what has been recorded and closes the sinks. From then on the middleware is closed: since 2026.10.05 a late record_event, from the loop or a thread, returns False, and the event is dropped and counted in mw.dispatcher.drop_reasons["dispatcher closed"], never written. Dropped is still lost, so wait first, and bound the wait so a stuck thread cannot hold shutdown up.
from fastmcp_feedback.instrumentation import InstrumentationMiddleware, MemorySink
JOB_EXIT_S = 5.0 # the longest shutdown waits for cancelled jobs
def wait_all(running: list[Job], budget_s: float) -> None:
end = time.monotonic() + budget_s
for job in running:
job.done.wait(max(0.0, end - time.monotonic()))
async def shutdown(recorder, running: list[Job], wait: bool = True) -> None:
for job in running:
job.cancelled.set()
if wait:
await asyncio.to_thread(wait_all, running, JOB_EXIT_S)
await recorder.aclose() # last: flush what the threads just recorded

In a real server this is the end of the lifespan, after the sessions close and the jobs are cancelled.

This is a race, so a test can pass by luck: a cancelled thread that reports quickly enough lands its event before aclose() even without the wait. mcardupilot found that, and its shutdown test slows the report by half a second so that only the waiting version keeps the event. Without the wait the finish arrives after the close and is counted as dropped:

async def stop_while_running(wait: bool):
sink = MemorySink()
recorder = InstrumentationMiddleware([sink])
job = Job(uid=str(uuid.uuid4()), build_id="plane-build", report_delay=0.5)
recorder.record_event("job.started", key=job.uid)
threading.Thread(target=run_build, args=(recorder, job, False), daemon=True).start()
await shutdown(recorder, [job], wait=wait)
await asyncio.to_thread(job.done.wait, 2) # let the thread end either way
events = [(e.kind, e.attrs.get("status")) for e in sink.events]
return events, recorder.dispatcher.drop_reasons
assert asyncio.run(stop_while_running(wait=True)) == (
[("job.started", None), ("job.finished", "cancelled")],
{},
)
assert asyncio.run(stop_while_running(wait=False)) == (
[("job.started", None)],
{"dispatcher closed": 1},
)

Before 2026.10.05 a record that arrived after aclose() while the loop was still running restarted delivery and was written to the sinks that had just been closed; once the loop stopped, it was dropped without being counted.