How mcqemu handles failure

An MCP server that manages VMs is in an unusual position. Its callers are often models, which are good at acting on a plausible-sounding sentence and less good at doubting one. Its subjects are long-lived processes holding gigabytes of state that no one wants to lose. And it works on a host that is doing other things at the same time.

Four rules follow from that. None of them is clever; all of them are the boring option, chosen deliberately.

Never delete without verifying what you waited for

Waiting for something and then assuming it happened is the standard way to destroy data. sandbox_destroy is the only tool in mcqemu that deletes files, so it is where this rule is enforced hardest.

It sends quit over the monitor, then polls until the PID is actually gone, giving it ten seconds. If the process is still there it escalates to SIGKILL and polls again for five. If the process is still alive after that, it stops and fails, and nothing is deleted. The error names the PID and says the overlay is still open by that process.

That last branch is the point. Unlinking a disk image that QEMU still has open does not free the space (the kernel keeps the inode alive for the open file descriptor) and it does not stop the guest writing, so you end up with a running VM writing into a file that no longer has a name, consuming disk you cannot account for. Refusing to delete leaves you with a mess you can see, which is strictly better than a mess you cannot.

The same instinct shows up in the surrounding checks. sandbox_destroy refuses outright to touch a VM that was not created by sandbox_vm, because stop_vm and forget_vm never delete disks and users reasonably assume the same of everything else. It re-validates the VM name it was handed rather than trusting the registry key it came from, since a corrupted record is exactly the input that turns a delete into a disaster. It resolves the directories it is about to remove and confirms they are genuinely inside the state and runtime roots, and are not the roots themselves. Cleanup errors are collected rather than swallowed, so the result distinguishes “destroyed” from “mostly destroyed, here is what is left”.

Degrade instead of dying

If the registry file is unreadable, mcqemu does not refuse to start. It moves the damaged file aside to a timestamped quarantine name, starts with an empty registry, and records what happened. list_vms returns that record in registry_warnings alongside the (now empty) VM list. Individual records that fail to parse are skipped one at a time with their own warning, rather than taking the whole file down with them.

The reasoning is a direct consequence of the process model described in Architecture. VMs outlive this server. A server that will not start because its notebook is corrupt is a server that cannot stop the runaway VM eating the host’s memory, and the notebook has no bearing on whether that VM exists. Bookkeeping failures must not disarm the controls.

The warning matters as much as the degradation. A quietly empty list would be read as “no VMs are running”, which in this situation is exactly wrong: VMs may well be running untracked, and their QMP sockets are still sitting under the runtime directory where attach_vm can pick them up. So the tool’s contract makes the warning list impossible to miss, and its own description tells the caller to report it rather than treat the list as complete.

A registry written by a newer mcqemu is handled the other way: if the schema version is higher than this build understands, it is not loaded and not quarantined, because reading it optimistically and writing it back would damage records the newer build owns. Refusing and saying so is the safe direction.

The same preference for a usable degraded state appears at launch. If QEMU exits zero (which means the VM is up and its monitor is listening) but the pidfile cannot be read, the launcher does not raise. Raising there would leave a running, unregistered VM that no tool could find or stop, which is the worst outcome available. It returns a record without a PID, and only fails when neither a pidfile nor a live QMP socket can be found, at which point it prints the pgrep command to locate the process by hand.

Do not guess a cause you have not checked

The most consequential error message in mcqemu is the one for a failed QMP connect, and it has two forms.

If the process is not alive, the message says the VM has likely exited and points at the QEMU log. If the process is alive, it says something quite different: that QEMU’s monitor accepts one client at a time, that another tool call or an external qmp-shell may be holding it, and, explicitly, not to relaunch the VM.

The difference is not politeness. A caller told “the VM has exited” will do the sensible thing and start it again, and if the VM was merely busy that means two QEMU processes fighting over one disk image, or a launch that fails confusingly on a locked image, or in the worst case a second guest writing to a filesystem the first one is already mounting. The wrong diagnosis produces a wrong action that produces real damage. So the liveness check runs before the message is written, and the message only claims what was checked.

The same principle appears in a few other places:

Waiting for a shutdown polls the process, not just the event queue. The client library’s event queue never wakes on a dropped connection, so a VM that crashes mid-wait would burn the caller’s entire timeout and then be reported as a guest that ignores ACPI, sending them to force=True for a machine that already died. Polling liveness once a second turns that into “process exited during shutdown”, which is what happened.

The identity check on a PID is deliberately one-sided. It compares the -name in /proc/<pid>/cmdline against the registry key, and returns “this is not a different VM” whenever it cannot prove otherwise: unreadable cmdline, no -name present, a truncated argument list. Its job is to veto a mistaken identification, so an inconclusive read must not be allowed to contradict a positive liveness check.

Liveness for an attached VM connects to its socket rather than stat-ing it, because a SIGKILLed QEMU leaves the socket file behind and the file’s existence proves nothing. The same check protects launch: a leftover socket file under a name you are reusing is deleted, but only after confirming that nothing is listening on it, since an unregistered but running QEMU would otherwise lose its monitor and become permanently unreachable.

And when a tool cannot tell, it says so. Timeouts report the events actually seen while waiting. Guest-agent failures name the specific thing that is missing (qemu-guest-agent installed and running in the guest) rather than reporting a generic connection error, and a guest agent that refuses a command on a RHEL-family guest gets the specific advice about BLOCK_RPCS in /etc/sysconfig/qemu-ga.

Bound every wait

Every operation that waits has a deadline. An unbounded wait in a server managing multiple VMs does not just hang one call, it holds that VM’s session lock and blocks every subsequent call against the same machine, so one wedged guest takes out an entire VM’s tooling.

Wait	Bound	Why that number
QMP connect	5s	Either the socket is there or it is not
QMP command	30s	Ordinary monitor commands are fast; 30s means wedged
Guest-agent handshake	3s	Doubles as the “is there an agent?” probe
Guest-agent call	10s	The handshake succeeded, so the agent was alive a moment ago
guest_exec	Caller’s timeout, plus an outer bound	The poll loop checks the clock between awaits, so a single hung call needs its own ceiling
stop_vm (graceful)	30s, configurable	Long enough for a normal shutdown, short enough to notice ACPI being ignored
savevm / loadvm	900s	Writing or reading all of guest RAM legitimately takes minutes
sandbox_vm agent wait	90s, configurable	A cold boot plus the agent’s own startup

savevm is the interesting one. Snapshotting a large VM writes its entire RAM into the qcow2 file and can honestly take minutes, so a 30 second bound would turn a working operation into a spurious failure. The response is a much longer bound rather than no bound at all, because “this can take a while” is not the same statement as “this can take forever”, and a wedged snapshot still needs to surface eventually.

sandbox_vm shows the other half of bounding a wait well: it does not simply sleep until its deadline. While waiting for the guest agent it also checks whether the VM is still alive, so a guest that dies during boot is reported as having exited, with the tail of the QEMU log, at the moment it happens, instead of being blamed ninety seconds later on a missing guest agent.

Concurrency is a failure mode too

Two mcqemu instances can share one registry file, and nothing prevents it, so every write is a read-modify-write under an exclusive file lock and the atomic rename uses a temp name carrying the writer’s PID. Without the lock the second writer’s snapshot silently erases the first writer’s VMs, and those VMs keep running with nobody tracking them. list_vms re-reads the file before answering so records another instance created are visible.

Within one process, calls against the same VM are serialized by a per-VM lock (the monitor socket takes one client at a time), and a launch reserves its name for the duration, because launching involves awaits between checking that a name is free and registering it, and two concurrent launches would otherwise both pass the check and race over the same sockets and pidfile.

What all of this costs

More calls fail than would otherwise. A tool that refuses to delete, refuses to guess, and gives up after a timeout will return errors in situations where a more optimistic implementation would have carried on and usually been fine. That is the trade being made, and it is made on purpose: the failures it avoids are the ones that lose data or leave two VMs on one disk, and those are not recoverable by retrying.

A bounded wait can also fire on a slow but perfectly healthy machine, which is why the bounds most likely to be wrong for your host are parameters rather than constants. stop_vm takes a timeout, guest_exec takes a timeout, and sandbox_vm takes wait_agent_s. If you are running a guest under TCG emulation on a busy laptop, raise them rather than fighting the defaults.

The errors are written to be acted on, which is worth knowing if you are reading them as a model: they usually name the next tool to call, and when the safe move is to do nothing they say that explicitly. Full parameter details for every tool mentioned here are in the Tool reference.