Seeing and driving a VM

Most tooling that automates a computer needs the computer’s cooperation. SSH needs a running sshd, and a network. A guest agent needs a package installed and a service started. Configuration management needs an OS that already booted. Every one of those requires that somebody already got the machine into a working state, which is precisely the part that is tedious to do by hand.

mcqemu can see and drive a VM without any of it. Screenshots come out of QEMU’s framebuffer and keystrokes go into QEMU’s emulated keyboard controller, so both work at a BIOS setup screen, a GRUB menu, a partitioner, a login prompt, or a kernel panic. That is the capability the display tools exist for.

Screenshots come from the framebuffer, not the guest

vm_screenshot issues QMP’s screendump, which asks QEMU to render whatever its emulated display device currently holds and write it to a file on the host as a PNG. The server reads that file, returns the image, and deletes it. The guest is not asked, not interrupted, and does not need to be capable of anything.

Each call writes to a filename unique to that call. A shared name would let a concurrent screenshot swap the frame between capture and use (which matters for vm_click, described below), and would leave a picture of the guest’s screen sitting on disk afterwards.

Two things follow from “this is the framebuffer, not a rendering of guest state”. A blank image is a real answer, not a failure: consoles blank, guests switch to a mode with nothing drawn yet, and a VM sitting at a black screen may be fine. And the resolution is whatever the guest last programmed, so it changes when an installer switches from text mode to a graphical mode, or when a display manager starts. Coordinates from an old screenshot do not survive a mode change.

For text-mode guests there is a cheaper channel. If the guest writes to its serial port (a kernel booted with console=ttyS0, or a text installer), vm_serial_read returns the tail of the serial log as text, which is far easier to read than pixels and keeps scrollback that a screenshot cannot. It returns nothing useful for a guest that only draws to the screen.

Keyboard input is synthesized at the scancode level

vm_send_keys uses QMP’s send-key, which injects key events into the emulated keyboard controller as QEMU’s qcode key identifiers. The guest sees exactly what it would see from a physical keyboard on the emulated hardware. Nothing above the hardware layer is involved, which is why this works before an OS exists.

Each entry in the keys list is one press. A single key is "ret", "esc", "f2", "a"; a chord is written with hyphens ("ctrl-alt-f2", "ctrl-c") and its keys are held together. Friendly aliases are accepted, so "enter", "space" and "escape" resolve to ret, spc and esc. Between entries there is a configurable delay, and each press has a hold time, because firmware menus and some bootloaders sample the keyboard on a timer and simply miss a press that is too brief.

vm_type_text is the same mechanism with a translation layer: each character is mapped through a US layout to a qcode plus an optional shift, and sent as its own press. This has consequences you should expect rather than discover. The layout is fixed, so a guest configured for a non-US keyboard will produce different characters than you typed, most visibly for symbols and for anything involving AltGr. And every character is a separate round trip over the monitor, so typing is slow and is capped at 4096 characters. Writing a config file by typing it into an editor works and is sometimes the only option; when the guest agent is available, guest_file_write is the right tool for anything longer than a command line.

Pointer input has two modes, and they behave differently

Absolute (the default). VMs launched by mcqemu get a virtio-tablet-pci device, which reports positions rather than movements, exactly like a touchscreen or a drawing tablet. vm_click takes pixel coordinates that match what vm_screenshot showed you. It takes a screenshot first, both to prove a display exists and to read the resolution, checks that your coordinates are inside it, scales them into QEMU’s absolute coordinate space (0 to 32767 on each axis), then sends the position and the button events. There is no cursor to chase: the pointer arrives where you said.

The catch is that the guest needs a driver for that device. Modern Linux, recent Windows with virtio drivers, and many live images have one. A guest from before about 2010, or a minimal install without the driver, does not, and vm_click will appear to do nothing at all. That silent non-effect is the symptom to recognise.

Relative (the fallback). vm_mouse_move drives the emulated PS/2 mouse through the human monitor, which every guest with any mouse support understands. The trade is that you are no longer specifying a position, you are specifying movement, and the guest decides what to do with it. Pointer acceleration means a 50 pixel delta may move the cursor 50 pixels, or 90, depending on the guest’s settings. Large deltas also desync some guests outright, which is why motion is sent in small packets (32 pixels by default, never more than 120, because PS/2 deltas are small signed values).

Because you cannot address a position directly, the reliable technique is to manufacture one. Pass home="bottom-right" (or any corner) and mcqemu pushes the cursor hard against that corner, overshooting deliberately, which pins it to a known location regardless of where it was. From there you move toward the target in small steps, take a screenshot to see where the cursor actually landed, correct with further small moves, and only then click. The tool’s own response says as much, because getting this wrong is the default outcome and a click in the wrong place in an installer is expensive.

The look-act-look loop

The single most useful habit when driving a VM is: take a screenshot, perform one action, take another screenshot. It feels wasteful and it is not.

Blind sequences of keystrokes go wrong for reasons that have nothing to do with the sequence being incorrect. Guests take unpredictable time to react, and a VM under TCG emulation can be an order of magnitude slower than the same guest with KVM, so timing tuned once does not transfer. Input sent while the guest is probing devices or switching video modes is dropped on the floor with no error anywhere; the keystroke simply never happened. Installers reorder or insert screens depending on what hardware they find, what mirror they reach, and whether a disk already has a partition table. Focus moves on its own when a dialog appears. And the failure mode compounds: once one keystroke lands on the wrong screen, every subsequent keystroke in the sequence is being interpreted by a program you did not intend, and the machine can end up in a state that is worse than where it started (a partitioner that has silently selected the wrong disk is the memorable example).

None of this is detectable from the return value of a key-send. The keystroke was delivered to the emulated keyboard successfully; that is all the success of that call means. The only evidence about what actually happened is the next screenshot.

So the loop is: look at where you are, decide one action, take it, look again to confirm it did what you expected. When a step is slow (a package installation, a reboot, a filesystem being written) look repeatedly rather than sleeping for a guess. When something unexpected appears, you find out one action after it happened instead of twenty. Drive an installer applies this to a concrete case, and your first sandbox is a gentler place to get a feel for it.

What this does not give you

The model is reading pixels. There is no accessibility tree, no DOM, no list of widgets, and no text extraction: a button is a shape with letters drawn on it, so low-contrast themes, unusual fonts, small text and scaled displays all degrade the read. Coordinates are display-space and go stale on a mode change. There is no way to ask what has focus, so a screenshot has to be interpreted for it.

The compensation is that all of this works on absolutely any guest, including one whose OS does not exist yet. Once a guest is booted and has the agent installed, guest_exec is faster, more precise and much easier to check for success, and it is the right tool for anything reachable that way. Screenshots and scancodes are for getting to that point, and for the times when the guest cannot help you.