Automating isolated systems over USB

Headless or embedded computers are all around us, from ticketing terminals and point of sale (pos) and self-checkout to industrial or factory machines and vending machines. Maintenance for these machines typically involves a human technician and debugging peripherals - but which can be automated.

Automated Human interface devices (HID)

At its core, a "Human Interface Device" or "HID" refers to any device meant to allow human interaction with a system, like keyboards, pointer devices (mouse, trackball), touch screens etc. But in addition to the interactive HIDs most people know, there are also automated variants that are programmed to input a specific sequence of inputs as soon as they are connected.

Automated devices have many different names depending on vendor and purpose. Names like "maintenance key", "field configuration device", "commissioning tool", "automation dongle" are common around industrial and kiosk automation, reset/recovery-focused ones are often named "reset key", "endpoint recovery token" or "break-glass hid device", and penetration testers refer to them as a "rubber ducky" or "BadUSB".

All of these really just mean a USB thumb drives that acts like a USB keyboard, inputting a preprogrammed sequence of key presses automatically. Especially "rubber ducky" and "BadUSB" have gotten a bad reputation because of its usage by hackers, but there is nothing inherently wrong with the devices themselves, and using them for automation is neither legally questionable nor problematic for internal security if handled with care (treat them like keys). If you need approval from your manager to consider implementing USB automation, make sure to use one of the other names in your presentation to not throw them off.

Automation targets

When automating systems over USB, there are typically two approaches:

1. If the vendor supports it, they may offer automation using simple thumb drives by storing digitally signed executable files on them and plugging them into systems, which execute instructions once verified. We are not talking about this variant in this article.

2. Much less known is a variant where you use an automated HID like a rubber ducky to automate a device that was not meant for automation this way, but allows a technician/maintainer to connect a USB keyboard for manual interaction. The HID effectively pretend to be both the human and the keyboard, preventing typos, human error and removing the need for a certified technician to be present. We are focusing on this type of automation.

Usage for office environments

Not all air-gapped systems are isolated from networks by choice. Consider a company with multiple office locations scattered across countries, with a single centralized it support department. Tech support can mostly be done over remote desktop, but if a workstation experiences connectivity issues, they either need to try and guide an office worker through repairs over the phone, dispatch a technician to repair it on-site, or replace it with a stand-in and ship it back to the it department for repairs.

All options involve significant downtime bundled with either risk or cost. A good middle ground could be to have a preprogrammed USB devices around that either resets the machine to default configuration or runs common network fixes, in an attempt to get the workstation back online for remote desktop support to function.

The device itself is cheap, so keeping one or two per site is reasonable, any office worker can easily plug it in, and if it even works a single time, it will have already paid for itself in saved billable technician hours.

Configuring or updating air-gapped devices

Intentionally isolated devices in high-security environments can automate planned maintenance and software updates through preloaded USB devices, bundled with cryptographically signed software packages and scripts. Using automatic HID devices prevents human error and increases technician speed, especially when managing multiple devices.

Air-gapped kiosk systems on the other hand could save even more money by automating configuration adjustments or firmware updates in a manner that removes the need for an on-site technician.

Suppose a company running multiple hotels with vending machines decides to set new sleep schedules or update prices. They could preload HID drives with all necessary commands and ship them to every hotel, where even technically uneducated staff like receptionists or building management can safely apply the changes on their own.

Resetting authentication

Devices and systems protected by an authentication or encryption layer, like office computers or factory machines, may need a last-resort way to circumvent forgotten credentials and lost encryption keys. These are often called break-glass HID devices, and are often just emulated headless USB keyboards. You can build those yourself if you fear forgetting credentials may be a problem for you.

Such applications of HID automation are a tradeoff between security and operational uptime, so the storage and access to the USB device must be tightly controlled.

An authentication reset device will typically be used to reset credentials or encryption keys to default values, so operators can log in and change them to a secure variant again.

Automating embedded systems

Embedded systems, especially industrial or factory machinery, can be seriously cost-optimized using automated HID USB devices.

For example, staff training can be sped up by inducing specific test conditions or error states reliably within seconds during technician exams, or to quickly run through and test all available functions without pressing buttons manually - even unattended, by going around the entire factory and plugging a USB device into every machine, then looping around to the first and checking if the machine succeeded. A single technician suddenly becomes much more productive without adding subjective workload to their schedule.

On the other hand, human operator time can also be reduced during normal operation. If a machine has a feature to self-clean, that could be executed unattended at a time where workers are not near the danger zone anymore, perhaps an hour after they clock out. An EOL device not supported by its manufacturer may show annoying warnings periodically, which could be automatically dismissed by a scheduled input script as well.

Vendors of new machinery may also find use for them by looping specific function sequences, so the machine can perform them continuously during a presentation or sales meeting. Showing devices in action is likely to impress clients and boost sales figures.

While this category has the most potential to save significantly on human work hours, it is also the most dangerous. Automation like this is blind, so it should never make decisions that could harm humans in any way, and must have physical stopping measures in place (like emergency power off buttons or thermal sensor shutoff)

Important considerations for automated inputs

Getting an automated HID is extremely easy: Just search for "USB rubber ducky" in tech stores and you will quickly find affordable models. Writing the script often just means typing the exact keys into a text file and copying it onto the device, but writing a reliable script involves a few caveats to consider:

1. You need to be more careful about errors. A failed command must prevent other commands from being executed, or you will create unpredictable debugging hellscapes. Check for errors after every command or use global error failure flags like set -euo pipefail for bash-compatible devices.

2. Timing matters. After typing a command, the device may take some time to process it. Depending on the system, scripts may potentially execute some tasks asynchronously, jumping to the next instruction before the previous has completed. Make sure to apply timeouts or sleep windows where necessary, preferably too long than too short. A device's performance can vary based on age, climate or physical wear, so plan some headroom for unexpectedly slow response times.

3. Automated and headless does not mean silent. A human will still be plugging in the device, and needs some feedback to know when to remove it. Ideally two, one for success and one for failure, so they know if they are good to resume their work or need to call a technician. Such signals can be anything, like flashing lights on a vending machine in a specific sequence or playing consecutive pos checkout beeps that humans could easily count.

4. You need bulletproof documentation. If an unqualified operator is supposed to use the device in a self-service manner, they need bulletproof instructions on how to do it properly. Do not assume they know what a USB port looks like, or what a reboot is. Focus on providing unambiguous descriptions, like sending pictures of exact USB port locations and describing events like reboots by visible/audible side effects like cooling fans stopping or a control beep from a starting pos checkout.

After writing your script, hand the drive to someone you know has little to no experience with the devices in question and see if they can successfully complete the task with only your instructions. If they pass, you can start deploying it in earnest.