One floor, three layers, nothing taking a step unless it is told it may
Our warehouse robot platform now runs end to end on a simulated site: a dashboard for operators, an edge server that drives the floor, and the devices themselves. This post shows what it does, how it keeps people and machines safe, and what we have run to prove it.
Three layers, one shared language
The platform controls a fleet of forklifts, robot arms, conveyors and cameras in a warehouse. It is built as three layers that talk only through defined interfaces, so each can be replaced or upgraded on its own.
Dashboard and cloud service
What operators, supervisors and maintenance staff see. A live floor map, alarms, commands, history and analytics across several sites, signed in through an identity provider with seven roles.
The edge: the robot server
Runs on site. It admits devices, keeps the live picture of the floor, publishes the permit to operate, routes the forklifts without collisions, turns what the cameras see into pallet jobs, and survives the loss of a node or of the WAN.
The devices and the world they stand in
An agent on every machine and a physical simulation around them: geometry, batteries, radio dead zones, people walking the aisles, MuJoCo arm physics and rendered camera pictures. Robots, shapes and instructions are data in YAML packs, not code.
What happens when an operator pauses the fleet
One action touches every layer. This is the path a fleet pause takes, and the one we time.
Operator presses Pause
Sent as the signed-in person, with an idempotency key so a retry never acts twice. Resuming asks for a step-up sign-in.
sent once, as a named personCloud records it
The request is written to the audit trail in Postgres with who, what and when, then sent to the site.
audit + lifecycleEdge takes it north to south
The command service passes it to the safety supervisor. The stop goes out before the store is waited for, so a slow disk never delays it.
no wait on storagePermit withdrawn
The supervisor marks every device held in the next permit. Vehicles on VDA 5050 get a pause order, one by one.
permit every 500 msDevices hold
Every device is in HOLD and the dashboard shows who is held and why. Resuming the fleet needs a second person.
measured ~100 ms · limit 1 s
What each part can do today
Dashboard
- Live floor map at 10 Hz, with staleness and device flags: held for, signal quality, presence.
- Alarm console in the ISA-18.2 style: first-out, suppressed counts, acknowledge by a person. 300 alarms in 60 s block the browser for less than 50 ms at a time.
- Commands to one device or the fleet, with every terminal state shown: done, rejected, expired, failed, cancelled.
- Multi-site with each site behind its own WAN. A 10-minute outage replays with zero duplicate, out-of-order or missing rows.
- History and analytics over 30 days: OEE, MTBF, MTTR, throughput, daily CSV report, 2-hour incident replay, maintenance tickets raised from motor and battery alarms.
- Configuration, maps and OTA rolled out ring by ring and halted by a rollback.
- Digital twin in Unity, live video over WebRTC, and an interface in English and Japanese.
Edge
- Five protocols admitted at one door, with devices enrolled over EST and let in over mTLS.
- Safety supervisor that publishes the permit to operate and applies nine supervisory rules.
- Traffic control by reservation: over 8 simulated hours and 6,826 journeys, no two forklifts ever overlapped.
- Missions: pallets seen by the cameras become jobs, forklifts are matched to stations and sent to charge when idle. 96 to 103% of what the stations could take.
- Raw camera vision finds pallets, unknown slots and askew pallets. 240 of 240 empty slots and 120 of 120 pallet types read right on test scenes.
- High availability: three nodes, a leader by lease. When the leader is killed, another serves after about 5 s.
- Edge to cloud over a leaf node, safety records first, each once and in order.
Devices and world
- Device agent speaking NRP, VDA 5050 or JSON over MQTT, with a write-ahead log so safety events survive a power cut.
- A world with physics: bodies, forks, belts, batteries, a hall's radio with roaming gaps and dead zones, and people.
- MuJoCo arm cell, checked against an independent arm model.
- Perception: labelled picture sets from a seed, byte for byte repeatable. Pose within 3 cm and 2 degrees.
- Hardware in the loop: the same device agent that ships on a machine runs against a simulated I/O bench.
- Definition packs: add a robot, a shape or an instruction in YAML. A floor scrubber was added this way.
- 3D view of the floor in Foxglove or a browser, at 10 Hz.
The running system, captured
Screenshots taken on 18 October 2026 with all three layers running together: the simulated floor, the edge, and the cloud and dashboard. We struck the faults on purpose and cleared them afterwards. The yellow labels are ours; everything else is what the screens showed.
3D viewer
The simulated world drawn in the browser, and streamed to Foxglove or Lichtblick as well. Here a person has stepped into arm-01's cell (F24) and agf-03's E-stop is pressed (F01). The panel on the left lists what is struck, what is held and what is offline.
- Live link to the world, 10 updates a second
- Faults struck now: F24 person in the cell, F01 E-stop
- Held by its own agent: arm-01 waits for a reset
- arm-01 red: protective stop, person in cell
- agf-03 in red: E-stop pressed
- Radio dead zone (pink) behind a 6 m rack
- Pallets in inbound slots
Physical simulation
What camera cam-01 sees, rendered by the world from a seed: 6 m up, 9.4 mm of floor per pixel, with sunlight, shadows, lens blur and sensor noise. Each picture comes with its depth, a segmentation and instance masks, and the true pose of every pallet. The edge's pallet detector is held to that truth, within 3 cm and 2 degrees.
- Pallet 0.40 m off its slot, turned 10.3°
- Shrink-wrapped pallet, square in slot-a2
- Painted slot outline
- A forklift (AGF-1600) driving into view
- Shadows from a sun at 268°, 449 lux
Depth
Segmentation
InstancesDevice management
Every device on the site, live, in one list. Each class reports what matters for it, and a class the dashboard has never seen is shown from the factsheet the device sends. The registry is the edge's own record of what it has in service.
- LIVE: the site streams; LOST greys every row
- Filter by class
- Forklift: battery, load, job, motor
- Arm: boxes left, cycles, picks
- Camera: slots occupied
- Conveyor PLC, over OPC UA
- Unknown class, from factsheet
- Age, judged by each device's period
- Served by the edge: revision 2
- Model and serial
- Older firmware
- Link, as the edge sees it
Live floor map
The floor from the site package, with every vehicle moving as the site reports it. Arms, conveyors and cameras stand where the package puts them, filled while they work.
- Forklift, pointing where it heads
- Arm: filled while it works
- Conveyor: filled while it runs
- Camera, between the two slots it watches
- Inbound slot
- No Wi-Fi coverage
- Charger
Fault tolerance
An E-stop on agf-03 (F01) and a radio dead zone on agf-04 (F05), struck through the edge. agf-04 never looks live while it is silent: it greys out with the age of its last position. The edge declares it LOST and keeps its reserved floor until it reports again (rule R1).
- agf-03 in red: E-stop pressed
- agf-04 in a dead zone: greyed, aged 4 s
- Alarm count
- Arms keep working
- Someone entered arm-01's cell
- The supervisory rule behind it
- Came back twice: counted once
- The station goes down with its arm
- Device back: the alarm waits for a person
- Acknowledged by a person, under their name
Devices must be told they may move
Most systems stop a robot when something goes wrong. This one inverts it. A machine starts new motion only while it holds a fresh permit, so silence from the server is itself a stop.
The permit to operate
The edge sends a permit every 500 ms. A permit is valid for one second on the device's own clock, and it names any device that is held. If permits stop, because the edge crashed, the broker went down or the network was cut, every supervised device holds within a second of the last one.
A permit from an older epoch is ignored, so a frozen leader that wakes up after a takeover cannot send a machine on its way. The highest epoch seen is kept across a loss of power.
A device goes silent
Alarm raised, nothing is sent to it, and a forklift keeps its reserved floor until it is heard again.
Forklift off its reservation
Over 0.5 m outside what is reserved: it and its neighbours are held until an operator places it, even through a restart.
Station interlock
An arm is held first. The forklift is released into the station only when the arm's own state shows the hold.
Person in a robot cell
A light curtain or laser scanner sees someone: the arm is held within one permit, about 150 ms, and stays held until the guard is reset.
Vehicle lost its position
Localization score under 0.6: paused until it is back at 0.8. An operator can vouch for it for 5 minutes at most.
Clock off by 100 ms
Alarm raised and the device's data is ordered by when it arrived, not by what it claims.
Bus not healthy for 500 ms
The permit is withdrawn and everything that needs one holds.
Traffic controller silent 5 s
Every vehicle it routes is paused, and released only by the supervisor that paused it.
A device reports a safety stop
A P0 alarm stands until the device says it is over. Nothing is dispatched to it meanwhile.
Stored before acknowledged
Safety events and events are written durably before the edge confirms them. A device keeps its own write-ahead log and resends until the edge confirms. No safety event was lost in the storm, failover and outage runs listed below.
Fenced leadership
A three-node edge elects one leader by lease. Two never serve at once. A frozen leader that wakes stops its edge before it sends anything.
People and permission
Seven roles. Step-up sign-in before a resume, a second person before a fleet resume, and every command in an audit trail with an idempotency key.
Trusted devices, signed changes
Devices enrol over EST and connect with mTLS. Configuration bundles are signed with cosign keys (ECDSA P-256), proven on probation and rolled back if a device does not come up healthy.
People and motion
Machines
Pallets and cameras
Network, power and software
Each fault above can be struck on the simulated floor on demand, and a test checks what the edge does against the simulator's ground truth. Two more, a state stall and a failed update, are tested the same way.
Built on public standards, not private formats
Where an industry standard exists we use it, so vendors' vehicles, tools and viewers work without translation.
Each limit we set, and how much of it we use
Every row has a limit written down before the test. The bar shows the worst figure we measured as a share of that limit. All were measured in simulation on a single machine.
Worst measured figure, as a share of its limit
Recovery is measured in seconds, not milliseconds: a reboot storm of 120 devices is fully back in about 3 s (limit 30 s). At 1,000 devices all return 29.3 s after the storm. After the leader is killed, another member serves after about 5 s and every device is back ONLINE 9 to 10 s after the kill.
How many tests, and what they check
The automated tests behind the platform, and what they check.
Tests by area
Between layers
- The interfaces between layers hold on every implementation: the simulator, the edge and the devices, including over TLS.
- Signed configuration bundles verify end to end, from the cloud to the edge.
- Message formats, scenario files and site packages stay valid.
Dashboard and cloud
- Live state reaches the screen in time, and 300 alarms in a minute do not freeze the page.
- Commands reach every terminal state and each one renders. Step-up and second-person rules are enforced.
- Outages: a 10-minute WAN cut replays with no duplicate, out-of-order or missing rows.
- Analytics, roles, audit trail, bundles and maps behave as specified. Every widget is checked in its live, stale, lost, empty and alarm-flood states.
- The Unity twin reads the feed exactly as the Go side writes it.
The edge
- Every fault is struck and the edge's response is measured against the simulator's ground truth.
- Safety: the fleet pause, the permit, the nine supervisory rules, holds that survive a restart or takeover.
- Traffic and missions: no two forklifts overlap, no deadlock lasts a minute, pallets become jobs.
- Availability: leader killed, cut off, wrong clock, full disk. Never two leaders, nothing lost.
- Protocols and vision: five protocols, certificates, and the detector against rendered scenes.
Devices and world
- The device interface on every device agent, however the agents are deployed.
- Physics and invariants checked at every control period of simulated shifts.
- Timing trials that must keep time to 50 ms, and agents killed without warning.
- Perception: the same seed gives the same pictures byte for byte, pose within 3 cm and 2 degrees.
- The arm on MuJoCo agrees with an independent model, and the device agent is exercised against a simulated I/O bench.
| Recorded long run | What it did | Result |
|---|---|---|
| Traffic, 8 simulated hours | 20 forklifts, 6,826 journeys on a larger simulated site | 0 footprint overlaps · longest deadlock 10.9 s |
| A full shift at 20x speed | The edge and the simulated floor as separate systems: 149 pallets arrived, 22,381 m driven, 2,500 routes | 0 safety violations · 0 orders resent |
| 72-hour WAN outage | 438 safety events, 8,760 events and 291,610 telemetry samples held back, then replayed | Each once, in order, safety first · 10 runs of 10 passed |
| Reboot storm, 1,000 devices | Every device power-cycled at once | All back in 29.3 s · no safety event lost |
| Leader killed without warning | Three edge nodes, then the new leader frozen past its lease | Successor in 3.4 to 7.7 s · never two leaders · woken leader stands by |
| Eight-hour mixed-vendor shift | Forklifts of two vendor profiles on one floor | 21.0 pallets/h through the peak · none of 3,169 routes cancelled |
| Eight days of floor time | 192 hours of operation, run at 20x speed | No leak · 4,293 pallets |
| Operator load | 500 dashboard sessions each following every device of three sites | p95 153 ms against a 500 ms limit |
Counts are of tests, not of checks: one test can cover several cases.
What this is not yet
All of the figures above come from simulation. No physical robot has run on the platform yet.
No hardware yet
Devices, people, radio and physics are simulated. A real machine's I/O is exercised only on a simulated bench. On a real floor the cell's own safety circuit acts first; our hold is the second layer.