Business Problem
A facilities maintenance company ran dispatch for 1,200 field technicians on a system where job assignments were pushed by a polling mobile app every 10 minutes. Technicians regularly drove to jobs that had been cancelled or reassigned, and dispatchers worked from a board that was minutes stale. The company estimated 6-8% of technician hours were wasted on stale assignments.
The commercial driver was SLA penalties: their largest contracts carried response-time commitments of 2-4 hours for priority faults, and missed windows triggered fee clawbacks. They needed dispatch decisions to reach technicians in seconds, a live view for dispatchers, and the whole thing had to tolerate technicians going offline in basements and rural areas for an hour at a time.
Architecture
I designed an event-driven core around a Redis Streams event log. Every state change, job created, assigned, accepted, en route, on site, completed, was an immutable event. A Laravel API handled commands and validation, while a Node.js WebSocket layer fanned events out to dispatcher dashboards and technician apps within a couple of seconds of the source change.
The scheduling engine consumed the same event stream to maintain a materialized view of technician availability, skills, and location, and produced ranked assignment suggestions. Dispatchers could accept a suggestion in one click or override it; either path emitted the same assignment event, so downstream consumers never cared how a decision was made.
Mobile offline tolerance came from an append-only sync protocol: the app kept a local event journal, and on reconnect both sides exchanged events since the last acknowledged sequence number. Conflicts, like a technician completing a job that dispatch had reassigned, resolved through explicit domain rules rather than last-write-wins.
System Design
Event log as the source of truth with consumer groups per subsystem, so the dispatch board, scheduling engine, and reporting each replayed independently after a failure.
Sequence-number-based mobile sync that survived hour-long offline gaps without full resyncs, keeping reconnect payloads under 50 KB in the typical case.
Materialized availability view rebuilt from events, making the scheduling engine stateless and trivially restartable.
Domain-rule conflict resolution for offline edits, with a small set of documented outcomes instead of silent data loss.
Geospatial indexing of technician positions in Redis for sub-100 ms nearest-qualified-technician queries.
Dead-letter handling with automated replay tooling, so a bad event never silently stalled a consumer group.
My Responsibilities
Owned backend architecture end to end, from the event schema to the sync protocol specification the mobile team built against.
Built the scheduling engine, including the skills-and-proximity ranking model and the availability materializer.
Wrote the Node.js WebSocket fan-out service and its Redis-backed presence tracking.
Designed the offline sync protocol and led three joint debugging weeks with the mobile team to harden it against real network conditions.
Set up load testing that replayed a full production day at 5x speed before each major release.
Mentored two mid-level engineers who took over the reporting consumers, and ran the on-call rotation for the dispatch core.
Implementation
The event schema came first and was treated as a public contract: versioned, documented, and validated at the producer. That discipline paid off when the reporting team and a third-party integrator both built consumers without ever needing changes to the core.
The scheduling engine started as a straightforward weighted scoring function over skills match, travel distance, current workload, and SLA priority. We resisted the temptation to make it clever early; instead we logged every suggestion alongside the dispatcher's actual choice, and used three months of that data to tune weights. Suggestion acceptance rate went from 55% to 84%.
Rollout was region by region. The first region ran two weeks in shadow mode, with the new system generating assignments that were compared against dispatcher decisions but never sent to devices. That surfaced a timezone bug and a skills-taxonomy mismatch before a single technician was affected.
Results
Dispatch-to-device latency dropped from a 10-minute polling worst case to under 2 seconds via WebSocket push.
Missed SLA appointment windows fell 38% in the first quarter after full rollout, directly reducing contractual penalty payments.
Wasted technician drive time from stale assignments was cut by an estimated 5,000 hours per year.
Scheduling suggestion acceptance reached 84%, letting dispatchers manage 50% more technicians per seat.
The platform processed 18k job events per day with the core services running on a deliberately modest 6-container footprint.
Zero data-loss incidents from offline sync across two years of operation.
Lessons Learned
Offline-first sync is a domain modelling problem wearing a networking costume; enumerate the conflicts with the business before writing resolution code.
Shadow mode is the cheapest insurance available for replacing a human workflow with an automated one.
Logging the human override alongside the machine suggestion is the highest-value training data you can collect.
Per-entity ordering with cross-entity parallelism is almost always the right event-stream partitioning answer.
Future Improvements
Replace the weighted scoring model with a learned ranking model trained on the accumulated suggestion/override dataset.
Add predictive travel-time estimates from a routing API instead of straight-line distance.
Move the event log from Redis Streams to a managed Kafka-compatible service for longer retention and simpler compliance replay.
Expose a public webhook API so client companies can consume job events into their own systems.
Challenges
Offline conflicts with real-world consequences
A technician marking a job complete while dispatch reassigns it offline is not a merge conflict, it is a business dispute. I catalogued the eight realistic conflict pairs with operations staff and encoded an explicit resolution for each, with an audit trail. Disputed-job tickets fell to near zero once outcomes became predictable.
Event ordering across consumers
Early prototypes processed events out of order under load, producing dispatch boards that briefly showed impossible states. Partitioning streams by job ID guaranteed per-job ordering while keeping parallelism across jobs, which fixed the anomalies without serializing the whole system.
WebSocket scale on a modest budget
1,200 concurrent mobile connections plus 80 dispatcher dashboards had to run on a small AWS footprint. Careful payload design (deltas, not snapshots), connection-level backpressure, and horizontal sharding by region kept the fan-out tier on three t3.medium instances.