Business Problem
Every sizable customer wanted slightly different process automation: notify the owner when an inspection is overdue, escalate maintenance jobs stuck in quote for 5 days, send a review request 3 days after move-in. Engineering was drowning in these as bespoke feature requests, with a backlog of 200+ automation asks and a six-month wait that was cited in churn interviews.
The bet was a no-code workflow builder: triggers from platform events, conditional logic, delays, and actions like emails, SMS, task creation, and webhooks, composable by an operations manager rather than an engineer. The engineering challenge was execution semantics: workflows with multi-day delays had to survive deploys and infrastructure changes, actions had to not double-fire, and a customer building an accidental infinite loop could not take down the platform.
Architecture
I designed the engine around durable workflow runs stored in MySQL, with Redis queues driving execution. Each run was a state machine: it advanced step by step, persisting state before every side effect, so a worker crash or deploy mid-run resumed exactly where it left off. Delays were not sleeping processes but persisted wake-up times scanned by a scheduler, which is what let runs span weeks across hundreds of deploys.
Triggers came from the platform's existing domain events. A trigger-matching service evaluated event streams against active workflow subscriptions, tens of thousands of workflows across tenants, using an inverted index on event type and tenant so matching stayed O(relevant workflows) rather than O(all workflows). Matched events spawned runs onto per-tenant fair-share queues.
Actions executed through an adapter layer with per-action idempotency keys, timeout budgets, and retry policies. Every external effect (email via SendGrid, SMS, webhooks to customer systems) recorded an outcome, and the run timeline UI showed users exactly what happened at each step, which turned out to be as important as the execution engine itself for trust and self-service debugging.
System Design
State-persisted-before-side-effect execution, making every run resumable across crashes and deploys with at-least-once step processing plus idempotent actions.
Persisted wake-up scheduling for delays, supporting multi-week waits with no live process and a scan cost independent of how many runs are sleeping.
Inverted-index trigger matching keeping event evaluation fast as workflow count grew past 30k active definitions.
Loop and runaway protection: per-workflow run-rate budgets, cycle detection on workflow graphs at save time, and a circuit breaker that pauses a workflow after anomalous firing with owner notification.
Per-tenant fair-share execution queues so one tenant's 100k-run burst never delayed another tenant's time-sensitive automation.
Versioned workflow definitions: in-flight runs complete on the version they started with, edits apply to new runs only, eliminating a whole category of mid-run mutation bugs.
My Responsibilities
Led the four-engineer team; owned the execution engine design and wrote its core state machine and scheduler myself.
Designed the workflow definition model and versioning semantics with the product team.
Built the trigger-matching service and its inverted-index subscription store.
Defined the action adapter contract and reviewed all 40+ adapter implementations for idempotency and timeout compliance.
Designed the abuse and runaway protections after red-teaming the builder with deliberately hostile workflow constructions.
Established the load-testing regime that replayed production event traffic at 10x against a staging tenant set before each release.
Implementation
We built the engine before the builder UI, driving it with hand-written JSON definitions for six internal use cases. Proving execution semantics, resumability, idempotency, delay correctness, against real usage first meant the UI team built on solid ground and the engine API barely changed afterwards.
The action adapter contract was deliberately strict: declared timeout, declared idempotency mechanism, structured outcome reporting, no exceptions crossing the boundary. Strictness paid compound interest, 40+ adapters written by multiple teams over two years and none of them ever destabilized the engine.
Launch was gated behind a template library rather than a blank canvas. Twenty pre-built workflows covering the most-requested automations gave customers working starting points, drove 3x higher activation than blank-canvas beta cohorts, and taught users the mental model before they built from scratch.
Results
2M+ workflow runs per month within a year of launch, across 30k+ active workflow definitions.
Automation feature-request backlog (200+ items) effectively dissolved; 85% of the requests were satisfiable by customers self-serving.
Trigger-to-first-action p95 of 4.2 seconds for immediate workflows; delay wake-up accuracy within 30 seconds of scheduled time.
Zero duplicate external side effects measured over 18 months against provider delivery logs.
Customers citing automation in retention interviews rose to 40%; the feature anchored the top pricing tier.
Support tickets for the automated process categories fell 55% as customers debugged their own workflows via the run timeline.
Lessons Learned
Build and prove the execution engine against real use cases before the visual builder; semantics are harder to change than UI.
Idempotency belongs in the platform contract, not in each integration author's memory.
Users trust automation they can inspect; the run timeline was as valuable as the engine underneath it.
Templates beat blank canvases for activation in any builder product.
Design for the hostile customer configuration on day one; the circuit breaker was the feature that let us sleep.
Future Improvements
Add an approval-step primitive so workflows can pause for human sign-off, the top post-launch request.
Workflow analytics showing conversion through multi-step sequences, not just per-run outcomes.
An LLM-assisted builder that drafts a workflow from a plain-language description of the desired process.
Cross-tenant workflow sharing with a curated community template marketplace.
Challenges
Exactly-once effects on at-least-once infrastructure
Queues deliver at least once, and customers will not accept a tenant getting two rent-reminder SMS messages. Every action executes with a deterministic idempotency key derived from run ID and step ID, checked at the adapter layer before any external call. Duplicate side effects dropped to zero measured against SendGrid and SMS provider delivery logs.
Customer-built infinite loops
A workflow that fires on record update and then updates that record is one checkbox away from a self-sustaining storm. Save-time cycle detection catches direct loops; the runtime rate circuit breaker catches indirect ones spanning multiple workflows. In the first year the breaker fired 60+ times, each one a platform incident that did not happen.
Long-running runs versus schema evolution
Runs sleeping for 30 days would wake into a world where the workflow, or the engine, had changed. Definition versioning solved the former. For the latter, engine state was serialized in a versioned envelope with explicit migration functions, tested by replaying a corpus of captured in-flight run states against every engine change in CI.