Blog
Practical writing on production RAG, queue design, idempotency, event-driven consistency, Kubernetes cost, integrations, observability, and developer productivity.
Outages, price changes, and models that shift under a stable name are operational facts. What a routing layer has to do — and the ways a naive fallback makes reliability worse rather than better.
Long-lived connections break the assumptions a request/response stack is built on. The four things that actually matter: connection budget, deploys, presence semantics, and the fan-out that quietly melts your broker.
Blocked loops, connection pools that multiply by replica count, backpressure, cold starts, and graceful shutdown — what breaks async Python services under real load.
Regulation is arriving as an engineering requirement. Classifying features by risk, the artifacts an auditor asks for, and building governance that does not stall delivery.
Both remove the separate API, and they fail differently under latency, complex interaction, and team composition. An honest comparison from shipping both.
Open table formats plus an embedded engine give you snapshots, schema evolution and ACID over object storage — queryable from a laptop, a container, or a scheduled job.
Agents are good at the mechanical sweep and unreliable at judgement. Where to place them in the review pipeline, keeping signal high, and the trap of reviewing generated code casually.
Choosing a shard key you can live with, consistent hashing versus explicit ranges, and the online resharding procedure that does not require a maintenance window.
One slow dependency should not take down the system. Deadline propagation, ratio-based retry budgets, load shedding, and hedged requests.
Most design docs are written after the decision and read by nobody. Writing for the reviewer you need, the sections that earn their place, and when not to write one at all.
Residency requirements arrive as a sales blocker and become an architecture. Regional partitioning, control plane separation, and the metadata that quietly crosses borders.
Decisions get made in meetings and lost in six months. Recording the reasoning, and encoding the constraints as automated checks so the system stays the shape you agreed.
The prompt is a data flow your privacy review has never seen. Redaction, tenant isolation in retrieval, erasure of embeddings, and why your traces are the real leak.
Uploads through your application server is the first mistake. Presigned uploads, validation you cannot skip, derivative generation, and serving without paying for it twice.
Most memory layers are a vector store of everything that ever happened. Write policies, structured facts over raw transcripts, forgetting, and how memory fails in multi-tenant systems.
Why enterprise agents fail in production and the four controls that fix it — tool contracts, hard budgets, least-privilege blast radius, and trajectory evals.
Loops, types and tests in infrastructure — and the failure modes of treating the cloud like an application. An honest comparison against declarative HCL.
Laravel makes everything reachable from everywhere, which is why boundaries erode. Module structure, enforced dependencies, in-process events, and extraction when it is time.
The FDE role is having a moment, and most descriptions of it are wrong. Field notes on living in the customer's environment, closing the feedback loop, and building the product's argument in code.
Deep work is not a personality trait, it is a schedule property. Batching interrupts, an explicit on-call rotation for questions, and asynchronous defaults that actually hold.
Wall clocks disagree across machines, which quietly corrupts ordering, deduplication and debugging. Logical clocks, hybrid timestamps, and where each one belongs.
Debt is not messy code, it is a deliberate trade with interest. Making the interest visible, choosing what to pay down, and getting the work funded without a rewrite pitch.
Scanning tells you what might be vulnerable; runtime tells you what is happening. Kernel-level visibility, detections worth writing, and keeping the noise survivable.
Clients should not make eleven calls to render a screen, and your gateway should not become a second application. Where aggregation belongs and what to keep out of it.
Offline eval suites go stale, and vibes do not scale. Building judges you can trust, calibrating them against humans, and monitoring quality on live traffic.
Your webhook is only as reliable as somebody else's endpoint. Signing, retries with backoff, ordering, replay, and the dashboard that stops the support tickets.
Model calls are slow, fail differently, and cost money per request. Queues, streaming, timeouts, cost control, and keeping AI features out of the request path.
Columnar memory and a query optimiser change what a single node can do. Where Polars replaces a Spark cluster, where it does not, and the streaming engine that made it viable.
A year of agentic coding: MCP servers as the integration layer, where agents genuinely accelerate senior engineers, and where they quietly fail.
What consensus actually buys you, why leader election via a lock is usually wrong, fencing tokens, and the operational realities of running a quorum system.
Reading a repository top to bottom is the slowest way in. Tracing a real request, following the data, and using the tests and history as documentation.
WebAuthn ends credential phishing by design. Account recovery, device loss, the synced-versus-device-bound decision, and a rollout that does not strand users.
Long-lived branches defer integration pain and compound it. Short-lived branches, flags for incomplete work, and deployment pipelines that make frequent releases boring.
The same word means different things to different departments, and forcing one model on all of them is how schemas become unusable. Finding boundaries and translating between them.
The decision is not about capability, it is about which failure you have. Knowledge gaps, behaviour gaps, and the LoRA economics that decide whether training is worth it.
Relevance is a product decision expressed as ranking. Analysis and tokenisation, ranking signals beyond text match, faceting, and measuring whether search is working.
Self-hosting inference is an economics decision. Continuous batching, KV cache memory math, GPU autoscaling, and the utilisation number that decides whether it makes sense.
Role checks scattered through controllers become unmaintainable at the third customer. Policies, ability-based permissions, tenant scoping, and testing the refusals.
Active-active is not a deployment topology, it is a consistency decision. Data partitioning by region, conflict handling, and why most teams want active-passive instead.
The five-minute task done twice a week costs a week a year. Picking what to automate, keeping the automation from becoming its own maintenance burden, and making it discoverable.
Harvest-now-decrypt-later makes this a today problem for long-lived secrets. Hybrid key exchange, the signature migration that is genuinely hard, and an inventory-first plan.
The pyramid is a heuristic, not a law. Choosing test types by failure mode, contract tests between services, and why most integration failures are shape mismatches.
Constrained decoding makes schema-valid output a guarantee rather than a hope. Schema design that improves accuracy, the fields that quietly hurt, and validation you still need.
One model serving both a complex domain and a fast query surface satisfies neither. Where the split earns its complexity, and the eventual consistency you must design around.
Async work breaks most tracing setups. Context propagation through message headers, span links, and the queue metrics that actually matter.
The decisions that are cheap on day one and expensive in year three — cursor pagination, additive change, machine-readable errors, and partial responses.
Fire-and-forget tasks are how async services leak. TaskGroups, cancellation scopes, timeouts that actually propagate, and shutdown that finishes what it started.
Tests coupled to implementation break on every change and get deleted. Testing at the right seam, fast database strategies, and architecture tests that enforce structure.
CDC turns the write-ahead log into an event stream. Schema evolution, snapshot and backfill, ordering guarantees, and the difference between a CDC event and a domain event.
A monorepo without a task graph is a slow polyrepo in one directory. Affected-project detection, content-addressed caching, and where the model breaks down.
Encryption in use, attestation as the actual control, and an honest read on when a trusted execution environment unblocks a deal versus when it is expensive reassurance.
The technical fix is rarely the hard part. Declaring early, separating command from investigation, and writing a review that produces action rather than blame.
Hexagonal architecture is about protecting the part of the code that encodes business rules. Where the boundary pays, where the indirection is waste, and how to keep it honest.
Extended reasoning trades latency and tokens for accuracy on a narrow class of problems. Where thinking budgets earn their cost, where they are pure waste, and how to route.
Shipping code and releasing behaviour are different events. Flag types with different lifecycles, safe rollout mechanics, and the flag debt that quietly accumulates.
Removing the GIL does not make your web service faster. Where free-threaded builds genuinely help, the ecosystem and memory trade-offs, and how to decide whether to care yet.
N+1 is the famous one and rarely the worst. Chunking, lazy collections, selective loading, and the indexes that decide whether your ORM is a problem.
Conflict-free replicated data types make offline editing and real-time collaboration converge without locks — and move the hard problems to intent, storage growth, and permissions.
A hub architecture for Microsoft Graph, Stripe, and SendGrid: anti-corruption layers, canonical events, and one place where retries live.
An agent's output quality tracks the context you give it. What belongs in a repository instruction file, what to leave out, and why most of them are too long to work.
Generating an SBOM nobody reads is compliance cosplay. Build provenance, signature verification at admission, and pinning that actually stops a compromised dependency.
Most review comments are things a machine should have caught. Automating the mechanical layer, sizing changes so review is possible, and reviewing what actually breaks systems.
A million-token window is not permission to use it. Prompt caching, structural prompt ordering, compaction strategies, and why retrieval still beats stuffing.
Event choreography scales teams and hides the workflow. Orchestration makes the process explicit and centralises coupling. When each one is the right trade.
Fan-out, preferences, deduplication, digests and delivery guarantees — the parts of notifications that are genuinely hard, and the ones that only look easy.
CPU is the wrong signal for an I/O-bound worker. Scaling on queue depth and lag with KEDA, scale-to-zero, and the stampede patterns to avoid.
Shared schema, schema-per-tenant, or database-per-tenant — the decision drives migrations, backups, onboarding cost, and the leak you have to design against.
Exactly-once processing is real, narrowly scoped, and easy to misuse. Partition key design, transactional writes, and the rebalance behaviour that quietly breaks ordering.
A 40-minute pipeline changes engineer behaviour in ways no policy can undo. Test selection, caching, parallelism, and deciding what actually has to run before merge.
Static service credentials are the credential that never rotates and never expires. SPIFFE, mTLS, and short-lived identity built into the platform instead of the app.
The deploy is not atomic and neither is your schema change. Backward-compatible steps, locking behaviour that surprises people, and backfills that do not saturate the database.
Rewrites fail because the old system keeps moving. Routing at the edge, seams, dual-running for verification, and choosing which slice to take first.
Most multi-agent systems are one agent with extra latency. The patterns that earn their complexity — supervisor handoffs, parallel fan-out, and explicit state graphs.
A cache is a correctness decision disguised as a performance one. Cache-aside versus write-through, stampede protection, negative caching, and staleness you choose on purpose.
Right-sizing requests, spot node pools with honest disruption budgets, and autoscaling that follows real load — a field report.
Resolution in milliseconds, a real lockfile, and one tool for environments, dependencies and Python versions — plus what to fix in CI and Docker to actually feel it.
Arrays of strings are where bugs live. Backed enums, value objects, readonly classes and casts that push validation to the boundary and keep it there.
Temporal-style durable execution replaces hand-rolled sagas and state machines. What it actually guarantees, where determinism bites, and when a queue is still the better answer.
Onboarding that takes two days is a build system problem. Containerised and declarative environments, what each solves, and keeping the fast inner loop fast.
The attack does not come through the chat box. It arrives inside a document, a ticket, or a web page your agent was told to read — and the defense is architectural.
Most alerting pages on causes nobody is affected by. Picking indicators users feel, burn-rate alerts that fire at the right speed, and using the budget as a decision tool.
Effective debugging is not intuition, it is a search strategy. Forming falsifiable hypotheses, halving the space, and the reasons hard bugs stay hidden.
Model Context Protocol turns your product into something agents can use. Tool granularity, auth, resource design, and the review questions security will ask.
Service boundaries buy independent deployment and cost you distributed transactions. A decision framework based on team topology, failure domains, and what you can still refactor.
An internal platform succeeds when the supported way is the easiest way. Treating it as a product, what belongs in the paved road, and the escape hatch that keeps it honest.
Fixed windows let double the traffic through at the boundary. Token buckets, sliding windows, distributed counters, and limiting the right dimension.
An append-only history that answers who changed what, when, and why — without becoming the largest table in your database or a compliance liability of its own.
Persistent workers remove bootstrap from every request and remove the safety net with it. Where the speed comes from, the state leaks to expect, and what to check before switching.
Dual writes are the quiet killer of event-driven systems. The transactional outbox fixes them with one table and a relay.
Exactly-once delivery doesn't exist. Exactly-once effects can — with idempotency keys, atomic claims, and stored responses.
Queue segregation, backpressure, tuned Horizon supervisors, and the failure modes that only show up under real throughput.
Chunking strategy, hybrid retrieval, reranking, and evals — the four decisions that separate a RAG demo from a system enterprise users trust.