Adding LLM Features to a Laravel Application Without Breaking It

The first LLM feature in a Laravel application is usually a synchronous call inside a controller. It works in development, and in production it holds a PHP worker for eleven seconds, exhausts the pool under modest load, and takes down endpoints that have nothing to do with the feature.

Model calls break every assumption a typical request path makes: they are slow, they fail in ways retries do not always fix, they cost money proportional to use, and their latency varies by an order of magnitude between identical requests. Treating them as an external integration with its own failure domain — rather than as a function call — is most of the work.

Get it out of the request

Anything that does not need to stream to a waiting user belongs in a queue: summarisation, classification, extraction, enrichment, document processing. The request enqueues a job and returns immediately, the job does the work with a sensible timeout, and the result arrives by broadcast, polling or notification.

class SummariseInspection implements ShouldQueue
{
    public int $tries = 3;
    public int $timeout = 120;
    public array $backoff = [10, 60, 180];

    // Its own connection and queue: a slow provider must not starve the
    // queue that sends password resets.
    public string $queue = 'ai';

    public function middleware(): array
    {
        return [
            // Provider rate limits are a shared resource across all workers.
            new RateLimitedWithRedis('llm-provider'),
            // A model call is not free: never let a retry storm run twice.
            new WithoutOverlapping("summarise:{$this->inspection->id}"),
        ];
    }

    public function handle(LlmClient $llm): void
    {
        $result = $llm->structured(
            prompt: SummaryPrompt::for($this->inspection),
            schema: InspectionSummary::schema(),
            timeout: 60,
        );

        $this->inspection->update(['summary' => $result->summary]);
        AiUsage::record($this->inspection->tenant_id, $result->usage);   // cost, per tenant
        SummaryReady::dispatch($this->inspection);                       // broadcast to the UI
    }

    public function failed(Throwable $e): void
    {
        // Degrade visibly. A silent failure looks like the feature is broken.
        $this->inspection->update(['summary_status' => 'unavailable']);
    }
}

A separate queue connection for AI work is the detail that prevents the worst incident. When a provider slows down, jobs pile up; if they share a queue with transactional work, everything behind them waits, and a degraded AI feature becomes a degraded product.

Streaming, when the user is waiting

For chat-style interactions, streaming is not a nicety — the difference between a token appearing in 400ms and a complete answer arriving in nine seconds is the difference between a feature people use and one they assume is broken. Server-sent events over a streamed response work well, and with a persistent runtime you are not holding a traditional worker for the duration.

• Record token usage per tenant on every call. Without it, the first monthly bill is an unpleasant conversation with no data behind it.
• Set a per-tenant spend cap and enforce it before the call. Rate limits protect the provider; caps protect you.
• Cache aggressively on a hash of the prompt inputs; identical requests are far more common than people expect.
• Store the prompt version alongside every stored output, or you cannot explain why last month's summaries differ from this month's.
• Always design the feature to work, in degraded form, when the provider is down. An unavailable summary is acceptable; an unavailable page is not.

An AI feature that can take down your checkout is not an AI feature. It is a dependency you forgot to isolate.

The architectural rule I apply now is simply that model calls live behind a queue or a stream, never in a synchronous request path that something important depends on. It costs a little product flexibility — some things genuinely want to be immediate — and it means a provider outage degrades one feature rather than the application.