Field engineers upload twenty photographs per inspection from a phone on a marginal connection. Routing those through the application server means each upload occupies a worker for the duration of a slow mobile transfer, the request times out around the eighth photo, and the retry starts again from the first.
The structural fix is that large files should never touch your application servers. The client uploads directly to object storage with a short-lived presigned URL, and your backend handles authorisation and bookkeeping — which is the part it is good at.
Presigned uploads, with the constraints baked in
@app.post('/uploads')
def create_upload(req: UploadRequest, user=Depends(current_user)):
authorize(user, 'inspection.attach', req.inspection_id)
key = f'tenant/{user.tenant_id}/inspections/{req.inspection_id}/{uuid4().hex}'
# Constraints are enforced by the storage service at upload time, not by
# trusting the client to respect them.
post = s3.generate_presigned_post(
Bucket=UPLOAD_BUCKET, Key=key,
Fields={'Content-Type': req.content_type},
Conditions=[
['content-length-range', 1, 25 * 1024 * 1024], # hard size limit
['starts-with', '$Content-Type', 'image/'],
{'x-amz-server-side-encryption': 'AES256'},
],
ExpiresIn=900,
)
Attachment.create(key=key, status='pending', inspection_id=req.inspection_id)
return post
# The upload landing in the bucket triggers processing. Never trust the client
# to tell you it finished — it will not, on a mobile connection.
@on_object_created(UPLOAD_BUCKET)
def process(event):
verify_magic_bytes(event.key) # the extension and Content-Type both lie
scan_for_malware(event.key)
strip_exif_location(event.key) # GPS in a field photo is personal data
generate_derivatives(event.key, sizes=['thumb', 'mobile', 'full'])
promote_to_serving_bucket(event.key)Two details there are load-bearing. Uploads land in a quarantine bucket and are only promoted to the serving bucket after validation, so an unscanned file is never reachable. And content type is verified from the actual bytes — a file named photo.jpg with a Content-Type of image/jpeg can be anything at all, and treating either as evidence is how stored-XSS and malware distribution happen.
Processing and delivery
- Generate derivatives asynchronously and make the UI tolerate their absence for a few seconds; blocking the upload response on transcoding is a timeout waiting to happen.
- Strip EXIF metadata — location, device, timestamps — unless you have a specific reason to keep it and have told the user.
- Serve through a CDN with signed URLs for private media. Short expiry, and never a public bucket you intended to keep private.
- Deduplicate by content hash. In field applications the same photograph is frequently uploaded several times by a retrying client.
- Set lifecycle rules from the start: originals to cold storage after ninety days, derivatives regenerable on demand. Storage is the cost that grows quietly forever.
Any file a user can upload and another user can fetch is a code execution question until proven otherwise.
The failure that caught me out once was resumability. Twenty photographs over a rural connection means partial uploads are routine, and a design that restarts from zero on failure is effectively unusable in the field. Multipart uploads with client-side resume turned a feature people complained about into one they stopped mentioning — which, for infrastructure of this kind, is the target.