4/16/26 Service Disruption: Postmortem

NOTE: This is unrelated to Bluesky’s appview outage on 4/16/26

On 4/16/26, we experienced a server outage in the SE-STO-1 region, which took down our PDS and our moderation service for 15 minutes.

BlueAT Network status

Welcome to BlueAT Network status page for real-time and historical data on system performance.

On April 16 at around 12:30 EST, we performed server maintenance to free up some disk space.

We deleted our /pds/pds.log to free up 3 GB of space and removed some old Docker containers, which freed about 5 GB of disk space.

Before we perform server maintenance we always do backups.

During maintenance, we accidentally deleted our caddy configuration file while clearing Docker containers, causing our PDS and Moderation services to be inaccessible to the public.

Luckily, we have a public copy of our Caddy file; you can find it here. We recreated our Caddy file, pasted in the configuration, and did a quick server restart to apply the changes.

From now on, when we perform server maintenance, we will always double-check what commands do before running them.