We have to move to object storage to have efficient image storage. We are currently at 70% of our disk storage.
We are sorry for the short notice and we don’t know how long it will take though we believe it will take multiple hours.
Small post mortem :
- The migration required us to be down - we thought we could run without images
- It took much longer than initially anticipated
- It broke at close to 2:30AM - 3 hours into the migration and I couldn’t fix it so the server has been down doing nothing for hours now
- We’re booting back up from the backup made before the migration was attempted, we’ll try another strategy.
There might’ve been some data loss in images, we’re looking into it. In the meanwhile, if your profile picture or banner is broken, feel free to re-upload them.
Update :
Hi Beeple!
We’re trying this again.
This time, Beehaw should remain available though no pictures will be able to be uploaded. The error will likely be weird because Lemmy will think it is possible but we will block the upload from happening.
We’ll take a snapshot of the pictures before the migration in 60 minutes at 21:00 UTC - it should take around one hour, do the migration and testing on our own before shipping on Beehaw.
Once we’ve resolved all these kinks, Beehaw will momentarily go down and then back up with the migration complete without losing any old pictures.
It can be though? Sites & service have been doing this for decades now. My example of e-hentai using distributed workers hosted on users machines (given they pass the networking & storage requirements) to serve up images is one of those.
The problem is the bulk of the work is on Lemmy developers to design such a solution, and then together with the FOSS community, make it accessable.
Media is the low hanging fruit, and has largely been a solved problem for quite some time. And even has semi-functional fediverse solutions. Distributed workers for encoding and compressing media is also a solved problem. And in many cases has been made as easy as downloading an executable or spinning up s docker container.
So, yes, for a set of workloads you don’t have to choose. And haven’t had to for years.
Actual distributed transactional workloads is a whole other beast, which is a problem that needs solving if we ever want to have robust and survivable communities that can deal with scaling issues without risk of dying because of a lack of funds or because someone ran off with the funds.