cross-posted from: https://beehaw.org/post/282116
We’ve posted a number of times about our increasing storage issues. We’re currently at the cusp of using 80% of the 25gb we have available in the current tier for the online service we run this instance on. This has caused some issues with the server crashing in recent days.
We’ve been monitoring and reporting on this progress occasionally, including support requests and comments on the main lemmy instance. Of particular note, it seems that pictures tend to be the culprit when it comes to storage issues.
The last time a discussion around pict-rs came up, the following comment stuck out to me as a potential solution
Storage requirements depend entirely on the amount of images that users upload. In case of slrpnk.net, there are currently 1.6 GB of pictrs data. You can also use s3 storage, or something like sshfs to mount remote storage.
Is there anyone around who is technically proficient enough to help guide us through potential solutions using “something like sshfs” to mount remote storage? As it currently exists, our only feasible option seems to be upgrading from $6/month to $12/month to double our current storage capacity (25GB -> 50 GB) which seems like an undesirable solution.
How would you accomplish this? I have plenty of bandwidth and plenty of storage I can subsection as a possible solution (hell even buying a raspberry pi and an old hard drive wouldn’t be all that expensive and potentially a fun project) but I really don’t even have an idea of how to connect this to the lemmy instance
See the link above. You can configure Pict-rs 4.0 (beta, unreleased) to a S3 compatible storage. The Garage project is a S3 compatible storage especially aimed at distributed self-hosters, but with some latency caveats aside something like Minio would probably also work.
S3 storage allows to redirect users directly to a storage location (with a public IP) instead of the main server loading images from a storage location and serving that to the user itself. Kind of like a CDN works.
Is there any way to do this and avoid having to use S3? I don’t want a surprise bill from Amazon because we exceeded some thresholds they have on the free tier (nor do I want to have to make new free tiers every 12 months).
s3 is only the api. Every S3 compatible storage should work afaik
Okay so I need to be sure I have something that can make sense of s3 calls to storage, I feel like we’re getting closer, just still way out of my own technological depth.
Pict-rs that Lemmy uses as the image storage is able to do S3 compatible storage API calls in the upcoming 4.0 version.
There are also many self-hosted options that you can install on your server to provide a S3 compatible storage API. The probably best known open-source software for that is called Minio. It is however more meant for data-centers with fast low latency connections or local network only.
The above mentioned Garage software is unique in that it is specifically designed to work in less than ideal networking conditions typical to self-hosted servers.
backblaze and wasabi have s3 compatible storage I think