EDIT
TO EVERYONE ASKING TO OPEN AN ISSUE ON GITHUB, IT HAS BEEN OPEN SINCE JULY 6: https://github.com/LemmyNet/lemmy/issues/3504
June 24 - https://github.com/LemmyNet/lemmy/issues/3236
TO EVERYONE SAYING THAT THIS IS NOT A CONCERN: Everybody has different laws in their countries (in other words, not everyone is American), and whether or not an admin is liable for such content residing in their servers without their knowledge, don’t you think it’s still an issue anyway? Are you not bothered by the fact that somebody could be sharing illegal images from your server without you ever knowing? Is that okay with you? OR are you only saying this because you’re NOT an admin? Different admins have already responded in the comments and have suggested ways to solve the problem because they are genuinely concerned about this problem as much as I am. Thank you to all the hard working admins. I appreciate and love you all.
ORIGINAL POST
You can upload images to a Lemmy instance without anyone knowing that the image is there if the admins are not regularly checking their pictrs database.
To do this, you create a post on any Lemmy instance, upload an image, and never click the “Create” button. The post is never created but the image is uploaded. Because the post isn’t created, nobody knows that the image is uploaded.
You can also go to any post, upload a picture in the comment, copy the URL and never post the comment. You can also upload an image as your avatar or banner and just close the tab. The image will still reside in the server.
You can (possibly) do the same with community icons and banners.
Why does this matter?
Because anyone can upload illegal images without the admin knowing and the admin will be liable for it. With everything that has been going on lately, I wanted to remind all of you about this. Don’t think that disabling cache is enough. Bad actors can secretly stash illegal images on your Lemmy instance if you aren’t checking!
These bad actors can then share these links around and you would never know! They can report it to the FBI and if you haven’t taken it down (because you did not know) for a certain period, say goodbye to your instance and see you in court.
Only your backend admins who have access to the database (or object storage or whatever) can check this, meaning non-backend admins and moderators WILL NOT BE ABLE TO MONITOR THESE, and regular users WILL NOT BE ABLE TO REPORT THESE.
Aren’t these images deleted if they aren’t used for the post/comment/banner/avatar/icon?
NOPE! The image actually stays uploaded! Lemmy doesn’t check if the images are used! Try it out yourself. Just make sure to copy the link by copying the link text or copying it by clicking the image then “copy image link”.
How come this hasn’t been addressed before?
I don’t know. I am fairly certain that this has been brought up before. Nobody paid attention but I’m bringing it up again after all the shit that happened in the past week. I can’t even find it on the GitHub issue tracker.
I’m an instance administrator, what the fuck do I do?
Check your pictrs images (good luck) or nuke it. Disable pictrs, restrict sign ups, or watch your database like a hawk. You can also delete your instance.
Good luck.
FYI to all admins: with the next release of pict-rs, it should be much easier to detect orphaned images, as the pict-rs database will be moved to postgresql. I am planning to build a hashtable of “in-use” images by iterating through all posts and comments by lemm.ee users (+ avatars and banners of course), and then I will iterate through all images in the pict-rs database, and if they are not in the “in-use” hash table, I will purge them.
Of course, Lemmy can be improved to handle this case better as well!
Perhaps someone should create a script to purge orphan images
Seems like the logical fix
Very much needed.
I’m an instance administrator, what the fuck do I do?
There’s one more option. The awesome @db0@lemmy.dbzer0.com has made this tool to detect and automatically remove CSAM content from a pict-rs object storage.
This is a nice tool but orphaned images still need to be purged. Mentioned on the other thread that bad actors can upload spam to fill up object storage space.
That is also very true. I think better tooling for that might come with the next pict-rs version, which will move the storage to a database (right now it’s in an internal ky-value storage). Hopefully that will make it easier to identify orphaned images.
I tried getting this to run in a container, but I was unable to access my GPU in the container. Does anyone have any tips on doing that?
Sorry I haven’t ran this myself yet nor have any experience with that kind of issues. But may I ask why you were concerned with running it inside of a container? Seems rather unnecessary to me.
Running anything in a container isn’t necessary. It just makes it easier to run, as it comes with all the dependencies. And if you decide you don’t want it anymore, you can just remove the container and it and all its dependencies are gone, which is really clean. It also makes the environment extremely repeatable, so people on all distros can run it with the exact same steps. And you don’t need to worry about what version of python you have and if it’s compatible with the dependencies. For example, the dependencies for this script require python 3.10 exactly. You can’t use 3.9 or 3.11.
So really the only reason was I wanted to make it easier for everyone.
I see. I considered the dependency problem but only thought of using a venv to fix that, however you are right, the python version is also often the cause of compatibility issues.
You can go one step further and use a conda env, which would also include the proper python version. All you need then is the micromamba binary. I might develop that as all it would need is to run a shell script to start
Honestly, I’m a little sick of needing to make a venv for each python script, which is why I’m trying to put all my python scripts in containers. I already got db0’s project to the point where anyone can run it with one single command line that you can just copy/paste (assuming you have docker installed already). It is just running on the CPU, which is painfully slow.
Same, it’s the reason why I can’t stand working with python.
Thank you for doing this, btw. Once you have something working on your hands you could consider spreading the word, maybe to db0 himself. I sure would love a convenient way to run that script, and many other admins probably would too.
You need a GPU for that. Most $5 VPSs don’t have that.
Yeah I know. It’s supposed to be ran from your computer, not the VPS.
Would I mount the the pictrs folders as a network folder locally?
No. Unfortunately it only works with storages on object storages like S3 buckets, not with filesystem storages. Meaning it access the files remotely one at a time from the bucket, downloading them over the internet (I assume, I didn’t make this).
But the more important thing is that, as it states in the readme, no files get saved to your disk, they only stay in your RAM while they are being processed and everything is deleted right after. This is relevant because even having had CSAM on your disk at some point can put you in trouble in some countries, with this tool it never happens.
Which btw is the same reason why mounting the pict-rs folder to your local computer is probably not a good idea.
theoretically this tool could be adjusted to go via scp and read your filesystem pict-rs storage as well, Just someone has to code it.
Interesting. That would be a nice extension, I think most small admins are using the filesystem (I know I am lol).
'm an instance administrator, what the fuck do I do?
Check your pictrs images (good luck) or nuke it. Disable pictrs, restrict sign ups, or watch your database like a hawk. You can also delete your instance.
How? I have checked, and there doesn’t seem to be any way to see the photos on my server.
I actually shut down pictrs entirely on my instance. Running pictrs in its current state is criminally negligent imo.
They are stored in the pctrs folder. They don’t have file extensions but are viewable with many image programs.
Oh, I see. I only use command line on my server, so I didn’t realize they were actual photos. Thanks!
FYI this requires a JWT so if registrations are closed on your instance you don’t have to worry
This is for public instances.
It seems like self-hosting your own Lemmy instance with registrations, communities, and pretty much anything else turned off is still very safe to do. I still want to end up self-hosting my own Lemmy instance some time when I have more time. Though I’d rather wait for things to be more stable first, there’s bugs I’d like to be ironed out before doing that probably, like one example is I still find it annoying that upvoting a comment in a thread deletes whatever comment you’re currently typing.
A lot of web software does this (Github and Gmail for example). I like it but always thought it could be abused.
You mean Gmail drafts? I know from at least one case where criminals used this, they shared the Gmail account password and messaged each other only via the drafts function. So technically there was never a mail send.
They probably have the tools to deal with it. Lemmy certainly doesn’t.
Oh wow. I always assumed the images are deleted if you don’t submit the post.
😬
Sadly not the case
How come this hasn’t been addressed before?
Because pictrs and most other components of Lemmy was designed for a much smaller use case by a very small development team. It was designed primarily by people volunteering their time and expertise. Most of the contributors have other things to do on a full-time basis. If you really want to see a change like this implemented NOW, then code it in yourself, file a new issue directly on their page with potential solutions, or donate to the people working on it.
Your post is good for the most part, but my patience is limited for the kind of entitled attitude you show under that heading specifically. Thanks for hearing me out.
OP is flagging a legitimate issue that can actually put instance owners at risk. Raising the issue that instance owners can unwittingly host illegal content and be liable for it - how is that entitled?
Totally understand that Lemmy devs are a small team, but the growth of use of the software is exploding now, and not being able to keep up is a problem of scale - gatekeeping others from raising issues does not help it get better and in fact discourages issue reports and promotes a head-in-the-sand culture.
I understand and raising the issue and discussion is fine. With all due respect to OP, I take it personally when the discussion is framed with the implication that the developers should not have released a project with some bugs and they should have put more effort here or there. I’ve contributed to Lemmy both in coding, translation and small donations, but I’m not here for people to push blame on devs. This is why bringing up the question “Why hasn’t anything been done?”, while I recognize it is a question on some people’s minds, it gets on my nerves. It bothers me like a clickbait/ragebait title does for many.
I would rather the discussion focus on where efforts are made or will be made to mitigate and fix the problem.
Entitlement? The “Subtitles” are acting as a Question the reader may have, and below the answer, OP is not demanding anything
Fair point. That question itself is what bothers me even if it is a valid one people have on their minds. The answer to that question should highlight more clearly what has been done, and if OP doesn’t know, then IMO it would best be to not include that question/answer.
I have no problems with OP’s post and the fact to bring up this issue and dicsuss it. Including that question with an incomplete answer bothers me like a clickbait headline for an article does, or how Tucker Carlson’s show asks questions. This serves little purpose but put the people working on fixes in a bad light acting like they haven’t been working on anything.
Entitled attitude? I’m just bringing it up again. It was brought up some time ago but wasn’t given attention so I’m bringing it up again after the recent CSAM attacks.
I didn’t demand anything in the post. I brought up the issue, explained why it’s important, and what admins could do about it.
I don’t know how to code but that doesn’t mean I’m not allowed to bring this issue to light…
I have no issue with your post itself and discussing this issue it is important to highlight things like this. Thank you for bringing it up, and sorry if I sound mad at you for doing that.
I will point out, the specific thing that bothers me is that the heading
How come this hasn’t been addressed before?
contains an incomplete answer that ignores work that is currently in progress by devs to address. I don’t blame you for not knowing the answer but for including and answering that question when you don’t know the answer. To me it’s reminiscent of Tucker Carlson-style questioning, where some issue is brought up, questions are asked but then the answer is sparsely researched and the viewer is expected to come to some conclusion of who to blame. This specifically is what gets on my nerves.
If you can include where work to rectify the issue had been discussed and is in progress like github issues, discussion throughout Lemmy and other things, I’ll edit my first reply to note my concern is assuaged.
E: Here are some of the relevant issues and discussion:
- This sort of thing has been discussed since 2020
- Adding admin purging of DB items and pictures. #904 #1331 #1809
- #3504, #2277, these and related issues are still open.
I don’t care if you don’t like my English writing. I brought up the issue and if people don’t care about it then whatever. We’ll just have to wait until it’s abused then maybe people will be actually concerned.
I’m glad to live in a world where concern about safety is considered entitlement somehow
Because anyone can upload illegal images without the admin knowing and the admin will be liable for it.
The admin/company isn’t liable until it is reported to them and they don’t do anything about it… That’s how all social media sites work, Google isn’t immediately liable if you upload illegal materials to GDrive and share it anonymously.
Doesn’t change the fact that this is an issue. Besides, do you think American law applies everywhere?
This can be solved very easily by a cron job to clean out the folder periodically, if you’re worried about it.
Very easily you say? Maybe tell us what this cron job is so we can all add it?
Just make a cron that runs the
rmcommand every day or whatever to clean out the files. Then run a SQL query at the same time to truncate any draft posts in the database. There’s no logic to this method, it just clears out the files and records related to draft posts, but it’s fast and effective.There’s a small chance it might fuck somebody up if they were writing a post at that exact moment, but you can schedule the cron for when your instance is the quietest.
wouldn’t it be just as easy to whitelist DNS?
Yes it would, if the problem had anything to do with the DNS.
The problem is people and people have DNS.
Explain.
He probably means whitelisting domains when posting already uploaded images, clearly not having read the post
That’s another issue. Also a necessary feature.
No I mean the user’s DNS should be whitelisted to permit uploads. If DNS not on whitelist then no upload, period.
What do you mean by “the user’s DNS” exactly??
Other than fulling up storage, what is the actual issue? If the image is orphaned then surely nobody can actually access the content? Sure you could be blind hosting things but if nobody can get the content back out then the abuse is surely minimal apart from say a complex cyber and physical targetted campaign or simply fulling up storage…
The issue is that you can share the image link to other people. People CAN get the content back out and admins or moderators WILL NOT KNOW about it.
So if someone uploads an illegal image in the comments, copies the link and does not post the comment, then they have a link of an illegal image hosted on someone’s Lemmy instance. They can share this image to other people or report it to the FBI. Admins won’t know about this UNLESS they look at their pictrs database. Nobody else can see it so nobody can report it.