A Telegram user who advertises their services on Twitter will create AI-generated porn of anyone for a price, and has also targeted minors.
A Telegram user who advertises their services on Twitter will create an AI-generated pornographic image of anyone in the world for as little as $10 if users send them pictures of that person. Like many other Telegram communities and users producing nonconsensual AI-generated sexual images, this user creates fake nude images of celebrities, including images of minors in swimsuits, but is particularly notable because it plainly and openly shows one of the most severe harms of generative AI tools: easily creating nonconsensual pornography of ordinary people.
"Djinn", specifically, being the correct word choice. We're way past fun-loving blue cartoon Robin Williams genies granting wishes, doing impressions of Jack Nicholson and getting into madcap hijinks. We're back into fuckin'... shapeshifting cobras woven of fire and dust by the archdevil Iblis, hiding in caves and slithering out into the desert at night to tempt mortal men to sin. That mythologically-accurate shit.
People are going to do what they're going to do, and the existence of this isn't an argument to put spyware on everyone's computer to catch it or whatever crazy extreme you can take it to.
But distributing nudes of someone without their consent, real or fake, should be treated as the clear sexual harassment it is, and result in meaningful criminal prosecution.
While I agree in spirit, any law surrounding it would need to be very clearly worded, with certain exceptions carved out. Which I'm sure wouldn't happen.
I could easily see people thinking something was of them, when in reality it was of someone else.
I'm not familiar with the US laws, but… isn't it already some form of crime or something to distribute nude of someone without their consent? This should not change whether AI is involved or not.
This is not new. People have been Photoshopping this kind of thing since before there was Photoshop. Why "AI" being involved matters is beyond me. The result is the same: fake porn/nudes.
And all the hand wringing in the world about it being non consensual will not stop it. The cat has been out of the bag for a long time.
I think we all need to shift to not believing what we see. It is counterintuitive, but also the new normal.
This feels entirely non-sequitur, to the point of damaging any point you're trying to make. Whether I paint a nude or the modern Leonardi DaVinci paints a nude our rights (and/or the rights of the model, depending on your perspective on this issue) should be no different, despite the enormous chasm that exists between our artistic skill.
A kid at my high school in the early 90s would use a photocopier and would literally cut and paste yearbook headshots onto porn photos. This could also be done in bulk and doesn't require any skills that a 1st grader doesn't have.
I hate this: "Just accept it women of the world, accept the abuse because it's the new normal" techbro logic so much. It's absolutely hateful towards women.
We have legal and justice systems to deal with this. It is not the new normal for me to be able to make porn of your sister, or mother, or daughter. Absolutely fucking abhorrent.
I don't know why you're being down voted. Sure, it's unfortunately been happening for a while, but we're just supposed to keep quiet about it and let it go?
I'm sorry, putting my face on a naked body that's not mine is one thing, but I really do fear for the people whose likeness gets used in some degrading/depraved porn and it's actually believable because it's AI generated. That is SO much worse/psychologically damaging if they find out about it.
How do you propose to deal with someone doing this on their computer, not posting them online, for their "enjoyment"? Mass global surveillance of all existing devices?
It's not a matter of willingly accepting it; it's a matter of looking at what can be done and what can not. Publishing fake porn, defaming people, and other similar actions are already (I hope… I am not a lawyer) illegal. Asking for the technology that exists, is available, will continue to grow, and can be used in a private setting with no witness to somehow "stop" because of a law is at best wishful thinking.
It's not normal but neither is new: you already could cut and glue your cousin's photo on a Playboy girl, or Photoshop the hot neighbour on Stallone's muscle body. Today is just easier.
I suck at Photoshop and Ive tried many times to get good at it over the years. I was able to train a local stable diffusion model on my and my family's faces and create numerous images of us in all kinds of situations in 2 nights of work. You can get a snap of someone and have nudes of them tomorrow for super cheap.
I agree there is nothing to be done, but it's painfully obvious to me that the scale and ease of it that makes it much more concerning.
Also the potential for automation/mass-production. Photoshop work still requires a person to sit down to do the actual photoshop. You can try to script things out, but it's hardly an easy affair.
By comparison, generative models are much more hands-free. Once you get the basics set up, you can just have it go, and churn things at rates well surpassing what a single human could reasonably do (if you have the computing power for it).
Exactly this. And rather believe cryptographically sighed images by comparing hashes with the one supplied by the owner. Then it's rather a question of trusting a specific source for a specific kind of content. A news photo of the war in Ukraine by the BBC? Check hash on their site. Their reputation is fini if a false image has been found.
At the same time, that does introduce an additional layer of work. Most people aren't going to do that just for the extra work that it would involve, in much the same way that people today won't track down an image back down to the original source, but usually just go by the one that they saw.
Especially for people who aren't so cryptographically or technologically inclined that they know what a hash is, where to find one, and how to compare it (without just opening them both and checking personally).
This is something I can't quite get through to my wife. She does not like that I dismiss things to some degree when it does not makes sense. We get into these convos where Im like I have serious doubts about this and she is like. Are you saying it did not happen and im like. no. It may have happened but not in quite the way they say or its being portrayed in a certain manner. Im still going to take video and photos for now as being likely true but I generally want to see it from independent sources. like different folks with their phones along with cctv of some kind and such.
I'll admit I used to look at celeb deepfakes, but once I saw that video I stopped immediately and avoid it as much as I possibly can. I believe porn can be done correctly with participant protection and respect. Regarding deepfakes/revenge porn though that statistic about suicidal ideation puts it outside of healthy or ethical. Obviously I can't make that decision for others or purge the internet, but the fact that there's such regular and extreme harm for the (what I now know are) victims of non-consensual porn makes it personally immoral. Not because of religion or society but because I want my entertainment to be at minimum consensual and hopefully fun and exciting, not killing people or ruining their happiness.
I get that people say this is the new normal, but it's already resulted in trauma and will almost certainly continue to do so. Maybe even get worse as the deepfakes get more realistic.
once non-consensual pornography (which deepfakes are classified as) is made public over half of people involved will have the urge to kill themselves.
Not saying that they are justified or anything but wouldn't people stop caring about them when they reach a critical mass?
I mean if everyone could make fakes like these, I think people would care less since they can just dismiss them as fakes.
The analogy given is it’s like watching video the next day of yourself undergoing sex without consent as if you’d been drugged.
You want a world where people just desensitise themselves to things that make them want to die through repeated exposure. I think you'll get a whole lot of complex PTSD instead.
I think this is realistically the only way forward. To delegitimize any kind of nudes that might show up of a person. Which could be good. But I have no doubt that highschools will be flooded with bullies sending porn around of innocent victims. As much as we delegitimize it as a society, it'll still have an effect. Like social media, though it's normal for anyone to reach you at any time, It still makes cyber bullying more hurtful.
I'm wondering if this may already be illegal in some countries. Revenge porn laws now exist in some countries, and I'm not sure if the legislation specifies how the material should be produced to qualify. And if the image is based on a minor, that's often going to be illegal too - some places I hear even pornographic cartoons are illegal if they feature minors. In my mind people who do this shit are doing something pretty similar to putting hidden cameras in bathrooms.
That's a ripoff. It costs them at most $0.1 to do simple stable diffusion img2img. And most people could do it themselves, they're purposefully exploiting people who aren't tech savvy.
I have no sympathy for the people who are being scammed here, I hope they lose hundreds to it. Making fake porn of somebody else without their consent, particularly that which could be mistaken for real if it were to be seen by others, is awful.
I wish everyone involved in this use of AI a very awful day.
But fuck dude they aren't taking advantage of anyone buying the service. That's not how the fucking world works. It turns out that even you have money you can post for people to do shit like clean your house or do an oil change.
NOBODY on that side of the equation are bring exploited 🤣
In my experience with SD, getting images that aren't obviously "wrong" in some way takes multiple iterations with quite some time spent tuning prompts and parameters.
It's not like deep fake pornography is "built in" but Stable Diffusion can take existing images and generate stuff based on it. That's kinda how it works really. The de-facto standard UI makes it pretty simple, even for someone who's not too tech savvy: https://github.com/AUTOMATIC1111/stable-diffusion-webui
FR is not generative AI, and people need to stop crying about FR being the boogieman. The harm that FR can potentially cause has been covered and surpassed by other forms of monitoring, primarily smartphone and online tracking.
We are acting as if through out history we managed to orient technology so as to to only keep the benefits and eliminate negative effects. while in reality most of the technology we use still comes with both aspects. and it is not gonna be different with AI.
I'd like to share my initial opinion here. "non consential Ai generated nudes" is technically a freedom, no? Like, we can bastardize our president's, paste peoples photos on devils or other characters, why is Ai nudes where the line is drawn? The internet made photos of trump and putin kissing shirtless.
They're making pornography of women who are not consenting to it when that is an extremely invasive thing to do that has massive social consequences for women and girls. This could (and almost certainly will) be used on kids too right, this can literally be a tool for the production of child pornography.
Even with regards to adults, do you think this will be used exclusively on public figures? Do you think people aren't taking pictures of their classmates, of their co-workers, of women and girls they personally know and having this done to pictures of them? It's fucking disgusting, and horrifying. Have you ever heard of the correlation between revenge porn and suicide? People literally end their lives when pornographic material of them is made and spread without their knowledge and consent. It's terrifyingly invasive and exploitative. It absolutely can and must be illegal to do this.
Given that it can be done in a private context and there is absolutely no way to enforce it without looking into random people's computer unless they post it online publicly, you're just asking for a new law to reassure people with no effect. That's useless.
Seems to fall under any other form of legal public humiliation to me, UNLESS it is purported to be true or genuine. I think if there’s a clear AI watermark or artists signature that’s free speech. If not, it falls under Libel - false and defamatory statements or facts, published as truth. Any harmful deep fake released as truth should be prosecuted as Libel or Slander, whether it’s sexual or not.
The internet made photos of trump and putin kissing shirtless.
And is that OK? I mean I get it, free speech, but just because congress can't stop you from expressing something doesn't mean you actually should do it. It's basically bullying.
Imagine you meet someone you really like at a party, they like you too and look you up on a social network... and find galleries of hardcore porn with you as the star. Only you're not a porn star, those galleries were created by someone who specifically wanted to hurt you.
AI porn without consent is clearly illegal in almost every country in the world, and the ones where it's not illegal yet it will be illegal soon. The 1st amendment will be a stumbling block, but it's not an impenetrable wall - congress can pass laws that limit speech in certain edge cases, and this will be one of them.
The internet made photos of trump and putin kissing shirtless.
And is that OK?
I'm going to jump in on this one and say yes - it's mostly fine.
I look at these things through the lens of the harm they do and the benefits they deliver - consequentialism and act utilitarianism.
The benefits are artistic, comedic and political.
The "harm" is that Putin and or Trump might feel bad, maaaaaaybe enough that they'd kill themselves. All that gets put back up under benefits as far as I'm concerned - they're both extremely powerful monsters that have done and will continue to do incredible harm.
The real harm is that such works risk normalising this treatment of regular folk, which is genuinely harmful. I think that's unlikely, but it's impossible to rule out.
Similarly, the dissemination of the kinds of AI fakes under discussion is a negative because they do serious,measurable harm.
I think the biggest thing with that is trump and Putin live public lives. They live lives scrutinized by media and the public. They bought into those lives, they chose them. Due to that, there are certain things that we push that they wouldn't necessarily be illegal if we did them to a normal, private citizen, but because your life is already public we turn a bit of a blind eye. And yes, this applies to celebrities, too.
I don't necessarily think the above is a good thing, I think everyone should be entitled to some privacy, having the same thing done to a normal person living a private life is a MUCH more clear violation of privacy.
Public figures vs private figures. Fair or not a public figure is usually open season. Go ahead and make a comic where Ben Stein rides a horse home to his love nest with Ben Stiller.
Lemme put it this way. Freedom of speech isn't freedom of consequences. You talk shit, you're gonna get hit. Is it truly freedom if you're infringing on someone else's rights?
Don't get me wrong it's unsettling, but I agree, I don't see the initial harm. I see it as creating a physical manifestation of someone's inner thoughts. I can definitely see how it could become or encourage dangerous situations, but that's like banning alcohol because it could lead to drunk driving or sexual assault.
Innocently drinking alcohol is in NO WAY compared to creating deepfakes of people without consent.
One is an innocent act that has potentially harsh consequences, the other is a disgusting and invasively violating act that has the potential to ruin an innocent persons life.
It's gonna suck no matter what once the technology became available. Perhaps in a bunch of generations there will be a massive cultural shift to something less toxic.
May as well drink the poison if I'm gonna be immersed in it. Cheers.
I was really hoping that with the onset of AI people would be more skeptical of content they see online.
This was one of the reasons. I don't think there's anything we can do to prevent people from acting like this, but what we can do as a society is adjust to it so that it's not as harmful. I'm still hoping that the eventual onset of it becoming easily accessible and useable will help people to look at all content much more closely.
It's stuff like this that makes me against copyright laws. To me it is clear and obvious that you own your own image, and it is far less obvious to me that a company can own an image whose creator drew multiple decades ago that everyone can identify. And yet one is protected and the other isn't.
What the hell do you own if not yourself? How come a corporation has more rights than we do?
Every time this comes up, all the tech nerds here like to excuse it as fine and not a bad thing at all. I am hoping this won't happen this time, but knowing lemmys audience...
I think part of the difficulty discussing this is the discussions usually combine two different things. The production and distribution.
I was informed elsewhere in this thread people can already produce these images/videos on their own machines with no third parties involved or remote processing. I can't think of a single thing that can be done about that so acceptance is all we've got.
Nonconsensual sharing, on the other hand, we can and should do something about. The legal system won't be able to stop it altogether but it can push it to the fringes and stop it from becoming mainstream so any victims wouldn't see fake images/videos of themselves proliferating everywhere.
It's not a matter of excusing it. Distribution of someone's picture without their explicit consent, and anything like that, is inexcusable. But we're talking about the generation of said content, which technically can't be stopped without seriously restraining everything.
I'm not saying it's not a bad thing but it's inevitable. The problem will just be getting worse and there's no stopping it. It's something we're just going to need to accept as a new normal. If we can deal with living under the constant threat of nuclear armageddon then I think we can live with fake nudes aswell.
Yeah it's this shit I'm talking about. We have a whole legal and justice system to deal with this. No kne needs to accept sexual abuse as a new normal. This shit is weird.
God, generative ai is such a fucking caustic technology. I honestly don't see anything positive and not disgusting enabled by this tech.
Edit: I see people don't agree, but like why can't ai stick to translating stuff and being useful rather than making horrifically unethical porn, taking the humanity out of art, and replacing peoples jobs with statistical content generation. I hate it here.
The distinction is that I can see worthwhile use cases for non-generative ai, and not for generative ai, and generative ai is built on theft of creative labor
I'm not angry at people who use generative ai, I'm angry at the people who built it by stealing from creatives to build a commercial tool that can seemingly only be used in awful ways.
I say stop antagonizing the AI.
The only difference between a skilled artist making it with Photoshop and someone using a Neural Net, is the amount of time and effort put into creating the instance.
If there are to be any laws against these, they need to apply to any and every fake that's convincing enough, no matter the technology used.
Don't blame the gunpowder for the dead person.
People were being killed by thrown stones before that.
The laws that oppress us on a daily basis suck ass I'll give yall that for fucking sure.... but downvoting someone wishing for the law equally being applied to all?
OooOo!
That's some high number of dwnv0t3s!
I wouldn't have realised unless you had replied here.
Nice, but it's also good that everyone is at least free to downvote and see the number of downvotes, unlike YouTube.
All over history, there has been this trend of people misusing technology and then blaming the technology instead of those that misuse it.
This trend is detrimental to the technological progress of a civilisation and is one of the driving forces, causing the cycle that our civilisation is stuck in (of losing all tech and history every once a while and then having to start over from the dark ages).
Technology, gives someone the ability to do something, but it is their will that makes them want to do so. If the "something" is considered "bad" for society, then instead of taking away the ability, we need to insist on getting the person to understand, why and how, said "something" is a problem for the society.
Until this problem is fixed, we are going to be stuck at the barrier and the next levels of civilisation shall stay a part of Fiction.
The root problem is government not enforcing the law on internet. Deepfakes existed for years.
The law enforcement should be more proactive on internet.