Researchers jailbreak AI chatbots with ASCII art -- ArtPrompt bypasses safety measures to unlock malicious queries
Researchers jailbreak AI chatbots with ASCII art -- ArtPrompt bypasses safety measures to unlock malicious queries
ArtPrompt bypassed safety measures in ChatGPT, Gemini, Clause, and Llama2.
You're viewing a single thread.
ooh hot take. reasearchers should stop doing security testing for OpenAI for free. aren’t they just publishing these papers, with full details on how it might be fixed, with no compensation for that labor?
bogus. this should work more like pen testing or finding zero day exploits. make these capitalist “oPeN” losers pay to secure the shit they create.
(pls tell me why im wrong if i am instead of downvoting, just spitballing here)
I highly doubt that OpenAI or any other AI developer would see any real repercussions, even if they had a security hole that someone managed to exploit to cause harm. Companies exist to make money, and OpenAI is no exception; if it's more profitable to release a dangerous product than a safe one, and they won't get in trouble for it, they'll likely have no issues with releasing their product with security holes.
Unfortunately, the question can't be "should we be charging them for this?" Nobody is going to force them to pay, and they have no reason to do it on their own. Barring an entire cultural revolution, the question instead must be "should we do it anyway to prevent this from being used in harmful ways?" And the answer is yes. Our society is designed to maximize profits, usually for people who already have money, so if you're working within the confines of that society, you need to factor that into your reasoning.
Companies have long since decided that ethics is nothing more than a burden getting in the way of their profits, and you'll have a hard time going against the will of the companies in a capitalist country.
oh! i see we have two different definitions of “security,” both of which are valid to discuss, but yours is not the one that relates to my point.
you understood “security” in a harm-reduction sense. i.e., that an LLM should not be permitted to incite violence, should not partake in emotional manipulation or harrasment of the user, and a few other examples like it shouldn’t be exploitable to leak PII. well and good, i agree that researchers publishing these harm-reduction security issues is good and should be continued.
my original definition of “security” is distinct and might be called “brand security.” OpenAI primarily wants to make use of their creation by selling it to brands for use in human-facing applications, such as customer service chat bots. (this is already happening and a list of examples can be found here.) as such, it behooves OpenAI to not only make a base-level secure product, but also one that is brand-friendly. the image in the article is one example—it’s not like human users can’t use google to find instructions to build a bomb. but it’s not brand friendly if users are able to ask the Expedia support bot or something for those instructions. other examples include why openAI have intentionally kept the LLM from saying the n-word (among other slurs), kirby doing 9-11 or writing excessively unkind or offensive output for users.
these things don’t directly cause any harm, but they would harm the brand.
I think that researchers should stop doing this “brand security” work for free. I have noticed this pattern where a well-meaning researcher publishes their findings of ways they were able to manipulate the brand-unsafe blackbox they published, quickly followed by a patch once the news spreads. In essence these corps are getting free QA for their products when they should just be hiring and paying these researchers for their time.
Ah, I see. It's true that these issues cast a negative light on AI, but I doubt most people will even hear about most of them, or even really understand them if they do. Even when talking about brand security, there's little incentive for these companies to actually address the issues - the AI train is already full-steam ahead.
I work with construction plans in my job, and just a few weeks ago I had to talk the CEO of the company I work for out of spending thousands on a program that "adds AI to blueprints." It literally just added a chatgpt interface to a pdf viewer. The chat wasn't even able to actually interact with the PDF in any way. He was enthralled by the "demo" that a rep had shown him at an expo, that I'm sure was set up to make it look way more useful than it really was. After that whole fiasco, I lost faith that the people in charge of whether or not AI programs are adopted will actually do their due diligence to ensure they're actually helpful.
Having a good brand image only matters if people are willing to look.
glad i was able to clarify.
there’s little incentive for these companies to actually address these (brand security) issues
this is where i disagree, and i think the facts back me up here. bing’s ai no longer draws kirby doing 9/11. openAI continues to crack down on ChatGPT saying slurs. it’s apparent to me that they have total incentive to address brand security, because brands are how they are raking in cash.
Oh, I'm sure they'll patch anything that gets exposed, absolutely. But that's just it - there are already several examples of people using AI to do non-brand-friendly stuff, but all the developers have to do is go "whoops, patched" and everyone's fine. They have no need to go out of their way to pay people to catch these issues early; they can just wait until a PR issue happens, patch whatever caused it, and move on.
the fact that you explained the problem doesn’t make it not a problem
Bug bounty programs are a thing.
Yes, an exploitative thing that mostly consists of free labour for big orgs.
yes i am aware? are they being used by openai?
That's how open software works. It's there for anyone to do whatever they want with it. Bonus if you charge money for it