Need to let loose a primal scream without collecting footnotes first? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid: Welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.
The post Xitter web has spawned soo many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.
(Semi-obligatory thanks to @dgerard for starting this.)
You must log in or register to comment.
hells yeah it’s time for some action - Drew DeVault is organizing a sit-in protest of Jack Dorsey’s keynote at FOSDEM 2025.
eh? I don’t see Jackie D’s keynote in the schedule, did the threat of a sit-in make them delete it? https://fosdem.org/2025/schedule/ edit: oh, it’s linked from Drew’s post.
The people of Muskogee will have their revenge on Jack Dorsey
So, the Wikipedia article about “prompt engineering” is pretty terrible. First source: OpenAI. Second: a blog. Third: OpenAI. Fourth: OpenAI’s blog. ArXiv, arXiv, arXiv… 43 times. Hop on over to the Talk page, and we find this gem:
It is sometimes necessary to make assumptions to write an article (see WP:MNA).
Spoiler alert: that link doesn’t justify anything. It basically advises against going off on tangents: There’s no need to rehash the fact that evolution is a fact on every damn biology page. It does not say that Wikipedia should have an article on some creationist fantasy, like baraminology or flood geology, based entirely on creationist screeds that all cite each other.
I have spent the last half-hour in the angry dome
Read a rumor that zuck’s marriage is falling apart, which scans.
A second divorced man is about to hit the tower.
i wonder where his roman empire fascination fits the pattern? also, who’s next, maybe thiel?
my first thought on hearing the same was “if this is what he’s like in public, imagine what it’s like at home”
He has two sisters, and one of them wrote a book about the manosphere/redpill (from a non rw way) shit in 2018. So fam gatherings will be wild.
This does have “Divorced Dad” vibes tbh
Does creating a cover of Get Low called “Z-Pain” for your wife, or commissioning the creation of a giant green statue for your wife strike you as the sorts of things someone in an unhealthy marriage would do? I think not!
I wish that men would go to therapy instead of going full fascist. Fuck this whole society. The West cannot fall sooner.
Computer understanders, click this link for some psychic damage! :D (aside: I tried to upload this as an image but it didn’t work)
Coiners are terminally brain poisoned by financialization of everything. HTTP represented by three payment processors (and I don’t even know if paying with Google or Apple pay involves HTTP but whatever).
Yet the money protocol is Bitcoin, apparently.
here you go
12 of the most valuable protocols on earth!
Counting like a chatbot.
Ah, so Google Search runs on UDP, interesting, that must be why it’s so unreliable now.
HTTP: famous for doing Google pay, Apple Pay, PayPal, and nothing else. no other use has been found for http
(also, FTP! as indicated by these apps that famously don’t use or support it!)
Well obviously they can’t be the money protocol, whatever that even means. Surely something like FIX would be the closest thing to an actual protocol for money, as opposed to a system.
huh, pict-rs has been acting up a lot lately. I’m gonna give the whole node a quick reboot later and see if that fixes things
ow.
The fact that DNS is “Domain Name Protocol” rather than the actual acronym (Domain Name System) is baffling and maddening.
Ah, the Image Upload Protocol must have gone woke.
Paul I am begging you to actually write out a fucking timeline. Apparently woke started in the 80s in universities when the (white) civil rights protestors of the 70s got tenure in the 60s, as an inevitable and predictable extension of political correctness in the 90s. From the title you’re obviously going to indulge the conservative fantasy that “wokeness” is a coherent thing rather than a political tool to dismiss calls for action to actually address blatant injustice. But if you’re going to bullshit me, at least do it competently and have an internally consistent narrative that allows for the natural passage of time.
If you can’t get through two short paragraphs without equating Stalinism and “social justice”, you may be a cockwomble.
Man wrote nearly 5k words of pure unfiltered cap:
I’m not sure how someone can read all this without capping themselves. We could sneer this all fucking day.
The opening statement is also quite silly already (and makes me belief in a companion to the dead internet theory, the dementia internet theory, as I was sure we have had conversations like this as ‘the internet’ already, Zuck turning manospherian all of a sudden also makes me thing this (same with the fight over H-1B on the US right, they had that in 2018 already, Trump likes H-1B)).
We had the whole ‘they act like they are morally superior’ discussion already a lot, and that was about vegans. Only one problem, they are morally superior on almost all ethical/moral/ideological systems you can think of. Sure hedonists, stoics (who are not allowed to complain), sadists, accelerationist extinctionists, ironic nihilistic status quo pushing postmodernists, all disagree they are superior morally but who cares about the opinion of those people. Sure some of them might be annoying to people, but annoying people can be morally superior.
His statements about how politically correctness comes from the 80’s is also wrong (it predates that, and has quite a complex history of being used by various different groups for different meanings), but at that moment I knew I was going to be wasting my time reading this as I would disagree with every paragraph. (as I have seen these types of articles before, they were popular a decade ago or so).
E2: Whoops that edit should have been on a different post. E3: bonus content: Two articles sneering at Paul, Paul Graham and the Cult of the Founder and Paul Graham, proto-techbro..
As many writers (perhaps most eloquently George Orwell) have observed, women seem more attracted than men to the idea of being moral enforcers.
Ah, thanks Paul for validating my disdain for Orwell at least.
George was writing his stories in the 40s, so at least has “product of his time” as an excuse.
Paul’s just a flat out piece of shit to be writing this nearly 100 years later.
Fair, though in Orwell’s case the misogyny is not accidental either, but an essential aspect of the mostly conservative ideology he adopted for 1984 (contempt for the working class, linguistic purism, just really being a little too enamoured with his perfect crystal of unending oppression etc).
I’ve never heard of anyone describing 1984 that way, could you elaborate on your points or link to some analysis?
I read it in high school. Iirc, the main character in 1984 deeply hates a woman he works with and his violent fantasies about her are tied up in his desire to rebel against the regime. He later overcomes his desire to commit violence against her by having sex with her. His contempt for her fairly leapt off the page when I read it. I’m sure it’s arguable what Orwell meant or intended.
In another scene, the middle-class protagonists watch a working-class woman hanging out washing and tell themselves that if there was any hope for freedom, it lay in “the proles” (members of the mass underclass, like that woman). But the way they look at her and talk about her is dehumanizing.
It’s probably easier to just read 1984 yourself and make up your own mind. it’s not a very long book.
Isn’t Julia a member of some sort of anti-sex league, meaning there’s a lot of bad faith involved in their relationship from the get go?
Also with respect to the attitudes on women and proles, although I don’t think it’s entirely written in the character’s point of view it feels like there’s a lot of unreliable narration going on, or at least you get a lot of stuff from the perspective of a person who grew up in one of the most absurdly totalitarian regimes in literature. Which is to say, it didn’t feel prescriptive most of the time to me.
See also: “proles”, as in the contempt is baked in to the language, which we know the regime is actively trying to hold in a tight leash.
I don’t think it’s a coincidence that the only viewpoint you get is that of a middle class bureaucrat. It’s the assumed audience, and it’s where Orwell would place himself as well. The narrative loses a lot of impact if you align yourself with the proles. Winston could live a real life if he really wanted to. I don’t think this point is intended by the novel.
Isn’t Julia a member of some sort of anti-sex league, meaning there’s a lot of bad faith involved in their relationship from the get go?
That’s a problem in itself, don’t you think? It’s all very “Feminists hate sex and they want to erase the differences between the genders”. Julia gets a taste of freedom and her right place in the world by putting on makeup and girly clothes and having a lot of sex.
Her lips were deeply reddened, her cheeks rouged, her nose powdered; there was even a touch of something under the eyes to make them brighter. It was not very skillfully done, but Winston’s standards in such matters were not high. He had never before seen or imagined a woman of the Party with cosmetics on her face. The improvement in her appearance was startling. With just a few dabs of color in the right places she had become not only very much prettier, but, above, all, far more feminine.
Also she’s a flighty moron.
I have not read it in ages, but did hear somebody has written something (not sure if book or play or etc) of the book from Julias perspective.
linguistic purism
That must have been really subtle, all I remember is a concern specifically about how a sufficiently totalitarian regime may try to weaponize language as a further means of subjugation, not that language evolving is bad in principle.
I think the premise of total control through language is in itself silly, though that can be excused by the book being satire. But Orwell, for good or ill, was undeniably a linguistic purist, as one can gather from a close reading of “Politics and the English Language”.
I said earlier that the decadence of our language is probably curable. Those who deny this would argue, if they produced an argument at all, that language merely reflects existing social conditions, and that we cannot influence its development by any direct tinkering with words and constructions. So far as the general tone or spirit of a language goes, this may be true, but it is not true in detail. Silly words and expressions have often disappeared, not through any evolutionary process but owing to the conscious action of a minority. Two recent examples were explore every avenue and leave no stone unturned, which were killed by the jeers of a few journalists. There is a long list of fly-blown metaphors which could similarly be got rid of if enough people would interest themselves in the job; and it should also be possible to laugh the not un- formation out of existence, to reduce the amount of Latin and Greek in the average sentence, to drive out foreign phrases and strayed scientific words, and, in general, to make pretentiousness unfashionable.
Huh.
I guess it stands to reason that the guy who made such a fuss about abusing language as a means to nefarious ends would himself have ideas about how it could be abused ethically.
Considering popes, priests in general, politicians etc are usually male (historically) i have a feeling these quotes also exclude some groups from being moral enforcers.
It also neatly ignores social pressures, which provides good reasons for women being into certain types of ‘moral enforcement’. Either because ‘it is their duty to protect the kids’ or the revolutionary idea that people are all people and should have equal rites, bodily autonomy, a political voice etc.
But nope: “me and the boys agree, this wokeness stuff is for girls”.
This all makes me wonder, we know he has proofreaders who help him. Did he either get rid of all the people who disagree with him, or did they give up, as some people dont want understand the other side they just want to argue their forever cause they believe they are correct (so disagreement is a massive waste of time).
E:
Thanks to Sam Altman, Ben Miller, Daniel Gackle, Robin Hanson, Jessica Livingston, Greg Lukianoff, Harj Taggar, Garry Tan, and Tim Urban for reading drafts of this. [emph mine, the names that really jumped out to me]
Ah. Also 1 name which jumps out to me as prob a woman. Let me google her. Ah right. His wife, and co-founder.
I’m not reading that shit but for the masochists out there who like to read HN licking VC boots, here ya go
I’ve been a big fan of HN comments lately
Did my regular check in of a q-pilled family member’s facebook page. Zuckerberg’s new fash turn is not being received well as he is being read as the worm that he is. i.e. they are still mad about the anti-vax fact checking.
I am sorry you have this family member.
It’s ok, it’s not your fault!
some of the first research science on promptfondlers and model-affine dipshits is starting to see the light of day and, in what will surprise probably 0% of our regulars, it confirms some things
(I have grumped about their desire for outsourced thinking in the past myself)
This is extremely tangential to the areas of sneer interest, but seeing as this is the only technology related community I am in, I’m putting it here.
This song has been making the rounds on the charts/social media and I refuse to believe that it isn’t about the package management tool apt
yoooooooooooooo
of course it is one in a long tradition of tech-tangential songs, including this banger about the inevitable collapse of tech bubbles
I remember boten anna as a tech song example
I hope Toni Basil and the Ting Tings get big royalty cheques.
the writers of hey mickey have a writing credit on it, but the ting tings got shafted.
A shame. Seems like they were clearly inspired by it … I guess you can’t copyright a vibe.
Welp, time to start the thread with fresh Awful for everyone to regret:
r/phenotypes
Thanks, I hate it.
Many of these threads are made in good faith and out of curiousity, but often times the comments become filled with hatred, ignorance, and trolling.
You don’t say.
Really? Right in the front of the clean new thread?
image description
Extremely simplistic drawing of a somewhat sad looking person. Drawn by the artist ‘flork of crows’[sic]. Reference to the ‘Really? Right In Front Of My X?’ meme
Some news on the people who own/run protonmail. They are pro Trump
mv protonmail protrumpmail
fucking hell. well i guess moving all my shit elsewhere has become my midweek project instead of my procrastinate-project.
EDIT: [rereads and comprehends the tuta addendum] well. cool. i think i’m going to reschedule that “move everything” to after my “scream and cry into a pillow”. anyone have any other recs?proton’s ceo is, at this moment, trying to post through it on multiple official company accounts
thank fuck I switched to tuta
[I hold up a hand to my earpiece and make a horrified face]
a second milkshake has hit the duck (cw: fetishized racism, you know the one)
I’m waiting to hear back from tuta about what the fuck but this might be the reason why I hop providers again
image description (but see cw)
so somebody at tuta had the idea to write this post on their official account on Twitter:
Don’t get cucked by Big Tech’s invasive policies. Keep your private data safe and secure with state of the art end-to-end encryption with Tutanota.
and then the image attached to the post is a cuckold porn meme I started to describe here but won’t??? because describing the image put me in a really terrible mood??? it’s SFW but trust me you don’t want it. the Black men in the picture have had their faces covered by logos for Microsoft, Amazon, Google, Apple, and Facebook in that order. the white woman in the picture does not have her face covered. impact font bottom text: “YOUR PERSONAL DATA”
Shit’s so diseased its a sequel to Tuskegee. They cooked this cack with CIA-grade crack.
(I have no rational way to respond to this - I’m just trying some Dracula Flow-type shit and hoping it lands.)
jfc
Someone should write a dissertation about that porn video and the memes.
there it is, sammy has gone and said people are just prompting the model wrong (I recall we’ve had that bit said here earlier)
but in true sammy grift: you just need to be asking the right questions to trump intelligence. “why do you want to suck, as a human?” sammy asks, not understanding a moment of humanity
how do you even define “raw, intellectual horsepower” and how does it differ from knowing how to formulate questions mother fucker
Pretty sure there’s gonna be a contract somewhere that defines raw intelligence as “the amount of money you make for OpenAI.”
“Raw, intellectual horsepower” means fucking an intellectual horse without a condom.
Oh, wait, that’s rawdogging intellectual horsepower, my mistake.
shot:
Von Neumann arguably had the highest processor-type “horsepower” we know of plus his breadth of intellectual achievements is unparalleled.
chaser:
But imo Grothendieck is a better comparison point for ASI as his intelligence, while being strangely similar to LLMs in some dimensions
oh my god
I never thought I’d say this but… don’t slander category theory like that, compared to LLMs it’s downright useful
the bit about it that I find subtly glorious (in how remarkably fuckwitted it is) is the baseline idea of “intellectual horsepower”
I’m not surprised that this is a view they (of the company that’s effectively going “just 12 more DCs bro it’ll be enough compute bro I promise bro just watch”) hold and consider in such a simple mechanism-rating scale
but it is funny as fuck
What they promise: intelligence systems, revolutionary agent which can take over tasks!
What we get: https://youtu.be/izazdBpraC8
a couple weeks back, I was (bc reasons) looking around to see how to turn off goog’s annoying gemini bullshit in an account, and you can!
except then even after doing that, accounts in that org still got prompts (in the form of in-app banners, and sparklebuttons in shit like gmail) to Try The Model
it looks like people aren’t biting enough, because now you get it whether you like it or not, for the low low price of pushing up your base account fee! and I checked in one org - “Gemini App” is disabled org-wide, but the fucking prompt is immediately in the UI (and you get a modal popover opening gmail)
fuck these people so much
Oh well. Nothing screams healthy business like force-feeding your product to every customer who can’t hammer the conveniently hidden opt-out button fast enough. I’m sure Gemini is doing great.
oh, no no
nooooo no no no
there isn’t an opt-out button
there is only:
- “Continue”,
- “Learn More”
Hello, I’d like to punch you in the groin. Will you accept ~or would you like to learn more~?
Sorry, I didn’t quite catch that. Did you say you accept?
Ah, you don’t want to be punched in the groin. That’s OK, I understand. We value your painless existence very much.
Now, obviously we cannot let you opt out of the Strictly Necessary punches in the groin. Surely you understand that if it’s necessary to punch you in the groin, your permission or lackthereof is irrelevant. Rest assured, this applies only when we really have to punch you in the groin.
What, do you want me to list all the possible circumstances in which one might be obligated to punch you in the groin? Don’t be unreasonable, now. I’m sure you know it when you see it.
That aside, I presume we can punch you in the groin for functional purposes? The kind that may not be strictly necessary, but serve a purpose in the functioning of our service.
Oh, we can’t? It’s OK, you have the right to make that choice. We don’t judge. Anyway, we take it that you’re probably at least cool with us punching you in the groin for the purposes of analyzing your behavior to improve our groin punching. Let me know if you decide you don’t want us to do that anymore.
Oh, I thought you were cooler than that. Alright, if you hate the working class and want to make it harder for the poor, overworked developers to improve your experience, we’ll do it your way. I guess we’ll have to make do with just the groin punches that are strictly necessary or for marketing purposes.
Ah, aren’t you observant. Have you ever noticed that all the adverts you get are really terrible? That’s because advertisers need to be able to punch you in the groin to find out what you like and to make their ads more appealing to you. Just food for thought. But if you really insist…
Fine, fine. Marketing groin punches are out. As for your question, no we don’t identify as an advertising company per se. But we are partnered with other companies that are in fact advertising companies. Would you like to adjust your preferences for our groin punching partners?
Well maybe to you it looks like the opt-out process we just went through should also cover this part but can we really know if we don’t look?
Who’s a good puppy? You’re a good puppy, yes you are! ❤️
Will you deny us permission to punch you in the groin on behalf of AAAAAAAAAAA Inc. or will you not?
OK, so we can only punch you in the groin on behalf of AAAAAAAAAAA Inc. for the purposes of Legitimate Interest?
It means the kinds of purposes where there is a legitimate interest to punch you in the groin.
Why would you ask if you didn’t want me to answer? Fine, that’s a no for Legitimate Interest based groin punching on behalf of AAAAAAAAAAA Inc.
Will you deny us permission to punch you in the groin on behalf of AAAAAAAAAAB Inc. or will you not?
Oh, we have a total of six hundred and sixteen thousand six hundred and sixty-six partners in our crotch impactizing network.
Indeed, we are proud to have such a wide network of trusted allies.
Ugh, fine. I guess I can check the end of the list to see if there’s a way to make a selection for all of them at once. Honestly, this form is starting to make me a bit dizzy as well.
Wow, who knew flipping through all those pages would take so long. There’s a line in here that says “disagree to all”, but there’s no checkbox or anything. It’s just there. Clicking it doesn’t seem to change anything. You can probably assume it worked.
Please calm down, we’re almost done. Would you like to accept and save?
Well it sounds like I mean “accept and save the options you just set”, not the ones we offered initially, doesn’t it?
Your groin punching settings have been applied. I don’t think there were any mistakes, but if you need to change the settings, you can find the form hidden somewhere in this house, assuming we remembered to put it there.
this is so wildly on point
yours?
(it should become an internet copypasta and drift into mass consciousness)
Thanks. I wrote this last night not expecting it to become so long, but I like to think the real work was done by thousands of very clever people with highly sophisticated moral compasses pretending not to understand privacy legislation.
Im gonna build a special circle in hell for these people, together with the ‘yes’ or ‘ask me again later’ people. On this circle all the software your stack depends on will break your build and releases a new release every friday at 5. Whohahhahah
Google+ but worse
little known historical fact: G+ was actually the mark that service got on its popularity exam
Possibly I’m the last to hear about this one, but seeing as proton mail has come up here a few times before: the founder and ceo Andy Yen is apparently a Trump fan.
Great pick by @realDonaldTrump. 10 years ago, Republicans were the party of big business and Dems stood for the little guys, but today the tables have completely turned. People forget that the current antitrust actions against Big Tech were started under the first Trump admin.
(from the beginning of december, on the nomination of trump staffer Gail Slater to antitrust post at the doj)
Sorry I and some others missed this post talking about the tweet so we have two top level ones atm. My more sensational post about the subject got a bit more attention. So you were not late but early.
Looks like LW/Lightcone managed to convince enough people to give then $2M, which will totally not be used to settle sexual assault lawsuits in the future.
It will instead be used to purchase another castle in which to perform SA
Crying in LW
It was a stately home goddammit! Stop weakmanning meeeeeeeeee
But how are we supposed to grift our absurd ideology into respectability without having a stately English manor to entertain people in??
Like every other ‘charity’, rent a room at some super expensive hotel, hire the most expensive cooks and give all your potential donors a free invite, and massive gifts.
Don’t confuse the Conference Castle with the (Non)Consent Castle!
Remember how OAI claimed that O3 had displayed superhuman levels on the mega hard Frontier Math exam written by Fields Medalist? Funny/totally not fishy story haha. Turns out OAI had exclusive access to that test for months and funded its creation and refused to let the creators of test publicly acknowledge this until after OAI did their big stupid magic trick.
From Subbarao Kambhampati via linkedIn:
"𝐎𝐧 𝐭𝐡𝐞 𝐬𝐞𝐞𝐝𝐲 𝐨𝐩𝐭𝐢𝐜𝐬 𝐨𝐟 “𝑩𝒖𝒊𝒍𝒅𝒊𝒏𝒈 𝒂𝒏 𝑨𝑮𝑰 𝑴𝒐𝒂𝒕 𝒃𝒚 𝑪𝒐𝒓𝒓𝒂𝒍𝒍𝒊𝒏𝒈 𝑩𝒆𝒏𝒄𝒉𝒎𝒂𝒓𝒌 𝑪𝒓𝒆𝒂𝒕𝒐𝒓𝒔” hashtag#SundayHarangue. One of the big reasons for the increased volume of “𝐀𝐆𝐈 𝐓𝐨𝐦𝐨𝐫𝐫𝐨𝐰” hype has been o3’s performance on the “frontier math” benchmark–something that other models basically had no handle on.
We are now being told (https://lnkd.in/gUaGKuAE) that this benchmark data may have been exclusively available (https://lnkd.in/g5E3tcse) to OpenAI since before o1–and that the benchmark creators were not allowed to disclose this *until after o3 *.
That o3 does well on frontier math held-out set is impressive, no doubt, but the mental picture of “𝒐1/𝒐3 𝒘𝒆𝒓𝒆 𝒋𝒖𝒔𝒕 𝒃𝒆𝒊𝒏𝒈 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒐𝒏 𝒔𝒊𝒎𝒑𝒍𝒆 𝒎𝒂𝒕𝒉, 𝒂𝒏𝒅 𝒕𝒉𝒆𝒚 𝒃𝒐𝒐𝒕𝒔𝒕𝒓𝒂𝒑𝒑𝒆𝒅 𝒕𝒉𝒆𝒎𝒔𝒆𝒍𝒗𝒆𝒔 𝒕𝒐 𝒇𝒓𝒐𝒏𝒕𝒊𝒆𝒓 𝒎𝒂𝒕𝒉”–that the AGI tomorrow crowd seem to have–that 𝘖𝘱𝘦𝘯𝘈𝘐 𝘸𝘩𝘪𝘭𝘦 𝘯𝘰𝘵 𝘦𝘹𝘱𝘭𝘪𝘤𝘪𝘵𝘭𝘺 𝘤𝘭𝘢𝘪𝘮𝘪𝘯𝘨, 𝘤𝘦𝘳𝘵𝘢𝘪𝘯𝘭𝘺 𝘥𝘪𝘥𝘯’𝘵 𝘥𝘪𝘳𝘦𝘤𝘵𝘭𝘺 𝘤𝘰𝘯𝘵𝘳𝘢𝘥𝘪𝘤𝘵–is shattered by this. (I have, in fact, been grumbling to my students since o3 announcement that I don’t completely believe that OpenAI didn’t have access to the Olympiad/Frontier Math data before hand… )
I do think o1/o3 are impressive technical achievements (see https://lnkd.in/gvVqmTG9 )
𝑫𝒐𝒊𝒏𝒈 𝒘𝒆𝒍𝒍 𝒐𝒏 𝒉𝒂𝒓𝒅 𝒃𝒆𝒏𝒄𝒉𝒎𝒂𝒓𝒌𝒔 𝒕𝒉𝒂𝒕 𝒚𝒐𝒖 𝒉𝒂𝒅 𝒑𝒓𝒊𝒐𝒓 𝒂𝒄𝒄𝒆𝒔𝒔 𝒕𝒐 𝒊𝒔 𝒔𝒕𝒊𝒍𝒍 𝒊𝒎𝒑𝒓𝒆𝒔𝒔𝒊𝒗𝒆–𝒃𝒖𝒕 𝒅𝒐𝒆𝒔𝒏’𝒕 𝒒𝒖𝒊𝒕𝒆 𝒔𝒄𝒓𝒆𝒂𝒎 “𝑨𝑮𝑰 𝑻𝒐𝒎𝒐𝒓𝒓𝒐𝒘.”
We all know that data contamination is an issue with LLMs and LRMs. We also know that reasoning claims need more careful vetting than “𝘸𝘦 𝘥𝘪𝘥𝘯’𝘵 𝘴𝘦𝘦 𝘵𝘩𝘢𝘵 𝘴𝘱𝘦𝘤𝘪𝘧𝘪𝘤 𝘱𝘳𝘰𝘣𝘭𝘦𝘮 𝘪𝘯𝘴𝘵𝘢𝘯𝘤𝘦 𝘥𝘶𝘳𝘪𝘯𝘨 𝘵𝘳𝘢𝘪𝘯𝘪𝘯𝘨” (see “In vs. Out of Distribution analyses are not that useful for understanding LLM reasoning capabilities” https://lnkd.in/gZ2wBM_F ).
At the very least, this episode further argues for increased vigilance/skepticism on the part of AI research community in how they parse the benchmark claims put out commercial entities."
Big stupid snake oil strikes again.
That o3 does well on frontier math held-out set is impressive, no doubt
I think there is plenty of room for doubt still. elliotglazer on reddit writes:
Epoch’s lead mathematician here. Yes, OAI funded this and has the dataset, which allowed them to evaluate o3 in-house. We haven’t yet independently verified their 25% claim. To do so, we’re currently developing a hold-out dataset and will be able to test their model without them having any prior exposure to these problems.
My personal opinion is that OAI’s score is legit (i.e., they didn’t train on the dataset), and that they have no incentive to lie about internal benchmarking performances. However, we can’t vouch for them until our independent evaluation is complete.
(emphasis mine). So there is good reason to doubt that the “held-out dataset” even exists.
