People are worried about Large Language Models

Plus—what's the value of LLM benchmarks, and what lessons can we learn from Israel's pager attack

People are worried about Large Language Models

A large, influential contingent of Silicon Valley believes that large language models could pose an existential threat to humanity. (A different, less hip contingent also believes LLMs can be harmful, but less existentially so—they worry it can be used by bad actors for misinformation, or can further human biases.) When a new product might have negative externalities one way we as a society deal with this is via regulation. Nobody expects the federal government to take any meaningful action here in the near future; everybody expects the EU to take action but at this point we’ve all written down Europe’s growth prospects to 0 so no one particularly cares. That just leaves California. Over the past year momentum for a CA bill regulating LLMs has steadily built, and we’re now near the finish line.

SB 1047, authored by State Senator Scott Wiener, sets up thresholds in terms of the cost and number of FLOPs needed to train a model and requires models that exceed these thresholds to have “reasonable assurances” (an emergency stop button, safety protocols, incident reporting) that they will avoid “critical harm” (>$500 million in damage). The bill has passed both state houses and is awaiting signature or veto from our governor, Gavin Newsom.

It’s created some unusual bedfellows. Those in favor include Geoffrey Hinton, Elon Musk, Anthropic, and pretty much all of the Democratic state legislators. Those opposed include Meta, Google, OpenAI, a16z, Nancy Pelosi, Ro Khanna, and Fei-Fei Li.

What will Newsom do? He made some comments at Dreamforce recently that signaled potential concerns with the bill, and the market odds of passage dropped from ~50% to ~28% as a result.

Regardless of what Newsom chooses, there does seem to be a decent chance of a scenario that I’ve seen little discussion about: the losing side fighting back via ballot proposition. This, after all, is what Lyft and Uber successfully did with Prop 22 after AB 5 was passed.

If you’re asking yourself: isn’t a subject like AI regulation better suited for the state legislature over a referendum? then you have not yet grasped the insanity that is CA propositions. Voters are expected to weigh-in on regulating dialysis clinics, adult films, and ambulance worker paid breaks. This year’s list of propositions includes a measure that purports to be about prescription drug pricing but is actually a vicious proxy fight between a NIMBY non-profit foundation and the California Apartment Association.

I for one can’t wait to be flooded with ads attempting to convince ordinary Californians to have meaningful opinions on LLM safety.

LLM Benchmarks

Scott Alexander points out an unusual dynamic: AI skeptics will measure model performance on certain benchmarks (Codeforces ELO rating, AIME score, IQ), and point to poor results as evidence that LLMs are dumb/not harmful/do not need regulation. Then OpenAI will release a new model that blows past these thresholds and nobody ever updates their beliefs. Indeed, OpenAI’s o1 model is now so good at passing our existing benchmarks that Scale is putting out a call for extremely hard questions for a new benchmark called “Humanity’s Last Exam”.

And yet—are the skeptics wrong exactly? GPT-4 has been released for a year and a half and it’s decidedly not the case that society has been utterly transformed as a result. There’s been no budge in productivity statistics, for example.

Is it just that technology naturally takes time to find good applications? That’s probably part of it, but there is currently mind-melting amounts of capital being deployed in Silicon Valley searching for revenue opportunities for LLMs; you’d think that we’d have found something beyond coding assistants at this point.

How is it possible to possess a technology that can score a 13 on the AIME and have nothing useful to do with it? Potentially the answer is that human intelligence is just very different from machine intelligence, and benchmarks that are useful in assessing the former are just not as useful for the latter. A TI-84 can perform 5 digit multiplication problems with 100% accuracy, and this puts it well above many humans at this task, but this has little implications for general purpose intelligence. What we’re finding out is that the AIME, SAT, and Olympiads are a lot closer to this type of test than we might have assumed.

As Tim Lee put it when assessing GPT-o1:

I’m quite impressed by the o1 models, but I do want to point out something that all of my examples have in common: they contain all necessary information within the four corners of a relatively short problem statement.

Most problems in the real world aren’t like this. Human workers spend decades accumulating knowledge that makes us more effective at our jobs. Sometimes solving a problem requires remembering facts from a conversation we had, or a research paper we read, months or years ago. Sometimes we’re missing key information and have to figure out what to read or who to talk to in order to get it.

I don’t think OpenAI is close to mastering this kind of problem.

This feels true to me; as a software engineer working at a coding assistant start-up, I work every day with models that accomplish incredible feats of software engineering while still coming nowhere close to automating away my entire job. There’s just more to human work than tests can capture.

So, what would it take to come up with a benchmark that actually measures general, human intelligence; a benchmark that when passed actually signals that models are ready to kick us into a new era of productivity? I’m not sure.

Israel’s Pager Attack

What should the tech industry take away from Israel’s audacious attack on Hezbollah’s pager and walkie-talkie network? Initially it appeared to be the boldest supply chain exploit of all time—Israel must have somehow intercepted the pagers, lined their battery packs with explosives and modified them to explode when receiving a signal.

But later reporting from the New York Times suggests that it was much simpler—rather than compromising an existing supply chain Israel just created their own, establishing a Hungary-based company called B.A.C. Consulting which produced ordinary pagers for ordinary clients and exploding pagers just for Hezbollah. There isn’t much demand for pagers these days, so it’s relatively easy to corner the market.

It has echoes of the FBI’s sting operation in which they created an encrypted messaging app, advertised it to criminal gangs in Australia, and then read all their messages.

On the other hand, it’s very different from the near-miss that was the xz backdoor in which a likely state actor tried to compromise a very popular compression library. To protect against the latter, you have to… audit every dependency and dependency of a dependency in your tech stack. To protect against the former you just have to not use sketchy apps/do a little more due diligence when purchasing niche hardware.

That being said, if Israel needed to pull of a real supply chain poisoning attack I have no doubt they could do it. It’s simply very hard to protect yourself against ridiculously sophisticated state actors. In the words of James Mickens:

Basically, you’re either dealing with Mossad or not-Mossad. If your adversary is not-Mossad, then you’ll probably be fine if you pick a good password and don’t respond to emails from ChEaPestPAiNPi11s@ virus-basket.biz.ru. If your adversary is the Mossad, YOU’RE GONNA DIE AND THERE’S NOTHING THAT YOU CAN DO ABOUT IT. The Mossad is not intimidated by the fact that you employ https://. If the Mossad wants your data, they’re going to use a drone to replace your cellphone with a piece of uranium that’s shaped like a cellphone, and when you die of tumors filled with tumors, they’re going to hold a press conference and say “It wasn’t us” as they wear t-shirts that say “IT WAS DEFINITELY US,” and then they’re going to buy all of your stuff at your estate sale so that they can directly look at the photos of your vacation instead of reading your insipid emails about them.

Hezbollah’s adversary was Mossad, so yeah, they were screwed.