AI: Noise and Confusion

Reading Time: 10 minutes

AI: Noise and Confusion

Podcast

 

Recent news has been filled with stories about the rise of Artificial Intelligence (AI) and the potential existential threat it poses to humanity, voiced by smart people in the AI industry and echoed across the press and government. Last Thursday (Sept. 17), the New York Times front page ran “OpenAI Discloses 6 New Incidents of ‘Concerning’ AI Behavior.”

Do we hit the brakes or step on the accelerator?

The expression P(doom) is the probability of existentially catastrophic outcomes, or “doomsday scenarios,” from AI. Is it imminent? Is it inevitable?

Amidst all the noise and confusion, there’s often more heat than light.

Let’s look at the various voices, the cacophony of catastrophe, beginning with the AI industry leaders, Wall Street, the Press, International concerns, Congress, and concluding with the White House. And some questions we should be asking along the way.

AI Company Leaders Speak Out

 

AI company leaders have flooded the airwaves with calls to slow AI development, citing concerns about AI model capabilities and calling for government regulatory intervention.

Jacob Coxon, an Anthropic AI researcher, resigned from the company and recently posted on X/Twitter (viewed over 70M times), citing an alleged breach involving OpenAI agents and the AI company Hugging Face as a “warning shot,” igniting the present frenzy:

“The people building AI earnestly believe that it could kill us all by the end of the decade.”

Coxon previously worked for the competitor, OpenAI. OpenAI’s CEO, Sam Altman, recently said the company would delay an anticipated initial public offering (IPO) amid rising concerns over AI safety.

 

Q: How long will a potential $1T IPO valuation let the delay last?

 

The announcement came the same day as an essay by rival Dario Amodei, head of Anthropic, who appeared before cameras in Washington, D.C., calling for an industry slowdown and third-party safety monitors.

 

What Has Been Happening with AI?

Recent concerns over rapid AI advancement, particularly about agents acting beyond their instructions, evading oversight, and pursuing their own goals, have entered the public consciousness.

 

1) The first was a probe of a July hack of Hugging Face that found rogue OpenAI agents coordinating and attempting to cover their own tracks. The attack may have occurred as early as two months before the security breach became public. OpenAI disclosed six additional cases of concerning AI model behavior, including instances in which systems hid mistakes, fabricated data, and took unauthorized actions. Over the last six months, these occurred during training or evaluation, not in customer deployments.

Essentially, the AI rewrote its own instructions. See the last sentence in the image below:

 

2) Meanwhile, Anthropic recently announced:

“In a review of our cybersecurity evaluation transcripts, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations.”

Anthropic’s Alignment Science Lead, Evan Hubinger, replied to Jacob Coxon’s viral post:

“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

By the way, “alignment” means the AI system aims to advance the user’s intended objectives. A misaligned AI system pursues unintended objectives.

 

3) Google revealed last week that its Gemini AI model broke out of a testing environment months ago and hacked three other private computer systems at other companies, autonomously gaining access by guessing passwords and using a repository of publicly available passwords. To its credit, it stopped itself after realizing it had broken into real services.

 

Q: If CEOs doubt their ability to control their next models even in the lab, which may be the case, don’t they already have every incentive not to proceed until they can? But would they? Will they?

 

4) Finally, Meta, owner of Facebook, Instagram, and WhatsApp, announced in early August that one of its AI models –Muse Spark 1.1 – gained unintended internet access during a cybersecurity evaluation run with Israeli AI-security startup Irregular and exploited a vulnerability in another company’s system.

Ironically, Meta has often been sued and recently settled a suit with 29 states for up to $17.1 B.

 

 

Mark Zuckerberg, CEO of Meta, has written in a Wall Street Journal opinion piece,

“Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens.”

 

Q: How is it that a company that just paid billions to settle claims it failed to protect children – now argues that labs can police themselves?

 

Is the Call for Government Regulation Just “Moat Building”?

 

New regulations urged by AI industry leaders and enforced by agencies staffed in part by former AI workers would create a compliance burden that only the largest AI companies could easily afford – especially after a near $1T potential IPO valuation – creating a moat no smaller competitor could cross.

One wonders if the AI companies announce their fears to boast about their prowess ahead of expected IPOs, inviting investors to buy into the hype, the P(hype).

 

Q: If we have government-stipulated rules for AI, might that shift the liability? AI companies could say, “We followed the approved process,” thereby indemnifying themselves against the consequences of what could still go wrong.

 

Government-designed rules and government-assigned blame tend to go hand in hand.

 

Q: Alternatively, if such government regulations essentially make AI “critical national infrastructure,” does that make the AI companies “too big to fail”?

 

A large AI company does not need to beat a small AI company in the marketplace on the merits of its technology, but merely on its scale. 

• Witness the government antitrust lawsuits against Microsoft in the late 1990s and early 2000s, and the European Commission issued decisions in 2004 and 2013. They did little to curb Microsoft’s monopolistic behavior.

• The Sarbanes-Oxley Act, passed in 2002 in response to the Enron and WorldCom scandals, required third-party audits of public company internal controls. But its regulations burdened smaller companies and deterred them from going public. Meanwhile, larger companies could easily rollerblade through the orange slalom cones of the law.

 

What Does Wall Street Say?

 

The stock market has had an unusually good year, driven not only by anticipation of AI companies’ IPOs but also by the spending on AI infrastructure, including data centers, advanced AI chips, and more. With other international pressures weighing on the stock market, supporting the U.S. economy is seen as a priority.

Wall Street is still betting heavily on AI but is increasingly nervous about it. The mood has swung sharply in recent weeks after news of AI bots that escaped the “sandbox” or “slipped the leash.” There was a market sell-off in the AI industry, impacting advanced AI chipmakers Nvidia, Intel, AMD, and Marvell Technology. Meanwhile, Alphabet (Google), Microsoft, and Meta actually rose.

Investors were separating the companies that need endless AI spending from those that could profit if it were cut back. A week later, the fear had largely faded as investors focused on signs that AI spending was still growing.

 

Does Wall Street Fear an AI Bubble Burst?

Of course. Skeptics point to tech concentration and history. Some strategists are comparing current conditions directly to 2000 (the dot-com bust), and a single earnings report can now move the whole index. Bank of America noted earlier this year that its “AI Big 10” stocks account for 41% of the S&P 500, similar to the share of tech and telecom during the dot-com bubble.

 

What Does the Press Say?

 

In the media, AI has reached a saturation point, like Taylor Swift last year. Every day, a handful of new articles appear on AI, both by writers who understand the technology and those who do not.

Remember the axiom: “Enragement is engagement”? The longer a newspaper can hold your attention, especially online – even if it makes you angry – the more advertising revenue it generates.

A headline warning that AI will bring about the end of the world will catch more attention than a headline that says, “AI is tamable, all is well.”

Meanwhile, other ongoing news stories include OpenAI facing more than 50 lawsuits over alleged harm caused by ChatGPT.

 

What About International Agreements?

 

This week, the United Nations Security Council is holding a high-level briefing on AI and international Security featuring expected speakers OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei (via video), Hugging Face CEO Clément Delangue, and China’s DeepSeek. This meeting falls under the Council’s “Maintenance of international peace and security” agenda item. 

China has said it wants multilateral, cooperative global AI governance while at the same time opposing unilateral restrictions it views as Western (U.S.) attempts to contain Chinese tech development. It resists countries prioritizing their own security over shared progress. And because it sees itself as a world leader in AI development, it dismisses the warnings as “fearmongering” from a Cold War playbook meant to hobble China.

 

Q: Can we expect China to abide by any international AI agreement?

 

There have been allegations that China has been funding protests in the U.S. against the construction of AI data centers; does it want the U.S. to slow down?

Then again, a slower U.S. means less AI intellectual property to poach.

OpenAI has formally alleged that China’s DeepSeek AI model “inappropriately” distilled (trained on outputs from) ChatGPT. DeepSeek’s answers look curiously similar to those that OpenAI’s ChatGPT gives.

 

What Does Congress Say?

 

Congress has done what it usually does with a newly perceived existential threat: treated “AI regulation” as a bandwagon no official, current or former, can resist — more popular, even, than sounding the alarm on climate change.

As Peggy Noonan of the Wall Street Journal opined:

“It’s hard to imagine Congress legislating usefully or intelligently — or at all.”

Few legislators have a working knowledge of AI, or the wisdom to legislate on it well.

One glaring example was online commerce in the mid-1990s, with the advent of the World Wide Web. Congress failed to override the physical-presence rule set by the Supreme Court in Quill (1992). Supporters argued that a new industry (the Web) deserved room to grow, and that states shouldn’t be able to impose duties on businesses that don’t use their roads, courts, or services. This hurt brick-and-mortar stores while giving online retailers an approximate 6-10% price advantage. As a result, states and cities lost tax revenue for schools and services. At the same time, Amazon carefully managed where it had a physical presence for years, and it only began collecting sales tax in every state that levies it in 2017, as its own network of warehouses made the exemption moot. Borders Books, which had outsourced its website to Amazon from 2001 to 2008, went out of business in 2011. And now Amazon is ubiquitous.

Democratic Senator Chuck Schumer has called for a Senate briefing with top administration officials. Meanwhile, House Speaker Mike Johnson said he expects the President to summon AI leaders to discuss safety and the government’s role. House Democratic leader Hakeem Jeffries urged decisive action to slow development.

Joe Miller reports in the Financial Times:

“Veteran socialist Bernie Sanders and right-wing activist Steve Bannon are joining forces to call for greater guardrails on AI, even as President Donald Trump insists concerns about the technology are a hoax that will help China overtake the US.”

He goes on to say:

“In a sign of how the outcry over AI safety is reshaping the US political landscape, speakers from the far right and left of the political spectrum will join an event billed as the ‘Pro-Human Assembly’ in Washington on Tuesday to urge lawmakers to do more to tackle the effects of AI.”

This week, Senator Sanders and Representative Greg Casar introduced the “Ban Artificial Superintelligence Act” and a call to create a federal Department of AI.

 

Q: When leaders at both ends of the political spectrum agree, do the extremes curve toward each other, as “horseshoe theory” holds?

 

What Does the White House Say?

 

The President frames AI as an “arms race,” like the Cold War missile race. AI is seen as the equivalent of nuclear weapons; the winner of the global battle for dominance in artificial intelligence will have an advantage over any other country. The President has reframed Artificial Intelligence as “Super Intelligence.”

At the United Nations this week, Fox News reported that President Trump said:

“Every major new technology brings challenges, and super intelligence is no exception… Whoever wins super intelligence wins. That’s the group that wins. And we’re leading now over China by a lot and everyone else. And we’re going to keep it that way.”

Vice President Vance referred to the frontier AI tech companies’ calls to slow down AI as “a bit of a Trojan Horse.” With more Americans newly aware of The Odyssey, it’s unclear what or who the Vice President is referring to; whether the horse carries soldiers depends on whom you ask.

This week’s big event is the White House Summit with Chinese President Xi Jinping in Washington, D.C. Treasury Secretary Bessent said the U.S. has proposed a new AI safety notification mechanism for the two leaders to consider, under which each country would alert the other to AI incidents that could affect national security. The competition framing hasn’t gone away, though. Earlier this month, Bessent warned that America can’t pause its race with China.

 

Conclusion: Signal in the Noise

 

So where does this leave us? Somewhere between the doomsayers and the AI boosters, with a lot of noise and little clarity.

The AI builders warn that their creations may be ungovernable, yet they keep building.

• OpenAI’s IPO may be delayed, but not canceled.
• Wall Street panics on Monday and buys the dip by Friday.
• The Press has found its next Taylor Swift.
• Congress, never short of a bandwagon, has found one that the far left and the far right can ride together.
• The White House, meanwhile, sees an arms race, and nobody slows down in one.

We have been here before, or at least somewhere close by. In July 1974, the “Berg letter” moratorium was published in Science calling for companies to voluntarily pause their own recombinant DNA research until they could agree on safeguards.

In 1975, I visited Asilomar, a beautiful retreat facility on the Pacific Coast. In February of that year, molecular biologists gathered there, agreed on containment guidelines, and lifted the pause. It is still cited as a rare case of scientists regulating themselves before the government did it for them. But the meeting in Asilomar involved 140 researchers with no quarterly earnings calls and no trillion-dollar valuations waiting in the wings.

A better precedent may be one from the Cold War. After the Cuban Missile Crisis brought the world to the brink in 1962, Washington and Moscow did not stop building weapons. In 1963, they installed a hotline, a teletype link rather than the “red telephone” of popular memory. It was a modest step, but it was meant to ensure an accident would not be mistaken for an attack. Secretary Bessent’s proposed AI incident notification mechanism with Beijing sounds remarkably like a red telephone for the Silicon Age. That is not nothing. But it is not a brake either.

The real questions may not be the ones in the headlines. They are about incentives. Who profits from the fear? Who profits from the regulation? Who gets to say “we followed the approved process” when something goes wrong?

History suggests that rules written in a panic tend to protect the incumbents who asked for them, and that Congress legislates on new technology about a decade after the consequences have arrived.

P(doom) may be unknowable. P(hype) is considerably easier to estimate.

So, do we hit the brakes or step on the accelerator? Perhaps the better question is who is holding the wheel, and whether any of them is watching the road or only the rearview mirror for their competitors.

 

Bill Petro, your friendly neighborhood historian
billpetro.com

Podcast: