Anthropic opens safety reviews to outside evaluators

Welcome to the Signal from Superpower Daily with Maya and Theo We are your weekly digest of the past week's most important AI development Right and we curate this specifically for founders builders operators and investors Exactly We try to filter out the noise and give you the fundamental shifts driving the industry And today we are previewing a massive lead story We are looking at Anthropic CEO Dario Amodei He is making this urgent call for a slowdown in frontier capabilities Yeah and he is backing it up by opening Anthropic's doors to outside safety reviews So let's dive straight in We are looking at Anthropic giving independent safety

evaluators permanent access to their internal safety work Right and this is not just casual you know corporate oversight No not at all This is employee level access And meanwhile Amodei is calling on the entire industry to slow the pace of capability gains Yeah and this new arrangement at Anthropic it takes effect immediately The independent evaluators get the exact same system permissions as the internal risk teams Which is wild It is And crucially they have the right to publish their findings They can do this without any editorial control from Anthropic I mean that lack of editorial control is just a staggering detail It's incredibly rare in

the tech sector It means the company cannot sanitize the reports before they reach you the public Right Like think about traditional software audits for a second Usually a company hires an outside firm The firm finds vulnerabilities right Yeah And then they give the report privately to the company Exactly The company fixes the issues quietly Then they release a heavily redacted summary But Anthropic is bypassing that entirely They are handing the megaphone directly to the auditors But this immediate action it's just the concrete part of a much broader proposal Amadé laid this out in a very detailed new essay He did He says frontier AI developers

must actively slow the pace of capability gains And they need to use that bot time to improve safety and judgment Which is a huge ask It is He's pushing for common safety standards among democratic nations But he actually goes further He's also calling for coordination with authoritarian states on shared risks Yeah He specifically cited AI developed bioweapons as the prime example of a shared risk Right So we have to unpack the distinction here between Anthropic's immediate actions and their proposed framework Yeah that's crucial The immediate action is the evaluator access That is real That is happening right now inside their servers Exactly But the proposed

framework it's entirely voluntary It depends on competitors deciding to opt in It depends on adversarial governments deciding to opt in Right Which highlights the unresolved operating problem of this entire AI sector There is this massive gap between shared concern and shared agreement Yeah Everyone might be concerned about the risks But getting everyone to agree to hit the brakes is a completely different challenge See I want to push back on the practicality of this entire framework Because I mean Anthropic can absolutely open its own doors They have full control over their own building right Sure They control their servers But they cannot single handedly slow an

industry wide race If Anthropic slows down but others do not what actually happens That is the big question Does this framework actually prevent the risks Amadei is warning about Or does it just change who gets to the finish line first Well that is the prisoner's dilemma of the AI industry You are entirely right If one lab pauses to build better guardrails they risk losing their market position Yeah And they lose talent Exactly They risk losing top engineering talent Engineers want to work on the cutting edge They do not want to sit idle while a rival ships a superior model Right So Amadei is trying to

turn a voluntary company level governance commitment into an industry wide norm He is trying to lead by example But the limitation is severe A single company cannot enforce a speed limit on a global highway Let's look closely at what Amadei is actually warning us about though Because he is not just talking about deepfakes interfering with elections He is not talking about automated spam He is talking about recursive self improvement Right The big one Yeah This is the idea of AI systems creating their own successors He believes this dynamic could move faster than humans can understand Yeah And move faster than we can control Recursive self

improvement is the theoretical tipping point in artificial intelligence It is a core fear driving this entire safety movement For sure Imagine an AI model that is good enough to write the code for a slightly better AI model It acts as an autonomous software engineer Right It finds efficiencies in its own architecture Exactly It rewrites its own neural network Then that new model writes the code for an even better one The loop tightens The speed increases exponentially It's like a factory that builds factory building machines Eventually the human managers cannot even track the blueprints That's a great way to put it Amadei is warning that an

unchecked recursive loop leads to severe real world consequences He specifically points to cyberattacks Oh yeah A self improving AI can pro global infrastructure for zero day vulnerabilities thousands of times a second He points to bioterrorism He points to massive economic disruption Right He is arguing that we need the safety rails installed permanently before the models start building themselves Because once that loop starts humans are no longer the bottleneck Compute power becomes the only bottleneck Right Which makes his call for international coordination incredibly urgent But I have to say it also sounds slightly utopian Utopian is a good word for it He wants democratic governments to

adopt common safety standards That is hard enough Have you ever seen five democratic countries agree on technology regulation quickly Unlikely It takes years Yeah exactly But he also wants coordination with authoritarian states He wants agreements against using AI to develop biological weapons How do you even begin to verify that kind of agreement Well verification is the Achilles heel of any international treaty on software Right It is not like counting nuclear warheads Right You can use satellites to count missile silos Yeah You can send inspectors to uranium enrichment facilities But you cannot do that with code A frontier AI model can be hidden in a server

farm anywhere on the planet It just looks like a data center Amadei knows this But he is making a geopolitical point based on mutual assured destruction Like mutual assured destruction for biology Yes He is saying that certain risks like a synthetic pandemic do not respect borders A bioweapon engineered with AI would threaten authoritarian regimes just as much as democratic ones Because a virus does not ask for a passport Exactly If an unchecked AI helps a rogue actor synthesize a highly contagious pathogen it destabilizes the entire globe He is hoping that this shared existential dread creates a baseline for negotiation So Amadei is basically putting his

own company out there first He is saying look at us We are opening our doors We are inviting independent scrutiny You should do it too It is a bold strategic move If Anthropix submits to permanent independent evaluation it puts immense pressure on their rivals Yeah It asks a silent question to the market It asks the question to enterprise customers Why are the other labs not doing this What are they hiding It turns safety into a competitive wedge So we need to watch how the market actually reacts over the next few quarters Does this immediate transparency measure become an accepted model for others or does it

remain an isolated move Right Competitors might just look at this nod politely and maintain their existing pace They might decide that the commercial cost of slowing down is simply too high That is the exact metric you need to track Do we see other frontier labs announcing similar independent evaluation boards Do we see them granting employee level access to outsiders without editorial control If we do not see those announcements in the next few months then Amadei's proposal remains just an essay The race will continue at its current velocity That wraps up our lead story Well the tension around safety timelines connects directly to our next major

development Yes it really does OpenAI and CEO Sam Altman have made a fascinating financial decision They will not pursue an IPO in 2026 Right Altman stated this explicitly in a recent Fortune interview He actually called a 2026 public offering ill advised Which is quite a phrase It is She cited unresolved work on AI safety alignment and society's ability to handle more capable systems This is a remarkable pivot OpenAI is essentially making safety and alignment work the stated condition for public financing Yeah Think about how strange that is for a technology company of this scale They are not pointing to revenue targets No they are not

pointing to market conditions or interest rates They are pointing to a technology capability calendar Altman is explicitly saying that safety standards are not yet where they should be Right He does not want the pressure of public markets pushing capabilities much further until those standards improve I have to question the motives here just a little bit Is this genuinely about holding the line on safety Or does citing safety just buy the business more time to mature That's a fair question Because an IPO is a brutal punishing process It requires immense financial discipline It requires predictable revenue forecasting Maybe the company just needs a few more years

to stabilize its enterprise product lines Safety makes a much better headline than saying your accounting department is not ready for Wall Street Well it is highly likely to be a mix of both But we must analyze the specific three part readiness test Altman laid out in that interview Right The three parts Yeah He said three things must be ready before an IPO First business readiness Second technical alignment Third the societal context OK The problem is the absolute lack of metrics for two of those three things They're like how do you measure technical alignment How do you measure society's readiness Let's unpack that third point for

a second Society's ability to handle more capable systems What does that actually mean in practice Right Does it mean the global economy needs time to absorb massive job disruptions Does it mean regulators need time to write new copyright laws Does it mean we need a universal basic income framework It likely means all of the above Altman is acknowledging a fundamental truth here Releasing a highly capable autonomous model into an unprepared society is dangerous It is destabilizing Yeah And he explicitly noted in the interview that AI beyond human control is absolutely possible That is a stark admission That is the CEO of the leading AI lab

in the world validating the core fear of the safety movement Yeah If you truly believe your product could slip beyond human control you delay your IPO You figure out the control mechanisms first OpenAI actually has a unique corporate structure that allows them to make this choice Right The capped profit thing Yes They are a capped profit company but they are governed by a nonprofit board Altman highlighted the structure in the interview He said it is the exact vehicle that allows them to prioritize their mission over shareholder timelines That structural detail is so crucial A traditional tech startup would be under immense pressure from venture capitalists

right now Massive pressure Venture capital funds have life cycles They usually run for seven to ten years Right The investors need an exit event to return capital to their limited partners They need an IPO or an acquisition The capped profit structure shields OpenAI from that immediate pressure It shields them from Wall Street Wall Street demands quarterly growth Wall Street demands constant product shipping If you miss a quarter your stock price gets hammered OpenAI is saying they might need to pause at new capability levels They might need to stop shipping core product updates to do more safety work You cannot easily tell public shareholders that you

are pausing your primary revenue driver for a year to work on philosophical alignment Definitely not But the most important limitation here is the lack of a new timeline The decision removes a firm near term deadline It takes 2026 off the table But it establishes no replacement date Right Altman did not commit to 2027 He offered no detailed safeguards He offered no external oversight plans like Anthropic did It is basically a delay without a destination That is what you need to watch next We need to look for concrete evidence that OpenAI's safety and social cases are actually maturing because a delayed financial move is just a

delay It buys time But what do they do at that time We should watch for actual capability pauses If they train GBT 7 and sit on it for two years that validates the safety claim Or if we watch for shared coordination agreements with other labs or governments if we see those concrete steps the safety rationale is validated Right But if we only see financial restructuring and new revenue tiers the rationale looks much weaker Exactly Next up we are looking at some massive predictions about the intersection of computing biology and robotics Yeah this one is wild Arm CEO Rene Haas has made some incredibly bold claims

In a recent BBC interview he predicted AI could help cure cancer in his lifetime Wow Yeah And he also predicted that humanoid robots could see widespread use within five years These predictions really highlight the massive contrast between long range competing ambitions and current physical realities Haas is envisioning a future where compounding compute power simply solves biology He believes that more capable models running on vastly more powerful chips designed by his company will basically unlock the answers to cancer I just love the timeline he chose for the cancer prediction Within my lifetime It is a brilliant executive forecast It really is It sounds incredibly visionary It

generates fantastic headlines but it completely avoids putting a firm date on the calendar Yeah Nobody can hold him to a specific financial quarter for a cancer cure If it happens he looks like a profit If it does not well he just ran out of time It is the ultimate safe prediction for a hardware executive But let's look at the actual science behind that ambition We are currently at the better magnifying glass stage of medical AI Haas is predicting the autodoctor stage Just look at what is actually happening in medicine today The NHS recently announced that AI powered tools helped four million patients receive faster lung

X ray diagnoses in 2026 That is an incredible milestone Faster diagnoses save lives But it is very narrow It is image recognition It is a pattern matching problem The AI looks at millions of healthy lungs It looks at millions of sick lungs It learns to spot the anomaly That is entirely different from discovering a biological cure Curing cancer is a dynamic biological modeling problem It is not static image recognition No not at all Haas admits this limitation himself He acknowledges that current AI systems cannot even model how cancer affects a single DNA marker reliably Yeah the complexity of cellular biology is orders of magnitude beyond

what today's models can handle You have to simulate millions of atomic interactions happening simultaneously within a cell You have to predict how proteins fold You have to predict how molecules bind Exactly That requires an exponential leap in compute power But his second prediction about humanoid robots is much more immediate He said five years for widespread use Which is a firm timeline Right He envisions machines moving flexibly between tasks He imagines a robot making a bed then pivoting immediately to clean a dustbin That flexibility is the holy grail of physical AI We currently have highly capable robots in factories But they are rigid Yeah They do

one highly programmed motion on an assembly line They weld a car door They move a box If you move the box two inches to the left the robot fails We want general purpose robots for the home and the office We want robots that can navigate the messiness of the physical world Right A living room is much harder to navigate than a warehouse You have pets moving around You have furniture in unpredictable places You have varying lighting conditions It requires immense real time processing to navigate that environment safely You need highly advanced inference chips embedded directly in the robots themselves The robot cannot rely on a

cloud connection with high latency to decide if it should step over a child's toy It has to process that locally And you also need massive server farms training the spatial models in the background But there is a massive caveat here too Let's hear it Haas admits this five year vision is currently bottlenecked by severe global chip shortages Arm designs the architecture for many of these advanced chips If Haas is saying shortages are constraining deployment that is a hard physical limit on the five year timeline Absolutely The software might be ready The models might be capable of understanding how to fold laundry But if you cannot

manufacture the specialized silicon at scale the robots stay in the laboratory The physical supply chain dictates the speed of the revolution So we have two distinct things to watch here based on these predictions First can the industry secure the physical chip supply needed for this robot revolution Will new fabrication plants come online fast enough Right Second can AI show measurable progress on increasingly difficult cancer research tasks We need to see steps beyond reading static x rays We need to see AI actually modeling complex biological interactions and predicting clinical trial outcomes Meanwhile we have a major shift in how developers are actually working with AI day

to day Yeah OpenAI has published new official guidance for its Codex model They are urging developers to write leaner more targeted prompts for the new GPT 6 Astra system This sounds like a minor technical update but it reveals a massive evolution in how we manage autonomous AI agents We're moving entirely away from generic handholding Right In the past developers built massive scaffolding around older models like GPT 5 6 Law They would write huge prompt files They would tell the AI every single rule about the entire codebase They forced the model to read everything before doing anything Let me use an analogy here It is like

hiring a brilliant senior developer but every time they want to fix a single typo on the landing page you force them to read the company's entire human resources manual That's agonizing Yeah You force them to read the architectural history of the company They just get bogged down They get confused by all the irrelevant information They stop working You need to just point them directly to the typo That is a perfect analogy OpenAI is saying that old approach is now actively harmful Those massive instruction files consume useful context Every AI model has a context window That is its short term memory If you fill the working

memory with generic rules it forgets the actual task It causes the new Astra model to trigger irrelevant work The model just gets confused by too much generalized information So OpenAI basically told developers to cut back long skill descriptions They told them to stop using blanket document reading rules in their repository files The technical quirk driving this is fascinating actually The new Astra model is highly literal It actually stops earlier than the older GPT 5 6 Sol model if workflows lack an explicit completion state Wow So it just stops Yeah If you do not tell it exactly what finished looks like it just pauses It does

not try to guess your intention So developers need to explicitly define done You cannot just say implement the login feature You have to say implement the login feature then run the local code then inspect the visual output then execute the automated test checks then repair any failures you find in those tests You have to map the entire micro workflow step by step They're also changing how repository guides work Developers use files commonly called Agent S not MD to instruct the AI on how to navigate a project Yeah OpenAI says these files should no longer be mandatory reading assignments Okay They should be conditional maps If

the agent is changing service boundaries direct it to the architecture guide If it is changing a database schema direct it to the database guide Keep the routing layers small and precise Only give the AI the map it needs for that specific journey But here's the most important caveat You are trimming all this obsolete caution You are removing the friction for routine tasks You want the agent to quickly run local tests or fixed typos without reading a novel Yes The hardest part is doing that without deleting the critical approval gates You still need friction for dangerous work You want to move fast but you do not

want a leaner prompt playbook to accidentally authorize a destructive database migration No You do not want the agent pushing experimental code to the live production server without human review You do not want it handling sensitive security credentials autonomously You need to keep the heavy blast doors locked Teams have to audit their repositories very carefully right now They have to distinguish between useless overhead and necessary safety friction They have to rewrite their access controls specifically for AI agents rather than human engineers That is what you need to watch next if you are a developer or engineering manager Can development teams successfully audit their systems Can they

speed up the routine work while keeping those blast doors locked It is a delicate balance of trust and control In other news we have a massive story about trust control and national security The Defense Department has officially shifted 90 of its classified AI work off Anthropic They have set a September 2026 completion target to remove the remaining 10 This is a remarkable breakdown in negotiations The Pentagon is actively removing a leading frontier model from its highly classified environments Projects involving Maven Smart Systems and Palantir have already been migrated away from Anthropic Right This transforms a basic software procurement dispute into a live high stakes test

of multi model military strategy The root of this dispute is contractual limitations It is basically about ethics colliding with military reality Anthropic wanted strict safeguards Yeah They wanted explicit contractual restrictions against using their models for mass surveillance They wanted explicit restrictions against fully autonomous lethal weapons They wanted to ensure their AI was never pulling the trigger without a human actively in the loop But the Pentagon demanded unconstrained use They insisted the software must be available for all lawful purposes under military code They rejected Anthropic's guardrails entirely Right They argued that civilian tech companies cannot dictate the rules of engagement for the United States military Yeah

And then they took a drastic step They slapped Anthropic with a national security supply chain risk designation I want to highlight the deep irony here Anthropic explicitly restricts its models because they want to ensure safety and ethical use They view autonomy and weaponry as a global risk Yeah The Pentagon looks at those exact safety restrictions and declares them a literal supply chain risk Usually a supply chain risk means you found a compromised microchip manufactured by a foreign adversary It means physical sabotage Here the risk is philosophical Right The military cannot depend on a vendor who might suddenly decide a specific mission violates their corporate terms

of service If a tech company revokes software access during an active conflict warfighters die The mission fails Under Secretary of Defense Emil Michael argued this point directly He said relying on a single frontier model is a major military risk anyway If that single provider falls behind the technology curve the military suffers Or in this specific case if the provider imposes usage limits the military is paralyzed So the Pentagon is hedging its bets They're shifting to a multi model architecture The NUMIX heavily features OpenAI Google and XAI It is entirely about vendor diversity If one vendor balks at a specific classified use case or if one

vendor has a prolonged server outage the military can seamlessly pivot to another model in their stack They are building a resilient supplier agnostic architecture for classified work But Anthropic is not walking away quietly They have filed a formal challenge in the D C Circuit Court regarding that blacklist designation Being labeled a national security supply chain risk is terrible for enterprise business Absolutely terrible That case is still pending But the migration is happening regardless The Pentagon is not waiting for a judge to tell them what to do They are ripping the software out now What you should watch next is the calendar Will the Pentagon actually

hit that September migration target Meta's personal AI agent called Muse has hit number two on the overall U S App Store chart SensorTower estimates the app has passed 83 000 iOS downloads in the United States It jumped up from number four just the day before Muse is interesting because it is a true action taking agent It is not just a chatbot It runs on a dedicated virtual machine A virtual machine is essentially a quarantine zone Right If the AI hallucinates and tries to delete files or act maliciously it is only doing it inside a fake disposable computer environment This sandbox allows Muse to securely book

travel send emails and make purchases on your behalf We need context for those download numbers The data only covers iOS in the U S market It completely excludes people using Muse on the desktop web It excludes the massive WhatsApp integration And Android totals are currently unavailable On Google Play it is sitting way down at number 338 in the productivity category It is also a much smaller debut compared to Meta's previous product launches Meta AI reached 108 000 U S downloads in its debut The Threads app generated a massive 4 3 million downloads on its launch day Huge difference Yeah The real test for Muse is

not this initial spike The real test is whether people are actually comfortable giving an AI agent permission to access their live credit cards and personal email accounts Next up Roblox is massively expanding its AI tools They are rolling out their natural language AI game creator called Build to a global audience They're adding desktop access and huge automated asset libraries They also announced plans for prompt based scene generation You will be able to type out a description of a world and Build will generate the 3D environment instantly Wow Yeah They are even planning to introduce NPC characters that can automatically play test your games to find

bugs and exploit physics glitches The biggest shift here is distribution Roblox is breaking out of its walled garden By the end of the year they plan to let creators share their AI generated games via simple browser links They're also planning standalone app distribution for mobile PC and consoles You will not need the main Roblox app to play a Roblox built game And they are changing how creators get paid for this work They are introducing a real money daily payout system called the Roblox Wallet They are partnering with Airwallex for this infrastructure starting with U S creators That's a big deal It is They are building

a full autonomous pipeline You build a game with AI you test it with AI you distribute it everywhere You get paid daily in real currency Meanwhile OpenAI has halted new signups for its 200 a month ChatGPT Pro plan They also halted upgrades to that tier from lower plans The massive compute demand for the new GPT 6 Astra model simply strained their physical infrastructure too much Existing Pro subscribers keep their access but if you cancel your subscription you are locked out until OpenAI lifts the pause The lower tiers like the 100 plan the Basic Plus plan and the developer API access all remain open This is

a stark example of physical compute limits hitting consumer software OpenAI is actively sacrificing new high tier recurring revenue They are doing it solely to protect service reliability for their current premium users The 200 tier offers the highest usage caps It places the highest strain on the system Capping new signups was the most direct way to control capacity while they build more data centers Finally Apple has detailed the durability technology for its first foldable phone the iPhone Duo Pre orders open October 16th The engineering focus is entirely on the mechanical hinge They are using an AI guided manufacturing process on the assembly line The AI analyzes

millions of variations and pairs each 100 component hinge with its mathematically perfect housing counterpart That's wild Then they scan the metal surface with a confocal laser This laser maps the microscopic topography of the material Based on that microscopic scan they use 3D printing to apply up to 25 custom photopolymer layers directly onto the hinge This fills in any microscopic gaps It reduces any residual waviness in the metal Right They also added a nano textured finish and a stiffer polymer cover to the screen itself It is a remarkable production use of AI They are using AI to solve a physical hardware tolerance problem But the unproven

question remains the exact same Does this intense precision manufacturing actually translate into long term durability in the real world Foldables face repeated mechanical stress dirt and friction every single day We will not know if this AI assisted manufacturing works until people have used the Duo for years We are now moving to our three takeaways from the week First the tension between capabilities and control is no longer theoretical We are seeing it play out in real time business decisions Anthropic is voluntarily bringing in permanent evaluators to monitor their internal safety work because they fear recursive self improvement Yeah OpenAI is explicitly delaying its 2026 IPO because

safety and alignment standards are not yet met The major players are acknowledging that they cannot just ship blindly anymore The existential risks are altering corporate governance Second compute and infrastructure are actively dictating business reality We see this with OpenAI having to cap its 200 Pro sign ups just to keep the servers running for existing users Right We also see it with the Arm CEO admitting that the humanoid robot revolution is being bottlenecked by global chip shortages The software models are highly capable but the physical hardware and data centers are struggling to keep up with the processing demand The definition of an agent is rapidly solidifying

around taking action not just generating text Metasmuse is running in a secure virtual machine to book your travel and send your emails autonomously OpenAI is forcing developers to write leaner codex prompts that explicitly define how an agent finishes a complex coding task Real world autonomy is becoming the new baseline for product development Keep an eye on the Pentagon next week The end of September deadline to completely rip Anthropic out of their classified systems is looming We will be watching for any sudden court movement in the DC circuit that could complicate that migration Head over to superpowerdaily com for all the latest updates Thank you for

listening We'll see you next week