The Miami-based AI startup Subquadratic came out of stealth mode last month with a huge claim. It announced that it had solved a mathematical bottleneck that had been holding back large language models for almost a decade.
The details were thin, and many people were unconvinced. But Subquadratic has started to bring the receipts, sharing the results of an independent evaluation of its new tech. The results suggest that the company’s claims might be worth paying attention to.
According to Subquadratic, it has developed a new kind of LLM, called SubQ, that is faster and cheaper and uses a lot less energy than any other model on the market. The company also claims that SubQ is able to process up to 12 times as much text at once as most other models, allowing it to carry out a range of data-heavy tasks, such as analyzing hundreds of documents or entire code bases.
What’s more, Subquadratic says, SubQ does this while more or less matching the performance of the best models put out by Google DeepMind, OpenAI, and Anthropic on key tasks like coding.
The problem was that the company at first provided little evidence for its claims beyond a handful of self-published test scores. And it has yet to make SubQ widely available for people to try out themselves.
So it’s no surprise that Subquadratic’s claims were met with skepticism. Dan McAteer, an artificial-intelligence engineer, captured the overall response on X: “SubQ is either the biggest breakthrough since the Transformer … or it’s AI Theranos.”
“We expected healthy skepticism,” says Subquadratic cofounder and chief technology officer Alex Whedon. “In hindsight, releasing the third-party benchmarks alongside the initial announcement would have preempted much of the skepticism, which is why we’re taking the time to make sure any future results are fully verified before putting them out.”
Subquadratic asked Appen, which evaluates other companies’ models, to run its tests on SubQ. The results seem to back up a lot of Subquadratic’s claims. “That was really exciting to me, it validated their architecture,” says Jeanine Sinanan-Singh, Appen’s director of generative AI research.
“I was like, ‘Wow, this could be a game changer,’ because models struggle with speed and inefficiency,” she adds. “But when you have kind of shocking results, it’s really not as credible when you say it yourself.”
SubQ won’t replace existing top models across the board, but it could offer huge increases in speed at a fraction of the typical cost for certain tasks. Subquadratic insists that in the long run, though, its breakthrough could change how LLMs are built. “We hope we’re kicking off a new age of efficiency,” says Justin Dangel, the firm’s cofounder and CEO. “We don’t think anybody will be building on transformers in a few years.”
Attention!
To understand why Subquadratic’s claims are a big deal, let’s dig into how most LLMs work. The key mechanism inside an LLM is a type of neural network called a transformer, which runs a process known as dense attention. Today’s LLMs typically chain together multiple transformers. (The foundational paper of the LLM era, published by researchers at Google in 2017, was titled “Attention Is All You Need.”)
Dense attention works like this: When a transformer processes a chunk of text, it first encodes each word (or part of a word, known as a token) with a number. To capture the meaning of the full text, it then multiplies each of those numbers with every other number for that text. For example, a piece of text 10,000 words long would kick off almost 50 million individual multiplications. That’s a lot of computation and the main reason that LLMs are notorious power hogs.
“If you want to summarize The Great Gatsby, you have to look at the first word and the last word together, and then you have to look at every other combination,” says Dangel.
As the length of the text increases, the number of computations skyrockets. That’s because each additional number must be multiplied by all other previous numbers. Double the number of words, and you roughly quadruple the number of computations, a rate of increase known as a quadratic expansion.
(You can picture this yourself: Draw a circle and mark dots around its edge. Each dot is a token. Then draw lines between pairs of dots to represent the multiplication of those two tokens. A circle with five dots will have 10 lines crossing it. Make it 10 dots and you will have 45 lines, 20 dots and you will have 190 lines, and so on.)
Slashing costs
Subquadratic’s solution is to ditch dense attention, the core operation of a transformer, in favor of what’s known as sparse attention, which slashes the number of computations needed. Instead of multiplying the number assigned to each token by every other number, sparse attention selects just some of the numbers to multiply. The idea is that not all relationships between words in a piece of text matter.
“Sparse attention says not all of those relationships are important, because they’re not,” says Whedon. “If you’re reading a book, you’re not going to look at the first and second words, first and third—that’s insane.”
It’s a simple approach, and Subquadratic is not the first to try it. “Pretty much everything under the sun has been attempted,” says Will Depue, an independent AI researcher who previously worked at OpenAI. “It’s not impossible, but it’s akin to running a four-minute mile.”
Previous techniques for selecting which numbers to multiply and which to ignore have not produced a mechanism that can capture the meaning of a document as well as dense attention can.
Subquadratic claims to have cracked the problem at last. It pitches SubQ as the first sparse-attention LLM that rivals mainstream dense-attention models in performance.
“Historically, most mechanisms have used fixed patterns, like always comparing the first word to the fifth,” says Whedon. “That’s pretty limiting. Language is too sophisticated for that. And so, one of the things that makes our mechanism unique is that we dynamically select which ones are important.”
The firm won’t say exactly how SubQ chooses which words to focus on, but the selection is calculated on the fly and differs for each piece of text the model is given. “That’s kind of where the secret sauce is,” says Whedon.
Testing, testing
The upshot is that for certain tasks, SubQ may be faster and cheaper to run than most other models. Appen evaluated SubQ on a handful of standard tests. In a straight-up speed test, which sets a baseline for how fast a model can operate in theory rather than assessing what a model can actually do, Appen found that SubQ was 56 times faster than models using FlashAttention, a previous sparse-attention technique.
On LiveCodeBench, a test that looks at how well models perform on competitive coding problems taken from real contests, SubQ scored 89.7%, putting it in the same ballpark as other top coding models. “This model continues to provide frontier-level performance in coding,” says Appen’s Sinanan-Singh.
Subquadratic’s claims about cost are harder to verify because SubQ is not yet widely available. According to Dangel, it costs $2,600 to run Anthropic’s LLM Opus 4.6 through RULER 128, a test developed by Nvidia to assess a model’s ability to retrieve information from large data sets. And SubQ? “It cost us eight dollars,” he says.
SubQ does seem to be able to handle a lot of text at once. The model has a context window (roughly akin to a working memory) up to 12 million tokens long. Most top models today have context windows one million tokens long. In a demo that Whedon ran for me, he asked SubQ to perform a task that required it to reason about information contained in 400 documents. It responded in seconds. When he gave Perplexity—a popular LLM-powered search engine—the same task, it failed to load all 400 documents.
Appen put SubQ through the Needle-in-a-Haystack test, which, like RULER, assesses how well a model retrieves specific information buried in a large data set. In its report, Appen states that Subquadratic’s model scored 98% with context windows six million and 12 million tokens long, “sustaining near-perfect long-context retrieval at scales few models are tested at.”
Too good to be true?
Despite the high scores, benchmarks paint an incomplete picture of what a model can and cannot do. Testing under very specific conditions is not a substitute for running a model on a wide range of real tasks.
Subquadratic is offering SubQ as a model tailored to coding and to searching very large data sets. It says that tens of thousands of potential users have already signed up for early access, including more than 500 enterprise customers. But there’s a long waitlist, and the firm has given very few people access so far. Subquadratic’s response is that it is a new, small company with limited resources and cannot serve too many people at once.
Until more people get their hands on the model and try it out for themselves, some skepticism is justified. One nagging issue is that Subquadratic reused the weights (values set within a model during training that determine how it will behave) from a version of the Chinese open-source model Qwen to bootstrap SubQ, rather than training it from scratch. That’s a common approach for model makers to take, but it cuts across Subquadratic’s claim that it has fully reinvented how LLMs work.
“They may have built something real and useful,” says Depue. “But the public evidence does not yet justify the stronger claim that they have solved the quadratic attention bottleneck.”
In the meantime, Subquadratic cofounder Whedon insists that making something different was his only option. If you want to build a competitive model, you have to have new ideas, he says: “We’re more up against it than OpenAI is.”
At the end of a tense and scoreless first half of a soccer match between the English men’s team and rival Germany, millions of Brits let out a collective sigh and did what they so often do in moments of stress: They made tea. That wave of electric kettles clicking on, however, caused a different kind of stress: a huge and sudden increase in demand for electricity. But National Grid, which operates the local transmission network, was ready.
Just as those kettles started heating up, an AI program sent instructions to a data center in London to slow down some of the facility’s power-hungry chips. This reduction helped make sure there was enough supply to match demand, staving off potential blackouts or damage to electrical hardware. For data centers, which normally guzzle power without consideration for anyone or anything else’s needs, it was a radical departure.
It was also a test. The software was controlling a real data center, but there was no game happening at the time, in December 2025. Engineers were doing a trial run for a new breed of data center built to be flexible about its electricity needs, so they re-created the energy demand facing the UK’s grid during a match from the 2020 Euro tournament. They wanted to see how their software, called Conductor, would have responded had it been online at the time.
Conductor is the signature product of Emerald AI, a firm based in Washington, DC, that’s part of a wave of companies trying to figure out whether data centers can work within the confines of the existing electric grid.
This year, Emerald is set to deploy Conductor in a new facility in the part of Virginia known as Data Center Alley, this time connected to the live grid. When overall demand spikes, Conductor will turn down the power used by the data center, while making sure its servers still carry out their timeliest and most important jobs. Emerald’s partners on the project—which include Nvidia and the giant data-center operator Digital Realty—bill it as one of the world’s first “power-flexible AI factories.”
Demonstrating that data centers can participate in this kind of give-and-take could ease what many tech leaders identify as the bottleneck in getting facilities online: It takes far longer to get approval for, construct, and connect new power plants than to build data centers. PJM, the grid operator in Virginia and the largest one in the US, for instance, needs eight years to bring new generation online, according to RMI, an energy research and advocacy group. “We need to solve the energy equation,” says Josh Parker, head of sustainability at Nvidia. “AI factory flexibility is the bridge between the incredible demand for AI and the immediate limitations of our energy grid.”
Speed, though, is only one of the issues. Once facilities do plug in, neighbors often criticize them for drawing too much electricity and contributing to rising prices. They say the data centers generate more noise than they do long-term jobs, contribute to pollution, and threaten to put people out of work. Organizers stalled over $150 billion worth of projects in 2025, according to Data Center Watch, and policymakers alert to the public mood are starting to impose limitations on development.
More than a dozen states are considering bans, and local moratoriums are in effect in places like Minneapolis and DeKalb County in Georgia. At the federal level, the GRID Act, a bipartisan bill in the US Senate, proposes to sever new data centers from public grids entirely. Some operators are already moving that way by trying to develop their own power generation.
Rather than rushing to build new power plants, companies could find part of the solution to the crunch right under our noses—or, more precisely, in the transmission lines under our feet and above our heads. The existing system operates near its full capacity during only a small number of high-demand hours throughout the year. This means, some grid experts argue, that if data centers can limit the power they draw during those stretches, they won’t need to wait for big infrastructure upgrades or build their own off-grid generation.
Indeed, a growing number of studies have shown there could be plenty of power available for data centers that can flex. A widely discussed 2025 report from researchers at Duke University found that the US grid could offer an additional 76 gigawatts—about 5% of its entire capacity, and about enough to accommodate projected data-center growth in the US through 2030—to facilities that are willing to reduce their usage just 0.25% of the time. That’s about 22 hours a year. And when researchers from Princeton University and two grid-modernization companies looked at locations for new data centers in the PJM region, their report, which was funded by Google, found that a 500-megawatt facility capable of flexing for less than 1% of the year could reach full operation three to five years faster than one that’s inflexible.
Flexible power connections could also help data centers address some of their PR problems. By decreasing their draw at times of grid stress, for instance, they could avoid diverting power from where it’s most needed, thus boosting stability. By using existing capacity, they might be able to reduce the need for new fossil-fuel power plants and spread fixed costs over more electricity users, pushing prices down.
The AI power pinch is attracting resources and research into strategies for grid flexibility overall, which could help negotiate a tricky period: Taken together with electric vehicles, air-conditioning, and other sectors, data centers are helping drive what analysts predict will be a 25% increase in US electricity demand by 2030 compared with 2023 levels.
Ideally, flexibility gives grid operators more control over the flow of electrons, making them leaders of a harmonious ensemble rather than hostages to inflexible electricity requirements. That will help them manage demand spikes across the entire system and deal more effectively with the intermittent nature of renewables like wind and solar. “Demand flexibility is incredibly useful for power grids,” says Johanna Mathieu, a grid expert at the University of Michigan. “It helps reduce electricity costs and improve grid reliability.”
But while advocates see plenty of benefits, the concept brings complexity. For data centers, compromising on energy needs can be a hard sell. Flexibility requires utilities and grid operators, which tend to be operationally conservative, to change long-held practices. And some skeptics also say that flexibility distracts from the very real need to build more grid infrastructure faster, and could even pose risks to our electricity supply.
Still, some technologists, grid operators, and utilities are hoping to show that flexibility works—not only in white papers or simulations but in real life.
The poster children for data-center growth default toward inflexibility. Hyperscalers like Microsoft and Oracle have proposed enormous new centers, many of which would rely on off-grid, natural-gas-burning power plants. When xAI wanted to speed up the buildout of the Colossus site outside Memphis, Tennessee, it rolled up with gas turbines on flatbed trucks. The facility, now in operation, is facing blowback from regulators and residents about the spike it’s causing in emissions and other pollution. In any case, there aren’t enough gas turbines worldwide to meet the demand from data-center operators.
One big obstacle for anyone demanding a lot of power is that our grids are mostly rigid. They’re designed to supply enough power to meet total demand when it’s highest, even if that’s for only a relatively small number of hours a year. That conservative approach is a simple route to reliability, but it means that the grid has quite a bit of headroom. “The grid is already overbuilt by a lot. If you were an airline running at 30% utilization, you would not buy more planes,” says Amit Narayan, the cofounder and CEO of GridCare, a company developing flexibility technologies, referring to a 2025 Stanford study of transmission lines in western North America. “If you are running a grid at 30% utilization, there’s no scientific reason you can’t go to 60.”
“If you were an airline running at 30% utilization, you would not buy more planes. If you are running a grid at 30% utilization, there’s no scientific reason you can’t go to 60.”
To be fair, the idea of flexibility isn’t entirely foreign to grid operators. For decades, they’ve practiced a technique called demand response: When it looks as if demand will get too close to supply, as it might during a heat wave when many people turn on the AC at the same time, they call large commercial or industrial facilities and ask them to shut down parts of their operations. This method can help avoid the need to fire up so-called peaker plants, which run on fossil fuels, but it’s slow, imprecise, and hard to scale.
In the 2000s, as the adoption of technologies like electric cars and solar panels presented new challenges, more internet-connected grids also provided new means of flexibility. Virtual power plants, or VPPs, offered a smarter, faster, more granular alternative. Electricity customers ranging from factories to homeowners with smart thermostats, solar panels, or big batteries would allow the utility to adjust their draw to help meet demand—often getting paid for their (frequently unnoticed) trouble.
After the generative AI boom began with the release of ChatGPT in 2022, some companies began to see flexibility as a way to get data centers set up more easily, efficiently, and affordably. If they bring AI money into existing grids and reduce or defer the need for expensive upgrades, data centers could actually help spread out fixed costs so as to lower rates for other users. A study from Duke University published this past February, for instance, found that flexibility could reduce rates by 0.5% to 2.8%.
PETRA PÉTERFFY
The trick is figuring out how data centers, notorious power hogs, can keep operating when their flexible connections are throttled. Flexibility specialists envision three possible ways. The simplest is for the new data center to install on-site backup power storage or generation to tap when the grid is maxed out—at their own expense, of course.
A facility could also fill the gap by drawing on a VPP. The utility would turn down the electricity going to users who signed up for the VPP, and the data center would pay them for their flexibility. This method wouldn’t require any major infrastructure, but it would require the utility to have a big VPP program and to coordinate the exchange at a time when the grid was under stress. While VPPs exist to some extent in nearly 40 states, the rules governing them vary widely, and they are empowered to do more in some areas than in others.
Finally, a data center could simply use less power at peak times. The conventional wisdom is that they won’t go for such limits, particularly when every number-crunching server can feel like a goose potentially laying little golden eggs. But some experts are betting that the value of getting up and running quickly is enough to change their minds. “There is a clear and growing trend,” says Ayse Coskun, chief scientist at Emerald AI. “Operators are increasingly willing to trade some level of flexibility for faster grid interconnection.”
GridCare, a startup based in Silicon Valley, was one of the first companies to use flexibility to get data centers online quickly. Instead of looking at grids only in worst-case scenarios when electricity demand is highest, the company analyzes the system under all conditions, explains CEO Narayan, who studied smart grids at Stanford. It feeds every part of the grid—including power plants, lines, substations, and homes—into a generative AI model that creates a “digital twin” for different grid configurations. It then picks out results that could unlock capacity while maintaining reliability, and it feeds those into another model trained on the physics of electrical components like resistors and capacitors to make sure they’re realistic.
GridCare found its first customer in the Silicon Forest, an area in the Pacific Northwest named for the trees that dominate the landscape and the IT industry that has more recently sprouted up there. The local grid needed more capacity to support more data centers. “Data centers wanted ‘speed to power,’” says Isaac Barrow, a manager of data-center relations at Portland General Electric, or PGE, the local power generator and distributor, “but transmission buildout is a long process that’s very costly.”
In 2024, Aligned Data Centers came to PGE wanting to expand its operation in Hillsboro, Oregon, and PGE followed a recommendation from GridCare. Aligned will install a 31-megawatt battery, set to be in service in May 2027, and decrease its draw by up to that amount when the grid becomes congested. Bundled with other flexibility measures, that battery has allowed PGE to increase the capacity it can offer Aligned and other nearby operators by 80 megawatts without any new power plants. Though the buildout of data centers in Hillsboro has faced plenty of pushback from locals, Barrow points out that it could have the knock-on effect of lowering costs for ratepayers, because it spreads out the tab.
Other companies are promoting different flavors of flexibility. Google has been moving processing loads from facilities in areas experiencing demand spikes to those in less stressed spots since 2023. It’s signed agreements with five utilities, including the Tennessee Valley Authority and Indiana Michigan Power, that add as much as a gigawatt of flexibility.
Voltus, a major VPP provider across the US and Canada, markets a “bring your own capacity” program in which a data-center company can fund a VPP nearby. The grid operator can use the VPP to decrease demand at busy times, and participants get a financial thank-you. “We can spin up new VPPs on the order of months,” says Emily Orvis, Voltus’s vice president of energy markets. In June, the company signed their first such data-center deal: a three-year plan in which Google will bankroll a VPP in the PJM interconnection.
Of all the approaches to flexibility, Emerald AI’s may be the most ambitious: asking data centers to dial into the grid’s needs. The company’s Conductor software, which can run on premises or in the cloud, builds on the research of chief scientist Coskun. Her group at Boston University showed in a pair of 2013 papers that a data center could watch the grid and help balance big power fluctuations, such as the intermittent effects of solar and wind power. By 2022, she and her colleagues had tested their methods on a cluster of 36 research servers and shown that the system could respect power limits without breaking the processes it was running.
One of the most important questions for Conductor is deciding which AI processes can be slowed down to save energy without kneecapping performance. A lot of companies label their jobs by priority—a real-time chatbot query, for instance, might outrank something like a web search that’s part of a deep research project. When they don’t, Emerald AI tries to infer priority from the nature of the job. Conductor then analyzes the AI workload to determine how tweaking the power to a given processor will affect the performance and help meet the usage limits set by the grid operator.
“The performance curve changes for different kinds of workloads,” says Coskun. “Each AI job is going to have a different location on that curve. Our intelligence is figuring out where you are on that curve.”
PETRA PÉTERFFY
Last year, Emerald AI began assessing the technology’s readiness for real-world use in a series of tests, raising the difficulty each time. The trials were carried out in partnership with the Data Center Flexible Load Initiative—a collaboration among tech companies like Google and Nvidia, utilities like Duke Energy, and grid operators like PJM that aims to help establish a repeatable framework for power-flexible data centers.
The first challenge was in Phoenix, a fast-growing computing hub. For the test, Conductor took control of a group of server racks laden with 256 Nvidia A100 GPUs—hardware that can use about as much power as around 170 US homes. When presented with a simulation of a busy grid, Conductor reduced the power to the chips by 25% for three hours, while maintaining acceptable computing performance. Emerald AI and its partners reported the results in a paper in Nature Energy in December 2025.
The next trial forced the system to juggle surprise grid fluctuations without advance warning and redirect AI jobs from a data center in Virginia to a less busy one in Chicago. Then, in London, Conductor took the reins of equipment beyond the main GPU processors and faced a more complicated mix of fluctuations, including very short and long bouts of congestion—plus the notorious teakettle effect.
The progress so far shows that flexibility can work, at least in some situations, but only a small fraction of operators have pursued it as yet. “We’re just in the beginning innings of the game,” says Jesse Jenkins, one of the authors of the 2025 Princeton study and cofounder of Firma, a startup that works on data-center flexibility. “People are recognizing that this is a potential solution. The motivation is there; there are some bespoke examples. But there’s no uniform solution set that’s the default option, which is where we need to get.”
While data centers are going up across the US, no place on Earth comes close to the accumulated computing muscle in Northern Virginia’s Data Center Alley. The region is home to around 500 compute-crunching facilities, which represent 13% of the entire world’s capacity; the next two hot spots, Beijing and Oregon, contain 6% each.
There are proposals to build hundreds more facilities in Virginia, but a government study found that the state’s electricity demand will increase 183% (around 26 gigawatts) by 2040 if they all go forward, and supporting even half would be difficult. The power-flexible data center that Emerald AI, Nvidia, Digital Realty, and their partners are building in the suburb of Manassas could demonstrate how data centers can squeeze the power they need out of existing capacity. The facility, slated to come online later this year, is intended to give Conductor the chance to manage power at the largest scale yet and to respond to conditions on a live grid for the first time. In the UK demonstration, Conductor managed a 130-kilowatt AI cluster; in Manassas, it will pull the strings of a 96-megawatt hyperscale AI factory.
Some degree of flex will play a key role as we transition away from fossil fuels and toward a future that has to juggle technologies like solar and wind power, batteries, and electric cars.
For PJM, the Manassas facility points to a potential path through the current power crunch. “We think data-center flexibility, in different forms, will be essential for the reliable integration of data-center load over the short to mid term,” says Scott Baker, who manages demand-side markets at PJM.
But not all grid experts are so sanguine. PJM’s market monitor, which oversees the grid operator, says there are no workarounds when it comes to adding capacity. “The notion that large amounts of data-center load can be added without adding new generation is magical thinking,” says Joseph Bowring, an economist and the head of PJM’s market monitor since 1999.
One problem, he says, is that there’s no way to guarantee that a data center will actually take less power when demand is high. That is, absent any legal or regulatory push for flexibility or compliance, the utility won’t be able to step in to help prevent, say, a blackout. Utilities can rely on resources like power plants, but they can’t control or rely on data centers. “They do not want to be fully interruptible,” Bowring says of the facilities.
Stephen Empedocles, an advisor for technology companies, views flexibility as more of a tool than a silver bullet. “These approaches are excellent for improving grid reliability and getting more out of the infrastructure we already have,” he says, “but they are optimization tools.” They’re not substitutes for the “generation, transmission, and distribution expansion that will still be required,” he continues.
Flexibility advocates agree that over the long term, whether or not AI continues to boom, electrification will drive a need for more generation and transmission. Some degree of flex will play a key role in using grid infrastructure better as we transition away from fossil fuels and toward a future that has to juggle technologies like solar and wind power, batteries, and electric cars. A report published by the International Renewable Energy Agency in January 2026 found that grids around the world will need three times as much flexibility in 2030 as they had in 2019—and 10 times as much by 2050—to balance increasing demand with fluctuating supplies of renewable energy.
The challenge of powering AI could provide just the spark we need to do the work of designing and building smarter, more flexible grids, says Coskun. “I think with a crisis like this, there’s no quick solution,” she says. “Sometimes a crisis like this creates an opportunity to do something differently.”
Amos Zeeberg is a freelance science and technology journalist based in Bucharest. He’s developing a book about technology networks, including electric grids.
This story was updated on June 20, 2026 to clarify details about Emerald AI’s test in London in 2025.
This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.
When I landed in Seoul after a grueling 12-hour flight from San Francisco, I walked through an unmanned immigration checkpoint, where a machine scanned my face and passport. On the subway home, people were glued to their phones (powered by flawless 5G even underground), as we raced past platforms lined with LED screens of ads celebrating K-pop idols’ birthdays. When I got off the station in Gangnam, a cartoon-eyed robot on wheels was waiting patiently at a crosswalk to deliver someone’s dinner. Internet cafés dotted the sidewalks, crammed with teenagers playing computer games, maybe hoping to become the next legendary pro gamer.
I stood at a bus stop with interactive touch screens showing real-time bus schedule updates. It will soon become an “AI bus stop,” the Gangnam district announced in June, with a kiosk that answers riders’ questions in multiple languages. The news didn’t surprise me. Having grown up in the city, I’ve watched Seoul transform from a scrappy boomtown into the gleaming tech capital it is today.
South Korea loves AI.
While a public backlash against AI is brewing across the US, South Koreans are optimistic. Only 16% say they are more concerned than excited about AI—the lowest of any of the 25 countries surveyed by the Pew Research Center—while 50% of Americans were more worried than excited. A majority of Koreans use AI every day, either as a sort of personal assistant or to do tasks at work, according to surveys by the Ministry of Culture, Sports, and Tourism and Korea Chamber of Commerce and Industry.
One of the most wired countries in the world, South Korea loves to street-test every new technology on the block—AI webcomics, virtual K-pop idols, and humanoid monks. And the appetite for experimentation doesn’t stop with ordinary citizens. Government agencies are early adopters too, deploying AI textbooks in schools and AI eldercare robots in welfare centers. South Koreans share a deep conviction that embracing technology is integral to modernizing the country and cementing its place in the global order. Their fascination with AI is just the latest incarnation of that ethos—and it’s making them anxious to stay ahead.
Engineered enthusiasm
All this techno-optimism has largely been engineered by South Korea’s national agenda to make AI a motor of economic growth. “The South Korean government has designated an AI-powered Fourth Industrial Revolution as the country’s path forward and aggressively promoted and invested in it,” says Chihyung Jeon, a professor of science and technology policy at the Korea Advanced Institute of Science and Technology. “South Koreans have consistently and relentlessly been told by the government about AI’s potential to create a better future.”
As South Korea rose from the ashes of the Korean War, technology lifted the nation from poverty into an economic powerhouse. In the 1970s, South Korea manufactured steel and ships, then semiconductors in the 1980s, broadband in the 1990s, and smartphones in the 2000s. Today, Samsung and SK Hynix supply most of the world’s high-bandwidth memory chips, which power the cutting-edge Nvidia hardware used to train AI models. South Korea’s economy now orbits these two semiconductor giants: The country’s main equity index, Kospi, surged to record highs in 2026, powered by the soaring share prices of both companies, each valued above $1 trillion.
Lee Jae-myung, president of South Korea, has pledged to vault the country into the ranks of the “top three AI powers” alongside the US and China. After taking office in 2025, he launched the Presidential Council on National AI Strategy to help buy massive amounts of computing power and a sovereign AI foundation model project that funds Korean companies to develop homegrown AI models. The government has also supported semiconductor titans, including Samsung and SK Hynix, through generous tax credits and low-interest financing.
South Korea’s policy posture also prioritizes accelerating AI development over safety considerations. In 2024, South Korea’s legislature passed the AI Basic Act, one of the world’s first comprehensive AI laws, to promote AI development and establish light-touch regulatory guardrails. Seventy percent of South Koreans say advancing science and medicine through AI innovation is a bigger priority than protecting industries through regulation, according to the 2026 Stanford AI Index.
All of that effort might be paying off. The same index ranked South Korea as having the third largest number of notable AI models in the world, based on criteria such as state-of-the-art advancements or high citation rates. For many small countries like South Korea, AI is a chance to punch above their weight.
The blind spots
But that single-mindedness can crowd out critical reflection on AI’s broader societal impacts. “Because the national agenda on AI prioritizes economic development,” says Jeon, the professor of science and technology policy, “there isn’t much reflection on the social, political, ethical dimensions of the technology.” In 2025, the South Korean government faced a fierce backlash for rolling out AI textbooks riddled with factual inaccuracies and data privacy risks without testing them first in a pilot program to evaluate how they affect student learning.
And despite their optimism, South Koreans are still worried that AI could displace them from their jobs. After Hyundai announced in January that it will deploy Atlas humanoid robots across its car factories, the Hyundai Motor Group union protested vehemently. “Without labor-management agreement, not a single robot using new technology will be allowed to enter the workplace,” the union said. Sixty-four percent of South Koreans fear AI could displace human labor and exacerbate inequality, although 52% believe it could also increase productivity.
On a recent Friday night in the Seoul Central Market, I went out with my cousins to a pocha, a late-night restaurant that serves fish cakes stacked in neat pyramids. As we clinked our cups of soju cut with beer—the scrappy staple cocktail of every Korean night out—one cousin asked me if I’d asked ChatGPT about my saju, a traditional Korean fortune-telling practice.
A 29-year-old insurance agent in Seoul praying for a new job and a boyfriend, she said asking ChatGPT about work and dating was her favorite pastime. She pulled up her phone and punched my birth date into the chatbot.
Addicted to their screens, trapped between unemployment and dead-end jobs, and priced out of marriage and homeownership, 46% of South Koreans in their 20s have used a chatbot to read their fortunes, according to a survey by Korea Gallup.
My cousin said she also asks ChatGPT for tips on trading stocks, dreaming big about making bank on her investment accounts into which she’s been pouring her salary. ChatGPT, she believes, is her portal out of reality into a better future.
Despite how fond she is of the chatbot as her shaman and financial advisor, she fears losing her job to AI. She still uses ChatGPT feverishly at work, as all her coworkers do, afraid of falling behind.
“I sometimes fear AI, but for now, it’s just so useful,” she said.
Google DeepMind is funding research into the potential dangers of situations where millions of different AI agents interact with each other online.
According to Rohin Shah, who directs the company’s AGI safety and alignment research, the mass-market arrival of agents that can carry out tasks without human oversight and follow instructions given to them by other agents creates a whole new class of risk.
In an effort to address this, Google DeepMind—which made agent-based tools a centerpiece of Google I/O last month—has teamed up with several other organizations to announce a $10 million funding pot for researchers to study the behavior of multi-agent systems and come up with ways to prevent unsafe scenarios. Joining Google DeepMind are Schmidt Sciences, a philanthropic foundation set up by Eric and Wendy Schmidt; ARIA, the UK government’s moonshot agency; the Cooperative AI foundation, a UK-based nonprofit research outfit; and Google’s charitable arm, Google.org.
I asked Shah and James Fox, who leads the Science of Trustworthy AI program at Schmidt Sciences, what they hope to achieve with that $10 million. It’s no small sum, but it’s dwarfed by the budgets commanded by Google DeepMind’s own research teams.
The aim is to kick-start research outside tech companies, says Shah: “The strength of academia is that it can look really quite far into the future and do the kind of work that isn’t top of mind at industry labs.”
“The main issue is that there just isn’t really a field of research for multi-agent safety yet,” he adds. “And we would like there to be.”
The concern is that as more and more AI agents get deployed and begin working together, we could hit a tipping point where imagined scenarios become real. “We see this with humanity, too,” says Shah. “Our institutions can accomplish things that no individual human can.”
Shah thinks we have a few more months to go before agents are deployed throughout the economy in numbers that make potential risks a real concern. He wants to get ahead of that moment.
Risky business
What risks are we talking about, exactly? The possibilities that Shah and Fox have in mind mostly boil down to supercharged versions of bad things that happen on the internet already: scams, prompt injections (where an AI agent is fed malicious instructions, turning it into a self-guiding piece of malware), other forms of cyberattack. We look at what humans do now and ask what the agent version of that would be, says Shah.
“We’ve got this digital commons that is integral to how society works, and you really want to ensure that this doesn’t descend into just absolute anarchy,” says Fox.
(I asked Shah if they were considering any worst-case scenarios more on the doomer end of the spectrum, such as widespread economic collapse. “Certainly not if we’re talking by the end of the year,” he said. That’s only six months away! He laughed. “Okay, a while after that.”)
Shah and Fox both think that the only way to understand what might happen when large numbers of multi-agent systems interact with each other is to run realistic simulations. They want researchers to drop AI agents into sandboxes and study what they do.
You can’t predict what’s going to happen by studying single agents, or even small groups of agents, in isolation. You can’t assume that AI agents underpinned by LLMs will always act rationally, says Fox. And the complexity comes from having huge numbers of interactions at once.
Google DeepMind is not the only top AI firm warning about the risks of the technology it is building. A couple of weeks ago, Anthropic published guidelines for deploying AI agents based on an approach to cybersecurity known as zero trust, which starts with the assumption that a computer system is vulnerable, an agent is an attacker, and a breach will happen.
Refael Angel, cofounder and CTO of Akeyless, a cybersecurity firm based in Tel Aviv, agrees that understanding the new risks introduced by agent-based systems is crucial.
Every approach to security in the past has assumed that the machine in question was software written by a human, doing fixed things on fixed paths, says Angel: “An agent breaks all of those assumptions. It reasons, it improvises, and it can be hijacked by a single sentence buried in a document it was asked to read.”
Angel welcomes this new funding. “No single lab should author the safety standards everyone else has to trust,” he says. But he cautions that safety researchers can overlook boring problems that are already here in favor of more exotic hypothetical ones.
And yet, Fox notes, risks that were hypothetical a few years ago are now very real: “The future’s come more quickly than perhaps expected.”
As adoption of AI agents looks set to surge by as much as 300% in the next two years, leadership teams are carefully considering the implications of a hybrid human-AI workforce.
Unlike existing enterprise-level automation that relies on manual input, AI agents are capable of autonomously coordinating complex tasks, interacting with multiple tools and environments across an organization. In early applications that center on customer service, HR, and sales, adoption of agentic AI has led to productivity gains of 30-50%.
Their autonomy positions agents more as collaborators than tools, working side-by-side with human employees in blended teams that look poised to upend traditional workplace dynamics.
More than three-quarters of HR leaders believe that the deployment of AI agents will transform existing workplace norms, driving a complete reappraisal of how roles and responsibilities are distributed, how skills are prioritized, and how workplace culture is shaped.
Though many admit they’re in the early or preparatory phase of this shift, 86% of chief HR officers predict that navigating digital labor shaped by agentic AI will be a central component of their role in the years ahead.
Fluency in the change management aspect of agentic AI adoption will be a crucial differentiator when it comes to unlocking the full potential of the technology going forward, believes Ateet Jayaswal, chief culture and employee experience officer at Wipro, a leading technology services and consulting company. This moment is one that he says, “calls for a mindset shift in how HR leaders would enable their organizations.”
Redeploying roles to enable higher-value work
As AI agents assume ownership of more complex and integral tasks, the distribution of roles and responsibilities within an organization will undergo significant change. It’s estimated that three-quarters of current roles will require redesign, reskilling, or redeployment by 2030 as a result of agentic AI.
For leadership, this shift should be about reskilling employees toward higher-value work in order to optimize the potential of an agent-human hybrid workforce, says Jayaswal.
For example, Wipro is a complex organization of 240,000 employees across 65 countries. It previously had multiple policies, documents, and knowledge fragmented across different systems, which delayed response to employee queries.
But the company has recently integrated a custom agentic AI assistant—an agent co-created in partnership with enterprise agentic AI platform Ema Unlimited—that can swiftly navigate this complex system, assuming responsibility for 50 HR tasks that had previously fallen to human employees. With the help of an AI agent, average response time to queries has lowered from 48 hours to five seconds.
Human employees have more time to focus on work “that requires a creative and imaginative mind and cross-functional collaboration, leveraging diverse ideas and thoughts to problem-solve,” says Jayaswal. The AI agent, meanwhile, handles rote administrative tasks like sorting timesheets or helping employees navigate policies and take actions in the flow of work.
When reallocating employee responsibilities, though, it is imperative that humans remain in the loop, Jayaswal caveats. When agentic AI is incorporated into enterprise technology, it must work with sensitive and personal data and therefore needs even more stringent guardrails and constraints than consumer applications. “When you expose an AI agent to organizational data, when you integrate it into multiple enterprise systems, then pathways around the AI agent become extremely important,” he says. “It’s an evolving space that leadership needs to have front-of-mind.” Governance should include robust data privacy rules and the establishment of governance layers, such as an AI council, he suggests.
At a fundamental level, the adoption of AI agents will force a re-evaluation of human roles, believes Jayaswal. Rather than employees primarily performing repetitive tasks or troubleshooting, a significant proportion of their time will shift to designing, teaching, and optimizing an AI agent that can do this work for them with far greater speed and predictability and without the agent getting bored.
“The nature of your job changes from being the hero who comes in to solve the problem to designing the hero who can solve the problem,” he summarizes. “The individuals who I have seen thrive in this environment are the ones who make this shift.”
An evolving employee skillset
Just as roles and responsibilities will be reconfigured to reflect the input of AI agents, the core skills of human employees will be reprioritized. More than four in five HR leaders say they’re planning to reskill workers to become more competitive in a market shaped by AI agents.
Technical skills will be increasingly important. Leading employers such as Salesforce, Danone, and Walmart are already rolling out dedicated AI and digital skills programs that aim to equip everyone from frontline workers to C-suite executives with a baseline level of AI literacy in response to the pervasiveness of the technology.
But desirable soft skills will also evolve, Jayaswal points out. Employees who assign tasks to an AI agent need to plainly articulate what modular steps may be needed to accomplish a task, what the desired outcome should be, and what parameters or guardrails need to be in place to ensure the agent doesn’t access or share confidential data.
As HR executives adapt to a blended workforce, three skills are emerging as top priorities during recruitment, according to a recent survey: relationship building, like forging constructive partnerships and account management; collaboration; and adaptability.
Maintaining a healthy workplace culture
In freeing up human employees to focus on higher-value tasks, the hope is that AI agents can elevate the employee experience, deepening fulfilment and satisfaction in the workplace.
“At Wipro, our vision is to improve the life of Wiproites,” says Jayaswal. “We are taking away non-value added work by embracing modern ways of collaborating, engaging, and transacting, leaving associates with higher order work content.”
But leadership teams embracing agentic AI will also need to plan for the new pressures and stressors that the technology can place on a workforce.
There is already confusion and knowledge gaps, with 73% of HR leaders reporting their employees don’t yet understand how digital labor will impact their work. Many organizations have opted to define AI agents as teammates or colleagues on org charts, but new research says this could erode trust and a sense of professional identity. It also raises new questions around accountability and ownership.
The role of management in addressing these concerns is critical, says Jayaswal. To maintain healthy dynamics, managers need to become skilled at orchestrating blended systems, splitting their focus between supervising AI agents and motivating human employees as they also build and supervise AI agents.
Upgrading employee well-being programs will be a core part of maintaining a robust workplace culture. “As there are more interactions with AI agents, you are losing some of the human touch that was provided by service delivery partners or leaders, or often even by colleagues and peers,” Jayaswal says. Employee services that encourage social connection and empathetic communication may help teams navigate this.
A breakneck transformation
Agentic AI looks set to scale at breakneck speed across many enterprises, and it will significantly transform how these organizations operate.
Carefully considering and deciding how to adapt to this newly blended workforce is now a top priority for leadership teams. Reviewing and refining organizational strategies is essential for optimizing both technological gains and the employee experience.
This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff. It was researched, designed, and written by human writers, editors, analysts, and illustrators. This includes the writing of surveys and collection of data for surveys. AI tools that may have been used were limited to secondary production processes that passed thorough human review.
At SXSW London last week I gave a talk called “Five things you need to know about AI,” in which I shared what I think are the biggest themes in AI right now.
I pulled a few things from our first AI10 list, an annual guide to the most important trends in this buzzy world, but I also veered off on a number of tangents. In my half-hour slot, I tried to cover the key talking points that I think help to make sense of what’s going on in tech—and thus the economy—today.
So: This is how I’m thinking about AI midway through 2026. Let me know if you would pick different points!
1. Strictly speaking, I didn’t need to show up to give this talk.
Tongue in cheek? Maybe. But generative AI tools have already become mundane, used by millions to automate everyday office tasks (including producing and delivering talks). It’s no surprise that one of the biggest questions out there right now is what this all means for jobs. People are confused and scared.
The frustrating answer is that despite the hype coming from the top about the potential for AI to join the workforce soon—and viral social media posts yelling that something big is happening—there is almost no data to say either way what kind of effect this technology will have on employment and the economy overall. That’s not to say it won’t have an impact, even a huge one, but it’s just too soon to tell.
In theory, teams of agents working together toward common goals could become assembly lines for white-collar work, doing to offices this century what Henry Ford’s innovations did to factories in the 20th century.
In theory. Because in order to know what will happen to jobs, we need to know what will happen inside the companies that create those jobs. But most companies are still figuring that out.
What’s happened instead is that many of the worst near-term, real-world fears have come true.
Take deepfakes, AI-generated images or videos of people doing things they didn’t actually do. Deepfakes have been used to incite violence, swing votes, and sow distrust. Trump’s White House is among those creating and publishing fake images.
Many deepfakes are also used to abuse women and girls. One study found that 98% of deepfakes are pornographic and 99% involve women.
Another concern is the rise of dangerous and delusional relationships with chatbots. Many people turn to chatbots to seek private advice and to feel heard. But there are now multiple lawsuits against AI companies alleging that the technology encouraged or aided suicides and other forms of self-harm.
AI is also being used in warfare in new and worrying ways. LLMs are now giving advice, not just being used for analysis. One US defense official told my colleague James O’Donnell that you could now give a military chatbot a list of targets and ask which one to hit first. Anyone who uses AI knows that its output needs to be reviewed carefully. In fact-paced, high-stress active conflict, the risk that corners get cut is high.
3. A lot of people really hate AI.
I checked out an anti-AI protest in London earlier this year and found a very broad mix of complaints. Banners proclaiming the end times bounced along to chants of “Stop the slop! Stop the slop!” Protests are getting more organized and drawing larger crowds.
There’s pushback from fans of films and video games, who object to the use of generative AI in their favorite titles. In one notable case, the acclaimed 2025 game Clair Obscur was stripped of an award when the developers admitted to using AI in just one small, specific part of its production.
And there’s the data center backlash. The US has more than 5,400 data centers and counting. With the energy demands of AI growing, people are unhappy about the environmental impact and their rising electricity bills. Activists are managing to stall development in a number of places.
Regulation is becoming politically popular. Grassroots movements like QuitGPT have gained momentum. A small number have turned to violence; a few weeks ago somebody threw a Molotov cocktail at Sam Altman’s house. It’s not clear where all this leads. But the apocalyptic hype from tech leaders is not helping people stay calm.
4. AI for science is a very big deal.
It’s early days yet, but the potential for AI to help make a genuine and important scientific discovery is greater than ever.
Mathematicians are excited too. Fundamental math underpins many everyday technologies, from internet security to video streaming. The last few months have seen a string of claims that AI has cracked unsolved math problems. And software that can solve really hard math problems will be able—so the argument goes—to solve more general-purpose real-world problems too.
What are the downsides? Some scientists are warning that an overreliance on AI tools could narrow the scope of research because scientists may choose problems that are most suited to AI assistance. There are also concerns that AI-assisted research will lead to a flood of inaccurate or fake results: science slop.
5. AI is everywhere all at once.
So where does that leave us? There are a lot of exciting things, a lot of worrying things, and a lot of hot air. It can be exhausting to keep up, and yet it all feels inescapable. Some people will tell you we’re in a race to the top; some will tell you we’re in a race to the bottom. But it’s really not clear where we’re headed.
Something is happening. Maybe even something comparable to the invention of electricity or the internet. But technologies like that take time to settle and bring lasting change.
Get ready for a marathon, not a sprint.
This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.
On June 5, 404 Mediareported that attackers had been using Meta’s AI customer support agent to steal Instagram accounts. Their approach was simple: They asked the agent to link the accounts to email addresses that they controlled, and the agent complied. One attacker broke into the dormant Obama White House account and made pro-Iran posts; others took over accounts with valuable, single-word handles, possibly in order to sell them.
AI cybersecurity concerns are nothing new. Since Anthropic announced in April that its Mythos model was too good at hacking to be released to the general public, commentators, researchers, and federal officials alike have fixated on the idea that superpowered AI systems could lay waste to our computer infrastructure. That’s not quite what this Instagram hack was: There, AI was the target rather than the attacker, and the method was far simpler than anything Mythos would cook up. But as companies offload more work to AI, these comparatively unsophisticated attacks could wreak their own havoc.
“As AI becomes more and more widely used—especially when AI is more and more widely used to automate our work flows, like account recovery—I think attackers are going to be more and more motivated to attack AI itself,” says Neil Gong, a professor of electrical and computer engineering at Duke University.
Gong and other scholars have been issuing warnings about the security vulnerabilities of AI agents for a while. They publish papers and blog posts detailing exploits such as indirect prompt injection, which involves hijacking agents using commands hidden in websites, emails, or other seemingly anodyne data sources. Compared with these techniques, the Meta hack was practically mindless. The only complication that hackers had to overcome was using a VPN that matched the true account owner’s location; then they directly asked the support agent to change the account’s email address, and it complied.
Meta has not commented publicly on how this vulnerability slipped through the cracks. But given the simplicity of the exploit, Gong says, it should have been uncovered easily, before the agent was deployed. “It’s really surprising,” he says. “I don’t understand why they didn’t find this simple problem.”
Jessica Ji, a senior research analyst at Georgetown’s Center for Security and Emerging Technology, agrees. “It raises questions like: Were there even guardrails in place?” she says. “Did anyone think to test for this kind of scenario?” She notes that the oversight is particularly striking coming from a company like Meta, which has extensive expertise in both AI and cybersecurity. Meta did not respond to a request for comment for this article, but on Monday a Meta spokesperson said on X that the vulnerability had been resolved.
As embarrassing a moment as this might be for Meta in particular, it also highlights some core vulnerabilities shared by all AI agents. Unlike traditional software, agents can respond in flexible—and unexpected—ways to new circumstances, which is why they might be able to substitute for human customer support agents. But AI agents can also be tricked in ways that humans wouldn’t be, and because they can take real-world actions, those mistakes have consequences. “A human would say, ‘Okay, why do you want to change the email address?’ and maybe respond with a security question,” says Somesh Jha, a professor of computer science at the University of Wisconsin–Madison. “What is going on with these agents is they’re very eager to finish the task. It’s almost like some elementary school student who just wants to please the teacher.”
There are ways to mitigate the risks. Companies can use traditional software to build guardrails that make sure agents follow strict rules, such as always asking for answers to security questions before sending sensitive account information to a new email address. And the experts consulted for this article all agree that agents should undergo rigorous red-teaming, a process in which developers try their best to attack a system in order to discover its vulnerabilities before it is deployed.
But there are also countervailing forces. Companies want to deploy capable agents, and the more power an agent has—and the fewer guardrails it is subject to—the more work it can potentially take on. “Security and utility always have a trade-off,” says Bo Li, a professor of computer science at the University of Illinois Urbana-Champaign. And adequate red-teaming can be expensive. Defenders have to expend more resources than attackers do, because attackers only need to discover a single exploit, while defenders try to discover and patch as many as they can. When attackers are working toward something as valuable as a single-word Instagram handle, they’ll pour resources into finding exploits, so defenders have to spend even more money to protect that prize.
As AI models continue to improve, hardening their defenses might actually get easier. Though the probabilistic nature of large language models means that LLM agents will always be vulnerable to some forms of attack, a more sophisticated model might have identified an attempt to change the email associated with the Obama White House account as suspicious. And AI systems can be used for agent red-teaming, much as participants in Anthropic’s Project Glasswing use Mythos to identify vulnerabilities in their software.
Still, experts expect that the problem of securing AI agents will only become more pressing in the future. As agents grow more capable, companies that adopt them may want to give them more power, both to provide more services with fewer humans and to avoid being left behind by their competitors. In the fast-moving world of AI, the time needed to carefully secure risky agentic systems might seem like an unconscionable delay.
“Everybody wants to be the first to do something and just push things out without careful scrutiny and red-teaming,” Jha says. “I think it’s a very dangerous thing.”
Most days in her chambers, Judge Maritza Braswell, a federal magistrate judge in Colorado, sifts through stacks of documents written by people without a lawyer. Many of them can’t afford to hire a lawyer, and others have cases too weak or too small to interest one. She reads each one carefully, mindful of how daunting it is to walk into the courtroom alone.
Lately, like many judges across the US, she has seen a noticeable uptick in such filings. According to a new study that examined 4.5 million federal civil cases from 2005 to 2026, the share of lawsuits brought by self-represented people increased from 11% in 2022 to 16.8% in 2025. Within those cases, the number of filings made more than doubled from pre-2023 levels.
Judge Braswell puts that jump down to AI.
“I do correlate that to AI in part because I see AI use,” she says. As a tech-savvy judge who uses AI to vet court documents, she’s learned to recognize how large language models write. She can tell from the prose and at times, hallucinated cases and fabricated quotes.
“I’m also actually seeing better-drafted pleadings,” she says.
But while AI appears to be expanding access to justice, it doesn’t seem to be improving people’s chances of winning. Judges are also starting to question what kinds of rights and responsibilities large language models should bear as they step into lawyers’ shoes. For example, they ask whether a chatbot has a duty to provide good advice, as a human lawyer does. And a growing number of lawmakers across the US are starting to grapple with who should pay the price when chatbots dish out bad legal advice.
AI supercharges lawsuits
To test whether AI was driving the increase in lawsuits filed by people without a lawyer, the authors of the study, Anand Shah at MIT and Joshua Levy at the University of Southern California, ran 1,600 randomly sampled court documents through Pangram, a commercial AI-text detector. The share flagged as containing AI-generated writing rose from 1% in 2023 to 18% in 2026.
To Judge Braswell, that’s not necessarily a cause for concern. While the surge of AI-assisted filings might be adding to their workloads, she and many other judges find the cases easier to rule on because AI is helping people without legal training better articulate their arguments.
Court documents written by people without lawyers are notoriously hard to decipher. Some arrive as handwritten scrawls bordering on gibberish that judges take a while to decode. However cryptic, judges are required to read them charitably.
These days, Judge Braswell has been churning through motions drafted by AI faster than the ones written by the litigants. “I have to be really careful because some of them contain hallucinations and errors, but I can generally understand what they’re arguing better with AI assistance from them than without it,” she says.
The clearer filings let Judge Braswell hear them better. “If I understand an argument a little bit better, I’m probably going to be able to help a little bit more,” she says.
Online communities are springing up to trade self-help guides on using AI to sue. In December 2024, a viral Reddit post walked immigration applicants through suing the United States Citizenship and Immigration Services over delayed review of their applications: draft a writ of mandamus with Microsoft Copilot, pay a lawyer $150 to polish it, and file in the expedient District of Vermont. Cases filed by people without lawyers in Vermont rose from about 45 a year before 2022 to more than 1,100 in 2024.
Even so, people without lawyers are far more likely to lose their case than people with lawyers, and that’s not changing even with the addition of AI, the study found.
“It turns out that mounting a lawsuit is a complex, multifaceted task. Not all of it is just drafting text,” says Levy.
Chatbot-client privilege
Judge William Garfinkel, a federal magistrate judge in Connecticut, has served on the bench for three decades, pondering all sorts of questions about lawyers’ relationship with their clients. Lately, he has been wondering whether people’s conversations with chatbots dispensing legal advice should be privileged, the way their conversations with lawyers are.
“You can make a good argument that … conversations with large language models like Claude or ChatGPT or Grok should deserve some protection,” he says.
Courts are starting to grapple with this question. In February, a federal court in Michigan ruled that a self-represented person’s conversations with ChatGPT to prepare her case were work product—legal work that is shielded from the opposing side.
The decision came on the same day a federal court in New York held that documents a criminal defendant had generated using Claude were not privileged attorney-client conversations or work product. The court argued that Claude is not an attorney and that a user has no “reasonable expectation of confidentiality in his communication” with it because AI companies can disclose user data to third parties.
In March, Judge Braswell ruled that a self-represented person’s use of a chatbot should stay off limits. “It is true that AI systems like ChatGPT, Claude, Gemini, and others … collect user data for training and other purposes. But … that does not eliminate all expectations of privacy,” she wrote. Courts have since remained split on the issue.
Malpractice without a pulse
Some judges are also wondering whether a chatbot, like a lawyer, has a duty to provide good legal advice. Judge Allison Goddard, a federal magistrate judge in California, has noticed that people without lawyers often get the wrong advice from ChatGPT when trying to assess the value of their case during settlement negotiations. In one case, a plaintiff who slipped and fell in a store asked for $700,000 from the store, which was wildly more than the case was worth.
“Where are you getting the idea that you’re getting $700,000? Did you go to ChatGPT?” Judge Goddard asked. “Well …” the plaintiff mumbled. She then walked the person through the law to explain why ChatGPT was wrong and suggested a lower amount. “It’s like Dr. Google went to law school,” she says.
Then there’s the question of who’s liable when a chatbot makes such mistakes. In March, Nippon Life Insurance Company sued OpenAI alleging that ChatGPT practiced law without a license and helped a woman reopen a lawsuit that was already settled, flooding the court with frivolous filings. “ChatGPT is not an attorney,” the lawsuit said.
In May, OpenAI asked the court to dismiss the case, arguing that ChatGPT does not practice law. “ChatGPT is not a person and neither has nor uses any degree of legal knowledge or skill,” OpenAI said in its filing. The case is still pending before the court.
States have started to weigh legislation that would hold AI companies liable when their chatbots offer bad legal advice. New York introduced a bill in March that would bar chatbots from impersonating lawyers, even if they notify users that they are interacting with chatbots. In Congress, a series of bills have been proposed to ban chatbots from posing as lawyers, doctors, and other licensed professionals. The bills have yet to gain traction.
For now, people will continue turning to AI to be their lawyer. For many of them, the rewards outweigh the risks. Not long ago, when Judge Braswell asked self-represented litigants why they wanted a particular piece of evidence, they mumbled timidly. Now, they answer her questions confidently, having rehearsed with a chatbot.
“This is a really tough system to navigate. With AI, though, it gets a little less complex,” she says.
The global health care sector is under increasing strain.
Decades of chronic underinvestment and constraints in recruitment have coincided with a surge in demand for services for aging populations. Gaps in provision are already taking a toll, with fragmented access to care and high rates of stress and burnout among staff. And it’s getting worse. The World Health Organization has warned that current shortfalls will increase to 11 million workers by 2030.
In their urgent hunt for a solution, many health-care providers are now pinning their hopes on agentic AI, with more than two-thirds (68%) having already adopted AI agents into their workforce, according to KPMG.
The technology is being deployed to automate complex back-office processes, collaborate with medical teams, and even triage patients, all in a bid to reduce the cognitive load on clinicians and improve quality of care for patients as the supply of human health-care workers dwindles.
A different type of digitalization
Until now, the benefits of digitalization within health care have been limited.
Many staff have blamed slow or outdated technology for adding to the administrative burden rather than alleviating it. For example, U.S. patient data was migrated to electronic health records (EHRs) in the early 2000s, but this data remains fragmented and reliant on manual inputs.
New telehealth services and digital care tools, like remote monitors, have had similar shortcomings, says Ashis Barad, MD, chief digital and technology officer at Hospital for Special Surgery (HSS), an academic medical center in New York that focuses on musculoskeletal health. Both technologies have helped improve access to health care by removing geographical barriers, he says, but they’ve failed to replicate the quality of in-person care or win trust from patients.
Agentic AI is different from these existing technologies, he insists.
Rather than relying on manual inputs or defaulting to human workers for any case that sits slightly outside a rigid framework, AI agents can handle nuanced, complex scenarios. They can make autonomous decisions, retrieve information from expert clinical sources, and iterate over time, freeing clinicians to focus on higher-level patient care. As Dr. Barad puts it: “Agentic AI takes your workflow and collapses it, augments it, supercharges it, and makes it more performant.”
At HSS, AI agents have already been deployed in multiple areas. They handle complex backend processes, such as insurance claims that previously took several weeks to complete and involved both HSS staff and a third-party contractor to handle the volume. Now, says Dr. Barad, AI agents complete 1,100 claims per month. They’ve reduced the appeals stage from 45 minutes to five and improved the success rate of those appeals from 65% to 100% in the nine months since implementation. HSS now handles all claims in-house.
Building on that success, HSS is now deploying AI agents in non-clinical patient-facing settings with an AI scheduling and triage service, as part of a collaboration with enterprise agentic AI developer Ema Unlimited. The service is accessible 24/7 via web, text, or phone. It uses conversational AI to ask patients clarifying questions about their condition and then books appointments with the most appropriate clinician, factoring in location, insurance coverage, and physician availability. “It completes the whole loop,” says Dr. Barad. The AI agent is trained on “all of our context, all of our rules, and all of our knowledge base,” he adds, providing patients with streamlined access to highly specialist knowledge from world-leading surgeons.
Given the high-stakes decisions delegated to AI agents, the triage service has built-in safeguards—sensitive, complex, or uncertain scenarios are escalated to human specialists. Every decision made by the AI agent is auditable and human staff can step in at any point. Patient data is kept secure and the system is trained on all HSS protocols, policies, and care pathways. By keeping humans in the loop, Ema says its technology strikes the balance between efficient automation, patient-first safety, and human-informed decision making.
As the technology becomes more prolific, it will be incumbent on providers to ensure they have these sorts of guardrails embedded into systems, says Dr. Barad. At HSS all decisions around the technology are filtered through an AI subcommittee that Dr. Barad co-chairs alongside a senior nursing executive. AI agents that may touch on patient care will be scrutinized with far more rigor than, say, backend processes, he explains.
AI agents prompt systems-level change
For example, Dr. Barad has plans to create a dedicated AI lab at the HSS main campus in New York City—a move that aims to democratize access to the technology across the organization. It will be open to all staff looking to understand or build AI agents, he explains, with informative classes and one-on-one training. “We’re getting agentic AI into everybody’s hands,” he says. This echoes research by Deloitte, which found that leading agentic AI adopters in health care were far more likely to have opted for multiagent solutions, redesigning end-to-end workflows rather than sticking to narrow solutions or individual use cases.
The key, it appears, is to integrate AI agents across the entire enterprise, treating them as a general-purpose technology. As Dr. Barad puts it: “It’s wrong to think of agentic AI in use cases… It’s a general-purpose technology, analogous to electricity.”
In practice, this means health-care providers need to set the right foundation to achieve value with agentic AI. This includes creating a unified data strategy, one that integrates fragmented data sources across an organization to create a single, comprehensive source of truth. In health care, data is often split across multiple departments and providers, each with their own legacy IT system.
In systems that rely on fragmented data sources, metrics often lack standardized definitions too. For example, Dr. Barad says that each hospital he’s worked in has had a slightly different definition for “time to start surgery,” a metric commonly used to gauge operating room efficiency. This level of fragmentation impedes AI agents from retrieving information from different sources or applications and assimilating the tacit knowledge that differentiates them from other technologies.
By creating greater interoperability of data at HSS, patient-facing AI agents can draw from a patient’s clinical care history and existing recommendations from their clinician, combine this information with current symptoms, and decide whether a situation requires escalation before notifying the correct specialist and informing the patient.
Building better outcomes
For Dr. Barad, the potential for AI agents to overhaul health care and alleviate the current pressures on resources, access, and patient care is huge.
He envisions a future in which 90% of non-clinical health-care tasks could be administered by AI agents, freeing clinicians up for what he calls white-glove work, meaning the most complex, specialized, and sensitive cases.
Most health-care providers seem equally optimistic. According to research by KPMG, 84% of providers are already comfortable handing decision making about specific processes over to AI agents.
“We’re spending so much time on keyboards and computers right now that we’re actually not doing what we should be doing,” says Dr. Barad. “This is going to rehumanize health care.”
This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff. It was researched, designed, and written by human writers, editors, analysts, and illustrators. This includes the writing of surveys and collection of data for surveys. AI tools that may have been used were limited to secondary production processes that passed thorough human review.
This article is from Making AI Work, MIT Technology Review’s limited-run newsletter examining how to apply LLMs across industries. To receive it in your inbox,sign up here.
From accounting to design to market research and product development, there’s a staggering breadth of skills needed to run a business. A large company can hire experts to handle these tasks, but small businesses don’t always have this luxury.
That’s where AI comes in. Today’s AI models do a decent job at these tasks. The trick for small businesses is to understand where AI is good enough and where it’s not.
One place where a “good enough” AI can already be quite valuable to small business owners is in providing secretarial skills and handling basic administrative matters. Let’s take a look at how one private tutor is using it to improve his recordkeeping and free up his time.
Case study
Sam Finnegan-Dehn works in fundraising for a charity, but he moonlights as a math and philosophy tutor for university students from his home in London. Through this part-time business, he can leverage his degrees in philosophy and share his love of the subject with clients.
But meeting with students is only a fraction of the work it takes to be a good tutor. He also plans lessons and finds fresh reading materials, creates assignments, sends invoices, and keeps up with new research—all on top of his regular job. Given these demands, Finnegan-Dehn doesn’t have as much time as he’d like to grow his tutoring roster.
So he’s turned to AI for some help in managing the day-to-day aspects of his business. He says AI has taken on a secretarial role across all of his digital notebooks, where he jots down reminders about his clients’ progress and new readings to keep himself up-to-date. He describes using AI as kind of like having a second memory that helps him connect ideas he’s written down in various places.
While he has experimented with different tools like Claude and ChatGPT, he’s now landed on Notion AI because it integrates better with his tutoring notes, which live across his notebook tabs in the Notion app. Finnegan-Dehn doesn’t use AI to create teaching materials, but he does let Notion AI record meetings with his clients (after getting their consent), and then uses its automated summaries to refine his teaching strategy. For example, if he notices from the AI’s summary that it seems like a certain technique was not helping a student, he may change how he approaches the subject next time.
Beyond this, Notion AI also helps him with goal-setting, drafting lesson notes, invoicing, and generating and syncing social media posts. For goal-setting, for example, Finnegan-Dehn says he understands his long-term goals for his business but not always the concrete steps to build to them. He uses AI to help fill in these gaps. He starts by writing down a “North Star” goal—say, to have a certain number of clients by the end of the year. Next, he asks his AI to generate the steps that he needs to take to get there, given the profile he has built up in the app. Then, he can reflect on the results and choose which tasks to tackle first.
The tool
Notion has been a big player in note-taking software for many years. Its AI add-on, released in late 2023, now has tools that enable it to interact with many other online productivity platforms. There’s an email client, calendar integrations, and a newly released agent. And while this level of access has raised privacy concerns, it can also make for a pretty powerful virtual assistant.
Many of the tasks targeted by Notion AI are less creative and more rote: syncing information across documents or searching through old scribbles, for example. This makes the tool especially appealing to small business owners, who have limited bandwidth, particularly for menial work.
Other companies are developing tools targeted at specific industries. For example, Grandma’s Quilt Shop in Yuma, Arizona, uses Rain, which has a software suite tailored to craft companies, to generate inventory descriptions and pricing for its stock of fabric designs. The owners claim this AI tool cuts the time it takes to list items by 60 to 80%.
There are drawbacks, though, as Finnegan-Dehn described some of Notion AI’s idiosyncrasies as “clunky” at times. And the AI add-on for Notion costs $20 per month. As with all new tools, small business owners should carefully assess how the potential gains and headaches measure up against the cost of just doing the job themselves.
User tips
Consider these points when thinking about whether AI might be able to help you run a business, or make any part of your work life just a little bit easier.
Look before you leap. Since LLMs feed on the data you input to answer your queries or complete tasks, you want to give them information in a way that’s convenient for you and for the model. For many of these notebook AI services, this means, for example, using their platform for notetaking so you don’t have to input or upload notes later. Because of this, it’s a good idea to weigh your options carefully before committing to an AI-powered ecosystem.
Work to your strengths. Think about what skills you lack in-house, and see if AI can either help with training or take these tasks on for you. Just be aware: AI hallucinates and makes mistakes, so think about where accuracy is needed and keep humans in charge there.
AI isn’t always the best tool. It’s okay to use something off the shelf when that’s the better choice. It’s going to be safer, for example, to use existing payment processing platforms like Shopify or Square than to vibe-code one using AI.
Consider using local models for any sensitive information. Our reporting has covered the risks that online AI models have in leaking sensitive data, and there have been many reports about how AI companies collect your data when you ask their chatbots questions. Even if your business doesn’t handle personal information, there can still be some things you’d prefer not to share publicly. In these cases, using an open-source model that makes inferences on your prompts locally can be a great option, instead of ChatGPT or Claude or other proprietary models. Thankfully, some LLMs can now be run off of laptops and small desktops. Here’s how to set one up and start using it.
Sign up for Making AI Work,MIT Technology Review’s limited-run newsletter examining how to apply LLMs across healthcare, climate tech, education, and more.
Login
