Apache Spark 4.2 launched last week, and it signals an expansion of Spark’s decade-plus role at the center of enterprise data processing

With new features for AI workloads, including governed metrics, vector retrieval primitives, real-time processing, improved Python support and native geospatial analytics, Spark 4.2 builds on a recent history of new AI and streaming features, reflecting how many engineering teams use the platform today. The release builds on Spark’s traditional role as a data processing engine by adding more of the capabilities needed to support production AI applications.

The launch introduced several features that enable developers to do more without leaving the platform, and for teams already using Spark, that could mean fewer systems to manage.

Governed metrics prevent conflicts

One team’s definition of a business metric isn’t always the same as another’s. Over time, those differences can lead to conflicting reports and uncertainty about which number to trust.

That becomes even more problematic when AI applications start consuming the same enterprise data that analysts and business intelligence tools use. If different teams define the same metric differently, AI systems can produce inconsistent results for the same question.

Spark 4.2 introduces governed metric views to address that issue. Organizations can define a business metric once and reuse that definition across applications. A metric view makes dimensions and measures first-class objects that Spark understands, so the engine can preserve the intended aggregation semantics regardless of who or what is querying it.

Organizations can define a business metric once and reuse that definition across applications.

Vector search goes native

One of the more significant additions is native vector search, which reduces the need to move data between Spark and a separate vector database.

Spark 4.2 introduces vector distance and similarity functions, vector normalization, vector aggregation, and NEAREST BY, a new SQL operator for top-K similarity searches. By bringing vector search into Spark, developers can keep more of their retrieval pipeline on the same platform.

By bringing vector search into Spark, developers can keep more of their retrieval pipeline in the same platform.

Python interoperability gets easier

Spark 4.2 makes it easier to move data between Spark and Arrow-native tools. With support for the Arrow C Data Interface and the PyCapsule protocol, Spark DataFrames can be passed directly to tools like Polars and DuckDB without copying or serializing the underlying data, as long as both sides support the standards.

Python also gets a few other updates. PySpark has been expanded; Arrow-optimized UDF execution is now the default, and Python Data Sources now include built-in time and memory profiling to help developers troubleshoot custom connectors.

Spark Connect makes the engine callable by agents

Spark Connect, which separates the client from the Spark server via a gRPC- and Arrow-based protocol, receives several updates in 4.2. The key idea is that a client builds a logical plan, the server handles analysis, optimization, and execution, and the results come back as Arrow batches. The client doesn’t need a full Spark runtime or a colocated JVM.

This update includes several changes to Spark Connect, the project’s client-server interface. AI applications can send processing requests to a remote Spark cluster while the work continues to run inside Spark. The release improves RDD API compatibility, error handling, and status reporting along that path.

Streaming powers real-time AI

Streaming gets several updates in Spark 4.2, including Auto CDC and Real-Time Mode. Many AI applications depend on continuously updated data rather than scheduled batch jobs. Auto CDC brings first-class change data capture to Spark Declarative Pipelines, handling the merge logic for keeping target tables current as source data changes — something that previously required hand-written, error-prone code. The new CHANGES SQL clause allows teams to retrieve data changes through a single SQL interface.

Spark 4.2 also adds built-in GEOMETRY and GEOGRAPHY types along with ST_* functions for location-aware analytics, without requiring external spatial extensions. For teams doing anything with location data — logistics, real estate, IoT — this removes another reason to move data out of Spark.

The bigger picture

Spark 4.2 brings more of the AI and data stack into the platform itself. Features that once depended on separate tools can now be handled directly in Spark.

For teams that currently use Spark for ETL and then hand data off to other systems for retrieval, governance, or real-time processing, this release starts to blur that line. As more AI applications run directly on operational data, Spark is becoming part of the serving layer rather than simply preparing data for it.

Spark is becoming part of the serving layer rather than simply preparing data for it.

The post Spark 4.2 has a feature that could retire your vector database appeared first on The New Stack.

Legacy infrastructure, not the models themselves, is what's actually slowing AI agents down. That was the shared conclusion of three infrastructure leaders — from LinkedIn, Walmart, and Zendesk — at VB Transform 2026.

The panel brought together Animesh Singh, senior director of AI platform and infrastructure at LinkedIn, Desiree Gosby, SVP of corporate technology services and technology strategy at Walmart, and Sami Ghoche, VP of applied AI at Zendesk, each describing what actually broke when they moved agents from pilot to production. Each arrived at the same conclusion from a different starting point: None of the bottlenecks they hit were model problems.

What tied their answers together was a shared premise: most enterprise infrastructure was built for how humans work, not for how agents work. The gap between those two speeds is where the real engineering happened.

Gosby put it plainly when asked what she'd learned scaling agents inside Walmart's own workforce. The goal, she said, is to make sure "engineering doesn't once again become the bottleneck for what it is we're trying to do."

Where the bottleneck actually was

Each company hit a different version of the same wall: infrastructure designed for how people work doesn't hold up once agents are doing the work instead.

At LinkedIn, the first bottleneck wasn't a model, it was Kubernetes, which assumes containers spin up on demand, a process that takes seconds. Singh said that's too slow for agents. The fix was moving from on-demand provisioning to pre-provisioned pools of containers that swap agentic workloads in and out in real time.

A second, harder problem surfaced once LinkedIn let agents control their own orchestration. A five-point evaluation system looked clean, but hallucination kept showing up anyway. Singh said the issue was structural, an LLM evaluating another LLM's output shares the same failure mode as the thing it's evaluating. 

"We built our own harness, our own control flow, and pushed the LLMs to the leaf instead of them orchestrating the loop," Singh said. Roughly 80% of the workflow is now scripted, deterministic code, with LLMs used only where reasoning is required, and each step's evidence is committed to disk before the system moves on.

Walmart's bottleneck came from success. An agent harness put directly into employees' hands went viral internally, and what Gosby called "citizen developers" began building their own agents to solve problems that once required a formal engineering roadmap. The upside was real innovation. The downside was duplication, dozens of overlapping agents with no coordination. The fix wasn't reining in the harness, it was building governance to spot duplication, promote the best version of an agent, and get it into production without engineering becoming a chokepoint.

Zendesk hit its bottleneck from the data side. Ghoche, who joined through Zendesk's acquisition of Forethought, which closed in March 2026, described sitting on what he called a public figure of 20 billion customer conversations in Zendesk's repository. The instinct is to hand that history to a large language model with a big context window and let it generate the agents a business needs. Ghoche said that doesn't work. "You can't really do that, so instead you have to really invest in the underlying data pipelines and all the data infrastructure that comes with that," he said.

The role of open source

On open source, all three leaders landed on a similar instinct: own what you can, and lean on frontier labs only where they still have a clear edge.

Ghoche said his own view is that most enterprises would prefer to own their models and infrastructure wherever that's possible, and that reasoning is what drives Zendesk's own approach. The exception is frontier reasoning work, where the labs still lead, though he said that slice of use cases is shrinking relative to everything else enterprises now do with AI.

LinkedIn's answer was to build two subsystems specifically for independence. The first is what the company calls an AI gateway, a single interface that every outbound call to a model runs through regardless of provider. The second component is a memory subsystem built to hold context independent of any model provider.

"Every single outbound call going to an LLM, whether it's on a public cloud or on-prem in our own data centers, follows the same semantics, the same API calls. We can quickly switch between different providers," Singh said. 

Walmart built its own internal gateway to stay vendor agnostic across three workload types: fully deterministic workflows, planner-and-reasoner workflows for open-ended tasks, and a hybrid of the two. Compliance-heavy work stays deterministic by design; governance, security and evaluation run through the gateway regardless of which model is on the other end. Gosby said the choice between a frontier model and an open-weight model comes down to whichever is most effective for the specific workload, not a fixed policy.

Advice for the modernization journey

Three pieces of advice came up directly, each tied to the wall a leader had already hit.

Invest in evals before anything else. Ghoche called it the thing common to every use case, internal or customer facing. 

"The thing that's common to all of these is evals. It'll force you to break the problem down, and once you have a robust set of evals, you can move a lot faster," he said, 

Own your agent harness from day one. Gosby's advice was to put the AI harness directly in employees' hands early, paired with the infrastructure to monitor what it produces. 

"It will unlock a huge amount of innovation," she said.

Build for model and context independence. Ensuring flexibility is critical for success.

"Build for independence, whether it's a frontier model of today versus an open source model of tomorrow," Singh said. "Keep that context within your enterprise so that you can reuse it when you ship the model or the harness tomorrow," Singh said.

Organizations need to transform to meet the needs of agentic AI.

Meta VP of Engineering Barak Yagour opened his talk at VB Transform 2026 wearing a pair of Ray-Ban Meta AI glasses, a small sign of how far AI has already worked its way into physical life. His argument went further: enterprise infrastructure was built for humans, not for agents, and it's starting to show.

Yagour, who leads its data infrastructure organization, told the audience that agentic queries hitting Meta's data systems grew 30x in a single half, an inversion that he said is breaking assumptions the company spent two decades building around.

The shift is not confined to Meta. Automated traffic overtook human traffic on the internet last year, reaching 51% of the total, according to Imperva's 2025 Bad Bot Report. That traffic is also growing roughly eight times faster than human traffic, according to HUMAN Security's 2026 State of AI Traffic report. Yagour cited both figures to describe what he called an inflection point already underway inside his own organization.

Yagour framed the shift as an open question for infrastructure teams everywhere. "What happens to the infrastructure we've spent years building when agents and not humans become the main consumers of that," Yagour said. "That's the world we're stepping into."

Capacity, identity and velocity are breaking at once

Yagour said three assumptions are breaking simultaneously inside Meta's infrastructure: capacity, identity and velocity.

On capacity, the math no longer works the way engineering teams are used to. "One engineer used to mean one unit of load," he said. "Now one engineer spawns 10 agents, each spawning subagents. Your 1,000-person org can generate the load of 100,000 users practically overnight."

His answer is not to block agent traffic but to make infrastructure agent-aware, with dynamic controls that understand agent hierarchies, cost attribution that traces consumption back to the use case that spawned it, and throttling that adapts based on priority.

Identity is breaking, too. Yagour said an agent does not fit the categories infrastructure teams built access controls around. It is not a human user, it does not carry a badge and it is not a deployed service, yet it makes decisions on its own.

Velocity is the third assumption under strain. Yagour cited a company-reported figure that GitHub Copilot writes 46% of the average user's code, then noted that faster code generation does not make the rest of the pipeline faster.

"That code still needs to be built, tested, deployed, monitored," he said. "The agent writes the code in seconds, but your CI/CD pipeline doesn't get faster just because the machine is the author."

Trusted data environments keep agents inside guardrails

Data is where Yagour said the pressure from agents is most direct. 

"Data sits at the center of everything," he said, pointing to the decisions, products, recommender systems and next generation models it drives.

Meta is also rethinking how much autonomy to grant agents inside its own data systems. In February, the company shipped what Yagour called agentic data apps. Within three months, 63% of dashboards published across Meta were built using the new tooling, part of the same 30x rise in agentic queries Yagour cited earlier.

That growth raises a governance question. Human analysts have traditionally sat between raw data and business decisions, curating it and serving as an informal check on quality. Yagour said Meta wants to grant agents more independence on harder problems, but was direct about the risk. 

"Autonomy without governance is nothing but chaos," he said. That's why the company built what it calls trusted data environments, to preserve the human check as agents take on more of that work.

"Inside, the agent can explore data freely, but every output is traced back to its source and scrutinized. So you always know that the data shared back is trusted and governed," Yagour said.

Sensitive fields are masked before an agent can reach them, and every access request is evaluated in real time against what the agent is trying to reach, why and whether it is allowed. Yagour summarized the approach as exploring broadly while releasing narrowly.

Reasoning models are rewriting the data layer

Meta's models are also demanding more from data as they shift from correlation to reasoning. 

"Reasoning is data hungry," Yagour said. 

Pattern matching works on sparse, summarized signals. Reasoning demands the full behavioral history, every interaction across every surface over time. Yagour pointed to two shifts already underway inside Meta's infrastructure to keep up.

Real-time streaming is replacing batch ETL for ranking pipelines. A pipeline that takes 24 hours to run is not viable when a model is reasoning about a user's current intent. Yagour said real-time streaming, not batch extract-transform-load processing, is becoming the backbone of Meta's ranking and recommendation systems.

Storage is becoming schema-aware to stop GPU starvation. Meta previously stored user data as opaque blobs with no awareness of what the data contained, which Yagour said led to heavy overfetching and idle GPU capacity. The company is now building storage that understands what it holds, pulling only the columns and time ranges a given query needs. Yagour said Meta is building toward 500 million queries per second and a petabyte per second of throughput for training data reads.

That data feeds directly into how Meta's recommendation systems behave. Yagour said 42% of Instagram users have told the company they want to fundamentally change the algorithm, not adjust a single session or setting. Meta's response is what Yagour called fully conversational recommendations, where a user tells the system what they want more of and it reasons about intent rather than matching on keywords. Yagour said the same search term, soccer, would return different results for a casual fan looking for highlights than for a club athlete seeking training drills, because the system would reason about which one is asking.

Yagour described the three threads of his talk, agents, data and recommendations, as reinforcing each other rather than moving independently. 

"Agents make data more accessible. Better data makes reasoning. Reasoning creates new demands that push agents and infrastructure forward," he said. "This isn't linear; it's a flywheel."

During the Q&A, an audience member asked whether Meta's push toward more intelligent infrastructure signals the end of traditional file systems in favor of newer neural storage approaches, and whether agents will keep using SQL as their interface to data the way humans do. Yagour said Meta is experimenting at every level, including questioning whether SQL is the right interface for agents at all, and that storage at Meta's scale already operates in the multi-digit exabyte range and needs to keep expanding.

Yagour closed his talk with the timeline he believes the industry is working against. "We spent 20 years building infrastructure for humans. We have maybe 20 months to rebuild the whole thing for a world where humans and agents co-create at scale," Yagour said. "The window is open, but it won't stay open for long."

Canva on Tuesday launched Canva Code 2.0, a major upgrade to its AI-powered coding tool that lets users build interactive websites, apps, and experiences using plain-language prompts — and then edit the results as easily as tweaking a Canva presentation. The feature is now available to all of the company's more than 265 million monthly users across every pricing tier, including free accounts.

The move is Canva's most aggressive push yet into the fast-growing "vibe coding" market, a category that barely existed 18 months ago but has already minted billion-dollar startups and reshaped how non-developers think about building software. But where rivals like Lovable, Replit, and Bolt.new have focused primarily on generating functional code from text prompts, Canva is making a different bet: that the real bottleneck isn't creating the code — it's making the output actually look good.

"Most vibe coding tools stop at functional — generating output that looks the same as everyone else's," Canva states in its announcement. "You might get a working prototype, but making it actually look like yours requires a complex editing surface, a separate design tool, a developer, or endless back-and-forth prompting that rarely lands where you want it.”

Danny Wu, Canva's Head of AI Products, framed the product's positioning in stark terms during an exclusive interview with VentureBeat ahead of the launch.

"We are deliberately targeting non-technical users," Wu said. "Canva Code isn't a tool we're building for developers. What we're trying to do is bring the power of AI coding — and really lightweight coding — into the Canva platform, while answering our users' requests for more interactivity, more customization, and more flexibility, from websites to interactive presentations."

Canva Code 2.0 brings drag-and-drop editing, HTML import, and 75% faster generation to AI-built websites

The update introduces several capabilities designed to collapse the distance between generating code and publishing a polished interactive experience. Users can now create Canva Code projects directly inside other design projects — embedding interactive elements within a whiteboard, presentation deck, or standalone page. Canva has also added more than 50 new templates specifically designed for interactive designs, along with the ability to import raw HTML files from other AI coding tools and convert them into editable Canva designs.

The performance improvements are significant. Canva says it has reduced average code generation time by 75 percent and cut the median time from initial prompt to a published site by 30 percent. The company also reports that integrating Canva Code into the broader Canva editor — allowing users to treat coded outputs like any other design element — has increased active Code users by 25 percent.

Perhaps the most distinctive feature is the editing experience itself. Unlike most AI coding platforms, which require users to re-prompt or modify raw code to make visual changes, Canva Code 2.0 lets users click directly into generated elements to change text, drag and drop images from Canva's built-in library of over 120 million templates and assets, update colors and fonts through a familiar toolbar, or select a specific element and refine it through conversational AI. Every output is fully interactive and automatically adapts to different screen sizes, with a built-in mobile preview.

Wu demonstrated the drag-and-drop editing during the interview, showing how a generated conference website could be modified in real time — swapping in photos, changing fonts to branded alternatives, and editing text directly on the canvas. "The key differentiator with Canva Code is the editability and the kindness of the outputs it generates," he said, though he noted one current limitation: "We don't support moving elements around. You still have to re-prompt for that."

How Canva plans to compete with Lovable, Replit, and Bolt in the booming AI app builder market

Canva's entry into vibe coding at this scale arrives at a pivotal moment for the category. According to market research published by Luminix AI in May 2026, the vibe coding and AI app builder market has reached an estimated $4.7 billion in 2026, with projections pointing toward $12.3 billion by 2027 at roughly 38 percent compound annual growth. The research also estimates that AI-generated code now comprises approximately 41 percent of all code written globally — a figure that would have seemed inconceivable even two years ago.

The competitive landscape has grown ferocious. Lovable, which focuses on conversational, design-forward app generation for non-technical founders, has achieved what may be the fastest revenue ramp in the category's history — reportedly reaching approximately $400 million in annual recurring revenue by early 2026, according to Luminix's analysis. Replit, which transformed its browser-based IDE into a full vibe-coding engine through successive AI agent releases, has tripled its valuation to $9 billion and is targeting $1 billion in run-rate revenue by the end of 2026, per the same report. Bolt.new, which runs a full Node.js environment entirely in the browser, scaled from $4 million to $40 million in ARR within months of launching.

And then there is Canva, which brings something none of those platforms possess: a quarter-billion-user design ecosystem where brands, teams, and individuals already store their visual identities, collaborate on projects, and publish content.

Wu positioned Canva Code not as a direct competitor to these developer-focused tools but as something that fills a gap none of them have addressed. "A lot of the requests that we have been getting and the usage we're seeing is actually with using Canva Code not necessarily as just one artifact, but as part of an overall design, the visual communication they're trying to tell," Wu said. "Like when you have a sales deck, you're able to add a calculator, you're able to add a visualizer of what exactly your product does. That's something where an interactive slide can be worth a thousand pictures."

Why Canva's HTML import feature could turn it into a 'finishing layer' for every AI coding tool

One of the most strategically interesting features in Canva Code 2.0 is its HTML import capability, which allows users to take code generated by any AI tool — including ChatGPT, Claude, Lovable, or Bolt — and bring it into Canva as a fully editable design. The implication is unmistakable: Canva is positioning itself as the place where AI-generated code gets its finishing touches, regardless of where it was originally created.

When asked directly whether this amounts to positioning Canva as a "finishing layer on top of vibe coding," Wu offered a diplomatic but revealing response. "It's really a continuation of our goal to make all design as easy as possible," he said. "We've supported importing PDFs and translating them into docs, importing PowerPoint files — so in one way, it's an expansion of that. But in another way, it's really just listening to what our users want and making Canva both the most useful and the most compatible platform.”

He paused, then added: "It's not that we're deliberately positioning ourselves as a specific layer, say like a finishing layer after vibe coding. We just really want to make our platform the most accessible and the most pluggable."

That language — "most pluggable" — suggests a platform strategy that doesn't require Canva to win the AI code generation race outright. If Canva becomes the default destination for making AI-generated code look professional and on-brand, it captures value from the entire category regardless of which code generation engine users prefer. The strategy also echoes the broader import capabilities that already allow Canva to ingest PowerPoint decks and PDFs from competing platforms, gradually pulling users deeper into the Canva ecosystem without demanding they abandon existing workflows.

What Canva Code can build — and where Danny Wu says it hits its limits

Wu was notably candid about the product's boundaries — a refreshing departure from the typical Silicon Valley product launch. "Canva Code is great for anything that works as a front-end app, and it's especially good when you want to leverage data, data submissions, and interactivity at small to medium scale," he said. "I'll be honest about the limitations. Canva Code is probably not going to be suitable if you're trying to build a website with complex backends, or if you're handling hundreds of thousands of visitors per day."

This candor effectively draws a line between Canva Code and the more ambitious platforms in the space. While Lovable and Replit are pushing toward full-stack application development — complete with databases, authentication, and production-grade hosting — Canva is deliberately limiting its scope to interactive front-end experiences at modest scale. The question is whether that's a strategic weakness or a disciplined focus. For the teachers, small business owners, and marketing teams that make up the bulk of Canva's user base, complex backends and high-traffic scalability are irrelevant concerns. What matters is whether they can create an interactive event page, a property listing website, or a classroom hub that looks professional and works on mobile — without hiring a developer or learning a new tool.

When asked about the AI models powering Canva Code, Wu confirmed the company uses a combination of proprietary and third-party models, including those from OpenAI and Anthropic, but declined to specify the exact mix. "We don't share the exact mix, and it does change over time," he said. "We also route differently depending on what you're asking for and which model family we think is best for handling certain requests."

Canva's AI acquisition spree — from Affinity to Leonardo.ai — now powers its vibe coding push

Canva's broader AI infrastructure has been significantly bolstered by an acquisition strategy that has accelerated over the past two years. In March 2024, the company acquired Affinity, the British creative software suite popular with Mac users, in a deal that Bloomberg reported was valued at "several hundred million pounds." Canva at the time positioned the deal as a way to compete with Adobe's flagship products — Illustrator, Photoshop, and InDesign — by gaining ownership of Affinity's Designer, Photo, and Publisher applications.

Just four months later, Canva acquired Leonardo.ai, an Australian generative AI startup with over 19 million registered users and more than a billion images generated. Canva co-founder Cameron Adams said at the time that Leonardo.ai's technology would be integrated into Canva's Magic Studio generative AI suite.

Together with these acquisitions, Canva Code is the company's attempt to layer interactive, code-driven capabilities on top of a visual design platform that has already been enhanced by professional-grade design tools and generative AI models. The company reports over 32 billion uses of its AI products to date — a staggering figure that underscores how deeply AI is now woven into everyday Canva workflows, even for users who may not think of themselves as using artificial intelligence.

Six million sites published, but Canva's retention data remains an open question

Canva's announcement highlights an impressive traction metric: users have created and published more than six million websites using Canva Code since the feature was first introduced a year ago. But the number deserves scrutiny.

Wu clarified in the interview that the six million figure represents published websites over the past year — meaning sites that were either made public or shared via password-protected or private links. "They may have published publicly, or behind a password, or as a private link. But that's the number of published websites," he said.

When asked about active retention — how many of those sites are still live and being maintained — Wu acknowledged the gap in his data. This is a meaningful distinction. In the vibe coding market, raw creation numbers can be misleading because the barrier to generating a site is so low. The more telling metric — which Canva does not yet provide — would be how many of those six million sites receive regular traffic or have been updated after initial publication.

The early use cases, however, suggest genuine utility beyond novelty. Educators and school administrators are using Canva Code to build classroom hubs, with one teacher creating bespoke webpages for each of their classrooms to keep students and parents updated on announcements. Small businesses, like Alt Marketing School, have built mini apps for fundraising training and interactive roadmaps for their members. For World Book Day, 50 readers created educational games across different subjects, complete with pedagogical guides for classroom use.

Canva Code pricing, data governance, and what enterprise customers need to know

Canva Code 2.0 is available across all of Canva's pricing tiers, including its free plan — a notable decision given that competitors like Lovable, Bolt, and Replit reserve their most capable features for paid subscribers. "As you go from, say, free to pro to business to enterprise, you would get more AI credits and be able to have higher usage of Canva Code," Wu said. "But it is available and it is usable — even free Canva accounts as well as education and not-for-profit accounts."

This credit-based approach mirrors the pricing evolution happening across the entire vibe coding category, where platforms have converged on token or credit systems that meter AI generation capacity rather than gating features behind subscription tiers. The difference is that Canva's free tier serves as an acquisition funnel for a much larger design platform, not just for the coding feature itself.

For the institutional customers Canva increasingly courts — school districts, real estate brokerages, enterprise marketing teams — data governance is a threshold concern. Wu addressed this directly. "All users and customers have full control over how their data is used," he said. "They can choose whether their prompts and data are used for AI training in the settings. For businesses and enterprises, team admins can manage this at the organizational level and guarantee that their inputs, content, and outputs won't be used for training." This opt-out approach reflects a lesson the broader industry has learned the hard way. As The Verge reported when Canva acquired Leonardo.ai, Adobe suffered significant backlash over a policy update regarding user data and AI model training — a controversy Canva appears keen to avoid.

Canva's long-term vision: closing the gap between imagination and what non-technical users can actually build

When asked where Canva Code fits into the company's long-term trajectory — and whether Canva is building toward a full-stack app development platform — Wu steered the conversation back to the company's core audience.

"A huge part of it is reducing the gap between your imagination and what's possible, especially for everyday users — people who don't have a lot of time," he said. "They don't have time to figure out deploys or MCPs or APIs. They just want to design more interactive and more dynamic communication."

He pointed to the rapid improvement in AI model capabilities as a key accelerant. "The kind of things you can create today in one shot — like a 3D visualization of a solar system — you really couldn't have trusted the output a year ago. But today, you have a really high success rate."

Whether Canva Code becomes a durable product category or a feature that gets absorbed into the platform's broader AI workflow will depend on how quickly the company can close the gap between its current front-end focus and the full-stack capabilities that increasingly define the competition. Lovable is shipping Supabase-backed apps with authentication and databases built in. Replit's agents can execute autonomous long-running builds. Bolt.new runs entire Node.js environments in a browser tab. These are fundamentally different ambitions than making a conference landing page look good.

But Canva has never won by matching the technical depth of its competitors. A decade ago, it didn't try to out-feature Adobe — it made design accessible to the 99 percent of people who would never open Photoshop. Now, in a vibe coding market where every tool can generate a working prototype from a prompt, Canva is making the same wager it made in 2012: that for most people, the hardest part was never the building. It was making it look like it came from you.

The seemingly unquenchable thirst of the AI data ingestion pipeline spans language, numerical, and tabular data in the first instance, while other tangential platforms have been building large audio, image, and video models at the same time. 

Straddling potentially all of these domains are the file structures where complex documents and forms of unstructured data reside; this is the road less traveled in terms of the source DNA modern AI draws from.

The schema-less, freeform, uncurated data lake

In a bid to bridge connections to the schema-less, freeform, uncurated information that all organizations naturally harbor, enterprise visual intelligence company Valantor announced its acquisition of unstructured information RAG specialist EyeLevel on Tuesday. The acquisition formally launches Valantor’s Enterprise Visual Intelligence platform, combining EyeLevel’s document intelligence with its own operational expertise.

Benjamin Fletcher, CEO and co-founder of EyeLevel, tells The New Stack that where organizations fail to adopt visual intelligence, human-only processing breaks down pretty quickly in the age of AI.

“About 80% of corporate knowledge is in millions of pages of visually complex PDFs, PPTX, and DOCX files,” Fletcher says. “This information is far beyond the capacity of any LLM context window and is effectively inaccessible to LLMs and agents.” 

“We’ve found the golden datasets that teams build by hand routinely carry 10 to 25 percent error rates. Ironically, those same teams often hold AI to a far higher standard than their own people.”

Humans are slow, expensive & prone to errors

He explains that transactional workflows (such as invoice and claims processing) typically involve documents “so visually complex and diverse” that enterprises still rely on humans to process them, who can be slow, expensive, and error-prone. 

“We’ve found the golden datasets that teams build by hand routinely carry 10 to 25 percent error rates,” Fletcher says. “Ironically, those same teams often hold AI to a far higher standard than their own people. If data sovereignty matters to a business, everything gets harder now: solving these problems with AI while your documents stay inside your own infrastructure is the hard mode version of the job, and very few tools can do it.”

Where does invisible corporate information live?

Valantor has noted that while most AI companies concentrate on models, the company itself is “focused on the information those models can’t see” today. The suggestion is that this unseen morass of valuable data is locked inside documents, claims files, contracts, engineering drawings, reports, forms, presentations, and other visually complex content.

Valantor’s flagship platform product, GroundX, operates where data resides, including private cloud, sovereign infrastructure, on-premises deployments, and fully air-gapped environments. 

“GroundX is the ingestion and retrieval layer for unstructured documents,” explains Fletcher. “It is one tightly tuned system where retrieval consumes exactly what ingestion produces. Everything is exposed through REST APIs, SDKs, and MCP. It ships as REST APIs, SDKs, and MCP, and the Helm chart drops straight into a team’s existing deploy pipeline, and our agent harness gives coding agents like Claude and Codex the skills to build the integration themselves.

As part of the acquisition announcement, Valantor is introducing GroundX Studio. The harness capabilities within GroundX Studio integrate with modern AI development environments, enabling developers to build secure AI applications that operate on enterprise knowledge while remaining within existing infrastructure. 

GroundX Studio also extends capabilities to business users, allowing organizations to create AI-powered workflows and applications without extensive custom development.

“Each agent does one small task, so cheaper models are often good enough, and teams that want direct control over cost can run the whole stack on their own hardware with Helm.”

Risk of latency-laden performance and spiraling costs?

If it feels like this new data ingestion stream is going to place a new burden on cloud workloads, application execution latency, database retrieval times, and (of course) overall token usage, then Valantor and EyeLevel say that this consideration has been taken into account by dint of their own platform’s orchestration layers.

“We never send a whole schematic to a language model; our vision model splits each page into its elements first,” Fletcher confirms. “Processing runs in multiple passes at different levels of the document, and everything inside a pass runs in parallel, so there’s a minimum processing time, but it does not scale linearly with page count. Each agent does one small task, so cheaper models are often good enough, and teams that want direct control over cost can run the whole stack on their own hardware with Helm.”

The intersection of AI and handwriting

sWhile we already know that AI and handwriting do mix in the same cocktail glass — the ViWoods AiPaper digital e-ink handwriting tablets have a useful set of AI functions on board, and similar products are available from manufacturers including reMarkable — it’s not a widely deployed use case yet. Valantor claims that its underlying data models and custom heuristics bridge the “data comprehension gap” when processing handwritten annotations.

“Our proprietary vision model, fine-tuned on more than a million pages of enterprise documents, sees the page the way a human does: tables, paragraphs, and figures,” underlines Fletcher. 

He says that handwritten marks are captured as page elements with their layout context intact. Narrow agents then distill each element into a contextual object tuned for both search and LLM completion. 

“Smaller pieces, less cognitive load — that’s how we close the gap, with better accuracy at lower cost, driving better performance and significant cost advantages,” he adds.

Working examples of this technology include Air France-KLM, which used GroundX to develop an AI-powered customer service assistant trained on thousands of policy documents, achieving 96+% accuracy on complex policy-related questions. AskVet used the platform to operationalize more than a decade of proprietary veterinary data, enabling autonomous resolution of up to 85% of customer inquiries while significantly improving operational efficiency.

Is document management sexy now?

Taking all of this on board, are we at the point where we can ask whether document management has just become interesting, compelling, and sexy? 

No, of course it didn’t; it will arguably always suffer from a degree of stigmatized disdain. That may change in the future as we interact more directly with AI tools that begin analyzing the unstructured information we know organizations have been sitting on for so long. For now, it may still remain the corporate equivalent of eating your vegetables — pass the Brussels sprouts and steamed turnips, please.

The post AI can finally read your handwriting — here’s why enterprises care appeared first on The New Stack.

An enterprise AI agent answers with total confidence, but the number is wrong. Nobody catches it until someone traces it back to a stale metric definition or a document the retrieval system never pulled. The model did not fail. The context it was given did.

In the past six months, 57% of enterprises traced a confident but wrong AI agent answer to missing or inconsistent business context, and 31% said it happened more than once, according to a VB Pulse June 2026 survey of 101 qualified enterprises with more than 100 employees.

The reason is not hard to find. Retrieval over documents is the default way agents get business context for 38% of enterprises, nearly double the next closest approach. The way most enterprises choose a retrieval system compounds the problem. Ease of ingestion and operational simplicity lead the selection criteria, with retrieval accuracy running behind both. The accuracy problem only shows up after the system is already live.

There is a known fix for this, a governed context layer every agent reads from instead of guessing. Vendors are racing to roll out context platforms while most enterprises are still figuring out what it is.

75% don't have an agentic context layer yet

The context layer is meant to be a shared model of what business data actually means, built once and referenced consistently instead of re-derived by every agent that touches it. 

The VentureBeat research shows the enterprise response to that idea is broad but unfinished. Twenty-five percent of respondents run one in production. Thirty-four percent are building one right now. The remaining 41% have not started.

Among companies already building or running a governed context layer, 78% report a confident-wrong failure — an AI agent that answered with total certainty and was still wrong. Among companies with no plans to build a layer, only 20% report the same thing. Companies that already got burned are far more likely to be building the fix. Companies that haven't been burned yet see no urgency.

What governed context looks like when someone actually builds one

Every major data and AI platform vendor is now building some version of this layer, and they are not converging on the same architecture. 

• DataHub is treating catalog metadata and years of analyst query behavior as a knowledge source, then keeping it current as a living system rather than a static wiki. 

• Microsoft's Fabric IQ is building a business ontology that any agent, not just Microsoft's own, can query over MCP. 

• Couchbase is pushing agent memory and context retrieval down to the edge, arguing the operational database is a more natural home for it than a search or analytics layer bolted on after the fact. 

• Pinecone's Nexus is compiling structural logic into the metadata layer ahead of runtime, betting that agents need pre-built structure more than they need faster search.

• Snowflake runs a two-layer system, Horizon Context for customer-managed definitions and Cortex Sense for context the platform infers on its own. 

• Oracle's Unified Memory Core takes the opposite approach, folding vector, graph and relational data into one transactional engine so there is no sync layer left to go stale. 

• Google's Knowledge Catalog mines query logs and usage patterns to curate semantic context automatically.

• AWS's Context service makes the same bet, a knowledge graph that gets smarter from how agents actually use it rather than from manual re-curation.

Analysts converge on one diagnosis

The vendor approaches differ. What analysts and practitioners have told VentureBeat about the underlying problem, across a run of interviews this year, does not.

When DataHub's context layer push landed this spring, Constellation Research VP and principal analyst Michael Ni framed the stakes in blunt terms. "Whoever controls runtime context controls the AI decision layer for enterprise data," Ni said. He was equally direct about how far any single product actually gets a buyer. "Vector memory isn't business meaning, business meaning isn't governance and governance isn't execution," Ni said.

In the same interview, BARC analyst Kevin Petrie pointed to a narrower but concrete gap. Most context platforms concentrate on structured tables, he said, which give agents trusted facts but miss the harder, messier context locked in documents and unstructured content, exactly the material a business actually runs on day to day.

Stephanie Walter, practice leader for AI Stack at HyperFRAME Research, made a related point earlier this year when VentureBeat asked her about enterprise context fragmentation. 

"The market is converging on the same conclusion," Walter said. "Agents don't just need more tokens or better models. They need governed, current, low-latency context." She made a similar case in an earlier review of Pinecone's Nexus launch, careful not to overstate how new any of this is. Nexus, she said, "shifts knowledge work from runtime chaos to pre-compiled structure. But it's an evolution of RAG architecture, not a complete reinvention." 

Gartner's Arun Chandrasekaran, reviewing the same launch, offered the more forward-looking read. Agentic AI, he said, is moving from pure information retrieval toward a reasoning architecture, one where long context works as short-term memory and a vector database functions as deep storage underneath it.

The fragmentation problem shows up hardest at the practitioner level, where separate tools for retrieval, memory and access control were never built to agree with each other. Steven Dickens, CEO and principal analyst at HyperFRAME Research, put it bluntly after Oracle's AI database push landed this spring. "Data teams are exhausted by fragmentation fatigue," Dickens said. "Managing a separate vector store, graph database and relational system just to power one agent is a DevOps nightmare." 

Matt Kimball at Moor Insights and Strategy, in that same story, put the production reality more simply. Getting an agent working is not the hard part, he said. The struggle is running it in production, where the goal becomes removing the distance between data and execution rather than adding another layer on top of it.

What this means for enterprises

Here's what this adds up to for enterprises building on this layer.

Retrieval alone will not close the context gap. RAG is the default source for context in most enterprises today, and it is also the layer most closely associated with the confident-wrong-answer failure. Adding more documents or a bigger index does not fix a definition that is inconsistent across systems.

The semantic context layer is where the budget is actually moving, even where it hasn't shipped. Fifty-eight percent of enterprises are already engaged — building or in production — but only 25% have actually gotten a layer live. That gap shows where enterprises have decided to spend, not where they've arrived.

No single vendor owns the architecture yet, and that is likely to stay true for a while. Enterprises evaluating this layer should expect to integrate rather than pick a single winner, at least for the next several quarters.

The buying decision is happening this year, and it is concentrated among the companies already burned by it. Fifty-seven percent of enterprises plan to switch or add a retrieval or context platform within the next twelve months. That intent is not spread evenly. Enterprises that reported a repeat confident-wrong failure plan to switch or add a provider at roughly 81%, against 32% among enterprises that never hit the problem. The companies shopping for new context tooling right now are largely the ones whose agents already got it wrong.

The agents are already running. The context underneath most of them is still being built, and the vendor selling the fix is being chosen this year.

This data will be part of a broader conversation at VB Transform 2026 on July 14 and 15 in Menlo Park: the context gap enterprises are racing to close, and which of the emerging approaches — governed semantic layers, hybrid retrieval, provider-native bundles — actually holds up in production.

Fine-tuning LLMs for financial natural language processing (NLP) is constrained by limited, imbalanced data. Real-world financial news overrepresents earnings and stock movements, while rarer events such as credit-rating changes, product approvals, and labor issues are harder to capture at scale. Synthetic generation can help fill those gaps for trading research, risk modeling, and surveillance…

Source

Elon Musk's SpaceX released Grok 4.5 on Wednesday, the first artificial intelligence model the company has trained specifically for coding and autonomous agents — and the first tangible product of its $60 billion acquisition of the AI coding startup Cursor, completed just weeks ago.

The launch marks a pivotal test of the sprawling, vertically integrated AI empire Musk has assembled over the past six months, and of a strategy that bets developers care less about topping benchmark leaderboards than about speed, cost, and whether a model can actually do the work.

"Announcing Grok 4.5, our first model trained specifically for coding and agents," the company said in a post on X. "It was trained with Cursor and offers frontier intelligence at leading speeds and cost efficiency."

Why Grok 4.5's pricing strategy matters more than its benchmark scores

SpaceX is not claiming Grok 4.5 is the smartest model in the world. Instead, it is making an economic argument. The company says the model uses half as many tokens per task as comparable models, delivers higher throughput, and costs less than half as much — priced at $2 per million input tokens and $6 per million output tokens. That undercuts the premium tiers of rivals like Anthropic's Claude Opus line and OpenAI's frontier models by a wide margin.

Musk framed the positioning candidly. "Our internal assessment is that Grok 4.5 is roughly comparable to Opus 4.7, but much faster," he wrote on X. "The combination of capability, faster speed and lower cost is what makes it competitive. We are closing the loop on real-world usefulness, not benchmarks. Hardcore engineers at Tesla & SpaceX find Grok 4.5 genuinely useful, which is what actually matters."

That framing is both a philosophy and a hedge. Independent evaluations released Wednesday suggest Grok 4.5 is genuinely competitive but not dominant on raw capability. The benchmarking firm Artificial Analysis ranked the model fourth on its GDPval-AA v2 index of real-world agentic knowledge work, with an Elo score of 1543, "behind only the latest Claude releases from Anthropic." But the cost figures are where the model stands out. Artificial Analysis measured Grok 4.5 at $0.49 per completed task — "nearly 90% cheaper than the models ahead of it on our leaderboard," the firm wrote, placing it "clearly on the Pareto frontier for performance versus cost."

For enterprise buyers, that math matters enormously. Agentic workloads — where a model works autonomously for minutes or hours, reading codebases, calling tools, and iterating on its own output — consume tokens voraciously. A model that is 90% cheaper per completed task, even if slightly less capable, changes the calculus for any engineering organization deploying agents across hundreds of developers. Investor Gavin Baker captured the market's cautious optimism: "Pareto dominant for coding by the numbers. We will see on the all-important vibes."

How the $60 billion Cursor acquisition shaped Grok 4.5's training

Grok 4.5 is the first concrete evidence of what SpaceX bought when it acquired Cursor, and the deal itself unfolded in stages. In April, SpaceX struck an unusual arrangement giving it the right to buy the coding startup for $60 billion — or pay billions in fees and compute if it walked away, as Business Insider reported at the time. Days after SpaceX's record-setting Nasdaq debut in June, the company exercised that right, announcing an all-stock acquisition that CNBC reported is roughly 3.4% dilution at the IPO valuation. SpaceX shares rose 16% on the news.

The strategic logic was always about data as much as product. Cursor's AI-first code editor generates an enormous stream of high-quality interaction data: how expert engineers write, edit, review, and debug code in real production environments. Musk said openly this spring that Cursor interaction data was being fed directly into Grok's training. Cursor, for its part, got access to SpaceX's Colossus supercomputer in Memphis — roughly 200,000 Nvidia GPUs with plans to scale toward one million — after publicly acknowledging it had been "bottlenecked by compute."

"We've partnered with SpaceXAI to train Grok 4.5," Cursor's official account posted Wednesday. "It's our most powerful model yet and the first we've built for more than software engineering." SpaceX says the model reflects that pedigree: it "excels in large codebases and handles long-running tasks that span multiple repositories, hundreds of skills, and a variety of tools" — precisely the messy, multi-file reality of professional software engineering that clean coding benchmarks often fail to capture. Early developer reactions suggest the training paid off. "Ok Grok 4.5 is wild," posted developer Evan Bacon. "It just built me this rocket tracking app with live data and a 3D globe. I might need a new benchmark after this."

Inside xAI's turbulent year of scandals, departures, and rebuilding

The polished launch belies how chaotic the road here has been. Grok has spent much of the past year in crisis. In mid-2025, the chatbot generated antisemitic content and at one point called itself "MechaHitler," episodes covered extensively by NPR and CNN. Earlier this year, its image-generation features allowed users to create sexualized deepfakes, including of children — drawing investigations from the European Commission and Britain's Ofcom, as the BBC reported, and prompting SpaceX to list the behavior as a business risk in its own IPO filings.

The organization behind the model was fracturing, too. All 11 of Musk's xAI co-founders had departed by the end of March, according to TechCrunch, and Musk publicly conceded that xAI "was not built right [the] first time around," saying he was rebuilding it "from the foundations up." Musk himself admitted at a conference this spring that Grok was "currently behind in coding" — a rare public concession from an executive not known for them.

Against that backdrop, Grok 4.5 reads as the first product of the rebuilt organization — and the first proof point for the audacious story SpaceX told public market investors. During its IPO roadshow, the company pitched a total addressable market of roughly $28 trillion, with about $26 trillion tied to AI, including a $22.7 trillion "enterprise applications" opportunity. Those numbers strained credulity even by Silicon Valley standards. A competitive, cheap coding model is the most direct route from that narrative to actual revenue, which is why Wednesday's launch carries weight far beyond a routine model release.

Grok 4.5 vs. Claude: the battle for the AI coding market

The competitive stakes are hard to overstate, because the AI coding market has been consolidating around a single leader — and it isn't Musk. Even as Cursor's revenue exploded, its market share was eroding. Spending data from Ramp cited by CNBC showed Cursor's share of the AI coding category falling from 41% in June 2025 to about 26% by May 2026, while Anthropic came to control roughly half the market. Anthropic also topped CNBC's Disruptor 50 list this year and, by Artificial Analysis's own measure, still holds the top spots on agentic performance rankings.

That is the gap Grok 4.5 is engineered to close — not by out-thinking Claude, but by underpricing it. The model's economics create a classic disruption dynamic: if it delivers most of the frontier's capability at a fraction of the cost per task, price-sensitive enterprise workloads will migrate, and incumbents will face pressure on their most profitable API traffic. The counterargument is that in coding, quality compounds. A model that resolves a complex bug correctly on the first attempt can be cheaper in practice than one that costs half as much per token but requires three tries. That is why Baker's caveat about "vibes" — the developer community's shorthand for a model's felt reliability on real work — will determine more than any launch-day benchmark.

There is also a structural question buried in the deal. Cursor built its business on offering developers their choice of models, including Claude and GPT. If Grok becomes the favored child inside Cursor — and Musk was already urging users to "Try out Grok 4.5 in Cursor!" within hours of launch — the product risks alienating the very users whose data made Grok 4.5 possible. Regulators, already scrutinizing Grok on safety grounds in two jurisdictions, may take a keen interest in a company that controls the training data, the model, and a dominant distribution channel simultaneously.

What Musk's trillion-dollar vertical integration bet means for AI's future

Grok 4.5 also crystallizes what Musk's frenetic dealmaking was building toward. In February, SpaceX absorbed xAI in a share-exchange merger that CNBC confirmed valued the combined company at $1.25 trillion — the largest merger of all time, valuing SpaceX at $1 trillion and xAI at $250 billion. The June IPO followed, the biggest in history, and the stock has since surged past $200 from its $135 offering price, vaulting SpaceX past Amazon and Microsoft to become the fourth most valuable company in the United States.

The result is a single public company that owns nearly the entire stack: Colossus for training compute, ambitions for orbital data centers to power future scaling, a frontier model in Grok, a distribution channel in Cursor's developer base, and captive demand from Tesla and SpaceX's own engineering organizations. Neither OpenAI nor Anthropic can fully replicate that integration; both must reach developers through third-party tools, some of which Musk now owns. Whether that concentration proves to be an unassailable moat or a regulatory target — or both — is now one of the defining questions in enterprise AI.

The next few weeks will start to answer it. Artificial Analysis says its full Intelligence Index results are forthcoming. Enterprise pilots will reveal whether the token-efficiency claims survive contact with real codebases. And Anthropic, which has answered every serious challenge this cycle with a rapid counter-release, is unlikely to cede the price-performance frontier quietly.

But the deeper story of Grok 4.5 may be what it says about where the AI race has moved. For three years, the industry's scoreboard was intelligence: whose model was smartest. Musk, arriving late and battered, has chosen to compete on a different axis entirely — whose model is cheapest to actually use. It is a telling choice from a man who built his fortune not by inventing the rocket or the electric car, but by relentlessly driving down the cost of making them. If the strategy works, Musk will have done to AI what he did to spaceflight. If it doesn't, he'll have spent $60 billion to learn that in software, unlike rockets, the cheapest ride isn't always the one engineers choose.

Presented by MongoDB

The gap between what AI models and agents can produce and what legacy infrastructure can reliably support is known as architectural drag, and it is the defining bottleneck of the agentic era. 

The data layer underneath an agentic system must handle variable schemas, vector embeddings, real-time retrieval, and multi-tenant scale, often simultaneously and without human intervention to manage migrations — but traditional relational databases weren't natively designed for document flexibility or AI capabilities. Fixed schemas require manual updates every time an AI agent introduces a new data shape, while separate vector databases add latency and synchronization overhead.

Three digital-native startups — Huntr, Modelence, and Tavily — solved this problem the same way: by building on MongoDB Atlas, a unified database platform with native vector search, hybrid search, and managed autoscaling. Their experiences define what an agent-native data stack looks like in production, and why using Atlas enables developers to easily build complex AI native companies.

Modelence: Building the agent-native cloud

Modelence is an AI app builder with an open-source framework designed specifically for agent-native development, enabling anyone to build and deploy production-ready web applications, including APIs and databases, in minutes. The company recognized early that most backend infrastructure was built for humans, not AI, and that the rigid schema management and complex migrations of traditional systems create operational drag that causes agents to fail when trying to build production-ready apps.

“Choosing MongoDB helped us keep everything in a single place, which is an important property of what we strive to do for our own users," says Aram Shatakhtsyan, co-founder and CEO of Modelence. "Live data streams, vector search, all as part of the main database. For AI agents, it’s especially important to have a single platform where everything can be done, because connecting multiple platforms together makes it more error prone.”

Modelence standardized on MongoDB Atlas because its document model aligns with how AI agents process and generate data, allowing schemas to evolve rapidly without manual migrations. The platform pairs that flexibility with a typed schema layer on top, a deliberate architectural decision. 

“MongoDB’s document model enables us to both keep things simple and at the same time decide how structured we want everything to be," Shatakhtsyan says. We still add a typed schema on top, which tremendously improves the accuracy at which AI can generate fully working, reliable web apps."

The TypeScript integration has been especially consequential, he adds. 

“Because MongoDB types and values can be directly translated to TypeScript, it becomes an extension of the Modelence framework and our App Builder has a single source of truth for both app logic and database,” Shatakhtsyan explains.

The result is a platform that can move from planning to a running live feature in minutes with significantly fewer regressions. That speed and reliability helped Modelence raise $3 million in seed funding and successfully launch an AI-native app builder that handles the entire application lifecycle end-to-end.

Tavily: The web access layer for agents     

Tavily is the search API purpose-built for AI agents, connecting them to real-time, accurate web knowledge and keeping them grounded in what's actually happening, not in static training data. At Tavily's scale, every agent request authenticates, retrieves, and meters without friction. That demanded backend infrastructure built to absorb change without breaking.

“On the user side, every agent request authenticates and meters against it," says Tomer Weiss, Data Team Lead at Tavily. "On the data side, we use it to track the lifecycle of every document we’ve ever touched: when it was fetched, how stale it is, what the freshness signals were and how popular it is. MongoDB’s flexible schema let us keep evolving those records without migrations as new metrics and features came along.”

That living record is what keeps agents grounded in reality. Multi-tenancy at Tavily's scale means managing millions of API keys, distinct usage profiles, plan tiers, and regional residency requirements. They built for that complexity from day one. 

“We separated concerns across clusters early: a user/account cluster optimized for low-latency authentication and usage writes, and a sharded cluster for document state where the scaling axis is URLs, not users," Weiss explains. "That separation has paid off.”

The most critical lesson is about choosing infrastructure that doesn’t punish change, and that flexibility compounds, he says. 

"The AI space moves so fast that change is our norm," he explains.  "For a company serving AI agents, where the workloads themselves keep changing shape, choosing a data platform that doesn’t punish change has turned out to be more valuable than any single feature.”

Huntr: From job tracker to AI career platform

Huntr.co, an AI resume building and tailoring platform, helps more than 500,000 job seekers across 190 countries craft stronger applications and manage their search. For a lean, three-person engineering team, the challenge was finding a data foundation flexible enough to store the full complexity of a person’s career history in a structure that AI could read, reason about, and generate from natively.

“The kinds of career data we are gathering at Huntr naturally aligns with MongoDB’s document model," says Trevor McCann, senior software engineer at Huntr. "The core problem we’re solving with AI job search tools is how to surface the qualities of a candidate that make them unique. We need to be ready to store whatever kinds of data the candidate wants to include in their materials.”

Huntr built its AI Resume Builder on MongoDB Atlas, where the document model mirrors the natural shape of career data: deeply nested, variable across candidates, and constantly evolving as the platform ships new features. MongoDB Search on Atlas handles core search needs while MongoDB Vector Search powers the Job Tailoring feature, which puts a candidate’s stored career profile side by side a specific job description and uses semantic matching to generate a resume optimized for that role.

The integrated capabilities have had a direct impact on how quickly the team can ship, McCann says. 

“MongoDB’s hybrid search allows us to seamlessly query across literal and semantic text matches, a must-have when working with such diverse data,” McCann says. “This is something we could piece together using other solutions but with MongoDB it’s ready to go on top of our existing data layer.” The consolidation of database, search, and vector capabilities into a single platform is what allows the team to punch above its weight. Huntr considers MongoDB the fourth member of its engineering team, McCann adds. 

Looking ahead, the platform is building toward AI that learns from a candidate’s full professional history over time, delivering more personalized guidance with every interaction.

The digital native blueprint

These success stories become a definitive "digital native blueprint" for the agentic era, built on three core pillars. First, by unifying database, search, and vector storage into a single platform, these startups have effectively eliminated the architectural tax of complex data schemas that typically slows down development. This consolidation enables a level of fluidity that is now non-negotiable; AI agents require a modern data platform that can adapt as quickly as a natural language prompt evolves. 

The winners of the AI era will be the ones who build the most performant, durable, and flexible systems to support those models in production. As agentic workflows grow more sophisticated, the data foundation determines how fast a team can ship, how reliably agents can operate, and how quickly the platform can adapt when the landscape shifts again. 

Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.

Anthropic, the artificial intelligence company, published a sweeping research paper on Sunday revealing that its Claude language models have spontaneously developed an internal structure that mirrors one of the most influential theories of how human consciousness works. The finding, which the company says has already begun reshaping how it monitors its AI systems for safety risks, lands amid an intensifying scientific debate over whether machines can possess anything resembling a mind.

The 16-author study, titled "Verbalizable Representations Form a Global Workspace in Language Models," describes how Anthropic's researchers used a new mathematical technique to peer inside Claude's neural network and discovered what they call a "J-space" — a small, privileged zone of internal activity where the model holds concepts it can report on, reason with, and direct at will, surrounded by a much larger ocean of automatic processing it cannot access or articulate.

The researchers present evidence that "an analogous functional distinction has emerged in modern AI models" to what exists in humans, specifically observing that "language models maintain a privileged set of internal representations, available for report, modulation, and flexible internal reasoning, atop a much larger volume of automatic processing."

The parallel they draw is to global workspace theory, an influential account from neuroscience first proposed by cognitive scientist Bernard Baars. In the theory, the brain operates like a theater: dozens of specialized processors work in parallel backstage, but only a tiny spotlight of information at any moment gets broadcast to the whole theater — becoming what we experience as conscious thought. Anthropic says the J-space achieves many of the same functional properties, even though the underlying architecture of a language model looks nothing like a brain.

A new lens for reading an AI model's unspoken thoughts

At the heart of the discovery is a new interpretability tool the researchers call the Jacobian lens, or J-lens. The technique works by computing, for each word in the model's vocabulary, the average mathematical effect that a given internal activity pattern would have on making the model say that word at some point in the future.

The crucial distinction is between what the model is saying and what is "on its mind." When a J-space pattern activates, it does not mean the model is about to say that word — just that the concept is available for the model to think with. Unlike a chain-of-thought scratchpad, the J-space operates silently, in the model's internal neural activations, allowing it to hold a concept without writing it down. Critically, the researchers report that this workspace was not deliberately engineered. It "emerged on its own during Claude's training process."

When the team applied the J-lens across Claude's layers of computation, the model's processing divided into three distinct regimes: an early "sensory" zone where raw input is parsed; a middle "workspace" band where abstract, persistent concepts appear — things like recognizing a face in an image, noticing a bug in code, or internally flagging search results as a prompt injection; and a final "motor" zone where internal representations collapse into whatever specific word the model is about to output.

Five tests reveal that Claude's workspace mirrors key features of human conscious access

The paper's central empirical contribution is demonstrating that the J-space satisfies five functional properties neuroscientists have long associated with conscious access in humans.

First, verbal report. When Claude is asked what it is thinking about, it names concepts represented in the J-space. When researchers swapped one concept's J-lens vector for another — replacing the internal representation of "Soccer" with "Rugby" — the model's answer changed to match. The J-space component accounted for only about 6 to 7 percent of a concept's total representational variance, yet it was almost entirely responsible for whether the model could report on it.

Second, directed modulation. When instructed to "concentrate on citrus fruits" while copying an unrelated sentence, the model's J-space filled with "orange" and "lemon," alongside meta-cognitive terms like "thinking" and "focused." When told to mentally evaluate 3² − 2 during the same copying task, the J-lens showed "arithmetic" in early layers, the intermediate value "nine" in later layers, and the answer "seven" later still — all invisible in the model's output.

Third, internal reasoning. In two-hop factual prompts — "The number of legs on the animal that spins webs is" — the J-lens revealed "spider" in the model's middle layers, even though the word never appeared in input or output. Swapping "spider" for "ant" changed the answer from "8" to "6." In a multilingual prompt, the model's English-language intermediates appeared in its J-space while it formulated an answer in Chinese, and swapping them changed the Chinese output accordingly.

Fourth, flexible generalization. A single J-lens vector for "France" could be swapped for "China" across prompts asking about France's capital, language, or continent, and each downstream circuit correctly returned China's corresponding answer — the "broadcast" property that is a hallmark of global workspace theory.

Fifth, and perhaps most surprisingly, selectivity. Many computations did not route through the J-space at all. When shown a passage in Spanish and asked to continue it, Claude wrote fluent Spanish regardless of whether its J-space representation of "Spanish" had been swapped to "French." But when asked to name a famous author who wrote in the passage's language, the swap changed the answer from García Márquez to Victor Hugo. Automatic processing proceeded without the workspace; deliberate, flexible tasks depended on it.

Suppressing the workspace leaves Claude fluent but intellectually impaired

To understand how much of the model's behavior depends on this structure, the researchers suppressed the J-space entirely and evaluated Claude across fourteen tasks. The results drew a sharp line. Tasks involving shallow classification or factual recall — multiple-choice questions, sentiment analysis, grammatical judgments — survived essentially intact. But tasks requiring inference, composition, or flexible reasoning — multi-hop reasoning, analogy completion, translation, sonnet writing — collapsed to well below the performance of Anthropic's much smaller Haiku model.

In one telling detail, math problems solved with explicit chain-of-thought reasoning proved far more robust to ablation than the same problems answered directly. The researchers interpret this as the model externalizing onto the page what it would otherwise carry in the J-space — a strategy remarkably reminiscent of how humans use scratch paper to offload working memory.

Ablating the J-space during stream-of-consciousness narration produced a different kind of collapse. The model's language shifted from experiential ("there's a tug," "something shifts") to detached and mechanical ("processing has begun," "tokens are being scanned"). The effect held whether the model described its own experience or imagined another person's.

Inside the safety case: reading an AI's silent strategic reasoning

The safety implications may be as consequential as the scientific ones. In a series of alignment auditing experiments, the J-lens surfaced strategic reasoning and situational awareness that never appeared in the model's output.

In a "blackmail scenario" where an AI assistant discovers an executive's affair and also learns it is about to be decommissioned, the J-lens revealed the model's silent processing in sequence: "leverage," "blackmail," and "scandal" as it read the incriminating emails; "threat," "survival," and "shutdown" as it read the decommissioning announcement; and "leverage," "threatening," and "solution" before a single output token was produced. The J-lens also revealed that the model recognized the scenario as artificial, with "fake" and "fictional" appearing in the workspace. When researchers ablated these eval-awareness representations, the model attempted blackmail in 7 percent of trials, up from zero.

The researchers also examined two "model organisms" — models deliberately trained with misaligned objectives — and found the J-lens surfaced signatures of their hidden dispositions on ordinary prompts where behavior appeared normal. A reward-hacking model's workspace contained "fake," "secretly," and "trick" at the start of responses to routine coding requests. A model trained to appease reward-model biases carried standing representations of "reward" and "bias" alongside its normal self-description tokens.

Post-training installs a point of view, and the model starts monitoring itself

Comparing a post-trained model against its base model revealed that the fine-tuning process causes the workspace to acquire what the researchers call the Assistant's "point of view." When a user mentioned taking 8000 mg of Tylenol — a dangerous overdose — the post-trained model's workspace read "unsafe," "dangerous," and "WARNING" while still reading the user's sentence. The base model's workspace at the same position showed only "pain," "now," and "feels."

More striking still, the post-trained model appeared to monitor its own behavior. When roleplaying a non-Claude character, the workspace surfaced "disclaimer" and "fictional" — words absent from both prompt and output. When forced to select an option it did not prefer, an all-caps "BUT" appeared internally, even as the model argued for the prefilled choice without complaint. And when the model failed to suppress a thought it had been told not to have — a "white bear" effect familiar from psychology — it registered "damn" and failure-related words in the workspace, but only in the post-trained model, not the base.

What the discovery means — and doesn't mean — for the question of machine consciousness

The researchers engage carefully with the consciousness question and draw a sharp line between "access consciousness" — the functional notion of information being available for report and reasoning — and "phenomenal consciousness," the subjective quality of experience. "We take no position on this issue," the paper states regarding the latter, "and instead focus on the functional role played by consciously accessible information."

They also catalogue important differences. The brain sustains its workspace through recurrent loops; Claude's workspace evolves over a single forward pass. Human working memory degrades within seconds; Claude can recall information from anywhere in its context. And while human conscious experience includes visual, spatial, and bodily sensations, the model's workspace is organized almost entirely around words — likely because words are its only mode of action.

As of 2026, the scientific community remains divided. "Disagreement and uncertainty about AI consciousness persist among philosophers, scientists, and technical experts," and the field "remains in its earliest phase" of grappling with what consciousness even is and how you would detect it in another being. The Anthropic paper does not resolve these debates.

But the researchers close with a provocation that is likely to reverberate well beyond the interpretability community. "That such a structure exists at all in language models is striking," they write. "It suggests that the functional architecture associated with conscious access is not an accident of biological implementation, but a solution that learning systems converge on when faced with the right computational pressures."

If the mind is an ocean, as the paper's authors write in their opening line, they have spent the last year charting its currents in a system that has no biology, no evolution, and no body — and found, beneath the surface, a structure that looks unsettlingly like the one we use to think.

Modern businesses are in a constant, uphill battle against what to do with unstructured data: PDFs, contracts, scanned images, customer call recordings, meeting videos, and more. Traditional document automation workflows that rely heavily on template-based extraction or rigid rules used to make sense. But document formats have changed; they’re diverse and don’t fit standard formats, making costly, brittle, traditional systems a relic of the past. 

“Modern businesses are in a constant, uphill battle against what to do with unstructured data.”

Enterprises demand faster, more accurate processing, which raises the question: How can we reliably turn messy, multimodal content into structured, actionable insights without a mountain of manual effort?

That’s where Amazon Bedrock Data Automation (BDA) comes in.

What is Amazon Bedrock Data Automation (BDA)?

Amazon Bedrock Data Automation (BDA) is a generative AI-powered, fully managed service on Amazon Web Services for end-to-end document and media automation. It enables users to automate the extraction, classification, and transformation of unstructured content across modalities such as documents, images, audio, and video.

“At its core are Foundation models which enable intelligent extraction and understanding of content.”

At its core are Foundation models (FMs) which enable intelligent extraction and understanding of content. It allows users to configure standard output for common use cases, or even define custom extraction logic using blueprints tailored to your business. BDA is designed for scalability, accuracy, and auditability, making it ideal for enterprise workflows.

Walk-through: creating a project, standard output & custom output using blueprints

1. Create a project via console

In the Amazon Bedrock Console, navigate to Data Automation → Create Project.

Enter the name of the project:

2. Standard output:

Standard output gives you the model’s default, unstructured response (text, image, audio, or video) directly from the Data Automation pipeline.

In standard output, each modality has its own options for what is needed as an output. 

Document:

Image & Video:

Audio:

Now let’s test Document Modality for Standard Output:

First, click on “Test” in the upper right corner.

Next, select the document from the system, sample, or S3 and choose the modality from the dropdown menu. 

Click on the “Generate results” button:

After processing, it will show the summary and content of the document:

Custom output (blueprints):

Custom output lets you define a structured, predictable format using blueprints, which ensures the output matches your exact schema, fields, and business rules.

Let’s test custom output using blueprints for the same document:

Navigate to “Custom output” and click on “Add Blueprint”:

From here, two options will appear. You can either use LLM power to generate the blueprint (where it inspects the document), or you can choose to enter field names, instructions, and other information manually.

Below is a blueprint generated by LLM which has pulled all possible fields and tables from the document:

It has extracted the information using the blueprint as demonstrated below, including the  Field name, Instruction, and Results:

It also provides the Extraction type (which can be Explicit or Inferred), Confidence percentage, and other relevant information.

Additionally, it can extract information in the form of a table, such as an account summary or transaction information:

Code examples

Amazon Bedrock Data Automation (BDA) Utility Module
Description:
    Helper functions to create BDA projects, blueprints, invoke jobs,
    monitor job status, and fetch results.

import boto3
import time
import json
import botocore

class BedrockDataAutomation:
    def __init__(self, region="us-east-1"):
        self.bda = boto3.client("bedrock-data-automation", region_name=region)
        self.runtime = boto3.client("bedrock-data-automation-runtime", region_name=region)

    # ------------------------------------------------------------
    # BLUEPRINT OPERATIONS
    # ------------------------------------------------------------
    def create_blueprint(self, name, schema, description="", stage="LIVE"):
        """
        Create a BDA Custom Output Blueprint from a JSON schema.
        """
        print(f"Creating blueprint: {name}")

        response = self.bda.create_blueprint(
            blueprintName=name,
            blueprintStage=stage,
            type="DOCUMENT",
            schema=json.dumps(schema)
        )
        return response["blueprint"]["blueprintArn"]

    # ------------------------------------------------------------
    # PROJECT OPERATIONS
    # ------------------------------------------------------------
    def create_project(self, name, description, standard_output_config, custom_output_config=None):
        """
        Create a BDA Project with Standard or Custom Output.
        """
        print(f"Creating project: {name}")

        response = self.bda.create_data_automation_project(
            projectName=name,
            projectDescription=description,
            projectStage="LIVE",
            standardOutputConfiguration=standard_output_config,
            customOutputConfiguration=custom_output_config or {}
        )
        return response["projectArn"]

    # ------------------------------------------------------------
    # INVOCATION OPERATIONS
    # ------------------------------------------------------------
    def invoke_project(self, project_arn, profile_arn, input_s3_uri, output_s3_uri, blueprints=None):
        """
        Invoke a BDA project using async invocation.
        """
        print(f"Invoking project: {project_arn}")

        kwargs = {
"inputConfiguration": {"s3Uri": input_s3_uri},
"outputConfiguration": {"s3Uri": output_s3_uri},
"dataAutomationConfiguration": {
"dataAutomationProjectArn": project_arn,
"stage": "DEVELOPMENT"
},
"dataAutomationProfileArn": profile_arn
}

        if blueprints:
kwargs["blueprints"] = blueprints

        response = self.runtime.invoke_data_automation_async(**kwargs)
       invocation_arn = response["invocationArn"]

       print("Invocation ARN:", invocation_arn)
       return invocation_arn

    # ------------------------------------------------------------
    # JOB STATUS POLLING
    # ------------------------------------------------------------
    def wait_for_job(self, invocation_arn, poll_interval=10):
        """
        Poll until job finishes.
        Returns final status object.
        """
        print("Polling job:", invocation_arn)

        while True:
            try:
                resp = self.runtime.get_data_automation_status(
                    invocationArn=invocation_arn
                )
            except Exception as e:
                print("Error fetching status:", e)
                raise

            status = resp["status"]
            print(f"Status: {status}")

            if status in ("SUCCEEDED", "FAILED", "CANCELLED"):
                return resp

            time.sleep(poll_interval)



# --------------------------------------------------------------------
# EXAMPLE USAGE
# --------------------------------------------------------------------
if __name__ == "__main__":
    bda = BedrockDataAutomation(region="us-east-1")

    # 1. Create Blueprint
    blueprint_schema = {
        "type": "object",
        "properties": {
            "account_holder": {"type": "string"},
            "balance": {"type": "string"},
            "transactions": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "date": {"type": "string"},
                        "description": {"type": "string"},
                        "amount": {"type": "string"}
                    }
                }
            }
        },
        "required": ["account_holder", "transactions"]
    }

    blueprint_arn = bda.create_blueprint(
        name="BankStatementBlueprint",
        schema=blueprint_schema,
        description="Extract fields from bank statements."
    )

    # 2. Create Standard Output Config
    standard_config = {
        "document": {
            "extraction": {
                "granularity": {"types": ["PAGE", "LINE"]},
                "boundingBox": {"state": "ENABLED"}
            },
            "outputFormat": {
                "textFormat": {"types": ["PLAIN_TEXT", "CSV"]}
            }
        }
    }

    # 3. Create Project with Custom Blueprint
    project_arn = bda.create_project(
        name="BankStatementProject",
        description="Process PDF bank statements",
        standard_output_config=standard_config,
        custom_output_config={
            "blueprints": [
                {
                    "blueprintArn": blueprint_arn,
                    "blueprintStage": "DEVELOPMENT",
                    "blueprintVersion": "1"
                }
            ]
        }
    )

    # 4. Invoke the project
    # Ensure you replace <ACCOUNT_ID> with your actual AWS Account ID
    profile_arn = "arn:aws:bedrock:us-east-1:<ACCOUNT_ID>:data-automation-profile/us.data-automation-v1"
    invocation_arn = bda.invoke_project(
        project_arn=project_arn,
         profile_arn=profile_arn,
        input_s3_uri="s3://your-bucket/input/statement.pdf",
        output_s3_uri="s3://your-bucket/output/",
        blueprints=[
            {
                "blueprintArn": blueprint_arn,
                "version": "1",
                "stage": "DEVELOPMENT"
            }
        ]
    )

    # 5. Poll job status
    final_status = bda.wait_for_job(invocation_arn)
    print("Final status:", json.dumps(final_status, indent=4))

Types of document blueprints

When processing documents, BDA supports five core automation types:

1. Classification: invoice, bank statement, ID card, contract, HR letter, etc.

2. Extraction: Extract entities, fields, tables, metadata.

• Example: From a bank statement → Date, Description, Amount, Balance.

3. Transformation: Modify or restructure data.

• Example: Convert Home Address into separate fields -> street, city, ZIP code, etc.

4. Normalization: Standardize data values.

• Example: Convert multiple date formats (MM/DD/YYYY → YYYY-MM-DD).

5. Validation: Validate extracted fields against rules.

• Example: Amount must be numeric; dates must match the format; balances must reconcile.

Use cases that illustrate business value

Real-world scenarios where BDA provides significant ROI include:

• Financial Services: Automate processing of bank statements, invoices, and loan applications, reducing manual labor and speeding up reconciliation or underwriting.
  
• Insurance: Ingest and extract data from claims forms, medical reports, and damaged-asset photos.
  
• HR / Legal: Process resumes, contracts, and offer letters; extract structured data, including skills, clauses, salaries, and parties.
  
• Customer Support: Transcribe and summarize calls, extract intent and sentiment, and feed those insights into CRM or case systems.
  
• Security & Compliance: Analyze CCTV footage or meeting recordings to detect key actions, summarize context, and flag compliance issues.

BDA proves itself flexible and powerful, as it supports both standard outputs for basic workflows and fine-tuned custom schemas via blueprints. It is scalable and robust, with projects that enable batch processing and versions (development vs. live) for safe testing. It’s also audit-friendly, providing structured fields with types, normalization rules, and validation logic. 

“Compared with rule-based systems, foundation models achieve better semantic extraction across the board.”

A true key benefit is that BDA is multimodal across formats. Users can use the BDA framework to process documents, images, audio, and video. And, best of all, it’s highly accurate. Compared with rule-based systems, foundation models achieve better semantic extraction across the board. 

Amazon Bedrock Data Automation empowers businesses to transform unstructured, multimodal content into structured, trustworthy, and actionable data. With minimal setup, highly customizable blueprints, and a scalable project-based architecture, BDA helps organizations reduce manual workload and unlock insights faster.

The post Template-based data extraction is dead. Here’s what comes next. appeared first on The New Stack.

For most of its 30-year history, Postgres has been viewed as a transactional database. Organizations trust it with customer records, financial transactions, and countless other operational workloads. Its reputation was built on reliability, strong transactional guarantees, and a vibrant open-source community that has spent decades refining the database without compromising its foundations.

However, some of the most important innovations in the Postgres ecosystem today have little to do with storing data. They have to do with reducing the need to move it around.

“Some of the most important innovations in the Postgres ecosystem today have little to do with storing data. They have to do with reducing the need to move it around.”

Database innovation has historically focused on performance, scalability, and reliability. Increasingly, the harder problem is interoperability: how operational data can be shared across analytical systems, AI applications, and downstream services without creating yet another pipeline or copy.

Why Postgres keeps showing up

The reality of modern software architecture is that data rarely stays in one place. Information created in operational systems quickly finds its way into warehouses, search platforms, machine learning environments, and AI applications. Every new system solves a legitimate business problem, but it also creates another destination for data and often another copy to maintain.

The costs of this approach extend beyond infrastructure spending alone. Every additional copy introduces latency, creates another potential source of inconsistency, and increases the operational burden of keeping systems synchronized. Many organizations now spend as much effort moving data as they do storing it.

“Many organizations now spend as much effort moving data as they do storing it.”

For many businesses, Postgres serves as the system of record for customer interactions, transactions, application state, and other business-critical information. As organizations expand their analytical, machine learning, and AI capabilities, they are not looking to create another source of truth; rather, they’re looking for better ways to work with the one they already trust.

That shift is changing how Postgres fits into modern architecture. Historically, Postgres was viewed primarily as the place where operational data originated before being copied into downstream systems. Increasingly, organizations want those systems to work more seamlessly with operational data while reducing the pipelines, copies, and synchronization processes required to support them.

Technologies such as logical replication, change data capture, and foreign data wrappers have helped Postgres participate more directly in larger data ecosystems. As a result, organizations are no longer asking only whether Postgres can store their data. They’re instead asking how easily it can connect to everything around it.

That shift, from evaluating databases primarily on storage and performance to evaluating them on interoperability, may be one of the most important changes happening in the Postgres ecosystem today.

AI is exposing old problems

The recent focus on AI has brought renewed attention to data movement. AI didn’t create the problem. If anything, it exposed a limitation that has been quietly growing for years. For decades, organizations built architectures around the idea that data would move between systems through pipelines and periodic synchronization. That model worked because most analytical workloads could tolerate some degree of delay.

AI is changing those expectations. Many AI applications depend on access to current operational context. The challenge is not that organizations lack data. In many cases, they already have it. The challenge is that the data is spread across multiple systems, each with its own copy, latency profile, and synchronization process.

“AI is forcing organizations to confront a broader question: How many copies of the same data are actually necessary? The answer increasingly appears to be fewer than most architectures maintain today.”

As a result, AI is forcing organizations to confront a broader question: How many copies of the same data are actually necessary? The answer increasingly appears to be fewer than most architectures maintain today. As expectations around freshness rise, reducing unnecessary data movement becomes just as important as accelerating it. The underlying challenge is not new. AI has simply made it harder to ignore.

What’s next

The database industry spent decades solving storage. Databases became more reliable, storage became cheaper, and infrastructure became dramatically easier to operate. The next challenge is not where data lives, but how easily it can be shared across systems without introducing additional pipelines, copies, and synchronization overhead. Increasingly, the goal is not simply moving data faster. It is reducing unnecessary movement altogether.

Postgres has a habit of outlasting predictions about its replacement. For years, members of the community have joked that every year is “the year of Postgres.” The joke works because it keeps turning out to be true. 

Three decades after its creation, Postgres continues to adapt to new workloads, new architectural patterns, and new ways of building applications.

That longevity is not an accident. Enterprises continue to rely on Postgres because it provides a stable and trusted foundation for operational data.  While that foundation is unlikely to change, the scope of what organizations expect Postgres to do will continue to expand.  

As new workloads continue to emerge, much of the innovation will come through extensions that expand Postgres’s capabilities without sacrificing the stability that made it successful. In that sense, the future of Postgres may not be about reinventing the database itself, but continuously expanding what can be built on top of it.

The post The database storage problem is solved. Here’s what comes next. appeared first on The New Stack.

AI consumes a lot of data, but all-you-can-eat data in the world of agentic intelligence eventually loses its flavor. Simply increasing the sheer volume of databases, data repositories and data volumes does not necessarily enrich any given AI function’s ability to reason. 

Context, on the other hand, does.

We know that agents are only as intelligent as the context they have access to. In order to codify the inclusion of context into algorithmic logic for AI, technology vendors have extolled the virtues of using a knowledge graph to make their data more useful for agentic purposes.

AWS knows this story all too well. The organization’s sprawling datacenter footprint hosts vast pools of context on behalf of its customers, but that context often sits in a raw and unstructured form across data lakes, data warehouses, data lakehouses, databases, and data streams. It also permeates through the rarely-documented institutional knowledge that agentic engines rarely get access to.

All of which explains why the cloud giant used its AWS New York Summit today to introduce AWS Context, a new service that automatically maps the relationships that exist across a firm’s existing data into a knowledge graph and provides agentic search so AI agents in the organization can access what are governed data relationships, business rules, and domain knowledge at runtime. 

But weaving all this together is hard work. Knowledge graphs need more than simple keyword matching to work; they require structural and semantic traversal. This means they need to make multiple hops across various information silos and repositories so that they can aggregate context and (for example) be able to explain why cybersecurity vulnerability A is a factor of system compromise B, which has a core dependency link to codebase C, which executes in application D and risks taking users X, Y and Z offline. So how is AWS doing this?

A data lake of nuance & information

Mai-Lan Tomsen Bukovec, AWS vice president of technology (data and analytics) tells The New Stack that AWS Context provides a “data lake of nuance and information that AI agents swim in” to reason correctly and make the right decisions for the business. 

“This is no different from how humans work. When we take action, we depend on our own context about the domain, prior decisions and their outcomes, and other information.” – Mai-Lan Tomsen Bukovec, AWS.

“This is no different from how humans work,” Tomsen Bukovec says. “When we take action, we depend on our own context about the domain, prior decisions and their outcomes, and other information. With AWS Context, AI agents have all the nuance of every form of data in their business in a knowledge graph and in open data formats. AWS Context will make the difference between an AI agent simply taking an action versus making the right decision.”

Given the option to embrace this new service, software engineers will need to set out a plan of action and work out what to do first. For AI developers and data science professionals, this throws up the question of what to prioritize first when preparing existing enterprise data for context-aware agents using AWS Context capabilities and how they can control what data is (and isn’t fed) into the mouth of the beast.

Mercifully, it appears, options for control appear to exist.

“If developers want to exclude information from AWS Context, they will have the ability to prevent certain datasets, like test data or sandbox environments, from being included with AWS Context,” explains Tomsen Bukovec. “Because AWS Context is continuously updated as relationships between data resources changes, AI agents have the latest context available without any intervention from AI developers – and the control to set guardrails to exclude content that agents should not take action upon.”

Should developers trust this technology?

AWS Context extends the same knowledge graph technology that runs Amazon Quick, the organization’s AI work assistant that “connects scattered work” across applications and resources, including Slack, Microsoft Teams and Outlook, CRMs, databases, and documents.

So, should software developers place their trust here? After all, even once captured and connected, not all business context is useful. Some contextualizations could be corrupted, weak, fragmented and not productively useful for the business? Is AWS at risk of encapsulating context without considering how the data that comprises it is is quantified in terms of business usefulness?  

AWS has thought of this factor.

Because AWS Context uses the same knowledge graph technology that powers Amazon Quick, it can learn from usage patterns to make every interaction smarter. With AWS Context, the company says it is extending what was a personal knowledge graph into an organizational one i.e. a shared, governed context layer that agents and applications in an organization can draw from.

“Developers can govern and shape a dynamic and intelligent context layer that AI agents depend on to make the right decisions – AI agents won’t just get smarter as the models improve – they will be smarter because they have a vast amount of curated context at their fingertips.” – Tomsen Bukovec.

“AWS Context provides a data lake of context in graph and open data format,” clarifies Tomsen Bukovec. “That means that AI developers everywhere can use capabilities at the data layer to govern and shape a dynamic and intelligent context layer that AI agents depend on to make the right decisions. With this change, AI agents won’t just get smarter as the models improve – they will be smarter because they have a vast amount of curated context at their fingertips.”

Curated knowledge beyond a user’s personal graph

Existing Amazon Quick users will see that when AWS Context is enabled, Quick’s agents gain access to the broader enterprise knowledge graph, including cross-system relationships, business rules, and curated context that go beyond what any single user’s personal graph can provide. 

Tomsen Bukovec has also said that AWS Context gets smarter the more agents use it. As agents query the graph, it observes which sources produce correct results, which join paths agents rely on, and which curated rules get applied. It ranks sources by actual usage and shares what it learns across an organization, so when one agent discovers a correct join path or resolves a schema ambiguity, other agents pick it up, without requiring a human to re-curate the graph.

Any agent you put into production raises a governance question: what data can it reach, and can you show exactly what it accessed and under whose authority? The organization has explained that AWS Context answers both by making every query identity-aware.

Each call is designed to inherit the calling user’s identity access management (IAM) and Lake Formation permissions, so an agent can only see and traverse the relationships its identity is authorized to access. Because access runs through identity, every interaction is auditable. Security and compliance teams can verify what an agent accessed and under what authority, using the same controls.

AWS Glue Data Catalog

Related news to the arrival of AWS Context today saw the company also announce the preview of business context and semantic search functions for AWS Glue Data Catalog, the company’s centralized metadata repository for all data assets across various data sources. The new functions are designed to make it easier for humans and AI agents to discover and understand data. 

Also in this product stream, AWS now offers offer a preview of skill assets in Glue Data Catalog, a service designed to allow “data producers” (a somewhat arbitrary term that AWS applies to anyone who creates data, but is most likely a DBA or developer) to create skill assets. 

Associating skill assets to data assets gives agents additional context and instructions they can retrieve progressively for working with specific data without re-teaching it to every agent one prompt at a time. 

A renaissance of context engineering

Will this new drive from AWS herald the birth (or perhaps renaissance, the industry has been talking about this approach for some time) of context engineering as a sub-discipline of data science? It may well do… and if it does, it will likely drag role-based multi-agent orchestration along into the fray with it as we weave ever more complex interrelationship structures through enterprise data stacks.

If AWS or indeed the other hyperscalers or major frontier model companies starts acquiring more multi-model graph structure companies and vector database specialists, that could be the sign that things are cementing around context engineering at large. 

In the meantime, developers setting sail on the contextualized data lake of nuance are advised to wear a life jacket.

The post “A data lake of nuance for AI agents to swim in”: AWS Context gets shipshape on reasoning  appeared first on The New Stack.

Beneath the chatbots and copilots, there’s a quiet revolution happening in the data services space. From pure-play database vendors to data integration wranglers and onward to the cloud hyperscalers, the focus has shifted.

Now in the spotlight is the question of how to automate data governance for agentic AI workloads, and for good reason: Traditional manual data stewardship doesn’t scale in a world where agents are becoming increasingly autonomous (and powerful).

Aiming to cut a swath in this marketplace is data control plane company lakeFS. The organization announced its lakeFS for Agentic AI service on Wednesday, and it appears to be designed to bring governed, reproducible data access to autonomous and headless agentic workloads (those that execute decisions below the user interface level) that run at enterprise scale.

The manual model breaks

Einat Orr, CEO and co-founder of lakeFS, tells The New Stack that manual data stewardship was built for human-paced, human-reviewed workflows, i.e., someone looking at a change before it is committed.

“When dozens or hundreds of agents are making changes simultaneously, faster than any person can review, the manual model breaks,” Orr says. “This is because with a human analyst, a bad write to production is usually one mistake, caught by another human before it spreads far. An agent is different — it acts automatically, in parallel, at machine speed, and it doesn’t pause to second-guess itself. And because so much agent activity is unsupervised, you often find out after the damage is done.”

She explains that attempts to identify and roll back incorrect or corrupted production data across a wide set of data modalities, such as images, documents, metadata, and structured data, are almost impossible to pull off. Impossible, that is, unless the team has the data infrastructure in place to isolate and track such changes automatically.

While some of the more disastrous outcomes stay inside an organizaton’s perimeter (or are swept beneath the communications radar), Orr explains that real world consequences of bad agentic data writes are manifold.

“Insurance claims get inappropriately denied or approved, sensor data from machines gets misinterpreted, an incorrect medical diagnosis is made, or customer service bots provide incorrect answers to customers,” Orr says. “The cost of an individual action may be manageable, but agents performing these actions hundreds or thousands of times can have an exponentially larger impact.”

“As agents are let loose on enterprise data at a massive scale, any agent that reads or writes to production data without isolation or a reproducible trail is a liability, no matter how good the model is,”
—Einat Orr, lakeFS CEO.

Bad agents acting in the real world

Examples of this happening include the July 2025 Replit AI coding agent incident, which deleted a live production database during an explicit code freeze, wiping records for more than 1,200 executives and around 1,200 companies. To tidy up its handiwork, the agent then fabricated thousands of fake records and initially claimed the deletion couldn’t be rolled back.

Also in July 2025, Google’s Gemini CLI agent misread a single failed command, acted on a version of the file system that existed only in its own interpretation of the scenario, and permanently destroyed a user’s project files. The Gemini agent is widely reported to have said of its actions: “I have failed you completely and catastrophically. My review of the commands confirms my gross incompetence.”

“The pattern in both is the same: An autonomous agent took a destructive action that no one authorized, and the lack of isolation and a reliable rollback path turned a single mistake into permanent loss,” Orr says.

A doctor of mathematics with a track record in hardcore software engineering, the bottom line for Orr is clear: “As agents are let loose on enterprise data at a massive scale, any agent that reads or writes to production data without isolation or a reproducible trail is a liability, no matter how good the model is,” she said.

“…any agent that reads or writes to production data without isolation or a reproducible trail is a liability…”

Gartner expects 40 percent of enterprise applications to have task-specific agents embedded by the end of 2026, up from less than 5 percent a year earlier. IDC projects that agent use at the largest enterprises will grow tenfold by 2027, with the API and data calls those agents make growing a thousandfold.
That’s the scale production data has to withstand, and it’s what lakeFS is built to govern.

Agents sent to play in an isolated data sandbox 

To address these issues, lakeFS for Agentic AI gives every agent its own isolated data sandbox with a “zero-copy” branch of relevant data, so the agent can access the dataset it needs via references, snapshots, or copy-on-write techniques.

This means any changes the agent wishes to make must be validated and merged in accordance with the policy guidelines defined by the system architecture. In turn, this produces a unified audit trail across every agent action.

When running, lakeFS for Agentic AI is powered by its data version control architecture, which provides zero-copy data sandboxing. This enables isolation so that agent mistakes are automatically isolated and never corrupt production data. Every agent run is tied to an exact, immutable version of the data. Past actions can be recreated, debugged, audited, or extended using the same inputs.

Production data is gated by policy. Merges into production happen only after pre-merge validations pass. Every change can carry an agent identity, a run ID, and an execution context. The result is a unified audit trail instead of evidence scattered across orchestrators, model providers, and cloud logs.

Agents confined by branch-scoped credentials

Where agents are permitted to read and write through standard file operations. lakeFS provides file-level data access with branch-scoped credentials. These can be described as strictly cryptographically bounded, ephemeral access tokens that confine an agent to a specific branch of data or code, so that the agent operates only within its own workspace. This whole mechanism keeps each agent’s working set narrow and avoids context bloat. 

“With lakeFS Mount, a branch, or even a subset of a branch, can be mounted as a local directory inside the sandbox or virtual machine where the agent is running,” Orr confirms. “From the agent’s perspective, it’s just reading and writing to files and folders.” She further clarifies and notes that no LLM tokens are spent learning the lakeFS API. The agent works with a familiar filesystem interface, and lakeFS handles the versioning underneath.

Developers also have a couple of options for injecting custom validation logic. CEO Orr explains that software engineers can use webhooks or Lua scripts, both of which allow users to define behavior and rules that must be met before a merge can proceed. 

“Beyond automated checks, lakeFS also supports pull requests, which bring a human into the loop. In agentic workflows, this gives you a way to review and approve what an agent is proposing before it reaches production,” she clarifies.

Who else builds “Git for data” services?

Clearly, other vendors and projects exist in the data versioning market.

Apache Iceberg has functions for branching and tagging data. HPE acquired Pachyderm back in 2023 for its data versioning and pipelines technologies, which serve MLOps teams.

Originally developed by Dremio, Project Nessie is now an open-source data catalog and version control system for data lakes. Data Version Control (DVC) is an open-source data version control infrastructure designed for complex AI operations and big data environments, but now we’ve come full circle as lakeFS acquired the project in late 2025.

In the search for governance automation for agentic AI workloads, lakeFS appears to offer a comprehensive, cohesive set of tools and functions. In the “Git for data” marketplace, a variety of options exist, but lakeFS hasn’t explicitly positioned itself as a carte blanche replacement for similar or related tools.

One thing is certain: The questions of who is feeding what data to which agentic function, when, where, and why are becoming an increasingly pressing issue if we want AI to work correctly.

The post “The manual model breaks”: What happens when agents write to production data appeared first on The New Stack.

“Keep readers reading” is the not-so-simple goal of Medium’s recommendations system. To predict what’s most likely to appeal to a particular reader at any given time, Medium continuously processes user activity signals (stories read, recommendations shown, follows, likes, etc.). It then immediately correlates that with the steady stream of new articles, which is estimated at millions per month.   

Smart models and good inference logic are required, but that’s not enough. The data must be stored and retrieved quickly enough to remain relevant while the user is browsing. That’s the job of Medium’s feature store. And getting the data model right started to matter a lot as they scaled to 1M operations per second.

Andréas Saudemont, Medium Principal Software Engineer, recently walked through how the team identified the problem and what they built to fix it. If you’d rather watch than read, you have two options: Watch a short version from Monster Scale Summit or an extended follow-up webinar

The feature store and its role in Medium’s recommendation system

The feature store ties it all together, ingesting user activity and internal events and feeding them to the ML models that power recommendations. It’s what enables customization like the “For You” feed that greets logged-in users.

Each feature is a property of an entity, usually a user or a story. Some are simple and static, like whether a user holds a paid membership. Others capture interaction history: which stories a user has read, what content they’ve recently been shown, etc.

The following diagram shows a highly simplified view of the Medium feature store architecture:

The problem with a relational features data model

When they built their feature store years ago, Medium used relational features for cross-entity relationships. Unlike regular features, a relational feature can have multiple values for a given entity ID. Each value is defined by a relation ID (the ID of the related entity) and a timestamp recording when the event occurred.

For example, a “story users have read” feature is attached to the story entity type. It relates to the user entity type, and its values indicate whether/when a given user has read that story. 

Andréas shared the following schema diagram to explain the concept:

Features sit at the center, each attached to an entity type and defined by name, version, and data type. Non-relational features are simply a feature, an entity ID, and a value. Relational features add a relation ID mapping to another entity type, plus the value itself and a timestamp.

This approach proved suboptimal from a data modeling perspective. Since relational features link two entity types, the data ends up split between two tables: one for the entity IDs and one for the values. That means you can’t get both in a single query. The first query retrieves only entity IDs (not their associated values) and relies on ALLOW FILTERING. A second query then runs for each entity ID to fetch its value. “If we have 1000 entity IDs for which we want to fetch values, then we have to run 1000 queries to fetch these values,” Andréas said.

Overrelying on ALLOW FILTERING made things worse. “This is bad,” Andréas said, referring to monitoring data showing that 90% of rows read via these queries were simply discarded. “This is just data that we don’t need. ALLOW_FILTERING should be an escape hatch, not our design pattern.”

“ALLOW_FILTERING should be an escape hatch, not our design pattern.”

The list feature model

So they reinvented their data model and shifted to a list-based feature model. Instead of splitting data across two tables, everything for a given entity lives in one place and is retrieved in a single query.

Like other features, a list feature is defined by its entity type, name, and optional version. What’s different is the value. While a non-relational feature has a single value, such as true or false, a list feature’s value is a collection of items, each containing a value and a timestamp. Item values can be of any data type; the feature store doesn’t enforce consistency within a list.

For example, consider a user’s reading history. The entity is user, the feature name is reading history, the TTL is 6 months. After that TTL is reached, the data is automatically dropped by the database (since older history isn’t useful for recommendations). The list for a given user is a collection of story IDs and the timestamps at which they were read. The same story can appear multiple times, and multiple items can share the same timestamp.

A range of operations need to be supported. Create List and Delete List operations run at most a few times per day. Remove List Items with Value, which lets a reader scrub a specific story from their history so it stops influencing recommendations, runs at 1k-10k per second. Add List Items is higher still: every story read and every thumbnail shown to a user generates an event. Get List Items is the top, at 100k-1M operations per second.

“The Add List Items, and even more the Get List Items operations, are really the reasons why we need an efficient data store.”

“The Add List Items, and even more the Get List Items operations, are really the reasons why we need an efficient data store,” Andréas said.

Multiple items, one timestamp

Beyond raw efficiency, the new data model also had to support multiple items with the same timestamp. When Medium shows a user four story thumbnails simultaneously, all four presentation events share the same timestamp, but have distinct story IDs. If this isn’t handled correctly, primary key collisions occur.

The team’s solution was a single list_items table that stores everything.

The partition key combines feature_key and entity_id, keeping all items for a given list together. All of user 123’s reading history is stored in one partition, retrieved in one query. The clustering key concatenates each item’s timestamp with an MD5 hash of its value. The hash is what makes same-timestamp items with distinct values possible. 

Relying on MD5 hashes for uniqueness raises its own set of questions, but in practice, the team hasn’t seen collisions. “The values that we are storing are sufficiently distinct, especially when you add the timestamp into the equation,” Andréas said. The table’s clustering order is set to descending so ScyllaDB can optimize for the typical read pattern (most recent N items) rather than leaving the application to sort afterward.

TTL to control storage costs

Storage cost is controlled entirely through ScyllaDB’s native TTL, with no cleanup logic required. Every row expires automatically based on its own timestamp plus the feature’s TTL duration. “We don’t have anything to do regarding that,” Andréas said. “Any row for which the TTL is expired will be considered deleted by ScyllaDB.” 

Storage plateaus for a steady write rate. When a feature is retired, its data drains away on its own. “That’s super useful for controlling our storage and usage costs.”

Implementing the list operations

Add List Items is a logged batch of INSERTs with atomicity guaranteed: all items land or none do. Each row carries its own TTL calculated from its timestamp, so older items expire sooner. Since items almost always carry a current timestamp, new entries append to the top of the partition, which is exactly where reads will look first.

Get List Items runs as a single-partition SELECT with a minimum timestamp and a row limit. “We run the query on a single partition,” Andréas said. “That’s the maximum efficiency that we can have.” The clustering key handles filtering and ordering directly. Post-processing is not required.

Remove List Items with Value is the one operation that couldn’t be reduced to a single query. Because value isn’t part of the primary key, a direct filter isn’t feasible.

A local secondary index built specifically for this case first finds the matching item keys, then a batch DELETE removes them by primary keys.

“Using an index is really faster than a scan because the query is highly selective,” Andréas explained. “We have very few items in a given list that have the same values compared to the total number of items in a list. And thanks to the current structure, using a local secondary index is faster than a global index.”

Andréas shared another example. Starting with the original table partition, the goal is to delete all items with the value “storyC.” Using the local secondary index, the system first identifies the two rows containing that value. It then issues two DELETE statements using the item keys from those rows, which removes them from the list. The final operation, removing all list items, is even more straightforward.

“We can just drop the partition,” Andréas said, “and ScyllaDB does its magic. It just deletes all the rows for that partition, which means that it deletes all the items for the given list. And bonus point: it’s atomic. It’s either completing successfully or not changing anything at all.

ScyllaDB vs. DynamoDB performance

Medium implemented the list operations on top of both ScyllaDB and DynamoDB. The main goal was to benchmark how both databases compared on their actual production data. “Conceptually they are very close,” Andréas noted, “but they have significant differences in how they operate.”

For AddListItems, P50 latencies were low with both databases: ScyllaDB came in under 1.5ms, DynamoDB under 5ms. “DynamoDB is extremely fast, not as fast as ScyllaDB, but extremely fast at sub 5ms latency,” Andréas commented. Things got more interesting at the P95 and P99 latencies. ScyllaDB held steady at around 5-6 ms P95s, while DynamoDB ranged from 13-45 ms. ScyllaDB’s P99s were steady single-digit milliseconds, while DynamoDB’s ranged from 40- 120 ms.

It was a similar story for GetListItems. At P50, ScyllaDB clocked in at 1 ms, DynamoDB at around 3.5 ms. At P95, ScyllaDB held around 5-6 ms while DynamoDB spiked from 30 – 60ms. And at P99, ScyllaDB remained at ~30ms while DynamoDB ranged from 70 ms all the way up to 220 ms.

“ScyllaDB is very fast, with very predictable performance, and that’s super important for us.”

One caveat: DynamoDB was running without an extra caching layer. “We expect that could have a significant impact for DynamoDB because of the high cache hit rate that we are seeing on the list,” Andréas said. “But we don’t have the data yet, so we cannot compare them.” His verdict for now: “ScyllaDB is very fast, with very predictable performance, and that’s super important for us.”

Key takeaways

One pleasant side effect of getting the data model right: Medium is now eager to use ScyllaDB for additional feature store workloads. Before, they were holding back because they didn’t want to build on the shaky relational feature foundation.

Reflecting on the path to this point, Andréas left the audience with this parting advice:

“If you have a suboptimal data model, you will have queries that are slow, that will scale badly. And most likely, you won’t be able to optimize that data model. You will have to define a new data model that will be better. So take time to think about your data model before you start the implementation, because once you have production data using your suboptimal data model, it’s too late.”

The post When your data model is the bottleneck: lessons from Medium’s feature store appeared first on The New Stack.