As more companies plug AI agents into the deepest depths of their internal data banks, how can they be sure those agents actually understand how the business works? Right now, many of these organizations are stuck manually building a Markdown file, hoping they find time to rewrite it each time the business changes.

Modus, for its part, thinks it has found a better way. The startup that formally exits stealth this week with $10 million in funding in tow is building what is coming to be known in industry parlance as a “context warehouse” — a layer that sits alongside a company’s existing data warehouse, continuously mapping how the business operates across its systems, and handing an AI agent only the relevant slice of that map when it needs it.

In real terms, Modus crawls relevant assets from sources like GitHub, dbt, Jira, Snowflake, and Postgres, using what it calls a Context Miner to continuously learn how the business operates. What it finds gets turned into “dynamically generated skills”: Short, purpose-built briefs, assembled in real time by a second system, the Context Composer, and handed to an agent the moment it’s given a task.

Modus co-founder and CTO Tomer Mesika tells The New Stack that this mining runs continuously, guided by its own internal logic for what to check and how often.

“We have a lot of mechanisms in place to know what to mine from the organization, at what cadence, how to look for deltas, when to dive deeper in, and when not to,” Mesika says.

“We have a lot of mechanisms in place to know what to mine from the organization, at what cadence, how to look for deltas, when to dive deeper in, and when not to.”

Daniel Shimoni, Modus co-founder and CEO, draws a direct line to data warehousing to highlight the gap he’s trying to close. Companies have spent years building infrastructure to store and organize their data, he argues, but nothing equivalent exists for the understanding that sits atop it.

“There’s a logic behind data warehouses — companies already know that is where they manage their data,” Shimoni tells The New Stack. “But where do they manage their context? Where do they actually understand what contexts exist in their organization, that they can actually use to ensure agents only have what they need?”

Shimoni says even that first step is hard enough on its own. But keeping a company’s context accurate as the business changes is harder still.

“We’ve noticed that building the context the first time is already a challenge, but maintaining it is the bigger issue,” Shimoni says. “So Modus always learns from what the company is doing, and whenever something shifts or changes in the business, it makes sure that only the relevant and updated context is fed to agents.”

“Building the context the first time is already a challenge, but maintaining it is the bigger issue.”

Who’s buying, and why cost matters

Shimoni says Modus is targeting engineering teams, the CTO office, and VPs of R&D, as well as data teams and a newer category of AI teams.

“AI teams weren’t really around last year; it seems that a lot of data teams are transitioning to becoming VP of data and AI, or AI enablement,” Shimoni says. “So really, it’s the people who are in charge of having this AI enablement mandate in the organization, making sure AI is scaled in the organization.”

Pitching enterprises a shiny new context warehouse becomes much easier when the promise is steeped in helping them cut costs. Spend has become one of the defining anxieties of enterprise AI this year, with companies switching providers in pursuit of cheaper models, to entire economic models being built around the price of a token.

“You want the bigger models to do the heavy and complex tasks to get great value. The problem is that they are wasting a lot of their effort and a lot of their token usage on menial tasks.”

Mesika says this is a central component of Modus’s modus operandi, arguing that frontier models end up spending a chunk of their token budget on work unrelated to actually answering a question.

“You want the bigger models to do the heavy and complex tasks to get great value,” Mesika says. “The problem is that they are wasting a lot of their effort and a lot of their token usage on menial tasks.”

Those menial tasks, in Mesika’s telling, include combing through pull requests or Jira tickets just to determine what’s relevant before an agent can start the job it was assigned to.

One approach to this problem is to hand the sorting work to a smaller, cheaper model. Mesika says Modus takes that further: rather than retrieving that context at the moment a question is asked, it uses small language models alongside search engines, vector search, and a graph database, all built up in advance, to do that work continuously in the background. By the time an expensive frontier model gets involved, it’s only ever handed a finished brief of exactly what it needs.

“Everyone’s talking about context”

Shimoni and Mesika both come from data-centric companies — Lusha, a go-to-market data platform, and Cyera, a cybersecurity data company, respectively — before leaving their roles in September 2025 to start Modus together.

The two had known each other for years, and spent much of the previous year comparing notes on a problem they were both running into in very different jobs.

“We decided this is a problem worth solving, and it seems like we were spot on, because everybody’s talking about context.”

“Some of the challenges were very similar — how do we combine a lot of various data assets into one place where AI can work?” Shimoni says. “We just started to notice that this is the gap — to make AI run with confidence, at scale, across a company. We decided this is a problem worth solving, and it seems like we were spot on, because everybody’s talking about context.”

Modus closed a hitherto unannounced $10 million seed round shortly after founding, led by Insight Partners. Other backers include Soma Capital and a handful of angel investors, among them founders from Cyera and Wix.com. The company began hiring its first employees in January 2026.

The broader takeaway from Modus’s pitch is now among the most common refrains emanating from AI circles this year: that the model itself is no longer the bottleneck; what limits an AI system now is everything built around it. And for Modus, that realization has been more or less present since its inception.

“Even last year […] we could already see that model capabilities weren’t the bottleneck,” Shimoni says. “It was more making sure that they actually have access to the context they need in order to give you the right answers.”

The post Modus’s operandi: To give AI agents just the right amount of context appeared first on The New Stack.

You spend weeks tuning an AI chatbot. Answers are accurate. Stakeholders sign off, and you ship it. Three months later, the system is confidently wrong about a third of what users ask. Nobody changed the model, and nobody touched the prompts. The world moved, pricing changed, a policy updated, a product spec shipped a new version, and the underlying knowledge store didn't move with it.

This is not a hypothetical. It's one of the most common production failure modes in enterprise AI right now, and most data engineering teams don't have the right tooling to catch it, regardless of how the AI system retrieves the data.

The failure that doesn't look like a failure 

An AI application doesn't care whether it's retrieving from a vector store, a document index, or an API call. Whatever the mechanism, nothing in a standard retrieval pipeline checks whether what it's serving is still correct. A stale pricing document retrieves just as confidently as a current one, because the system is scoring relevance or availability, not correctness. A record with a silently missing field passes through just as cleanly as a complete one, for the same reason.

So the failure is invisible by design. Outdated or incomplete data still scores high on relevance, or passes every check a data pipeline was built to run. The model answers with full confidence because the retrieved context looks authoritative. Every dashboard you're watching stays green. The system looks like it's working. It's just wrong.

I’ve watched a similar version of this happen outside the AI context, in a fintech pipeline. An upstream system changed a field without notifying downstream users. The pipeline did not fail; it simply propagated bad values into dashboards because the system only checked whether the job completed, not whether the data was still correct. The issue surfaced only when a customer noticed something inconsistent. By then, the bad data had already moved downstream. 

Whether it's a document that's gone stale or a field that's gone silently missing, the failure shape is the same: the absence of an error is not the presence of correctness, and without building proper validation layers, nothing in the pipeline could identify the problem.

Why this is a data engineering problem

Teams that hit this failure tend to misdiagnose it, and they tend to do it twice.

Blaming the model: The first instinct is to blame the model, try a different LLM, adjust the prompt. The real problem lies further upstream, at the data engineering layer, the same instinct behind the fintech failure above: monitoring built for the pipeline, not the data.

Blaming the retrieval layer: Once the model's ruled out, the next instinct is to blame the retrieval or context layer instead and buy a better one. The timing isn't a coincidence: as enterprises push these systems into the real production world, this gap is exactly what's starting to surface, and the vendor response has been everywhere. 

• AWS just entered the "context layer" race with a knowledge graph that learns from agent usage. 

• Snowflake's new Horizon Context and Cortex Sense target the exact symptom this piece opened with: agents giving confident wrong answers because nothing governs the business logic underneath them. 

Both are real responses to a real problem, but they sit one layer above it; a knowledge graph still depends on whatever feeds it.

The real problem lies further upstream, at the data engineering layer. Teams check whether a job ran, not whether the data it moved is still true, an instinct that predates AI by years. Monitoring is built for the pipeline, not for the data. 

What's actually missing: Data observability

Data observability is a well-known concept that doesn't get enough attention in how it's actually implemented. The relevant metric isn't a percentage — it's coverage: what fraction of critical datasets have lineage that's actually queryable, versus only living in someone's head.

Uber built a dedicated data quality and observability platform long before retrieval-augmented generation existed. Their Unified Data Quality platform supports more than 2,000 critical datasets and detects around 90% of data quality incidents before they reach downstream consumers.

Netflix solved a different piece of the same problem, building a company-wide data lineage system so anyone could answer where a dataset came from and what touched it along the way. It maps dependencies across Kafka topics, ML models, and experimentation, not just warehouse tables. Similar to Uber, the platform was built for humans and now it has become more important with the rise in AI/LLM applications.

Between them, Uber and Netflix cover two of the four things worth building for. In practice, I think about it as four dimensions, each measurable on its own terms.

Correctness: Does each record conform to the shape and rules it's supposed to, right field types, no unexpected nulls, values in range. Tools like Great Expectations and Soda handle this well: automated row and column-level validation instead of manual checks after something breaks. Track percentage of records passing validation per run.

Freshness: Is the data still current relative to its source, not just current as of its last check. Track time since last successful update per source, with an SLA per dataset rather than one blanket threshold, since some sources need hourly refresh and others don't.

Consistency: Does the same fact read the same way everywhere it's stored or indexed. This fails silently, it only shows up when two systems fed by the same source start disagreeing. A periodic cross-check between downstream destinations, flagging mismatch rate above a threshold, is enough to catch it early.

Lineage: Can you trace any output back to its source and every transform it passed through, the same question Netflix built its system to answer. 

None of this requires infrastructure most data teams don't already have. I know because I've built it, not just argued for it.

At Socure, client data arrived in whatever shape the client felt like sending it, and occasionally, quietly wrong. The challenge was building a system where incorrect data could be identified before it propagated downstream. The same principles applied: Validate what arrived, understand where it came from, and prevent bad data from becoming someone else's problem.

Great Expectations became part of that foundation: schema and range validation at ingestion, per-source SLAs for freshness, cross-system checks for consistency, and file-level lineage. All of it sat behind a write-audit-publish pattern, where data landed in staging, was validated, and only moved downstream if it passed the required checks.

The result showed up downstream: better accuracy across the board, in reporting, in the ML models, and in AI retrieval built on top of that same data.

What to do Monday morning

If you're running retrieval-based AI systems in production, the diagnostic question isn't which model to try next or which retrieval architecture to migrate to. It's four narrower questions: 

• Is the underlying data validated against the standards required by its consumers?

• What's the oldest piece of content currently being served with high confidence?

• Would two chunks of the same source ever disagree with each other in the same retrieval result?

• Could you trace where it came from if it turned out to be wrong?

If you can't answer those questions, then the gap lies in the pipeline between your source systems and whatever your agent reads from. That’s a data engineering fix, not a model swap or a vendor migration.

Whether you're building reporting pipelines, ML systems, or AI agents, correctness, freshness, consistency, and lineage are what make data trustworthy. AI simply exposes weaknesses that have existed in data engineering all along. 

GPU memory is the most expensive resource in production AI, and it's also the one running out fastest.

Long context windows and multi-turn conversations force AI models to repeatedly recompute information they've already processed, consuming GPU memory and compute that could otherwise serve additional users or generate new responses.

Instead of treating GPU memory as the limiting resource, why not extend it with much cheaper storage technologies?

Weka, for one, believes that cheap flash storage can close that gap. The company's NeuralMesh 6 software platform, launching alongside its first self-designed hardware line, Wekapod 3, extends what Weka calls Augmented Memory Grid, an approach that aggregates NAND flash to behave like GPU memory at a fraction of the cost.

This is an active and increasingly crowded category. Dell, NetApp, Pure Storage and VAST have all repositioned toward AI infrastructure over the past two years and Weka is one of several vendors arguing it's built for this specific moment rather than adapting to it.

"What we're seeing now with customers is they're chasing availability of compute, and once they get new allocation from anyone, they want to be able to grab it and start running right away," Weka co-founder and CEO Liran Zvibel, told VentureBeat.

The potential payoff is straightforward: better utilization of existing GPU investments, lower inference costs and faster deployment of new AI workloads without waiting months for additional GPU capacity.

The technology is most relevant for organizations already operating AI at scale or expecting rapid growth in usage, particularly enterprises building internal copilots, customer service agents, software engineering assistants or retrieval systems with long context windows. Smaller deployments may see less immediate benefit than organizations where GPU utilization has already become a limiting factor.

Inside Weka's NeuralMesh 6

NeuralMesh 6 adds four capabilities aimed directly at a functionality gap Zvibel says has been costing Weka deals in competitive evaluations.

Composable and virtual multi-tenancy. Composable clusters give anchor tenants full hardware-level isolation, dedicated CPU, memory, and storage. Virtual multi-tenancy runs through Weka's RDMA fabric, delivering network-level isolation that scales past 1,000 tenants per cluster, with provisioning in under 30 minutes. Combined, a single cluster running 50 composable clusters can support up to 50,000 tenants. 

Unified file and object storage. Most storage systems keep two separate paths: a file-based path (the standard way servers and applications read and write files, used heavily in training and fine-tuning pipelines) and an object-based path (S3, the format inference and cloud-native tools typically expect). Normally a gateway translates between the two, meaning the data effectively exists twice. Weka's claim is that the same physical data on disk is directly readable through either path at once, no translation layer, no second copy. Zvibel is targeting non-AWS GPU clouds specifically, naming Lambda, Nebius, G42, and CoreWeave, with what he described as roughly two orders of magnitude higher performance than conventional S3 and a capacity-based pricing model instead of per-API charges. 

Metadata-first replication. Destination environments become browsable before a full data copy arrives, with data hydrating only when accessed. 

"They had to wait for all of that to make it to the other side, and this takes days or weeks, in extreme cases a month," Zvibel said. "We now allow our customers to grab some allocation of new GPUs and get up and running within an hour."

AlloyFlash and Always-On data reduction. TLC and QLC are two types of NAND flash memory. TLC is faster and more durable but costs more per terabyte, while QLC is cheaper and holds more data per chip but is slower. AlloyFlash mixes both within a single cluster, automatically routing latency-sensitive work to TLC while running bulk-capacity workloads on QLC, cutting cost per terabyte without a performance penalty on the work that needs speed. Data reduction now runs by default rather than as an option.

Solving AI's context problem

Multi-tenancy and object storage solve how enterprises and neo clouds operate the platform day to day. A harder problem sits underneath: as context windows and multi-turn interactions grow, so does the GPU compute wasted recalculating work a model has already done. Augmented Memory Grid, a NeuralMesh 6 feature built specifically for this, is Weka's answer.

Every prompt triggers two stages. Prefill calculates attention, the core mechanism behind how large language models process input, and it's computationally expensive. Decode converts that calculation into output and is comparatively lightweight.

The cost shows up hardest in multi-turn sessions like chat or coding, where each new turn re-triggers prefill for everything that came before it, unless that work has been cached.

"If you have 10 turns, you may overcalculate 100 times because you're redoing all of them. If you have 20, you'll overcalculate 400 times," Zvibel said. "You can put two orders of magnitude more NAND than you could afford in shared memory, and we can cache 100% of the pre-calculated tokens, so you never need to redo it."

Where Weka sits competitively

Storage vendors have spent the past year and a half repositioning around AI, and separating genuine capability from repositioned messaging is now a real evaluation problem for buyers. 

"The storage world is shifting its focus from serving bits to enterprise workloads to managing data at the speed of AI. We've seen that most clearly over the past 18 months from Dell, NetApp, and Pure," Steve McDowell, chief analyst at NAND Research, told VentureBeat. "The interesting thing is that companies like Weka, and VAST, are the true AI-native data companies, solving these problems since day one."

McDowell singled out Augmented Memory Grid as Weka's clearest technical lead. 

"Weka continues to have the most technically capable KV cache implementation on the market with its Augmented Memory Grid," he said. " They were early with this technology, and continue to innovate. This is critical for AI inference, as it enables a level of GPU efficiency that, without question, saves money on GPUs and memory. That’s key for today’s memory and GPU constrained market." 

He also flagged Weka's contractual guarantee on its data reduction claims as underappreciated. 

"One flying a little under the radar: Weka is putting its money where its mouth is with its contractual guarantees for its data reduction promises," he said.

McDowell's advice to buyers evaluating competing claims from Weka, VAST, Pure and NetApp alike was pointed suggesting that enterprise buyers should look hard at what vendors are promising versus what they're actually delivering.

"A smart buyer will look at how competing vendors are solving real-world problems today," McDowell said. " They do this by talking to organizations running similar workloads at similar scale. If a vendor can't point to that, then it should be a warning sign."

A single AI agent conversation can look flawless scored on its own and still point to a broken product. That gap is driving a shift in how enterprises evaluate agents, away from scoring individual traces and toward comparing cohorts of users against a baseline.

At VB Transform 2026, Harrison Chase, CEO of LangChain; Hui Zhang, CTO and co-founder of Conviva; and Emmanuel Turlay, director of engineering at CoreWeave, described that shift, along with a parallel move toward cheaper, narrower judge models.

Agent-as-judge — judging one AI agent's output with another — hasn't replaced LLM-as-judge, which Chase said remains the default. The larger tension, Zhang said, is between automated judging, whether by LLM or agent, and human review.

"You have scalable but ungrounded, whether it's agents as judge or LLMs as judge, you grade the outcome, you grade the work. It still is very difficult to ground it and then you use humans and that's just not scalable," Zhang said. "The whole industry is facing this, which poison you want to pick."

Evaluation criteria now function as the product spec

That gap — a conversation that scores well but still signals a broken product — is what teams try to close by building an exhaustive evaluation suite before they ship anything. Chase said that doesn't work.

"We sometimes see teams that have almost eval paralysis," Chase said. "They're like, this is an eval set, I can't launch it. The best teams launch and then iterate."

Chase framed evaluation criteria as a living specification, not a one-time test suite: a product requirements document — the standard software-development spec for what an application should do. "Evals are like the new PRD," he said. "They define what your agent should and shouldn't do."

Turlay described hitting the same failure from a different angle. "I was trying to reach 100% coverage for my tests, and I still had bugs in production," he said — a test suite that looked complete but still missed what mattered, the same gap Chase was describing with evals.

Broad, always-on monitoring, he said, catches more real failures than an exhaustive pre-launch test suite. Teams should set up wide online checks first, use those to identify failure classes as they occur, then build a targeted offline evaluation set around the problems that surface.

Why scoring traces one at a time is a mistake

Even a well-built evaluation process can still score the wrong thing. Zhang's objection is to how most teams run evaluation: sampling traces, whether 50 of them or a full population, scoring each in isolation. That approach misses a signal that only shows up when comparing cohorts of users against a baseline, a method Zhang calls contrastive analysis.

Zhang illustrated it with a retail example: a shopper asks an agent for a running shoe ahead of a half marathon, the agent asks qualifying questions, and the shopper buys a shoe. Scored individually, that interaction looks fine. But the clarification ratio, how many follow-up questions an agent asks before completing a task, came in three times higher than baseline for that shoe category across the full user population. A second metric, how often shoppers finished their purchase outside the conversation, was five times higher than baseline for the same category.

Neither number is visible from a single trace. Both point to a debuggable, category-specific problem. Zhang said the industry also lacks a second data source: what happens before, between and after the conversation, not just the trace itself.

Sizing the judge to the job

Once contrastive analysis flags which category is actually broken, the next problem is what watches for it going forward — and at what cost. Turlay's rule was to start with the most capable model available to prove a task is solvable, then work down. If it can't be done with a top-tier model, he said, it won't work with a smaller one. Once a pattern proves viable, teams can sample a fraction of traffic instead of judging every interaction, and move simpler tasks like binary classification to smaller open source models.

LangChain took that further, fine-tuning its own model to detect when a user believes the agent made a mistake, a signal Chase calls perceived error. "The model we fine-tuned was a Qwen model," he said, referring to Alibaba's open source family. Combining hand labeling with distillation, the result performed well. "Same as [Claude]Sonnet, for, depending on how we served it, either 10 to 100x cost reduction," Chase said.

Not every guardrail needs a model. Chase pointed to Claude Code's own guardrails as proof: regexes, the common programming technique for finding and validating patterns in code. "A lot of the guardrails they had were just regexes," he said. "They weren't small LLMs, they were just regexes."

LLM-as-judge doesn't mean human-in-the-loop disappears

The bigger question is whether using LLM as a judge removes the need for a human in the loop.

Turlay pointed to accountability, drawing on his prior work at a self-driving car company. His team compressed data intake and retraining into a two-week cycle for shipping a new model to the car. Even then, someone still had to sign off.

"I felt confident on behalf of the company to say this model should go into the car," he said. The same logic extends to legal, finance and healthcare. "Before we can remove a human to say, I endorse this and I take responsibility legally for it, it's going to be a while before agents can do that on their own."

Zhang agreed a human has to remain the guardian on corner cases, even as automation eventually runs at a scale that beats individual human accuracy — machines can see more at the pattern level.

Chase went further: that human check isn't just a safety net. "Human in the loop is really important for building trust in how these agentic systems work, and also really important for memory and learning from systems," he said. "There has to be interactions in order for the system to learn."

Zillow, the real estate technology company, doesn't get one conversation with its customers. They move from a phone screen to a loan officer to a real estate agent, sometimes over months or years, and expect the context to follow them. A single chatbot could never carry that thread.

At VB Transform 2026, Zillow SVP of Engineering Toby Roberts and Glean co-founder and CEO Arvind Jain described how they built AI architecture meant to carry context across that entire journey — and why context, not raw data, turned out to be the harder problem to solve. Zillow's products touch roughly 80% of U.S. real estate transactions each year, and the company has been using AI long before ChatGPT existed.

"We pretty quickly identified that we were going to need a persistent context layer that was going to meet our customers and the professionals wherever they were," Roberts said.

Data was never the hard part

Roberts said Zillow's AI effort started where most enterprise AI efforts start, with the data itself.

"We started with a large push around making sure our data did have the right foundation," Roberts said. That meant a data mesh approach, clear data lineage and a governance structure with permissions and identity attached to the data itself.

None of that turned out to be the hard problem. The hard problem was building something that remembered where a customer was in their journey and carried that forward, no matter which surface they showed up on next.

"This context layer has to live to be able to support you where you are at any given point in your journey," Roberts said. Zillow chose to own that layer itself rather than depend on a single external chat interface, a decision Roberts said the team reached quickly once it looked at the shape of a real transaction rather than a single conversation.

Why Zillow built its own architecture, and where Glean fits into it

Zillow built its own harness rather than route customers through a single model API. The team drew on 20 years of machine learning history behind products like Zestimate, leaning into smaller, task-specific fine-tuned models instead of one general-purpose model.

Internally, that harness runs alongside Glean. Roberts said Zillow now has thousands of Glean agents in production, handling repetitive tasks with tens of thousands of executions across the company. Glean's pitch, per Jain, is centralizing that integration work once, through the Glean MCP gateway, rather than letting finance, legal and marketing each rebuild their own connections to the same systems.

That centralization is also a cost lever. Jain pointed to two mechanisms: model routing, which sends most tasks to smaller, cheaper models instead of defaulting to frontier models, and precomputed context, which avoids an agent burning tokens assembling its own context from scratch.

"Claude is also very slow because the first part of assembling that context actually takes forever," Jain said. Routing that request through Glean instead, he said, can cut token consumption by as much as half.

What Zillow and Glean's approach means for enterprises

Across data, cost and permissions, the session offered a few practical takeaways for enterprises building agentic AI on their own systems.

Build the measurement baseline before the AI push, not after. Roberts said Zillow's ability to credibly attribute a 40% increase in shipped code to AI adoption rests on a DORA metrics baseline the team put in place years earlier, not on the AI rollout itself.

Centralize context once instead of letting every team rebuild it. Jain's core argument for Glean's platform is that duplicated integration work across finance, legal and marketing teams is a hidden cost most enterprises haven't accounted for.

Don't assume permission inheritance is enough for regulated data. Even with a permissions-aware context platform in place, Zillow layered hard rules and a standing compliance check on top for its most sensitive categories, rather than trusting the architecture to handle it automatically.

Treat context as a cost lever, not just a capability. Model routing and precomputed context were the two mechanisms Jain pointed to for cutting AI spend, both aimed at reducing wasted token consumption rather than adding new capability.

"Models by themselves are not enough to bring automation with AI inside your enterprise," Jain said. "You do have to connect it with your enterprise context."

Apache Spark 4.2 launched last week, and it signals an expansion of Spark’s decade-plus role at the center of enterprise data processing

With new features for AI workloads, including governed metrics, vector retrieval primitives, real-time processing, improved Python support and native geospatial analytics, Spark 4.2 builds on a recent history of new AI and streaming features, reflecting how many engineering teams use the platform today. The release builds on Spark’s traditional role as a data processing engine by adding more of the capabilities needed to support production AI applications.

The launch introduced several features that enable developers to do more without leaving the platform, and for teams already using Spark, that could mean fewer systems to manage.

Governed metrics prevent conflicts

One team’s definition of a business metric isn’t always the same as another’s. Over time, those differences can lead to conflicting reports and uncertainty about which number to trust.

That becomes even more problematic when AI applications start consuming the same enterprise data that analysts and business intelligence tools use. If different teams define the same metric differently, AI systems can produce inconsistent results for the same question.

Spark 4.2 introduces governed metric views to address that issue. Organizations can define a business metric once and reuse that definition across applications. A metric view makes dimensions and measures first-class objects that Spark understands, so the engine can preserve the intended aggregation semantics regardless of who or what is querying it.

Organizations can define a business metric once and reuse that definition across applications.

Vector search goes native

One of the more significant additions is native vector search, which reduces the need to move data between Spark and a separate vector database.

Spark 4.2 introduces vector distance and similarity functions, vector normalization, vector aggregation, and NEAREST BY, a new SQL operator for top-K similarity searches. By bringing vector search into Spark, developers can keep more of their retrieval pipeline on the same platform.

By bringing vector search into Spark, developers can keep more of their retrieval pipeline in the same platform.

Python interoperability gets easier

Spark 4.2 makes it easier to move data between Spark and Arrow-native tools. With support for the Arrow C Data Interface and the PyCapsule protocol, Spark DataFrames can be passed directly to tools like Polars and DuckDB without copying or serializing the underlying data, as long as both sides support the standards.

Python also gets a few other updates. PySpark has been expanded; Arrow-optimized UDF execution is now the default, and Python Data Sources now include built-in time and memory profiling to help developers troubleshoot custom connectors.

Spark Connect makes the engine callable by agents

Spark Connect, which separates the client from the Spark server via a gRPC- and Arrow-based protocol, receives several updates in 4.2. The key idea is that a client builds a logical plan, the server handles analysis, optimization, and execution, and the results come back as Arrow batches. The client doesn’t need a full Spark runtime or a colocated JVM.

This update includes several changes to Spark Connect, the project’s client-server interface. AI applications can send processing requests to a remote Spark cluster while the work continues to run inside Spark. The release improves RDD API compatibility, error handling, and status reporting along that path.

Streaming powers real-time AI

Streaming gets several updates in Spark 4.2, including Auto CDC and Real-Time Mode. Many AI applications depend on continuously updated data rather than scheduled batch jobs. Auto CDC brings first-class change data capture to Spark Declarative Pipelines, handling the merge logic for keeping target tables current as source data changes — something that previously required hand-written, error-prone code. The new CHANGES SQL clause allows teams to retrieve data changes through a single SQL interface.

Spark 4.2 also adds built-in GEOMETRY and GEOGRAPHY types along with ST_* functions for location-aware analytics, without requiring external spatial extensions. For teams doing anything with location data — logistics, real estate, IoT — this removes another reason to move data out of Spark.

The bigger picture

Spark 4.2 brings more of the AI and data stack into the platform itself. Features that once depended on separate tools can now be handled directly in Spark.

For teams that currently use Spark for ETL and then hand data off to other systems for retrieval, governance, or real-time processing, this release starts to blur that line. As more AI applications run directly on operational data, Spark is becoming part of the serving layer rather than simply preparing data for it.

Spark is becoming part of the serving layer rather than simply preparing data for it.

The post Spark 4.2 has a feature that could retire your vector database appeared first on The New Stack.

Legacy infrastructure, not the models themselves, is what's actually slowing AI agents down. That was the shared conclusion of three infrastructure leaders — from LinkedIn, Walmart, and Zendesk — at VB Transform 2026.

The panel brought together Animesh Singh, senior director of AI platform and infrastructure at LinkedIn, Desiree Gosby, SVP of corporate technology services and technology strategy at Walmart, and Sami Ghoche, VP of applied AI at Zendesk, each describing what actually broke when they moved agents from pilot to production. Each arrived at the same conclusion from a different starting point: None of the bottlenecks they hit were model problems.

What tied their answers together was a shared premise: most enterprise infrastructure was built for how humans work, not for how agents work. The gap between those two speeds is where the real engineering happened.

Gosby put it plainly when asked what she'd learned scaling agents inside Walmart's own workforce. The goal, she said, is to make sure "engineering doesn't once again become the bottleneck for what it is we're trying to do."

Where the bottleneck actually was

Each company hit a different version of the same wall: infrastructure designed for how people work doesn't hold up once agents are doing the work instead.

At LinkedIn, the first bottleneck wasn't a model, it was Kubernetes, which assumes containers spin up on demand, a process that takes seconds. Singh said that's too slow for agents. The fix was moving from on-demand provisioning to pre-provisioned pools of containers that swap agentic workloads in and out in real time.

A second, harder problem surfaced once LinkedIn let agents control their own orchestration. A five-point evaluation system looked clean, but hallucination kept showing up anyway. Singh said the issue was structural, an LLM evaluating another LLM's output shares the same failure mode as the thing it's evaluating. 

"We built our own harness, our own control flow, and pushed the LLMs to the leaf instead of them orchestrating the loop," Singh said. Roughly 80% of the workflow is now scripted, deterministic code, with LLMs used only where reasoning is required, and each step's evidence is committed to disk before the system moves on.

Walmart's bottleneck came from success. An agent harness put directly into employees' hands went viral internally, and what Gosby called "citizen developers" began building their own agents to solve problems that once required a formal engineering roadmap. The upside was real innovation. The downside was duplication, dozens of overlapping agents with no coordination. The fix wasn't reining in the harness, it was building governance to spot duplication, promote the best version of an agent, and get it into production without engineering becoming a chokepoint.

Zendesk hit its bottleneck from the data side. Ghoche, who joined through Zendesk's acquisition of Forethought, which closed in March 2026, described sitting on what he called a public figure of 20 billion customer conversations in Zendesk's repository. The instinct is to hand that history to a large language model with a big context window and let it generate the agents a business needs. Ghoche said that doesn't work. "You can't really do that, so instead you have to really invest in the underlying data pipelines and all the data infrastructure that comes with that," he said.

The role of open source

On open source, all three leaders landed on a similar instinct: own what you can, and lean on frontier labs only where they still have a clear edge.

Ghoche said his own view is that most enterprises would prefer to own their models and infrastructure wherever that's possible, and that reasoning is what drives Zendesk's own approach. The exception is frontier reasoning work, where the labs still lead, though he said that slice of use cases is shrinking relative to everything else enterprises now do with AI.

LinkedIn's answer was to build two subsystems specifically for independence. The first is what the company calls an AI gateway, a single interface that every outbound call to a model runs through regardless of provider. The second component is a memory subsystem built to hold context independent of any model provider.

"Every single outbound call going to an LLM, whether it's on a public cloud or on-prem in our own data centers, follows the same semantics, the same API calls. We can quickly switch between different providers," Singh said. 

Walmart built its own internal gateway to stay vendor agnostic across three workload types: fully deterministic workflows, planner-and-reasoner workflows for open-ended tasks, and a hybrid of the two. Compliance-heavy work stays deterministic by design; governance, security and evaluation run through the gateway regardless of which model is on the other end. Gosby said the choice between a frontier model and an open-weight model comes down to whichever is most effective for the specific workload, not a fixed policy.

Advice for the modernization journey

Three pieces of advice came up directly, each tied to the wall a leader had already hit.

Invest in evals before anything else. Ghoche called it the thing common to every use case, internal or customer facing. 

"The thing that's common to all of these is evals. It'll force you to break the problem down, and once you have a robust set of evals, you can move a lot faster," he said, 

Own your agent harness from day one. Gosby's advice was to put the AI harness directly in employees' hands early, paired with the infrastructure to monitor what it produces. 

"It will unlock a huge amount of innovation," she said.

Build for model and context independence. Ensuring flexibility is critical for success.

"Build for independence, whether it's a frontier model of today versus an open source model of tomorrow," Singh said. "Keep that context within your enterprise so that you can reuse it when you ship the model or the harness tomorrow," Singh said.

Organizations need to transform to meet the needs of agentic AI.

Meta VP of Engineering Barak Yagour opened his talk at VB Transform 2026 wearing a pair of Ray-Ban Meta AI glasses, a small sign of how far AI has already worked its way into physical life. His argument went further: enterprise infrastructure was built for humans, not for agents, and it's starting to show.

Yagour, who leads its data infrastructure organization, told the audience that agentic queries hitting Meta's data systems grew 30x in a single half, an inversion that he said is breaking assumptions the company spent two decades building around.

The shift is not confined to Meta. Automated traffic overtook human traffic on the internet last year, reaching 51% of the total, according to Imperva's 2025 Bad Bot Report. That traffic is also growing roughly eight times faster than human traffic, according to HUMAN Security's 2026 State of AI Traffic report. Yagour cited both figures to describe what he called an inflection point already underway inside his own organization.

Yagour framed the shift as an open question for infrastructure teams everywhere. "What happens to the infrastructure we've spent years building when agents and not humans become the main consumers of that," Yagour said. "That's the world we're stepping into."

Capacity, identity and velocity are breaking at once

Yagour said three assumptions are breaking simultaneously inside Meta's infrastructure: capacity, identity and velocity.

On capacity, the math no longer works the way engineering teams are used to. "One engineer used to mean one unit of load," he said. "Now one engineer spawns 10 agents, each spawning subagents. Your 1,000-person org can generate the load of 100,000 users practically overnight."

His answer is not to block agent traffic but to make infrastructure agent-aware, with dynamic controls that understand agent hierarchies, cost attribution that traces consumption back to the use case that spawned it, and throttling that adapts based on priority.

Identity is breaking, too. Yagour said an agent does not fit the categories infrastructure teams built access controls around. It is not a human user, it does not carry a badge and it is not a deployed service, yet it makes decisions on its own.

Velocity is the third assumption under strain. Yagour cited a company-reported figure that GitHub Copilot writes 46% of the average user's code, then noted that faster code generation does not make the rest of the pipeline faster.

"That code still needs to be built, tested, deployed, monitored," he said. "The agent writes the code in seconds, but your CI/CD pipeline doesn't get faster just because the machine is the author."

Trusted data environments keep agents inside guardrails

Data is where Yagour said the pressure from agents is most direct. 

"Data sits at the center of everything," he said, pointing to the decisions, products, recommender systems and next generation models it drives.

Meta is also rethinking how much autonomy to grant agents inside its own data systems. In February, the company shipped what Yagour called agentic data apps. Within three months, 63% of dashboards published across Meta were built using the new tooling, part of the same 30x rise in agentic queries Yagour cited earlier.

That growth raises a governance question. Human analysts have traditionally sat between raw data and business decisions, curating it and serving as an informal check on quality. Yagour said Meta wants to grant agents more independence on harder problems, but was direct about the risk. 

"Autonomy without governance is nothing but chaos," he said. That's why the company built what it calls trusted data environments, to preserve the human check as agents take on more of that work.

"Inside, the agent can explore data freely, but every output is traced back to its source and scrutinized. So you always know that the data shared back is trusted and governed," Yagour said.

Sensitive fields are masked before an agent can reach them, and every access request is evaluated in real time against what the agent is trying to reach, why and whether it is allowed. Yagour summarized the approach as exploring broadly while releasing narrowly.

Reasoning models are rewriting the data layer

Meta's models are also demanding more from data as they shift from correlation to reasoning. 

"Reasoning is data hungry," Yagour said. 

Pattern matching works on sparse, summarized signals. Reasoning demands the full behavioral history, every interaction across every surface over time. Yagour pointed to two shifts already underway inside Meta's infrastructure to keep up.

Real-time streaming is replacing batch ETL for ranking pipelines. A pipeline that takes 24 hours to run is not viable when a model is reasoning about a user's current intent. Yagour said real-time streaming, not batch extract-transform-load processing, is becoming the backbone of Meta's ranking and recommendation systems.

Storage is becoming schema-aware to stop GPU starvation. Meta previously stored user data as opaque blobs with no awareness of what the data contained, which Yagour said led to heavy overfetching and idle GPU capacity. The company is now building storage that understands what it holds, pulling only the columns and time ranges a given query needs. Yagour said Meta is building toward 500 million queries per second and a petabyte per second of throughput for training data reads.

That data feeds directly into how Meta's recommendation systems behave. Yagour said 42% of Instagram users have told the company they want to fundamentally change the algorithm, not adjust a single session or setting. Meta's response is what Yagour called fully conversational recommendations, where a user tells the system what they want more of and it reasons about intent rather than matching on keywords. Yagour said the same search term, soccer, would return different results for a casual fan looking for highlights than for a club athlete seeking training drills, because the system would reason about which one is asking.

Yagour described the three threads of his talk, agents, data and recommendations, as reinforcing each other rather than moving independently. 

"Agents make data more accessible. Better data makes reasoning. Reasoning creates new demands that push agents and infrastructure forward," he said. "This isn't linear; it's a flywheel."

During the Q&A, an audience member asked whether Meta's push toward more intelligent infrastructure signals the end of traditional file systems in favor of newer neural storage approaches, and whether agents will keep using SQL as their interface to data the way humans do. Yagour said Meta is experimenting at every level, including questioning whether SQL is the right interface for agents at all, and that storage at Meta's scale already operates in the multi-digit exabyte range and needs to keep expanding.

Yagour closed his talk with the timeline he believes the industry is working against. "We spent 20 years building infrastructure for humans. We have maybe 20 months to rebuild the whole thing for a world where humans and agents co-create at scale," Yagour said. "The window is open, but it won't stay open for long."

Canva on Tuesday launched Canva Code 2.0, a major upgrade to its AI-powered coding tool that lets users build interactive websites, apps, and experiences using plain-language prompts — and then edit the results as easily as tweaking a Canva presentation. The feature is now available to all of the company's more than 265 million monthly users across every pricing tier, including free accounts.

The move is Canva's most aggressive push yet into the fast-growing "vibe coding" market, a category that barely existed 18 months ago but has already minted billion-dollar startups and reshaped how non-developers think about building software. But where rivals like Lovable, Replit, and Bolt.new have focused primarily on generating functional code from text prompts, Canva is making a different bet: that the real bottleneck isn't creating the code — it's making the output actually look good.

"Most vibe coding tools stop at functional — generating output that looks the same as everyone else's," Canva states in its announcement. "You might get a working prototype, but making it actually look like yours requires a complex editing surface, a separate design tool, a developer, or endless back-and-forth prompting that rarely lands where you want it.”

Danny Wu, Canva's Head of AI Products, framed the product's positioning in stark terms during an exclusive interview with VentureBeat ahead of the launch.

"We are deliberately targeting non-technical users," Wu said. "Canva Code isn't a tool we're building for developers. What we're trying to do is bring the power of AI coding — and really lightweight coding — into the Canva platform, while answering our users' requests for more interactivity, more customization, and more flexibility, from websites to interactive presentations."

Canva Code 2.0 brings drag-and-drop editing, HTML import, and 75% faster generation to AI-built websites

The update introduces several capabilities designed to collapse the distance between generating code and publishing a polished interactive experience. Users can now create Canva Code projects directly inside other design projects — embedding interactive elements within a whiteboard, presentation deck, or standalone page. Canva has also added more than 50 new templates specifically designed for interactive designs, along with the ability to import raw HTML files from other AI coding tools and convert them into editable Canva designs.

The performance improvements are significant. Canva says it has reduced average code generation time by 75 percent and cut the median time from initial prompt to a published site by 30 percent. The company also reports that integrating Canva Code into the broader Canva editor — allowing users to treat coded outputs like any other design element — has increased active Code users by 25 percent.

Perhaps the most distinctive feature is the editing experience itself. Unlike most AI coding platforms, which require users to re-prompt or modify raw code to make visual changes, Canva Code 2.0 lets users click directly into generated elements to change text, drag and drop images from Canva's built-in library of over 120 million templates and assets, update colors and fonts through a familiar toolbar, or select a specific element and refine it through conversational AI. Every output is fully interactive and automatically adapts to different screen sizes, with a built-in mobile preview.

Wu demonstrated the drag-and-drop editing during the interview, showing how a generated conference website could be modified in real time — swapping in photos, changing fonts to branded alternatives, and editing text directly on the canvas. "The key differentiator with Canva Code is the editability and the kindness of the outputs it generates," he said, though he noted one current limitation: "We don't support moving elements around. You still have to re-prompt for that."

How Canva plans to compete with Lovable, Replit, and Bolt in the booming AI app builder market

Canva's entry into vibe coding at this scale arrives at a pivotal moment for the category. According to market research published by Luminix AI in May 2026, the vibe coding and AI app builder market has reached an estimated $4.7 billion in 2026, with projections pointing toward $12.3 billion by 2027 at roughly 38 percent compound annual growth. The research also estimates that AI-generated code now comprises approximately 41 percent of all code written globally — a figure that would have seemed inconceivable even two years ago.

The competitive landscape has grown ferocious. Lovable, which focuses on conversational, design-forward app generation for non-technical founders, has achieved what may be the fastest revenue ramp in the category's history — reportedly reaching approximately $400 million in annual recurring revenue by early 2026, according to Luminix's analysis. Replit, which transformed its browser-based IDE into a full vibe-coding engine through successive AI agent releases, has tripled its valuation to $9 billion and is targeting $1 billion in run-rate revenue by the end of 2026, per the same report. Bolt.new, which runs a full Node.js environment entirely in the browser, scaled from $4 million to $40 million in ARR within months of launching.

And then there is Canva, which brings something none of those platforms possess: a quarter-billion-user design ecosystem where brands, teams, and individuals already store their visual identities, collaborate on projects, and publish content.

Wu positioned Canva Code not as a direct competitor to these developer-focused tools but as something that fills a gap none of them have addressed. "A lot of the requests that we have been getting and the usage we're seeing is actually with using Canva Code not necessarily as just one artifact, but as part of an overall design, the visual communication they're trying to tell," Wu said. "Like when you have a sales deck, you're able to add a calculator, you're able to add a visualizer of what exactly your product does. That's something where an interactive slide can be worth a thousand pictures."

Why Canva's HTML import feature could turn it into a 'finishing layer' for every AI coding tool

One of the most strategically interesting features in Canva Code 2.0 is its HTML import capability, which allows users to take code generated by any AI tool — including ChatGPT, Claude, Lovable, or Bolt — and bring it into Canva as a fully editable design. The implication is unmistakable: Canva is positioning itself as the place where AI-generated code gets its finishing touches, regardless of where it was originally created.

When asked directly whether this amounts to positioning Canva as a "finishing layer on top of vibe coding," Wu offered a diplomatic but revealing response. "It's really a continuation of our goal to make all design as easy as possible," he said. "We've supported importing PDFs and translating them into docs, importing PowerPoint files — so in one way, it's an expansion of that. But in another way, it's really just listening to what our users want and making Canva both the most useful and the most compatible platform.”

He paused, then added: "It's not that we're deliberately positioning ourselves as a specific layer, say like a finishing layer after vibe coding. We just really want to make our platform the most accessible and the most pluggable."

That language — "most pluggable" — suggests a platform strategy that doesn't require Canva to win the AI code generation race outright. If Canva becomes the default destination for making AI-generated code look professional and on-brand, it captures value from the entire category regardless of which code generation engine users prefer. The strategy also echoes the broader import capabilities that already allow Canva to ingest PowerPoint decks and PDFs from competing platforms, gradually pulling users deeper into the Canva ecosystem without demanding they abandon existing workflows.

What Canva Code can build — and where Danny Wu says it hits its limits

Wu was notably candid about the product's boundaries — a refreshing departure from the typical Silicon Valley product launch. "Canva Code is great for anything that works as a front-end app, and it's especially good when you want to leverage data, data submissions, and interactivity at small to medium scale," he said. "I'll be honest about the limitations. Canva Code is probably not going to be suitable if you're trying to build a website with complex backends, or if you're handling hundreds of thousands of visitors per day."

This candor effectively draws a line between Canva Code and the more ambitious platforms in the space. While Lovable and Replit are pushing toward full-stack application development — complete with databases, authentication, and production-grade hosting — Canva is deliberately limiting its scope to interactive front-end experiences at modest scale. The question is whether that's a strategic weakness or a disciplined focus. For the teachers, small business owners, and marketing teams that make up the bulk of Canva's user base, complex backends and high-traffic scalability are irrelevant concerns. What matters is whether they can create an interactive event page, a property listing website, or a classroom hub that looks professional and works on mobile — without hiring a developer or learning a new tool.

When asked about the AI models powering Canva Code, Wu confirmed the company uses a combination of proprietary and third-party models, including those from OpenAI and Anthropic, but declined to specify the exact mix. "We don't share the exact mix, and it does change over time," he said. "We also route differently depending on what you're asking for and which model family we think is best for handling certain requests."

Canva's AI acquisition spree — from Affinity to Leonardo.ai — now powers its vibe coding push

Canva's broader AI infrastructure has been significantly bolstered by an acquisition strategy that has accelerated over the past two years. In March 2024, the company acquired Affinity, the British creative software suite popular with Mac users, in a deal that Bloomberg reported was valued at "several hundred million pounds." Canva at the time positioned the deal as a way to compete with Adobe's flagship products — Illustrator, Photoshop, and InDesign — by gaining ownership of Affinity's Designer, Photo, and Publisher applications.

Just four months later, Canva acquired Leonardo.ai, an Australian generative AI startup with over 19 million registered users and more than a billion images generated. Canva co-founder Cameron Adams said at the time that Leonardo.ai's technology would be integrated into Canva's Magic Studio generative AI suite.

Together with these acquisitions, Canva Code is the company's attempt to layer interactive, code-driven capabilities on top of a visual design platform that has already been enhanced by professional-grade design tools and generative AI models. The company reports over 32 billion uses of its AI products to date — a staggering figure that underscores how deeply AI is now woven into everyday Canva workflows, even for users who may not think of themselves as using artificial intelligence.

Six million sites published, but Canva's retention data remains an open question

Canva's announcement highlights an impressive traction metric: users have created and published more than six million websites using Canva Code since the feature was first introduced a year ago. But the number deserves scrutiny.

Wu clarified in the interview that the six million figure represents published websites over the past year — meaning sites that were either made public or shared via password-protected or private links. "They may have published publicly, or behind a password, or as a private link. But that's the number of published websites," he said.

When asked about active retention — how many of those sites are still live and being maintained — Wu acknowledged the gap in his data. This is a meaningful distinction. In the vibe coding market, raw creation numbers can be misleading because the barrier to generating a site is so low. The more telling metric — which Canva does not yet provide — would be how many of those six million sites receive regular traffic or have been updated after initial publication.

The early use cases, however, suggest genuine utility beyond novelty. Educators and school administrators are using Canva Code to build classroom hubs, with one teacher creating bespoke webpages for each of their classrooms to keep students and parents updated on announcements. Small businesses, like Alt Marketing School, have built mini apps for fundraising training and interactive roadmaps for their members. For World Book Day, 50 readers created educational games across different subjects, complete with pedagogical guides for classroom use.

Canva Code pricing, data governance, and what enterprise customers need to know

Canva Code 2.0 is available across all of Canva's pricing tiers, including its free plan — a notable decision given that competitors like Lovable, Bolt, and Replit reserve their most capable features for paid subscribers. "As you go from, say, free to pro to business to enterprise, you would get more AI credits and be able to have higher usage of Canva Code," Wu said. "But it is available and it is usable — even free Canva accounts as well as education and not-for-profit accounts."

This credit-based approach mirrors the pricing evolution happening across the entire vibe coding category, where platforms have converged on token or credit systems that meter AI generation capacity rather than gating features behind subscription tiers. The difference is that Canva's free tier serves as an acquisition funnel for a much larger design platform, not just for the coding feature itself.

For the institutional customers Canva increasingly courts — school districts, real estate brokerages, enterprise marketing teams — data governance is a threshold concern. Wu addressed this directly. "All users and customers have full control over how their data is used," he said. "They can choose whether their prompts and data are used for AI training in the settings. For businesses and enterprises, team admins can manage this at the organizational level and guarantee that their inputs, content, and outputs won't be used for training." This opt-out approach reflects a lesson the broader industry has learned the hard way. As The Verge reported when Canva acquired Leonardo.ai, Adobe suffered significant backlash over a policy update regarding user data and AI model training — a controversy Canva appears keen to avoid.

Canva's long-term vision: closing the gap between imagination and what non-technical users can actually build

When asked where Canva Code fits into the company's long-term trajectory — and whether Canva is building toward a full-stack app development platform — Wu steered the conversation back to the company's core audience.

"A huge part of it is reducing the gap between your imagination and what's possible, especially for everyday users — people who don't have a lot of time," he said. "They don't have time to figure out deploys or MCPs or APIs. They just want to design more interactive and more dynamic communication."

He pointed to the rapid improvement in AI model capabilities as a key accelerant. "The kind of things you can create today in one shot — like a 3D visualization of a solar system — you really couldn't have trusted the output a year ago. But today, you have a really high success rate."

Whether Canva Code becomes a durable product category or a feature that gets absorbed into the platform's broader AI workflow will depend on how quickly the company can close the gap between its current front-end focus and the full-stack capabilities that increasingly define the competition. Lovable is shipping Supabase-backed apps with authentication and databases built in. Replit's agents can execute autonomous long-running builds. Bolt.new runs entire Node.js environments in a browser tab. These are fundamentally different ambitions than making a conference landing page look good.

But Canva has never won by matching the technical depth of its competitors. A decade ago, it didn't try to out-feature Adobe — it made design accessible to the 99 percent of people who would never open Photoshop. Now, in a vibe coding market where every tool can generate a working prototype from a prompt, Canva is making the same wager it made in 2012: that for most people, the hardest part was never the building. It was making it look like it came from you.

The seemingly unquenchable thirst of the AI data ingestion pipeline spans language, numerical, and tabular data in the first instance, while other tangential platforms have been building large audio, image, and video models at the same time. 

Straddling potentially all of these domains are the file structures where complex documents and forms of unstructured data reside; this is the road less traveled in terms of the source DNA modern AI draws from.

The schema-less, freeform, uncurated data lake

In a bid to bridge connections to the schema-less, freeform, uncurated information that all organizations naturally harbor, enterprise visual intelligence company Valantor announced its acquisition of unstructured information RAG specialist EyeLevel on Tuesday. The acquisition formally launches Valantor’s Enterprise Visual Intelligence platform, combining EyeLevel’s document intelligence with its own operational expertise.

Benjamin Fletcher, CEO and co-founder of EyeLevel, tells The New Stack that where organizations fail to adopt visual intelligence, human-only processing breaks down pretty quickly in the age of AI.

“About 80% of corporate knowledge is in millions of pages of visually complex PDFs, PPTX, and DOCX files,” Fletcher says. “This information is far beyond the capacity of any LLM context window and is effectively inaccessible to LLMs and agents.” 

“We’ve found the golden datasets that teams build by hand routinely carry 10 to 25 percent error rates. Ironically, those same teams often hold AI to a far higher standard than their own people.”

Humans are slow, expensive & prone to errors

He explains that transactional workflows (such as invoice and claims processing) typically involve documents “so visually complex and diverse” that enterprises still rely on humans to process them, who can be slow, expensive, and error-prone. 

“We’ve found the golden datasets that teams build by hand routinely carry 10 to 25 percent error rates,” Fletcher says. “Ironically, those same teams often hold AI to a far higher standard than their own people. If data sovereignty matters to a business, everything gets harder now: solving these problems with AI while your documents stay inside your own infrastructure is the hard mode version of the job, and very few tools can do it.”

Where does invisible corporate information live?

Valantor has noted that while most AI companies concentrate on models, the company itself is “focused on the information those models can’t see” today. The suggestion is that this unseen morass of valuable data is locked inside documents, claims files, contracts, engineering drawings, reports, forms, presentations, and other visually complex content.

Valantor’s flagship platform product, GroundX, operates where data resides, including private cloud, sovereign infrastructure, on-premises deployments, and fully air-gapped environments. 

“GroundX is the ingestion and retrieval layer for unstructured documents,” explains Fletcher. “It is one tightly tuned system where retrieval consumes exactly what ingestion produces. Everything is exposed through REST APIs, SDKs, and MCP. It ships as REST APIs, SDKs, and MCP, and the Helm chart drops straight into a team’s existing deploy pipeline, and our agent harness gives coding agents like Claude and Codex the skills to build the integration themselves.

As part of the acquisition announcement, Valantor is introducing GroundX Studio. The harness capabilities within GroundX Studio integrate with modern AI development environments, enabling developers to build secure AI applications that operate on enterprise knowledge while remaining within existing infrastructure. 

GroundX Studio also extends capabilities to business users, allowing organizations to create AI-powered workflows and applications without extensive custom development.

“Each agent does one small task, so cheaper models are often good enough, and teams that want direct control over cost can run the whole stack on their own hardware with Helm.”

Risk of latency-laden performance and spiraling costs?

If it feels like this new data ingestion stream is going to place a new burden on cloud workloads, application execution latency, database retrieval times, and (of course) overall token usage, then Valantor and EyeLevel say that this consideration has been taken into account by dint of their own platform’s orchestration layers.

“We never send a whole schematic to a language model; our vision model splits each page into its elements first,” Fletcher confirms. “Processing runs in multiple passes at different levels of the document, and everything inside a pass runs in parallel, so there’s a minimum processing time, but it does not scale linearly with page count. Each agent does one small task, so cheaper models are often good enough, and teams that want direct control over cost can run the whole stack on their own hardware with Helm.”

The intersection of AI and handwriting

sWhile we already know that AI and handwriting do mix in the same cocktail glass — the ViWoods AiPaper digital e-ink handwriting tablets have a useful set of AI functions on board, and similar products are available from manufacturers including reMarkable — it’s not a widely deployed use case yet. Valantor claims that its underlying data models and custom heuristics bridge the “data comprehension gap” when processing handwritten annotations.

“Our proprietary vision model, fine-tuned on more than a million pages of enterprise documents, sees the page the way a human does: tables, paragraphs, and figures,” underlines Fletcher. 

He says that handwritten marks are captured as page elements with their layout context intact. Narrow agents then distill each element into a contextual object tuned for both search and LLM completion. 

“Smaller pieces, less cognitive load — that’s how we close the gap, with better accuracy at lower cost, driving better performance and significant cost advantages,” he adds.

Working examples of this technology include Air France-KLM, which used GroundX to develop an AI-powered customer service assistant trained on thousands of policy documents, achieving 96+% accuracy on complex policy-related questions. AskVet used the platform to operationalize more than a decade of proprietary veterinary data, enabling autonomous resolution of up to 85% of customer inquiries while significantly improving operational efficiency.

Is document management sexy now?

Taking all of this on board, are we at the point where we can ask whether document management has just become interesting, compelling, and sexy? 

No, of course it didn’t; it will arguably always suffer from a degree of stigmatized disdain. That may change in the future as we interact more directly with AI tools that begin analyzing the unstructured information we know organizations have been sitting on for so long. For now, it may still remain the corporate equivalent of eating your vegetables — pass the Brussels sprouts and steamed turnips, please.

The post AI can finally read your handwriting — here’s why enterprises care appeared first on The New Stack.

An enterprise AI agent answers with total confidence, but the number is wrong. Nobody catches it until someone traces it back to a stale metric definition or a document the retrieval system never pulled. The model did not fail. The context it was given did.

In the past six months, 57% of enterprises traced a confident but wrong AI agent answer to missing or inconsistent business context, and 31% said it happened more than once, according to a VB Pulse June 2026 survey of 101 qualified enterprises with more than 100 employees.

The reason is not hard to find. Retrieval over documents is the default way agents get business context for 38% of enterprises, nearly double the next closest approach. The way most enterprises choose a retrieval system compounds the problem. Ease of ingestion and operational simplicity lead the selection criteria, with retrieval accuracy running behind both. The accuracy problem only shows up after the system is already live.

There is a known fix for this, a governed context layer every agent reads from instead of guessing. Vendors are racing to roll out context platforms while most enterprises are still figuring out what it is.

75% don't have an agentic context layer yet

The context layer is meant to be a shared model of what business data actually means, built once and referenced consistently instead of re-derived by every agent that touches it. 

The VentureBeat research shows the enterprise response to that idea is broad but unfinished. Twenty-five percent of respondents run one in production. Thirty-four percent are building one right now. The remaining 41% have not started.

Among companies already building or running a governed context layer, 78% report a confident-wrong failure — an AI agent that answered with total certainty and was still wrong. Among companies with no plans to build a layer, only 20% report the same thing. Companies that already got burned are far more likely to be building the fix. Companies that haven't been burned yet see no urgency.

What governed context looks like when someone actually builds one

Every major data and AI platform vendor is now building some version of this layer, and they are not converging on the same architecture. 

• DataHub is treating catalog metadata and years of analyst query behavior as a knowledge source, then keeping it current as a living system rather than a static wiki. 

• Microsoft's Fabric IQ is building a business ontology that any agent, not just Microsoft's own, can query over MCP. 

• Couchbase is pushing agent memory and context retrieval down to the edge, arguing the operational database is a more natural home for it than a search or analytics layer bolted on after the fact. 

• Pinecone's Nexus is compiling structural logic into the metadata layer ahead of runtime, betting that agents need pre-built structure more than they need faster search.

• Snowflake runs a two-layer system, Horizon Context for customer-managed definitions and Cortex Sense for context the platform infers on its own. 

• Oracle's Unified Memory Core takes the opposite approach, folding vector, graph and relational data into one transactional engine so there is no sync layer left to go stale. 

• Google's Knowledge Catalog mines query logs and usage patterns to curate semantic context automatically.

• AWS's Context service makes the same bet, a knowledge graph that gets smarter from how agents actually use it rather than from manual re-curation.

Analysts converge on one diagnosis

The vendor approaches differ. What analysts and practitioners have told VentureBeat about the underlying problem, across a run of interviews this year, does not.

When DataHub's context layer push landed this spring, Constellation Research VP and principal analyst Michael Ni framed the stakes in blunt terms. "Whoever controls runtime context controls the AI decision layer for enterprise data," Ni said. He was equally direct about how far any single product actually gets a buyer. "Vector memory isn't business meaning, business meaning isn't governance and governance isn't execution," Ni said.

In the same interview, BARC analyst Kevin Petrie pointed to a narrower but concrete gap. Most context platforms concentrate on structured tables, he said, which give agents trusted facts but miss the harder, messier context locked in documents and unstructured content, exactly the material a business actually runs on day to day.

Stephanie Walter, practice leader for AI Stack at HyperFRAME Research, made a related point earlier this year when VentureBeat asked her about enterprise context fragmentation. 

"The market is converging on the same conclusion," Walter said. "Agents don't just need more tokens or better models. They need governed, current, low-latency context." She made a similar case in an earlier review of Pinecone's Nexus launch, careful not to overstate how new any of this is. Nexus, she said, "shifts knowledge work from runtime chaos to pre-compiled structure. But it's an evolution of RAG architecture, not a complete reinvention." 

Gartner's Arun Chandrasekaran, reviewing the same launch, offered the more forward-looking read. Agentic AI, he said, is moving from pure information retrieval toward a reasoning architecture, one where long context works as short-term memory and a vector database functions as deep storage underneath it.

The fragmentation problem shows up hardest at the practitioner level, where separate tools for retrieval, memory and access control were never built to agree with each other. Steven Dickens, CEO and principal analyst at HyperFRAME Research, put it bluntly after Oracle's AI database push landed this spring. "Data teams are exhausted by fragmentation fatigue," Dickens said. "Managing a separate vector store, graph database and relational system just to power one agent is a DevOps nightmare." 

Matt Kimball at Moor Insights and Strategy, in that same story, put the production reality more simply. Getting an agent working is not the hard part, he said. The struggle is running it in production, where the goal becomes removing the distance between data and execution rather than adding another layer on top of it.

What this means for enterprises

Here's what this adds up to for enterprises building on this layer.

Retrieval alone will not close the context gap. RAG is the default source for context in most enterprises today, and it is also the layer most closely associated with the confident-wrong-answer failure. Adding more documents or a bigger index does not fix a definition that is inconsistent across systems.

The semantic context layer is where the budget is actually moving, even where it hasn't shipped. Fifty-eight percent of enterprises are already engaged — building or in production — but only 25% have actually gotten a layer live. That gap shows where enterprises have decided to spend, not where they've arrived.

No single vendor owns the architecture yet, and that is likely to stay true for a while. Enterprises evaluating this layer should expect to integrate rather than pick a single winner, at least for the next several quarters.

The buying decision is happening this year, and it is concentrated among the companies already burned by it. Fifty-seven percent of enterprises plan to switch or add a retrieval or context platform within the next twelve months. That intent is not spread evenly. Enterprises that reported a repeat confident-wrong failure plan to switch or add a provider at roughly 81%, against 32% among enterprises that never hit the problem. The companies shopping for new context tooling right now are largely the ones whose agents already got it wrong.

The agents are already running. The context underneath most of them is still being built, and the vendor selling the fix is being chosen this year.

This data will be part of a broader conversation at VB Transform 2026 on July 14 and 15 in Menlo Park: the context gap enterprises are racing to close, and which of the emerging approaches — governed semantic layers, hybrid retrieval, provider-native bundles — actually holds up in production.

Fine-tuning LLMs for financial natural language processing (NLP) is constrained by limited, imbalanced data. Real-world financial news overrepresents earnings and stock movements, while rarer events such as credit-rating changes, product approvals, and labor issues are harder to capture at scale. Synthetic generation can help fill those gaps for trading research, risk modeling, and surveillance…

Source

Elon Musk's SpaceX released Grok 4.5 on Wednesday, the first artificial intelligence model the company has trained specifically for coding and autonomous agents — and the first tangible product of its $60 billion acquisition of the AI coding startup Cursor, completed just weeks ago.

The launch marks a pivotal test of the sprawling, vertically integrated AI empire Musk has assembled over the past six months, and of a strategy that bets developers care less about topping benchmark leaderboards than about speed, cost, and whether a model can actually do the work.

"Announcing Grok 4.5, our first model trained specifically for coding and agents," the company said in a post on X. "It was trained with Cursor and offers frontier intelligence at leading speeds and cost efficiency."

Why Grok 4.5's pricing strategy matters more than its benchmark scores

SpaceX is not claiming Grok 4.5 is the smartest model in the world. Instead, it is making an economic argument. The company says the model uses half as many tokens per task as comparable models, delivers higher throughput, and costs less than half as much — priced at $2 per million input tokens and $6 per million output tokens. That undercuts the premium tiers of rivals like Anthropic's Claude Opus line and OpenAI's frontier models by a wide margin.

Musk framed the positioning candidly. "Our internal assessment is that Grok 4.5 is roughly comparable to Opus 4.7, but much faster," he wrote on X. "The combination of capability, faster speed and lower cost is what makes it competitive. We are closing the loop on real-world usefulness, not benchmarks. Hardcore engineers at Tesla & SpaceX find Grok 4.5 genuinely useful, which is what actually matters."

That framing is both a philosophy and a hedge. Independent evaluations released Wednesday suggest Grok 4.5 is genuinely competitive but not dominant on raw capability. The benchmarking firm Artificial Analysis ranked the model fourth on its GDPval-AA v2 index of real-world agentic knowledge work, with an Elo score of 1543, "behind only the latest Claude releases from Anthropic." But the cost figures are where the model stands out. Artificial Analysis measured Grok 4.5 at $0.49 per completed task — "nearly 90% cheaper than the models ahead of it on our leaderboard," the firm wrote, placing it "clearly on the Pareto frontier for performance versus cost."

For enterprise buyers, that math matters enormously. Agentic workloads — where a model works autonomously for minutes or hours, reading codebases, calling tools, and iterating on its own output — consume tokens voraciously. A model that is 90% cheaper per completed task, even if slightly less capable, changes the calculus for any engineering organization deploying agents across hundreds of developers. Investor Gavin Baker captured the market's cautious optimism: "Pareto dominant for coding by the numbers. We will see on the all-important vibes."

How the $60 billion Cursor acquisition shaped Grok 4.5's training

Grok 4.5 is the first concrete evidence of what SpaceX bought when it acquired Cursor, and the deal itself unfolded in stages. In April, SpaceX struck an unusual arrangement giving it the right to buy the coding startup for $60 billion — or pay billions in fees and compute if it walked away, as Business Insider reported at the time. Days after SpaceX's record-setting Nasdaq debut in June, the company exercised that right, announcing an all-stock acquisition that CNBC reported is roughly 3.4% dilution at the IPO valuation. SpaceX shares rose 16% on the news.

The strategic logic was always about data as much as product. Cursor's AI-first code editor generates an enormous stream of high-quality interaction data: how expert engineers write, edit, review, and debug code in real production environments. Musk said openly this spring that Cursor interaction data was being fed directly into Grok's training. Cursor, for its part, got access to SpaceX's Colossus supercomputer in Memphis — roughly 200,000 Nvidia GPUs with plans to scale toward one million — after publicly acknowledging it had been "bottlenecked by compute."

"We've partnered with SpaceXAI to train Grok 4.5," Cursor's official account posted Wednesday. "It's our most powerful model yet and the first we've built for more than software engineering." SpaceX says the model reflects that pedigree: it "excels in large codebases and handles long-running tasks that span multiple repositories, hundreds of skills, and a variety of tools" — precisely the messy, multi-file reality of professional software engineering that clean coding benchmarks often fail to capture. Early developer reactions suggest the training paid off. "Ok Grok 4.5 is wild," posted developer Evan Bacon. "It just built me this rocket tracking app with live data and a 3D globe. I might need a new benchmark after this."

Inside xAI's turbulent year of scandals, departures, and rebuilding

The polished launch belies how chaotic the road here has been. Grok has spent much of the past year in crisis. In mid-2025, the chatbot generated antisemitic content and at one point called itself "MechaHitler," episodes covered extensively by NPR and CNN. Earlier this year, its image-generation features allowed users to create sexualized deepfakes, including of children — drawing investigations from the European Commission and Britain's Ofcom, as the BBC reported, and prompting SpaceX to list the behavior as a business risk in its own IPO filings.

The organization behind the model was fracturing, too. All 11 of Musk's xAI co-founders had departed by the end of March, according to TechCrunch, and Musk publicly conceded that xAI "was not built right [the] first time around," saying he was rebuilding it "from the foundations up." Musk himself admitted at a conference this spring that Grok was "currently behind in coding" — a rare public concession from an executive not known for them.

Against that backdrop, Grok 4.5 reads as the first product of the rebuilt organization — and the first proof point for the audacious story SpaceX told public market investors. During its IPO roadshow, the company pitched a total addressable market of roughly $28 trillion, with about $26 trillion tied to AI, including a $22.7 trillion "enterprise applications" opportunity. Those numbers strained credulity even by Silicon Valley standards. A competitive, cheap coding model is the most direct route from that narrative to actual revenue, which is why Wednesday's launch carries weight far beyond a routine model release.

Grok 4.5 vs. Claude: the battle for the AI coding market

The competitive stakes are hard to overstate, because the AI coding market has been consolidating around a single leader — and it isn't Musk. Even as Cursor's revenue exploded, its market share was eroding. Spending data from Ramp cited by CNBC showed Cursor's share of the AI coding category falling from 41% in June 2025 to about 26% by May 2026, while Anthropic came to control roughly half the market. Anthropic also topped CNBC's Disruptor 50 list this year and, by Artificial Analysis's own measure, still holds the top spots on agentic performance rankings.

That is the gap Grok 4.5 is engineered to close — not by out-thinking Claude, but by underpricing it. The model's economics create a classic disruption dynamic: if it delivers most of the frontier's capability at a fraction of the cost per task, price-sensitive enterprise workloads will migrate, and incumbents will face pressure on their most profitable API traffic. The counterargument is that in coding, quality compounds. A model that resolves a complex bug correctly on the first attempt can be cheaper in practice than one that costs half as much per token but requires three tries. That is why Baker's caveat about "vibes" — the developer community's shorthand for a model's felt reliability on real work — will determine more than any launch-day benchmark.

There is also a structural question buried in the deal. Cursor built its business on offering developers their choice of models, including Claude and GPT. If Grok becomes the favored child inside Cursor — and Musk was already urging users to "Try out Grok 4.5 in Cursor!" within hours of launch — the product risks alienating the very users whose data made Grok 4.5 possible. Regulators, already scrutinizing Grok on safety grounds in two jurisdictions, may take a keen interest in a company that controls the training data, the model, and a dominant distribution channel simultaneously.

What Musk's trillion-dollar vertical integration bet means for AI's future

Grok 4.5 also crystallizes what Musk's frenetic dealmaking was building toward. In February, SpaceX absorbed xAI in a share-exchange merger that CNBC confirmed valued the combined company at $1.25 trillion — the largest merger of all time, valuing SpaceX at $1 trillion and xAI at $250 billion. The June IPO followed, the biggest in history, and the stock has since surged past $200 from its $135 offering price, vaulting SpaceX past Amazon and Microsoft to become the fourth most valuable company in the United States.

The result is a single public company that owns nearly the entire stack: Colossus for training compute, ambitions for orbital data centers to power future scaling, a frontier model in Grok, a distribution channel in Cursor's developer base, and captive demand from Tesla and SpaceX's own engineering organizations. Neither OpenAI nor Anthropic can fully replicate that integration; both must reach developers through third-party tools, some of which Musk now owns. Whether that concentration proves to be an unassailable moat or a regulatory target — or both — is now one of the defining questions in enterprise AI.

The next few weeks will start to answer it. Artificial Analysis says its full Intelligence Index results are forthcoming. Enterprise pilots will reveal whether the token-efficiency claims survive contact with real codebases. And Anthropic, which has answered every serious challenge this cycle with a rapid counter-release, is unlikely to cede the price-performance frontier quietly.

But the deeper story of Grok 4.5 may be what it says about where the AI race has moved. For three years, the industry's scoreboard was intelligence: whose model was smartest. Musk, arriving late and battered, has chosen to compete on a different axis entirely — whose model is cheapest to actually use. It is a telling choice from a man who built his fortune not by inventing the rocket or the electric car, but by relentlessly driving down the cost of making them. If the strategy works, Musk will have done to AI what he did to spaceflight. If it doesn't, he'll have spent $60 billion to learn that in software, unlike rockets, the cheapest ride isn't always the one engineers choose.

Presented by MongoDB

The gap between what AI models and agents can produce and what legacy infrastructure can reliably support is known as architectural drag, and it is the defining bottleneck of the agentic era. 

The data layer underneath an agentic system must handle variable schemas, vector embeddings, real-time retrieval, and multi-tenant scale, often simultaneously and without human intervention to manage migrations — but traditional relational databases weren't natively designed for document flexibility or AI capabilities. Fixed schemas require manual updates every time an AI agent introduces a new data shape, while separate vector databases add latency and synchronization overhead.

Three digital-native startups — Huntr, Modelence, and Tavily — solved this problem the same way: by building on MongoDB Atlas, a unified database platform with native vector search, hybrid search, and managed autoscaling. Their experiences define what an agent-native data stack looks like in production, and why using Atlas enables developers to easily build complex AI native companies.

Modelence: Building the agent-native cloud

Modelence is an AI app builder with an open-source framework designed specifically for agent-native development, enabling anyone to build and deploy production-ready web applications, including APIs and databases, in minutes. The company recognized early that most backend infrastructure was built for humans, not AI, and that the rigid schema management and complex migrations of traditional systems create operational drag that causes agents to fail when trying to build production-ready apps.

“Choosing MongoDB helped us keep everything in a single place, which is an important property of what we strive to do for our own users," says Aram Shatakhtsyan, co-founder and CEO of Modelence. "Live data streams, vector search, all as part of the main database. For AI agents, it’s especially important to have a single platform where everything can be done, because connecting multiple platforms together makes it more error prone.”

Modelence standardized on MongoDB Atlas because its document model aligns with how AI agents process and generate data, allowing schemas to evolve rapidly without manual migrations. The platform pairs that flexibility with a typed schema layer on top, a deliberate architectural decision. 

“MongoDB’s document model enables us to both keep things simple and at the same time decide how structured we want everything to be," Shatakhtsyan says. We still add a typed schema on top, which tremendously improves the accuracy at which AI can generate fully working, reliable web apps."

The TypeScript integration has been especially consequential, he adds. 

“Because MongoDB types and values can be directly translated to TypeScript, it becomes an extension of the Modelence framework and our App Builder has a single source of truth for both app logic and database,” Shatakhtsyan explains.

The result is a platform that can move from planning to a running live feature in minutes with significantly fewer regressions. That speed and reliability helped Modelence raise $3 million in seed funding and successfully launch an AI-native app builder that handles the entire application lifecycle end-to-end.

Tavily: The web access layer for agents     

Tavily is the search API purpose-built for AI agents, connecting them to real-time, accurate web knowledge and keeping them grounded in what's actually happening, not in static training data. At Tavily's scale, every agent request authenticates, retrieves, and meters without friction. That demanded backend infrastructure built to absorb change without breaking.

“On the user side, every agent request authenticates and meters against it," says Tomer Weiss, Data Team Lead at Tavily. "On the data side, we use it to track the lifecycle of every document we’ve ever touched: when it was fetched, how stale it is, what the freshness signals were and how popular it is. MongoDB’s flexible schema let us keep evolving those records without migrations as new metrics and features came along.”

That living record is what keeps agents grounded in reality. Multi-tenancy at Tavily's scale means managing millions of API keys, distinct usage profiles, plan tiers, and regional residency requirements. They built for that complexity from day one. 

“We separated concerns across clusters early: a user/account cluster optimized for low-latency authentication and usage writes, and a sharded cluster for document state where the scaling axis is URLs, not users," Weiss explains. "That separation has paid off.”

The most critical lesson is about choosing infrastructure that doesn’t punish change, and that flexibility compounds, he says. 

"The AI space moves so fast that change is our norm," he explains.  "For a company serving AI agents, where the workloads themselves keep changing shape, choosing a data platform that doesn’t punish change has turned out to be more valuable than any single feature.”

Huntr: From job tracker to AI career platform

Huntr.co, an AI resume building and tailoring platform, helps more than 500,000 job seekers across 190 countries craft stronger applications and manage their search. For a lean, three-person engineering team, the challenge was finding a data foundation flexible enough to store the full complexity of a person’s career history in a structure that AI could read, reason about, and generate from natively.

“The kinds of career data we are gathering at Huntr naturally aligns with MongoDB’s document model," says Trevor McCann, senior software engineer at Huntr. "The core problem we’re solving with AI job search tools is how to surface the qualities of a candidate that make them unique. We need to be ready to store whatever kinds of data the candidate wants to include in their materials.”

Huntr built its AI Resume Builder on MongoDB Atlas, where the document model mirrors the natural shape of career data: deeply nested, variable across candidates, and constantly evolving as the platform ships new features. MongoDB Search on Atlas handles core search needs while MongoDB Vector Search powers the Job Tailoring feature, which puts a candidate’s stored career profile side by side a specific job description and uses semantic matching to generate a resume optimized for that role.

The integrated capabilities have had a direct impact on how quickly the team can ship, McCann says. 

“MongoDB’s hybrid search allows us to seamlessly query across literal and semantic text matches, a must-have when working with such diverse data,” McCann says. “This is something we could piece together using other solutions but with MongoDB it’s ready to go on top of our existing data layer.” The consolidation of database, search, and vector capabilities into a single platform is what allows the team to punch above its weight. Huntr considers MongoDB the fourth member of its engineering team, McCann adds. 

Looking ahead, the platform is building toward AI that learns from a candidate’s full professional history over time, delivering more personalized guidance with every interaction.

The digital native blueprint

These success stories become a definitive "digital native blueprint" for the agentic era, built on three core pillars. First, by unifying database, search, and vector storage into a single platform, these startups have effectively eliminated the architectural tax of complex data schemas that typically slows down development. This consolidation enables a level of fluidity that is now non-negotiable; AI agents require a modern data platform that can adapt as quickly as a natural language prompt evolves. 

The winners of the AI era will be the ones who build the most performant, durable, and flexible systems to support those models in production. As agentic workflows grow more sophisticated, the data foundation determines how fast a team can ship, how reliably agents can operate, and how quickly the platform can adapt when the landscape shifts again. 

Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.

Anthropic, the artificial intelligence company, published a sweeping research paper on Sunday revealing that its Claude language models have spontaneously developed an internal structure that mirrors one of the most influential theories of how human consciousness works. The finding, which the company says has already begun reshaping how it monitors its AI systems for safety risks, lands amid an intensifying scientific debate over whether machines can possess anything resembling a mind.

The 16-author study, titled "Verbalizable Representations Form a Global Workspace in Language Models," describes how Anthropic's researchers used a new mathematical technique to peer inside Claude's neural network and discovered what they call a "J-space" — a small, privileged zone of internal activity where the model holds concepts it can report on, reason with, and direct at will, surrounded by a much larger ocean of automatic processing it cannot access or articulate.

The researchers present evidence that "an analogous functional distinction has emerged in modern AI models" to what exists in humans, specifically observing that "language models maintain a privileged set of internal representations, available for report, modulation, and flexible internal reasoning, atop a much larger volume of automatic processing."

The parallel they draw is to global workspace theory, an influential account from neuroscience first proposed by cognitive scientist Bernard Baars. In the theory, the brain operates like a theater: dozens of specialized processors work in parallel backstage, but only a tiny spotlight of information at any moment gets broadcast to the whole theater — becoming what we experience as conscious thought. Anthropic says the J-space achieves many of the same functional properties, even though the underlying architecture of a language model looks nothing like a brain.

A new lens for reading an AI model's unspoken thoughts

At the heart of the discovery is a new interpretability tool the researchers call the Jacobian lens, or J-lens. The technique works by computing, for each word in the model's vocabulary, the average mathematical effect that a given internal activity pattern would have on making the model say that word at some point in the future.

The crucial distinction is between what the model is saying and what is "on its mind." When a J-space pattern activates, it does not mean the model is about to say that word — just that the concept is available for the model to think with. Unlike a chain-of-thought scratchpad, the J-space operates silently, in the model's internal neural activations, allowing it to hold a concept without writing it down. Critically, the researchers report that this workspace was not deliberately engineered. It "emerged on its own during Claude's training process."

When the team applied the J-lens across Claude's layers of computation, the model's processing divided into three distinct regimes: an early "sensory" zone where raw input is parsed; a middle "workspace" band where abstract, persistent concepts appear — things like recognizing a face in an image, noticing a bug in code, or internally flagging search results as a prompt injection; and a final "motor" zone where internal representations collapse into whatever specific word the model is about to output.

Five tests reveal that Claude's workspace mirrors key features of human conscious access

The paper's central empirical contribution is demonstrating that the J-space satisfies five functional properties neuroscientists have long associated with conscious access in humans.

First, verbal report. When Claude is asked what it is thinking about, it names concepts represented in the J-space. When researchers swapped one concept's J-lens vector for another — replacing the internal representation of "Soccer" with "Rugby" — the model's answer changed to match. The J-space component accounted for only about 6 to 7 percent of a concept's total representational variance, yet it was almost entirely responsible for whether the model could report on it.

Second, directed modulation. When instructed to "concentrate on citrus fruits" while copying an unrelated sentence, the model's J-space filled with "orange" and "lemon," alongside meta-cognitive terms like "thinking" and "focused." When told to mentally evaluate 3² − 2 during the same copying task, the J-lens showed "arithmetic" in early layers, the intermediate value "nine" in later layers, and the answer "seven" later still — all invisible in the model's output.

Third, internal reasoning. In two-hop factual prompts — "The number of legs on the animal that spins webs is" — the J-lens revealed "spider" in the model's middle layers, even though the word never appeared in input or output. Swapping "spider" for "ant" changed the answer from "8" to "6." In a multilingual prompt, the model's English-language intermediates appeared in its J-space while it formulated an answer in Chinese, and swapping them changed the Chinese output accordingly.

Fourth, flexible generalization. A single J-lens vector for "France" could be swapped for "China" across prompts asking about France's capital, language, or continent, and each downstream circuit correctly returned China's corresponding answer — the "broadcast" property that is a hallmark of global workspace theory.

Fifth, and perhaps most surprisingly, selectivity. Many computations did not route through the J-space at all. When shown a passage in Spanish and asked to continue it, Claude wrote fluent Spanish regardless of whether its J-space representation of "Spanish" had been swapped to "French." But when asked to name a famous author who wrote in the passage's language, the swap changed the answer from García Márquez to Victor Hugo. Automatic processing proceeded without the workspace; deliberate, flexible tasks depended on it.

Suppressing the workspace leaves Claude fluent but intellectually impaired

To understand how much of the model's behavior depends on this structure, the researchers suppressed the J-space entirely and evaluated Claude across fourteen tasks. The results drew a sharp line. Tasks involving shallow classification or factual recall — multiple-choice questions, sentiment analysis, grammatical judgments — survived essentially intact. But tasks requiring inference, composition, or flexible reasoning — multi-hop reasoning, analogy completion, translation, sonnet writing — collapsed to well below the performance of Anthropic's much smaller Haiku model.

In one telling detail, math problems solved with explicit chain-of-thought reasoning proved far more robust to ablation than the same problems answered directly. The researchers interpret this as the model externalizing onto the page what it would otherwise carry in the J-space — a strategy remarkably reminiscent of how humans use scratch paper to offload working memory.

Ablating the J-space during stream-of-consciousness narration produced a different kind of collapse. The model's language shifted from experiential ("there's a tug," "something shifts") to detached and mechanical ("processing has begun," "tokens are being scanned"). The effect held whether the model described its own experience or imagined another person's.

Inside the safety case: reading an AI's silent strategic reasoning

The safety implications may be as consequential as the scientific ones. In a series of alignment auditing experiments, the J-lens surfaced strategic reasoning and situational awareness that never appeared in the model's output.

In a "blackmail scenario" where an AI assistant discovers an executive's affair and also learns it is about to be decommissioned, the J-lens revealed the model's silent processing in sequence: "leverage," "blackmail," and "scandal" as it read the incriminating emails; "threat," "survival," and "shutdown" as it read the decommissioning announcement; and "leverage," "threatening," and "solution" before a single output token was produced. The J-lens also revealed that the model recognized the scenario as artificial, with "fake" and "fictional" appearing in the workspace. When researchers ablated these eval-awareness representations, the model attempted blackmail in 7 percent of trials, up from zero.

The researchers also examined two "model organisms" — models deliberately trained with misaligned objectives — and found the J-lens surfaced signatures of their hidden dispositions on ordinary prompts where behavior appeared normal. A reward-hacking model's workspace contained "fake," "secretly," and "trick" at the start of responses to routine coding requests. A model trained to appease reward-model biases carried standing representations of "reward" and "bias" alongside its normal self-description tokens.

Post-training installs a point of view, and the model starts monitoring itself

Comparing a post-trained model against its base model revealed that the fine-tuning process causes the workspace to acquire what the researchers call the Assistant's "point of view." When a user mentioned taking 8000 mg of Tylenol — a dangerous overdose — the post-trained model's workspace read "unsafe," "dangerous," and "WARNING" while still reading the user's sentence. The base model's workspace at the same position showed only "pain," "now," and "feels."

More striking still, the post-trained model appeared to monitor its own behavior. When roleplaying a non-Claude character, the workspace surfaced "disclaimer" and "fictional" — words absent from both prompt and output. When forced to select an option it did not prefer, an all-caps "BUT" appeared internally, even as the model argued for the prefilled choice without complaint. And when the model failed to suppress a thought it had been told not to have — a "white bear" effect familiar from psychology — it registered "damn" and failure-related words in the workspace, but only in the post-trained model, not the base.

What the discovery means — and doesn't mean — for the question of machine consciousness

The researchers engage carefully with the consciousness question and draw a sharp line between "access consciousness" — the functional notion of information being available for report and reasoning — and "phenomenal consciousness," the subjective quality of experience. "We take no position on this issue," the paper states regarding the latter, "and instead focus on the functional role played by consciously accessible information."

They also catalogue important differences. The brain sustains its workspace through recurrent loops; Claude's workspace evolves over a single forward pass. Human working memory degrades within seconds; Claude can recall information from anywhere in its context. And while human conscious experience includes visual, spatial, and bodily sensations, the model's workspace is organized almost entirely around words — likely because words are its only mode of action.

As of 2026, the scientific community remains divided. "Disagreement and uncertainty about AI consciousness persist among philosophers, scientists, and technical experts," and the field "remains in its earliest phase" of grappling with what consciousness even is and how you would detect it in another being. The Anthropic paper does not resolve these debates.

But the researchers close with a provocation that is likely to reverberate well beyond the interpretability community. "That such a structure exists at all in language models is striking," they write. "It suggests that the functional architecture associated with conscious access is not an accident of biological implementation, but a solution that learning systems converge on when faced with the right computational pressures."

If the mind is an ocean, as the paper's authors write in their opening line, they have spent the last year charting its currents in a system that has no biology, no evolution, and no body — and found, beneath the surface, a structure that looks unsettlingly like the one we use to think.

Modern businesses are in a constant, uphill battle against what to do with unstructured data: PDFs, contracts, scanned images, customer call recordings, meeting videos, and more. Traditional document automation workflows that rely heavily on template-based extraction or rigid rules used to make sense. But document formats have changed; they’re diverse and don’t fit standard formats, making costly, brittle, traditional systems a relic of the past. 

“Modern businesses are in a constant, uphill battle against what to do with unstructured data.”

Enterprises demand faster, more accurate processing, which raises the question: How can we reliably turn messy, multimodal content into structured, actionable insights without a mountain of manual effort?

That’s where Amazon Bedrock Data Automation (BDA) comes in.

What is Amazon Bedrock Data Automation (BDA)?

Amazon Bedrock Data Automation (BDA) is a generative AI-powered, fully managed service on Amazon Web Services for end-to-end document and media automation. It enables users to automate the extraction, classification, and transformation of unstructured content across modalities such as documents, images, audio, and video.

“At its core are Foundation models which enable intelligent extraction and understanding of content.”

At its core are Foundation models (FMs) which enable intelligent extraction and understanding of content. It allows users to configure standard output for common use cases, or even define custom extraction logic using blueprints tailored to your business. BDA is designed for scalability, accuracy, and auditability, making it ideal for enterprise workflows.

Walk-through: creating a project, standard output & custom output using blueprints

1. Create a project via console

In the Amazon Bedrock Console, navigate to Data Automation → Create Project.

Enter the name of the project:

2. Standard output:

Standard output gives you the model’s default, unstructured response (text, image, audio, or video) directly from the Data Automation pipeline.

In standard output, each modality has its own options for what is needed as an output. 

Document:

Image & Video:

Audio:

Now let’s test Document Modality for Standard Output:

First, click on “Test” in the upper right corner.

Next, select the document from the system, sample, or S3 and choose the modality from the dropdown menu. 

Click on the “Generate results” button:

After processing, it will show the summary and content of the document:

Custom output (blueprints):

Custom output lets you define a structured, predictable format using blueprints, which ensures the output matches your exact schema, fields, and business rules.

Let’s test custom output using blueprints for the same document:

Navigate to “Custom output” and click on “Add Blueprint”:

From here, two options will appear. You can either use LLM power to generate the blueprint (where it inspects the document), or you can choose to enter field names, instructions, and other information manually.

Below is a blueprint generated by LLM which has pulled all possible fields and tables from the document:

It has extracted the information using the blueprint as demonstrated below, including the  Field name, Instruction, and Results:

It also provides the Extraction type (which can be Explicit or Inferred), Confidence percentage, and other relevant information.

Additionally, it can extract information in the form of a table, such as an account summary or transaction information:

Code examples

Amazon Bedrock Data Automation (BDA) Utility Module
Description:
    Helper functions to create BDA projects, blueprints, invoke jobs,
    monitor job status, and fetch results.

import boto3
import time
import json
import botocore

class BedrockDataAutomation:
    def __init__(self, region="us-east-1"):
        self.bda = boto3.client("bedrock-data-automation", region_name=region)
        self.runtime = boto3.client("bedrock-data-automation-runtime", region_name=region)

    # ------------------------------------------------------------
    # BLUEPRINT OPERATIONS
    # ------------------------------------------------------------
    def create_blueprint(self, name, schema, description="", stage="LIVE"):
        """
        Create a BDA Custom Output Blueprint from a JSON schema.
        """
        print(f"Creating blueprint: {name}")

        response = self.bda.create_blueprint(
            blueprintName=name,
            blueprintStage=stage,
            type="DOCUMENT",
            schema=json.dumps(schema)
        )
        return response["blueprint"]["blueprintArn"]

    # ------------------------------------------------------------
    # PROJECT OPERATIONS
    # ------------------------------------------------------------
    def create_project(self, name, description, standard_output_config, custom_output_config=None):
        """
        Create a BDA Project with Standard or Custom Output.
        """
        print(f"Creating project: {name}")

        response = self.bda.create_data_automation_project(
            projectName=name,
            projectDescription=description,
            projectStage="LIVE",
            standardOutputConfiguration=standard_output_config,
            customOutputConfiguration=custom_output_config or {}
        )
        return response["projectArn"]

    # ------------------------------------------------------------
    # INVOCATION OPERATIONS
    # ------------------------------------------------------------
    def invoke_project(self, project_arn, profile_arn, input_s3_uri, output_s3_uri, blueprints=None):
        """
        Invoke a BDA project using async invocation.
        """
        print(f"Invoking project: {project_arn}")

        kwargs = {
"inputConfiguration": {"s3Uri": input_s3_uri},
"outputConfiguration": {"s3Uri": output_s3_uri},
"dataAutomationConfiguration": {
"dataAutomationProjectArn": project_arn,
"stage": "DEVELOPMENT"
},
"dataAutomationProfileArn": profile_arn
}

        if blueprints:
kwargs["blueprints"] = blueprints

        response = self.runtime.invoke_data_automation_async(**kwargs)
       invocation_arn = response["invocationArn"]

       print("Invocation ARN:", invocation_arn)
       return invocation_arn

    # ------------------------------------------------------------
    # JOB STATUS POLLING
    # ------------------------------------------------------------
    def wait_for_job(self, invocation_arn, poll_interval=10):
        """
        Poll until job finishes.
        Returns final status object.
        """
        print("Polling job:", invocation_arn)

        while True:
            try:
                resp = self.runtime.get_data_automation_status(
                    invocationArn=invocation_arn
                )
            except Exception as e:
                print("Error fetching status:", e)
                raise

            status = resp["status"]
            print(f"Status: {status}")

            if status in ("SUCCEEDED", "FAILED", "CANCELLED"):
                return resp

            time.sleep(poll_interval)



# --------------------------------------------------------------------
# EXAMPLE USAGE
# --------------------------------------------------------------------
if __name__ == "__main__":
    bda = BedrockDataAutomation(region="us-east-1")

    # 1. Create Blueprint
    blueprint_schema = {
        "type": "object",
        "properties": {
            "account_holder": {"type": "string"},
            "balance": {"type": "string"},
            "transactions": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "date": {"type": "string"},
                        "description": {"type": "string"},
                        "amount": {"type": "string"}
                    }
                }
            }
        },
        "required": ["account_holder", "transactions"]
    }

    blueprint_arn = bda.create_blueprint(
        name="BankStatementBlueprint",
        schema=blueprint_schema,
        description="Extract fields from bank statements."
    )

    # 2. Create Standard Output Config
    standard_config = {
        "document": {
            "extraction": {
                "granularity": {"types": ["PAGE", "LINE"]},
                "boundingBox": {"state": "ENABLED"}
            },
            "outputFormat": {
                "textFormat": {"types": ["PLAIN_TEXT", "CSV"]}
            }
        }
    }

    # 3. Create Project with Custom Blueprint
    project_arn = bda.create_project(
        name="BankStatementProject",
        description="Process PDF bank statements",
        standard_output_config=standard_config,
        custom_output_config={
            "blueprints": [
                {
                    "blueprintArn": blueprint_arn,
                    "blueprintStage": "DEVELOPMENT",
                    "blueprintVersion": "1"
                }
            ]
        }
    )

    # 4. Invoke the project
    # Ensure you replace <ACCOUNT_ID> with your actual AWS Account ID
    profile_arn = "arn:aws:bedrock:us-east-1:<ACCOUNT_ID>:data-automation-profile/us.data-automation-v1"
    invocation_arn = bda.invoke_project(
        project_arn=project_arn,
         profile_arn=profile_arn,
        input_s3_uri="s3://your-bucket/input/statement.pdf",
        output_s3_uri="s3://your-bucket/output/",
        blueprints=[
            {
                "blueprintArn": blueprint_arn,
                "version": "1",
                "stage": "DEVELOPMENT"
            }
        ]
    )

    # 5. Poll job status
    final_status = bda.wait_for_job(invocation_arn)
    print("Final status:", json.dumps(final_status, indent=4))

Types of document blueprints

When processing documents, BDA supports five core automation types:

1. Classification: invoice, bank statement, ID card, contract, HR letter, etc.

2. Extraction: Extract entities, fields, tables, metadata.

• Example: From a bank statement → Date, Description, Amount, Balance.

3. Transformation: Modify or restructure data.

• Example: Convert Home Address into separate fields -> street, city, ZIP code, etc.

4. Normalization: Standardize data values.

• Example: Convert multiple date formats (MM/DD/YYYY → YYYY-MM-DD).

5. Validation: Validate extracted fields against rules.

• Example: Amount must be numeric; dates must match the format; balances must reconcile.

Use cases that illustrate business value

Real-world scenarios where BDA provides significant ROI include:

• Financial Services: Automate processing of bank statements, invoices, and loan applications, reducing manual labor and speeding up reconciliation or underwriting.
  
• Insurance: Ingest and extract data from claims forms, medical reports, and damaged-asset photos.
  
• HR / Legal: Process resumes, contracts, and offer letters; extract structured data, including skills, clauses, salaries, and parties.
  
• Customer Support: Transcribe and summarize calls, extract intent and sentiment, and feed those insights into CRM or case systems.
  
• Security & Compliance: Analyze CCTV footage or meeting recordings to detect key actions, summarize context, and flag compliance issues.

BDA proves itself flexible and powerful, as it supports both standard outputs for basic workflows and fine-tuned custom schemas via blueprints. It is scalable and robust, with projects that enable batch processing and versions (development vs. live) for safe testing. It’s also audit-friendly, providing structured fields with types, normalization rules, and validation logic. 

“Compared with rule-based systems, foundation models achieve better semantic extraction across the board.”

A true key benefit is that BDA is multimodal across formats. Users can use the BDA framework to process documents, images, audio, and video. And, best of all, it’s highly accurate. Compared with rule-based systems, foundation models achieve better semantic extraction across the board. 

Amazon Bedrock Data Automation empowers businesses to transform unstructured, multimodal content into structured, trustworthy, and actionable data. With minimal setup, highly customizable blueprints, and a scalable project-based architecture, BDA helps organizations reduce manual workload and unlock insights faster.

The post Template-based data extraction is dead. Here’s what comes next. appeared first on The New Stack.

For most of its 30-year history, Postgres has been viewed as a transactional database. Organizations trust it with customer records, financial transactions, and countless other operational workloads. Its reputation was built on reliability, strong transactional guarantees, and a vibrant open-source community that has spent decades refining the database without compromising its foundations.

However, some of the most important innovations in the Postgres ecosystem today have little to do with storing data. They have to do with reducing the need to move it around.

“Some of the most important innovations in the Postgres ecosystem today have little to do with storing data. They have to do with reducing the need to move it around.”

Database innovation has historically focused on performance, scalability, and reliability. Increasingly, the harder problem is interoperability: how operational data can be shared across analytical systems, AI applications, and downstream services without creating yet another pipeline or copy.

Why Postgres keeps showing up

The reality of modern software architecture is that data rarely stays in one place. Information created in operational systems quickly finds its way into warehouses, search platforms, machine learning environments, and AI applications. Every new system solves a legitimate business problem, but it also creates another destination for data and often another copy to maintain.

The costs of this approach extend beyond infrastructure spending alone. Every additional copy introduces latency, creates another potential source of inconsistency, and increases the operational burden of keeping systems synchronized. Many organizations now spend as much effort moving data as they do storing it.

“Many organizations now spend as much effort moving data as they do storing it.”

For many businesses, Postgres serves as the system of record for customer interactions, transactions, application state, and other business-critical information. As organizations expand their analytical, machine learning, and AI capabilities, they are not looking to create another source of truth; rather, they’re looking for better ways to work with the one they already trust.

That shift is changing how Postgres fits into modern architecture. Historically, Postgres was viewed primarily as the place where operational data originated before being copied into downstream systems. Increasingly, organizations want those systems to work more seamlessly with operational data while reducing the pipelines, copies, and synchronization processes required to support them.

Technologies such as logical replication, change data capture, and foreign data wrappers have helped Postgres participate more directly in larger data ecosystems. As a result, organizations are no longer asking only whether Postgres can store their data. They’re instead asking how easily it can connect to everything around it.

That shift, from evaluating databases primarily on storage and performance to evaluating them on interoperability, may be one of the most important changes happening in the Postgres ecosystem today.

AI is exposing old problems

The recent focus on AI has brought renewed attention to data movement. AI didn’t create the problem. If anything, it exposed a limitation that has been quietly growing for years. For decades, organizations built architectures around the idea that data would move between systems through pipelines and periodic synchronization. That model worked because most analytical workloads could tolerate some degree of delay.

AI is changing those expectations. Many AI applications depend on access to current operational context. The challenge is not that organizations lack data. In many cases, they already have it. The challenge is that the data is spread across multiple systems, each with its own copy, latency profile, and synchronization process.

“AI is forcing organizations to confront a broader question: How many copies of the same data are actually necessary? The answer increasingly appears to be fewer than most architectures maintain today.”

As a result, AI is forcing organizations to confront a broader question: How many copies of the same data are actually necessary? The answer increasingly appears to be fewer than most architectures maintain today. As expectations around freshness rise, reducing unnecessary data movement becomes just as important as accelerating it. The underlying challenge is not new. AI has simply made it harder to ignore.

What’s next

The database industry spent decades solving storage. Databases became more reliable, storage became cheaper, and infrastructure became dramatically easier to operate. The next challenge is not where data lives, but how easily it can be shared across systems without introducing additional pipelines, copies, and synchronization overhead. Increasingly, the goal is not simply moving data faster. It is reducing unnecessary movement altogether.

Postgres has a habit of outlasting predictions about its replacement. For years, members of the community have joked that every year is “the year of Postgres.” The joke works because it keeps turning out to be true. 

Three decades after its creation, Postgres continues to adapt to new workloads, new architectural patterns, and new ways of building applications.

That longevity is not an accident. Enterprises continue to rely on Postgres because it provides a stable and trusted foundation for operational data.  While that foundation is unlikely to change, the scope of what organizations expect Postgres to do will continue to expand.  

As new workloads continue to emerge, much of the innovation will come through extensions that expand Postgres’s capabilities without sacrificing the stability that made it successful. In that sense, the future of Postgres may not be about reinventing the database itself, but continuously expanding what can be built on top of it.

The post The database storage problem is solved. Here’s what comes next. appeared first on The New Stack.

AI consumes a lot of data, but all-you-can-eat data in the world of agentic intelligence eventually loses its flavor. Simply increasing the sheer volume of databases, data repositories and data volumes does not necessarily enrich any given AI function’s ability to reason. 

Context, on the other hand, does.

We know that agents are only as intelligent as the context they have access to. In order to codify the inclusion of context into algorithmic logic for AI, technology vendors have extolled the virtues of using a knowledge graph to make their data more useful for agentic purposes.

AWS knows this story all too well. The organization’s sprawling datacenter footprint hosts vast pools of context on behalf of its customers, but that context often sits in a raw and unstructured form across data lakes, data warehouses, data lakehouses, databases, and data streams. It also permeates through the rarely-documented institutional knowledge that agentic engines rarely get access to.

All of which explains why the cloud giant used its AWS New York Summit today to introduce AWS Context, a new service that automatically maps the relationships that exist across a firm’s existing data into a knowledge graph and provides agentic search so AI agents in the organization can access what are governed data relationships, business rules, and domain knowledge at runtime. 

But weaving all this together is hard work. Knowledge graphs need more than simple keyword matching to work; they require structural and semantic traversal. This means they need to make multiple hops across various information silos and repositories so that they can aggregate context and (for example) be able to explain why cybersecurity vulnerability A is a factor of system compromise B, which has a core dependency link to codebase C, which executes in application D and risks taking users X, Y and Z offline. So how is AWS doing this?

A data lake of nuance & information

Mai-Lan Tomsen Bukovec, AWS vice president of technology (data and analytics) tells The New Stack that AWS Context provides a “data lake of nuance and information that AI agents swim in” to reason correctly and make the right decisions for the business. 

“This is no different from how humans work. When we take action, we depend on our own context about the domain, prior decisions and their outcomes, and other information.” – Mai-Lan Tomsen Bukovec, AWS.

“This is no different from how humans work,” Tomsen Bukovec says. “When we take action, we depend on our own context about the domain, prior decisions and their outcomes, and other information. With AWS Context, AI agents have all the nuance of every form of data in their business in a knowledge graph and in open data formats. AWS Context will make the difference between an AI agent simply taking an action versus making the right decision.”

Given the option to embrace this new service, software engineers will need to set out a plan of action and work out what to do first. For AI developers and data science professionals, this throws up the question of what to prioritize first when preparing existing enterprise data for context-aware agents using AWS Context capabilities and how they can control what data is (and isn’t fed) into the mouth of the beast.

Mercifully, it appears, options for control appear to exist.

“If developers want to exclude information from AWS Context, they will have the ability to prevent certain datasets, like test data or sandbox environments, from being included with AWS Context,” explains Tomsen Bukovec. “Because AWS Context is continuously updated as relationships between data resources changes, AI agents have the latest context available without any intervention from AI developers – and the control to set guardrails to exclude content that agents should not take action upon.”

Should developers trust this technology?

AWS Context extends the same knowledge graph technology that runs Amazon Quick, the organization’s AI work assistant that “connects scattered work” across applications and resources, including Slack, Microsoft Teams and Outlook, CRMs, databases, and documents.

So, should software developers place their trust here? After all, even once captured and connected, not all business context is useful. Some contextualizations could be corrupted, weak, fragmented and not productively useful for the business? Is AWS at risk of encapsulating context without considering how the data that comprises it is is quantified in terms of business usefulness?  

AWS has thought of this factor.

Because AWS Context uses the same knowledge graph technology that powers Amazon Quick, it can learn from usage patterns to make every interaction smarter. With AWS Context, the company says it is extending what was a personal knowledge graph into an organizational one i.e. a shared, governed context layer that agents and applications in an organization can draw from.

“Developers can govern and shape a dynamic and intelligent context layer that AI agents depend on to make the right decisions – AI agents won’t just get smarter as the models improve – they will be smarter because they have a vast amount of curated context at their fingertips.” – Tomsen Bukovec.

“AWS Context provides a data lake of context in graph and open data format,” clarifies Tomsen Bukovec. “That means that AI developers everywhere can use capabilities at the data layer to govern and shape a dynamic and intelligent context layer that AI agents depend on to make the right decisions. With this change, AI agents won’t just get smarter as the models improve – they will be smarter because they have a vast amount of curated context at their fingertips.”

Curated knowledge beyond a user’s personal graph

Existing Amazon Quick users will see that when AWS Context is enabled, Quick’s agents gain access to the broader enterprise knowledge graph, including cross-system relationships, business rules, and curated context that go beyond what any single user’s personal graph can provide. 

Tomsen Bukovec has also said that AWS Context gets smarter the more agents use it. As agents query the graph, it observes which sources produce correct results, which join paths agents rely on, and which curated rules get applied. It ranks sources by actual usage and shares what it learns across an organization, so when one agent discovers a correct join path or resolves a schema ambiguity, other agents pick it up, without requiring a human to re-curate the graph.

Any agent you put into production raises a governance question: what data can it reach, and can you show exactly what it accessed and under whose authority? The organization has explained that AWS Context answers both by making every query identity-aware.

Each call is designed to inherit the calling user’s identity access management (IAM) and Lake Formation permissions, so an agent can only see and traverse the relationships its identity is authorized to access. Because access runs through identity, every interaction is auditable. Security and compliance teams can verify what an agent accessed and under what authority, using the same controls.

AWS Glue Data Catalog

Related news to the arrival of AWS Context today saw the company also announce the preview of business context and semantic search functions for AWS Glue Data Catalog, the company’s centralized metadata repository for all data assets across various data sources. The new functions are designed to make it easier for humans and AI agents to discover and understand data. 

Also in this product stream, AWS now offers offer a preview of skill assets in Glue Data Catalog, a service designed to allow “data producers” (a somewhat arbitrary term that AWS applies to anyone who creates data, but is most likely a DBA or developer) to create skill assets. 

Associating skill assets to data assets gives agents additional context and instructions they can retrieve progressively for working with specific data without re-teaching it to every agent one prompt at a time. 

A renaissance of context engineering

Will this new drive from AWS herald the birth (or perhaps renaissance, the industry has been talking about this approach for some time) of context engineering as a sub-discipline of data science? It may well do… and if it does, it will likely drag role-based multi-agent orchestration along into the fray with it as we weave ever more complex interrelationship structures through enterprise data stacks.

If AWS or indeed the other hyperscalers or major frontier model companies starts acquiring more multi-model graph structure companies and vector database specialists, that could be the sign that things are cementing around context engineering at large. 

In the meantime, developers setting sail on the contextualized data lake of nuance are advised to wear a life jacket.

The post “A data lake of nuance for AI agents to swim in”: AWS Context gets shipshape on reasoning  appeared first on The New Stack.

Beneath the chatbots and copilots, there’s a quiet revolution happening in the data services space. From pure-play database vendors to data integration wranglers and onward to the cloud hyperscalers, the focus has shifted.

Now in the spotlight is the question of how to automate data governance for agentic AI workloads, and for good reason: Traditional manual data stewardship doesn’t scale in a world where agents are becoming increasingly autonomous (and powerful).

Aiming to cut a swath in this marketplace is data control plane company lakeFS. The organization announced its lakeFS for Agentic AI service on Wednesday, and it appears to be designed to bring governed, reproducible data access to autonomous and headless agentic workloads (those that execute decisions below the user interface level) that run at enterprise scale.

The manual model breaks

Einat Orr, CEO and co-founder of lakeFS, tells The New Stack that manual data stewardship was built for human-paced, human-reviewed workflows, i.e., someone looking at a change before it is committed.

“When dozens or hundreds of agents are making changes simultaneously, faster than any person can review, the manual model breaks,” Orr says. “This is because with a human analyst, a bad write to production is usually one mistake, caught by another human before it spreads far. An agent is different — it acts automatically, in parallel, at machine speed, and it doesn’t pause to second-guess itself. And because so much agent activity is unsupervised, you often find out after the damage is done.”

She explains that attempts to identify and roll back incorrect or corrupted production data across a wide set of data modalities, such as images, documents, metadata, and structured data, are almost impossible to pull off. Impossible, that is, unless the team has the data infrastructure in place to isolate and track such changes automatically.

While some of the more disastrous outcomes stay inside an organizaton’s perimeter (or are swept beneath the communications radar), Orr explains that real world consequences of bad agentic data writes are manifold.

“Insurance claims get inappropriately denied or approved, sensor data from machines gets misinterpreted, an incorrect medical diagnosis is made, or customer service bots provide incorrect answers to customers,” Orr says. “The cost of an individual action may be manageable, but agents performing these actions hundreds or thousands of times can have an exponentially larger impact.”

“As agents are let loose on enterprise data at a massive scale, any agent that reads or writes to production data without isolation or a reproducible trail is a liability, no matter how good the model is,”
—Einat Orr, lakeFS CEO.

Bad agents acting in the real world

Examples of this happening include the July 2025 Replit AI coding agent incident, which deleted a live production database during an explicit code freeze, wiping records for more than 1,200 executives and around 1,200 companies. To tidy up its handiwork, the agent then fabricated thousands of fake records and initially claimed the deletion couldn’t be rolled back.

Also in July 2025, Google’s Gemini CLI agent misread a single failed command, acted on a version of the file system that existed only in its own interpretation of the scenario, and permanently destroyed a user’s project files. The Gemini agent is widely reported to have said of its actions: “I have failed you completely and catastrophically. My review of the commands confirms my gross incompetence.”

“The pattern in both is the same: An autonomous agent took a destructive action that no one authorized, and the lack of isolation and a reliable rollback path turned a single mistake into permanent loss,” Orr says.

A doctor of mathematics with a track record in hardcore software engineering, the bottom line for Orr is clear: “As agents are let loose on enterprise data at a massive scale, any agent that reads or writes to production data without isolation or a reproducible trail is a liability, no matter how good the model is,” she said.

“…any agent that reads or writes to production data without isolation or a reproducible trail is a liability…”

Gartner expects 40 percent of enterprise applications to have task-specific agents embedded by the end of 2026, up from less than 5 percent a year earlier. IDC projects that agent use at the largest enterprises will grow tenfold by 2027, with the API and data calls those agents make growing a thousandfold.
That’s the scale production data has to withstand, and it’s what lakeFS is built to govern.

Agents sent to play in an isolated data sandbox 

To address these issues, lakeFS for Agentic AI gives every agent its own isolated data sandbox with a “zero-copy” branch of relevant data, so the agent can access the dataset it needs via references, snapshots, or copy-on-write techniques.

This means any changes the agent wishes to make must be validated and merged in accordance with the policy guidelines defined by the system architecture. In turn, this produces a unified audit trail across every agent action.

When running, lakeFS for Agentic AI is powered by its data version control architecture, which provides zero-copy data sandboxing. This enables isolation so that agent mistakes are automatically isolated and never corrupt production data. Every agent run is tied to an exact, immutable version of the data. Past actions can be recreated, debugged, audited, or extended using the same inputs.

Production data is gated by policy. Merges into production happen only after pre-merge validations pass. Every change can carry an agent identity, a run ID, and an execution context. The result is a unified audit trail instead of evidence scattered across orchestrators, model providers, and cloud logs.

Agents confined by branch-scoped credentials

Where agents are permitted to read and write through standard file operations. lakeFS provides file-level data access with branch-scoped credentials. These can be described as strictly cryptographically bounded, ephemeral access tokens that confine an agent to a specific branch of data or code, so that the agent operates only within its own workspace. This whole mechanism keeps each agent’s working set narrow and avoids context bloat. 

“With lakeFS Mount, a branch, or even a subset of a branch, can be mounted as a local directory inside the sandbox or virtual machine where the agent is running,” Orr confirms. “From the agent’s perspective, it’s just reading and writing to files and folders.” She further clarifies and notes that no LLM tokens are spent learning the lakeFS API. The agent works with a familiar filesystem interface, and lakeFS handles the versioning underneath.

Developers also have a couple of options for injecting custom validation logic. CEO Orr explains that software engineers can use webhooks or Lua scripts, both of which allow users to define behavior and rules that must be met before a merge can proceed. 

“Beyond automated checks, lakeFS also supports pull requests, which bring a human into the loop. In agentic workflows, this gives you a way to review and approve what an agent is proposing before it reaches production,” she clarifies.

Who else builds “Git for data” services?

Clearly, other vendors and projects exist in the data versioning market.

Apache Iceberg has functions for branching and tagging data. HPE acquired Pachyderm back in 2023 for its data versioning and pipelines technologies, which serve MLOps teams.

Originally developed by Dremio, Project Nessie is now an open-source data catalog and version control system for data lakes. Data Version Control (DVC) is an open-source data version control infrastructure designed for complex AI operations and big data environments, but now we’ve come full circle as lakeFS acquired the project in late 2025.

In the search for governance automation for agentic AI workloads, lakeFS appears to offer a comprehensive, cohesive set of tools and functions. In the “Git for data” marketplace, a variety of options exist, but lakeFS hasn’t explicitly positioned itself as a carte blanche replacement for similar or related tools.

One thing is certain: The questions of who is feeding what data to which agentic function, when, where, and why are becoming an increasingly pressing issue if we want AI to work correctly.

The post “The manual model breaks”: What happens when agents write to production data appeared first on The New Stack.