As more companies plug AI agents into the deepest depths of their internal data banks, how can they be sure those agents actually understand how the business works? Right now, many of these organizations are stuck manually building a Markdown file, hoping they find time to rewrite it each time the business changes.

Modus, for its part, thinks it has found a better way. The startup that formally exits stealth this week with $10 million in funding in tow is building what is coming to be known in industry parlance as a “context warehouse” — a layer that sits alongside a company’s existing data warehouse, continuously mapping how the business operates across its systems, and handing an AI agent only the relevant slice of that map when it needs it.

In real terms, Modus crawls relevant assets from sources like GitHub, dbt, Jira, Snowflake, and Postgres, using what it calls a Context Miner to continuously learn how the business operates. What it finds gets turned into “dynamically generated skills”: Short, purpose-built briefs, assembled in real time by a second system, the Context Composer, and handed to an agent the moment it’s given a task.

Modus co-founder and CTO Tomer Mesika tells The New Stack that this mining runs continuously, guided by its own internal logic for what to check and how often.

“We have a lot of mechanisms in place to know what to mine from the organization, at what cadence, how to look for deltas, when to dive deeper in, and when not to,” Mesika says.

“We have a lot of mechanisms in place to know what to mine from the organization, at what cadence, how to look for deltas, when to dive deeper in, and when not to.”

Daniel Shimoni, Modus co-founder and CEO, draws a direct line to data warehousing to highlight the gap he’s trying to close. Companies have spent years building infrastructure to store and organize their data, he argues, but nothing equivalent exists for the understanding that sits atop it.

“There’s a logic behind data warehouses — companies already know that is where they manage their data,” Shimoni tells The New Stack. “But where do they manage their context? Where do they actually understand what contexts exist in their organization, that they can actually use to ensure agents only have what they need?”

Shimoni says even that first step is hard enough on its own. But keeping a company’s context accurate as the business changes is harder still.

“We’ve noticed that building the context the first time is already a challenge, but maintaining it is the bigger issue,” Shimoni says. “So Modus always learns from what the company is doing, and whenever something shifts or changes in the business, it makes sure that only the relevant and updated context is fed to agents.”

“Building the context the first time is already a challenge, but maintaining it is the bigger issue.”

Who’s buying, and why cost matters

Shimoni says Modus is targeting engineering teams, the CTO office, and VPs of R&D, as well as data teams and a newer category of AI teams.

“AI teams weren’t really around last year; it seems that a lot of data teams are transitioning to becoming VP of data and AI, or AI enablement,” Shimoni says. “So really, it’s the people who are in charge of having this AI enablement mandate in the organization, making sure AI is scaled in the organization.”

Pitching enterprises a shiny new context warehouse becomes much easier when the promise is steeped in helping them cut costs. Spend has become one of the defining anxieties of enterprise AI this year, with companies switching providers in pursuit of cheaper models, to entire economic models being built around the price of a token.

“You want the bigger models to do the heavy and complex tasks to get great value. The problem is that they are wasting a lot of their effort and a lot of their token usage on menial tasks.”

Mesika says this is a central component of Modus’s modus operandi, arguing that frontier models end up spending a chunk of their token budget on work unrelated to actually answering a question.

“You want the bigger models to do the heavy and complex tasks to get great value,” Mesika says. “The problem is that they are wasting a lot of their effort and a lot of their token usage on menial tasks.”

Those menial tasks, in Mesika’s telling, include combing through pull requests or Jira tickets just to determine what’s relevant before an agent can start the job it was assigned to.

One approach to this problem is to hand the sorting work to a smaller, cheaper model. Mesika says Modus takes that further: rather than retrieving that context at the moment a question is asked, it uses small language models alongside search engines, vector search, and a graph database, all built up in advance, to do that work continuously in the background. By the time an expensive frontier model gets involved, it’s only ever handed a finished brief of exactly what it needs.

“Everyone’s talking about context”

Shimoni and Mesika both come from data-centric companies — Lusha, a go-to-market data platform, and Cyera, a cybersecurity data company, respectively — before leaving their roles in September 2025 to start Modus together.

The two had known each other for years, and spent much of the previous year comparing notes on a problem they were both running into in very different jobs.

“We decided this is a problem worth solving, and it seems like we were spot on, because everybody’s talking about context.”

“Some of the challenges were very similar — how do we combine a lot of various data assets into one place where AI can work?” Shimoni says. “We just started to notice that this is the gap — to make AI run with confidence, at scale, across a company. We decided this is a problem worth solving, and it seems like we were spot on, because everybody’s talking about context.”

Modus closed a hitherto unannounced $10 million seed round shortly after founding, led by Insight Partners. Other backers include Soma Capital and a handful of angel investors, among them founders from Cyera and Wix.com. The company began hiring its first employees in January 2026.

The broader takeaway from Modus’s pitch is now among the most common refrains emanating from AI circles this year: that the model itself is no longer the bottleneck; what limits an AI system now is everything built around it. And for Modus, that realization has been more or less present since its inception.

“Even last year […] we could already see that model capabilities weren’t the bottleneck,” Shimoni says. “It was more making sure that they actually have access to the context they need in order to give you the right answers.”

The post Modus’s operandi: To give AI agents just the right amount of context appeared first on The New Stack.

Apache Spark 4.2 launched last week, and it signals an expansion of Spark’s decade-plus role at the center of enterprise data processing

With new features for AI workloads, including governed metrics, vector retrieval primitives, real-time processing, improved Python support and native geospatial analytics, Spark 4.2 builds on a recent history of new AI and streaming features, reflecting how many engineering teams use the platform today. The release builds on Spark’s traditional role as a data processing engine by adding more of the capabilities needed to support production AI applications.

The launch introduced several features that enable developers to do more without leaving the platform, and for teams already using Spark, that could mean fewer systems to manage.

Governed metrics prevent conflicts

One team’s definition of a business metric isn’t always the same as another’s. Over time, those differences can lead to conflicting reports and uncertainty about which number to trust.

That becomes even more problematic when AI applications start consuming the same enterprise data that analysts and business intelligence tools use. If different teams define the same metric differently, AI systems can produce inconsistent results for the same question.

Spark 4.2 introduces governed metric views to address that issue. Organizations can define a business metric once and reuse that definition across applications. A metric view makes dimensions and measures first-class objects that Spark understands, so the engine can preserve the intended aggregation semantics regardless of who or what is querying it.

Organizations can define a business metric once and reuse that definition across applications.

Vector search goes native

One of the more significant additions is native vector search, which reduces the need to move data between Spark and a separate vector database.

Spark 4.2 introduces vector distance and similarity functions, vector normalization, vector aggregation, and NEAREST BY, a new SQL operator for top-K similarity searches. By bringing vector search into Spark, developers can keep more of their retrieval pipeline on the same platform.

By bringing vector search into Spark, developers can keep more of their retrieval pipeline in the same platform.

Python interoperability gets easier

Spark 4.2 makes it easier to move data between Spark and Arrow-native tools. With support for the Arrow C Data Interface and the PyCapsule protocol, Spark DataFrames can be passed directly to tools like Polars and DuckDB without copying or serializing the underlying data, as long as both sides support the standards.

Python also gets a few other updates. PySpark has been expanded; Arrow-optimized UDF execution is now the default, and Python Data Sources now include built-in time and memory profiling to help developers troubleshoot custom connectors.

Spark Connect makes the engine callable by agents

Spark Connect, which separates the client from the Spark server via a gRPC- and Arrow-based protocol, receives several updates in 4.2. The key idea is that a client builds a logical plan, the server handles analysis, optimization, and execution, and the results come back as Arrow batches. The client doesn’t need a full Spark runtime or a colocated JVM.

This update includes several changes to Spark Connect, the project’s client-server interface. AI applications can send processing requests to a remote Spark cluster while the work continues to run inside Spark. The release improves RDD API compatibility, error handling, and status reporting along that path.

Streaming powers real-time AI

Streaming gets several updates in Spark 4.2, including Auto CDC and Real-Time Mode. Many AI applications depend on continuously updated data rather than scheduled batch jobs. Auto CDC brings first-class change data capture to Spark Declarative Pipelines, handling the merge logic for keeping target tables current as source data changes — something that previously required hand-written, error-prone code. The new CHANGES SQL clause allows teams to retrieve data changes through a single SQL interface.

Spark 4.2 also adds built-in GEOMETRY and GEOGRAPHY types along with ST_* functions for location-aware analytics, without requiring external spatial extensions. For teams doing anything with location data — logistics, real estate, IoT — this removes another reason to move data out of Spark.

The bigger picture

Spark 4.2 brings more of the AI and data stack into the platform itself. Features that once depended on separate tools can now be handled directly in Spark.

For teams that currently use Spark for ETL and then hand data off to other systems for retrieval, governance, or real-time processing, this release starts to blur that line. As more AI applications run directly on operational data, Spark is becoming part of the serving layer rather than simply preparing data for it.

Spark is becoming part of the serving layer rather than simply preparing data for it.

The post Spark 4.2 has a feature that could retire your vector database appeared first on The New Stack.

The seemingly unquenchable thirst of the AI data ingestion pipeline spans language, numerical, and tabular data in the first instance, while other tangential platforms have been building large audio, image, and video models at the same time. 

Straddling potentially all of these domains are the file structures where complex documents and forms of unstructured data reside; this is the road less traveled in terms of the source DNA modern AI draws from.

The schema-less, freeform, uncurated data lake

In a bid to bridge connections to the schema-less, freeform, uncurated information that all organizations naturally harbor, enterprise visual intelligence company Valantor announced its acquisition of unstructured information RAG specialist EyeLevel on Tuesday. The acquisition formally launches Valantor’s Enterprise Visual Intelligence platform, combining EyeLevel’s document intelligence with its own operational expertise.

Benjamin Fletcher, CEO and co-founder of EyeLevel, tells The New Stack that where organizations fail to adopt visual intelligence, human-only processing breaks down pretty quickly in the age of AI.

“About 80% of corporate knowledge is in millions of pages of visually complex PDFs, PPTX, and DOCX files,” Fletcher says. “This information is far beyond the capacity of any LLM context window and is effectively inaccessible to LLMs and agents.” 

“We’ve found the golden datasets that teams build by hand routinely carry 10 to 25 percent error rates. Ironically, those same teams often hold AI to a far higher standard than their own people.”

Humans are slow, expensive & prone to errors

He explains that transactional workflows (such as invoice and claims processing) typically involve documents “so visually complex and diverse” that enterprises still rely on humans to process them, who can be slow, expensive, and error-prone. 

“We’ve found the golden datasets that teams build by hand routinely carry 10 to 25 percent error rates,” Fletcher says. “Ironically, those same teams often hold AI to a far higher standard than their own people. If data sovereignty matters to a business, everything gets harder now: solving these problems with AI while your documents stay inside your own infrastructure is the hard mode version of the job, and very few tools can do it.”

Where does invisible corporate information live?

Valantor has noted that while most AI companies concentrate on models, the company itself is “focused on the information those models can’t see” today. The suggestion is that this unseen morass of valuable data is locked inside documents, claims files, contracts, engineering drawings, reports, forms, presentations, and other visually complex content.

Valantor’s flagship platform product, GroundX, operates where data resides, including private cloud, sovereign infrastructure, on-premises deployments, and fully air-gapped environments. 

“GroundX is the ingestion and retrieval layer for unstructured documents,” explains Fletcher. “It is one tightly tuned system where retrieval consumes exactly what ingestion produces. Everything is exposed through REST APIs, SDKs, and MCP. It ships as REST APIs, SDKs, and MCP, and the Helm chart drops straight into a team’s existing deploy pipeline, and our agent harness gives coding agents like Claude and Codex the skills to build the integration themselves.

As part of the acquisition announcement, Valantor is introducing GroundX Studio. The harness capabilities within GroundX Studio integrate with modern AI development environments, enabling developers to build secure AI applications that operate on enterprise knowledge while remaining within existing infrastructure. 

GroundX Studio also extends capabilities to business users, allowing organizations to create AI-powered workflows and applications without extensive custom development.

“Each agent does one small task, so cheaper models are often good enough, and teams that want direct control over cost can run the whole stack on their own hardware with Helm.”

Risk of latency-laden performance and spiraling costs?

If it feels like this new data ingestion stream is going to place a new burden on cloud workloads, application execution latency, database retrieval times, and (of course) overall token usage, then Valantor and EyeLevel say that this consideration has been taken into account by dint of their own platform’s orchestration layers.

“We never send a whole schematic to a language model; our vision model splits each page into its elements first,” Fletcher confirms. “Processing runs in multiple passes at different levels of the document, and everything inside a pass runs in parallel, so there’s a minimum processing time, but it does not scale linearly with page count. Each agent does one small task, so cheaper models are often good enough, and teams that want direct control over cost can run the whole stack on their own hardware with Helm.”

The intersection of AI and handwriting

sWhile we already know that AI and handwriting do mix in the same cocktail glass — the ViWoods AiPaper digital e-ink handwriting tablets have a useful set of AI functions on board, and similar products are available from manufacturers including reMarkable — it’s not a widely deployed use case yet. Valantor claims that its underlying data models and custom heuristics bridge the “data comprehension gap” when processing handwritten annotations.

“Our proprietary vision model, fine-tuned on more than a million pages of enterprise documents, sees the page the way a human does: tables, paragraphs, and figures,” underlines Fletcher. 

He says that handwritten marks are captured as page elements with their layout context intact. Narrow agents then distill each element into a contextual object tuned for both search and LLM completion. 

“Smaller pieces, less cognitive load — that’s how we close the gap, with better accuracy at lower cost, driving better performance and significant cost advantages,” he adds.

Working examples of this technology include Air France-KLM, which used GroundX to develop an AI-powered customer service assistant trained on thousands of policy documents, achieving 96+% accuracy on complex policy-related questions. AskVet used the platform to operationalize more than a decade of proprietary veterinary data, enabling autonomous resolution of up to 85% of customer inquiries while significantly improving operational efficiency.

Is document management sexy now?

Taking all of this on board, are we at the point where we can ask whether document management has just become interesting, compelling, and sexy? 

No, of course it didn’t; it will arguably always suffer from a degree of stigmatized disdain. That may change in the future as we interact more directly with AI tools that begin analyzing the unstructured information we know organizations have been sitting on for so long. For now, it may still remain the corporate equivalent of eating your vegetables — pass the Brussels sprouts and steamed turnips, please.

The post AI can finally read your handwriting — here’s why enterprises care appeared first on The New Stack.

Modern businesses are in a constant, uphill battle against what to do with unstructured data: PDFs, contracts, scanned images, customer call recordings, meeting videos, and more. Traditional document automation workflows that rely heavily on template-based extraction or rigid rules used to make sense. But document formats have changed; they’re diverse and don’t fit standard formats, making costly, brittle, traditional systems a relic of the past. 

“Modern businesses are in a constant, uphill battle against what to do with unstructured data.”

Enterprises demand faster, more accurate processing, which raises the question: How can we reliably turn messy, multimodal content into structured, actionable insights without a mountain of manual effort?

That’s where Amazon Bedrock Data Automation (BDA) comes in.

What is Amazon Bedrock Data Automation (BDA)?

Amazon Bedrock Data Automation (BDA) is a generative AI-powered, fully managed service on Amazon Web Services for end-to-end document and media automation. It enables users to automate the extraction, classification, and transformation of unstructured content across modalities such as documents, images, audio, and video.

“At its core are Foundation models which enable intelligent extraction and understanding of content.”

At its core are Foundation models (FMs) which enable intelligent extraction and understanding of content. It allows users to configure standard output for common use cases, or even define custom extraction logic using blueprints tailored to your business. BDA is designed for scalability, accuracy, and auditability, making it ideal for enterprise workflows.

Walk-through: creating a project, standard output & custom output using blueprints

1. Create a project via console

In the Amazon Bedrock Console, navigate to Data Automation → Create Project.

Enter the name of the project:

2. Standard output:

Standard output gives you the model’s default, unstructured response (text, image, audio, or video) directly from the Data Automation pipeline.

In standard output, each modality has its own options for what is needed as an output. 

Document:

Image & Video:

Audio:

Now let’s test Document Modality for Standard Output:

First, click on “Test” in the upper right corner.

Next, select the document from the system, sample, or S3 and choose the modality from the dropdown menu. 

Click on the “Generate results” button:

After processing, it will show the summary and content of the document:

Custom output (blueprints):

Custom output lets you define a structured, predictable format using blueprints, which ensures the output matches your exact schema, fields, and business rules.

Let’s test custom output using blueprints for the same document:

Navigate to “Custom output” and click on “Add Blueprint”:

From here, two options will appear. You can either use LLM power to generate the blueprint (where it inspects the document), or you can choose to enter field names, instructions, and other information manually.

Below is a blueprint generated by LLM which has pulled all possible fields and tables from the document:

It has extracted the information using the blueprint as demonstrated below, including the  Field name, Instruction, and Results:

It also provides the Extraction type (which can be Explicit or Inferred), Confidence percentage, and other relevant information.

Additionally, it can extract information in the form of a table, such as an account summary or transaction information:

Code examples

Amazon Bedrock Data Automation (BDA) Utility Module
Description:
    Helper functions to create BDA projects, blueprints, invoke jobs,
    monitor job status, and fetch results.

import boto3
import time
import json
import botocore

class BedrockDataAutomation:
    def __init__(self, region="us-east-1"):
        self.bda = boto3.client("bedrock-data-automation", region_name=region)
        self.runtime = boto3.client("bedrock-data-automation-runtime", region_name=region)

    # ------------------------------------------------------------
    # BLUEPRINT OPERATIONS
    # ------------------------------------------------------------
    def create_blueprint(self, name, schema, description="", stage="LIVE"):
        """
        Create a BDA Custom Output Blueprint from a JSON schema.
        """
        print(f"Creating blueprint: {name}")

        response = self.bda.create_blueprint(
            blueprintName=name,
            blueprintStage=stage,
            type="DOCUMENT",
            schema=json.dumps(schema)
        )
        return response["blueprint"]["blueprintArn"]

    # ------------------------------------------------------------
    # PROJECT OPERATIONS
    # ------------------------------------------------------------
    def create_project(self, name, description, standard_output_config, custom_output_config=None):
        """
        Create a BDA Project with Standard or Custom Output.
        """
        print(f"Creating project: {name}")

        response = self.bda.create_data_automation_project(
            projectName=name,
            projectDescription=description,
            projectStage="LIVE",
            standardOutputConfiguration=standard_output_config,
            customOutputConfiguration=custom_output_config or {}
        )
        return response["projectArn"]

    # ------------------------------------------------------------
    # INVOCATION OPERATIONS
    # ------------------------------------------------------------
    def invoke_project(self, project_arn, profile_arn, input_s3_uri, output_s3_uri, blueprints=None):
        """
        Invoke a BDA project using async invocation.
        """
        print(f"Invoking project: {project_arn}")

        kwargs = {
"inputConfiguration": {"s3Uri": input_s3_uri},
"outputConfiguration": {"s3Uri": output_s3_uri},
"dataAutomationConfiguration": {
"dataAutomationProjectArn": project_arn,
"stage": "DEVELOPMENT"
},
"dataAutomationProfileArn": profile_arn
}

        if blueprints:
kwargs["blueprints"] = blueprints

        response = self.runtime.invoke_data_automation_async(**kwargs)
       invocation_arn = response["invocationArn"]

       print("Invocation ARN:", invocation_arn)
       return invocation_arn

    # ------------------------------------------------------------
    # JOB STATUS POLLING
    # ------------------------------------------------------------
    def wait_for_job(self, invocation_arn, poll_interval=10):
        """
        Poll until job finishes.
        Returns final status object.
        """
        print("Polling job:", invocation_arn)

        while True:
            try:
                resp = self.runtime.get_data_automation_status(
                    invocationArn=invocation_arn
                )
            except Exception as e:
                print("Error fetching status:", e)
                raise

            status = resp["status"]
            print(f"Status: {status}")

            if status in ("SUCCEEDED", "FAILED", "CANCELLED"):
                return resp

            time.sleep(poll_interval)



# --------------------------------------------------------------------
# EXAMPLE USAGE
# --------------------------------------------------------------------
if __name__ == "__main__":
    bda = BedrockDataAutomation(region="us-east-1")

    # 1. Create Blueprint
    blueprint_schema = {
        "type": "object",
        "properties": {
            "account_holder": {"type": "string"},
            "balance": {"type": "string"},
            "transactions": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "date": {"type": "string"},
                        "description": {"type": "string"},
                        "amount": {"type": "string"}
                    }
                }
            }
        },
        "required": ["account_holder", "transactions"]
    }

    blueprint_arn = bda.create_blueprint(
        name="BankStatementBlueprint",
        schema=blueprint_schema,
        description="Extract fields from bank statements."
    )

    # 2. Create Standard Output Config
    standard_config = {
        "document": {
            "extraction": {
                "granularity": {"types": ["PAGE", "LINE"]},
                "boundingBox": {"state": "ENABLED"}
            },
            "outputFormat": {
                "textFormat": {"types": ["PLAIN_TEXT", "CSV"]}
            }
        }
    }

    # 3. Create Project with Custom Blueprint
    project_arn = bda.create_project(
        name="BankStatementProject",
        description="Process PDF bank statements",
        standard_output_config=standard_config,
        custom_output_config={
            "blueprints": [
                {
                    "blueprintArn": blueprint_arn,
                    "blueprintStage": "DEVELOPMENT",
                    "blueprintVersion": "1"
                }
            ]
        }
    )

    # 4. Invoke the project
    # Ensure you replace <ACCOUNT_ID> with your actual AWS Account ID
    profile_arn = "arn:aws:bedrock:us-east-1:<ACCOUNT_ID>:data-automation-profile/us.data-automation-v1"
    invocation_arn = bda.invoke_project(
        project_arn=project_arn,
         profile_arn=profile_arn,
        input_s3_uri="s3://your-bucket/input/statement.pdf",
        output_s3_uri="s3://your-bucket/output/",
        blueprints=[
            {
                "blueprintArn": blueprint_arn,
                "version": "1",
                "stage": "DEVELOPMENT"
            }
        ]
    )

    # 5. Poll job status
    final_status = bda.wait_for_job(invocation_arn)
    print("Final status:", json.dumps(final_status, indent=4))

Types of document blueprints

When processing documents, BDA supports five core automation types:

1. Classification: invoice, bank statement, ID card, contract, HR letter, etc.

2. Extraction: Extract entities, fields, tables, metadata.

• Example: From a bank statement → Date, Description, Amount, Balance.

3. Transformation: Modify or restructure data.

• Example: Convert Home Address into separate fields -> street, city, ZIP code, etc.

4. Normalization: Standardize data values.

• Example: Convert multiple date formats (MM/DD/YYYY → YYYY-MM-DD).

5. Validation: Validate extracted fields against rules.

• Example: Amount must be numeric; dates must match the format; balances must reconcile.

Use cases that illustrate business value

Real-world scenarios where BDA provides significant ROI include:

• Financial Services: Automate processing of bank statements, invoices, and loan applications, reducing manual labor and speeding up reconciliation or underwriting.
  
• Insurance: Ingest and extract data from claims forms, medical reports, and damaged-asset photos.
  
• HR / Legal: Process resumes, contracts, and offer letters; extract structured data, including skills, clauses, salaries, and parties.
  
• Customer Support: Transcribe and summarize calls, extract intent and sentiment, and feed those insights into CRM or case systems.
  
• Security & Compliance: Analyze CCTV footage or meeting recordings to detect key actions, summarize context, and flag compliance issues.

BDA proves itself flexible and powerful, as it supports both standard outputs for basic workflows and fine-tuned custom schemas via blueprints. It is scalable and robust, with projects that enable batch processing and versions (development vs. live) for safe testing. It’s also audit-friendly, providing structured fields with types, normalization rules, and validation logic. 

“Compared with rule-based systems, foundation models achieve better semantic extraction across the board.”

A true key benefit is that BDA is multimodal across formats. Users can use the BDA framework to process documents, images, audio, and video. And, best of all, it’s highly accurate. Compared with rule-based systems, foundation models achieve better semantic extraction across the board. 

Amazon Bedrock Data Automation empowers businesses to transform unstructured, multimodal content into structured, trustworthy, and actionable data. With minimal setup, highly customizable blueprints, and a scalable project-based architecture, BDA helps organizations reduce manual workload and unlock insights faster.

The post Template-based data extraction is dead. Here’s what comes next. appeared first on The New Stack.

For most of its 30-year history, Postgres has been viewed as a transactional database. Organizations trust it with customer records, financial transactions, and countless other operational workloads. Its reputation was built on reliability, strong transactional guarantees, and a vibrant open-source community that has spent decades refining the database without compromising its foundations.

However, some of the most important innovations in the Postgres ecosystem today have little to do with storing data. They have to do with reducing the need to move it around.

“Some of the most important innovations in the Postgres ecosystem today have little to do with storing data. They have to do with reducing the need to move it around.”

Database innovation has historically focused on performance, scalability, and reliability. Increasingly, the harder problem is interoperability: how operational data can be shared across analytical systems, AI applications, and downstream services without creating yet another pipeline or copy.

Why Postgres keeps showing up

The reality of modern software architecture is that data rarely stays in one place. Information created in operational systems quickly finds its way into warehouses, search platforms, machine learning environments, and AI applications. Every new system solves a legitimate business problem, but it also creates another destination for data and often another copy to maintain.

The costs of this approach extend beyond infrastructure spending alone. Every additional copy introduces latency, creates another potential source of inconsistency, and increases the operational burden of keeping systems synchronized. Many organizations now spend as much effort moving data as they do storing it.

“Many organizations now spend as much effort moving data as they do storing it.”

For many businesses, Postgres serves as the system of record for customer interactions, transactions, application state, and other business-critical information. As organizations expand their analytical, machine learning, and AI capabilities, they are not looking to create another source of truth; rather, they’re looking for better ways to work with the one they already trust.

That shift is changing how Postgres fits into modern architecture. Historically, Postgres was viewed primarily as the place where operational data originated before being copied into downstream systems. Increasingly, organizations want those systems to work more seamlessly with operational data while reducing the pipelines, copies, and synchronization processes required to support them.

Technologies such as logical replication, change data capture, and foreign data wrappers have helped Postgres participate more directly in larger data ecosystems. As a result, organizations are no longer asking only whether Postgres can store their data. They’re instead asking how easily it can connect to everything around it.

That shift, from evaluating databases primarily on storage and performance to evaluating them on interoperability, may be one of the most important changes happening in the Postgres ecosystem today.

AI is exposing old problems

The recent focus on AI has brought renewed attention to data movement. AI didn’t create the problem. If anything, it exposed a limitation that has been quietly growing for years. For decades, organizations built architectures around the idea that data would move between systems through pipelines and periodic synchronization. That model worked because most analytical workloads could tolerate some degree of delay.

AI is changing those expectations. Many AI applications depend on access to current operational context. The challenge is not that organizations lack data. In many cases, they already have it. The challenge is that the data is spread across multiple systems, each with its own copy, latency profile, and synchronization process.

“AI is forcing organizations to confront a broader question: How many copies of the same data are actually necessary? The answer increasingly appears to be fewer than most architectures maintain today.”

As a result, AI is forcing organizations to confront a broader question: How many copies of the same data are actually necessary? The answer increasingly appears to be fewer than most architectures maintain today. As expectations around freshness rise, reducing unnecessary data movement becomes just as important as accelerating it. The underlying challenge is not new. AI has simply made it harder to ignore.

What’s next

The database industry spent decades solving storage. Databases became more reliable, storage became cheaper, and infrastructure became dramatically easier to operate. The next challenge is not where data lives, but how easily it can be shared across systems without introducing additional pipelines, copies, and synchronization overhead. Increasingly, the goal is not simply moving data faster. It is reducing unnecessary movement altogether.

Postgres has a habit of outlasting predictions about its replacement. For years, members of the community have joked that every year is “the year of Postgres.” The joke works because it keeps turning out to be true. 

Three decades after its creation, Postgres continues to adapt to new workloads, new architectural patterns, and new ways of building applications.

That longevity is not an accident. Enterprises continue to rely on Postgres because it provides a stable and trusted foundation for operational data.  While that foundation is unlikely to change, the scope of what organizations expect Postgres to do will continue to expand.  

As new workloads continue to emerge, much of the innovation will come through extensions that expand Postgres’s capabilities without sacrificing the stability that made it successful. In that sense, the future of Postgres may not be about reinventing the database itself, but continuously expanding what can be built on top of it.

The post The database storage problem is solved. Here’s what comes next. appeared first on The New Stack.

AI consumes a lot of data, but all-you-can-eat data in the world of agentic intelligence eventually loses its flavor. Simply increasing the sheer volume of databases, data repositories and data volumes does not necessarily enrich any given AI function’s ability to reason. 

Context, on the other hand, does.

We know that agents are only as intelligent as the context they have access to. In order to codify the inclusion of context into algorithmic logic for AI, technology vendors have extolled the virtues of using a knowledge graph to make their data more useful for agentic purposes.

AWS knows this story all too well. The organization’s sprawling datacenter footprint hosts vast pools of context on behalf of its customers, but that context often sits in a raw and unstructured form across data lakes, data warehouses, data lakehouses, databases, and data streams. It also permeates through the rarely-documented institutional knowledge that agentic engines rarely get access to.

All of which explains why the cloud giant used its AWS New York Summit today to introduce AWS Context, a new service that automatically maps the relationships that exist across a firm’s existing data into a knowledge graph and provides agentic search so AI agents in the organization can access what are governed data relationships, business rules, and domain knowledge at runtime. 

But weaving all this together is hard work. Knowledge graphs need more than simple keyword matching to work; they require structural and semantic traversal. This means they need to make multiple hops across various information silos and repositories so that they can aggregate context and (for example) be able to explain why cybersecurity vulnerability A is a factor of system compromise B, which has a core dependency link to codebase C, which executes in application D and risks taking users X, Y and Z offline. So how is AWS doing this?

A data lake of nuance & information

Mai-Lan Tomsen Bukovec, AWS vice president of technology (data and analytics) tells The New Stack that AWS Context provides a “data lake of nuance and information that AI agents swim in” to reason correctly and make the right decisions for the business. 

“This is no different from how humans work. When we take action, we depend on our own context about the domain, prior decisions and their outcomes, and other information.” – Mai-Lan Tomsen Bukovec, AWS.

“This is no different from how humans work,” Tomsen Bukovec says. “When we take action, we depend on our own context about the domain, prior decisions and their outcomes, and other information. With AWS Context, AI agents have all the nuance of every form of data in their business in a knowledge graph and in open data formats. AWS Context will make the difference between an AI agent simply taking an action versus making the right decision.”

Given the option to embrace this new service, software engineers will need to set out a plan of action and work out what to do first. For AI developers and data science professionals, this throws up the question of what to prioritize first when preparing existing enterprise data for context-aware agents using AWS Context capabilities and how they can control what data is (and isn’t fed) into the mouth of the beast.

Mercifully, it appears, options for control appear to exist.

“If developers want to exclude information from AWS Context, they will have the ability to prevent certain datasets, like test data or sandbox environments, from being included with AWS Context,” explains Tomsen Bukovec. “Because AWS Context is continuously updated as relationships between data resources changes, AI agents have the latest context available without any intervention from AI developers – and the control to set guardrails to exclude content that agents should not take action upon.”

Should developers trust this technology?

AWS Context extends the same knowledge graph technology that runs Amazon Quick, the organization’s AI work assistant that “connects scattered work” across applications and resources, including Slack, Microsoft Teams and Outlook, CRMs, databases, and documents.

So, should software developers place their trust here? After all, even once captured and connected, not all business context is useful. Some contextualizations could be corrupted, weak, fragmented and not productively useful for the business? Is AWS at risk of encapsulating context without considering how the data that comprises it is is quantified in terms of business usefulness?  

AWS has thought of this factor.

Because AWS Context uses the same knowledge graph technology that powers Amazon Quick, it can learn from usage patterns to make every interaction smarter. With AWS Context, the company says it is extending what was a personal knowledge graph into an organizational one i.e. a shared, governed context layer that agents and applications in an organization can draw from.

“Developers can govern and shape a dynamic and intelligent context layer that AI agents depend on to make the right decisions – AI agents won’t just get smarter as the models improve – they will be smarter because they have a vast amount of curated context at their fingertips.” – Tomsen Bukovec.

“AWS Context provides a data lake of context in graph and open data format,” clarifies Tomsen Bukovec. “That means that AI developers everywhere can use capabilities at the data layer to govern and shape a dynamic and intelligent context layer that AI agents depend on to make the right decisions. With this change, AI agents won’t just get smarter as the models improve – they will be smarter because they have a vast amount of curated context at their fingertips.”

Curated knowledge beyond a user’s personal graph

Existing Amazon Quick users will see that when AWS Context is enabled, Quick’s agents gain access to the broader enterprise knowledge graph, including cross-system relationships, business rules, and curated context that go beyond what any single user’s personal graph can provide. 

Tomsen Bukovec has also said that AWS Context gets smarter the more agents use it. As agents query the graph, it observes which sources produce correct results, which join paths agents rely on, and which curated rules get applied. It ranks sources by actual usage and shares what it learns across an organization, so when one agent discovers a correct join path or resolves a schema ambiguity, other agents pick it up, without requiring a human to re-curate the graph.

Any agent you put into production raises a governance question: what data can it reach, and can you show exactly what it accessed and under whose authority? The organization has explained that AWS Context answers both by making every query identity-aware.

Each call is designed to inherit the calling user’s identity access management (IAM) and Lake Formation permissions, so an agent can only see and traverse the relationships its identity is authorized to access. Because access runs through identity, every interaction is auditable. Security and compliance teams can verify what an agent accessed and under what authority, using the same controls.

AWS Glue Data Catalog

Related news to the arrival of AWS Context today saw the company also announce the preview of business context and semantic search functions for AWS Glue Data Catalog, the company’s centralized metadata repository for all data assets across various data sources. The new functions are designed to make it easier for humans and AI agents to discover and understand data. 

Also in this product stream, AWS now offers offer a preview of skill assets in Glue Data Catalog, a service designed to allow “data producers” (a somewhat arbitrary term that AWS applies to anyone who creates data, but is most likely a DBA or developer) to create skill assets. 

Associating skill assets to data assets gives agents additional context and instructions they can retrieve progressively for working with specific data without re-teaching it to every agent one prompt at a time. 

A renaissance of context engineering

Will this new drive from AWS herald the birth (or perhaps renaissance, the industry has been talking about this approach for some time) of context engineering as a sub-discipline of data science? It may well do… and if it does, it will likely drag role-based multi-agent orchestration along into the fray with it as we weave ever more complex interrelationship structures through enterprise data stacks.

If AWS or indeed the other hyperscalers or major frontier model companies starts acquiring more multi-model graph structure companies and vector database specialists, that could be the sign that things are cementing around context engineering at large. 

In the meantime, developers setting sail on the contextualized data lake of nuance are advised to wear a life jacket.

The post “A data lake of nuance for AI agents to swim in”: AWS Context gets shipshape on reasoning  appeared first on The New Stack.

Beneath the chatbots and copilots, there’s a quiet revolution happening in the data services space. From pure-play database vendors to data integration wranglers and onward to the cloud hyperscalers, the focus has shifted.

Now in the spotlight is the question of how to automate data governance for agentic AI workloads, and for good reason: Traditional manual data stewardship doesn’t scale in a world where agents are becoming increasingly autonomous (and powerful).

Aiming to cut a swath in this marketplace is data control plane company lakeFS. The organization announced its lakeFS for Agentic AI service on Wednesday, and it appears to be designed to bring governed, reproducible data access to autonomous and headless agentic workloads (those that execute decisions below the user interface level) that run at enterprise scale.

The manual model breaks

Einat Orr, CEO and co-founder of lakeFS, tells The New Stack that manual data stewardship was built for human-paced, human-reviewed workflows, i.e., someone looking at a change before it is committed.

“When dozens or hundreds of agents are making changes simultaneously, faster than any person can review, the manual model breaks,” Orr says. “This is because with a human analyst, a bad write to production is usually one mistake, caught by another human before it spreads far. An agent is different — it acts automatically, in parallel, at machine speed, and it doesn’t pause to second-guess itself. And because so much agent activity is unsupervised, you often find out after the damage is done.”

She explains that attempts to identify and roll back incorrect or corrupted production data across a wide set of data modalities, such as images, documents, metadata, and structured data, are almost impossible to pull off. Impossible, that is, unless the team has the data infrastructure in place to isolate and track such changes automatically.

While some of the more disastrous outcomes stay inside an organizaton’s perimeter (or are swept beneath the communications radar), Orr explains that real world consequences of bad agentic data writes are manifold.

“Insurance claims get inappropriately denied or approved, sensor data from machines gets misinterpreted, an incorrect medical diagnosis is made, or customer service bots provide incorrect answers to customers,” Orr says. “The cost of an individual action may be manageable, but agents performing these actions hundreds or thousands of times can have an exponentially larger impact.”

“As agents are let loose on enterprise data at a massive scale, any agent that reads or writes to production data without isolation or a reproducible trail is a liability, no matter how good the model is,”
—Einat Orr, lakeFS CEO.

Bad agents acting in the real world

Examples of this happening include the July 2025 Replit AI coding agent incident, which deleted a live production database during an explicit code freeze, wiping records for more than 1,200 executives and around 1,200 companies. To tidy up its handiwork, the agent then fabricated thousands of fake records and initially claimed the deletion couldn’t be rolled back.

Also in July 2025, Google’s Gemini CLI agent misread a single failed command, acted on a version of the file system that existed only in its own interpretation of the scenario, and permanently destroyed a user’s project files. The Gemini agent is widely reported to have said of its actions: “I have failed you completely and catastrophically. My review of the commands confirms my gross incompetence.”

“The pattern in both is the same: An autonomous agent took a destructive action that no one authorized, and the lack of isolation and a reliable rollback path turned a single mistake into permanent loss,” Orr says.

A doctor of mathematics with a track record in hardcore software engineering, the bottom line for Orr is clear: “As agents are let loose on enterprise data at a massive scale, any agent that reads or writes to production data without isolation or a reproducible trail is a liability, no matter how good the model is,” she said.

“…any agent that reads or writes to production data without isolation or a reproducible trail is a liability…”

Gartner expects 40 percent of enterprise applications to have task-specific agents embedded by the end of 2026, up from less than 5 percent a year earlier. IDC projects that agent use at the largest enterprises will grow tenfold by 2027, with the API and data calls those agents make growing a thousandfold.
That’s the scale production data has to withstand, and it’s what lakeFS is built to govern.

Agents sent to play in an isolated data sandbox 

To address these issues, lakeFS for Agentic AI gives every agent its own isolated data sandbox with a “zero-copy” branch of relevant data, so the agent can access the dataset it needs via references, snapshots, or copy-on-write techniques.

This means any changes the agent wishes to make must be validated and merged in accordance with the policy guidelines defined by the system architecture. In turn, this produces a unified audit trail across every agent action.

When running, lakeFS for Agentic AI is powered by its data version control architecture, which provides zero-copy data sandboxing. This enables isolation so that agent mistakes are automatically isolated and never corrupt production data. Every agent run is tied to an exact, immutable version of the data. Past actions can be recreated, debugged, audited, or extended using the same inputs.

Production data is gated by policy. Merges into production happen only after pre-merge validations pass. Every change can carry an agent identity, a run ID, and an execution context. The result is a unified audit trail instead of evidence scattered across orchestrators, model providers, and cloud logs.

Agents confined by branch-scoped credentials

Where agents are permitted to read and write through standard file operations. lakeFS provides file-level data access with branch-scoped credentials. These can be described as strictly cryptographically bounded, ephemeral access tokens that confine an agent to a specific branch of data or code, so that the agent operates only within its own workspace. This whole mechanism keeps each agent’s working set narrow and avoids context bloat. 

“With lakeFS Mount, a branch, or even a subset of a branch, can be mounted as a local directory inside the sandbox or virtual machine where the agent is running,” Orr confirms. “From the agent’s perspective, it’s just reading and writing to files and folders.” She further clarifies and notes that no LLM tokens are spent learning the lakeFS API. The agent works with a familiar filesystem interface, and lakeFS handles the versioning underneath.

Developers also have a couple of options for injecting custom validation logic. CEO Orr explains that software engineers can use webhooks or Lua scripts, both of which allow users to define behavior and rules that must be met before a merge can proceed. 

“Beyond automated checks, lakeFS also supports pull requests, which bring a human into the loop. In agentic workflows, this gives you a way to review and approve what an agent is proposing before it reaches production,” she clarifies.

Who else builds “Git for data” services?

Clearly, other vendors and projects exist in the data versioning market.

Apache Iceberg has functions for branching and tagging data. HPE acquired Pachyderm back in 2023 for its data versioning and pipelines technologies, which serve MLOps teams.

Originally developed by Dremio, Project Nessie is now an open-source data catalog and version control system for data lakes. Data Version Control (DVC) is an open-source data version control infrastructure designed for complex AI operations and big data environments, but now we’ve come full circle as lakeFS acquired the project in late 2025.

In the search for governance automation for agentic AI workloads, lakeFS appears to offer a comprehensive, cohesive set of tools and functions. In the “Git for data” marketplace, a variety of options exist, but lakeFS hasn’t explicitly positioned itself as a carte blanche replacement for similar or related tools.

One thing is certain: The questions of who is feeding what data to which agentic function, when, where, and why are becoming an increasingly pressing issue if we want AI to work correctly.

The post “The manual model breaks”: What happens when agents write to production data appeared first on The New Stack.

“Keep readers reading” is the not-so-simple goal of Medium’s recommendations system. To predict what’s most likely to appeal to a particular reader at any given time, Medium continuously processes user activity signals (stories read, recommendations shown, follows, likes, etc.). It then immediately correlates that with the steady stream of new articles, which is estimated at millions per month.   

Smart models and good inference logic are required, but that’s not enough. The data must be stored and retrieved quickly enough to remain relevant while the user is browsing. That’s the job of Medium’s feature store. And getting the data model right started to matter a lot as they scaled to 1M operations per second.

Andréas Saudemont, Medium Principal Software Engineer, recently walked through how the team identified the problem and what they built to fix it. If you’d rather watch than read, you have two options: Watch a short version from Monster Scale Summit or an extended follow-up webinar

The feature store and its role in Medium’s recommendation system

The feature store ties it all together, ingesting user activity and internal events and feeding them to the ML models that power recommendations. It’s what enables customization like the “For You” feed that greets logged-in users.

Each feature is a property of an entity, usually a user or a story. Some are simple and static, like whether a user holds a paid membership. Others capture interaction history: which stories a user has read, what content they’ve recently been shown, etc.

The following diagram shows a highly simplified view of the Medium feature store architecture:

The problem with a relational features data model

When they built their feature store years ago, Medium used relational features for cross-entity relationships. Unlike regular features, a relational feature can have multiple values for a given entity ID. Each value is defined by a relation ID (the ID of the related entity) and a timestamp recording when the event occurred.

For example, a “story users have read” feature is attached to the story entity type. It relates to the user entity type, and its values indicate whether/when a given user has read that story. 

Andréas shared the following schema diagram to explain the concept:

Features sit at the center, each attached to an entity type and defined by name, version, and data type. Non-relational features are simply a feature, an entity ID, and a value. Relational features add a relation ID mapping to another entity type, plus the value itself and a timestamp.

This approach proved suboptimal from a data modeling perspective. Since relational features link two entity types, the data ends up split between two tables: one for the entity IDs and one for the values. That means you can’t get both in a single query. The first query retrieves only entity IDs (not their associated values) and relies on ALLOW FILTERING. A second query then runs for each entity ID to fetch its value. “If we have 1000 entity IDs for which we want to fetch values, then we have to run 1000 queries to fetch these values,” Andréas said.

Overrelying on ALLOW FILTERING made things worse. “This is bad,” Andréas said, referring to monitoring data showing that 90% of rows read via these queries were simply discarded. “This is just data that we don’t need. ALLOW_FILTERING should be an escape hatch, not our design pattern.”

“ALLOW_FILTERING should be an escape hatch, not our design pattern.”

The list feature model

So they reinvented their data model and shifted to a list-based feature model. Instead of splitting data across two tables, everything for a given entity lives in one place and is retrieved in a single query.

Like other features, a list feature is defined by its entity type, name, and optional version. What’s different is the value. While a non-relational feature has a single value, such as true or false, a list feature’s value is a collection of items, each containing a value and a timestamp. Item values can be of any data type; the feature store doesn’t enforce consistency within a list.

For example, consider a user’s reading history. The entity is user, the feature name is reading history, the TTL is 6 months. After that TTL is reached, the data is automatically dropped by the database (since older history isn’t useful for recommendations). The list for a given user is a collection of story IDs and the timestamps at which they were read. The same story can appear multiple times, and multiple items can share the same timestamp.

A range of operations need to be supported. Create List and Delete List operations run at most a few times per day. Remove List Items with Value, which lets a reader scrub a specific story from their history so it stops influencing recommendations, runs at 1k-10k per second. Add List Items is higher still: every story read and every thumbnail shown to a user generates an event. Get List Items is the top, at 100k-1M operations per second.

“The Add List Items, and even more the Get List Items operations, are really the reasons why we need an efficient data store.”

“The Add List Items, and even more the Get List Items operations, are really the reasons why we need an efficient data store,” Andréas said.

Multiple items, one timestamp

Beyond raw efficiency, the new data model also had to support multiple items with the same timestamp. When Medium shows a user four story thumbnails simultaneously, all four presentation events share the same timestamp, but have distinct story IDs. If this isn’t handled correctly, primary key collisions occur.

The team’s solution was a single list_items table that stores everything.

The partition key combines feature_key and entity_id, keeping all items for a given list together. All of user 123’s reading history is stored in one partition, retrieved in one query. The clustering key concatenates each item’s timestamp with an MD5 hash of its value. The hash is what makes same-timestamp items with distinct values possible. 

Relying on MD5 hashes for uniqueness raises its own set of questions, but in practice, the team hasn’t seen collisions. “The values that we are storing are sufficiently distinct, especially when you add the timestamp into the equation,” Andréas said. The table’s clustering order is set to descending so ScyllaDB can optimize for the typical read pattern (most recent N items) rather than leaving the application to sort afterward.

TTL to control storage costs

Storage cost is controlled entirely through ScyllaDB’s native TTL, with no cleanup logic required. Every row expires automatically based on its own timestamp plus the feature’s TTL duration. “We don’t have anything to do regarding that,” Andréas said. “Any row for which the TTL is expired will be considered deleted by ScyllaDB.” 

Storage plateaus for a steady write rate. When a feature is retired, its data drains away on its own. “That’s super useful for controlling our storage and usage costs.”

Implementing the list operations

Add List Items is a logged batch of INSERTs with atomicity guaranteed: all items land or none do. Each row carries its own TTL calculated from its timestamp, so older items expire sooner. Since items almost always carry a current timestamp, new entries append to the top of the partition, which is exactly where reads will look first.

Get List Items runs as a single-partition SELECT with a minimum timestamp and a row limit. “We run the query on a single partition,” Andréas said. “That’s the maximum efficiency that we can have.” The clustering key handles filtering and ordering directly. Post-processing is not required.

Remove List Items with Value is the one operation that couldn’t be reduced to a single query. Because value isn’t part of the primary key, a direct filter isn’t feasible.

A local secondary index built specifically for this case first finds the matching item keys, then a batch DELETE removes them by primary keys.

“Using an index is really faster than a scan because the query is highly selective,” Andréas explained. “We have very few items in a given list that have the same values compared to the total number of items in a list. And thanks to the current structure, using a local secondary index is faster than a global index.”

Andréas shared another example. Starting with the original table partition, the goal is to delete all items with the value “storyC.” Using the local secondary index, the system first identifies the two rows containing that value. It then issues two DELETE statements using the item keys from those rows, which removes them from the list. The final operation, removing all list items, is even more straightforward.

“We can just drop the partition,” Andréas said, “and ScyllaDB does its magic. It just deletes all the rows for that partition, which means that it deletes all the items for the given list. And bonus point: it’s atomic. It’s either completing successfully or not changing anything at all.

ScyllaDB vs. DynamoDB performance

Medium implemented the list operations on top of both ScyllaDB and DynamoDB. The main goal was to benchmark how both databases compared on their actual production data. “Conceptually they are very close,” Andréas noted, “but they have significant differences in how they operate.”

For AddListItems, P50 latencies were low with both databases: ScyllaDB came in under 1.5ms, DynamoDB under 5ms. “DynamoDB is extremely fast, not as fast as ScyllaDB, but extremely fast at sub 5ms latency,” Andréas commented. Things got more interesting at the P95 and P99 latencies. ScyllaDB held steady at around 5-6 ms P95s, while DynamoDB ranged from 13-45 ms. ScyllaDB’s P99s were steady single-digit milliseconds, while DynamoDB’s ranged from 40- 120 ms.

It was a similar story for GetListItems. At P50, ScyllaDB clocked in at 1 ms, DynamoDB at around 3.5 ms. At P95, ScyllaDB held around 5-6 ms while DynamoDB spiked from 30 – 60ms. And at P99, ScyllaDB remained at ~30ms while DynamoDB ranged from 70 ms all the way up to 220 ms.

“ScyllaDB is very fast, with very predictable performance, and that’s super important for us.”

One caveat: DynamoDB was running without an extra caching layer. “We expect that could have a significant impact for DynamoDB because of the high cache hit rate that we are seeing on the list,” Andréas said. “But we don’t have the data yet, so we cannot compare them.” His verdict for now: “ScyllaDB is very fast, with very predictable performance, and that’s super important for us.”

Key takeaways

One pleasant side effect of getting the data model right: Medium is now eager to use ScyllaDB for additional feature store workloads. Before, they were holding back because they didn’t want to build on the shaky relational feature foundation.

Reflecting on the path to this point, Andréas left the audience with this parting advice:

“If you have a suboptimal data model, you will have queries that are slow, that will scale badly. And most likely, you won’t be able to optimize that data model. You will have to define a new data model that will be better. So take time to think about your data model before you start the implementation, because once you have production data using your suboptimal data model, it’s too late.”

The post When your data model is the bottleneck: lessons from Medium’s feature store appeared first on The New Stack.