Databricks Pushes Feature Stores From Batch Lag to 200ms Freshness

A model deciding whether to approve a payment may need a user’s transaction activity from moments earlier, not the last scheduled data refresh. Databricks says its Feature Store now moves an event from Kafka to its online feature store in 200 milliseconds at the 99th percentile, using a continuously running Spark pipeline, Lakebase storage and Model Serving retrieval. The claim puts the emphasis on a usually awkward part of machine-learning systems: keeping the data supplied at inference time as current as the event stream.

Scheduled Spark jobs remain suited to historical baseline features, but Databricks says they introduce lag measured in minutes or hours. That can be acceptable when a model needs a stable long-range profile. It is a poorer fit for a short window, such as a customer’s transaction total over the previous 10 minutes, where each new event can change the feature a fraud model should receive.

The new architecture starts with events in Kafka. A serverless Lakeflow Spark Delta Pipeline running Spark Real-Time Mode computes rolling aggregations, then sends updated values through a streaming JDBC sink to Lakebase, which serves as the online feature store. Model Serving endpoints retrieve the latest features from Lakebase when a model is invoked.

Feature Store supports tumbling, sliding and rolling aggregation windows. Tumbling windows emit values at fixed interval boundaries, while sliding windows overlap on a set schedule. A rolling window instead looks backward from each event timestamp, giving it millisecond-level resolution and allowing a fresh aggregate to move whenever a new event arrives or an older event expires.

Traditional microbatch streaming collects events into discrete batches, processes stages sequentially and checkpoints at batch boundaries. Databricks says Real-Time Mode instead runs stages concurrently and processes rows without waiting for a microbatch to finish. For rolling features, each incoming row updates a local RocksDB state store and emits the revised aggregate downstream; expiration removes an old event’s contribution and produces a corrected value.

The design choices behind the latency claim

• Continuous row processing avoids waiting for a conventional microbatch to complete.
• Lakebase is intended to limit write amplification from the frequent, small updates created by streaming aggregates.
• Checkpoint planning and checkpoint work are amortized over longer intervals rather than imposed at every batch boundary.

The pipeline still needs durable state and recovery behavior. Databricks says Real-Time Mode maintains exactly-once processing guarantees while spreading checkpointing costs over longer intervals. Its stated trade-off is that, after a failure, the pipeline replays at most five minutes of Kafka data. The company presents that added replay volume as the price of lower steady-state latency.

The architecture joins feature computation, online storage and inference retrieval under one Databricks-managed path, reducing the need to assemble those components separately. The material open question is how the company-reported 200ms p99 result holds across customer workloads, event volumes and feature definitions. The stated figure measures arrival in Kafka through availability in the online feature store, rather than the model’s full decision time.