Real-time streaming data can be valuable for numerous applications and purposes across industries. In the case of the New York Stock Exchange (NYSE), streaming data is literally money.
The NYSE is one of the largest financial exchanges in the world and has a lengthy history of being able to share its financial market data.
A hundred years ago it used telegraph based ticker tape to share information. In the modern era it has developed its own low-latency, high-performance technologies deployed on-premises that other organizations can connect with.
Now it’s taking the next step forward, embracing a model based on the open-source Apache Kafka streaming technology that brings NYSE Best Quote and Trades (BQT) data to the AWS cloud.
To do that, NYSE partnered with streaming data platform vendor Redpanda, which has developed its own implementation of Kafka written in the C++ programming language.
NYSE’s deployment of Redpanda’s C++-based streaming platform achieved 4-5x performance improvements over traditional Kafka competitors, exposing fundamental limitations in how most organizations handle bursty data workloads.
This performance gap becomes critical as enterprises scale AI applications that demand consistent low-latency data access. Kafka-based data streaming also has potential to enable agent-to-agent communications, rivaling other approaches like Google’s A2A and it can also be extended to enable Model Context Protocol (MCP).
“The market thesis is that all of the large foundation models have really indexed the public data sets, and the next frontier is private data sets, and Redpanda really unlocks private data sets for agentic access,” Alex Gallego,founder and CEO of Redpanda told VentureBeat.
What the NYSE is building in the cloud
NYSE built its cloud streaming platform to serve customers who cannot access its data centers directly. The exchange targets fintech companies and retail broker-dealers who need AWS-based access to real-time market data.
“Not every consumer of our market data has the capacity to come to our data center, take the feed and use that feed,” Vinil Bhandari, head of cloud and full stack engineering at NYSE told VentureBeat. “But you know, a small shop in Hong Kong has access to creating their own AWS account, for example, and it’s those audiences that we are trying to cater to.”
NYSE streams its BQT (Best Quotes and Trades) feed, which aggregates real-time data from all seven NYSE exchanges. The deployment required building new infrastructure rather than extending existing systems.
Why NYSE chose Redpanda and how programming language choice matters
NYSE processes over 500 billion messages daily across seven exchanges. During market volatility, message volume can spike 1,000x above average within microseconds.
Traditional Java implementations struggle with these patterns because garbage collection creates unpredictable latency spikes.
“The classic Kafka implementation was written in the Java programming language, which makes this bursty kind of traffic, you know, not fair very well with Java’s garbage collection that happens in the programming language,” Bhandari explained. “Redpanda has done the Kafka implementation by rewriting Kafka protocol in C++ so whenever we get a burst of traffic from our market activity, the volatility, we are able to manage that streaming out of data better.”
The choice of programming language is also why NYSE went with Redpanda for data streaming instead of other options such as Confluent or Amazon Managed Streaming for Kafka (MSK).
This technical decision resulted in measurable performance improvements.
“We are safe to establish that we are at least four to five times faster in our data delivery using Redpanda as compared to some of our big ticket custom competitors who are using Kafka technology to stream similar data,” Bhandari noted.
For enterprises evaluating streaming platforms, this comparison highlights a critical consideration: Java-based implementations for data streaming may struggle during traffic spikes, while C++ based alternatives can maintain consistent performance.
Observability proves critical for mission-critical deployments
Bhandari emphasized observability as essential for production streaming deployments. Redpanda’s built-in telemetry capabilities provided immediate operational value.
“The more that a deployment like this can have observability and telemetry of what’s happening under the hood, the better the producer of the data and the consumers of the data are going to be,” Bhandari explained.
This observability enables proactive issue detection and resolution before problems impact customers. Without comprehensive monitoring, enterprises risk discovering performance issues only after they affect production workloads and customer experience.
Architecture philosophy shift: Streaming as an AI foundation
NYSE will be using the streaming data capabilities in a fairly traditional way, at least initially. That is data from its market exchanges is made available for users to consume.
The direction that Redpanda is headed points to a more agentic AI future, one that users such as NYSE will likely embrace in the years ahead. Redpanda CEO Gallego argues that enterprises should view streaming architecture differently in the AI era.
“Streaming has the right architectural pattern, not for speed, but because it is the right architecture for reactive and agentic applications,” Gallego explained.
Beyond solving traditional streaming performance problems, Redpanda has repositioned itself for what Gallego calls the agentic enterprise. The company has wrapped its data connectors in MCP (Model Context Protocol), enabling AI agents to access enterprise data sources directly.
This approach solves a computational complexity problem that emerges as enterprises deploy multiple AI agents.
“Without the Kafka API, you have an n squared communication problem where every agent has to have access to every other agent,” Gallego said. “And when you introduce the Kafka API, then it reduces from n squared computational complexity down to linear.”
According to Gallego, banks are already deploying hundreds of agents. One Redpanda customer plans to build 1,000 agents over the next two years. Another is currently building 130 agents for production deployment within 18 months. These scale requirements make agent coordination architecture decisions critical for long-term AI strategy success.
What this means for enterprise data strategy
Real-time streaming data is set to become an increasingly critical aspect of many organization’s operations.
NYSE’s evaluation process reveals critical decision criteria for enterprise decision-makers evaluating streaming infrastructure:
Java-based Kafka hits performance walls under burst traffic. Organizations handling unpredictable workloads should evaluate C++-based alternatives before scaling production deployments. The 4-5x performance difference isn’t marginal optimization but fundamental capability gap.
Cloud-first streaming strategies can achieve production-grade performance. This enables global data access patterns that were previously impractical due to latency constraints, opening new market opportunities for data-driven businesses.
Agent coordination requires streaming architecture. As AI deployments expand beyond single agents, streaming platforms become essential infrastructure rather than performance optimizations. The computational complexity advantages become critical at scale.
For organizations planning AI implementations it’s critical to prioritize streaming platforms that support both MCP integration and agent coordination. The computational complexity advantages become critical at scale and retrofitting coordination architecture after deploying multiple agents proves exponentially more difficult than building it correctly from the start.
Organizations waiting to adopt AI should recognize that streaming architecture decisions made today will constrain future AI capabilities more than most leaders realize.
The post The NYSE sped up its realtime streaming data 5X with Redpanda appeared first on Venture Beat.
