For anyone in AI, it’s no big news that “data is the real prize.” If you have strong data foundations, your models and the applications powered by them will be right on the money.
But that’s where it gets messy. Building that foundation is no piece of cake, especially when there are dozens of data sources, each hosting valuable information. You need to build and maintain integration pipelines for each source — a massive engineering burden for data teams juggling disparate ETL tools to centralize what’s needed to power AI workloads. At scale, these pipelines become rigid bottlenecks — hard to adapt, extend or expand.
Snowflake thinks it has an answer.
Today, at its annual summit, the company announced the general availability of Openflow — a fully managed data ingestion service that pulls any type of data from virtually any source, streamlining the process of mobilizing information for rapid AI deployment.
How does it work?
Powered by Apache NiFi, Openflow uses connectors — prebuilt or custom — with Snowflake’s embedded governance and security. Whether it’s unstructured multimodal content from Box or real-time event streams, Openflow plugs in, unifies, and makes all data types readily available in Snowflake’s AI Data Cloud.
“Data engineers often faced a critical tradeoff – if they wanted highly controllable pipelines, they encountered complexity and significant infrastructure management. If they wanted a simple solution, they encountered issues of limited privacy, flexibility and customization. Openflow meets customers where their data lives, providing deployment flexibility and guaranteeing security and governance along the way,” Chris Child, VP of Product, Data Engineering, at Snowflake, told VentureBeat.
While Snowflake has offered ingestion options like Snowpipe for streaming or individual connectors, Openflow delivers a “comprehensive, effortless solution for ingesting virtually all enterprise data.”
“Snowflake’s Snowpipe and Snowpipe Streaming remain a key foundation for customers bringing data into Snowflake, and focus on the ‘load’ of the ETL process. Openflow, on the other hand, handles the extraction of data directly from source systems, then performs the transform and load processes. It is also integrated with our new Snowpipe Streaming architecture, so data can be streamed into Snowflake once it is extracted,” he explained.
This ultimately unlocks new use cases where AI can analyze a complete picture of enterprise data, including documents, images, and real-time events, directly within Snowflake. Once the insights are extracted, they can return to the source system using the connector.
Over 200 connectors available
Openflow currently supports 200+ ready-to-use connectors and processors, covering services like Box, Google Ads, Microsoft SharePoint, Oracle, Salesforce Data Cloud, Workday and Zendesk.
“Box’s integration with Snowflake Openflow…leverages data extraction from Box using Box AI, honors the original permissions for secure access, and feeds that data into Snowflake for analysis. It also enables a two-way flow in which enriched insights or metadata can be written back to Box, making content smarter over time,” Ben Kus, CTO at Box, told VentureBeat.
Creating new connectors takes just a few minutes, speeding up time to value. Users also get security features such as role-based authorization, encryption in transit, and secrets management to keep data protected end-to-end.
“Organizations that require real-time data integration, deal with high volumes of data from various sources, or rely on unstructured data like images, audio, and video to derive value from will benefit immensely from Openflow,” Child added. A retail company, for instance, could unify siloed data from sales, ecommerce, CRM, and social media to deliver personalized experiences and optimized operations.
Snowflake customers Irwin, Securonix, and WorkWave are among those set to use Openflow to move and scale global data — though the company hasn’t disclosed exact adoption numbers.
What’s next?
As the next step, Snowflake aims to make Openflow the backbone of real-time, intelligent data movement across distributed systems – powering the age of AI agents.
“We’re focusing on moving events at a massive scale and enabling real-time, agent-to-agent bi-directional communication, so insights and actions flow seamlessly across distributed systems. For example, a Cortex Agent handing over events to other enterprise agents from other systems, like ServiceNow,” Child said.
The timeline for these upgrades remains unclear for now.
The post Snowflake’s Openflow tackles AI’s toughest engineering challenge: Data ingestion at scale appeared first on Venture Beat.
