We’re excited to officially launch Wingfoil today. Wingfoil is an ultra-low latency, highly-scalable stream processing framework used to receive, process and distribute streaming data. Built in Rust, it’s open source, simple to use and seamlessly supports both real-time and historical workloads.
Streaming data is data that is continuously generated and delivered in real, or near-real time. Unlike more traditional batch processing, where data is collected, stored and processed later, streaming data allows systems to handle information as it arrives, which in turn makes it possible to detect patterns, generate insights, and trigger actions without waiting for a complete dataset to be collected first. Streaming data has emerged over the last 10-15 years as a very significant component of the digital ecosystem, driven in part by the exponential growth of data and in part by the push to develop ever faster and more responsive systems. As a result, streaming data pipelines have become a common part of many organisations’ digital infrastructure, with new streaming technologies like Kafka and ReactiveX making it possible to scale digital applications and data in ways that were previously impossible.
Wingfoil is the only streaming data framework that offers ultra-low latency performance
We built Wingfoil because current data streaming systems struggle when it comes to ultra-low latency applications, e.g. systems where micro and nano seconds count , like electronic market places, trading engines, autonomous vehicle pipelines, and real-time monitoring systems. This is understandable, because of these frameworks were designed for throughput, e.g. clickstream data on highly scaled web applications. And while latency is consideration in these types of environments, the performance required is of a lower order than applications like electronic market places and real time AI systems in UAVs for example. This means developers have had to ‘hand roll’ their own streaming systems for these types of application – a complex and time consuming task. And there are tens of thousands of electronic market places with hundreds being added each year. And that’s just one applications Wingfoil can be used for.
Wingfoil is ultra low latency
Wingfoil prioritises end-to-end latency under 500µs for small messages, even under high throughput, ensuring systems respond immediately to changing conditions. It uses a zero-copy architecture to reduce serialisation and deserialisation overhead, and it handles back pressure efficiently without stalling the pipeline, maintaining system stability under load.
Wingfoil is super lightweight
Other streaming frameworks do a range of jobs, from acting as message buses and orchestrators of complex streaming environments to delivering streams. Wingfoil focusses on ultra-low latency streaming and is built to add the minimum of overhead to the process of streaming data.
Wingfoil is simple to deploy
Developing bespoke solutions for low latency streaming data in languages like Rust and C++ is complex. Like other frameworks, Wingfoil solves a problem so you don’t have to.
Wingfoil is highly scalable
Wingfoil offers simple, composable APIs so you can plug it directly into your machine learning models or risk engines and easily integrate with existing tools. Wingfoil has browser bindings in Python and Typescript and integrates with Tokio to simplify the setup of asynchronous I/O adapters, which means workloads can be efficiently distributed across multi-threaded and multi-server environments.
Wingfoil is open source
The Wingfoil ultra-low latency data streaming framework is open source. You can download from crates.io, read the documentation, review the benchmarks or jump straight into one of the examples. You can also download the wingfoil python module from pypi.
We built Wingfoil, because we wanted a better way to develop for real time applications. And we believe that real time applications are only going to become more important, whether those are monitoring systems for embodied AI or electronic market places.
For more details on Wingfoil and ultra-low latency streaming you can read our white paper.
And we’d love to hear from you! Wingfoil is an open source projects, so we’re always looking for contributors, and we’re happy to help out, there’s a feature you’d like to see or you have any other feedback, then let us know. And if you’re building something using Wingfoil too, then we’d also love to hear from you.
Please email us at hello@wingfoil.io or get involved in the wingfoil discussion. And you can take a look at the issues for ideas on ways to contribute.
