Project Overview
Aura is an open-source orchestration layer designed to execute stateless functions across a distributed fleet of worker nodes with minimal overhead.
The Problem
Modern data pipelines often struggle with the overhead of serialization and deserialization across network boundaries. Existing solutions like AWS Lambda or Kubernetes jobs require heavy container runtimes, leading to significant memory footprints and cold-start times exceeding several seconds. For high-frequency, low-latency financial data processing, this was unacceptable.
The Solution
By leveraging WebAssembly (WASM) on the edge and Rust for the core processing engine, Aura reduces cold start times to single-digit milliseconds while providing strict memory safety guarantees and near-native execution speed.
Architecture
The core engine uses a custom actor model designed specifically for high-throughput environments. Message passing is handled via zero-copy buffers, entirely bypassing traditional JSON serialization.
Technical Challenges
- WASM Memory Limits: We hit the 4GB WASM memory limit early on when processing large data frames. We solved this by implementing a streaming buffer approach.
- Deterministic Networking: Getting WASM modules to make reliable network calls required writing a custom host-function layer bridge.
Results & Impact
Lessons Learned
While WebAssembly is incredibly promising for backend workloads, the tooling ecosystem (especially around debugging) is still nascent. Building the custom host integrations took three times longer than anticipated, but the resulting performance benefits validated the architectural choice.