Using Goroutines and Channels for High-Throughput S3 Object Hashing
Revisiting the S3 Hasher, but this time in Go! We achieve similar performance to Rust, processing at just around 3 GB/s using Go’s built-in concurrency primitives.
Performance
High-Throughput S3 Object Hashing: 3GB/s with Rust and Tokio
A high-speed S3 object hasher built with Rust, processing at up to 3 GB/s. We use a channel-based architecture to concurrently stream and hash S3 objects. We process a 12 TB dataset in just over an hour, demonstrating Rust’s capability for efficient, large-scale data processing.