Yandex has just open-sourced a new tool called YaFF, designed to speed up data handling in heavy backend systems. It tackles a common bottleneck: the time and CPU power spent parsing Protobuf messages. YaFF lets programs read data directly from memory without unpacking it first. This can save up to 20% of CPU resources in production environments.

Protobuf is a popular format for sending structured data. It works well for many applications, but parsing its messages can be costly when systems handle millions of requests each second. The parsing step consumes a lot of CPU, which adds up to big infrastructure costs. YaFF changes the way Protobuf messages sit in memory, so the data can be accessed directly. This zero-copy approach avoids the usual overhead.

Unlike other zero-copy tools like FlatBuffers, YaFF keeps using the same Protobuf schemas and APIs. This means developers don’t have to rewrite their code or manage duplicate schema files. You can introduce YaFF incrementally, switching just the busiest parts of your system without disrupting the rest. That makes adopting it less risky and less work.

How YaFF Works and Why It Matters

YaFF does not replace Protobuf. Instead, it offers alternative memory layouts for Protobuf messages. These layouts control how data is arranged in memory without changing the message structure or the generated interfaces. YaFF supports four layouts: Fixed, Flat, Sparse, and Dynamic. Each offers a different balance between read speed and schema flexibility.

The Fixed layout is the fastest but requires a stable schema. Flat supports some schema changes and adds a small header. Sparse handles very flexible schemas but reads slower. Dynamic switches between Flat and Sparse depending on schema compatibility and is the default choice. This flexibility helps YaFF handle various data models efficiently.

Benchmarks show YaFF’s Flat layout reads data nearly four times faster than FlatBuffers and about 22 times faster than standard Protobuf parsing. It runs close to raw C++ structs in speed. This means applications can access hot data almost instantly, which is critical for high-throughput services.

Real-World Impact at Yandex

Yandex already uses YaFF in its advertising recommendation system. This system handles hundreds of thousands of requests every second, each processing tens of thousands of objects. By switching to YaFF, Yandex cut CPU usage by 10 to 20 percent. That frees up a lot of computing power and reduces infrastructure expenses.

YaFF also fits well with memory-mapped files. These files let a system store large data sets locally and access them without loading or parsing at startup. This saves time and memory when services restart or scale up. For large datasets, such as tens of gigabytes of ad banners, memory-mapping with YaFF means faster response times and fewer resource demands.

The ability to maintain the same Protobuf schema across the entire pipeline simplifies development. Teams can avoid costly data conversions between formats. This reduces bugs and speeds up feature rollouts. For companies dealing with big data and real-time processing, this is a huge win.

YaFF is open-source under the Apache 2.0 license and currently supports C++. Developers can experiment with it right away. Yandex plans to keep improving YaFF and hopes the community will contribute to its growth. This tool adds to Yandex’s portfolio of open-source projects aimed at making large-scale backend systems more efficient and reliable.