> If writing to object store vs disk, wouldn't this impact the performance?
Bufstream e2e latency is a little bit higher, but it is still fast enough for most workloads, and for a small latency tradeoff you get massive cost savings, operational simplicity, air gapped deployments, broker side data validation, role based access control, and much more).
Here is a relevant bit from the benchmark docs:
"Of course, there's no free lunch. Object storage is durable, scalable, and cost-efficient, but it's not as low-latency as local disk. For this benchmark, we used Bufstream's default configuration, which strikes a balance between end-to-end message delivery latency and cost. With that configuration, median end-to-end latency was 260 milliseconds and p99 latency was 500 milliseconds."
Very thoughtful article. Thank you!
Hello Vu, about the red and blue colors in the 1st diagram under "A bit of Kafka", blue should be "disk read" red is "disk write"..
or am i wrong ??
You're right. I fixed it. Thank you so much for the feedback ;)
Beautiful diagrams which website did you use to create diagrams ?
Oh thank you, I used draw.io + Canva
Thanks for sharing.
If writing to object store vs disk, wouldn't this impact the performance?
Do they support Delta?
> If writing to object store vs disk, wouldn't this impact the performance?
Bufstream e2e latency is a little bit higher, but it is still fast enough for most workloads, and for a small latency tradeoff you get massive cost savings, operational simplicity, air gapped deployments, broker side data validation, role based access control, and much more).
Here is a relevant bit from the benchmark docs:
"Of course, there's no free lunch. Object storage is durable, scalable, and cost-efficient, but it's not as low-latency as local disk. For this benchmark, we used Bufstream's default configuration, which strikes a balance between end-to-end message delivery latency and cost. With that configuration, median end-to-end latency was 260 milliseconds and p99 latency was 500 milliseconds."
Source
https://buf.build/docs/bufstream/cost/#predictable-performance
> Do they support Delta?
Not currently. Here are the docs that explain why Bufstream targeted Iceberg: https://buf.build/docs/bufstream/iceberg/
Thanks for great article.
Is there any open source alternative for bufstream or wrapstream?
To my knowledge, I don't think we have the open-source alternatives
i think its warpstream... I know, wrap is tempting :-)
may be autoMQ, i read they also use object storage.. but not sure about iceberg integration