Skip to content

feat(kafka): Enable support for zstd - #8588

Open
elramen wants to merge 1 commit into
masterfrom
elramen-kafkacodec
Open

elramen wants to merge 1 commit into
masterfrom
elramen-kafkacodec

Conversation

@elramen

@elramen elramen commented Oct 22, 2026

Copy link
Copy Markdown
Member

Enable zstd support for kafka in order to measure how much zstd outperforms lz4 in s4s2 (potential cost saving).

@elramen
elramen requested a review from a team as a code owner October 22, 2026 17:37
Comment thread rust_snuba/Cargo.toml
crate-type = ["cdylib", "rlib"]

[dependencies]
rdkafka = { version = "0.37", features = ["zstd"] }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to pull in rdkafka? We use arroyo instead. Also zstd is already a dependency of this repo.

@elramen elramen Oct 23, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arroyo's rdkafka dep doesn't use the zstd feature. But I can update arroyo and then bump the version here.

zstd being a dependency already is not enough.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would definitely prefer updating arroyo instead of adding rdkafka as a dependency here unless there's a good reason as to why we can't do it that way.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants