Kafka endpoint only supports a single broker address

[badaverse]

Describe the issue

The Kafka endpoint currently accepts only a single broker address in its URL format (kafka://host:port/topic). Kafka is typically deployed as a multi-broker cluster, and connecting to a single broker means the client cannot leverage cluster discovery, failover, or load balancing — core benefits of a Kafka cluster setup.

While a single broker address still allows cluster discovery under normal conditions (the client fetches full cluster metadata after the initial connection), it creates a single point of failure for bootstrapping. If that one broker is unavailable, the client cannot establish the initial connection and cluster discovery fails entirely — making it a half-measure for high availability.

The underlying library (github.com/IBM/sarama) already supports multiple broker addresses via NewSyncProducer([]string{...}, cfg), but the endpoint URL parser only extracts a single host:port pair.

Expected behavior

Support specifying multiple broker addresses in the Kafka endpoint URL, following the standard comma-separated format:

kafka://broker1:9092,broker2:9092,broker3:9092/topic-name

All specified brokers should be passed to the Sarama producer so the client can discover the full cluster topology and failover correctly.

Additional context

• Sarama's NewSyncProducer accepts []string of broker addresses but is currently called with a single-element slice.
• The SETHOOK command uses commas to separate multiple fallback endpoint URLs. The URL split logic needs to be updated to distinguish between commas inside a single endpoint (Kafka brokers) and commas between separate endpoints.

[iwpnd]

https://tile38.com/commands/sethook#endpoint-failover

[badaverse]

@iwpnd Thanks for the pointer! I'm aware of the endpoint failover feature, but this addresses a different layer.

Endpoint failover is a Tile38-level retry. each endpoint creates an independent Kafka producer connecting to a single broker, and Tile38 must fail a send (timeout/connection refused) before trying the next one.

Multiple bootstrap brokers (kafka://b1:9092,b2:9092/topic) is a Kafka client-level feature, a single producer receives multiple addresses for initial cluster discovery, and Sarama handles failover internally without needing a failed send first.

They complement each other rather than being interchangeable.

[badaverse]

I went ahead and worked on this in a branch: master...badaverse:tile38:feat/kafka-multiple-brokers

I'd like to discuss the approach with a maintainer first, so I opened this issue. Please take a look when you get a chance.

[iwpnd]

I agree - somewhat. This part of Tile38 hasn't been touched in ages and a technology like Kafka (and therefor their clients) evolves quickly. A check once in a while is a good idea for sure. But you make this sound more shiny than it actually is.

must fail a send (timeout/connection refused) before trying the next one.

This is exactly the same behaviour as in Sarama itself with the added benefit (?) that Sarama hides unhealthy brokers from the caller, and eventually retries them. At least that's what I get from their implementation and Wiki. I do see a benefit if Sarama performs frequent healthchecks on the list of brokers, but I cannot verify that they do.

From their Wiki:

The current implementation is ugly and not optimal, but works reasonably well in practice.

https://github.com/IBM/sarama/wiki/Cluster-metadata#current-implementation

Something that you could say from the current Tile38 implementation as well. 😄

When I understand you correctly, then what you want is for Sarama to handle retry and failover in the client versus having Tile38 handle it, and your argument is that "client can discover the full cluster topology and failover correctly". What actual limitation have you encountered in your use of Tile38s Kafka hook?

[badaverse]

To be transparent, I didn't hit a hard runtime limitation. I noticed this when I tried to configure Kafka hooks the same way I configure bootstrap servers in every other Kafka client (host1:port,host2:port), and it didn't work as expected because Tile38 split it into separate endpoints. That's what led me here.

You're right that the endpoint failover workaround works in practice. But from a usability standpoint, it's a Tile38-specific pattern that Kafka users wouldn't expect. You'd need to read the failover docs to understand why comma-separated brokers aren't behaving as bootstrap servers.

On the retry behavior, Sarama does run a background metadata refresher (Metadata.RefreshFrequency, default 10min) that proactively updates cluster topology, so it's not purely reactive. But I agree the practical difference at the retry level is small.

That said, I think there's also value in letting a Kafka client like Sarama fully own the responsibility of cluster discovery and broker failover, rather than working around it at the Tile38 layer. It's what the library is designed to do.

[iwpnd]

Agreed, and thanks for the additional context.