Async Geo-Replication and Cluster Down Scenarios

[shasank112001]

With Async Geo Replication between 2 clusters, I understand that there is a special replicator on each cluster that consumes incoming messages, adds metadata of the originating cluster and then publishes it to the other side. I also understand that on both sides, you can configure a QueueSize for each replicator so it reads from the ledger in Chunks.
However, in the metrics there is also a metric for replication-backlog which keeps track of how many messages need to be replicated to the other side. I couldn't find any property to control this replication backlog size.

I have the following scenario:

1. Cluster A with topic X configured to replicate towards Cluster B.
2. Cluster B with topic X configured to replicate towards Cluster A.

Now, lets assume that in a failure scenario, Cluster A goes down. Therefore I have clients producing and consuming only from cluster B. As the cluster A is down, the replicator cannot replicate any messages towards it. Therefore, the replication backlog keeps on increasing, while the topic backlog is near 0, as nearly all messages being produced are also consumed.

This scenario can pose a challenge, as it can lead to storage being full, because "acknowledged" messages are still waiting to be replicated.

Another question is how does messageTTL work with the replicator. If I send a message to Cluster B with TTL 5 seconds, and the replicator cannot replicate because Cluster A is down, does that mean after 5 seconds, the message will be removed from the replicator backlog as well?

[Denovo1998]

replicationBacklog is not a separately configurable buffer size in Pulsar. It is effectively the backlog of the replicator cursor, i.e. the number of source-topic entries that have not yet been acknowledged by the remote-cluster replication path.

The QueueSize setting you found only controls the internal replication producer pending queue and how aggressively the replicator reads from the ledger. It does not cap the durable backlog stored on disk. So in your A-down / B-still-serving scenario, yes: the B->A replication backlog can keep growing even if the local consumer backlog is close to zero.

That happens because local consumer acknowledgements only advance subscription cursors. The replicator cursor advances only after the message is successfully published to the remote cluster. Until that happens, the entries remain retained on cluster B. This is by design, because otherwise Pulsar would be silently discarding data that has not yet been replicated.

---

For TTL, the answer is also yes, with one important nuance: message expiry applies to replicator cursors too, so expired messages can be removed from the replication backlog and therefore will not be replicated once cluster A comes back. However, this is not guaranteed to happen exactly at T+5 seconds. TTL means the message becomes eligible for expiry after 5 seconds, and the actual removal depends on the broker’s periodic expiry check.

So there is no dedicated maxReplicationBacklog knob. If you want to bound this risk during a remote-cluster outage, the practical controls are backlog quota, TTL, or an explicit operational decision to clear the replicator backlog and accept data loss for the remote cluster.