[Bug][Zeta] Job stuck permanently after master failover, unable to complete (affects BATCH / bounded source / job shutdown phase)

[nzw921rx]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

After master node failover or restart, a BATCH job or bounded Streaming job gets permanently stuck in RUNNING state and never completes. The same issue occurs when a Streaming job triggers a cancel or savepoint and master failover happens during the shutdown phase.

Affected scenarios:

• BATCH jobs
• Streaming jobs with bounded sources (e.g. Kafka consuming up to a specified offset)
• Streaming jobs undergoing active cancel or savepoint when master failover occurs

Normal flow:

Source finishes reading → SourceSplitEnumeratorTask sends LastCheckpointNotifyOperation to master → CheckpointCoordinator adds task to readyToCloseStartingTask set → once all collected, triggers COMPLETED_POINT_TYPE → job completes normally.

Broken flow after master failover:

1. readyToCloseStartingTask is pure in-memory state (not persisted). When the old master dies, this set is lost.
2. SourceSplitEnumeratorTask sends LastCheckpointNotifyOperation only once, then transitions to PREPARE_CLOSE. After the new master takes over, the task is already in PREPARE_CLOSE state and never re-sends the notification.
3. Deadlock forms:
   • Coordinator waiting for tasks: readyToCloseStartingTask is empty, can never be filled, COMPLETED_POINT_TYPE is never triggered.
   • Tasks waiting for coordinator: stuck in PREPARE_CLOSE, waiting for the COMPLETED_POINT_TYPE barrier to proceed.
   • Both sides wait for each other indefinitely — job is permanently stuck.

Log pattern observed:

Periodic checkpoints keep completing normally, but COMPLETED_POINT_TYPE is never triggered. Checkpoint IDs increment from 30 to 50+ with each round completing successfully:

[seatunnel-coordinator-service-22] - pending checkpoint(30/...) notify finished!
[seatunnel-coordinator-service-22] - start notify checkpoint completed, job id: ..., pipeline id: 1, checkpoint id: 30
[seatunnel-coordinator-service-47] - wait checkpoint completed: 30
[seatunnel-coordinator-service-22] - pending checkpoint(31/...) notify finished!
[seatunnel-coordinator-service-22] - start notify checkpoint completed, job id: ..., pipeline id: 1, checkpoint id: 31
[seatunnel-coordinator-service-47] - wait checkpoint completed: 31
... (continues indefinitely, checkpoint id keeps incrementing)

Normal checkpoints succeed, confirming the engine is running. However, COMPLETED_POINT_TYPE is never triggered and the job never ends.

SeaTunnel Version

2.3.13

SeaTunnel Config

Not applicable (engine-internal issue, not config-dependent).

Running Command

Not applicable.

Error Exception

No exception thrown. The job stays in RUNNING state indefinitely without any error or completion log.
Symptom: periodic checkpoints keep completing normally (checkpoint id keeps incrementing), but COMPLETED_POINT_TYPE is never triggered and the job never ends.

Zeta or Flink or Spark Version

2.3.13

Java or Scala Version

jdk1.8

Screenshots

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[nzw921rx]

@DanielLeens Please help me confirm the issue

[DanielLeens]

Thanks for writing this up so clearly. I traced the current shutdown path and this does look like a real engine bug rather than a config issue.

The problematic path is around the prepare-close / completed-point handshake:

• SourceSplitEnumeratorTask sends LastCheckpointNotifyOperation once when it first observes prepareCloseStatus, then moves to PREPARE_CLOSE;
• CheckpointCoordinator.readyToCloseStartingTask is coordinator-memory state;
• after master failover, that set is lost, but the task does not re-send the ready-to-close signal, so the new coordinator can keep finishing normal periodic checkpoints without ever collecting the full ready-to-close set needed to trigger COMPLETED_POINT_TYPE.

That matches the symptom you described: the engine is still alive and normal checkpoints continue, but the job never reaches the final completed barrier.

As a temporary workaround, please avoid master failover/restart while a bounded job is finishing, or while cancel/savepoint is in progress, because the current recovery path does not reconstruct that close-handshake state.

This should stay open for a follow-up fix. A PR is welcome if you want to drive it.