bug(slurm): 95 zombie [srun] processes leaking under daphne PID 1

[ywatanabe1989]

Symptom

docker exec scitex-cloud-prod-django-1 ps aux shows 95 [srun] <defunct> zombie processes, all with PPID 1 (daphne).

     95 1   ← ppid 1 = daphne, zombies accumulate since Apr15

Root cause

Code that spawns srun via subprocess.Popen / subprocess.run is not awaiting/reaping the children. Daphne (asgi) runs as PID 1 and doesn't install a SIGCHLD handler / subprocess-reaper, so child exits stay as zombies.

Impact

• Not immediately fatal, but indicates a subprocess lifecycle bug.
• Fills the process table over time.
• Same code path likely retains file descriptors / pipes → contributes to the daphne memory growth in bug(django): daphne memory leak — container hitting 8 GiB cgroup limit, causing site-wide 504 #147.

Fix

1. Find the srun invocation site:

   grep -rn \"srun\" src/ apps/

2. Replace subprocess.Popen('srun ...') fire-and-forget with either:
   • asyncio.create_subprocess_exec(...) + await proc.wait() in async views
   • subprocess.run(..., check=False) in sync paths
3. Alternatively run daphne under tini (docker run --init) so PID 1 reaps orphans: add init: true in the django service of docker-compose.prod.yml.

Acceptance

After 24 h of prod traffic: ps -eo stat | grep -c ^Z in django container is 0.

[ywatanabe1989]

Partial fix landed in the same compose edit as #147: init: true added to the django service so Docker's built-in init runs as PID 1 and reaps orphans. This resolves the accumulation on redeploy. A full fix (calling await proc.wait() on srun subprocesses in code) is still desirable so we don't rely on init, and is tracked separately.

[ywatanabe1989]

Source audit (no code change yet)

grep -rn for srun/Popen across apps/:

• apptainer_runner.py:66 — subprocess.Popen(...) with timeout path; calls proc.kill() + proc.wait(). Clean on normal + timeout paths. Exception paths before proc.wait() could leak but rare.
• allocation.py:394 — subprocess.run(["srun", ...], timeout=10) — Python's subprocess.run with timeout cleans up on TimeoutExpired. Clean.
• session.py:279 and execution.py:196 — os.execvpe("srun", ...) — replaces process image; no zombie possible.
• dev_app_runner.py:156 — subprocess.run(cmd, ...) — clean.
• project_service_manager.py:304 — subprocess.Popen(..., start_new_session=True) returns the proc; line 212-213 later calls process.terminate() via psutil without process.wait() afterward. Orphaned terminate leaks zombies. Only called by cleanup_idle_services, which I can't find a caller for (grep cleanup_idle_services shows only the definition — no scheduler / celery beat entry / view wiring). So not currently the active leak.
• workflow_tasks.py:360 — (next to audit)
• unix_user.py:163 — returns Popen to caller; caller must wait.

Actual structural cause

Running container has daphne as PID 1, so tini (specified in docker-compose.prod.yml entrypoint) is not actually PID 1. Without PID 1 reaping, ANY child of daphne that exits without daphne calling .wait() becomes a zombie. This happens for any path that terminate()s without wait() — current code only has one such path (the dormant cleanup_idle_services) but historically there were likely more.

The init: true addition in my earlier (reverted) compose edit would have fixed this at the infra level. I'm leaving that structural fix out of #147 and filing it as a small separate edit to the django service in compose:

django:
  init: true   # docker-init reaps orphans regardless of whether
               # daphne replaces tini as PID 1

Will propose that as a standalone PR once the 504 root-cause investigation is complete.

[ywatanabe1989]

Auto-labeled scitex-cloud via lead msg#16257 bulk audit.