Support a single target resource for doneness/deletion trigger

[wseaton]

Single deletion target

In the llm-d project we have an interesting problem where we need to deploy pretty often:

1. 2 Deployments or LWS (using Kueue for this solves a lot of resource coordination problems with experiments)
2. A Job containing a simulated inference workload (like guidellm) or unit tests
3. A whole bunch of other configuration resources that can also be ephemeral but aren't scheduleable

It's my understanding that the AppWrapper never exits because the Deployments are never considered to be done.

It would be a great feature to allow defining a single target resource that is used to garbage collect everything else in the AppWrapper, then I wouldn't have to include a poison pill or self-destruct script that triggers after the benchmark is complete to tear down the inference deployment. I would set the target deletion resource to be the Job.

[wseaton]

Proposed API using TypedLocalObjectReference:

apiVersion: workload.codeflare.dev/v1beta2
kind: AppWrapper
metadata:
  name: my-benchmark
spec:
  completionPolicy:
    triggerRef:
      apiGroup: batch
      kind: Job
      name: benchmark-job
    # Optional: what to do when trigger completes
    action: DeleteAll  # or just UpdateStatus
  components:
    - template: ...  # Deployment (runs until deleted)
    - template: ...  # Job (triggers completion)

Go types:

type CompletionPolicy struct {
    // TriggerRef identifies the component that determines AppWrapper completion.
    // Uses TypedLocalObjectReference (apiGroup, kind, name).
    TriggerRef corev1.TypedLocalObjectReference `json:"triggerRef"`
    
    // Action specifies what happens when the trigger component completes.
    // DeleteAll (default): delete all other components
    // UpdateStatus: only update AppWrapper status, leave components running
    // +optional
    Action CompletionAction `json:"action,omitempty"`
}

type CompletionAction string

const (
    CompletionActionDeleteAll     CompletionAction = "DeleteAll"
    CompletionActionUpdateStatus  CompletionAction = "UpdateStatus"
)

[dgrove-oss]

The use case makes sense. The old MCAD scheduler (AppWrapper v1/beta1) had a way to designate one wrapped resource as being in control of termination.

I can commit to reviewing any PRs submitted towards implementing this, but do not have the bandwidth to do more than that.

[dgrove-oss]

An alternative API would be to use the annotations field of AppWrapperComponent to indicate that when that component completes, the entire AppWrapper should be considered completed. This keeps the information "close" to the resource and doesn't require additions to the API. Just logic in the controller to detect the annotation and take appropriate action.

I'd lean to taking that approach unless for some reason it is not viable.

[wseaton]

Thanks for the suggestion @dgrove-oss, I have time to implement this and throw up a PR this week.