Fix saturation detection and harden load generator

[Bslabe123]

Addresses: #300

Refactors LoadGenerator introducing a stepped load strategy. New approach continually monitors throughput, automatically detects saturation, and dynamically readjusts the testing range.

Motivation

Previous static load generation frequently failed on cold starts (0 successful requests) and required repetitive manual tuning to capture inflection points.

Changes

• Stepped Preprocessing: Ramps up load while monitoring throughput; detects saturation when rates drop below 90% tolerance.
• Automatically shifts the test window if initial rates are too high to capture the true saturation point.
• Replaces arbitrary start-at-1 logic with proportional scaling for better resolution at lower rates.

Verification

• Added tests/loadgen/test_saturation_robustness.py covering hard limits, early saturation, and tolerances.
• Updated tests/loadgen/test_preprocess.py reflecting new rate calculation.
• Validated manually using reproduce_rates.py for mathematical correctness.
• Added loadgen preprocessing sanity check to e2e test suite

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Bslabe123
Once this PR has been reviewed and has the lgtm label, please assign sergeykanzhelev for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jjk-g]

Can you show an example of how the rates and duration are selected as you go through finding saturation?

[Bslabe123]

Data from last round of testing across various models, model server versions, hardware, and dataset configs:

[jjk-g]

Any parameters we have hardcoded should be exposed with a reasonable default in the sweep config, or a reasonable reason (MAX vs a configurable) and then they should be moved outside the scope of the function.

[Bslabe123]

Maybe we consider an alternate approach before exposing any new parameters in the sweep config unless its guaranteed that the generated stages are a function the new parameters? I'm worried about exposing implementation details that might turn into breaking API changes if we ever decide change and/or harden the preprocessing algorithm.