Search before asking
Linkis Component
Description
Currently, Linkis uses a fixed queue configuration for Yarn jobs. This can lead to inefficient resource utilization when some queues are heavily loaded while others have available capacity. Users need the ability to automatically select the optimal queue based on real-time resource usage.
Steps to reproduce
- Configure a primary queue and a secondary queue
- Submit jobs to Linkis
- Jobs always use the primary queue even when it's heavily loaded
- Secondary queue capacity is wasted
Expected behavior
Linkis should automatically select the optimal queue based on:
- Resource usage threshold (configurable)
- Support for multiple engines (Spark, etc.)
- Support for specific Creator types
- Automatic fallback to primary queue if secondary queue is unavailable or overloaded
Your environment
- Linkis version used: 1.18.0
- Environment name and version:
- hadoop-3.3.4
- spark-2.4.3
- scala-2.11.12
- jdk 1.8.0_xxx
Anything else
This feature enables intelligent queue selection to optimize resource utilization and improve job performance.
Are you willing to submit a PR?
Search before asking
Linkis Component
Description
Currently, Linkis uses a fixed queue configuration for Yarn jobs. This can lead to inefficient resource utilization when some queues are heavily loaded while others have available capacity. Users need the ability to automatically select the optimal queue based on real-time resource usage.
Steps to reproduce
Expected behavior
Linkis should automatically select the optimal queue based on:
Your environment
Anything else
This feature enables intelligent queue selection to optimize resource utilization and improve job performance.
Are you willing to submit a PR?