Skip to content

refactor(test): Pre-create MSK/MQ in companion stack for testing#3909

Open
vicheey wants to merge 2 commits intodevelopfrom
integration-test-update
Open

refactor(test): Pre-create MSK/MQ in companion stack for testing#3909
vicheey wants to merge 2 commits intodevelopfrom
integration-test-update

Conversation

@vicheey
Copy link
Copy Markdown
Contributor

@vicheey vicheey commented Apr 14, 2026

What

Pre-create long-provisioning resources (MSK cluster, MQ broker) in the companion stack instead of creating them inline per-test. Also adds an API Gateway CloudWatch logging role to the companion stack.

Why

MSK clusters take 15-30 min to provision and MQ brokers take 15-20 min. When tests create these resources inline, a single test can take over 20 minutes. Pre-creating them in the companion stack lets tests reference existing resources via ARN parameters, reducing test execution time significantly.

The API Gateway logging role ensures test_api_settings tests don't fail with "CloudWatch Logs role ARN must be set in account settings" in environments where the role isn't pre-configured.

Changes

New file:

  • integration/setup/companion-stack.yaml — adds MSK cluster (kafka.t3.small), MQ broker (mq.t3.micro) with networking/secrets, API Gateway logging role + account resource. Uses CFN Conditions (CreateMskCluster, CreateMqBroker) so resources can be disabled in regions that don't support them.

Refactored templates (6 files):

  • 4 MSK templates — removed inline AWS::MSK::Cluster, added PreCreatedMskClusterArn parameter
  • 2 MQ templates — removed inline broker/VPC/subnet/security group/secret, added PreCreatedMqBrokerArn + PreCreatedMqBrokerSecretArn parameters

Updated expected outputs (6 files):

  • Removed expected resources for inline MSK/MQ that no longer exist in templates

Updated test code (6 files):

  • conftest.py — pass companion stack outputs as parameters to test stacks; region-aware disabling of MSK/MQ parameters
  • test_function_with_msk.py, test_function_with_mq.py — simplified to use companion stack ARNs instead of creating resources inline
  • helpers/base_test.py — support parameters in create_and_verify. Also moves output_dir to /tmp/ for compatibility with read-only filesystem environments. Existing test reporting is unaffected since output_dir is only used for intermediate CFN changeset files, not test reports.
  • helpers/resource.py — add get_s3_uri helper
  • helpers/stack.py — pass parameters through create_or_update to _deploy_stack

Risk

Low. The only difference is where the underlying MSK/MQ resources come from (companion stack parameter vs inline creation). Existing test workflows continue to work because the companion stack is already exit.

Examples?

Please reach out in the comments if you want to add an example. Examples will be
added to sam init through aws/aws-sam-cli-app-templates.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@vicheey vicheey requested a review from a team as a code owner April 14, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant