refactor(test): Pre-create MSK/MQ in companion stack for testing#3909
Open
refactor(test): Pre-create MSK/MQ in companion stack for testing#3909
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Pre-create long-provisioning resources (MSK cluster, MQ broker) in the companion stack instead of creating them inline per-test. Also adds an API Gateway CloudWatch logging role to the companion stack.
Why
MSK clusters take 15-30 min to provision and MQ brokers take 15-20 min. When tests create these resources inline, a single test can take over 20 minutes. Pre-creating them in the companion stack lets tests reference existing resources via ARN parameters, reducing test execution time significantly.
The API Gateway logging role ensures
test_api_settingstests don't fail with "CloudWatch Logs role ARN must be set in account settings" in environments where the role isn't pre-configured.Changes
New file:
integration/setup/companion-stack.yaml— adds MSK cluster (kafka.t3.small), MQ broker (mq.t3.micro) with networking/secrets, API Gateway logging role + account resource. Uses CFN Conditions (CreateMskCluster,CreateMqBroker) so resources can be disabled in regions that don't support them.Refactored templates (6 files):
AWS::MSK::Cluster, addedPreCreatedMskClusterArnparameterPreCreatedMqBrokerArn+PreCreatedMqBrokerSecretArnparametersUpdated expected outputs (6 files):
Updated test code (6 files):
conftest.py— pass companion stack outputs as parameters to test stacks; region-aware disabling of MSK/MQ parameterstest_function_with_msk.py,test_function_with_mq.py— simplified to use companion stack ARNs instead of creating resources inlinehelpers/base_test.py— supportparametersincreate_and_verify. Also movesoutput_dirto/tmp/for compatibility with read-only filesystem environments. Existing test reporting is unaffected sinceoutput_diris only used for intermediate CFN changeset files, not test reports.helpers/resource.py— addget_s3_urihelperhelpers/stack.py— pass parameters throughcreate_or_updateto_deploy_stackRisk
Low. The only difference is where the underlying MSK/MQ resources come from (companion stack parameter vs inline creation). Existing test workflows continue to work because the companion stack is already exit.
Examples?
Please reach out in the comments if you want to add an example. Examples will be
added to
sam initthrough aws/aws-sam-cli-app-templates.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.