Is this your first time submitting a feature request?
Describe the feature
Requesting more efficient resolution of macros during dbt parse time to reduce parse times.
In our (rather large) dbt project, inefficient macro namespace construction results in an estimated 73.07s of parse time overhead (~30% of parse time).
Context: During parse, dbt builds a MacroNamespace for every node so that {{ ... }} rendering can look up macros (dbt.run_query, my_pkg.my_macro, etc.). Today the namespace is rebuilt from scratch per node. Every macro in every package is wrapped in a fresh MacroGenerator, the package-level dict is constructed, and the global/dbt namespace is composed. On a 19k-path project this is roughly N×M work where N is "nodes parsed" and M is "macros visible to that node." The same work is repeated thousands of times because the macro graph itself is invariant for the duration of a parse.
Describe alternatives you've considered
- Switch to dbt-fusion: we'll be doing this locally but it's not robust enough for prod usage yet.
- Optimize something else: we're also looking into other options, but this is the lowest hanging fruit based on our profiling analysis.
Who will this benefit?
All users of dbt-core. Benefit will be larger for those with larger projects -- specifically those using many macros or with many models. That said, I tried this out and saw perf improvements even for a small project with a few hundred models when many macros get pulled in by several dbt packages.
Are you interested in contributing this feature?
Yes, just need review of the idea and code contributions.
Anything else?
Proposed implementation: lazy-load macro namespaces by restructuring the namespace into two tiers:
- a per-
Manifest NamespaceTemplate that is computed once and captures the package layout + which macros are visible from where, and
- a per-node
LazyMacroNamespace that defers MacroGenerator construction until a macro is actually accessed during rendering.
Most macros are never touched by most nodes. So the lazy path makes the per-node cost proportional to used macros rather than visible macros.
Details:
Manifest.get_namespace_template(root_package, search_package, internal_packages) computes a NamespaceTemplate that holds, per package, the dict of (macro_name → unique_id) visible from that root/search pair. This is pure metadata — no MacroGenerator instances, no node binding. It's memoized on the Manifest, so it's built at most once per (root, search, internals) tuple per Manifest instance.LazyMacroNamespace(manifest, template, node, ...) is a MacroNamespace subclass that wraps the template and constructs MacroGenerator instances on first access via a MacroDictProxy. This is a dict subclass whose magic methods materialize the generator on demand and cache it.- Unit-test override compatibility:
generate_runtime_unit_test_context mutates the namespace to inject test overrides, but the lazy package view is read-only. So the override path materializes affected packages to a plain dict before mutating.
Example implementation is here.
Is this your first time submitting a feature request?
Describe the feature
Requesting more efficient resolution of macros during dbt parse time to reduce parse times.
In our (rather large) dbt project, inefficient macro namespace construction results in an estimated 73.07s of parse time overhead (~30% of parse time).
Context: During parse, dbt builds a
MacroNamespacefor every node so that{{ ... }}rendering can look up macros (dbt.run_query,my_pkg.my_macro, etc.). Today the namespace is rebuilt from scratch per node. Every macro in every package is wrapped in a freshMacroGenerator, the package-level dict is constructed, and the global/dbt namespace is composed. On a 19k-path project this is roughly N×M work where N is "nodes parsed" and M is "macros visible to that node." The same work is repeated thousands of times because the macro graph itself is invariant for the duration of a parse.Describe alternatives you've considered
Who will this benefit?
All users of dbt-core. Benefit will be larger for those with larger projects -- specifically those using many macros or with many models. That said, I tried this out and saw perf improvements even for a small project with a few hundred models when many macros get pulled in by several dbt packages.
Are you interested in contributing this feature?
Yes, just need review of the idea and code contributions.
Anything else?
Proposed implementation: lazy-load macro namespaces by restructuring the namespace into two tiers:
ManifestNamespaceTemplatethat is computed once and captures the package layout + which macros are visible from where, andLazyMacroNamespacethat defersMacroGeneratorconstruction until a macro is actually accessed during rendering.Most macros are never touched by most nodes. So the lazy path makes the per-node cost proportional to used macros rather than visible macros.
Details:
Manifest.get_namespace_template(root_package, search_package, internal_packages)computes aNamespaceTemplatethat holds, per package, the dict of(macro_name → unique_id)visible from that root/search pair. This is pure metadata — noMacroGeneratorinstances, no node binding. It's memoized on theManifest, so it's built at most once per (root, search, internals) tuple per Manifest instance.LazyMacroNamespace(manifest, template, node, ...)is aMacroNamespacesubclass that wraps the template and constructsMacroGeneratorinstances on first access via aMacroDictProxy. This is a dict subclass whose magic methods materialize the generator on demand and cache it.generate_runtime_unit_test_contextmutates the namespace to inject test overrides, but the lazy package view is read-only. So the override path materializes affected packages to a plain dict before mutating.Example implementation is here.