[BUG] Max depth reached error when hot reloading via sphinx (e.g. with esbonio)

[MaximilianSoerenPollak]

There currently seems to be a bug where sphinx errors due to this extension as it reaches the max depth recursion.
Error:

ERROR: sphinx-build failed
Traceback (most recent call last):
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/sphinx/events.py", line 404, in emit
    results.append(listener.handler(self.app, *args))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/main.py", line 19, in write
    write_needscfg_file(app, app.config)
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 307, in write_needscfg_file
    safe_value = get_safe_config(value, f"needs.{config_name}", outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 192, in get_safe_config
    safe_value = get_safe_config(item, item_path, outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 183, in get_safe_config
    safe_value = get_safe_config(value, item_path, outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 183, in get_safe_config
    safe_value = get_safe_config(value, item_path, outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 192, in get_safe_config
    safe_value = get_safe_config(item, item_path, outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 183, in get_safe_config
    safe_value = get_safe_config(value, item_path, outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 183, in get_safe_config
    safe_value = get_safe_config(value, item_path, outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...

This continues until:

  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 183, in get_safe_config
    safe_value = get_safe_config(value, item_path, outpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/builder.py", line 95, in get_safe_config
    if matches_path_pattern(path, pattern):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/needs_config_writer/utils.py", line 76, in matches_path_pattern
    return re.fullmatch(pattern_regex, config_path) is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/cache/bazel/05454f223d7eceee02200e9d517999d9/external/rules_python++python+python_3_12_x86_64-unknown-linux-gnu/lib/python3.12/re/__init__.py", line 172, in fullmatch
    return _compile(pattern, flags).fullmatch(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vscode/.vscode-server/extensions/swyddfa.esbonio-1.0.0/bundled/libs/esbonio/sphinx_agent/handlers/__init__.py", line 168, in build_sphinx_app
    self.app.build()
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/sphinx/application.py", line 426, in build
    self.builder.build_update()
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 375, in build_update
    self.build(
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 403, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 508, in read
    self.events.emit('env-before-read-docs', self.env, docnames)
  File "/workspaces/docs-as-code/.venv_docs/lib/python3.12/site-packages/sphinx/events.py", line 415, in emit
    raise ExtensionError(
sphinx.errors.ExtensionError: Handler <function write at 0x74767523e200> for event 'env-before-read-docs' threw an exception (exception: maximum recursion depth exceeded)

Steps to reproduce:

• Cloning repo: https://github.com/eclipse-score/docs-as-code
• Opening in Devcontainer (so it installs Esbonio extension and server etc)
• Make sure .venv_docs exists and is selected as python interpreter
• In the output on the bottom select 'Esbonio' to see it's output
• Open any RST file
  You now hope fully should see a non error sphinx build accin to this:

Running Sphinx v8.2.3
loading translations [en]... done
calculate directory_hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 within 9.822845458984375e-05 seconds.
Read in collections ...
Clean collections ...
Executing collections ...
myst v5.0.0: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions={'colon_fence'}, disable_syntax=[], all_links_external=False, links_external_new_tab=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, fence_as_directive=set(), number_code_blocks=[], title_to_header=False, heading_anchors=0, heading_slug_func=None, html_meta={}, footnote_sort=True, footnote_transition=True, words_per_minute=200, substitutions={}, linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', enable_checkboxes=False, suppress_warnings=[], highlight_code_blocks=True)
building [mo]: targets for 0 po files that are out of date
writing output...
building [html]: targets for 32 source files that are out of date
updating environment: [new config] 32 added, 0 changed, 0 removed
Path 'needs.flow_configs.score_config' matches pattern 'needs_flow_configs.score_config' (prefix='!include ', suffix=None)
Relativizing embedded string path at 'needs.flow_configs.score_config': !include /var/cache/bazel/05454f223d7eceee02200e9d517999d9/execroot/_main/bazel-out/k8-fastbuild/bin/ide_support.runfiles/_main/src/extensions/score_layout/assets/puml-theme-score.puml -> !include ../src/extensions/score_layout/assets/puml-theme-score.puml
Path 'needs.external_needs[0].json_path' matches pattern 'needs_external_needs[*].json_path' (prefix=None, suffix=None)
Relativizing path at 'needs.external_needs[0].json_path': /workspaces/docs-as-code/bazel-bin/ide_support.runfiles/score_process+/needs_json/_build/needs/needs.json -> ../bazel-bin/external/score_process+/needs_json/_build/needs/needs.json
Merged TOML configuration from '/var/cache/bazel/05454f223d7eceee02200e9d517999d9/execroot/_main/bazel-out/k8-fastbuild/bin/ide_support.runfiles/_main/src/extensions/score_sync_toml/shared.toml'
Needs configuration written to '/workspaces/docs-as-code/docs/ubproject.toml'
reading sources...
Copying static files for sphinx-data-viewer support
Copying static files for sphinx-needs datatables support
Copying static style files for sphinx-needs
WARNING: the sphinx_agent.handlers.webview extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit
WARNING: doing serial read
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
Schema validation completed with 0 warning(s) in 0.166 seconds. Validated 8240 needs/s.
preparing documents... done
Writing evaluated template result to /workspaces/docs-as-code/_build/_static/basic.css
Writing evaluated template result to /workspaces/docs-as-code/_build/_static/language_data.js
Writing evaluated template result to /workspaces/docs-as-code/_build/_static/documentation_options.js
copying static files... done
copying extra files... done
copying assets... done
writing output...

genindex generating indices... done
search writing additional pages... done
copying images...
dumping search index in English (code: en)... done
dumping object inventory... done
Needs successfully exported
Final clean of collections ...
build succeeded, 2 warnings.

The HTML pages are in _build.

• Now change something in the rst file like removing some word or adding something
• Save to force esbonio to reload
• The error at the begining now should pop up.

This seems to be an issue with the Document update event in sphinx, somehow the extension can't handle something in there.

Happy to answer any questions or provide more info that is needed.

[arnoox]

Recursion Issue Analysis: get_safe_config Function

Summary

The get_safe_config function in builder.py is vulnerable to RecursionError when processing Sphinx configuration objects that contain:

1. Circular references (objects that reference each other)
2. Deeply nested structures (nesting depth > Python's recursion limit)
3. Self-referential data (objects that contain references to themselves)

All three scenarios have been successfully reproduced with test cases.

---

1. How the Builder Works - Design Overview

Purpose

The builder extracts Sphinx-Needs configuration from a Sphinx Config object and writes it to a ubproject.toml file in a TOML-serializable format.

Key Components

A. write_needscfg_file() - Main Entry Point

• Called during Sphinx build process
• Iterates over all config attributes starting with needs_
• Processes each config value to make it TOML-compatible

B. get_safe_config() - Recursive Config Processor

Location: builder.py:38-206

Responsibilities:

1. Recursively walk through config objects (dicts, lists, tuples, sets)
2. Filter out non-serializable types (None, functions, classes, etc.)
3. Convert special types (Path objects → strings)
4. Relativize absolute paths based on configuration rules
5. Return TOML-compatible data structures

Recursive Pattern:

def get_safe_config(obj, path="", outpath=None):
    # Base cases (primitives, Path objects)
    if isinstance(obj, (str, int, float, bool, ...)):
        return obj

    # Recursive case: dictionaries
    if isinstance(obj, dict):
        result = {}
        for key, value in obj.items():
            result[key] = get_safe_config(value, ...)  # RECURSION
        return result

    # Recursive case: lists/tuples/sets
    if isinstance(obj, (list, tuple, set)):
        items = []
        for item in obj:
            items.append(get_safe_config(item, ...))  # RECURSION
        return items

C. sort_for_reproducibility() - Recursive Sorter

Location: builder.py:212-280

• Also recursive, but processes already-filtered data from get_safe_config()
• Less likely to hit recursion limits since it works on clean data

---

2. Root Cause Analysis - What Can Go Wrong?

Problem: Naive Recursion Without Cycle Detection

The function uses unbounded recursion without:

• ✗ Cycle/circular reference detection
• ✗ Visited object tracking (e.g., using id())
• ✗ Depth limiting
• ✗ Iterative alternative

Scenario A: Circular References Between Objects

Example Sphinx Config:

# In Sphinx conf.py or extension code
needs_workflow_state_a = {"name": "StateA"}
needs_workflow_state_b = {"name": "StateB", "next": needs_workflow_state_a}
needs_workflow_state_a["next"] = needs_workflow_state_b  # Circular!

What Happens:

1. get_safe_config(state_a) → processes dict
2. Encounters state_a["next"] → calls get_safe_config(state_b)
3. Processes state_b["next"] → calls get_safe_config(state_a) ← LOOP!
4. Infinite recursion → RecursionError

Test Result: ✓ Reproduced in test_recursion.py

---

Scenario B: Deeply Nested Structures

Example Sphinx Config:

# Deeply nested configuration (e.g., from automated generation)
needs_schema = {
    "level1": {
        "level2": {
            "level3": {
                # ... 1000+ levels deep
            }
        }
    }
}

What Happens:

• Python's default recursion limit: 1000 (sys.getrecursionlimit())
• Each dict level adds one recursion frame
• Structure depth > 1000 → RecursionError

Real-World Triggers:

• Auto-generated config from deeply nested JSON schemas
• Complex organizational hierarchies
• Nested template expansions
• Bazel/build system generated configs

Test Result: ✓ Reproduced in test_recursion.py with depth=1500

---

Scenario C: Self-Referential Data Structures

Example Sphinx Config:

# Object contains reference to itself
needs_custom_list = [1, 2, 3]
needs_custom_list.append(needs_custom_list)  # Self-reference!

# Or via dict
needs_custom_dict = {"key": "value"}
needs_custom_dict["self"] = needs_custom_dict

What Happens:

• When processing the list/dict, it encounters itself as a member
• Infinite recursion → RecursionError

Test Result: ✓ Reproduced in test_recursion.py

---

3. When Can This Happen in Practice?

High-Risk Scenarios

1. Sphinx Extensions with Complex State

   • Extensions that maintain workflow graphs
   • State machines with bidirectional links
   • Dependency graphs with cycles

2. Auto-Generated Configuration

   • JSON/YAML configs converted to Python dicts
   • Build systems (Bazel, Buck) generating deep structures
   • Schema-driven configs with deep nesting

3. User Plugins/Custom Code

   • Users adding custom needs_* config attributes
   • Monkey-patching or dynamic config modification
   • Config merging from multiple sources creating unintended cycles

4. Sphinx Config Object Itself

   • config object has many cross-references
   • Some Sphinx internals may create circular links
   • Extension interactions creating unexpected references

Example from Real World

# A Sphinx-Needs extension adding workflow support
class WorkflowState:
    def __init__(self, name):
        self.name = name
        self.transitions = []

    def add_transition(self, target_state):
        self.transitions.append(target_state)

# In conf.py
start = WorkflowState("start")
review = WorkflowState("review")
approved = WorkflowState("approved")

start.add_transition(review)
review.add_transition(approved)
approved.add_transition(review)  # Cycle: review ↔ approved

needs_workflow_states = {
    "start": start,
    "review": review,
    "approved": approved
}

When get_safe_config() processes needs_workflow_states, it will:

1. Convert WorkflowState objects to dicts (if not filtered)
2. Process transitions list
3. Encounter the cycle between review and approved
4. RecursionError

---

4. Reproduction Evidence

Test Results

============================================================
TESTING RECURSION SCENARIOS
============================================================

=== Testing circular reference ===
SUCCESS: RecursionError caught as expected

=== Testing deeply nested structure ===
Current recursion limit: 1000
  Depth: 0-900
SUCCESS: RecursionError caught as expected

=== Testing self-referential list ===
SUCCESS: RecursionError caught as expected

============================================================
SUMMARY
============================================================
✓ PASS: Circular reference
✓ PASS: Deeply nested
✓ PASS: Self-referential list

All three problematic scenarios successfully reproduce the RecursionError.

---

5. Current Behavior Under Error

When RecursionError occurs:

• Sphinx build fails completely
• User sees cryptic stack trace (potentially 1000+ lines)
• No output file is written
• No helpful error message explaining which config caused the issue
• No path information showing where the cycle/depth problem exists

---

6. Design Implications

Why Recursion Was Chosen (Likely Intent)

Advantages:

• ✓ Clean, readable code
• ✓ Naturally handles nested structures
• ✓ Easy to maintain path tracking (path parameter)
• ✓ Works well for "normal" config structures

Implicit Assumptions:

• Config structures are acyclic (no circular references)
• Config structures are moderately nested (< 1000 levels)
• Sphinx config object is well-behaved

Why These Assumptions Can Fail

1. Python allows circular references - no compile-time checks
2. Sphinx config is user-extensible - can contain arbitrary objects
3. Extensions can modify config - after initial setup
4. Dynamic config generation - from external sources (JSON, TOML, databases)

---

7. Problem Space Summary

Core Issue

Unbounded recursion on untrusted/unverified data structures

Impact

• Severity: High (build fails completely)
• Likelihood: Low-Medium (depends on config complexity)
• User Experience: Poor (cryptic error, no actionable message)

Technical Debt

The current implementation:

1. Trusts input data structure to be acyclic
2. Assumes reasonable nesting depth
3. Lacks defensive programming (cycle detection, depth limits)
4. Provides poor error diagnostics