Skip to content

test(coverage): expand pytest coverage measurement beyond training/ directory #442

@WilliamBerryiii

Description

@WilliamBerryiii

Issue Description

Background

The root pyproject.toml coverage configuration only measures the training/ directory, which represents ~24% of the Python codebase. This means code exercised by tests in tests/inference/, data-management/tools/tests/, and any future evaluation/tests/ is not captured in Codecov reporting.

The codecov.yml pytest flag similarly limits paths to ["training/"], so even if coverage were generated for other domains, it would be filtered out.

Current overall Codecov coverage is 14.19%. OpenSSF Silver criterion regression_tests_added50 requires ≥50% statement coverage.

Current Configuration

pyproject.toml:

[tool.pytest.ini_options]
testpaths = ["tests", "training/tests", "data-management/tools/tests"]
addopts = ["-ra", "-m", "not e2e", "--strict-markers", "--strict-config",
           "--cov=training", "--cov-report=term-missing", "--cov-report=xml",
           "--junitxml=logs/pytest-results.xml"]

[tool.coverage.run]
source = ["training"]
branch = true
omit = ["training/**/conftest.py", "training/**/__init__.py"]

codecov.yml:

flag_management:
  individual_flags:
    - name: pytest
      paths: ["training/"]
      statuses:
        - type: patch
          informational: true

Impact

  • tests/inference/test_inference_common.py and tests/inference/test_inference_training.py run via testpaths = ["tests"] but coverage is NOT captured (not under source = ["training"])
  • data-management/tools/tests/ runs but coverage for data-management/tools/ is NOT captured
  • Any future evaluation/tests/ would similarly be invisible
  • The src/ directory contains only empty namespace stubs — it is NOT a coverage target

Suggested Fix

1. Expand pyproject.toml coverage sources

[tool.pytest.ini_options]
addopts = ["-ra", "-m", "not e2e", "--strict-markers", "--strict-config",
           "--cov=training", "--cov=evaluation", "--cov=tests",
           "--cov=data-management/tools",
           "--cov-report=term-missing", "--cov-report=xml",
           "--junitxml=logs/pytest-results.xml"]

[tool.coverage.run]
source = ["training", "evaluation", "tests", "data-management/tools"]
branch = true
omit = [
  "**/conftest.py",
  "**/__init__.py",
  "**/tests/**"
]

2. Update codecov.yml pytest flag paths

flag_management:
  individual_flags:
    - name: pytest
      paths: ["training/", "evaluation/", "tests/", "data-management/tools/"]

3. Validate

  • Run uv run pytest -v and verify coverage report includes all source domains
  • Confirm Codecov receives expanded coverage data after CI run
  • Verify no regressions in existing coverage numbers

Acceptance Criteria

  • pyproject.toml [tool.coverage.run].source includes all Python source domains
  • pyproject.toml addopts includes --cov flags for all source domains
  • codecov.yml pytest flag paths match expanded source list
  • tests/inference/ hypothesis tests contribute to coverage reports
  • CI coverage upload succeeds with expanded configuration
  • No regression in existing training/ coverage numbers

Related Issues

Context

This is a config-only change — no new test code is needed. This is the highest impact-to-effort ratio item for improving overall coverage numbers toward the OpenSSF Silver ≥50% threshold.

OpenSSF IDs: regression_tests_added50

Metadata

Metadata

Labels

ci/cdCI/CD pipeline and automationenhancementNew feature or improvement requesttestingTesting-related issues

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions