Skip to content

log_parsers: read unittest and Django test-runner output - #168

Open
k21993 wants to merge 2 commits into
huggingface:mainfrom
k21993:feat/unittest-parser
Open

k21993 wants to merge 2 commits into
huggingface:mainfrom
k21993:feat/unittest-parser

Conversation

@k21993

@k21993 k21993 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Parse verbose unittest and Django TextTestRunner output in both the generation parser and standalone verifier.
  • Detect python -m unittest, manage.py test and Django runtests.py, adding the verbosity each runner needs to expose stable test identities.
  • Group subtests under their parent with the worst status winning, including named subtests and nested parameter values, and keep unittest count footers out of the pytest fallback.

Test plan

  • uv run --all-extras pytest -q (2151 passed, 6 skipped)
  • uv run ruff check .
  • uv run ruff format --check .
  • Run the focused parser, normalizer and verifier tests (155 passed)
  • Generate named-subtest output with the real TextTestRunner and parse it through both parser copies
  • Verify the new normalization and stderr tests fail against the pre-fix main branch

Out of scope

Closes #167

@k21993
k21993 marked this pull request as ready for review September 27, 2026 08:32

@adithya-s-k adithya-s-k left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Useful addition, but a real unittest case is still dropped by both parser copies: with self.subTest("edge case", i=1): self.assertEqual(1, 2).

TextTestRunner prints test_named (pkg.T.test_named) [edge case] (i=1) ... FAIL, plus the matching failure footer. Both parsers return {} because _NAME_RE rejects the message. That loses the failing parent and prevents F2P detection.

Please support named subtests (with and without parameters), retain the parent ID and worst-status behavior, and add tests using real TextTestRunner output in both copies. #164 is now merged, so the stderr dependency is resolved.

Python unittest and Django commands previously fell back to the pytest parser and produced no oracle. Add TextTestRunner parsing in both parser copies, runner detection, and the verbosity flags needed to expose test identities.

Group subtests under stable parent identities, accept nested parameter values, preserve the worst status, and keep unittest count footers out of the pytest fallback.

Closes huggingface#167
TextTestRunner includes the subtest message before its parameters. Match and discard that full suffix so both parser copies retain the stable parent identity, including when the message contains an ellipsis.
@k21993
k21993 force-pushed the feat/unittest-parser branch from a798cac to 34e42c9 Compare September 29, 2026 09:05
@k21993

k21993 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in 34e42c9. The regression test now generates the exact output with a real TextTestRunner and sends it through both parser copies:

  • [edge ... case] (i=1) covers a named subtest with parameters and an ellipsis in the message.
  • [message only] covers a named subtest without parameters.

Both collapse to pkg.NamedSubTests.test_named, with the existing worst-status behavior. The parser now splits on the final result separator and accepts the full subtest suffix instead of only a parenthesized parameter block.

I also rebased onto current main, so the merged #164 commit is no longer in this PR. Full suite: 2151 passed, 6 skipped; ruff clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unittest and Django logs produce no runtime oracle

2 participants