Improve pytest parsing for beginner-friendly evaluation feedback

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item

test_program.py F                                                        [100%]

=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>

    def test_weekly_todo(monkeypatch, capsys):
        user_input = iter([
            "Laundry",
            "Groceries",
            "Gym",
            "Study",
            "Work",
            "Hike",
            "Rest",
        ])

        run_program_with(monkeypatch, user_input)

>       assert capsys.readouterr().out == (
            "Weekly to-do list for the 7 days of the week:\n"
            "Mon: Laundry\n"
            "Tues: Groceries\n"
            "Wed Gym\n"
            "Thur: Study\n"
            "Fri: Work\n"
            "Sat: Hike\n"
            "Sun: Rest\n"
        )
E       AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E         Skipping 34 identical leading characters in diff, use -v to show
E         - f the week:
E         + f the weeke:
E         ?           +
E           Mon: Laundry
E           Tues: Groceries...
E
E         ...Full output truncated (5 lines hidden), use '-vv' to show

test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
  state: 'failed',
  tests_run: 1,
  passed: 0,
  failed: 1,
  errors: 0,
  no_tests_collected: false,
  exit_code: 1,
  failure_details: [
    {
      test_case: 'test_weekly_todo',
      expected: '',
      received: '',
      error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
      rawout: '============================= test session starts ==============================\n' +
        'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
        'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
        'collected 1 item\n' +
        '\n' +
        'test_program.py F                                                        [100%]\n' +
        '\n' +
        '=================================== FAILURES ===================================\n' +
        '_______________________________ test_weekly_todo _______________________________\n' +
        '\n' +
        'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
        'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
        '\n' +
        '    def test_weekly_todo(monkeypatch, capsys):\n' +
        '        user_input = iter([\n' +
        '            "Laundry",\n' +
        '            "Groceries",\n' +
        '            "Gym",\n' +
        '            "Study",\n' +
        '            "Work",\n' +
        '            "Hike",\n' +
        '            "Rest",\n' +
        '        ])\n' +
        '    \n' +
        '        run_program_with(monkeypatch, user_input)\n' +
        '    \n' +
        '>       assert capsys.readouterr().out == (\n' +
        '            "Weekly to-do list for the 7 days of the week:\\n"\n' +
        '            "Mon: Laundry\\n"\n' +
        '            "Tues: Groceries\\n"\n' +
        '            "Wed Gym\\n"\n' +
        '            "Thur: Study\\n"\n' +
        '            "Fri: Work\\n"\n' +
        '            "Sat: Hike\\n"\n' +
        '            "Sun: Rest\\n"\n' +
        '        )\n' +
        "E       AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
        'E         \n' +
        'E         Skipping 34 identical leading characters in diff, use -v to show\n' +
        'E         - f the week:\n' +
        'E         + f the weeke:\n' +
        'E         ?           +\n' +
        'E           Mon: Laundry\n' +
        'E           Tues: Groceries...\n' +
        'E         \n' +
        "E         ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
        '\n' +
        'test_program.py:29: AssertionError\n' +
        '=========================== short test summary info ============================\n' +
        "FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
        '============================== 1 failed in 0.02s ===============================\n'
    }
  ],
  compilation_error: '',
  runtime_error: '',
  execution_time_exceeded: false,
  memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junit xml file that contains information about the tests that were run. if that is something you want to look into.