Skip to content

Use Python to combine verification output#16

Open
yrasool wants to merge 2 commits into
lanl:mainfrom
yrasool:codex/hard-bug-fix
Open

Use Python to combine verification output#16
yrasool wants to merge 2 commits into
lanl:mainfrom
yrasool:codex/hard-bug-fix

Conversation

@yrasool

@yrasool yrasool commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Change

find_last_output used cat to merge rank CSV files. This now uses Python file I/O, with the existing output order unchanged.

Checks

  • python tools/test_verify_output.py
  • ruff check tools
  • Python compile checks
  • codespell on changed files
  • local Sphinx build
  • GitHub documentation CI
  • git diff --check

CMake and C++ unit tests were not run locally because CMake and FleCSI are not installed here.

Closes #15

@yrasool yrasool changed the title Make verify output work on Windows Remove Unix-only cat from verify output Jul 10, 2026
@yrasool yrasool changed the title Remove Unix-only cat from verify output Use Python to combine verification output Jul 10, 2026
@yrasool
yrasool marked this pull request as ready for review July 10, 2026 01:21

@AndresYague AndresYague left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simple MR that makes the code more portable as it does not require that the system has cat. The code will be slower than running cat but not by a significant enough margin.

It may be useful to provide a buffer length of a few MB to shutil.copyfileobj to handle the 3D case output a bit faster.

@yrasool

yrasool commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Simple MR that makes the code more portable as it does not require that the system has cat. The code will be slower than running cat but not by a significant enough margin.

It may be useful to provide a buffer length of a few MB to shutil.copyfileobj to handle the 3D case output a bit faster.

Thanks, I added a 1 MiB buffer. In a local 128 MiB 3D-shaped merge, it was 1.34x faster than a 64 KiB buffer. The output test and docs check pass.

@AndresYague

Copy link
Copy Markdown
Contributor

Thanks, I added a 1 MiB buffer. In a local 128 MiB 3D-shaped merge, it was 1.34x faster than a 64 KiB buffer. The output test and docs check pass.

This all looks good to me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use Python to combine verification output

2 participants