Repository navigation
ccpp-capgen v1: a unified successor to ccpp-prebuild and ccpp-capgen v0 #759
Description
Description
Note. ccpp-capgen v1 was previously called ccpp-capgen-ng. Some of the supporting documents attached below (until 2026-06-28) use this terminology.
ccpp-capgen v1 is a next-generation code generator for the CCPP framework. It is not a criticism of the existing tools — both ccpp-prebuild and ccpp-capgen v0 have served their respective communities well and embody years of careful design. This project attempts to synthesize the strengths of both: the performance of ccpp-prebuild and the metadata richness and feature set of ccpp-capgen v0, while also taking advantage of relaxed code-generator requirements that have emerged over time and lessons learned from operating both tools in development, research, and production. It also aims to resolve several outstanding issues that have been blocking progress toward a long-discussed unification of the two code generators.
Supporting documents — including a redesign analysis, the LLM prompt used to bootstrap development, developer and user documentation, and a migration guide — will be provided as part of the draft pull request.
The code was developed with the assistance of an LLM (Claude, by Anthropic) as a coding partner. Where possible, existing code was reused and repackaged.
Testing
Python docstring tests, unit tests
Significantly expanded test coverage with Python docstring tests and 1500-ish unit tests. Run in CI, all pass.
End-to-end tests
The following tests and features are currently supported by ccpp-capgen v1. They run successfullly in CI and on my development boxes.
- Former ccpp-capgen v0 tests:
- advection
- advection_auto_clone (support legacy constituents auto-cloning from metadata)
- capgen_ng
- ddthost
- nested_suite
- var_compat
- Former ccpp-prebuild tests:
- chunked_data
-
blocked-datanot supported in capgen or capgen-ng - opt_arg
- Newly added features/tests:
- multi-instances
- multi-instances + constituents
- suite_allocate (exercises suite-owned variables with various types of runtime-known dimensions)
- constituents_dim (suite- and host-owned variables with dimensions from constituents objects)
Host model integration
- CCPP-SCM all suites compile and all tests run, results are b4b identical for all except a few tests that are also problematic in develop
- CAM-SIMA Python unit tests pass, system tests pass with Intel and GNU
- https://github.com/climbfuji/CAM-SIMA/tree/feature/capgen-ng
- Feature/capgen ng climbfuji/CAM-SIMA#1
- Feature/capgen ng climbfuji/atmospheric_physics#1
- All regression tests (
test/system) pass with both GNU and Intel, and with b4b-identical results except for two tests that also fail ondevelopment(SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_intel.cam-outfrq_se_cslam_multitape,SMS_Ln9.ne3pg3_ne3pg3_mg37.FADIAB.derecho_gnu.cam-outfrq_se_cslam)
- UFS all regression tests pass, answers are bit-for-bit identical
- Updates for ccpp-capgen v1 ufs-community/ufs-weather-model#3288
- Updates for ccpp-capgen v1 NOAA-EMC/ufsatm#1125
- Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1 #762
- Updates for ccpp-capgen v1 ufs-community/ccpp-physics#378
- Updates for ccpp-capgen v1 NOAA-GFDL/GFDL_atmos_cubed_sphere#419
- Updates for ccpp-capgen v1 NOAA-EMC/CMEPS#173
- NEPTUNE complete
- Pull requests merged in NRL Enterprise GitHub
- All 300+ tests pass with the three mandatory compilers (Intel LLVM, GCC, LLVM native)
- Regular, mid- and high-altitude physics covered (including proprietary NRL physics code)
Issues
A list of issues that will be addressed with this issue will be provided in the associated pull request.
Addressing this issue will also address/close the following issues if/when the participating host models switch to capgen-ng:
- Rethinking horizontal dimensions #760
- Weird Capgen fortran parsing error? #757
- Bug in capgen (test/capgen_test) ? surface_air_pressure outvar in ddt_suite #755
- Document --kind-type in ccpp_capgen.py CLI/help, and consider reversing the syntax for external kind modules #754
- Remove unused metadata attribute state_variable #753
- capgen: Add capability and test for passing mpi communicator as a flat field #751
- capgen: case-sensitivity when comparing metadata to Fortran #746
- Bug in
scripts/file_utils.py/move_modified_filesroutine #744 - Unallocated fields passed into Group Caps #690
- Capgen performance when using many suites #688
- Add support for MPI_Comm (mpi08) object in capgen #686
- Transition UFS weather model to Capgen #682
- Transition CCPP-SCM to Capgen #680
- Transition NEPTUNE to capgen #679
- Enhanced/Expanded Testing #595
- capgen's Fortran parser does not allow array bounds which are not Fortran identifiers #588
- Add "number of instances" dimension to module-level variables to support multi-instance usage #561
- capgen: check handling of type attribute for ddts #516
- Fortran/metadata comparison incorrectly reporting bug when comparing Fortran variable declarations and metadata for derived types #360
- CCPP Framework to provide API calls for suite inspection by host #380
To check/may not be implemented yet:
- capgen Fortran parser does not accept internal DDTs as part of a registered DDT #749
- capgen: Invalid character declaration, 'character(len=nf_file_length) ' #748
- capgen: dimension specifier foo(N) is not the same as foo(1:N) in Fortran code #747
- Errors when integer variable is output from one phase and used as an interstitial dimension in another #681
Working towards:
Supporting Materials
See doc/ folder in the feature/capgen-v1 - documents evolved over time (commit history).
The documents were produced by Claude during development and most have them have not been formally reviewed. They are shared for transparency and to give context for the design decisions made. They should be treated as starting points rather than authoritative references.
An update of the CCPP technical documentation is made in NCAR/ccpp-doc#81.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
bbakernoaa commentedon May 26, 2026
@climbfuji I'm really interested to get this moving forward. How can I help move this forward?
climbfuji commentedon May 27, 2026
@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.
To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.
bbakernoaa commentedon May 27, 2026
@climbfuji Thanks. I would really like to get this moving forward as fast as possible. @dustinswales how can I help?
climbfuji commentedon May 27, 2026
Realized that Dustin is out of the office this week. I suggest we work with EPIC to prioritize the metadata-fortran bug fixes this week and early next week start working on dropping capgen-ng in the UFS.
@ligiabernardet FYI
[-][PLACEHOLDER] ccpp-capgen-ng: a unified successor to ccpp-prebuild and ccpp-capgen[/-][+]ccpp-capgen-ng: a unified successor to ccpp-prebuild and ccpp-capgen[/+][-]ccpp-capgen-ng: a unified successor to ccpp-prebuild and ccpp-capgen[/-][+]ccpp-capgen v1: a unified successor to ccpp-prebuild and ccpp-capgen v0[/+]