GH-49537: [C++][FlightRPC] Windows CI to Support ODBC DLL & MSI Signing#49603
GH-49537: [C++][FlightRPC] Windows CI to Support ODBC DLL & MSI Signing#49603alinaliBQ wants to merge 4 commits into
Conversation
| - name: Upload the artifacts to GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release upload ${GITHUB_REF_NAME} \ | ||
| --clobber \ | ||
| arrow_flight_sql_odbc_unsigned.dll |
There was a problem hiding this comment.
Since both DLL and MSI need to be signed and unsigned DLL is harder to catch, uploading as arrow_flight_sql_odbc_unsigned.dll to make it clear on GitHub release if the DLL is unsigned.
|
I did an empty commit (a522393) and got this error: The same implementation worked yesterday (see https://github.com/apache/arrow/actions/runs/23622018872/job/68803175375). Seems that GitHub might have updated runner, I will look into this. |
This issue should be resolved now by commit 95bc75b |
alinaliBQ
left a comment
There was a problem hiding this comment.
Addressed comments. Please have another look, thanks!
amoeba
left a comment
There was a problem hiding this comment.
Thanks @alinaliBQ.
@raulcd do you want to have a look? Once we merge, I'll test the whole flow with the script (PR to come) that does the signing so we'll have more chances to fix any issues.
raulcd
left a comment
There was a problem hiding this comment.
Thanks for the work @alinaliBQ ! I am unsure this is ready yet, sorry it took me a couple days for review, it was Easter and had some family around. My main concern is around, the tag hasn't been updated to match the one on the packaging linux jobs, currently any tag will trigger the job and we only want to do it for RC's. The worlflow dispatch is still around and I thought this was not necessary with the tag.
@amoeba I am about to do the feature freeze, I think this is going to miss 24.0.0, is that ok?
| schedule: | ||
| - cron: | | ||
| 0 0 * * * | ||
| workflow_dispatch: |
There was a problem hiding this comment.
I am not sure I understand this. Didn't we agreed to use the tag for the RC? Why is the workflow dispatch needed and referenced below?
There was a problem hiding this comment.
The flow is a bit complicated, we need to trigger some jobs when the RC is created but the odbc-msvc-upload-msi step needs to be triggered manually so we use workflow_dispatch. The sequence is,
- Release manager tags release
- cpp_extra.yml jobs
odbc-msvc-upload-dllandodbc-dll-releaseare run automatically, unsigned DLL is added to release - Release manager locally runs script (TBD) that downloads, signs (w/ jsign), and uploads the signed DLL to the release
- Release manager triggers
odbc-msvc-upload-msiviaworkflow_dispatchwhich builds an unsigned MSI with a signed DLL inside and adds it to the release - Release manager locally runs script that downloads signs (w/ jsign), and uploads the signed MSI to the releasee
There was a problem hiding this comment.
I see, I did not understood the process. As the process isn't entirely automated and it requires several manual steps, can we document it on the release page?
https://github.com/apache/arrow/blob/main/docs/source/developers/release.rst
There was a problem hiding this comment.
@amoeba Could you kindly help with the documentation on release page?
There was a problem hiding this comment.
Absolutely. I'll file an issue and fill it in as we finish the final PRs.
There was a problem hiding this comment.
hi @raulcd, @amoeba has helped to document the process on release page:
arrow/docs/source/developers/release.rst
Lines 267 to 276 in 62cdda9
|
Thanks for taking a look @raulcd. It's okay if this doesn't make it for 24 but if it's at all possible that'd be great. Re: the jobs running on any tag, I'm not seeing what you see, could you point out which job needs updating?
I could see making the last one ( |
No worries @raulcd, hope you had a good Easter. Since having a |
53e361b to
c0d1bed
Compare
|
Rebased and resolved merge conflict |
c0d1bed to
ef617ad
Compare
|
cc @justing-bq Please help me with code review comments on this PR going forward |
0d6d212 to
cffb86d
Compare
cffb86d to
808dedb
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates the C++ “Extra” GitHub Actions workflow to support a two-step Windows ODBC release process where the DLL is built/uploaded unsigned first, then a signed DLL is injected into the MSI build so the resulting MSI contains the signed DLL.
Changes:
- Added
workflow_dispatchinput and two new Windows ODBC release jobs: upload unsigned DLL, and build/upload unsigned MSI using a signed DLL from the release. - Refactored the Windows ODBC build steps into a new composite action (
.github/actions/odbc-windows) to reduce duplication. - Removed the previous
odbc-releasejob and updated reporting dependencies accordingly.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
.github/workflows/cpp_extra.yml |
Adds manual release-step controls and new ODBC DLL/MSI release jobs; refactors ODBC Windows build usage to composite action; removes old ODBC release job. |
.github/actions/odbc-windows/action.yml |
Introduces a composite action encapsulating the Windows ODBC build setup (ccache, vcpkg, build, driver registration). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| odbc-msvc-upload-dll: | ||
| needs: check-labels | ||
| name: ODBC Windows Upload Unsigned DLL | ||
| runs-on: windows-2022 | ||
| if: >- | ||
| ${{ | ||
| startsWith(github.ref_name, 'apache-arrow-') && | ||
| contains(github.ref_name, '-rc') && | ||
| !inputs.odbc_release_step | ||
| }} | ||
| timeout-minutes: 240 | ||
| permissions: | ||
| packages: write | ||
| env: *odbc_msvc_env | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Build ODBC Windows | ||
| uses: ./.github/actions/odbc-windows | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| - name: Rename Unsigned ODBC DLL | ||
| run: | | ||
| Rename-Item ` | ||
| -Path build/cpp/${{ env.ARROW_BUILD_TYPE }}/arrow_flight_sql_odbc.dll ` | ||
| -NewName arrow_flight_sql_odbc_unsigned.dll | ||
| - name: Upload ODBC DLL to the job | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: flight-sql-odbc-dll | ||
| path: build/cpp/${{ env.ARROW_BUILD_TYPE }}/arrow_flight_sql_odbc_unsigned.dll | ||
| if-no-files-found: error |
There was a problem hiding this comment.
Why do we need this?
Can we upload the DLL in the odbc-msvc job instead?
There was a problem hiding this comment.
I kept the logic separate since odbc-msvc runs Arrow ODBC tests. I see other Github upload workflows like package_linux.yml don't run Arrow tests, so I followed that pattern.
| odbc-dll-release: | ||
| needs: odbc-msvc-upload-dll | ||
| name: Upload Unsigned ODBC DLL | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| # Upload to GitHub Release | ||
| contents: write | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Download the artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: flight-sql-odbc-dll | ||
| - name: Wait for creating GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| dev/release/utils-watch-gh-workflow.sh \ | ||
| ${GITHUB_REF_NAME} \ | ||
| release_candidate.yml | ||
| - name: Upload the artifacts to GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release upload ${GITHUB_REF_NAME} \ | ||
| --clobber \ | ||
| arrow_flight_sql_odbc_unsigned.dll | ||
|
|
||
| odbc-msvc-upload-msi: | ||
| needs: check-labels | ||
| name: ODBC Windows Build & Upload Unsigned MSI | ||
| runs-on: windows-2022 | ||
| if: inputs.odbc_release_step | ||
| timeout-minutes: 240 | ||
| permissions: | ||
| # Upload to GitHub Release | ||
| contents: write | ||
| packages: write | ||
| env: *odbc_msvc_env | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Download signed ODBC DLL | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release download $env:GITHUB_REF_NAME ` | ||
| --pattern arrow_flight_sql_odbc.dll ` | ||
| --clobber | ||
| - name: Build ODBC Windows | ||
| uses: ./.github/actions/odbc-windows | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| - name: Replace signed DLL with unsigned DLL | ||
| run: | | ||
| Move-Item ` | ||
| -Path ./arrow_flight_sql_odbc.dll ` | ||
| -Destination build/cpp/${{ env.ARROW_BUILD_TYPE }}/arrow_flight_sql_odbc.dll ` | ||
| -Force | ||
| - name: Install WiX Toolset | ||
| shell: pwsh | ||
| run: | | ||
| Invoke-WebRequest -Uri https://github.com/wixtoolset/wix/releases/download/v6.0.0/wix-cli-x64.msi -OutFile wix-cli-x64.msi | ||
| Start-Process -FilePath wix-cli-x64.msi -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait | ||
| echo "C:\Program Files\WiX Toolset v6.0\bin\" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append | ||
| - name: Build MSI ODBC installer | ||
| shell: pwsh | ||
| run: | | ||
| # Verify WiX version | ||
| wix --version | ||
| cd build/cpp | ||
| cpack | ||
| - name: Upload the artifacts to GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| cd build/cpp | ||
| gh release upload $env:GITHUB_REF_NAME ` | ||
| --clobber ` | ||
| Apache-Arrow-Flight-SQL-ODBC-*-win64.msi |
There was a problem hiding this comment.
How about moving these jobs to separated workflow (package_odbc.yml?) for easy to understand?
There was a problem hiding this comment.
Yes I think this is a good idea. cc @justing-bq we can look into this.
| odbc-dll-release: | ||
| needs: odbc-msvc-upload-dll | ||
| name: Upload Unsigned ODBC DLL | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| # Upload to GitHub Release | ||
| contents: write | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Download the artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: flight-sql-odbc-dll | ||
| - name: Wait for creating GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| dev/release/utils-watch-gh-workflow.sh \ | ||
| ${GITHUB_REF_NAME} \ | ||
| release_candidate.yml | ||
| - name: Upload the artifacts to GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release upload ${GITHUB_REF_NAME} \ | ||
| --clobber \ | ||
| arrow_flight_sql_odbc_unsigned.dll |
There was a problem hiding this comment.
How about uploading the unsigned DLL to GitHub Release in the odbc-msvc job?
We can use if: startsWith(github.ref_name, 'apache-arrow-') && contains(github.ref_name, '-rc') for a step that uploads the unsigned DLL to GitHub Release.
There was a problem hiding this comment.
I think odbc-dll-release can stay as a separate Ubuntu job since dev/release/utils-watch-gh-workflow.sh only runs on ubuntu-latest. Before I tried running dev/release/utils-watch-gh-workflow.sh on windows-2022 and it didn't work.
We can move the uploading logic to a separate file
808dedb to
2632c7d
Compare
* Add draft code for CI A and CI B Attempt workflow dispatch Only ODBC Windows original workflow should run. Later need to add `github.event_name != 'workflow_dispatch' ||` to all existing workflows after uncomment Use `GITHUB_REF_NAME` directly via push Add `workflow_dispatch` definitions Add `ODBC Windows Upload DLL` Use common ODBC Windows environment variables Use ODBC as composite action Create cpp_odbc.yml Initial draft temp disable test step Temp disable non-ODBC Windows workflows * Clean Up Code * Remove comments * Fix Installer path for MSI Trigger CI Fix issue with `secrets.GITHUB_TOKEN` Change `odbc_msi_upload` to boolean input Change `odbc-msvc-upload-dll` to be triggered via rc tag and can be invoked manually Work on Bryce's code review comments
2632c7d to
0f3fefd
Compare
alinaliBQ
left a comment
There was a problem hiding this comment.
Thanks for the review kou. I will work with Justin to address the comments
| exit 1 | ||
|
|
||
| odbc-msvc-upload-dll: |
There was a problem hiding this comment.
To address: #49603 comment
| exit 1 | |
| odbc-msvc-upload-dll: | |
| exit 1 | |
| # GH-49537 CPack only packages from a single build directory, so the build cannot be reused and we need to rebuild MSI in separate workflow. | |
| odbc-msvc-upload-dll: |
@justing-bq feel free to reword my draft comment.
| odbc-dll-release: | ||
| needs: odbc-msvc-upload-dll | ||
| name: Upload Unsigned ODBC DLL | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| # Upload to GitHub Release | ||
| contents: write | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Download the artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: flight-sql-odbc-dll | ||
| - name: Wait for creating GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| dev/release/utils-watch-gh-workflow.sh \ | ||
| ${GITHUB_REF_NAME} \ | ||
| release_candidate.yml | ||
| - name: Upload the artifacts to GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release upload ${GITHUB_REF_NAME} \ | ||
| --clobber \ | ||
| arrow_flight_sql_odbc_unsigned.dll |
There was a problem hiding this comment.
I think odbc-dll-release can stay as a separate Ubuntu job since dev/release/utils-watch-gh-workflow.sh only runs on ubuntu-latest. Before I tried running dev/release/utils-watch-gh-workflow.sh on windows-2022 and it didn't work.
We can move the uploading logic to a separate file
| odbc-msvc-upload-dll: | ||
| needs: check-labels | ||
| name: ODBC Windows Upload Unsigned DLL | ||
| runs-on: windows-2022 | ||
| if: >- | ||
| ${{ | ||
| startsWith(github.ref_name, 'apache-arrow-') && | ||
| contains(github.ref_name, '-rc') && | ||
| !inputs.odbc_release_step | ||
| }} | ||
| timeout-minutes: 240 | ||
| permissions: | ||
| packages: write | ||
| env: *odbc_msvc_env | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Build ODBC Windows | ||
| uses: ./.github/actions/odbc-windows | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| - name: Rename Unsigned ODBC DLL | ||
| run: | | ||
| Rename-Item ` | ||
| -Path build/cpp/${{ env.ARROW_BUILD_TYPE }}/arrow_flight_sql_odbc.dll ` | ||
| -NewName arrow_flight_sql_odbc_unsigned.dll | ||
| - name: Upload ODBC DLL to the job | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: flight-sql-odbc-dll | ||
| path: build/cpp/${{ env.ARROW_BUILD_TYPE }}/arrow_flight_sql_odbc_unsigned.dll | ||
| if-no-files-found: error |
There was a problem hiding this comment.
I kept the logic separate since odbc-msvc runs Arrow ODBC tests. I see other Github upload workflows like package_linux.yml don't run Arrow tests, so I followed that pattern.
| odbc-dll-release: | ||
| needs: odbc-msvc-upload-dll | ||
| name: Upload Unsigned ODBC DLL | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| # Upload to GitHub Release | ||
| contents: write | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Download the artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: flight-sql-odbc-dll | ||
| - name: Wait for creating GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| dev/release/utils-watch-gh-workflow.sh \ | ||
| ${GITHUB_REF_NAME} \ | ||
| release_candidate.yml | ||
| - name: Upload the artifacts to GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release upload ${GITHUB_REF_NAME} \ | ||
| --clobber \ | ||
| arrow_flight_sql_odbc_unsigned.dll | ||
|
|
||
| odbc-msvc-upload-msi: | ||
| needs: check-labels | ||
| name: ODBC Windows Build & Upload Unsigned MSI | ||
| runs-on: windows-2022 | ||
| if: inputs.odbc_release_step | ||
| timeout-minutes: 240 | ||
| permissions: | ||
| # Upload to GitHub Release | ||
| contents: write | ||
| packages: write | ||
| env: *odbc_msvc_env | ||
| steps: | ||
| - name: Checkout Arrow | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: Download signed ODBC DLL | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release download $env:GITHUB_REF_NAME ` | ||
| --pattern arrow_flight_sql_odbc.dll ` | ||
| --clobber | ||
| - name: Build ODBC Windows | ||
| uses: ./.github/actions/odbc-windows | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| - name: Replace signed DLL with unsigned DLL | ||
| run: | | ||
| Move-Item ` | ||
| -Path ./arrow_flight_sql_odbc.dll ` | ||
| -Destination build/cpp/${{ env.ARROW_BUILD_TYPE }}/arrow_flight_sql_odbc.dll ` | ||
| -Force | ||
| - name: Install WiX Toolset | ||
| shell: pwsh | ||
| run: | | ||
| Invoke-WebRequest -Uri https://github.com/wixtoolset/wix/releases/download/v6.0.0/wix-cli-x64.msi -OutFile wix-cli-x64.msi | ||
| Start-Process -FilePath wix-cli-x64.msi -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait | ||
| echo "C:\Program Files\WiX Toolset v6.0\bin\" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append | ||
| - name: Build MSI ODBC installer | ||
| shell: pwsh | ||
| run: | | ||
| # Verify WiX version | ||
| wix --version | ||
| cd build/cpp | ||
| cpack | ||
| - name: Upload the artifacts to GitHub Release | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| cd build/cpp | ||
| gh release upload $env:GITHUB_REF_NAME ` | ||
| --clobber ` | ||
| Apache-Arrow-Flight-SQL-ODBC-*-win64.msi |
There was a problem hiding this comment.
Yes I think this is a good idea. cc @justing-bq we can look into this.
52579a9 to
f4165e9
Compare
| - odbc-msvc-upload-dll | ||
| - odbc-msvc-upload-msi |
| push: | ||
| branches: | ||
| - '**' | ||
| - '!dependabot/**' | ||
| paths: | ||
| - '.github/actions/odbc-windows/action.yml' | ||
| - '.github/workflows/check_labels.yml' | ||
| - '.github/workflows/cpp_extra.yml' | ||
| - '.github/workflows/report_ci.yml' | ||
| - 'ci/scripts/ccache_setup.sh' | ||
| - 'ci/scripts/cpp_*' | ||
| - 'ci/scripts/download_tz_database.sh' | ||
| - 'ci/scripts/install_ccache.sh' | ||
| - 'cpp/**' | ||
| - 'format/Flight.proto' | ||
| - 'testing' |
| if [ "${PHASE_BUILD_MSI}" -gt 0 ]; then | ||
| echo "[4/8 Triggering odbc_release_step in cpp_extra.yml workflow..." | ||
| gh workflow run cpp_extra.yml \ | ||
| echo "[4/8 Triggering odbc_release_step in package_odbc.yml workflow..." |
Rationale for this change
GH-49537
What changes are included in this PR?
odbc-msvc-upload-dllUpload unsigned DLLodbc-msvc-upload-msiDownload signed DLL and upload unsigned MSIodbc-releaseCI that is replaced by the new CIs.Example of
07-flightsqlodbc-upload.shscript (not tested):We need to either 1) implement a way to get
RUN_IDand then callgh run watch,or 2) enter each command manually and wait for the CI to finish.
Documentation is in:
arrow/docs/source/developers/release.rst
Lines 267 to 276 in 62cdda9
Are these changes tested?
Are there any user-facing changes?
N/A