Build Windows debug extensions without debug Python

[Andrej730]

Feature or enhancement

Proposal:

---

Summary of a problem

On Windows when users are building a debug version of anything, including Python extensions, they typically provide /MDd/MTd flag to link against the debug version of the C runtime library.

Which automatically sets _DEBUG, which, historically, sets Py_DEBUG macro.

Py_DEBUG macro makes it impossible to link against release version of Python library:

• because of implicit linking against _d.lib by # pragma comment(lib,"python315_d.lib").
  This restriction was overcomed in Python 3.14 by introduction of Py_NO_LINK_LIB macro.

• Py_DEBUG also automatically sets Py_REF_DEBUG, which in turn enables additional symbols (namely Py_NegativeRefcount, Py_INCREF_IncRefTotal, Py_DECREF_DecRefTotal), which of course are not present in Release library, so linking Debug build against it is still impossible.
  But if we disable Py_REF_DEBUG when Py_DEBUG is set, then linking succeeds, so there are no more obstacles
  to build debug extensions against release Python.

Typically, users don't want to use debug Python for their extensions, they just want debug version of their own extension, as debug Python binaries are not widely available and even if they are, after build user now would also need debug versions of the entire extensions ecosystem they were using.

So it would be great to provide an option to finally figure this out after all those years.

---

Changes

There are two approaches to consider (PR currently implements both of them just to showcase the options, but we need to pick one):

• Py_NO_PY_REF_DEBUG - macro to avoid setting Py_REF_DEBUG based on Py_DEBUG.
  It's a minimal change change to overcome the current problem.
  It seems currently there are no implications when set Py_DEBUG would conflict with unset Py_REF_DEBUG, but Python developers will need to keep it in mind in future that Py_REF_DEBUG might be unset even in debug builds.

• Py_WIN_NO_PY_DEBUG - that would make _DEBUG not imply Py_DEBUG.
  Honestly, I would prefer this option, as it seems to be a fix for the root cause of the problem and makes flags more consistent between Windows and other platforms - making Py_DEBUG deliberate choice on both Windows and Unix systems.
  _DEBUG implying Py_DEBUG seems to be mostly historical thing introduced awhile ago (9f9fa6d) and they're not closely related.
  Adding Py_WIN_NO_PY_DEBUG also opens an opportunity in the future to completely deprecate _DEBUG setting Py_DEBUG.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/building-debug-version-of-extensions-on-windows-without-debug-binaries

Linked PRs

• gh-142760: Build Windows debug extensions without debug Python #142761

[Andrej730]

@zooba @vstinner

[vstinner]

I would prefer to only add Py_NO_PY_DEBUG (without WIN_ prefix), and not add Py_NO_PY_REF_DEBUG.

Py_DEBUG is wide, it enables many debug features which requires a different ABI. Py_REF_DEBUG is just an example of side effect of Py_DEBUG.

cc @encukou

[encukou]

Hmm, could this be solved by providing Py_NegativeRefcount, Py_INCREF_IncRefTotal, Py_DECREF_DecRefTotal in release builds (as no-ops)?

[Andrej730]

Hmm, could this be solved by providing Py_NegativeRefcount, Py_INCREF_IncRefTotal, Py_DECREF_DecRefTotal in release builds (as no-ops)?

Interesting idea, it's also an option, but it has similar downside to Py_NO_PY_REF_DEBUG - developers will need to keep in mind that Py_DEBUG is not neccesarily enables all debug features. And perhaps in the future there can be more debug options, so maybe it's better to make Py_DEBUG more deliberate in the first place.

I would prefer to only add Py_NO_PY_DEBUG (without WIN_ prefix), and not add Py_NO_PY_REF_DEBUG.

Py_DEBUG is wide, it enables many debug features which requires a different ABI. Py_REF_DEBUG is just an example of side effect of Py_DEBUG.

I was thinking about Py_WIN_NO_PY_DEBUG meaning not "disable Py_DEBUG", but rather "disable auto-setting Py_DEBUG on Windows". And with Py_WIN_NO_PY_DEBUG set, setting Py_DEBUG behaviour works consistently across all the platforms. So in theory, with time, this flag can be deprecated completely. Maybe flag name can be adjusted to make it more clear.

Regarding Py_NO_PY_DEBUG - it seems typically NO_XXX flag takes precedence over XXX.
But for this particular case, I thought build systems would be able to adapt Py_WIN_NO_PY_DEBUG set by default on Windows, when user is building extension in Debug configuration, as this is what the most users would likely to expect. And then if user really wants debug Python, they can also define -DPy_DEBUG next to it, so -DPy_WIN_NO_PY_DEBUG -DPy_DEBUG -> Py_DEBUG.

But since typically NO_XXX takes precedence, then on Unix it seems more intuitive that -DPy_NO_PY_DEBUG -DPy_DEBUG -> No Py_DEBUG. So I thought maybe better to keep this flag just on Windows.

[vstinner]

I was thinking about Py_WIN_NO_PY_DEBUG meaning not "disable Py_DEBUG", but rather "disable auto-setting Py_DEBUG on Windows". And with Py_WIN_NO_PY_DEBUG set, setting Py_DEBUG behaviour works consistently across all the platforms.

I would prefer to have a Py_NO_PY_DEBUG macro which would work the same on all platforms, not only Windows. "disable Py_DEBUG" and "disable auto-setting Py_DEBUG on Windows" sound like the same thing to me.

And then if user really wants debug Python, they can also define -DPy_DEBUG next to it, so -DPy_WIN_NO_PY_DEBUG -DPy_DEBUG -> Py_DEBUG.

That's strange. As you wrote, I would prefer that Py_NO_PY_DEBUG has the priority over Py_DEBUG.

So I thought maybe better to keep this flag just on Windows.

I don't follow your rationale. I don't see how it's an issue to have the Py_NO_PY_DEBUG macro available on all platforms.

[vstinner]

@encukou:

Hmm, could this be solved by providing Py_NegativeRefcount, Py_INCREF_IncRefTotal, Py_DECREF_DecRefTotal in release builds (as no-ops)?

If possible, I would prefer to avoid this way. The debug build has a different ABI and I would prefer to leave these debug-specific functions in the debug build (debug ABI).

[Andrej730]

I don't follow your rationale. I don't see how it's an issue to have the Py_NO_PY_DEBUG macro available on all platforms.

Py_NO_PY_DEBUG will be able to resolve the original problem too, but my concern is about trying to achieve Py_DEBUG usage consistency across the platforms.

E.g. let's say Py_NO_PY_DEBUG option is added and we're building extension in Debug configuration:

• On Windows -> Py_DEBUG is set by default (based on _DEBUG):

  • to enable Py_DEBUG - user shouldn't provide anything
  • disable Py_DEBUG - provide Py_NO_PY_DEBUG

• On Unix -> Py_DEBUG is not set by default:

  • enable Py_DEBUG - provide Py_DEBUG
  • disable Py_DEBUG - user shouldn't provide anything
  • undo Py_DEBUG if it was provided previously - provide Py_NO_PYDEBUG

So not providing Py_DEBUG has a different meaning depending on the platform and on Windows Py_DEBUG cannot be set deliberately.

But if we add Py_WIN_NO_PY_DEBUG option instead, that a build system can set by default on Windows and Debug config (assuming it's a commonly desired behaviour), then the usage for the end user becomes consistent for all platforms - Py_DEBUG is disabled by default and can be enabled by providing it explicitly.

Py_WIN_NO_PY_DEBUG basically resolves the same issue, but not by allowing the user to undo Py_DEBUG, but by decoupling it from _DEBUG on Windows. Possibly allowing us to deprecate _DEBUG -> Py_DEBUG connection completely in the future.

Maybe the name can be adjusted to something like Py_WIN_EXPLICIT_PY_DEBUG, Py_WIN_DISABLE_AUTO_PY_DEBUG, to avoid confusion like "why there's Py_NO_PY_DEBUG on Windows, but not on Unix".

[zooba]

On Windows when users are building a debug version of anything, including Python extensions, they typically provide /MDd/MTd flag to link against the debug version of the C runtime library.

It's easiest just to fix this flag, which then will link everything correctly without any further effort. Build with /MD instead of /MDd and you'll be fine - there are other options to disable optimisation and generate debug symbols.