Use `str.casefold` in `ConfigParser.optionxform`

[StSav012]

Feature or enhancement

Proposal:

As the Comparing Strings chapter reads, calling str.lower to normalize strings is only good for ASCII characters. When it comes to other Unicode ranges, it becomes much more complicated.

Yet, the current implementation keeps using lower to normalize strings.

So, I suggest the change of ConfigParser.optionxform in favor of str.casefold or even unicodedata.normalize.

It might break some code. However, I can't imagine how a sane person would miss such a behavior as optionxform currently conducts. I'd expect everyone unhappy with the usage of str.lower to overwrite optionxform a long time ago. Therefore, I'm afraid the change I suggest will not break some code but correct it.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

[StanFromIreland]

The unicodedata module is a heavy import, and is not ideal for such usage. If we want to consider this proposal, only st.casefold makes sense.

This would be backwards-incompatible, and contradicts the current documentation:

The default implementation returns a lower-case version

This would require warnings for several versions IMO

[serhiy-storchaka]

Do you have any real case whether this caused problem?

Unfortunately, neither str.lower(), nor str.casefold() can be correct answers if we talk about reproduction of case-insensitive behavior of other programs or OS, which can, for example, only ignore case in ASCII letters.

For example, if use str.lower(), then 'K' (U+212A) is equivalent to Latin 'K', but there is no guarantee that this is so in a third-party program that uses the ini-file. So Python will see option "Keyboard" overriding option "Keyboard", but other program can see them as different options. str.casefold() has more examples, so as 'ſ' (U+017F) equal to 's', and 'ß' (U+00DF) equal to 'ss'. Even if the third-party program ignore case for non-ASCII letters, we do not know what algorithm is used for this.

So, we do not know what, str.lower(), str.casefold(), or other variant, makes sense in most cases. Most likely different algorithms are used in different programs. Fortunately, you can simply override optionxform() in your program is there is a need.

[StSav012]

Fortunately for me, I've never used a package other than English in a config file. (Well, the only time it happened the config was processed with QSettings.) And only ASCII characters every time. That might've been a good habit.

For my work, I can overwrite the function indeed. And I do. I would rather not use any normalization not to confuse meV with MeV. None of the languages I use have such a doubling of a letter as we see in the str.casefold docs.

But it feels inconsistent to highlight the might of str.casefold in the docs and use str.lower in the implementation.

As for the third-party programs, I wouldn't care of them at all. If it comes to a guaranteed compatibility, everything should either fully implement the to-date Unicode support or reside to 7-bit ASCII codes and never dare a step away. Python has just got UTF-8 as the default encoding on Widows. So, that's a long road ahead.

[serhiy-storchaka]

I'm closing this as we don't have any solid evidence that using str.casefold() is better than using str.lower() here.