Other consequences of textwrap.wrap() changes in 3.15

[avylove]

Bug report

Bug description:

It seems like the change in #139902 had other effects than what the PR intended. If those changes are intentional, it would be good if they were documented. And if they were not intentional, then this seem like a bug. This change has since been backported (#139902, #139903), though I don't believe those backports have been released.

Given the following code:

import textwrap
textwrap.wrap(' ABCDEFG', width=1)

In 3.14.0, this returns [' ', 'A', 'B', 'C', 'D', 'E', 'F', 'G']
In 3.15.0a1, this returns ['A', 'B', 'C', 'D', 'E', 'F', 'G']

Since the PR only mentions trailing whitespace, it appears this change was unintended.

The documentation reads as follows.

drop_whitespace
(default: True) If true, whitespace at the beginning and ending of every line (after wrapping but before indenting) is dropped. Whitespace at the beginning of the paragraph, however, is not dropped if non-whitespace follows it. If whitespace being dropped takes up an entire line, the whole line is dropped.

I am not sure how to interpret this. This line would seem to indicate the space should be preserved.

Whitespace at the beginning of the paragraph, however, is not dropped if non-whitespace follows it.

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

• gh-140627: Fix textwrap.wrap removing whitespace at start of paragraph #140639

[picnixz]

cc @serhiy-storchaka

[serhiy-storchaka]

Interesting. This corner case was not foreseen. But I think that it was yet one bug.

Whitespace at the beginning of the paragraph, however, is not dropped if non-whitespace follows it.

But no non-whitespace follows it.

And the following sentence:

If whitespace being dropped takes up an entire line, the whole line is dropped.

implies that there should not be lines consisting of only whitespaces.

If you increase the number of leading spaces, you will see that all whitespace-only lines except one were already dropped in the old code (tested in 3.12):

>>> textwrap.wrap('     ABCDEFG', width=1)
[' ', 'A', 'B', 'C', 'D', 'E', 'F', 'G']

And #139070 just fixed that one bug.

[avylove]

And the following sentence:

If whitespace being dropped takes up an entire line, the whole line is dropped.

implies that there should not be lines consisting of only whitespaces.

I think it implies that's only the case if it's not the beginning of the paragraph, since white space is not dropped there. It seems the previous behavior was to chunk the whitespace by width and preserve the last chunk or partial chunk.

>>> textwrap.wrap('   ABCDEFG', width=1)
[' ', 'A', 'B', 'C', 'D', 'E', 'F', 'G']
>>> textwrap.wrap('   ABCDEFG', width=2)
[' A', 'BC', 'DE', 'FG']
>>> textwrap.wrap('   ABCDEFG', width=3)
['   ', 'ABC', 'DEF', 'G']

Only the first one seems broken in 3.15, but it looks like #140639 will restore the previous behavior. It might be worth adding the other test cases though.

[serhiy-storchaka]

The previous behavior was to chunk the whitespace by width and drops whitespace-only chunks, except sometimes preserve one of them.

>>> for i in range(10):
...     print(textwrap.wrap(' '*i + 'ABCDEFG', width=3))
... 
['ABC', 'DEF', 'G']
[' AB', 'CDE', 'FG']
['  A', 'BCD', 'EFG']
['   ', 'ABC', 'DEF', 'G']
[' AB', 'CDE', 'FG']
['  A', 'BCD', 'EFG']
['   ', 'ABC', 'DEF', 'G']
[' AB', 'CDE', 'FG']
['  A', 'BCD', 'EFG']
['   ', 'ABC', 'DEF', 'G']

Nothing in the documentation describes such behavior. It happened only before the split long word:

>>> for i in range(10):
...     print(textwrap.wrap(' '*i + 'A BCDEFG', width=3))
... 
['A B', 'CDE', 'FG']
[' A ', 'BCD', 'EFG']
['  A', 'BCD', 'EFG']
['A B', 'CDE', 'FG']
[' A ', 'BCD', 'EFG']
['  A', 'BCD', 'EFG']
['A B', 'CDE', 'FG']
[' A ', 'BCD', 'EFG']
['  A', 'BCD', 'EFG']
['A B', 'CDE', 'FG']

So this is just the same bug as #139065.

[avylove]

I see what you are saying about split long words, but I don't think it's the same bug, since that bug is about trailing whitespace and this is about preceding whitespace. Regardless, either there is currently a bug in the code or in the documentation.

If in the code, then it's both the issue I reported and that the behavior is not consistent when split with long words.

If in the documentation, then it's both because this is an undocumented change, and because the behavior is not clear.

So let's look at this again

If true, whitespace at the beginning and ending of every line (after wrapping but before indenting) is dropped. Whitespace at the beginning of the paragraph, however, is not dropped if non-whitespace follows it.

Based on this, the whitespace should be preserved because it is at the beginning of the paragraph. However, it does not say how much whitespace, which is inconstant, but appears to be a maximum of width characters.

If whitespace being dropped takes up an entire line, the whole line is dropped.

This only applies to whitespace that is dropped, not whitespace that is preserved.

I might be tempted to agree with you that lines that only contain whitespace should be dropped, though I don't really think wrap() should be dropping anything at the end or beginning of the string provided, since, if the user wanted those stripped, wouldn't they have stripped them? But if we assume wrap() should be doing this, then this is a documentation bug.

[serhiy-storchaka]

Let's look at this again.

First, it wraps the paragraph, i.e. split it on the chunks of maximum width size.

>>> for i in range(10):
...     print(textwrap.wrap(' '*i + 'A BCDEFG', width=3, drop_whitespace=False))
... 
['A B', 'CDE', 'FG']
[' A ', 'BCD', 'EFG']
['  A', ' BC', 'DEF', 'G']
['   ', 'A B', 'CDE', 'FG']
['   ', ' A ', 'BCD', 'EFG']
['   ', '  A', ' BC', 'DEF', 'G']
['   ', '   ', 'A B', 'CDE', 'FG']
['   ', '   ', ' A ', 'BCD', 'EFG']
['   ', '   ', '  A', ' BC', 'DEF', 'G']
['   ', '   ', '   ', 'A B', 'CDE', 'FG']

Then it drops whitespace at the beginning and ending of every line, unless a) it is at the beginning of the paragraph, and b) non-whitespace follows it.

Then it drops the whole lines that consisted of dropped whitespace. Then it adds indentations, etc.

The current code does exactly what is documented (even if this does not make much sense if the number of initial spaces larger than width). The old code preserved whitespace just before a wrapped long word (and this does not make any sense). If it was whitespace in the middle of paragraph, it was preserved as a trailing whitespace in the previous line (issue #139065). If it was whitespace at the beginning of paragraph, it was preserved as a separate whitespace-only line (this issue). This is essentially the same bug, and it was fixed by PR #139070.

[avylove]

Then this line is incorrect

If whitespace being dropped takes up an entire line, the whole line is dropped.

It says the line is removed only if the whitespace is dropped, and previous line says it won't be dropped. Therefore, the last line wouldn't apply. It should read something like this

drop_whitespace
(default: True) If true, whitespace at the beginning and ending of every line (after wrapping but before indenting) is dropped. However, whitespace at the beginning of the text is wrapped with the rest of the text, then any lines which only contain whitespace are dropped. This results in 0 to (width - 1) whitespace characters being preserved in the first line.

I think if the documentation was clarified and the change in behavior is documented in the release notes, then this issue could be closed.

[serhiy-storchaka]

The current behavior conforms the existing documentation. Your changes are not correct.

whitespace at the beginning of the text

No, it is at the beginning of the paragraph, as in the current documentation.

is wrapped with the rest of the text,

This is redundant, it is already said that the text has been wrapped. Repeating this here will add doubts about the meaning of "wrapping".

then any lines which only contain whitespace are dropped.

But lines consisting of whitespaces in the middle of the paragraph will be dropped too.

This results in 0 to (width - 1) whitespace characters being preserved in the first line.

This is redundant if it implies that there are non-whitespace characters after whitespace (which is said more directly in the current documentation but omitted in your variant). If you don't make that conclusion, you can expect a line with 0 to (width - 1) whitespace characters be preserved.

[avylove]

whitespace at the beginning of the text

No, it is at the beginning of the paragraph, as in the current documentation.

If it is paragraph, then you have another bug. This would be [' AB', 'CDE', ' EF', 'GHI'], but whitespace is not preserved at the beginning of paragraphs, it is only preserved at the beginning of the text.

>>> textwrap.wrap(' ABCDE\n EFGHI', width=3)
[' AB', 'CDE', 'EFG', 'HI']

is wrapped with the rest of the text,

This is redundant, it is already said that the text has been wrapped. Repeating this here will add doubts about the meaning of "wrapping".

Where does it say this? wrapping is only mentioned one place, in parenthesis, as a clarifying remark for something else.

then any lines which only contain whitespace are dropped.

But lines consisting of whitespaces in the middle of the paragraph will be dropped too.

Actually, that is not clear in either the existing text or my proposed text. And again, you mean text, not paragraph. wrap() with drop_whitespace does not honor paragraphs.

>> textwrap.wrap(' ABCDE\nEFGHI', width=3)
[' AB', 'CDE', 'EFG', 'HI']

This results in 0 to (width - 1) whitespace characters being preserved in the first line.

This is redundant if it implies that there are non-whitespace characters after whitespace (which is said more directly in the current documentation but omitted in your variant). If you don't make that conclusion, you can expect a line with 0 to (width - 1) whitespace characters be preserved.

I don't understand what you are trying to say here, but the existing documentation is ambiguous about when and how much whitespace is preserved.

Is it possible you are too close to this? Maybe run it by some others to get other opinions. I asked 6 people. None of them said it was clear how that option operates. To be fair, they said my proposed text was clearer, but they would like to see examples.

[serhiy-storchaka]

If it is paragraph, then you have another bug.

My mistake. The whole text is treated as a single paragraph in the textwrap module. But it is referred as a paragraph, not just text.

Where does it say this? wrapping is only mentioned one place, in parenthesis, as a clarifying remark for something else.

No, this is not something else. It says that the paragraph is wrapped before dropping whitespaces.

Actually, that is not clear in either the existing text or my proposed text.

"whitespace at the beginning and ending of every line [...] is dropped. [...] If whitespace being dropped takes up an entire line, the whole line is dropped."

This is enough clear.

I don't understand what you are trying to say here, but the existing documentation is ambiguous about when and how much whitespace is preserved.

This is ambiguous in the very unusual corner case -- when the number of initial whitespace characters is greater than or equal to the width. Because there is no good solution of this problem. This case is not supported, and what you get is just an implementation artifact, the result of applying the algorithm outside of its scope.

[avylove]

If it is paragraph, then you have another bug.

My mistake. The whole text is treated as a single paragraph in the textwrap module. But it is referred as a paragraph, not just text.

Paragraph has a specific meaning so this creates confusion. The parameter's name is text, so that's what I used. However I said "the text", where it should just be text (formatted as code), so it is clear the parameter is being referred to.

Where does it say this? wrapping is only mentioned one place, in parenthesis, as a clarifying remark for something else.

No, this is not something else. It says that the paragraph is wrapped before dropping whitespaces

And then the very next line negates this line for the case where the whitespace is at the beginning of the text. Draw a logic diagram and you'll see what I mean.

Actually, that is not clear in either the existing text or my proposed text.

"whitespace at the beginning and ending of every line [...] is dropped. [...] If whitespace being dropped takes up an entire line, the whole line is dropped."

This is enough clear.

It is not clear. Did you ask others? I found 0 people who believe this is clear. How many have you found?

I don't understand what you are trying to say here, but the existing documentation is ambiguous about when and how much whitespace is preserved.

This is ambiguous in the very unusual corner case -- when the number of initial whitespace characters is greater than or equal to the width. Because there is no good solution of this problem. This case is not supported, and what you get is just an implementation artifact, the result of applying the algorithm outside of its scope.

To the casual user, the documentation gives no indication of when and how much space is preserved at the beginning of text. It is predictable, but not documented. If you're saying it's not supported, then that should be documented.

Remember, this all came about because it was ambiguous if a code change created a bug or not. Someone should be able to read the documentation and know one way or another. Since it's ambiguous, we're pretty much relying on what is in your head. Unless you feel like having these conversations is a good use of time, then it's in your interest to make the documentation clearer.