A multi-byte character in new PyREPL requires two backspaces to delete

[grayjk]

Bug report

Bug description:

Bug report

To reproduce

1. Open the new REPL
2. Paste: é
3. Backspace to remove the pasted character

The paste-able string above is letter 'e' followed by \u0301. To create it use:

print('e\u0301')

The \u0301 is a combining mark with zero-width display length

Current behavior

Two backspaces are required to remove the displayed character

Expected behavior

One backspace is required to remove the displayed character

This is the behavior in the old REPL

CPython versions tested on:

3.14.0, 3.15.0a2

Operating systems tested on:

Linux

Linked PRs

• gh-142162: Make PyREPL navigate and edit by grapheme cluster #145332

[typenoob]

Seems a change with the new colored REPL since 3.13 (#111201)

Extra Info: Node.js REPL also requires two backspaces to delete. Do others think this is problematic?

[typenoob]

Interestingly, there seems two different é
copy this é(\xe9) and it requires single backspace to delete.
copy this é(e\u0301) and it requires two backspaces.

[yihong0618]

This is as expected I tried other repl like node and ipython
they are the same behavior.

[typenoob]

I also agree it's the right behavior.

After some searches and tries, I discover the fact that, ́ (\u0301) this is the Combining Acute Accent. It can come after any char, though it may behave very strange when come after non-letter char.

And when it came after a letter like é, it merged automatically. However, it still two chars and apparently need two backspaces to delete (even in github comment area)

However I believe there's something odd in the new REPL as you can see a unexpected space came after é. I can open a new issue about this.

[grayjk]

@typenoob compare the behavior of the old REPL with the new REPL. The behavior should be the same - one backspace to delete the combined character

[grayjk]

in the new REPL as you can see a unexpected space came after é.

@typenoob the space after issue was addressed in #139246 which is fixed in 3.14.1 and 3.15.0a2

[typenoob]

@typenoob compare the behavior of the old REPL with the new REPL. The behavior should be the same - one backspace to delete the combined character

In cmd, it's one backspace.
In powershell, it's two backspaces.
In bash, it's one backspace.

It behaves differently among the interactive shells and their versions, some requires two, but some requires one.

Also behaves differently among text edit softwares

We should make a specification for it.
Is there any existing spec we can follow, or we can make a spec to restrict the behavior for CPython?

[picnixz]

e\u0301 is a string of 2 characters which then becomes a glyph looking like 1 character. But it's still 2 characters, not 1, so we could say that it's also fine (one backspace = one character to delete).

Is there any existing spec we can follow, or we can make a spec to restrict the behavior for CPython?

That's what we call "Combining characters" in the Unicode standard. So we could filter which characters to delete in one go.

---

Now, the alternative would be to simply check if the character that was removed is a combining character and, if so, continue deleting. I'll try to see if such patch is easy to do or not.