web backend broken by ChatGPT UI redesign (composer / send button / turn markers / blob: image src)

[dobest1024]

Version: image-use 0.30.0, chrome-use 1.5.144, macOS, Chinese UI, Plus account, relay connected (doctor all ok for web).

Command

image-use "A cute orange cat sitting on a windowsill, soft morning light, cozy illustration style" --backend web --size 1024x1024

Symptoms (in order, each surfaced after fixing the previous one)

1. composer never appeared (likely a login wall …) — page is logged in, but the ProseMirror composer no longer has id="prompt-textarea" (it is now div.ProseMirror[contenteditable=true] inside a form).
2. ChatGPT Send button did not become ready — send button no longer has data-testid="send-button"; it is a plain button[type=submit] with aria-label="发送" inside the composer form.
3. could not confirm ChatGPT received the prompt — the prompt was sent (conversation created), but [data-message-author-role] no longer exists anywhere in the DOM.
4. Result image is now <img src="blob:https://chatgpt.com/…"> inside button[data-testid="generated-image-preview"] (under [data-testid="generated-image-gallery"]), so WEB_IMG_SRC_RE never matches.

New DOM observations (2026-09-29)

• Each turn is div[data-turn-key=…] containing both the user bubble ([data-user-message-bubble=true]) and an assistant heading h4[data-conversation-role=assistant] (present before the reply renders, so it's not a "reply done" signal on its own).
• The entire thread is rendered twice (every data-turn-key appears 2×), so naive counts are doubled — dedupe by data-turn-key.
• In-page fetch(blobSrc) works fine from chrome-use eval (got image/png, 2.1 MB), so _JS_FETCH needs no change.
• Stop button detection worked with an added button[aria-label*="停止"] (localized UI).
• Model picker not found (warning: model picker not found) — harmless, not investigated.

Local patch that makes it work end-to-end (verified: submit → generating → receiving image → download → conversation deleted, ~105 s):

Selector fallbacks only; old selectors kept so the old DOM still works.

image-use-chatgpt-redesign.patch

--- image-use.orig	2026-09-29 22:36:16
+++ image-use	2026-09-29 22:48:46
@@ -1380,7 +1380,7 @@
 _JS_STATE = r"""(() => {
   const seen = new Set(%s);
   const stop = !!document.querySelector(
-    'button[data-testid="stop-button"], button[aria-label*="Stop" i]'
+    'button[data-testid="stop-button"], button[aria-label*="Stop" i], button[aria-label*="停止"]'
   );
   const re = new RegExp(%s);
   // The generated image renders in <main> but OUTSIDE any message bubble (it's
@@ -1389,12 +1389,25 @@
   // user bubble with a fresh estuary/content src not in the pre-submit baseline.
   // So detect fresh main images but exclude any that belong to a user message.
   const userSrcs = new Set();
-  document.querySelectorAll('[data-message-author-role="user"] img')
+  document.querySelectorAll('[data-message-author-role="user"] img, [data-user-message-bubble] img')
     .forEach(i => { const s = i.currentSrc || i.src; if (s) userSrcs.add(s); });
+  // 2026-09 ChatGPT redesign: results render as blob: imgs inside
+  // [data-testid=generated-image-preview]; accept those alongside the regex.
   const fresh = [...document.querySelectorAll('main img')]
+    .filter(i => re.test(i.currentSrc || i.src) ||
+      (!!i.closest('[data-testid="generated-image-preview"]') && i.complete && i.naturalWidth > 0))
     .map(i => i.currentSrc || i.src)
-    .filter(s => re.test(s) && !seen.has(s) && !userSrcs.has(s));
-  const a = document.querySelectorAll('[data-message-author-role="assistant"]');
+    .filter(s => s && !seen.has(s) && !userSrcs.has(s));
+  let a = [...document.querySelectorAll('[data-message-author-role="assistant"]')];
+  if (!a.length) {
+    // Redesign: each [data-turn-key] holds the user bubble AND the reply; the
+    // reply counts once it has rendered content (markdown or an image gallery).
+    const turns = [...document.querySelectorAll('main [data-turn-key]')];
+    const t = turns[turns.length - 1];
+    const body = t && t.querySelector('[data-conversation-role="assistant"]') &&
+      t.querySelector('[data-markdown-copy], [data-testid="generated-image-gallery"]');
+    if (body) a = [body];
+  }
   const lastA = a[a.length - 1];
   const dlg = [...document.querySelectorAll('[role="dialog"]')]
     .map(d => d.textContent || '').join(' ');
@@ -1407,7 +1420,9 @@
 _JS_BASELINE = r"""(() => {
   const re = new RegExp(%s);
   const srcs = [...document.querySelectorAll('main img')]
-    .map(i => i.currentSrc || i.src).filter(s => re.test(s));
+    .filter(i => re.test(i.currentSrc || i.src) ||
+      !!i.closest('[data-testid="generated-image-preview"]'))
+    .map(i => i.currentSrc || i.src).filter(Boolean);
   return JSON.stringify(srcs);
 })()"""
 
@@ -1476,7 +1491,7 @@
   const dlg = [...document.querySelectorAll('[role="dialog"]')]
     .map(d => d.textContent || '').join(' ');
   return JSON.stringify({
-    composer: !!document.querySelector('#prompt-textarea'),
+    composer: !!document.querySelector('#prompt-textarea, div.ProseMirror[contenteditable=true]'),
     limited: /too many requests|requests too quickly/i.test(dlg),
   });
 })()"""
@@ -2130,7 +2145,7 @@
 # spacing) or textContent (which loses paragraph breaks). A ProseMirror trailing
 # BR is an empty-paragraph placeholder, not an extra newline.
 _JS_WEB_COMPOSER_STATE = r"""(() => {
-  const composer = document.querySelector('#prompt-textarea');
+  const composer = document.querySelector('#prompt-textarea, div.ProseMirror[contenteditable=true]');
   const form = composer && composer.closest('form');
   const read = n => {
     if (n.nodeType === 3) return n.textContent;
@@ -2147,7 +2162,7 @@
   }
   const images = form ? [...form.querySelectorAll('img')]
     .filter(i => /blob:|estuary\/content|oaiusercontent/.test(i.src || '')) : [];
-  const send = form && form.querySelector('button[data-testid="send-button"]');
+  const send = form && form.querySelector('button[data-testid="send-button"], button[type=submit]');
   return JSON.stringify({
     exists: !!composer, text,
     attachments: images.length,
@@ -2156,7 +2171,11 @@
     ready: images.filter(i => !i.src.startsWith('blob:')).length,
     busy: !!(form && form.querySelector('[role="progressbar"], [aria-busy="true"]')),
     send_ready: !!send && !send.disabled && send.getAttribute('aria-disabled') !== 'true',
-    user_turns: document.querySelectorAll('[data-message-author-role="user"]').length,
+    user_turns: document.querySelectorAll('[data-message-author-role="user"]').length +
+      new Set([...document.querySelectorAll('[data-user-message-bubble]')].map((b, i) => {
+        const t = b.closest('[data-turn-key]');
+        return t ? t.getAttribute('data-turn-key') : 'b' + i;
+      })).size,
   });
 })()"""
 
@@ -2253,7 +2272,7 @@
 def _clear_web_composer(ab, session, remaining) -> None:
     """Best-effort: empty the composer so our own text is not left as a draft."""
     try:
-        _ab(ab, "fill", "#prompt-textarea", "--stdin", input_text="",
+        _ab(ab, "fill", "#prompt-textarea, div.ProseMirror[contenteditable=true]", "--stdin", input_text="",
             session=session, timeout=min(20.0, remaining()))
     except GatewayError:
         pass
@@ -2275,7 +2294,7 @@
     if initial.get("text"):
         raise GatewayError("ChatGPT composer is not empty; prompt not submitted. " + _DRAFT_HINT)
     emit("pasting prompt")
-    _ab(ab, "fill", "#prompt-textarea", "--stdin", input_text=user_text,
+    _ab(ab, "fill", "#prompt-textarea, div.ProseMirror[contenteditable=true]", "--stdin", input_text=user_text,
         session=session, timeout=min(60.0, remaining()))
     # Everything below until the click is a "nothing was sent" failure. ChatGPT
     # keeps composer drafts across chats, so leaving our text in the box would
@@ -2309,7 +2328,7 @@
     emit(f"verified prompt ({len(user_text)} characters) and {n_refs} reference image(s)")
     send_error = None
     try:
-        _ab(ab, "click", 'button[data-testid="send-button"]',
+        _ab(ab, "click", 'form button[data-testid="send-button"], form button[type=submit]',
             session=session, timeout=min(20.0, remaining()))
     except GatewayError as error:
         send_error = error

image-use doctor output:

[ok]    python     3.10.13
[ok]    version    0.30.0 (latest)
[ok]    codex      OAuth token present (~/.codex/auth.json)
[ok]    chrome-use 1.5.144
[ok]    relay      connected (drives your open Chrome)
[ok]    profiles   1 logged-in: Default
→ auto would use: web  (spares Codex-usage)

[leeguooooo]

Thanks for the detailed report and the patch. Your DOM notes were enough to build the fix and its test fixture. It's fixed in v0.30.1 (#42).

What changed in the web backend:

• Composer: it now also matches form div.ProseMirror[contenteditable=true], alongside #prompt-textarea.
• Send: it now also matches a plain button[type=submit] in the composer's form. The label can be in any language (发送 included), and voice and dictation buttons are never matched.
• Turns: user messages are also detected from [data-user-message-bubble] and counted per distinct data-turn-key, so the doubled thread counts once. The reply counts only once it has content (markdown or generated-image-gallery), not just the h4 heading.
• Result image: a blob: <img> inside generated-image-preview is accepted once it has decoded, and its bytes are fetched inside the page. If ChatGPT revokes the blob first, the page re-encodes the decoded image as PNG instead.
• Stop button: it also matches localized labels (停止, 中止).

The old selectors are all still there, so accounts that still get the previous layout keep working. I checked that with a real run on the old layout. The tests now cover both layouts with DOM fixtures.

To upgrade:

image-use upgrade
image-use --version   # 0.30.1

If you use the Claude Code plugin, the marketplace already lists 0.30.1. If this is still broken on your account, please reopen with the full run output (progress lines included) and I'll take another look.