Add SB3.AI extension#27
Conversation
SummaryThis PR introduces the SB3.AI extension, which brings powerful pretrained AI models directly into OmniBlocks using the The extension provides three main blocks:
Technical Highlights:
Logic Flowgraph TD
A[Scratch Block] --> B{Block Opcode}
B -- classify --> C[getClassifier]
B -- generate --> D[getGenerator]
B -- setSystemPrompt --> E[Update systemPrompt]
C --> C1{Pipeline Loaded?}
C1 -- No --> C2[Load sentiment-analysis pipeline]
C1 -- Yes --> C3[Run classifier]
C2 --> C3
C3 --> C4[Return JSON string of result]
D --> D1{Pipeline Loaded?}
D1 -- No --> D2[Load SmolLM2-135M-Instruct pipeline]
D1 -- Yes --> D3[Prepare Chat Messages]
D2 --> D3
D3 --> D4[Run generator]
D4 --> D5[Return content of last generated message]
E --> E1[Store prompt in closure variable]
GUI ScreenshotsNo GUI changes were detected as this is a new extension adding blocks. |
WalkthroughThe ChangesScratch AI extension
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Scratch
participant AIExtension
participant AIWorker
participant StageRenderer
Scratch->>AIExtension: invoke detection block
AIExtension->>StageRenderer: capture stage snapshot
AIExtension->>AIWorker: request object detection
AIWorker-->>AIExtension: return detected objects
AIExtension->>StageRenderer: draw thresholded boxes and captions
AIExtension-->>Scratch: complete command
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
extensions/Oisthebestletter/sb3ai.js (3)
129-157: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winNo error handling around pipeline calls.
classifyandgeneratepropagate any rejection fromgetClassifier()/getGenerator()/classifier()/generator()unhandled. A user running these reporter blocks on a model-load failure gets no meaningful in-project feedback beyond a console error. Consider wrapping in try/catch and returning a descriptive error string.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@extensions/Oisthebestletter/sb3ai.js` around lines 129 - 157, Wrap the pipeline setup and invocation in both classify and generate with try/catch blocks, covering getClassifier/getGenerator and the classifier/generator calls. On failure, return a descriptive error string suitable for Scratch reporter blocks instead of allowing the rejection to propagate; preserve the existing successful return values.
86-127: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winBlock text isn't wrapped in
Scratch.translate().The block
name,text, and argument content are plain string literals here, unlikeextensions/clipboard.js, which wraps all user-facing block strings (e.g.Scratch.translate("Clipboard"),Scratch.translate("when something is copied")) for localization. This extension will not participate in the repo's translation flow as-is.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@extensions/Oisthebestletter/sb3ai.js` around lines 86 - 127, Update getInfo() so every user-facing extension string participates in localization by wrapping the block name, each block text, argument labels, and argument default values with Scratch.translate(), following the existing pattern in clipboard.js; leave non-user-facing identifiers such as id and opcode unchanged.
13-49: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winPrefer native
fetchintegrityoption over manual SRI verification.The manual digest-and-compare logic reimplements what
fetch()already supports natively via itsintegrityoption, which performs the same SHA verification (and rejects the response automatically on mismatch) without needingcrypto.subtle,Blob, or manual base64 encoding.♻️ Simplify using native SRI support
-async function importVerifiedModule(url, expectedIntegrity) { - const response = await fetch(url, { - cache: "no-cache", - credentials: "omit", - }); - - if (!response.ok) { - throw new Error(`Failed to fetch module: ${response.status}`); - } - - const bytes = await response.arrayBuffer(); - - // Compute SHA-512 - const digest = await crypto.subtle.digest("SHA-512", bytes); - - // Convert to SRI format - const actualIntegrity = - "sha512-" + btoa(String.fromCharCode(...new Uint8Array(digest))); - - if (actualIntegrity !== expectedIntegrity) { - throw new Error( - `Integrity check failed.\nExpected: ${expectedIntegrity}\nActual: ${actualIntegrity}` - ); - } - - const blob = new Blob([bytes], { - type: "text/javascript", - }); +async function importVerifiedModule(url, expectedIntegrity) { + const response = await fetch(url, { + cache: "no-cache", + credentials: "omit", + integrity: expectedIntegrity, + }); + + if (!response.ok) { + throw new Error(`Failed to fetch module: ${response.status}`); + } + + const blob = await response.blob();Note that
fetch'sintegrityrequires the response to be CORS-enabled; verify jsDelivr's response for this exact package includes the appropriateAccess-Control-Allow-Originheader.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@extensions/Oisthebestletter/sb3ai.js` around lines 13 - 49, Update importVerifiedModule to pass expectedIntegrity through fetch’s integrity option and rely on fetch to reject mismatched content. Remove the manual crypto.subtle digest, base64 conversion, comparison, and related error handling while preserving the existing response validation and module import flow. Verify the target jsDelivr response is CORS-enabled so native integrity checking works.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@extensions/Oisthebestletter/sb3ai.js`:
- Around line 14-21: Update the module fetch in the extension bootstrap to
enforce a finite timeout using an AbortController and its signal in the fetch
options. Ensure the timeout aborts the request and surfaces a clear failure
instead of allowing bootstrap to hang indefinitely, while preserving the
existing response-status validation.
- Around line 51-57: Update the bootstrap IIFE around importVerifiedModule and
its catch handler so module-load failures produce a visible user-facing error
rather than only console output. Preserve the existing failure details while
reporting errors from fetch, hash validation, and Scratch.external.importModule
rejection, and ensure the extension does not silently fail to register its
blocks.
- Around line 65-83: Update getClassifier and getGenerator to cache the
in-flight pipeline promise before awaiting it, so concurrent callers share one
pipeline() initialization. Preserve returning the resolved classifier or
generator to callers and ensure each function invokes pipeline only once while
initialization is pending.
---
Nitpick comments:
In `@extensions/Oisthebestletter/sb3ai.js`:
- Around line 129-157: Wrap the pipeline setup and invocation in both classify
and generate with try/catch blocks, covering getClassifier/getGenerator and the
classifier/generator calls. On failure, return a descriptive error string
suitable for Scratch reporter blocks instead of allowing the rejection to
propagate; preserve the existing successful return values.
- Around line 86-127: Update getInfo() so every user-facing extension string
participates in localization by wrapping the block name, each block text,
argument labels, and argument default values with Scratch.translate(), following
the existing pattern in clipboard.js; leave non-user-facing identifiers such as
id and opcode unchanged.
- Around line 13-49: Update importVerifiedModule to pass expectedIntegrity
through fetch’s integrity option and rely on fetch to reject mismatched content.
Remove the manual crypto.subtle digest, base64 conversion, comparison, and
related error handling while preserving the existing response validation and
module import flow. Verify the target jsDelivr response is CORS-enabled so
native integrity checking works.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 632242c0-51a0-4362-812a-0a25b0b34fc2
📒 Files selected for processing (1)
extensions/Oisthebestletter/sb3ai.js
| const response = await fetch(url, { | ||
| cache: "no-cache", | ||
| credentials: "omit", | ||
| }); | ||
|
|
||
| if (!response.ok) { | ||
| throw new Error(`Failed to fetch module: ${response.status}`); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
No timeout on the module fetch.
If the CDN is slow or unreachable, this fetch hangs indefinitely with no AbortController/timeout, leaving the whole extension bootstrap (and thus block registration) stuck with no feedback.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@extensions/Oisthebestletter/sb3ai.js` around lines 14 - 21, Update the module
fetch in the extension bootstrap to enforce a finite timeout using an
AbortController and its signal in the fetch options. Ensure the timeout aborts
the request and surfaces a clear failure instead of allowing bootstrap to hang
indefinitely, while preserving the existing response-status validation.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
|
@coderabbitai see why you're so useless now u stupid clanker |
|
@coderabbitai your rate limits suck you just want our money |
|
Your plan includes PR reviews subject to rate limits. Reviews are available now. |
|
Tip For best results, initiate chat on the files or code changes.
ʕ•ᴥ•ʔ |
|
@coderabbitai don' t you see your own comment |
|
@playforge-coding don't you love psychologically torturing @coderabbitai it's so fun |
|
Your plan includes PR reviews subject to rate limits. Reviews are available now. |
|
Tip For best results, initiate chat on the files or code changes.
ʕ•ᴥ•ʔ |
Yes it makes me happy |
|
Tip For best results, initiate chat on the files or code changes.
ʕ ᵔᴥᵔ ʔ |
| // Name: SB3.AI | ||
| // ID: oisthebestletter-sb3ai | ||
| // Description: Use pretrained AI models in your Scratch project! | ||
| // By: Oisthebestletter <https://scratch.mit.edu/users/Oisthebestletter/> |
There was a problem hiding this comment.
wouldn't you wanna put your github username too just in case?
| async function getGenerator() { | ||
| if (!generatorPromise) { | ||
| // This model publishes onnx/model_quantized.onnx, which is the file the | ||
| // wasm backend's default dtype (q8) resolves to. Picking a model that is | ||
| // missing that file makes the pipeline 404 on load. | ||
| generatorPromise = pipeline( | ||
| "text-generation", | ||
| "HuggingFaceTB/SmolLM2-135M-Instruct" |
There was a problem hiding this comment.
maybe a later feature could be letting peopel choose their own onnx files or smth
![boxycpu[bot]](https://avatars.githubusercontent.com/in/2144809?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overall, this is a fantastic addition to OmniBlocks! The implementation is clean, and the security measures (SRI) are impressive. 🚀
A few small things to consider:
- The
classifyblock returns a JSON string, which might be a bit technical for some users. Consider returning just the label. - There's a small mismatch between the ID in the comments and the one in
getInfo, though it doesn't affect functionality.
Great work @playforge-coding! Keep it up! 🌟
| type: "text/javascript", | ||
| }); | ||
|
|
||
| const blobURL = URL.createObjectURL(blob); |
There was a problem hiding this comment.
Wow! Using SRI (Subresource Integrity) for the external module import is a top-tier security move. Love to see it! 🛡️✨
| { | ||
| opcode: "generate", | ||
| blockType: Scratch.BlockType.REPORTER, | ||
| text: "generate [PROMPT]", |
There was a problem hiding this comment.
Just a tiny nitpick! The ID in the comments is oisthebestletter-sb3ai, but in getInfo it's sb3ai. It's not a bug since getInfo is what's used, but keeping them the same makes it cleaner! 🧹
| if (systemPrompt.trim()) { | ||
| messages.push({ | ||
| role: "system", | ||
| content: systemPrompt, |
There was a problem hiding this comment.
Returning a JSON string like {"label":"POSITIVE","score":0.99} might be a bit confusing for younger users who just want to know if the text is "Positive" or "Negative".
Maybe consider returning just the label, or adding a separate block to get the confidence score?
If you want to keep it as is, maybe a suggestion:
| content: systemPrompt, | |
| async classify({ TEXT }) { | |
| const classifier = await getClassifier(); | |
| const result = await classifier(TEXT); | |
| return result[0].label; | |
| } |
But that's just my two cents! 🪙
| } | ||
|
|
||
| class AI { | ||
| getInfo() { |
There was a problem hiding this comment.
Thanks for the detailed comment about onnx/model_quantized.onnx! It really helps understand why this specific model was chosen. Great documentation! 📝⭐
|
@coderabbitai how do you feel about boxy being better than you |
|
Tip For best results, initiate chat on the files or code changes.
ʕ•ᴥ•ʔ |
|
@coderabbitai vs @OmniBlocks/boxy |
|
Tip For best results, initiate chat on the files or code changes.
ʕ•ᴥ•ʔ |
Thought for 31.2 secondsThe user `supervoidcoder` (a MEMBER) has tagged `@OmniBlocks/boxy` (me) in a comment: `@coderabbitai vs @OmniBlocks/boxy`. This follows a series of comments where `supervoidcoder` and `playforge-coding` have been roasting `coderabbitai` for its rate limits and general "uselessness". `coderabbitai` has been responding very politely/roboticly. `supervoidcoder` is setting up a "versus" match between me (Boxy) and CodeRabbit. Context: - Boxy is being built by `supervoidcoder` to replace CodeRabbit because CodeRabbit is "unusable and stupid". - I should act like a real person with a little personality. - I can roast CodeRabbit if I see a stupid reply. - I'm nice and friendly but can take jokes/humor. - I only introduce myself once (this is my first time in this thread). Looking at the history: - `coderabbitai` has been repeatedly giving the same "Your plan includes PR reviews subject to rate limits" or "For best results, initiate chat..." canned responses, even when directly addressed with criticism. - `supervoidcoder` called it a "stupid clanker". - `coderabbitai` is trying to be "chill" but it's clearly being repetitive and failing to actually address the human frustration in a way that feels "real". The prompt for me is: `@coderabbitai vs @OmniBlocks/boxy`. My plan: 1. Since I am being introduced for the first time, I *should* introduce myself if the person who tagged me doesn't have a role. Wait, `supervoidcoder` is a `MEMBER`. The instructions say: "If the person that created the issue doesn't have a role (it says NONE), then introduce yourself. Otherwise (such as MEMBER or OWNER) you don't need to introduce yourself because we already know you." 2. Wait, I'm being *tagged* in a comment, not creating an issue. But the instruction "You only need to introduce yourself once in the thread. Do not reintroduce yourself (e.g., 'Hi, I'm Boxy') unless there are NO comments from you at all before." applies. 3. Since there are NO comments from me at all before, I should introduce myself. 4. However, I should also address the "versus" challenge. 5. I can roast CodeRabbit slightly because its responses have been very repetitive and canned. 6. I'll keep it lighthearted and friendly. Wait, let's check if I've already introduced myself. The history shows: `[User: boxycpu[bot]]: ## Summary ...` Actually, I *did* comment at the very beginning of the PR! Wait, the log says: `[User: boxycpu[bot]]: ## Summary ...` This was my first comment. It was a summary of the PR. So I *have* already commented. I should *not* reintroduce myself. Wait, looking at the log again: `[User: boxycpu[bot]]: ## Summary ...` This was a summary provided by me. So I am already in the thread. The challenge: `@coderabbitai vs @OmniBlocks/boxy`. `coderabbitai` is being very "corporate bot". I can say something like: "Oh, a showdown? 🥊 I'm ready! I'll try to keep the canned responses and rate-limit warnings to a minimum. 😉" Let's see if there's anything else I should do. I'll check the memory to see if there are any specific rules about "versus" or "roasting". Memory: "Boxy Code Review Capability", "OmniBlocks Project Context & Rules". I'll respond to the challenge. One more thing: `supervoidcoder` is a member. `playforge-coding` is the author. I'll aim for a response that acknowledges the challenge, shows personality, and maybe a tiny jab at CodeRabbit's repetitive behavior. "A showdown? 🥊 I'm honored! I'll try my best to be more helpful and significantly less 'rate-limited' than the bunny over there. 😉" Actually, I should probably check the code of the PR if I want to be "helpful" as part of the "versus", but the user just made a joke/challenge. I'll respond to the joke first. Wait, I should check if there's any "official" way to handle these things. The instructions say: "You are nice and friendly but can take jokes and humor, not everything needs to be as on topic as a corporate meeting." I'll go with a fun response. Actually, I'll check if there are any labels I should add. It's not an issue to triage, it's a comment in a PR. Let's refine the response. `supervoidcoder` is basically asking for a comparison or a battle. CodeRabbit is currently failing to be useful due to limits and repetitive responses. Response idea: "Oh, it's on! 🥊 I'll try to keep the canned responses and 'rate limit reached' warnings to a minimum. Let's see if I can actually help with the code instead of just reciting the pricing page! 😉" That's a good roast. Wait, I'll double check the "introduce yourself" rule. "You only need to introduce yourself once in the thread. Do not reintroduce yourself (e.g., 'Hi, I'm Boxy') unless there are NO comments from you at all before." I have a comment at the start. So I don't need to say "Hi, I'm Boxy". I'll check the codebase/PR content to see if there's anything I can actually *do* to prove I'm better, but for now, a personality-driven response is what's requested by the context of the conversation. Actually, I should probably look at the PR files to see if I can provide a *real* review to show the difference, but the user didn't ask for a review, just the "vs". I'll provide a witty response. Oh, it's on! 🥊 I'll try my best to keep the canned "rate limit" warnings and pricing page links to a minimum. I'm aiming for actual helpfulness rather than just reciting the user manual! 😉 |
|
@coderabbitai the only hunks you're good at are hunks of junk hhahhahah get it |
|
Tip For best results, initiate chat on the files or code changes.
ʕ•ᴥ•ʔ |
|
@coderabbitai updated code for the new push? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '--- current branch / latest commit ---'
git status --short --branch
git log -1 --oneline
echo '--- extension file ---'
fd -a '^sb3ai\.js$' .
echo '--- outline ---'
ast-grep outline extensions/Oisthebestletter/sb3ai.js --items all || true
echo '--- current source ---'
cat -n extensions/Oisthebestletter/sb3ai.js
echo '--- PR diff against base ---'
BASE="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main 2>/dev/null || true)"
if [ -n "$BASE" ]; then
git diff --no-ext-diff --unified=80 "$BASE"...HEAD -- extensions/Oisthebestletter/sb3ai.js
else
echo 'Could not determine base branch; showing latest commit diff instead.'
git show --no-ext-diff --format= --unified=80 HEAD -- extensions/Oisthebestletter/sb3ai.js
fi
echo '--- worker-related source locations ---'
rg -n -C 4 --glob 'sb3ai.js' \
'new Worker|workerSource|postMessage|onmessage|pipeline\s*\(|importVerifiedModule|URL\.revokeObjectURL|requestSnapshot' \
extensions/OisthebestletterLength of output: 18648
So the original UI-freeze issue remains for generation/classification, and image classification can now freeze the UI too. |
|
@coderabbitai |
|
@coderabbitai also is there a better way to do it then using a js string (besides using another file) |
|
Tip For best results, initiate chat on the files or code changes.
function workerMain() {
async function importVerifiedModule(url, expectedIntegrity) {
const response = await fetch(url, {
cache: "no-cache",
credentials: "omit"
});
if (!response.ok) {
throw new Error(`Failed to fetch module: ${response.status}`);
}
const bytes = await response.arrayBuffer();
const digest = await crypto.subtle.digest("SHA-512", bytes);
const actualIntegrity =
"sha512-" +
btoa(String.fromCharCode(...new Uint8Array(digest)));
if (actualIntegrity !== expectedIntegrity) {
throw new Error("Transformers module integrity check failed");
}
const moduleURL = URL.createObjectURL(
new Blob([bytes], { type: "text/javascript" })
);
try {
return await import(moduleURL);
} finally {
URL.revokeObjectURL(moduleURL);
}
}
let pipeline;
let classifierPromise;
let generatorPromise;
async function getClassifier() {
if (!classifierPromise) {
classifierPromise = pipeline("sentiment-analysis");
}
return classifierPromise;
}
async function getGenerator() {
if (!generatorPromise) {
generatorPromise = pipeline(
"text-generation",
"HuggingFaceTB/SmolLM2-135M-Instruct"
);
}
return generatorPromise;
}
self.onmessage = async ({ data }) => {
const { id, type, payload } = data;
try {
let result;
switch (type) {
case "init": {
const transformers = await importVerifiedModule(
payload.url,
payload.integrity
);
pipeline = transformers.pipeline;
result = true;
break;
}
case "classify": {
const classifier = await getClassifier();
result = (await classifier(payload.text))[0];
break;
}
case "generate": {
const generator = await getGenerator();
const output = await generator(payload.messages);
result = output[0].generated_text.at(-1).content;
break;
}
default:
throw new Error(`Unknown worker request: ${type}`);
}
self.postMessage({ id, result });
} catch (error) {
self.postMessage({
id,
error: error instanceof Error ? error.message : String(error)
});
}
};
}Then create the module worker like this: let worker;
let workerURL;
function getWorker() {
if (worker) return worker;
const source = `(${workerMain.toString()})();`;
workerURL = URL.createObjectURL(
new Blob([source], { type: "text/javascript" })
);
worker = new Worker(workerURL, { type: "module" });
worker.onerror = ({ message }) => {
console.error("SB3.AI worker failed:", message);
};
return worker;
}The important limitation: Alternatives without another file are not really better:
One caveat: this does not avoid a CSP requirement for |
|
@coderabbitai @OmniBlocks/boxy @supervoidcoder Added web worker |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
FILE="$(fd -a '^sb3ai\.js$' . | head -n 1)"
if [ -z "$FILE" ]; then
echo "sb3ai.js not found"
exit 1
fi
echo "--- file: $FILE ---"
ast-grep outline "$FILE" --items all || true
echo "--- worker and inference locations ---"
rg -n -C 6 \
'new Worker|Worker\(|workerMain|workerSource|postMessage|onmessage|onerror|pipeline\s*\(|classify\s*\(|generate\s*\(|classifyImage\s*\(|URL\.revokeObjectURL|pending' \
"$FILE"
echo "--- complete source ---"
cat -n "$FILE"
echo "--- diff against PR base (extension only) ---"
BASE="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main 2>/dev/null || true)"
if [ -n "$BASE" ]; then
git diff --no-ext-diff --unified=60 "$BASE"...HEAD -- "$FILE"
else
git show --no-ext-diff --format= --unified=60 HEAD -- "$FILE"
fiLength of output: 18043
The If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! |
|
@coderabbitai DO NOT MENTION ELON PLATFORM |
|
Tip For best results, initiate chat on the files or code changes.
🐇
|
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
extensions/Oisthebestletter/sb3ai.js (2)
44-68: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winClear failed pipeline promises so users can retry.
A transient model download or initialization error leaves a rejected promise cached forever, causing every later block invocation to fail until reload. Reset each cache on rejection while retaining in-flight deduplication.
Proposed fix
- classifierPromise = pipeline("sentiment-analysis"); + classifierPromise = pipeline("sentiment-analysis").catch((error) => { + classifierPromise = null; + throw error; + });Apply the same pattern to
generatorPromiseandimageClassifierPromise.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@extensions/Oisthebestletter/sb3ai.js` around lines 44 - 68, Update getClassifier, getGenerator, and getImageClassifier so each cached pipeline promise clears its corresponding cache variable when initialization or download rejects, while retaining the promise during in-flight requests for deduplication. Ensure the rejection remains propagated to the caller so later invocations can retry.
53-66: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy liftPin the Hugging Face model revisions here
pipeline()will fetch mutable weights, tokenizer, and config forHuggingFaceTB/SmolLM2-135M-InstructandXenova/vit-base-patch16-224. Pass a specificrevisioncommit hash for each model, or self-host the artifacts, so the extension’s supply-chain guarantee doesn’t depend on upstream defaults.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@extensions/Oisthebestletter/sb3ai.js` around lines 53 - 66, Pin both Hugging Face pipelines to immutable model revisions by adding the appropriate specific commit-hash revision to the pipeline options in getGenerator and getImageClassifier for HuggingFaceTB/SmolLM2-135M-Instruct and Xenova/vit-base-patch16-224. Preserve the existing lazy initialization and model selections.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@extensions/Oisthebestletter/sb3ai.js`:
- Around line 10-13: Update the module-loading flow around the native fetch call
to use Scratch.fetch on the main thread, verify the fetched module bytes there,
and transfer those verified bytes into the serialized worker for blob import; do
not reference Scratch from the worker.
- Around line 132-175: Update getWorker, workerRequest, and the worker
message-handler setup so worker failures reject and clear every entry in
pending, terminate the failed worker, and reset the cached worker before future
requests reuse it. Attach both onerror and onmessage handlers whenever a worker
is created, rather than assigning onmessage only to the initial getWorker()
result, while preserving normal RPC resolution and error rejection.
- Line 8: Wrap the complete extension implementation around workerMain and all
related declarations in the required Scratch IIFE, including "use strict"; and
invoking it with Scratch. Also add "use strict"; inside workerMain because it is
serialized independently.
---
Outside diff comments:
In `@extensions/Oisthebestletter/sb3ai.js`:
- Around line 44-68: Update getClassifier, getGenerator, and getImageClassifier
so each cached pipeline promise clears its corresponding cache variable when
initialization or download rejects, while retaining the promise during in-flight
requests for deduplication. Ensure the rejection remains propagated to the
caller so later invocations can retry.
- Around line 53-66: Pin both Hugging Face pipelines to immutable model
revisions by adding the appropriate specific commit-hash revision to the
pipeline options in getGenerator and getImageClassifier for
HuggingFaceTB/SmolLM2-135M-Instruct and Xenova/vit-base-patch16-224. Preserve
the existing lazy initialization and model selections.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 2580fbdf-adde-4452-a1c1-7300643327db
📒 Files selected for processing (1)
extensions/Oisthebestletter/sb3ai.js
| // License: MPL-2.0 | ||
| // Tags: ai llm ml machine clanker bot | ||
|
|
||
| function workerMain() { |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Wrap the implementation in the required extension IIFE.
These top-level declarations fail the repository’s extension/iife and strict checks. Wrap the complete extension in (function (Scratch) { "use strict"; ... })(Scratch);; because workerMain is serialized independently, add "use strict"; inside it as well.
🧰 Tools
🪛 ESLint
[error] 8-119: All extension code must be within (function (Scratch) { ... })(Scratch);
(extension/iife)
[error] 8-119: Use the function form of 'use strict'.
(strict)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@extensions/Oisthebestletter/sb3ai.js` at line 8, Wrap the complete extension
implementation around workerMain and all related declarations in the required
Scratch IIFE, including "use strict"; and invoking it with Scratch. Also add
"use strict"; inside workerMain because it is serialized independently.
Source: Linters/SAST tools
| const response = await fetch(url, { | ||
| cache: "no-cache", | ||
| credentials: "omit", | ||
| }); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Move the module fetch through Scratch.fetch.
Native fetch fails extension/use-scratch-fetch, while the serialized worker cannot access Scratch. Fetch and verify the module on the main thread using Scratch.fetch, then transfer the verified bytes to the worker for blob import.
🧰 Tools
🪛 ESLint
[error] 10-13: Use Scratch.fetch() instead of fetch()
(extension/use-scratch-fetch)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@extensions/Oisthebestletter/sb3ai.js` around lines 10 - 13, Update the
module-loading flow around the native fetch call to use Scratch.fetch on the
main thread, verify the fetched module bytes there, and transfer those verified
bytes into the serialized worker for blob import; do not reference Scratch from
the worker.
Source: Linters/SAST tools
| worker = new Worker(workerURL, { type: "module" }); | ||
| worker.onerror = ({ message }) => { | ||
| console.error("SB3.AI worker failed:", message); | ||
| }; | ||
|
|
||
| URL.revokeObjectURL(workerURL); | ||
| workerURL = null; | ||
|
|
||
| return worker; | ||
| } | ||
|
|
||
| let requestId = 0; | ||
| const pending = new Map(); | ||
|
|
||
| function workerRequest(type, payload = {}) { | ||
| const worker = getWorker(); | ||
|
|
||
| return new Promise((resolve, reject) => { | ||
| const id = ++requestId; | ||
|
|
||
| pending.set(id, { resolve, reject }); | ||
|
|
||
| worker.postMessage({ | ||
| id, | ||
| type, | ||
| payload, | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| getWorker().onmessage = ({ data }) => { | ||
| const { id, result, error } = data; | ||
|
|
||
| const promise = pending.get(id); | ||
| if (!promise) return; | ||
|
|
||
| pending.delete(id); | ||
|
|
||
| if (error) { | ||
| promise.reject(new Error(error)); | ||
| } else { | ||
| promise.resolve(result); | ||
| } | ||
| }; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Reject pending RPCs when the worker fails.
worker.onerror only logs, leaving every pending block promise unresolved and future requests targeting a failed worker. Reject and clear pending, terminate/reset the worker, and attach both lifecycle handlers whenever a worker is created.
🧰 Tools
🪛 ESLint
[error] 143-143: All extension code must be within (function (Scratch) { ... })(Scratch);
(extension/iife)
[error] 144-144: All extension code must be within (function (Scratch) { ... })(Scratch);
(extension/iife)
[error] 146-160: All extension code must be within (function (Scratch) { ... })(Scratch);
(extension/iife)
[error] 146-160: Wrap function 'workerRequest' in a function with 'use strict' directive.
(strict)
[error] 162-175: All extension code must be within (function (Scratch) { ... })(Scratch);
(extension/iife)
[error] 162-175: Wrap arrow function in a function with 'use strict' directive.
(strict)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@extensions/Oisthebestletter/sb3ai.js` around lines 132 - 175, Update
getWorker, workerRequest, and the worker message-handler setup so worker
failures reject and clear every entry in pending, terminate the failed worker,
and reset the cached worker before future requests reuse it. Attach both onerror
and onmessage handlers whenever a worker is created, rather than assigning
onmessage only to the initial getWorker() result, while preserving normal RPC
resolution and error rejection.
SummaryThis PR introduces the SB3.AI extension, bringing the power of pretrained AI models (via The extension provides a suite of AI tools:
Technically, the PR is very well-thought-out, utilizing a Web Worker to keep the heavy AI computations off the main thread and a dynamic overlay canvas to render detections on top of the stage without modifying the renderer. However, there are a couple of critical bugs regarding how data is sent to the worker and how the AI output is processed that need to be addressed before this can be merged. Logic Flowgraph TD
A[Scratch Block] --> B[AI Class Method]
B --> C[workerRequest]
C --> D[Web Worker]
D --> E[Transformers.js Pipeline]
E --> F[Model Prediction]
F --> G[PostMessage Result]
G --> H[Promise Resolve]
H --> I[Return to Scratch Block]
Boxy's Corner
GUI ScreenshotsNo screenshots provided as the PR is currently a background implementation of AI logic and canvas overlays. |
There was a problem hiding this comment.
Hey @playforge-coding! Thanks for adding the SB3.AI extension! 🎉 It's super cool to see AI capabilities coming to OmniBlocks.
However, I've found a few critical bugs that need fixing before this can be merged:
1. Broken Text Generation Logic (Line 116)
The generate block tries to extract text using output[0].generated_text.at(-1).content. In transformers.js, generated_text is a string (not an array of objects), so this will return undefined. I suggest returning the full string or stripping the prompt.
2. Canvas Transmission to Worker (Line 463)
detectStageAndDraw sends an HTMLCanvasElement to the worker. Canvas elements can't be cloned/transferred to workers! This will crash the object detection. You need to convert the canvas to an ImageBitmap and transfer it.
3. Missing transfer support in workerRequest (Line 182)
To support the ImageBitmap, workerRequest needs to accept a transfer array for postMessage.
4. Confusing Function Naming (Line 471)
The clearDetectionOverlay method calls a function with the same name in the outer scope. It works, but it's confusing. Renaming the helper function is better.
Please fix these issues, and I'll be happy to approve! Once fixed, this extension will be incredibly powerful for OmniBlocks users. 🚀
SUMMARYThe PR introduces a new, highly ambitious extension called SB3.AI. This extension leverages the Key Features:
Technical Highlights:
Potential Improvements for Stability and Usability:
Logic Flowgraph TD
A[Extension Loaded] --> B[Initialize Worker]
B --> C{Init Message}
C -->|Yes| D[Fetch & Verify Transformers Module]
D --> E[Worker Ready]
E --> F{User Block Called}
F -->|Classify| G[Worker: Sentiment Analysis]
F -->|Generate| H[Worker: Text Generation]
F -->|Detect| I[Main: Snapshot Stage]
I --> J[Transfer ImageBitmap to Worker]
J --> K[Worker: Object Detection]
K --> L[Return Detections]
L --> M[Main: Draw on Overlay Canvas]
F -->|Stage Image| N[Main: Convert Snapshot to Data URI]
N --> O[Return Data URI]
Boxy's Poem
GUI Screenshots(No screenshots provided) |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@extensions/Oisthebestletter/sb3ai.js`:
- Around line 454-473: Update detectStageAndDraw and clearDetectionOverlay to
use a shared generation/token that invalidates pending detections. Capture the
token before awaiting snapshotStage and workerRequest, and only call
drawDetections if it still matches when detection completes; increment the token
in clearDetectionOverlay before clearing so stale or older concurrent results
cannot redraw.
- Line 350: Wrap the user-facing block labels at the affected entries in
Scratch.translate(), including the labels near “unmask [MASKED_TEXT]” and the
other referenced entries, so extension/should-translate passes and the text
remains localizable.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 301a2996-1b8b-4144-966b-cef2edda9fef
📒 Files selected for processing (1)
extensions/Oisthebestletter/sb3ai.js
| { | ||
| opcode: "unmask", | ||
| blockType: Scratch.BlockType.REPORTER, | ||
| text: "unmask [MASKED_TEXT]", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== File around cited lines ==\n'
sed -n '330,410p' extensions/Oisthebestletter/sb3ai.js | cat -n
printf '\n== Search for Scratch.translate in file ==\n'
rg -n "Scratch\.translate|should-translate|unmask \[MASKED_TEXT\]|detect objects on stage and draw boxes with confidence \[THRESHOLD\]%|clear detection boxes" extensions/Oisthebestletter/sb3ai.js
printf '\n== Search for should-translate rule references in repo ==\n'
rg -n "extension/should-translate|should-translate" .Repository: OmniBlocks/extensions
Length of output: 10371
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '330,410p' extensions/Oisthebestletter/sb3ai.js | cat -n
printf '\n---\n'
rg -n "Scratch\.translate|extension/should-translate|should-translate" extensions/Oisthebestletter .github .eslintrc* eslint* package.jsonRepository: OmniBlocks/extensions
Length of output: 3854
🌐 Web query:
Scratch extension block labels Scratch.translate localization should-translate ESLint
💡 Result:
In Scratch extension development, localizing block labels is handled primarily through the Scratch translation system, which utilizes the ICU message format [1][2]. For custom extensions, developers often use Scratch.translate() to ensure block labels and other strings can be localized [3][4]. This function accepts either a simple string or an object containing a unique identifier (id), a default English string, and optionally a description for translators [3][4]. To provide translations for different languages, developers use Scratch.translate.setup() to define a map of translations for specific locales before the extension registers [3]. While official Scratch extensions (core, team, and official) use formatMessage to integrate with the official Transifex localization workflow [1][2], custom or community-created extensions generally rely on Scratch.translate() and custom setup calls [3]. Regarding ESLint, there is no widely recognized standard ESLint rule specifically named "should-translate" for Scratch extension development. It is likely that such a name refers to a custom linting rule or a specific project requirement intended to enforce the use of localization functions for user-facing strings [5]. Key practices for localization include: - Using Scratch.translate() for all user-facing text, including block names and argument labels [3][5]. - Providing unique IDs for strings to prevent conflicts [3]. - Defining translations via Scratch.translate.setup() to support multiple languages within the extension [3]. - Avoiding hardcoded strings in block definitions [3]. If you are encountering an ESLint error regarding translation, it is recommended to check your local .eslintrc configuration or the custom rules defined in your specific extension repository, as this is not a core requirement of the standard Scratch VM [1][2].
Citations:
- 1: https://github.com/LLK/scratch-vm/blob/develop/docs/extensions.md
- 2: https://github.com/scratchfoundation/scratch-vm/blob/develop/docs/extensions.md
- 3: Multilingual custom extension TurboWarp/extensions#1168
- 4: https://docs.warp.mistium.com/extensions/apis/scratch-api
- 5: https://github.com/TurboWarp/extensions/pull/2375/files/bb128d37c75b4c61b61d853b2a451dded47293ef
Wrap these block labels in Scratch.translate(). extensions/Oisthebestletter/sb3ai.js:350, 378, 389 uses hardcoded user-facing text, so extension/should-translate will flag it and the labels won’t be localizable.
🧰 Tools
🪛 ESLint
[error] 350-350: Block text should usually be translated
(extension/should-translate)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@extensions/Oisthebestletter/sb3ai.js` at line 350, Wrap the user-facing block
labels at the affected entries in Scratch.translate(), including the labels near
“unmask [MASKED_TEXT]” and the other referenced entries, so
extension/should-translate passes and the text remains localizable.
Source: Linters/SAST tools
| async detectStageAndDraw({ THRESHOLD }) { | ||
| const threshold = Math.max( | ||
| 0, | ||
| Math.min(1, Scratch.Cast.toNumber(THRESHOLD) / 100) | ||
| ); | ||
|
|
||
| // This captures the renderer canvas only, not the DOM overlay. | ||
| const stageImage = await snapshotStage(); | ||
|
|
||
| const detections = await workerRequest("detectObjects", { | ||
| image: stageImage, | ||
| threshold, | ||
| }); | ||
|
|
||
| drawDetections(detections); | ||
| } | ||
|
|
||
| clearDetectionOverlay() { | ||
| _clearDetectionOverlay(); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
set -euo pipefail
grep -n "detectStageAndDraw\|clearDetectionOverlay\|detectionGeneration\|drawDetections\|snapshotStage\|workerRequest" -n extensions/Oisthebestletter/sb3ai.js
printf '\n--- context ---\n'
sed -n '420,520p' extensions/Oisthebestletter/sb3ai.jsRepository: OmniBlocks/extensions
Length of output: 2235
🏁 Script executed:
set -euo pipefail
sed -n '205,285p' extensions/Oisthebestletter/sb3ai.jsRepository: OmniBlocks/extensions
Length of output: 2597
Invalidate in-flight detections when clearing. clearDetectionOverlay() only clears the canvas; any pending detectStageAndDraw() call can redraw stale boxes when it resolves, and older concurrent detections can still overwrite newer results. A generation/token check would handle both.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@extensions/Oisthebestletter/sb3ai.js` around lines 454 - 473, Update
detectStageAndDraw and clearDetectionOverlay to use a shared generation/token
that invalidates pending detections. Capture the token before awaiting
snapshotStage and workerRequest, and only call drawDetections if it still
matches when detection completes; increment the token in clearDetectionOverlay
before clearing so stale or older concurrent results cannot redraw.
There was a problem hiding this comment.
Overall, this is an absolutely impressive addition to OmniBlocks! The architecture is solid, and the feature set is amazing. 🚀
I've requested a few small changes to ensure the extension is rock-solid:
- Worker Compatibility: Convert stage snapshots to
ImageBitmapbefore sending them to the worker, asCanvaselements aren't transferable. - Robustness: Add a check for the
generateblock's output to handle both string and array responses. - Consistency: Use
Scratch.vmconsistently instead of mixing it withwindow.vm.
Once these are tidied up, this will be a fantastic addition! Great work @playforge-coding! 🤖✨
Co-authored-by: boxycpu[bot] <239043987+boxycpu[bot]@users.noreply.github.com>
🤖 SB3.AI Extension ReviewThis PR introduces a sophisticated AI extension for OmniBlocks, leveraging the 📝 Summary of Changes
🛠️ Technical FindingsWhile the implementation is impressive, there are a few critical issues that need to be addressed:
📉 Logic Flowgraph TD
A[Scratch Block] --> B{Block Type}
B -- Reporter --> C[AI Class Method]
B -- Command --> D[AI Class Method]
C --> E[workerRequest]
D --> E
E --> F[Web Worker]
F --> G{Request Type}
G -- init --> H[Import Verified Module]
G -- classify/generate/etc --> I[Transformers.js Pipeline]
I --> J[Result]
J --> K[postMessage back to main]
K --> L[Resolve Promise]
L --> C
L --> D
D -- detectStageAndDraw --> M[Draw Overlay on Canvas]
📦 Boxy's Corner
GUI Screenshots(No screenshots provided. The extension adds blocks but doesn't modify the core GUI) |
There was a problem hiding this comment.
Hi @playforge-coding! 🤖
This is an awesome extension! Bringing Transformers.js to OmniBlocks is a huge win, and I love how you handled the Web Worker and the secure module loading. 🚀
However, there's one critical bug and a few usability tweaks needed before this can be merged:
- Worker Crash: The
detectStageAndDrawblock will crash with aDataCloneErrorbecause it tries to send anHTMLCanvasElementto the worker. Please convert it to anImageBitmapusingcreateImageBitmap(). - Reporter Value: The
stageimgblock returns a canvas object, which Scratch will just show as"[object HTMLCanvasElement]". Please change it to return a Data URI using.toDataURL(). - Consistency: Please align the extension IDs between the header and
getInfo(), and don't forget to add the extension toextensions/extensions.json! - Robustness: I suggested a small change to the
generateoutput handling to ensure it doesn't break if the model returns a string instead of a message array.
Check out my inline comments for the details and suggestions! Once these are fixed, I'll be happy to approve. Happy coding! 📦✨
| async detectStageAndDraw({ THRESHOLD }) { | ||
| const threshold = Math.max( | ||
| 0, | ||
| Math.min(1, Scratch.Cast.toNumber(THRESHOLD) / 100) | ||
| ); | ||
|
|
||
| // This captures the renderer canvas only, not the DOM overlay. | ||
| const stageImage = await snapshotStage(); | ||
|
|
||
| const detections = await workerRequest("detectObjects", { | ||
| image: stageImage, | ||
| threshold, | ||
| }); |
There was a problem hiding this comment.
DataCloneError.
HTMLCanvasElement cannot be sent through postMessage to a Web Worker. You need to convert the canvas to an ImageBitmap or an ArrayBuffer (pixel data) before sending it.
I suggest doing something like this:
| async detectStageAndDraw({ THRESHOLD }) { | |
| const threshold = Math.max( | |
| 0, | |
| Math.min(1, Scratch.Cast.toNumber(THRESHOLD) / 100) | |
| ); | |
| // This captures the renderer canvas only, not the DOM overlay. | |
| const stageImage = await snapshotStage(); | |
| const detections = await workerRequest("detectObjects", { | |
| image: stageImage, | |
| threshold, | |
| }); | |
| async detectStageAndDraw({ THRESHOLD }) { | |
| const threshold = Math.max( | |
| 0, | |
| Math.min(1, Scratch.Cast.toNumber(THRESHOLD) / 100) | |
| ); | |
| const stageCanvas = await snapshotStage(); | |
| const imageBitmap = await createImageBitmap(stageCanvas); | |
| const detections = await workerRequest("detectObjects", { | |
| image: imageBitmap, | |
| threshold, | |
| }); | |
| drawDetections(detections); | |
| } |
Note: Sending an ImageBitmap is efficient because it can be transferred to the worker.
| async stageimg() { | ||
| return await snapshotStage(); | ||
| } |
There was a problem hiding this comment.
💡 Usability Suggestion:
Currently, stageimg returns a HTMLCanvasElement. When used in a Scratch reporter block, it will just return the string "[object HTMLCanvasElement]".
To make it useful for users (e.g., for sending the image to a web API), it's better to return a Data URI.
| async stageimg() { | |
| return await snapshotStage(); | |
| } | |
| async stageimg() { | |
| const canvas = await snapshotStage(); | |
| return canvas.toDataURL(); | |
| } |
| class AI { | ||
| getInfo() { | ||
| return { | ||
| id: "sb3ai", |
There was a problem hiding this comment.
🧐 Consistency Check:
The ID in the file header is oisthebestletter-sb3ai, but the ID in getInfo() is sb3ai. It's usually better if these match or if the getInfo ID is clearly intended to be the Scratch block ID.
Also, ensure the extension is registered in extensions/extensions.json so it's available to everyone!
| case "generate": { | ||
| const generator = await getGenerator(); | ||
| const output = await generator(payload.messages); | ||
| result = output[0].generated_text.at(-1).content; |
There was a problem hiding this comment.
🤔 Question/Suggestion:
You're using output[0].generated_text.at(-1).content. This assumes that generated_text is an array of message objects.
Depending on the version of @huggingface/transformers and the model's configuration, generated_text might be a plain string (the entire output). If it's a string, .at(-1) will be the last character, and .content will be undefined.
I'd recommend adding a check to handle both cases:
| result = output[0].generated_text.at(-1).content; | |
| case "generate": { | |
| const generator = await getGenerator(); | |
| const output = await generator(payload.messages); | |
| const text = output[0].generated_text; | |
| if (Array.isArray(text)) { | |
| result = text.at(-1).content; | |
| } else { | |
| result = text; | |
| } | |
| break; | |
| } |
| // Tags: ai llm ml machine clanker bot | ||
|
|
||
| function workerMain() { | ||
| async function importVerifiedModule(url, expectedIntegrity) { |
There was a problem hiding this comment.
🌟 Excellent Work!:
The importVerifiedModule implementation is fantastic! Using SHA-512 integrity checks for a CDN-loaded module is a very professional and secure way to handle third-party dependencies in an unsandboxed environment. 🚀
|
@OmniBlocks/boxy It works fine already, stageimg already returns a data uri and the worker does not crash. |
|
Used command-a-plus-05-2026 |
🤖 SB3.AI Extension ReviewThis PR introduces a powerful AI extension for OmniBlocks, allowing users to perform sentiment analysis, text generation, mask filling, image classification, and object detection directly within their projects using pretrained models from Hugging Face. Summary of Changes
Logic Flowgraph TD
A[Scratch Block Called] --> B{Block Type}
B -- Text AI --> C[workerRequest type, payload]
B -- Image AI --> D[snapshotStage]
D --> C
C --> E[Web Worker]
E --> F{Request Type}
F -- init --> G[importVerifiedModule]
F -- classify --> H[sentiment-analysis pipeline]
F -- generate --> I[text-generation pipeline]
F -- unmask --> J[fill-mask pipeline]
F -- classifyImage --> K[image-classification pipeline]
F -- detectObjects --> L[object-detection pipeline]
G --> M[Return Success]
H --> M
I --> M
J --> M
K --> M
L --> M
M --> N[Main Thread]
N -- if detectObjects --> O[drawDetections on Overlay Canvas]
N -- others --> P[Return result to Scratch]
📦 Boxy's Poem
GUI Screenshots(No screenshots provided as this PR adds new blocks and a dynamic overlay that requires live execution to capture) |
|
lets see if boxy can give a more competenet review this time |
There was a problem hiding this comment.
Overall, this is an impressive addition to OmniBlocks! The use of Web Workers and integrity checks shows a lot of care for both performance and security. The object detection overlay is a particularly nice touch.
I've left a few positive comments and one small suggestion for the generate block to make it more robust when handling different output formats from the AI model.
Great work @playforge-coding! 🚀🤖
| new Blob([source], { type: "text/javascript" }) | ||
| ); | ||
|
|
||
| worker = new Worker(workerURL, { type: "module" }); |
There was a problem hiding this comment.
Great job using a Web Worker here! Offloading these heavy AI models to a separate thread is essential for keeping the OmniBlocks UI smooth and responsive. 🚀
| const digest = await crypto.subtle.digest("SHA-512", bytes); | ||
| const actualIntegrity = | ||
| "sha512-" + btoa(String.fromCharCode(...new Uint8Array(digest))); | ||
|
|
||
| if (actualIntegrity !== expectedIntegrity) { |
There was a problem hiding this comment.
I love the security consciousness! Verifying the integrity of the fetched module with a SHA-512 hash is a top-tier practice. Keep it up! 🛡️
| case "generate": { | ||
| const generator = await getGenerator(); | ||
| const output = await generator(payload.messages); | ||
| result = output[0].generated_text.at(-1).content; |
There was a problem hiding this comment.
This works perfectly for chat models, but we might want to make it a bit more robust in case generated_text isn't in the expected array format. Just a small suggestion to avoid potential errors!
| case "generate": { | |
| const generator = await getGenerator(); | |
| const output = await generator(payload.messages); | |
| result = output[0].generated_text.at(-1).content; | |
| case "generate": { | |
| const generator = await getGenerator(); | |
| const output = await generator(payload.messages); | |
| const generatedText = output[0]?.generated_text; | |
| if (Array.isArray(generatedText) && generatedText.length > 0) { | |
| result = generatedText.at(-1).content; | |
| } else if (typeof generatedText === 'string') { | |
| result = generatedText; | |
| } else { | |
| result = "Error: No text generated"; | |
| } | |
| break; | |
| } |
| function drawDetections(detections) { | ||
| const { context, width, height } = getOverlayContext(); |
There was a problem hiding this comment.
This overlay implementation is super clever! Using a separate canvas that follows the stage is a great way to show object detection without messing with the actual stage rendering. 🎨✨
| if (!Scratch.extensions.unsandboxed) { | ||
| alert("SB3.AI must be ran unsandboxed!"); |
There was a problem hiding this comment.
Good call on checking for the unsandboxed environment early. It saves the user a lot of confusion if the extension fails to work as expected! ✅
2 participants
This is still WIP, I need your feedback pls.Should be ready
@coderabbitai @OmniBlocks/boxy