Skip to content

Blog: Inside Kueue : How Kubernetes decide what runs next - #266

Merged
saiyam1814 merged 2 commits into
kubesimplify:mainfrom
ekam-walia:blog-inside-kueue
Sep 16, 2026
Merged

saiyam1814 merged 2 commits into
kubesimplify:mainfrom
ekam-walia:blog-inside-kueue

Conversation

@ekam-walia

@ekam-walia ekam-walia commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes Issue

NONE

Changes proposed

Added a Blog

Screenshots

NONE

Note to reviewers

This PR covers the blog Inside Kueue: How Kubernetes decide what runs next

@ekam-walia

Copy link
Copy Markdown
Contributor Author

cc @saiyam1814

@ekam-walia ekam-walia changed the title Blog inside kueue Blog: Inside Kueue : How Kubernetes decide what runs next Aug 19, 2026

@saiyam1814 saiyam1814 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @ekam-walia, thanks for sending this in. I read the whole thing properly and also got a preview build up so I could see it rendered rather than just reading the diff.

Quick note on that: Cloudflare doesn't build previews for PRs that come from forks, so I pushed your commits to a branch on our repo to get one. You can see the post live here:

https://blog-inside-kueue.website-dab.pages.dev/inside-kueue-how-kubernetes-decides-what-runs-next

Worth opening that yourself before you start editing, because a few things read fine in the markdown but fall apart on the page.

The good part first, and I mean it. Your instinct for explaining things is the strongest thing about this post. The "four team members, one still waiting for a computer" framing of gang scheduling is the clearest version of that idea I have read anywhere, and Demo 1 has a real story to it: job A takes the room, job B waits, A leaves, B gets in. That is exactly the shape a good hands-on post has. The microservices vs batch comparison table is useful too.

So this is worth finishing. But I can't merge it as it stands, and I would rather be straight with you about why than send it back with vague notes.

What's blocking merge

minimumReservedResources is not a real Kueue field. Strategy 2 is built entirely on it, and it doesn't exist in the API. A ClusterQueue resource entry takes nominalQuota, borrowingLimit, and lendingLimit, and that's it (see the ClusterQueue concepts page). The API server would reject that YAML. If part of this came out of an AI draft, no judgment at all, that is a fine way to get a first draft, but everything has to get applied to a real cluster before it ships. The genuine answers to "job B is starving" are workload priorities with preemption, or cohorts with borrowingLimit so one queue can lend spare capacity without giving up its floor. Either one makes a better Strategy 2 than what's there now.

Strategy 1 would break admission rather than prevent deadlock. Both demo jobs request CPU and memory. A ClusterQueue whose coveredResources only lists cpu can't admit a workload that also asks for memory, it gets marked inadmissible. So splitting into a cpu-only queue and a memory-only queue doesn't route jobs by their dominant resource, it just stops both jobs from ever being admitted. If the point is isolation between job types, that's separate ClusterQueues each covering both resources, optionally in a cohort.

The version is about two years stale. You're pinning v0.6.1 and current is v0.19.2. Please re-run on a current release and update the URL in both places it appears (install step and the Resources list at the end).

The command output looks reconstructed rather than pasted. A few tells: kubectl get workloads shows the generated workload name, so it would be something like job-job-a-big-2f4c1, not job-a-big. The QUEUE column shows the LocalQueue the job went through, which in your setup is default, not demo-queue. And kubectl describe workload job-b-small wouldn't resolve, since that's the Job's name and not the workload's. Same thing in Demo 2. "Real commands, real output" is honestly the main thing we ask for (it's the first bullet under "What we look for" in WRITING-A-BLOG.md), and it's also the thing readers can smell instantly. Please paste what your terminal actually printed, hashes and odd column widths and all.

Demo 2 isn't really a deadlock. Job B waiting for job A to finish its sleep 600 is queuing, or starvation if you want to be dramatic about it. It resolves on its own, which the post half admits when it says "when Job A finishes, resources are released". The actual deadlock is the one you describe correctly up in the gang scheduling section: a distributed job gets 3 of its 4 pods placed, those 3 hold resources, and nothing progresses. That's the thing worth demoing. Either rename Demo 2 to what it is, or build the real one with two gang jobs competing.

Render issues

All of these are visible on the preview link:

  • Eight <h1>s on the page. Body headings need to start at ##, since the frontmatter title is already the H1. Right now seven of your section headings compete with the title and clutter the table of contents.
  • Prose stuck inside a code block. Under Strategy 2 the "How it works" explanation renders as YAML, because the fence never gets closed before it. Close the fence after the last YAML line.
  • A stray ** renders literally in "Strategy 2: Use minimumReservedResources**".
  • The arrow flowcharts collapse into run-on paragraphs. Markdown eats single newlines, so "Pod lands → Scheduler checks if it fits on any node." and the two lines under it become one blob. Same problem hits the "What Just Happened" recap, the Pod 1 through Pod 4 dependency chain, the car analogy, and the three links at the end. Make those bullet lists, or wrap them in a ```text block if you want the arrow-diagram look.
  • Unfenced output in Demo 1 Step 8: the Conditions: block runs together as a paragraph while every other output block is fenced.
  • Headings used as emphasis. "##### Kueue checks: Can I fit all 4?" and "##### Queue is stuck" are sentence fragments sitting in the heading hierarchy and the TOC. Bold text does that job.
  • Missing space: "Which job gets them?Jobs from all namespaces compete here" (and the quote before it never closes).

Style, small stuff

  • No em dashes on the blog. Two here: the seoDescription ("workloads—with a hands on demo") and "someone deploys a Batch Job—maybe it's...". A comma or a period does the job. While you're in that line, "hands on" wants a hyphen: hands-on.
  • Tags should be lowercase, hyphenated, 2 to 5 of them, no slashes. "AI/ML" breaks the tag page URL because of the slash. I'd go with ["kubernetes", "kueue", "scheduling", "batch-workloads"].
  • Images are heavy: cover is 1.2MB, and the two diagrams are 1.6MB and 1.8MB. Please get them under ~300KB each, WebP if you can.
  • Typos worth a pass: "schedular" should be scheduler, "Core principal" should be principle, "it's fix" should be its fix, and "until desired resources don't get vacant" reads oddly (try "until resources free up"). The heading "Now Lets jump on to how Fix this DEADLOCK" needs a rewrite, and "Now let's first understand what Batch Jobs are and the concept of Gang Scheduling, also how they create a DEADLOCK Issue through an example" is a whole sentence doing a heading's job. Curly quotes have crept in from a word processor in a few spots, straight quotes please.
  • The post opens directly on a heading and ends on a bare link dump. Two or three sentences up top saying who this is for, and a short wrap-up at the end, would round it off.

One process thing

None of the four commits have a DCO sign-off, which we do require. Easiest fix is to squash them into one:

git reset --soft origin/main
git commit -s -m "Add blog post: Inside Kueue: How Kubernetes Decides What Runs Next"
git push --force

That also cleans up the commit messages, which are currently "..", "...", and "Is this reqd".


I know that's a long list. It isn't a rejection, and none of it is about your voice, which is fine. It comes down to: fix the two strategies so they use mechanisms Kueue actually has, re-run the demos on a current version and paste what you really saw, and clean up the heading levels. Do that and this is a solid post that I'll be glad to publish with your byline on it.

Ping me here when you've pushed and I'll refresh the preview and take another look. If you get stuck on the priorities/preemption or cohort setup, ask, that part of Kueue is genuinely fiddly.

@ekam-walia
ekam-walia force-pushed the blog-inside-kueue branch 2 times, most recently from 32a4d31 to c829c59 Compare August 25, 2026 14:34
@ekam-walia

Copy link
Copy Markdown
Contributor Author

Hi @saiyam1814, I have made all the suggested changes, I have added a new block The Real Deadlock Demo (Gang Scheduling) in which i have explained why and how deadlock occurs, thru examples and have also added yaml files that can cause Deadlock Scenario and how Kueue resolves it

@saiyam1814 saiyam1814 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ekam-walia, this is a big step up and I can see the work that went into it. Most of what I flagged is genuinely fixed, so let me confirm that first and then give you the short list that's still open.

Fixed and verified:

  • minimumReservedResources is gone, and so is the Strategy 1 section that would have broken admission. Every coveredResources block now lists both cpu and memory.
  • You're on v0.19.2 instead of v0.6.1.
  • The workload names in the output are real now (job-job-a-big-cb1a1, job-job-b-small-c8c54), the QUEUE column shows the LocalQueue with RESERVED IN alongside it, and describe targets the workload rather than the job. That reads like an actual terminal, which is exactly what I was after. The couldn't assign flavors to pod set main: insufficient unused quota for cpu in flavor default, 3 more needed message is a nice touch.
  • All eight H1s are gone, the fences are balanced, the stray ** is gone.
  • Tags are down to four, lowercase and hyphenated.
  • Images are WebP and all under 105KB, down from 1.6MB and 1.8MB.
  • schedular, principal, "it's fix" and "don't get vacant" are all cleaned up.
  • Commits squashed to one with a proper DCO sign-off and a real message.

That's most of the list. Here's what's left, and unfortunately the first two are the kind that have to be right before it goes out.

The install command is broken

Line 219 currently reads:

kubectl apply -f kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.2/manifests.yaml

The kubectl apply -f prefix got duplicated, so the shell would treat the literal string kubectl as a filename. This is the very first command anyone runs, so the tutorial stops there. The version in your Resources section at the bottom is correct, just copy that one up:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.2/manifests.yaml

The closing takeaway isn't right

The wrap-up ends on this:

The best part? One line fixes it: suspend: true. That's it.

That's not what does the work, and I'd hate for the last thing a reader takes away to be the one wrong thing. What enrolls a Job with Kueue is the kueue.x-k8s.io/queue-name label. Kueue's webhook then sets spec.suspend itself when it admits or holds the job. You can confirm it from Kueue's own sample job, which has the queue-name label and no suspend field anywhere:

https://github.com/kubernetes-sigs/kueue/blob/main/site/static/examples/jobs/sample-job.yaml

This also affects your "Before Kueue" and "After Kueue" pair, because both of those YAMLs already carry the queue-name label. As written, Kueue would manage both of them identically, so the only difference you're showing isn't the thing doing the work. Rewriting that section around the label (and mentioning that Kueue handles suspension for you) would land the point properly, and it's honestly a better ending: you add one label and Kueue takes over admission.

Worth a look while you're in there: job-b-needs-6-gets-3 has completions: 1 with parallelism: 2, which is an odd combination. I think you want both set to 2, the way your "Before Kueue" example has it.

The deadlock section is a good explanation but still isn't a demo

The new "The Real Deadlock Demo (Gang Scheduling)" block explains partial admission much better than the old Demo 2 did, and the reasoning is correct. But everything in it is narrative and YAML, with no captured output showing the deadlock actually happening. Right now it's labelled a demo and reads as a description.

Two ways to fix it, either is fine. Rename it to something like "Understanding the Deadlock" so it doesn't promise a run. Or, better, actually run those two jobs and paste what kubectl get workloads and kubectl get pods show while it's stuck. Given you clearly have a working cluster (your Demo output proves it), the second option would make this the strongest section in the post.

Smaller things

  • Both em dashes are still there. Line 11 ("works great for microservices—until someone") and line 43 ("someone deploys a Batch Job—maybe it's"). Line 43 is the exact one I flagged last time. We don't use em dashes on the blog, a comma or a full stop is fine.
  • Arrow lines still collapse into paragraphs. You fixed several by wrapping them in ```text blocks, which is exactly right, but these are still bare and will render as run-on paragraphs: the Pod 1 through Pod 4 dependency chain, the "Result / Status / Wasted resources" block, the "What happens?" recap, the "Kubernetes scheduler doesn't understand gang scheduling" list, the "What Just Happened" recap after Step 9, and the three links under Resources. Same fix as the ones you already did.
  • them?Jobs in the ClusterQueue explainer still has no space, and the quote before it doesn't close.
  • Mixed API versions. ClusterQueue appears as v1beta1 in the demo and v1beta2 in the deadlock section. Both are served in v0.19.2 so nothing breaks, but pick one, and I'd use v1beta2 since that's the storage version now. Your LocalQueue is still v1beta1 too.
  • Heading levels jump around. You go from ## straight to #### and #####, and in one spot ######. Since the levels feed the table of contents, keeping to ## and ### will make the sidebar readable.
  • The heading at line 49 is still a full sentence ("Now let's first understand what Batch Jobs are and the concept of Gang Scheduling, also how they create a DEADLOCK Issue through an example."). Something like "Batch Jobs and Gang Scheduling" would do the same job.
  • Your bio in authors.json is still the placeholder "Writing some cool stuff ... ". Worth a real sentence, it shows on your author page.

Fix the install command and the suspend: true ending, decide what to do about the deadlock section, and do a pass on the arrow blocks and em dashes, and I'm happy to publish this. The demo section is in good shape now, which was the hard part.

@ekam-walia

Copy link
Copy Markdown
Contributor Author

Hi @saiyam1814, I have made all the changes that were suggested by you, I have made the whole deadlock block again
I have added the yaml files as well as the output one should receive in the deadlock scenario
Thank you !

@saiyam1814

Copy link
Copy Markdown
Member

Thanks @ekam-walia. The deadlock section is the real thing now: actual FailedCreate quota events, controller UIDs, your own shell prompt in the pastes. That was the hardest ask on the list and you delivered it. Fresh preview is building here:

https://blog-inside-kueue.website-dab.pages.dev/inside-kueue-how-kubernetes-decides-what-runs-next

Confirmed fixed from last round: the install command (line 228), the suspend: true ending replaced with the label, all API versions on v1beta2, the them?Jobs spacing and the unclosed quote, the odd completions: 1 / parallelism: 2 job is gone, the Pod 1-4 chain and the other arrow blocks you wrapped in text fences render correctly, and your bio is a real bio. Good pass.

One genuinely interesting thing came out of verifying this round, and it's the main thing left to fix.

The label isn't where you think it is, and your own demo proves it

I went to check why your outputs worked, because the YAMLs say they shouldn't have:

  • In the first demo, job-a-big (line 363) and job-b-small (line 429) carry kueue.x-k8s.io/queue-name only on the pod template labels. Kueue's Job integration reads that label from the Job's own metadata.labels, not the template (official sample).
  • In the deadlock demo, Job A (Step 5) has no queue-name label anywhere, yet your output shows workload job-job-a-takes-800m-c8bc8 admitted into smart-queue.

So why did it all work on your cluster? Because both of your LocalQueues are named default, and Kueue has LocalQueue defaulting (stable since v0.17): any unlabeled job in a namespace that has a LocalQueue named default gets adopted automatically. Every job in this post was enrolled by the queue's name, not by your labels. The template-level labels are doing nothing.

That matters because the post ends on "The magic? One label", and a reader who copies your YAMLs into a namespace whose LocalQueue is named anything else will watch Kueue ignore their jobs completely, with no idea why.

The fix is small: put kueue.x-k8s.io/queue-name: default in the Job's metadata.labels in every Kueue-managed YAML (Step 6 already has it there, just drop the duplicate from its template), add it to deadlock-demo Job A, and remove the template-only copies in the first demo. If you want, add one sentence noting that a LocalQueue named default adopts unlabeled jobs automatically. That's a nice pro tip and it explains your own terminal output honestly.

Output pastes that don't match the commands next to them

Same class of issue as round one, four spots:

  • Step 6's YAML says echo 'All pods running together!'; sleep 60 (line 943), but the describe output shows echo 'Pod started. Waiting for partner pod...'; sleep 600 (line 1015). One of them was edited after the capture. Re-run with the YAML as published, or change the YAML back to what you ran.
  • Line 713 says "check pods" but the command is kubectl get jobs, and the output below it is from kubectl get pods.
  • Line 660 gives kubectl get pods -n deadlock-demo -o wide, but the output has no NODE/IP columns and your prompt line shows you ran it without -o wide. Drop the flag.
  • Line 980: kubectl describe job-b-gang -n kueue-demo is missing the resource type, it would error as written. Your own paste right below has the correct kubectl describe job job-b-gang. Also that output fence starts with leftover scrollback (the previous get workloads run repeated), trim it.

The two demos collide if someone runs the post top to bottom

Demo 1 creates LocalQueue default in kueue-demo pointing at demo-queue. The deadlock solution then reuses the same namespace and applies LocalQueue default pointing at smart-queue. A reader following along hits a conflict there, and demo 1's ClusterQueue and jobs may still be holding quota. Add a short cleanup step between the two (kubectl delete ns kueue-demo plus kubectl delete clusterqueue demo-queue), or give part two its own namespace. Related: Step 1 creates kueue-demo with kubectl create ns and Step 4's YAML creates the same Namespace again, keep one.

The abstract deadlock story contradicts its own numbers

The "Real Problem (Without Kueue)" block runs on 12 CPUs, Job A takes 8, Job B needs 6 as 2 pods. But then "Pod 1 starts with 4 CPUs" (a pod requests a fixed amount, it can't soak up whatever is free; 2 pods needing 6 total is 3 each) and "Pod 2 is waiting for 3 CPUs" while the setup says only 4 of 6 were available. It also says "Kubernetes admits it anyway" with Pod 2 Pending, but your actual demo blocks pod creation via ResourceQuota, so Pod 2 never exists at all. Simplest fix: rewrite that block using the demo's real numbers (1500m quota, A takes 800m, B is 2 x 400m, pod 2 forbidden because 1200m + 400m > 1500m) so the story and the terminal output match.

Render and style pass

  • Bare lines that will collapse into run-on paragraphs, same fix as the ones you already did (bullets or a text fence): the "With gang scheduling (Kueue)" block at lines 136-144 (the two-space indent on the Yes?/No? lines isn't enough to make a code block), the "Kubernetes scheduler doesn't understand gang scheduling" list at 148-152, the "THIS is the difference! Intercepted / Calculated / Checked / Suspended" list at 1034-1039, and the wrap-up's "Without Kueue: / With Kueue: / The magic?" at 1050-1055.
  • Three new em dashes arrived with the new section: lines 726, 1039, 1059. And line 45, where the old one was deleted, now reads "a Batch Job maybe it's a machine learning..." and needs the comma that replaced it.
  • Curly quotes from a word processor at lines 28, 45, 47, 51, 158, 170. Straight quotes please.
  • Headings: line 51 is still a full sentence ("Now let's first understand what are Batch Jobs..."), something like "Batch Jobs, Gang Scheduling, and Deadlock" does the job. "## What happens?" at line 95 is a fragment sitting in the TOC as an H2, make it bold text. "Now Lets use Kueue to Solve this problem" at line 802 wants "Let's" and normal capitalization.
  • Line 592: kubectl create namespace deadlock-demo is fenced as yaml, should be bash.
  • "As we can see in line 13 of the terminal's output that is ... shows that how Kueue automatically suspended" reads broken. Try: "Note the Suspend: true line: Kueue set that itself when it held the job back."
  • Optional: the Resources links render as unclickable monospace inside the text fence, a plain bullet list would let people click them.

That's the whole list, and none of it is structural this time. Fix the label placement (the one that changes what readers learn), align the four pastes, add the cleanup step, and do the render pass, and this merges. I'll bump datePublished to the actual publish date when it goes in.

The instinct to verify why something works, not just that it works, is what separates this post from most Kueue content out there. You're one pass away.

@saiyam1814

Copy link
Copy Markdown
Member

Hey @ekam-walia any updates on this ?

@ekam-walia

Copy link
Copy Markdown
Contributor Author

Hi @saiyam1814, I have made all the changes that were suggested by you.

@saiyam1814

saiyam1814 commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

Thanks @ekam-walia, this is a good pass. Fresh preview is rebuilding here:

https://blog-inside-kueue.website-dab.pages.dev/inside-kueue-how-kubernetes-decides-what-runs-next

I test-merged the branch onto current main and ran a real build, so everything below is from the rendered page, not just reading the diff.

Confirmed fixed from round 3, and the big one landed: kueue.x-k8s.io/queue-name is now on the Job's metadata.labels in all four Kueue-managed YAMLs with the pod-template copies removed, including deadlock-demo Job A. The post now teaches the thing that actually enrolls a job. Also fixed: the namespace collision (the cleanup step between the demos), the abstract deadlock block rewritten on the demo's real numbers so the story and the terminal output finally agree, all four output pastes from the last list, the curly quotes, the headings at 51 / 95 / 808, the yaml fence on 592, and the Resources list is clickable now.

Two things still block, both small.

1. The whole conclusion renders as a code block

Lines 1043-1053 got wrapped in a ```text fence. On the built page that comes out as <pre data-language="text">: dark monospace, no bold, no paragraph breaks, horizontal scroll on mobile. It is the last thing a reader sees.

My last note asked for bullets on the four Without Kueue: / With Kueue: / The magic? lines specifically. The fence went around the entire section instead. Drop it and let the prose be prose, with the label on its own one-line code fence:

The magic? One label:

kueue.x-k8s.io/queue-name: default

The last remaining em dash is in there too, line 1051: no custom schedulers—just intelligent → no custom schedulers, just intelligent.

2. Line 980 still does not match its output

kubectl describe workloads job-b-gang -n kueue-demo

It picked up a resource type since last round, but the wrong one, and job-b-gang is not a Workload name. Your own paste two blocks above shows the Workload is job-job-b-gang-c597e. As written this errors with workloads.kueue.x-k8s.io "job-b-gang" not found, and the output underneath is kubectl describe job job-b-gang.

Either change the command to kubectl describe job job-b-gang -n kueue-demo to match what you pasted, or better: run kubectl describe workload job-job-b-gang-c597e -n kueue-demo and paste that. The Workload object carries the QuotaReserved: False condition with the reason, which is a stronger ending for that section than the Job description.

Worth doing in the same pass

  • Line 537, **NOTE:** LocalQueue named default adopts unlabeled jobs automatically is an orphan sentence now that every job in the post carries the label. Either cut it or give it its context: "Note: we label every Job explicitly. Kueue will also adopt an unlabeled Job when the namespace has a LocalQueue named default (LocalQueue defaulting), which is why a queue named anything else needs the label."
  • Line 391, drop the -w. The output below it is a single settled row, not a watch stream.
  • Line 529, "Job A finished, releasing resources": Step 9 deletes Job A, it does not wait for it to finish. Say deleted.
  • Line 854, "Step 4: Create Namespace and LocalQueue": only the LocalQueue is there now.
  • deadlock-demo is never cleaned up. Before Step 1 of the Kueue fix, add kubectl delete ns deadlock-demo. Job A's 800m and the orphan gang pod's 400m are still consuming real node CPU while the reader runs the Kueue version, and on a small kind cluster that changes what they see.
  • The payoff is claimed but never shown. The wrap-up says "When resources freed up, both pods started together", but part two ends with Job B suspended. Demo 1 closes its loop properly with the get workloads after the delete. Do the same here: delete Job A, then one kubectl get workloads -n kueue-demo showing job-job-b-gang-c597e admitted, and if you can, kubectl get pods with both gang pods Running at the same age. That side-by-side with the FailedCreate storm is the whole argument of the post.
  • About 20 of the 47 ```bash fences hold kubectl output rather than commands. Shiki shell-highlights them, so (base) ekamwalia % and the NAME/READY columns get coloured like syntax. ```text on the output blocks reads much cleaner. Same fix you already applied elsewhere.
  • Lines 57-63: **Example**: A web API serving user requests sits on the line directly after list item 3, so markdown absorbs it into that bullet. Confirmed on the built page. Blank line before it, both times.
  • Heading hierarchy. Line 51 "Batch Jobs, Gang Scheduling, and Deadlock" has one sentence under it and then line 55 "Traditional Microservices vs Batch Workloads" is a sibling ## that should be its ###. Line 225 "Installation of Kueue" is a major section sitting at ###. The sidebar TOC shows it.
  • 38 lines have trailing whitespace, including several headings. sed -i '' 's/[[:space:]]*$//' on the file.

The first diagram

kueue-job-sched.webp is excellent, genuinely one of the better Kueue diagrams out there, with the legend and the namespace / control-plane split. Keep it exactly as is.

default-job-scheduling.webp is the weak one next to it. It is AI clipart in a completely different visual language, the scoring podium reads "Spread across / across nodes / Image Locality" where the middle label is garbled, and it introduces predicates, scoring and binding that the prose above it never explains, so a beginner meets three new terms in an image and no definition in the text. Either redo it in the same style as the other diagram, or drop it: the ```text flow right above already makes the point.

At merge

I will bump datePublished (frontmatter says 2026-08-19, but your CRD paste is 08-25 and the Job start time is 09-03). Also bump the install URL to v0.19.4, which is current in the 0.19 line.

One nuance you might want a sentence on: what the demo proves is Kueue's all-or-nothing admission, which is what prevents the partial-creation deadlock. Full gang scheduling, every pod landing on a node simultaneously, also needs topology-aware scheduling. The post uses "gang scheduling" for both, and one line distinguishing them would make it more accurate than most of what is written about Kueue.

Fix the two blockers and the quick list, and this merges. The label investigation from last round is still the thing that makes this post better than the rest of the Kueue content out there.

Signed-off-by: ekam-walia <164983405+ekam-walia@users.noreply.github.com>
@ekam-walia

Copy link
Copy Markdown
Contributor Author

Hi @saiyam1814, I have made the suggested changes.

- Step 9 heading said "Delete Job B" while the command deletes Job A
- split the two admission verdicts in Step 7 onto their own bullets
- gang scheduling block: bare lines were collapsing into one paragraph
- wrap-up: the label fence broke out of the bullet list; retag it yaml,
  and say the partner pod was never created rather than "stuck forever",
  which is what the demo output actually shows
- drop default-job-scheduling.webp: garbled labels (Servec, Eted,
  spec.nodeRame) and it labels PostFilter as the ranking phase
- bump Kueue install to v0.19.4, set publish date, add draft: false
- strip trailing whitespace

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@saiyam1814

Copy link
Copy Markdown
Member

Thanks @ekam-walia. Both blockers from the last round are closed, and the big one landed properly: part two now shows the payoff instead of claiming it. Delete Job A, get workloads with job-job-b-gang-c597e admitted, get pods with both gang pods Running at the same age. That side by side with the FailedCreate storm is the whole argument of the post, and it is finally on the page. The conclusion is prose again, line 980 matches its output, deadlock-demo gets cleaned up, the fence pass landed (only one of your 36 bash blocks was still holding output), and the label note has its context.

Rather than send you round five for the leftovers, I pushed them myself in 6d7a2c1 and merged. Here is everything I changed, so nothing is a surprise.

One thing you should know about, because it is not your fault

The Cloudflare preview I have been linking is stale. It is still serving a build from before your 15 Sep push. I noticed because Step 9 read Free Up Resources (Delete Job B) in the branch but (Delete Job A) on the preview, which is backwards from what you would expect.

So (Delete Job B) was a regression that arrived in the last push, over a step whose command is kubectl delete job job-a-big, and the preview could not have shown it to either of us. Worth knowing for next time: when I say "fresh preview is rebuilding here", check the branch, not just the link. I will stop treating that URL as authoritative.

What I pushed

Correctness

  • Step 9 heading back to (Delete Job A), matching the command underneath and the point of the step.
  • Wrap-up said "one pod stuck forever". Your own demo shows the second pod was never created at all, that is what the FailedCreate events are. Changed it to say so. This is the same honesty pass we did on the abstract deadlock block two rounds ago, and it is the last place the story ran ahead of the terminal output.

Rendering

  • The label fence at the end had broken out of the bullet list (- The magic? One label: followed by an unindented fence closes the <ul>). It is prose plus a yaml fence now, which is what I was asking for last round. It was tagged bash for a YAML line.
  • The "With gang scheduling (Kueue)" block and the two admission verdicts in Step 7 were still bare consecutive lines that collapse into run-on paragraphs. Bullets now.
  • Trailing whitespace stripped (10 lines, including two headings).

The first diagram, which is the one editorial call I made for you

I dropped default-job-scheduling.webp. You clearly redid it and the new one is structurally much better than the clipart it replaced, so this is not "I ignored the work". But it still has garbled labels that will be visible on a published page: Commit Binding to API Servec, Record Pod Status in Eted, spec.nodeRame~node-01, Hig hest Score.

The one that decided it for me is technical, not cosmetic: the diagram labels PostFilter as "Rank Feasible Nodes". PostFilter is the preemption hook that runs when a pod is unschedulable; ranking is the Score / PostScore phase. In a post explaining how the scheduler decides what runs next, a diagram teaching the wrong phase is worse than no diagram, and the text flow directly above it already makes the point.

This is one line to revert if you disagree, and I would genuinely rather have a correct diagram than none. If you want to redo it, kueue-job-sched.webp is the bar: hand-built, correct, legible at blog width, and one of the better Kueue diagrams anywhere. Open a follow-up PR and I will merge it.

At merge, as promised

  • datePublished set to today.
  • Kueue install bumped to v0.19.4 in both places.
  • draft: false added to match our recent posts.

What I checked before merging

Rendered the markdown through a GFM pipeline and read the HTML for every region I touched: lists come out as lists, the label as a tagged yaml block, **Example** as its own paragraph. Frontmatter satisfies lib/blog.js, the author handle resolves in authors.json, all three fence languages (bash, text, yaml) are valid Shiki, both referenced images and your avatar exist on the branch, and the extracted TOC reads cleanly at 24 entries.

In fairness I should say what I did not do: I could not get a full next build running locally this time, so that check is on my GFM render and the pipeline contract rather than the real builder. Given how small and local the diff is, and that the previous state built fine, I was comfortable merging on that. If anything looks off on the live page, tell me and I will fix it directly.

The one nuance still worth a follow-up

What this demo proves is Kueue's all-or-nothing admission, which is what prevents the partial-creation deadlock you captured. Full gang scheduling, every pod landing on a node simultaneously, also needs topology-aware scheduling. The post uses "gang scheduling" for both. One sentence distinguishing them would make it more accurate than most of what is written about Kueue, and it is a natural follow-up PR alongside the diagram if you want one.

Merged

Four rounds is a lot, and you took every one of them seriously. The thing I keep coming back to is round three: you went to check why your outputs worked, found that LocalQueue defaulting had been enrolling your jobs the whole time and the labels you had written were doing nothing, and fixed the post to teach the thing that actually enrolls a job. Most people would have shipped it, because it worked on their cluster. That instinct, verifying why something works rather than that it works, is what makes this better than the Kueue content already out there.

Congratulations on the post. It goes live shortly at:
https://kubesimplify.com/inside-kueue-how-kubernetes-decides-what-runs-next

@saiyam1814
saiyam1814 merged commit 1e46b67 into kubesimplify:main Sep 16, 2026
saiyam1814 added a commit that referenced this pull request Sep 16, 2026
The committed output of scripts/generate-redirects.mjs had drifted from
content/blog:

- kubernetes-observability-in-2026-with-openobserve (#268) was missing from
  _redirects, vercel.json and _worker.js
- inside-kueue-how-kubernetes-decides-what-runs-next (#266) was missing from
  _worker.js (the PR committed _redirects and vercel.json but not the worker)

BLOG_SLUGS goes 201 -> 203. Nothing is removed; the diff is purely additive.

No live impact: prebuild regenerates all four files during the Pages build, so
both posts already route correctly. This just stops the repo shipping stale
artifacts, which is what made the omission invisible in the first place.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
saiyam1814 added a commit that referenced this pull request Sep 16, 2026
scripts/generate-redirects.mjs writes public/_redirects, public/_worker.js,
public/_routes.json and vercel.json. prebuild regenerates all four during every
Pages build, so a stale committed copy never breaks the live site -- which is
precisely why the drift is invisible.

It went unnoticed twice: #268 was missing from all four files, and #266 was
missing from _worker.js (the file that actually holds BLOG_SLUGS and does the
routing on Cloudflare). Both were only found by regenerating and diffing by hand.

This regenerates in CI and fails if the tree changes, with a message naming the
fix. Verified both ways: no diff on current main, and adding one post to
content/blog trips it (203 -> 204 slugs).

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants