Multiple kbatch jobs - running concurrently? On same server?

[Jesse Anderson (jessjaco)]

Do kbatch jobs run concurrently? Are they running on the same server, or does each have it's own? Just trying the best way to run some things.

[Jesse Anderson (jessjaco)]

My guess is that each runs on its own server but the total number of nodes available through dask are capped at the account level.

[Tom Augspurger (TomAugspurger)]

They should run concurrently. The number of nodes (Dask or just "singleuser" nodes) are limited and are a shared resource between all users of the Hub.

[Jesse Anderson (jessjaco)]

Then I guess I am seeing different performance on a jupyterhub instance vs using kbatch and I'm not sure why. I am processing some landsat data by scene, and logging the time it takes to process each scene (I am pulling all data for each year for that scene, masking clouds, calculating an index, and writing the result to blob storage). Here is a portion of the run on a jupyterhub instance, where each row is the log for a particular scene:

And this is the identical processing using kbatch (from the pod log), running at the same time:

As you can see, things are taking a lot longer using kbatch. They should be using the same number of workers, etc. (I haven't figured out how to log the number of workers actually being used in kbatch, or see the dask dashboard). The kbatch process has also been hung on this same scene for several hours, which does not seem to be unusual. Ultimately the jupyterhub processing took about an hour and a half total, and the kbatch version has been running for 2 days already.

[Jesse Anderson (jessjaco)]

Just an update that the stalling behavior appears to happen for processes that don't use the dask gateway as well as those that do. I think it might have to do with multiple kbatch jobs running concurrently, but I'm not positive. Trying to winnow down to a reproducible example.