11# Overview #
22
33` xdist ` works by spawning one or more ** workers** , which are controlled
4- by the ** master ** . Each ** worker** is responsible for performing
5- a full test collection and afterwards running tests as dictated by the ** master ** .
4+ by the ** controller ** . Each ** worker** is responsible for performing
5+ a full test collection and afterwards running tests as dictated by the ** controller ** .
66
77The execution flow is:
88
9- 1 . ** master ** spawns one or more ** workers** at the beginning of
10- the test session. The communication between ** master ** and ** worker** nodes makes use of
11- [ execnet] ( http ://codespeak.net/execnet/) and its [ gateways] ( http ://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters) .
9+ 1 . ** controller ** spawns one or more ** workers** at the beginning of
10+ the test session. The communication between ** controller ** and ** worker** nodes makes use of
11+ [ execnet] ( https ://codespeak.net/execnet/) and its [ gateways] ( https ://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters) .
1212 The actual interpreters executing the code for the ** workers** might
1313 be remote or local.
1414
15151 . Each ** worker** itself is a mini pytest runner. ** workers** at this
1616 point perform a full test collection, sending back the collected
17- test-ids back to the ** master ** which does not
17+ test-ids back to the ** controller ** which does not
1818 perform any collection itself.
1919
20- 1 . The ** master ** receives the result of the collection from all nodes.
21- At this point the ** master ** performs some sanity check to ensure that
20+ 1 . The ** controller ** receives the result of the collection from all nodes.
21+ At this point the ** controller ** performs some sanity check to ensure that
2222 all ** workers** collected the same tests (including order), bailing out otherwise.
2323 If all is well, it converts the list of test-ids into a list of simple
2424 indexes, where each index corresponds to the position of that test in the
2525 original collection list. This works because all nodes have the same
26- collection list, and saves bandwidth because the ** master ** can now tell
26+ collection list, and saves bandwidth because the ** controller ** can now tell
2727 one of the workers to just * execute test index 3* index of passing the
2828 full test id.
2929
30- 1 . If ** dist-mode** is ** each** : the ** master ** just sends the full list
30+ 1 . If ** dist-mode** is ** each** : the ** controller ** just sends the full list
3131 of test indexes to each node at this moment.
3232
33- 1 . If ** dist-mode** is ** load** : the ** master ** takes around 25% of the
33+ 1 . If ** dist-mode** is ** load** : the ** controller ** takes around 25% of the
3434 tests and sends them one by one to each ** worker** in a round robin
3535 fashion. The rest of the tests will be distributed later as ** workers**
3636 finish tests (see below).
@@ -40,36 +40,36 @@ The execution flow is:
40401 . ** workers** re-implement ` pytest_runtestloop ` : pytest's default implementation
4141 basically loops over all collected items in the ` session ` object and executes
4242 the ` pytest_runtest_protocol ` for each test item, but in xdist ** workers** sit idly
43- waiting for ** master ** to send tests for execution. As tests are
43+ waiting for ** controller ** to send tests for execution. As tests are
4444 received by ** workers** , ` pytest_runtest_protocol ` is executed for each test.
4545 Here it worth noting an implementation detail: ** workers** always must keep at
4646 least one test item on their queue due to how the ` pytest_runtest_protocol(item, nextitem) `
4747 hook is defined: in order to pass the ` nextitem ` to the hook, the worker must wait for more
48- instructions from master before executing that remaining test. If it receives more tests,
48+ instructions from controller before executing that remaining test. If it receives more tests,
4949 then it can safely call ` pytest_runtest_protocol ` because it knows what the ` nextitem ` parameter will be.
5050 If it receives a "shutdown" signal, then it can execute the hook passing ` nextitem ` as ` None ` .
5151
52521 . As tests are started and completed at the ** workers** , the results are sent
53- back to the ** master ** , which then just forwards the results to
53+ back to the ** controller ** , which then just forwards the results to
5454 the appropriate pytest hooks: ` pytest_runtest_logstart ` and
5555 ` pytest_runtest_logreport ` . This way other plugins (for example ` junitxml ` )
56- can work normally. The ** master ** (when in dist-mode ** load** )
56+ can work normally. The ** controller ** (when in dist-mode ** load** )
5757 decides to send more tests to a node when a test completes, using
5858 some heuristics such as test durations and how many tests each ** worker**
5959 still has to run.
6060
61- 1 . When the ** master ** has no more pending tests it will
61+ 1 . When the ** controller ** has no more pending tests it will
6262 send a "shutdown" signal to all ** workers** , which will then run their
6363 remaining tests to completion and shut down. At this point the
64- ** master ** will sit waiting for ** workers** to shut down, still
64+ ** controller ** will sit waiting for ** workers** to shut down, still
6565 processing events such as ` pytest_runtest_logreport ` .
6666
6767## FAQ ##
6868
6969> Why does each worker do its own collection, as opposed to having
70- the master collect once and distribute from that collection to the workers?
70+ the controller collect once and distribute from that collection to the workers?
7171
72- If collection was performed by master then it would have to
72+ If collection was performed by controller then it would have to
7373serialize collected items to send them through the wire, as workers live in another process.
7474The problem is that test items are not easily (impossible?) to serialize, as they contain references to
7575the test functions, fixture managers, config objects, etc. Even if one manages to serialize it,
0 commit comments