Skip to content

fix(cpu): cap generic thread defaults at the logical CPU count #182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Afloat16 wants to merge 1 commit into
base: main
Choose a base branch
from
+9 −3
Open

fix(cpu): cap generic thread defaults at the logical CPU count #182

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions edge0-infer/docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ The runtime opens a fixed set of devices (`devices`); a model can only use devic

| Flag | Native field | Default | Meaning |
| --- | --- | --- | --- |
| `--threads <n>` | `cpu_threads` | macOS 4; Android logical CPUs minus two (at least one); Windows `max(1, P - 2)` | CPU threads for decode; requires a CPU device; at most the logical CPU count |
| `--prefill-threads <n>` | `cpu_prefill_threads` | macOS 4; Android all logical CPUs; Windows `min(logical, P + 2)` | CPU threads for prefill (processing the input), set independently of decode |
| `--threads <n>` | `cpu_threads` | macOS/Linux up to 4 logical CPUs (at least one); Android logical CPUs minus two (at least one); Windows `max(1, P - 2)` | CPU threads for decode; requires a CPU device; at most the logical CPU count |
| `--prefill-threads <n>` | `cpu_prefill_threads` | macOS/Linux up to 4 logical CPUs (at least one); Android all logical CPUs; Windows `min(logical, P + 2)` | CPU threads for prefill (processing the input), set independently of decode |
| `--memory-budget <bytes>` | `memory_budget_bytes` | The total memory of the smallest opened device | The total budget every loaded model and session is admitted against; must be positive and not exceed the device memory |

The Android thread defaults were tuned on one device and may not suit other CPU layouts. On Windows, `P` is `logical / 2`. That rule assumes two threads per core and was measured on one 6-core CPU; set the counts explicitly when the CPU has no simultaneous multithreading or has efficiency cores. `info` prints the defaults; summaries report the resolved values and whether they were given explicitly.
Expand Down
3 changes: 2 additions & 1 deletion edge0-infer/src/platform/backend.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ CpuThreads default_cpu_threads(unsigned logical_cpus, bool android, bool windows
const int physical = std::max(1, logical / 2);
return {std::max(1, physical - 2), std::min(logical, physical + 2)};
}
return {4, 4};
const int count = static_cast<int>(std::max(1u, std::min(logical_cpus, 4u)));
return {count, count};
}
CpuThreads default_cpu_threads() {
#if defined(__ANDROID__)
Expand Down
5 changes: 5 additions & 0 deletions edge0-infer/tests/unit/platform/backend_test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ int main() {
using edge0::platform::default_cpu_threads;
const auto generic = default_cpu_threads(12, false);
CHECK(generic.decode == 4 && generic.prefill == 4);
const std::pair<unsigned, int> generic_cases[] = {{0, 1}, {1, 1}, {2, 2}, {3, 3}, {4, 4}, {128, 4}};
for (const auto& [logical, expected] : generic_cases) {
const auto limited = default_cpu_threads(logical, false);
CHECK(limited.decode == expected && limited.prefill == expected);
}
const auto android = default_cpu_threads(8, true);
CHECK(android.decode == 6 && android.prefill == 8);
// Windows rule: physical = logical / 2; decode = max(1, physical - 2),
Expand Down