Limit CPU usage

[gpanikian]

Dear Colleagues,
I am sending you this message to ask you about how I can limit CPU core usage when running GPy library.
I am applying a bootstrap technique for the classifier to evaluate model performance using the standard implementation as:
model = GPy.models.GPClassification(X=X_train, Y=Y_train, kernel=kernel )
The process is consuming all resources, I tried to set n_jobs or n_workers but I did not succeed doing it!

Can you please help me!

Thank you
Garo

[gpanikian]

I should also inform you that I tried to limit the number of threads to 2 but there was not any improvement.
Your help is much appreciated!

import os

Limit number of threads to 2

os.environ["OMP_NUM_THREADS"] = "2"
os.environ["OPENBLAS_NUM_THREADS"] = "2"
os.environ["MKL_NUM_THREADS"] = "2"
os.environ["VECLIB_MAXIMUM_THREADS"] = "2"
os.environ["NUMEXPR_NUM_THREADS"] = "2"

[lawrennd]

Hi Garo,

GPy doesn’t expose n_jobs on classifiers. CPU use comes from the BLAS backend under NumPy (and from whatever launches many fits in your bootstrap).

1. Try setting thread limits before Python starts, or at the very top of the script before any import numpy / import GPy:

OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 MKL_NUM_THREADS=2 \
VECLIB_MAXIMUM_THREADS=2 NUMEXPR_NUM_THREADS=2 python your_script.py

2. Or after imports, with threadpoolctl:

from threadpoolctl import threadpool_limits
with threadpool_limits(limits=2):
    model = GPy.models.GPClassification(X_train, Y_train, kernel)
    model.optimize()

3. If you use optimize_restarts(..., parallel=True), that spawns processes. You can use parallel=False (default) when bootstrapping, or limit outer bootstrap concurrency yourself.