bpnn kernel error: no kernel image is available for execution on the device

[PrabinKuSabat]

./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft -D 0
Running backprop-rodinia-2.0-ft
------------- NVBit (NVidia Binary Instrumentation Tool v1.7.4) Loaded --------------
NVBit core environment variables (mostly for nvbit-devs):
ACK_CTX_INIT_LIMITATION = 0 - if set, no warning will be printed for nvbit_at_ctx_init()
            NVDISASM = nvdisasm - override default nvdisasm found in PATH
            NOBANNER = 0 - if set, does not print this banner
       NO_EAGER_LOAD = 0 - eager module loading is turned on by NVBit to prevent potential NVBit tool deadlock, turn it off if you want to use the lazy module loading feature
---------------------------------------------------------------------------------
         INSTR_BEGIN = 0 - Beginning of the instruction interval where to apply instrumentation
           INSTR_END = 4294967295 - End of the instruction interval where to apply instrumentation
    EXCLUDE_PRED_OFF = 1 - Exclude predicated off instruction from count
      TRACE_LINEINFO = 0 - Include source code line info at the start of each traced line. The target binary must be compiled with -lineinfo or --generate-line-info
DYNAMIC_KERNEL_LIMIT_END = 0 - Limit of the number kernel to be printed, 0 means no limit
DYNAMIC_KERNEL_LIMIT_START = 0 - start to report kernel from this kernel id, 0 means starts from the beginning, i.e. first kernel
   ACTIVE_FROM_START = 1 - Start instruction tracing from start or wait for cuProfilerStart and cuProfilerStop. If set to 0, DYNAMIC_KERNEL_LIMIT options have no effect
        TOOL_VERBOSE = 0 - Enable verbosity inside the tool
       TOOL_COMPRESS = 1 - Enable traces compression
     TOOL_TRACE_CORE = 0 - write the core id in the traces
TERMINATE_UPON_LIMIT = 0 - Stop the process once the current kernel > DYNAMIC_KERNEL_LIMIT_END
USER_DEFINED_FOLDERS = 0 - Uses the user defined folder TRACES_FOLDER path environment
 TRACE_FILE_COMPRESS = 1 - Create xz-compressed tracefile
----------------------------------------------------------------------------------------------------
Random number generator seed: 7
Input layer size : 4096
Starting training kernel
WARNING: Do not call CUDA memory allocation in nvbit_at_ctx_init(). It will cause deadlocks. Do them in nvbit_tool_init(). If you encounter deadlocks, remove CUDA API calls to debug.
Performing GPU computation
bpnn kernel error: no kernel image is available for execution on the device
Error invoking nvbit on /home/cab-prj/Prabin/accelSim/accel-sim-framework/hw_run/traces/device-0/11.8/backprop-rodinia-2.0-ft/4096___data_result_4096_txt

Distributor ID: Ubuntu
Description: Ubuntu 24.04.2 LTS
Release: 24.04
Codename: noble

 nvidia-smi
Fri Mar 14 09:35:57 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX TITAN X     On  |   00000000:84:00.0 Off |                  N/A |
| 22%   46C    P8             32W /  250W |       2MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

cab-prj@dmacs-titanx ~/Prabin/accelSim/accel-sim-framework dev*
❯ nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

How to solve the above error in this environment setup?

[JRPan]

I think this is caused by gencode or ARCH flag missing in the tracer. It seems GTX TITAN X is a Maxwell card? Try change the tracer make file with SM_50

[PrabinKuSabat]

Yes, it’s a Maxwell card—thanks a lot for the solution!

I’d also like to take this opportunity to make a request. As part of my Master’s project, I’m planning to add support for an RTX 4090 configuration in Accel-Sim. Could you kindly guide me on how to get started with this?

Additionally, it would be really helpful if you could share some statistical verifications for the RTX 3070 configurations, so that we can cross-check them ourselves and ensure we’re able to reproduce the results.

Thanks once again for keeping this project alive—it has been an invaluable resource for learning and working with GPUs.

[JRPan]

We have a tuner documented here https://github.com/accel-sim/accel-sim-framework/tree/dev/util/tuner.
With some careful tuning, we got very good results on A100s and H100s with the tuner.

[PrabinKuSabat]

"For each new GPU generation, we have to crease ISA_def file that specifies the SASS instruction types and where each instruction should be executed."
This is what I found at https://github.com/accel-sim/accel-sim-framework/blob/d3ba55f398ac18f6e1043e897c852c7eef8bb1c0/gpu-simulator/README.md.

So is it not required to add the ISA for the Ada Lovelace Architecture?

[PrabinKuSabat]

Also, can you share the approx. MAE and corr. values for rtx 3070 for the rodinia 2.0-ft and rodinia-3.1_ft benchmarks?