Skip to content

NaNs during training when running the TinyYoloHouseNumberDetection example #1084

Description

@b005t3r

Issue Description

Please describe your issue, along with:

  • expected behavior: the network trains normally, no NaNs are reported
  • encountered behavior: training crashes after the first fit() call because of NaNs (I enabled NaN detection, see below)

Version Information

Please indicate relevant versions, including, if relevant:

  • Deeplearning4j version: 1.0.0-beta7 and 1.0.0-M2.1
  • platform information: macOS Sonoma 14.2.1

The mentioned example:
https://github.com/deeplearning4j/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/modelling/objectdetection/TinyYoloHouseNumberDetection.java

You'll need to add this as the first call in the main() method:

        Nd4j.getExecutioner().setProfilingConfig(ProfilerConfig.builder()
                .checkForINF(true)
                .checkForNAN(true)
                .checkElapsedTime(true)
                .checkLocality(true)
                .checkWorkspaces(true)
                .build());

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions