2026.08.30 miscellaneous improvements

TODO.md

@@ -54,17 +54,6 @@ E1
 !maintenance
 
 
-## Add --force_model_download to run_md_and_speciesnet
-
-run_detector_batch supports a --force_model_download argument to handle the case where model weights were partially downloaded; add a corresponding option to run_md_and_speciesnet.  It should apply to both the MD weights and the SpeciesNet weights.
-
-P0
-
-E0
-
-!feature
-
-
 ## Graceful handling of small images during tiling
 
 When running tiled inference, if either dimension of an image is smaller than the tiling size, that image fails.  This is OK, it's correctly recorded as an inference failure, but in most cases I would rather it fall back to a smaller tile size in that case.
@@ -443,21 +432,6 @@ E3
 !feature
 
 
-## Reference result updates
-
-* Reduce complexity of reference results: MD's test harness relies on .json files with pre-generated results for MDv5a and MDv5b, for a reference set of images and videos.  Because output varies slightly between PyTorch versions and between hardware environments, I have a number of results files.  This has gotten too complicated; remove most of the results files and increase the allowed tolerance during testing.
-* Add test results for MD1000 models: MD's test harness only has results for MDv5, so it tests the not-crashing-ness of the other models, but it does not test correctness.  Add test results for other MD1000 models.
-* Vehicle images: none of the test images include vehicles; add vehicle images to testing, including human/vehicle and animal/vehicle images
-* Images with lat/lon information in EXIF metadata; make sure EXIF extraction (especially GPS location) is working correctly.
-* The "magic zebra image" that causes problems on M1 Pro machines; this should be called out as a dedicated single-command test case
-
-P0
-
-E1
-
-!testing
-
-
 ## Test coverage improvements
 
 This is a placeholder for generally evaluating md_tests and the pytest harness, and deciding which scripts need additional testing.  Effort is highly variable; for example, adding tests for run_md_and_speciesnet is important and very easy.  Adding tests for postprocess_batch_results that actually verify correctness is a pain.  This work item almost certainly starts with asking AI what modules are not covered (or poorly covered) by tests.
@@ -678,17 +652,6 @@ E1
 !feature
 
 
-## Update Colab
-
-Nothing is "wrong" with the [MegaDetector Colab](https://github.com/agentmorris/MegaDetector/blob/main/notebooks/megadetector_colab.ipynb), but it hasn't been updated in a while.  It doesn't mention MDv1000 or SpeciesNet; it would be helpful to just give the Colab a once-over, make sure it's still in good shape, and add optional cells that demonstrate MDv1000 use and SpeciesNet inference (via run_md_and_speciesnet).
-
-P0
-
-E0
-
-!feature
-
-
 ## Explore compiled PyTorch
 
 [torch.compile](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) was introduced in 2023, but I haven't evaluated it for MegaDetector (or SpeciesNet).  Evaluate it for both MegaDetector and SpeciesNet.

megadetector/data_management/lila/get_lila_annotation_counts.py

@@ -95,7 +95,7 @@
 
 dataset_to_categories = {}
 
-# ds_name = 'NACTI'
+# ds_name = 'SWG Camera Traps'
 for ds_name in metadata_table.keys():
 
     taxonomy_mapping_available = (ds_name in datasets_with_taxonomy_mapping)
@@ -116,6 +116,12 @@
     # Collect list of categories and mappings to category name
     categories = data['categories']
 
+    # In some cases, a "count" field is in the actual .json file, remove this
+    # so we can re-count
+    for c in categories:
+        if 'count' in c:
+            del c['count']
+
     category_id_to_count = defaultdict(int)
     annotations = data['annotations']
 
@@ -125,9 +131,11 @@
 
     # c = categories[0]
     for c in categories:
+
        count = category_id_to_count[c['id']]
        if 'count' in c:
-           assert 'bbox' in ds_name or c['count'] == count
+           assert ('bbox' in ds_name) or (c['count'] == count)
+
        c['count'] = count
 
        # Don't do taxonomy mapping for bbox data sets, which are sometimes just binary and are
@@ -148,6 +156,8 @@
                assert sn is not None and len(sn) > 0
                c['scientific_name_from_taxonomy_mapping'] = sn
 
+    # ...for each category
+
     dataset_to_categories[ds_name] = categories
 
 # ...for each dataset

megadetector/detection/process_video.py

@@ -87,6 +87,11 @@ def __init__(self):
         #: Detector-specific options
         self.detector_options = None
 
+        #: Force downloading the model file if a named model (e.g. "MDV5A") is supplied,
+        #: even if the local file already exists (typically to overwrite a corrupted
+        #: model file)
+        self.force_model_download = False
+
         #: Write a checkpoint file (to resume processing later) every N videos;
         #: set to -1 (default) to disable checkpointing
         self.checkpoint_frequency = -1
@@ -153,7 +158,9 @@ def process_videos(options):
     if options.verbose:
         print('Processing videos from input source {}'.format(options.input_video_file))
 
-    detector = load_detector(options.model_file,detector_options=options.detector_options)
+    detector = load_detector(options.model_file,
+                             force_model_download=options.force_model_download,
+                             detector_options=options.detector_options)
 
     def frame_callback(image_np,image_id):
         return detector.generate_detections_one_image(image_np,
@@ -434,6 +441,12 @@ def main(): # noqa
         default='',
         help='Detector-specific options, as a space-separated list of key-value pairs')
 
+    parser.add_argument(
+        '--force_model_download',
+        action='store_true',
+        help=('If a named model (e.g. "MDV5A") is supplied, force a download of that model even if the ' +\
+              'local file already exists (typically to overwrite a corrupted model file).'))
+
     parser.add_argument(
         '--checkpoint_frequency',
         type=int,

megadetector/detection/run_detector_batch.py

@@ -1110,7 +1110,7 @@ def load_and_run_detector_batch(model_file,
         augment (bool, optional): enable image augmentation
         force_model_download (bool, optional): force downloading the model file if
             a named model (e.g. "MDV5A") is supplied, even if the local file already
-            exists
+            exists (typically to handle the case where the model file is corrupted).
         detector_options (dict, optional): key/value pairs that are interpreted differently
             by different detectors.  Can also be a list of k=v pairs, or a comma-delimited
             string containing a list of k=v pairs.
@@ -1889,7 +1889,7 @@ def main(): # noqa
         '--force_model_download',
         action='store_true',
         help=('If a named model (e.g. "MDV5A") is supplied, force a download of that model even if the ' +\
-              'local file already exists.'))
+              'local file already exists (typically to overwrite a corrupted model model).'))
     parser.add_argument(
         '--previous_results_file',
         type=str,

megadetector/detection/run_md_and_speciesnet.py

@@ -188,6 +188,13 @@ def __init__(self):
         #: Include raw (pre-rollup/geofence) classification scores in output
         self.include_raw_classifications = False
 
+        #: Force downloading the detector and classifier model files, even if the
+        #: local files already exist (typically to overwrite corrupted model files).
+        #:
+        #: Only relevant for named/remote models, i.e. this has no effect for models
+        #: that are specified as local files or folders.
+        self.force_model_download = False
+
         if self.time_sample is None and self.frame_sample is None:
             self.time_sample = DEFAULT_SECONDS_PER_VIDEO_FRAME
 
@@ -939,7 +946,8 @@ def _run_detection_step(source_folder: str,
                         skip_images: bool = False,
                         skip_video: bool = False,
                         frame_sample: int = None,
-                        time_sample: float = None) -> str:
+                        time_sample: float = None,
+                        force_model_download: bool = False) -> str:
     """
     Run MegaDetector on all images/videos in [source_folder].
 
@@ -956,6 +964,9 @@ def _run_detection_step(source_folder: str,
         skip_video (bool, optional): ignore videos, only process images
         frame_sample (int, optional): sample every Nth frame from videos
         time_sample (float, optional): sample frames every N seconds from videos
+        force_model_download (bool, optional): force downloading the detector model file
+            if a named model (e.g. "MDV5A") is supplied, even if the local file already
+            exists (typically to overwrite a corrupted model file)
     """
 
     print('Starting detection step...')
@@ -1009,7 +1020,8 @@ def _run_detection_step(source_folder: str,
             include_exif_tags=None,
             loader_workers=detector_worker_threads,
             preprocess_on_image_queue=True,
-            use_threads_for_queue=use_threads_for_queue
+            use_threads_for_queue=use_threads_for_queue,
+            force_model_download=force_model_download
         )
 
         # Write image results to temporary file
@@ -1029,9 +1041,14 @@ def _run_detection_step(source_folder: str,
 
         print('Running MegaDetector on {} videos...'.format(len(video_files)))
 
+        # If we also had images to process, we already forced a download of the
+        # detector model above, so there's no need to download it again here.
+        force_model_download_for_videos = force_model_download and (len(image_files) == 0)
+
         # Set up video processing options
         video_options = ProcessVideoOptions()
         video_options.model_file = detector_model
+        video_options.force_model_download = force_model_download_for_videos
         video_options.input_video_file = source_folder
         video_options.output_json_file = detector_output_file.replace('.json', '_videos.json')
         video_options.json_confidence_threshold = detection_confidence_threshold
@@ -1078,7 +1095,8 @@ def _run_classification_step(detector_results_file: str,
                              top_n_scores: int = DEFAULT_TOP_N_SCORES,
                              worker_type: str = DEFAULT_WORKER_TYPE,
                              include_raw_classifications: bool = False,
-                             rollup_target_confidence: float = DEFAULT_ROLLUP_TARGET_CONFIDENCE):
+                             rollup_target_confidence: float = DEFAULT_ROLLUP_TARGET_CONFIDENCE,
+                             force_model_download: bool = False):
     """
     Run SpeciesNet classification on detections from MegaDetector results.
 
@@ -1099,6 +1117,9 @@ def _run_classification_step(detector_results_file: str,
             classification scores in output
         rollup_target_confidence (float, optional): target confidence threshold for taxonomic
             rollup.  Ignored if enable_rollup is False.
+        force_model_download (bool, optional): force downloading the classifier model files
+            if a remote model (e.g. a "kaggle:" or "hf:" identifier) is supplied, even if
+            the local files already exist (typically to overwrite corrupted model files)
     """
 
     print('Starting classification step...')
@@ -1118,6 +1139,22 @@ def _run_classification_step(detector_results_file: str,
 
     print('Using SpeciesNet classifier: {}'.format(classifier_model))
 
+    # The classifier gets loaded in several places below (in the main thread and/or in
+    # worker threads/processes, depending on [worker_type]), and it doesn't make sense
+    # to force a download in all of those places.  Instead, if we've been asked to force
+    # a download, we load a throwaway instance here, on the CPU, just to make sure the
+    # model files are freshly downloaded (and loadable) before we do anything else.
+    if force_model_download:
+
+        print('Forcing a download of classifier model {}'.format(classifier_model))
+        throwaway_classifier = SpeciesNetClassifier(classifier_model,
+                                                    device='cpu',
+                                                    force_model_download=True)
+        del throwaway_classifier
+        print('Finished forced download of classifier model {}'.format(classifier_model))
+
+    # ...if we need to force a model download
+
     # Set multiprocessing start method to 'spawn' for CUDA compatibility
     if worker_type == 'process':
         original_start_method = multiprocessing.get_start_method()
@@ -1450,7 +1487,8 @@ def run_md_and_speciesnet(options):
             skip_video=options.skip_video,
             frame_sample=options.frame_sample,
             time_sample=options.time_sample,
-            worker_type=options.worker_type
+            worker_type=options.worker_type,
+            force_model_download=options.force_model_download
         )
 
     # Run SpeciesNet
@@ -1467,7 +1505,8 @@ def run_md_and_speciesnet(options):
         admin1_region=options.admin1_region,
         worker_type=options.worker_type,
         include_raw_classifications=options.include_raw_classifications,
-        rollup_target_confidence=options.rollup_target_confidence
+        rollup_target_confidence=options.rollup_target_confidence,
+        force_model_download=options.force_model_download
     )
 
     elapsed_time = time.time() - start_time
@@ -1584,6 +1623,11 @@ def main():
     parser.add_argument('--include_raw_classifications',
                         action='store_true',
                         help='Include raw (pre-rollup/geofence) classification scores in output')
+    parser.add_argument('--force_model_download',
+                        action='store_true',
+                        help='Force a download of both the detector and classifier models, even if ' + \
+                             'the local model files already exist (typically to overwrite corrupted ' + \
+                             'model files)')
 
     if len(sys.argv[1:]) == 0:
         parser.print_help()

megadetector/taxonomy_mapping/map_new_lila_datasets.py

@@ -15,10 +15,10 @@
 # Created by get_lila_category_list.py
 input_lila_category_list_file = os.path.expanduser('~/lila/lila_categories_list/lila_dataset_to_categories.json')
 
-output_file = os.path.expanduser('~/lila/lila_additions_2026.06.17.csv')
+output_file = os.path.expanduser('~/lila/lila_additions_2026.09.01.csv')
 
 datasets_to_map = [
-    'AMMonitor Camera Traps'
+    'Duck Pictures in Wetlands'
     ]
 
 
@@ -192,7 +192,7 @@
 
     # Use this when an iNat match includes an empty subgenus with the same name as the genus
     n_levels_to_pop = 0
-    q = 'animalia'
+    q = 'cygnus'
 
     taxonomy_preference = 'inat'
     m = get_preferred_taxonomic_match(q,taxonomy_preference)

megadetector/taxonomy_mapping/preview_lila_taxonomy.py

@@ -16,7 +16,7 @@
 import pandas as pd
 
 # lila_taxonomy_file = r"c:\git\agentmorrisprivate\lila-taxonomy\lila-taxonomy-mapping.csv"
-lila_taxonomy_file = os.path.expanduser('~/lila/lila_additions_2026.06.17.csv')
+lila_taxonomy_file = os.path.expanduser('~/lila/lila_additions_2026.09.01.csv')
 
 preview_base = os.path.expanduser('~/lila/lila_taxonomy_preview')
 os.makedirs(preview_base,exist_ok=True)