WIP: Migrate to datajoint v2

[thomasZen]

This branch already has changes from #72 integrated.

When testing the tutorial schema, make sure to create a new config based on https://github.com/eulerlab/djimaging/blob/migration/datajointv2/djimaging/djconfig/djconf_template.json, adapt the path for processed files, and run the make_user.py script to have access to the updated example notebook, and the mentioned folders for the processed files have to exists already.

The example notebook then partially works already, I have to rerun it and will check it again, though.

Note: datajoint writes a lot of files now into the folders.

To check

Moving Bar Stimulus warning

Stimulus().add_movingbar(
    ntrigger_rep=1,
    trial_info=[0, 180, 45, 225, 90, 270, 135, 315],
    bardx=300,
    bardy=1000,
    velumsec=1000,
    tmovedurs=4.0,
    skip_duplicates=True,
)
[djimaging/tables/core/stimulus.py:66](http://localhost:4444/lab/workspaces/auto-R/tree/GitRepos/djimaging/djimaging/user/dj_v2_try3_tz/notebooks/GitRepos/djimaging/djimaging/tables/core/stimulus.py#line=65): UserWarning: Number of triggers in trial_info=8 must match ntrigger_rep=1.
  warnings.warn(msg)

Snippets multiprocessing

processes=20 lead to some concurrency bug. I think two processes wrote the same metadata file, which lead to an error. Using processes=1 fixed it, but is a slow solution.

Snippets().populate(processes=1, display_progress=True)

Error message with processed=20:

Processes:   0%|          | 0[/3503](http://localhost:4444/3503) [00:00<?, ?it[/s](http://localhost:4444/s)]
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "[/usr/lib/python3.12/multiprocessing/pool.py", line 125](http://localhost:4444/usr/lib/python3.12/multiprocessing/pool.py#line=124), in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py", line 75](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py#line=74), in _call_populate1
    return process.table._populate1(key, process.jobs, **process.populate_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py", line 689](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py#line=688), in _populate1
    make(dict(key), **(make_kwargs or {}))
  File "[/gpfs01/euler/User/tzenkel/GitRepos/djimaging/djimaging/tables/core/snippets.py", line 145](http://localhost:4444/lab/workspaces/auto-R/tree/GitRepos/djimaging/djimaging/user/dj_v2_try4_tz/notebooks/GitRepos/djimaging/djimaging/tables/core/snippets.py#line=144), in make
    self.insert1(dict(
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py", line 718](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py#line=717), in insert1
    self.insert((row,), **kwargs)
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py", line 862](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py#line=861), in insert
    self._insert_rows(rows, replace, skip_duplicates, ignore_extra_fields)
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py", line 881](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py#line=880), in _insert_rows
    rows = list(self.__make_row_to_insert(row, field_list, ignore_extra_fields) for row in rows)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py", line 881](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py#line=880), in <genexpr>
    rows = list(self.__make_row_to_insert(row, field_list, ignore_extra_fields) for row in rows)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py", line 1472](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py#line=1471), in __make_row_to_insert
    self.__make_placeholder(name, row[name], ignore_extra_fields, row=row_dict)
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py", line 1395](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/table.py#line=1394), in __make_placeholder
    value = attr_type.encode(value, key=context, store_name=resolved_store)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/builtin_codecs/hash.py", line 82](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/builtin_codecs/hash.py#line=81), in encode
    return put_hash(value, schema_name=schema_name, store_name=store_name, config=config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/hash_registry.py", line 244](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/hash_registry.py#line=243), in put_hash
    backend.put_buffer(data, path)
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/storage.py", line 540](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/storage.py#line=539), in put_buffer
    safe_write(full_path, buffer)
  File "[/workspace/.venv/lib/python3.12/site-packages/datajoint/utils.py", line 150](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/utils.py#line=149), in safe_write
    temp_file.rename(filepath)
  File "[/usr/lib/python3.12/pathlib.py", line 1365](http://localhost:4444/usr/lib/python3.12/pathlib.py#line=1364), in rename
    os.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/gpfs01/euler/User/tzenkel/dj_storage/processed/_hash/ageuler_tzenkel_test/rcuanjtva5uvcqjpauupxf4tnm.saving' -> '[/gpfs01/euler/User/tzenkel/dj_storage/processed/_hash/ageuler_tzenkel_test/rcuanjtva5uvcqjpauupxf4tnm](http://localhost:4444/lab/workspaces/auto-R/tree/GitRepos/djimaging/djimaging/user/dj_v2_try4_tz/notebooks/dj_storage/processed/_hash/ageuler_tzenkel_test/rcuanjtva5uvcqjpauupxf4tnm)'
"""

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
Cell In[42], line 2
      1 # Use one process to avoid concurrent writes to identical DataJoint hash-store files.
----> 2 Snippets().populate(processes=20, display_progress=True)
      3 Snippets()

File [/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py:439](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py#line=438), in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, max_calls, display_progress, processes, make_kwargs, priority, refresh, *restrictions)
    427     return self._populate_distributed(
    428         *restrictions,
    429         suppress_errors=suppress_errors,
   (...)    436         refresh=refresh,
    437     )
    438 else:
--> 439     return self._populate_direct(
    440         *restrictions,
    441         suppress_errors=suppress_errors,
    442         return_exception_objects=return_exception_objects,
    443         max_calls=max_calls,
    444         display_progress=display_progress,
    445         processes=processes,
    446         make_kwargs=make_kwargs,
    447     )

File [/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py:507](http://localhost:4444/workspace/.venv/lib/python3.12/site-packages/datajoint/autopopulate.py#line=506), in AutoPopulate._populate_direct(self, suppress_errors, return_exception_objects, max_calls, display_progress, processes, make_kwargs, *restrictions)
    500     del self.connection._conn.ctx
    501 with (
    502     mp.get_context(_MP_START_METHOD).Pool(
    503         processes, _initialize_populate, (self, None, populate_kwargs)
    504     ) as pool,
    505     tqdm(desc="Processes: ", total=nkeys) if display_progress else contextlib.nullcontext() as progress_bar,
    506 ):
--> 507     for status in pool.imap(_call_populate1, keys, chunksize=1):
    508         if status is True:
    509             success_list.append(1)

File [/usr/lib/python3.12/multiprocessing/pool.py:873](http://localhost:4444/usr/lib/python3.12/multiprocessing/pool.py#line=872), in IMapIterator.next(self, timeout)
    871 if success:
    872     return value
--> 873 raise value

FileNotFoundError: [Errno 2] No such file or directory: '/gpfs01/euler/User/tzenkel/dj_storage/processed/_hash/ageuler_tzenkel_test/rcuanjtva5uvcqjpauupxf4tnm.saving' -> '[/gpfs01/euler/User/tzenkel/dj_storage/processed/_hash/ageuler_tzenkel_test/rcuanjtva5uvcqjpauupxf4tnm](http://localhost:4444/lab/workspaces/auto-R/tree/GitRepos/djimaging/djimaging/user/dj_v2_try4_tz/notebooks/dj_storage/processed/_hash/ageuler_tzenkel_test/rcuanjtva5uvcqjpauupxf4tnm)'

To test

• CellAssignment
• AutoRoi (the example data already had roi masks for all cells)