How the neetbox.integrations are designed

The problem is, as we want a not-dependency-intensive neetbox, we are trying to remove codes with heavy dependencies from core functions. To be specific, in order to avoid installing package 'thop'(which depends on torch) for non-torch users, we have removed import torch from the headings of each python file into the line where the packages are required to overcome the import errors for most of the packages. For example, in neetbox.torch.profile:

from thop import profile as _profile
def profile():
    case 1: ... return
    case 2: ... return
    case that needs thop: ... return

was changed to:

def profile():
    case 1: ... return
    case 2: ... return
    case that needs thop: 
            assert pkg.is_installed('thop')
            from thop import profile as _profile
            ... 
            return

It was simple until we added neetbox.integrations. I'm not a pro for Python things, but I found it much more clear if we move things related to third parties into neetbox.integrations. So what does the neetbox.integrations stand for? In general, neetbox.integrations contains plugins for the original neetbox codes which empowers neetbox with third-party-compbilities.

I first mentioned the idea in #19 , in which I suggest that we should move 'nvidia-smi' and 'conda' command line getters out of neetbox.utils. However, since I'm a little confused about how to effectively add plugins into python class, I'm not sure what to do next to form the general framework for neetbox.integrations. Anyway, I think I need some help. Further discussions and methods are welcomed and appreciated.

I think we can use modulefinder a builtin python module to integrate nvidia-smi or conda like command, and achieve the goal above. here is the doc. modulefinder

Currently, I'm trying with patch as a workaround. But I think it's not gonna be so easy. I will put my new ideas here as soon I get them.

Not to mention, an example here:

# package neetbox.integrations.tensorboard
from neetbox.logging.logger import Logger
from neetbox.utils.utils import patch
from neetbox.integrations import pkg
from typing import Any

assert pkg.is_installed('tensorboard',try_install_if_not=True)
import tensorboard

@patch
def write_tensorboard(self:Logger, what:Any,):
    pass # to write something into tensorboard

The code adds a new function caller 'write_tensorboard' into class Logger. So later in my code, I can do such thing:

from neetbox.logging import logger
import neetbox.integrations

logger.write_tensorboard(whatever)

It's a kinda solution, but idk how my IDEs perform with the code completion... I'll find it out later when I have time.

Another example for patch in neetbox.utils.utils:

@PaperCube

@PaperCube @PommesPeter What do you think? Do you find it plausible?

I'm running out of ideas. I'm rubbish indeed.

I think @patch is a nice design. Nevertheless, I think traditional OOP abstract classes might also be a feasible solution.

another approach scans all submodules in neetbox.integrations and let them register a self loading function after workspace is loaded. #72
cbcc66d

it's kinda piece of scaffold, I will make it into a stronger implementation later, just keep you guys up to date.

So, how should we design the neetbox.intergrations.

Navigation Menu

Repository navigation

How the `neetbox.integrations` are designed #20

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

How the neetbox.integrations are designed #20

Description

Activity

PommesPeter commented on Mar 24, 2023

visualDust commented on Apr 11, 2023

visualDust commented on Apr 11, 2023

visualDust commented on Apr 11, 2023

visualDust commented on Apr 11, 2023

visualDust commented on Apr 11, 2023

PaperCube commented on Apr 12, 2023

visualDust commented on Nov 26, 2023

PommesPeter commented on Nov 27, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

How the `neetbox.integrations` are designed #20