Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,80 @@ and this project adheres to [Semantic Versioning 2.0.0](https://semver.org/spec/
middle of the operation that reported it. Numbers and quoted strings keep
their colors.

- Carry out a scheduled snapshot restore in the directories of the server.
With `--base-directory`, the restore that runs at the next start worked on
the Manager's files and the `custom_nodes` directory inside the ComfyUI
directory instead of those of the base directory: it did not see the node
packs of the server, and ended with `Restore snapshot failed`.

- Start with a node pack whose `.git/config` Manager could not read. A url
with a `%`, such as one with `%20`, a key without a value, such as `bare`
alone, a byte order mark at the start of the file, or on Windows text that
the code page of the system cannot decode, such as a Korean user name
written in UTF-8, raised an error while Manager was loading: ComfyUI did not
start, or with `--enable-manager-legacy-ui` started without Manager. These
files are read now, and for a file that cannot be read the url of that pack
is unknown and a warning names the file once.

- Run the git executable set as `git_exe` in `config.ini`. The path was read
but not used: with the GitPython backend, git commands kept running the
`git` found on `PATH`, and without a `git` on `PATH` the pygit2 backend was
selected, or with the legacy UI Manager did not load. Both servers, the git
helper and `cm-cli` now run the configured executable. For the servers and
what they start it no longer has to be on `PATH`; `cm-cli` run by hand still
needs a git on `PATH`. A value that is not a working git is reported, and
the default git is used. The value is read as it is written: with a `%` in
it, as in a path of Windows, Manager read every setting of `config.ini` as
its default. The configured executable is also run when pygit2
is asked for with `CM_USE_PYGIT2=1` but is not installed.
- Start without a git. With neither a `git` on `PATH` nor pygit2, ComfyUI did
not start, or with the legacy UI started without Manager; Manager now loads
and reports that git is missing, and what needs git fails when it is used.
With pygit2 and no `git`, the legacy UI started without Manager; it now
loads on the pygit2 backend, but what the legacy server does with GitPython
itself still needs a git.
- Run the start-up setup of the default server. It was never called, so the
server did not read the ComfyUI revision, did not log the ComfyUI version,
did not call the handlers that custom nodes register for the revision, and
did not check for nodepacks in an invalid state.

- Report the steps of a snapshot restore that failed. The summary listed a
nodepack as `INSTALLED`, `DISABLED` or `ENABLED` without looking at the
result of the step: a clone that failed, for example because the directory
already existed or the url could not be reached, was `INSTALLED`. A step
that fails is listed as `FAILED`, and Manager itself is no longer listed as
`DISABLED`. A Registry that cannot be reached ended the restore at the first
nodepack to install from it, and so did a nodepack whose directory could not
be moved to disable or enable it; such a nodepack is listed as `FAILED`, and
the restore goes on.
- Restore the commit of the snapshot for every git nodepack. A git nodepack
that the restore had to clone stayed at the newest commit; it is checked out
at the commit of the snapshot, and its install script runs after that. An
installed git nodepack that the Registry does not know was checked out only
when `cm-cli` was run from `custom_nodes`, as a scheduled restore does; it
is checked out from any directory. The submodules of a git nodepack are
brought to the commits that the commit of the snapshot names; a submodule
that was never fetched is fetched only when the nodepack itself is checked
out.
- Install the requirements and run the install script of a Registry nodepack
that a snapshot restore installs. The nodepack was unpacked and nothing
more. It is listed as `FAILED` when they fail.
- Try a requirement or an install script that failed again. It was taken as
done until the next start: the next nodepack that names the requirement
skipped it, and installing or fixing the nodepack again did not run the
script.
- Update the submodules of a submodule with the pygit2 backend, and fail when
the submodules could not be updated. pygit2 updated the submodules of the
nodepack and not those below them, which git does. When it could not update
them and there was no `git` to do it in its place, or that `git` did not
answer in time, a message was printed and the install or the update went on
as if they were updated. On Windows an update still ends as done: the helper
that runs it prints the failure, and its result is not read.
- Record the commit of a git nodepack whose refs are packed, as after
`git gc`, in a snapshot. It was recorded as `unknown`. A `HEAD` that is not
UTF-8 raised an error where the commit of a nodepack is read; the commit of
such a nodepack is now unknown.

- Refuse a `db_mode` or an `update_policy` that Manager does not know in the
legacy server. With `--enable-manager-legacy-ui`, the two setting routes
stored whatever value a request carried and wrote it to `config.ini`. They
Expand Down
40 changes: 21 additions & 19 deletions cm_cli/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@


import utils.extra_config
from comfyui_manager.common.git_compat import setup_git_environment # before the modules that import GitPython
from comfyui_manager.common import manager_util
from comfyui_manager.common import cm_global
from comfyui_manager.legacy import manager_core as core
Expand Down Expand Up @@ -135,25 +136,26 @@ def set_no_deps(self, no_deps):
self.no_deps = no_deps

def set_user_directory(self, user_directory):
if user_directory is None:
return

extra_model_paths_yaml = os.path.join(user_directory, 'extra_model_paths.yaml')
if os.path.exists(extra_model_paths_yaml):
utils.extra_config.load_extra_path_config(extra_model_paths_yaml)

context.update_user_directory(user_directory)

if os.path.exists(context.manager_pip_overrides_path):
with open(context.manager_pip_overrides_path, 'r', encoding="UTF-8", errors="ignore") as json_file:
cm_global.pip_overrides = json.load(json_file)

if os.path.exists(context.manager_pip_blacklist_path):
with open(context.manager_pip_blacklist_path, 'r', encoding="UTF-8", errors="ignore") as f:
for x in f.readlines():
y = x.strip()
if y != '':
cm_global.pip_blacklist.add(y)
if user_directory is not None:
extra_model_paths_yaml = os.path.join(user_directory, 'extra_model_paths.yaml')
if os.path.exists(extra_model_paths_yaml):
utils.extra_config.load_extra_path_config(extra_model_paths_yaml)

context.update_user_directory(user_directory)

if os.path.exists(context.manager_pip_overrides_path):
with open(context.manager_pip_overrides_path, 'r', encoding="UTF-8", errors="ignore") as json_file:
cm_global.pip_overrides = json.load(json_file)

if os.path.exists(context.manager_pip_blacklist_path):
with open(context.manager_pip_blacklist_path, 'r', encoding="UTF-8", errors="ignore") as f:
for x in f.readlines():
y = x.strip()
if y != '':
cm_global.pip_blacklist.add(y)

# the configuration of the user directory is known from here on
setup_git_environment(core.get_config()['git_exe'])

def update_custom_nodes_dir(self, target_dir):
import folder_paths
Expand Down
119 changes: 97 additions & 22 deletions comfyui_manager/common/git_compat.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@

import os
import re
import shutil
import subprocess
import sys
from abc import ABC, abstractmethod
from collections import deque
Expand Down Expand Up @@ -52,25 +54,71 @@ def _to_https_url(url):
# Backend selection
# ---------------------------------------------------------------------------

def find_git_executable(git_exe, timeout=10):
"""The full path of the program `git_exe` if it answers `version` as git does; None otherwise."""
# A name without a directory is looked up on PATH; GitPython would take it
# as a file of the current directory.
path = shutil.which(git_exe)
if path is None:
return None

try:
res = subprocess.run([path, 'version'], capture_output=True, stdin=subprocess.DEVNULL, timeout=timeout)
except (OSError, subprocess.SubprocessError):
return None

if res.returncode != 0 or not res.stdout.startswith(b'git version'):
return None

return os.path.abspath(path)


_PYGIT2_REQUESTED = os.environ.get('CM_USE_PYGIT2', '').strip() == '1'
USE_PYGIT2 = _PYGIT2_REQUESTED

# The git set as `git_exe`, from prestartup or from the process that runs git_helper.
# GitPython is told before it is imported, so that a git that is not on PATH can be named.
_GIT_EXE = os.environ.get('GIT_EXE_PATH')
_GIT_EXE_FOUND = None


def _find_configured_git():
found = find_git_executable(_GIT_EXE)
if found is not None:
os.environ['GIT_PYTHON_GIT_EXECUTABLE'] = found
else:
print(f"[ComfyUI-Manager] `git_exe = {_GIT_EXE}` is not a working git executable. The default git is used.", file=sys.stderr)
return found


if not USE_PYGIT2 and _GIT_EXE:
_GIT_EXE_FOUND = _find_configured_git()

if not USE_PYGIT2:
try:
import git as _git
_git.Git().execute(['git', '--version'])
_git.Git().execute([_git.Git.GIT_PYTHON_GIT_EXECUTABLE, '--version'])
except Exception:
USE_PYGIT2 = True

if USE_PYGIT2:
# Without a git, GitPython raises already when it is imported, unless it
# is told to keep quiet. The legacy modules import it themselves.
os.environ.setdefault('GIT_PYTHON_REFRESH', 'quiet')

try:
import pygit2 as _pygit2
except ImportError:
# pygit2 not available either — fall back to GitPython and let it
# fail at the point of use, preserving pre-existing behavior.
# fail at the point of use.
USE_PYGIT2 = False
if _PYGIT2_REQUESTED and _GIT_EXE:
# pygit2 was asked for, so the setting was not looked at above
_GIT_EXE_FOUND = _find_configured_git()
_PYGIT2_REQUESTED = False
import git as _git
if not _git.GIT_OK:
print("[ComfyUI-Manager] No git executable was found and pygit2 is not installed. What needs git will fail.", file=sys.stderr)
else:
# Disable owner validation once at import time.
# Required for Desktop 2.0 standalone installs where repo directories
Expand Down Expand Up @@ -237,7 +285,7 @@ def checkout(self, ref): ...
def checkout_new_branch(self, branch_name: str, start_point: str): ...

@abstractmethod
def submodule_update(self): ...
def submodule_update(self, init=True): ...

@abstractmethod
def clear_cache(self): ...
Expand Down Expand Up @@ -424,8 +472,8 @@ def checkout(self, ref):
def checkout_new_branch(self, branch_name, start_point):
self._repo.git.checkout('-b', branch_name, start_point)

def submodule_update(self):
self._repo.git.submodule('update', '--init', '--recursive')
def submodule_update(self, init=True):
self._repo.git.submodule('update', *(['--init'] if init else []), '--recursive')

def clear_cache(self):
self._repo.git.clear_cache()
Expand Down Expand Up @@ -845,28 +893,45 @@ def checkout_new_branch(self, branch_name, start_point):
self._repo.checkout(branch)
self._repo.set_head(branch.name)

def submodule_update(self):
def _update_submodules(self, repo):
# One that `.gitmodules` alone names has no commit, and a commit that `.gitmodules` does not name has no url:
# neither can be fetched, and pygit2 refuses the second.
names = [submodule.name for submodule in repo.submodules
if submodule.head_id is not None and submodule.url is not None]
if names:
repo.submodules.init(names)
repo.submodules.update(names)

# as `--recursive`: the submodules of a submodule
for name in names:
self._update_submodules(repo.submodules[name].open())

def submodule_update(self, init=True):
try:
self._repo.submodules.init()
self._repo.submodules.update()
except Exception:
if init:
self._update_submodules(self._repo)
else:
# one that was never fetched is left alone; pygit2 refuses it and would not come to the others
fetched = [submodule.name for submodule in self._repo.submodules
if os.path.exists(os.path.join(self._working_dir, submodule.path, '.git'))]
if fetched:
self._repo.submodules.update(fetched)
except Exception as e:
import subprocess
try:
result = subprocess.run(
['git', 'submodule', 'update', '--init', '--recursive'],
['git', 'submodule', 'update', *(['--init'] if init else []), '--recursive'],
cwd=self._working_dir,
capture_output=True, timeout=120,
)
if result.returncode != 0:
raise GitCommandError(
f"submodule update failed (exit {result.returncode}): "
f"{result.stderr.decode(errors='replace')}")
except FileNotFoundError:
print("[ComfyUI-Manager] pygit2: submodule update requires system git (not installed)", file=sys.stderr)
except GitCommandError:
raise
except Exception as sub_e:
print(f"[ComfyUI-Manager] pygit2: submodule update failed: {sub_e}", file=sys.stderr)
# no git to do it in the place of pygit2: the caller must not take the update as done
raise GitCommandError(f"submodule update failed: {e}") from e

if result.returncode != 0:
raise GitCommandError(
f"submodule update failed (exit {result.returncode}): "
f"{result.stderr.decode(errors='replace')}")

def clear_cache(self):
pass
Expand Down Expand Up @@ -938,7 +1003,17 @@ def clone_repo(url, dest, progress=None):

def setup_git_environment(git_exe):
"""Configure the git executable path (GitPython only)."""
if USE_PYGIT2:
if USE_PYGIT2 or not git_exe:
return
if git_exe:
_git.Git().update_environment(GIT_PYTHON_GIT_EXECUTABLE=git_exe)

if git_exe == _GIT_EXE:
# checked, and reported if it is no git, when this module was imported
found = _GIT_EXE_FOUND
else:
found = find_git_executable(git_exe)
if found is None:
print(f"[ComfyUI-Manager] `git_exe = {git_exe}` is not a working git executable. The default git is used.", file=sys.stderr)

# GitPython may have been imported before this module, with the git on PATH
if found is not None and _git.Git.GIT_PYTHON_GIT_EXECUTABLE != found:
_git.refresh(found)
Loading
Loading