fastACCORD

A fast, scalable implementation of the ACCORD (based on the Semismooth Newton method and Proximal Gradient Descent) using PyTorch. Designed for large-scale data optimization, this project supports multiprocessing and GPU acceleration to efficiently solve massive-scale optimization problems.

Requirements

Install the required Python libraries using the command below before running the project. (If you are running in a GPU environment, it is highly recommended to install the appropriate version of PyTorch from the official website first.)

pip install -r requirements.txt

Configuration

This project manages hyperparameters and execution settings via YAML files. You can modify the YAML files in the cfg/ directory.

Key configuration parameters (cfg/config.yaml):

• data_file: Path to the input data file (NumPy .npy format). The data matrix is expected to be in the form of $X \in \mathbb{R}^{n \times p}$, and all entries are assumed to be normalized.
• out_file: Prefix for the output results.
• l12: List of L1 and L2 penalty values to be used. Each element is a tuple of the form $(l_1, l_2)$.
• max_round, max_outer: Maximum number of iterations for the algorithm.
• CUDA: Set to True to enable GPU acceleration.
• total_device: The number of worker processes to launch for multiprocessing.

Usage

You can execute the main script from the command line using the provided configuration file.

# Example of running the program with a configuration file and saving the log
python main.py --config cfg/config.yaml --log log/main.log

For the debiasing procedure, run the following command:

# Example of running the debiasing procedure with a configuration file and saving the log
python main.py --config cfg/config_debias.yaml --log log/main_debias.log

Note that for the debiasing procedure, you need to provide the path to the estimated omega matrix in the resume_from_whole parameter.

Usage as a Python Module

If you wish to import the core functions directly into your scripts or Jupyter Notebooks, you can use the module as shown below:

import numpy as np
import torch
from src.accord_snh import accord_snh
from src.logger import setup_logger
from scipy import sparse

# 1. Load data and convert to tensor
X_numpy = np.load("data/X_100.npy")
X_tensor = torch.from_numpy(X_numpy).to(torch.float64)

# 2. Prepare parameters and configuration
lamb_tup = (0.3, 0.01) # L1, L2 penalty
cfg = {
    "eps": 1e-5, "rho": 0.3, "max_outer": 100, "max_inner": 10,
    "tau_init": 0.01, "max_round": 10, "max_active": 1000,
    "float64": True, "sigma": 0.2, "gamma": 0.6, "log_interval": 10,
    "run_hash": "test_hash"
}
logger = setup_logger(cfg, rank=0)

# 3. Execute the algorithm
# Use the `part` argument to specify the row range for partial processing (e.g., [0, row_max] for full range)
omega_result, is_converged = accord_snh(
    X = X_tensor, 
    lamb_tup = lamb_tup, 
    cfg = cfg, 
    logger = logger, 
    part = (0, X_tensor.shape[0]),
    penalty_idx = 0,
    task_idx = 0
)

print("Result Sparse Matrix:", omega_result)
print("Converged:", is_converged)

Processing outputs

The raw outputs (which are estimated omega's, or the reparamaterized version of the precision matrix) are saved in the form of .npz files, and they are chunked according to the row_divide parameter. Check sample_process.ipynb for an example of how to merge them, transform them into the precision matrix, and finally into the correlation matrix. The hyperparamter selection with extended (psuedo-)BIC is also demonstrated in sample_process.ipynb.