Velocloud (VMware SD-WAN, Broadcom, Arista) VCO Event and Flow Logs Fetcher

This repository contains Python scripts to fetch event logs and flow visibility metrics from a Velocloud VCO (Virtual Cloud Orchestrator) instance using the Velocloud API. Velocloud technology has been known as VMware SD-WAN, and is now part of Broadcom and Arista Networks.

Background

Velocloud's (VMware SD-WAN, Broadcom, Arista) VCO API allows you to retrieve event logs and flow logs for your enterprise. However, due to a long-standing bug in the VCO portal, only the first page of results is returned by default, making it difficult to collect complete logs for analysis or archiving.

These scripts resolve this problem by automatically handling pagination, allowing you to fetch and save all available event and flow logs, not just the first page.

Note: Experience shows that the available data in the VCO is typically reset around the start of each new year. This means that, in practice, you can retrieve at most one year of logs, but sometimes the available data may be much less (close to zero), depending on when the reset occurred.

Features

• Fetches all event logs and flow visibility metrics from Velocloud VCO (VMware SD-WAN, Broadcom, Arista), overcoming the single-page limitation in VCO.
• Uses the official Velocloud API.
• Easy to configure via config.jsonc.
• Modular and easy to modify for your own needs.
• Can be scheduled or repeated as needed (e.g., via cron jobs).
• Output in JSON, which can easily be processed or loaded in visidata.
• Uses a concurrent process to fetch the data and to actually save the data to speed up the process when getting huge data
• Memory optimized: Uses pagination and json-"streaming" to disk to prevent the requirement of huge amount of memory when processing huge flow or event logs
• Cross-Platform & macOS Safe: Utilizes safe_qsize() to handle macOS POSIX semaphore limitations gracefully.

Cross-Platform & macOS Multiprocessing Notes

The extraction scripts use Python's multiprocessing.Queue to decouple API fetch workers from disk-writing loops.

On macOS (Darwin), calling multiprocessing.Queue.qsize() raises NotImplementedError because Apple's OS kernel does not implement POSIX sem_getvalue(). To maintain multi-OS compatibility:

• Both getVCOEnterpriseGetEnterpriseEvents.py and getVCOEnterpriseGetEdgeFlowVisibilityMetrics.py use a safe_qsize(queue) wrapper.
• On Linux systems, safe_qsize() returns the exact integer queue length.
• On macOS systems, safe_qsize() catches NotImplementedError and returns "N/A", allowing logging and event streaming to proceed uninterrupted.

Usage

Quickstart with Makefile

You can run make or make help at any time to list all available target commands.

# 1. Setup virtual environment & dependencies
make venv
source .venv/bin/activate

# 2. Bootstrap configuration file
make config  # Copies config.json.sample to config.jsonc

# 3. Data Extraction & Security Analysis
make fetch-events                            # Fetch enterprise event logs
make fetch-flows                             # Fetch edge flow metrics
make audit INPUT=output-EnterpriseEvents...  # Run security audit on JSON dump
make pipeline INPUT=output-EnterpriseEvents.. # Run audit + WHOIS/IP enrichment pipeline

# 4. Testing & Maintenance
make test                                    # Run test suite
make clean-logs                              # Clear log files

Unified CLI (velocloud.py)

You can also run all extraction and security analysis workflows through the unified velocloud.py CLI:

# Fetch enterprise events
python velocloud.py events --start_human "2026-08-01 00:00:00" --stop_human "2026-08-12 23:59:59"

# Fetch edge flow metrics
python velocloud.py flows --start_human "2026-08-01 00:00:00" --stop_human "2026-08-12 23:59:59"

# Security audit & IP enrichment
python velocloud.py audit -i output-EnterpriseEvents_2025-09-01_00-00-00_to_2026-08-12_23-59-59.json
python velocloud.py enrich -i output-EnterpriseEvents_2025-09-01_00-00-00_to_2026-08-12_23-59-59.security_summary.json

# Full automated pipeline (audit + WHOIS enrichment in sequence)
python velocloud.py pipeline -i output-EnterpriseEvents_2025-09-01_00-00-00_to_2026-08-12_23-59-59.json

Standalone Script Execution

• To fetch flow visibility metrics:

  python getVCOEnterpriseGetEdgeFlowVisibilityMetrics.py [--help] [--start_human "YYYY-MM-DD HH:MM:SS"] [--stop_human "YYYY-MM-DD HH:MM:SS"]

  • You can override the time range and other options via command-line arguments, environment variables, or config file. Command-line arguments take highest priority.
  • For all options, run:

  python getVCOEnterpriseGetEdgeFlowVisibilityMetrics.py --help

• To fetch event logs:

  python getVCOEnterpriseGetEnterpriseEvents.py [--help] [--start_human "YYYY-MM-DD HH:MM:SS"] [--stop_human "YYYY-MM-DD HH:MM:SS"]

• To perform a security audit on extracted event logs (user logins, source IPs, admin changes):

  python analyzeVCOEnterpriseEventsSecurityAudit.py [-i output-EnterpriseEvents_...json]

  • Generates matching .security_audit.json and .security_summary.json output files mapped dynamically from the input filename.
  • Free of hardcoded IP addresses, user accounts, or environment names.

• To enrich extracted IP addresses with WHOIS/ASN and geolocation metadata:

  python enrichVCOEnterpriseEventsIPOwnership.py [-i output-EnterpriseEvents_...security_summary.json]

  • Generates matching .ip_ownership_report.md report mapped dynamically from the summary filename.

  • Portable across any environment with zero hardcoded IP addresses.

  • For all options, run:

  python getVCOEnterpriseGetEnterpriseEvents.py --help

4. Run the test suite:
   • To run unit tests for the event log script:

     python3 -m unittest test_getVCOEnterpriseGetEnterpriseEvents.py

   • To run unit tests for the flow visibility metrics script:

     python3 -m unittest test_getVCOEnterpriseGetEdgeFlowVisibilityMetrics.py

   • To run the tests in verbose mode (see which tests are executed):

     python3 -m unittest -v test_getVCOEnterpriseGetEdgeFlowVisibilityMetrics.py

   • To run the event log script tests in verbose mode:

     python3 -m unittest -v test_getVCOEnterpriseGetEnterpriseEvents.py

   • To run extra event log tests (edge cases, error handling, formatting):

     python3 -m unittest -v test_getVCOEnterpriseGetEnterpriseEvents_extra.py

Running All Tests

You can run all tests in the project at once using either of these methods:

1. One-liner: Discover and run all tests automatically

This will find and run all test files matching test*.py in the current directory and subdirectories:

python3 -m unittest discover

For more detailed output, use:

python3 -m unittest discover -v

2. Run specific test files manually

You can also run individual test files directly, for example:

python3 -m unittest test_getVCOEnterpriseGetEnterpriseEvents.py
python3 -m unittest test_getVCOEnterpriseGetEdgeFlowVisibilityMetrics.py
python3 -m unittest -v test_getVCOEnterpriseGetEnterpriseEvents_extra.py
python3 -m unittest -v test_getVCOEnterpriseGetEdgeFlowVisibilityMetrics_extra.py

Both methods are valid; the first is most convenient for running all tests at once, while the second allows you to target specific test suites.

Python Environment Management

This project was developed and tested using pyenv to easily manage Python versions and prevent conflicts with the system Python and its modules. Using pyenv allows you to:

• Install and switch between multiple Python versions without affecting your system Python.
• Avoid dependency collisions with other Python projects or system tools.

Alternatively, using a standard Python venv (virtual environment) is also perfectly fine and recommended for most users. Both approaches will help you keep dependencies isolated and your system clean.

To use venv:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

To use pyenv, see the pyenv documentation for installation and usage instructions.

Customization

• The scripts are modular and well-commented, making it easy to adapt them to your specific requirements.
• You can schedule them using cron or any other scheduler for regular log collection.

Requirements

See requirements.txt for required Python packages.

Contributing

Contributions, bug reports, and suggestions are welcome! Please open an issue or submit a pull request.

License

This project is licensed under the MIT License. See LICENSE for details.

Disclaimer

These scripts are provided as-is and are not officially supported by Velocloud. Use at your own risk.

Why Not Use a Generated Library from the Velocloud Swagger Spec?

While Velocloud provides a Swagger/OpenAPI specification for its API, this project does not use a generated client library for several reasons:

• Incomplete/Outdated API Documentation: The official Swagger spec and API documentation from Velocloud have historically been incomplete or out of date. Critical features—such as the correct handling of pagination via nextPageLink—are either missing, poorly documented, or behave differently in practice than described.
• Pagination Issues: The nextPageLink mechanism, which is essential for retrieving more than the first page of results, has been especially under-documented and confusing. Many users have found that generated clients do not handle this correctly out of the box.
• Practical Reliability: By using direct, well-tested requests and custom logic, this project ensures robust handling of pagination and error cases, even when the API documentation is lacking or ambiguous.
• Transparency and Debugging: Writing the logic explicitly makes it easier to debug, adapt, and extend, especially when the API changes or behaves unexpectedly.

If Velocloud's API documentation and generated libraries improve in the future, it may become practical to use them. For now, this approach is the most reliable for real-world data collection.

Testing Across Multiple Python Versions (pyenv)

To ensure compatibility with all Python versions you have installed via pyenv, use the provided script:

bash test_all_pyenv_versions.sh

This script will:

• Loop through all Python versions installed by pyenv.
• For each version:
  • Create a temporary virtual environment.
  • Install all dependencies from requirements.txt.
  • Run all unittests.
  • Clean up the virtual environment.

Requirements:

• pyenv must be installed and initialized in your shell.
• All desired Python versions should be installed via pyenv (e.g., pyenv install 3.7.16).
• requirements.txt must be present in the project root.

This is a convenient way to verify that your code and dependencies work across all supported Python versions.

Data Conversion

You can easily convert the output JSON files (or other formats) using Python one-liners with pandas:

• Convert JSON to CSV:

  python -c "import pandas as pd; pd.read_json('input.json').to_csv('output.csv', index=False)"

• Convert JSON to Excel (XLSX):

  python -c "import pandas as pd; pd.read_json('input.json').to_excel('output.xlsx', index=False)"

• Convert CSV to JSON:

  python -c "import pandas as pd; pd.read_csv('input.csv').to_json('output.json', orient='records', lines=False)"

• Convert CSV to Excel:

  python -c "import pandas as pd; pd.read_csv('input.csv').to_excel('output.xlsx', index=False)"

• Convert Excel to CSV:

  python -c "import pandas as pd; pd.read_excel('input.xlsx').to_csv('output.csv', index=False)"

All of these require the pandas package (not in requirements.txt, so you may need to execute pip instal pandas)

---

Feel free to contribute improvements or report issues!