Repository navigation
Gitea available fix - #559
Closed
greghaynes wants to merge 147 commits into
Closed
greghaynes wants to merge 147 commits into
greghaynes wants to merge 147 commits into
Conversation
This specification document proposes a significant architectural evolution to transition idpbuilder from a CLI-driven installation model to a controller-based architecture. Key changes include: - New Platform CR to orchestrate the entire IDP platform - Dedicated controllers for ArgoCD, Gitea, and Nginx components - Enhanced GitRepository and new Package CRs - Clear separation between infrastructure and platform services - 7-phase implementation plan over 20 weeks - Migration strategy with backward compatibility - Support for production workloads and virtualized control planes The new architecture will enable Kubernetes-native management through kubectl and GitOps tools, improve observability and debugging, and provide better extensibility for third-party integrations. Signed-off-by: Greg Haynes <greg@greghaynes.net>
Make the implementation plan phase-based without specific week durations to allow for more flexible planning and execution. Signed-off-by: Greg Haynes <greg@greghaynes.net>
Enable flexible provider selection for key platform components: - Git Provider: Support for Gitea (in-cluster), GitHub (external), and GitLab (future) - Gateway Provider: Support for Nginx Ingress, Envoy Gateway, and Istio Gateway (future) Key changes: - Replaced GiteaComponent and NginxComponent with GitProvider and Gateway CRs - Added provider field to allow selection between implementations - Updated Platform CR to support provider-specific configuration - Added GitProviderReconciler and GatewayReconciler with pluggable provider factories - Included examples for GitHub + Envoy Gateway configuration - Updated all diagrams and documentation to reflect provider model This architecture enables: - Using external Git providers (GitHub/GitLab) without in-cluster Git - Modern gateway implementations (Envoy/Istio) alongside Nginx - Easier testing and development with different provider combinations - Future extensibility for additional provider implementations Signed-off-by: Greg Haynes <greg@greghaynes.net>
Change Git provider architecture to use individual CR types instead of a single GitProvider CR with embedded configuration. This improves modularity and enables multiple providers simultaneously. Key changes: - Introduce separate CR types: GiteaProvider, GitHubProvider, GitLabProvider - Each provider type has its own reconciler - Define duck-typed interface with common status fields: * endpoint: External URL for UI/cloning * internalEndpoint: Cluster-internal API URL * credentialsSecretRef: Authentication credentials - Platform CR references a collection of provider CRs instead of inline config - GitRepository and other consumers use unstructured access to read common fields from any provider type Benefits: - Support multiple Git providers simultaneously (e.g., Gitea + GitHub) - Clear separation of concerns - each provider is independent - Easy to add new provider types without changing Platform CR - Other controllers can use any provider generically via duck typing - Provider CRs can exist independently and be shared across platforms - Simpler testing - can test each provider type in isolation Gateway providers updated similarly with NginxGateway, EnvoyGateway, etc. Duck typing pattern follows Kubernetes conventions (like Service, Ingress) allowing flexible composition without tight coupling. Signed-off-by: Greg Haynes <greg@greghaynes.net>
Refactor gateway architecture to use separate provider CR types with duck-typed common interface, mirroring the Git provider pattern. Key changes: - Replace single Gateway CR with provider-specific types: * NginxGateway - Nginx Ingress Controller * EnvoyGateway - Envoy Gateway * IstioGateway - Istio service mesh gateway - Each gateway provider has dedicated reconciler - Define duck-typed status interface: * ingressClassName: For creating Ingress resources * loadBalancerEndpoint: External access URL * internalEndpoint: Cluster-internal API endpoint - Platform CR uses 'gateways' collection for multiple gateway references - Added utility functions for duck-typed gateway provider access - Included detailed reconciler implementations for each provider type Benefits: - Support multiple gateway implementations simultaneously (e.g., Nginx for public traffic, Envoy for internal/service mesh) - Clear separation of concerns - each gateway provider is independent - Easy to add new gateway types (Traefik, Kong, etc.) - Components can dynamically choose which gateway to use - Ingress resources work with any provider via ingressClassName - Simpler testing with isolated provider reconcilers Added examples: - Multi-gateway configuration showing Nginx + Envoy - Component controller using duck-typed gateway access - Ingress creation working with any gateway provider type This completes the duck-typing pattern across both Git and Gateway providers, enabling true composability and extensibility. Signed-off-by: Greg Haynes <greg@greghaynes.net>
Revise architecture diagram and description to accurately represent the new provider-based design with duck-typed interfaces. Changes: - Updated diagram to show separate provider controller groups * Git Provider Controllers box with Gitea/GitHub/GitLab reconcilers * Gateway Provider Controllers box with Nginx/Envoy/Istio reconcilers - Each provider reconciler shows exposed duck-typed fields - Platform Services layer shows multiple providers coexisting - Added concrete examples of provider combinations: * Gitea for dev + GitHub for production * Nginx for public + Envoy for service mesh Added Key Architecture Principles section: 1. Duck Typing - common status fields without shared interface type 2. Composition - Platform references providers by name and kind 3. Extensibility - new providers without Platform CR changes 4. Independence - providers can exist and be managed separately 5. Flexibility - runtime provider selection by components Updated PlatformReconciler description: - References provider CRs instead of creating component CRs - Focuses on orchestration and status aggregation Updated reconciler descriptions: - GitRepositoryReconciler works with ANY Git provider - PackageReconciler references Git providers dynamically The design now clearly shows the composable, multi-provider architecture that enables production-grade flexibility. Signed-off-by: Greg Haynes <greg@greghaynes.net>
Extend the duck-typing architecture to GitOps management, replacing ArgoCDComponent CR with separate provider types for ArgoCD and Flux. Key changes: - Replace ArgoCDComponent with GitOps provider CRs: * ArgoCDProvider - ArgoCD GitOps engine * FluxProvider - Flux GitOps toolkit - Each GitOps provider has dedicated reconciler - Define duck-typed status interface: * endpoint: External URL for web UI * internalEndpoint: Cluster-internal API endpoint * credentialsSecretRef: Admin credentials - Platform CR updated to use 'gitOpsProviders' collection - Platform CR now only contains provider references (no inline component config) - Platform status aggregates provider status from referenced CRs - Added utility functions for duck-typed GitOps provider access - Included detailed reconciler implementations for both providers Benefits: - Support multiple GitOps implementations simultaneously (e.g., ArgoCD for applications, Flux for infrastructure) - Clear separation of concerns - each provider is independent - Easy to add new GitOps providers (e.g., Jenkins X) - Package controllers can dynamically choose which GitOps provider to use - Different teams can use different GitOps tools - Simpler testing with isolated provider reconcilers Updated documentation: - Platform CR spec now purely references providers (no inline configs) - Platform status aggregates provider readiness - High-level design diagram shows GitOps Provider Controllers box - Platform Services layer includes GitOps Providers - Bootstrap flow reflects provider-based architecture - Component interaction diagram shows provider references - Examples show multi-GitOps provider configuration - RBAC section includes all provider reconciler permissions - Phase 1 and Phase 2 updated with GitOps provider CRDs and reconcilers Architecture principles: 1. Duck Typing - common status fields without shared interface 2. Composition - Platform references providers by name and kind 3. Extensibility - add providers without Platform CR changes 4. Independence - providers exist and managed separately 5. Flexibility - runtime provider selection by components This completes the duck-typing pattern across Git, Gateway, and GitOps providers, creating a fully composable platform architecture. Signed-off-by: Greg Haynes <greg@greghaynes.net>
- Phase 1: Core end-to-end with existing providers (Gitea, Nginx, ArgoCD) - Phase 2: CLI integration and migration path - Phases 3-7: Incremental alternative provider additions - Phase 8: Production features and stabilization This approach validates the architecture early with a narrow vertical slice before adding alternative providers, reducing risk and enabling faster value delivery. Signed-off-by: Greg Haynes <greg.haynes@siemens.com>
Controller based arch
Restructure Phase 1 into four sub-phases where each sub-phase implements Platform CR with one provider CR at a time, enabling independent implementation, validation, and merging: - Sub-Phase 1.1: Platform CR + GiteaProvider (Git Provider Foundation) - Sub-Phase 1.2: Add NginxGateway Provider (Gateway Support) - Sub-Phase 1.3: Add ArgoCDProvider (GitOps Support) - Sub-Phase 1.4: Integration, Testing, and Documentation Each sub-phase delivers end-to-end functionality that can be: - Implemented independently - Validated with tests - Merged without breaking existing functionality - Used immediately by early adopters Benefits: - Faster feedback loops - Lower risk of large merge conflicts - Early validation of duck-typing pattern - Progressive complexity (git → gateway → GitOps) - Ability to course-correct between sub-phases Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…e 1 sub-phase Add explicit MIGRATE and REMOVE instructions for each sub-phase: - Sub-Phase 1.1: Migrate Gitea from localbuild/gitea.go, remove after migration - Sub-Phase 1.2: Migrate Nginx from localbuild/nginx.go, remove after migration - Sub-Phase 1.3: Migrate ArgoCD from localbuild/argo.go, remove after migration - Sub-Phase 1.4: Final cleanup verification and old file removal Each sub-phase now clearly states: - Which specific files contain the code to migrate - Which functions/logic to migrate from LocalbuildReconciler - Which files to remove after migration is complete - Success criteria include verifying old code is removed - Deliverables include MIGRATED and REMOVED items This ensures that as each provider is implemented, the corresponding old implementation is removed, progressively cleaning up LocalbuildReconciler until it becomes a minimal compatibility shim by the end of Phase 1. Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…ller - Created v1alpha2 API version with Platform and GiteaProvider CRDs - Implemented duck-typing infrastructure for Git providers - Created GiteaProviderReconciler with Gitea installation logic - Generated CRD manifests and deepcopy code - Added unit tests for duck-typing utilities Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…site [WIP] Create a new skeleton static site for Cloudflare Pages
…-runner Migrate all workflows to homelab-mini-runner and fix build validation
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…ation discovery (#38) * Initial plan * Add owner reference pattern to controller architecture specifications Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Add troubleshooting guide and migration path for owner reference pattern Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…put modes and package-level progress (#34)
…Ingress resources (#42)
…ints (#43) * Initial plan * Add URL validation for GitRepository endpoint fields - Add comprehensive validation to ensure endpoint and internalEndpoint are not empty - Validate URLs start with http:// or https:// - Validate URLs have content beyond the protocol prefix - Add comprehensive test coverage for all validation scenarios - Fixes issue where empty/invalid endpoint causes GitRepository creation to fail Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Fix length validation for HTTPS URLs - Change from length comparison to exact string comparison - Explicitly check for both "http://" and "https://" edge cases - Add test case for "https://" to ensure it's properly rejected - Addresses code review feedback about incorrect length check Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Refactor URL validation to reduce code duplication - Extract URL validation logic into validateGitURL helper function - Add comprehensive unit tests for validateGitURL function - Improves code maintainability by avoiding duplicate validation logic - Addresses code review feedback about code duplication Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…reconciliation (#45) * Initial plan * Fix gitea-credential secret creation order to prevent deployment errors Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Refactor secret creation to reduce code duplication Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Use secret object properties instead of hardcoded values for consistency Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Final: Fix gitea-credential secret creation order Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…nce (#46) * Initial plan * Add nil receiver checks to Reporter methods to prevent panic with --status-output=none Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Add interface test to verify nil reporter fix Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Final verification and documentation complete Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…hitecture (#40) * Initial plan * Add test coverage improvement implementation plan Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Update implementation README to include test coverage plan Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Add test coverage improvement plan to docs site navigation Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Remove localbuild references and prioritize v2 controller architecture Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Add conflict error handling to GiteaProvider controller Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Fix incorrect success report when cancelling during package sync Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Handle GiteaProvider readiness check as requeue instead of error - Add errGiteaProviderNotReady sentinel error - Return sentinel error when GiteaProvider is not ready in reconcileGitRepo - In Reconcile, check for this error and requeue with delay instead of hard error - This prevents the ERROR log message and exponential backoff when GiteaProvider is still initializing Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Add scrolling to documentation sidebar navigation Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
… architecture spec (#44) * Initial plan * Document next steps to remove localbuild controller Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Add comprehensive implementation guides for removing localbuild controller Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Complete documentation for removing localbuild controller - all steps identified Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Convert ASCII diagrams to mermaid format for consistency Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…tput (#51) * Initial plan * Add gitea substep tracking in parallel with argocd Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Add tests for parallel argocd and gitea tracking Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Address code review feedback: add constants and helper function Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Use predefined error for gitea install timeout Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Add comprehensive test coverage for ArgoCD provider controller Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Apply go fmt to test file Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Add comprehensive tests for GiteaProvider controller - Phase 1 - Added 18 new test functions covering various aspects - Improved coverage from 20.1% to 46.4% - Tests cover: - Finalizer addition and removal - Phase transitions - Admin secret creation and management - Password generation - Configuration variations - Nginx webhook readiness checks - Status condition updates - Idempotency - Error handling - Deletion handling Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Add Phase 2 tests for GiteaProvider controller - reach 62% avg coverage - Added 19 more test functions (37 total) - Individual function coverage: - buildConfigFromSpec: 100% - ensureAdminSecretWithoutToken: 100% - createAdminSecretIfNotExists: 84.2% - isNginxAdmissionWebhookReady: 83.3% - isGiteaReady: 74.1% - installGiteaResources: 73.3% - reconcileGitea: 66.7% - handleDeletion: 62.5% - Reconcile: 37.9% - Tests added in this phase: - Nginx webhook readiness with/without endpoints - Deployment readiness checks - Configuration defaults and variations - Full reconciliation cycle - Error handling and phase transitions - Namespace creation with existing namespace - Multiple endpoint subsets - Protocol, host, and port defaulting Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Format test file with go fmt --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
…57) * Initial plan * Add ArgoCD provider creation and tracking in platform resource - Added createArgoCDProvider function to create ArgoCDProvider CR during build - Updated createPlatform to include ArgoCD provider in GitOpsProviders list - ArgoCD provider is now properly created in the argocd namespace with auto-generated credentials - Platform resource now tracks ArgoCD as a GitOps provider alongside Git providers Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Add ArgoCD provider examples and documentation - Created argocdprovider.yaml example showing ArgoCD provider configuration - Created platform-complete.yaml showing a complete platform with all provider types - Updated README.md to include ArgoCD provider in architecture overview - Added documentation for GitOps provider duck-typed fields - Updated quick start guide to include ArgoCD provider creation steps Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Add documentation sync validation tooling Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Improve validation script documentation and robustness Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
* Initial plan * Add CRD phase tracking for package installation - Added GetProviderPhase utility to extract phase from v1alpha2 CRDs - Extended status reporter with UpdateSubStepWithPhase method - Updated localbuild controller to track GiteaProvider, ArgoCDProvider, and NginxGateway phases - Phase information now displayed in sub-steps (e.g., "gitea (Installing)", "argocd (Ready)") - Added comprehensive tests for phase extraction and display Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> * Address code review feedback - Renamed gitea-specific constants to generic ones (statusPollInterval, installTimeout) - Fixed empty phase handling to clear previous phase information - Added test for empty phase clearing behavior - All tests pass Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: greghaynes <42657+greghaynes@users.noreply.github.com>
- Pass Protocol, Host, Port, and UsePathRouting from build config to GiteaProvider spec - Update controller defaults from http:8080 to https:8443 - Fix API readiness check to use external URL instead of internal service URL - Resolves issue where controller couldn't reach Gitea API on host machine
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.