feat: Enterprise AI Adoption Audit Meta-Skill - #8
Merged
Conversation
- Add comprehensive AI technology detection and Copilot integration analysis - Build skill orchestration engine with maturity-based execution paths - Create AI-focused ADR generation templates and rubrics - Update plugin.json to focus on AI adoption as sole exposed skill - Consolidate artifact/, repo-assessment/, recommend-workspace-pack/ functionality - Add comprehensive assessment scripts: ai-detector, copilot-analyzer, maturity-scoring, adr-generator - Implement result aggregation and skill routing orchestration - Create AI readiness rubric and Copilot integration checklist - Generate comprehensive AI adoption reports and strategic ADRs This transforms the plugin from general repository assessment to a specialized enterprise AI adoption audit toolkit with end-to-end workflow capabilities.
Review Summary by QodoEnterprise AI Adoption Audit Meta-Skill with Detection, Copilot Analysis, Maturity Scoring, and ADR Generation
WalkthroughsDescription• Implements comprehensive enterprise AI adoption audit skill with multi-component analysis system • Adds AI technology detection across multiple package managers (npm, pip, cargo, maven, gradle) with readiness scoring • Introduces GitHub Copilot integration maturity assessment with governance and adoption analysis • Develops AI maturity scoring framework across five dimensions (technology, Copilot, governance, organization, infrastructure) with six maturity levels (AI-Novice through AI-Native) • Creates 6-phase result aggregation pipeline for comprehensive analysis synthesis with insights, action items, and benchmarking • Generates Architecture Decision Records (ADRs) for AI adoption strategies with cost analysis, timeline planning, and ROI assessment • Implements sophisticated orchestration engine with maturity-based execution paths and targeted recommendations • Provides extensive documentation including AI readiness rubric, assessment checklists, ADR templates, and sample audit reports • Updates plugin configuration to reflect enterprise AI audit focus with simplified skill structure Diagramflowchart LR
A["Repository Codebase"] -->|"AI Detection"| B["AI Technology Scanner"]
A -->|"Copilot Analysis"| C["Copilot Integration Analyzer"]
B -->|"Technology Scores"| D["Maturity Scoring Engine"]
C -->|"Copilot Scores"| D
A -->|"Governance & Infrastructure"| D
D -->|"Component Scores"| E["Result Aggregator"]
E -->|"Synthesized Analysis"| F["ADR Generator"]
E -->|"Executive Summary & Roadmap"| G["Audit Report"]
F -->|"Strategic Decisions"| G
File Changes1. skills/enterprise-ai-adoption-audit/orchestration/result-aggregator.js
|
Code Review by Qodo
1. Unredacted report logged
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ebf26707d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+372
to
+376
| const enforcementLevel = | ||
| Object.values(governance).reduce( | ||
| (sum, g) => sum + g.enforcement.level, | ||
| 0 | ||
| ) / Object.keys(governance).length; |
There was a problem hiding this comment.
When no governance files are detected (a common case in new repos), Object.keys(governance).length is 0, so this division yields NaN; that NaN propagates into governance and overall maturity scores, and the final JSON serializes these as null, which breaks maturity classification and downstream routing/recommendations.
Useful? React with 👍 / 👎.
Comment on lines
+314
to
+316
| strategy: this.assessStrategicAlignment(), | ||
| structure: this.assessStructuralAlignment(), | ||
| culture: this.assessCulturalAlignment(), |
There was a problem hiding this comment.
These helper calls are not implemented on SkillRouter, so any run that reaches the scaling/advanced paths throws TypeError: ... is not a function and aborts the audit instead of producing a report; this makes non-foundation maturity paths non-executable.
Useful? React with 👍 / 👎.
Comment on lines
+1014
to
+1016
| return Math.round( | ||
| (avgCompleteness * 0.7 + Math.min(bestPracticesCount * 10, 30)) * 0.3 | ||
| ); |
There was a problem hiding this comment.
Applying * 0.3 to the already weighted configuration subtotal caps this component at 30 points, so configuration can never reach the 60/70 thresholds used elsewhere; that systematically under-scores Copilot maturity and skews recommendations/execution path selection even for well-configured repositories.
Useful? React with 👍 / 👎.
| const comparison = {}; | ||
|
|
||
| for (const [component, benchmark] of Object.entries(benchmarks)) { | ||
| const currentScore = scores[component]?.score || 0; |
There was a problem hiding this comment.
The benchmark loop assumes every score is an object with .score, but scores.overall is a number; this makes the overall benchmark current value fall back to 0, producing incorrect overall benchmark gaps/percentiles and misleading benchmarking output.
Useful? React with 👍 / 👎.
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+1213
to
+1215
| console.log('\n📊 Copilot Integration Analysis Report'); | ||
| console.log('======================================='); | ||
| console.log(JSON.stringify(report, null, 2)); |
There was a problem hiding this comment.
1. Unredacted report logged 📘 Rule violation ⛨ Security
New CLI scripts print full JSON reports to stdout, which can include sensitive configuration values from scanned repo files. This risks leaking secrets/PII into terminal output and CI/CD logs.
Agent Prompt
## Issue description
The CLI scripts print full JSON reports (including parsed configuration `content`) to stdout. If scanned config files include secrets (tokens/keys/passwords) or PII, they will be emitted into terminal output and CI logs.
## Issue Context
`CopilotAnalyzer` stores full parsed config objects in `this.findings.configurations` and later prints `JSON.stringify(report, null, 2)`.
## Fix Focus Areas
- skills/enterprise-ai-adoption-audit/scripts/copilot-analyzer.js[233-238]
- skills/enterprise-ai-adoption-audit/scripts/copilot-analyzer.js[1213-1215]
- skills/enterprise-ai-adoption-audit/scripts/maturity-scoring.js[1059-1064]
- skills/enterprise-ai-adoption-audit/scripts/ai-detector.js[621-627]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+1352
to
+1356
| const adrResult = await this.detectors.adr.generateADR(report, { | ||
| title: `AI Adoption Strategy - ${report.maturityAssessment.maturity.level}`, | ||
| owner: 'AI Strategy Team', | ||
| timeline: 'Q1 2024 - Q4 2024', | ||
| }); |
There was a problem hiding this comment.
2. Adr input type mismatch 🐞 Bug ✓ Correctness
SkillRouter.generateADR passes the comprehensive report object into ADRGenerator.generateADR, but ADRGenerator expects a maturity report with a top-level maturity field and immediately dereferences maturity.level, causing a runtime TypeError and failing the audit when ADR generation is enabled.
Agent Prompt
### Issue description
`SkillRouter.generateADR()` calls `ADRGenerator.generateADR()` with the *comprehensive* audit report object, but `ADRGenerator` expects a maturity report with a top-level `maturity` field. This causes `maturity` to be `undefined` and the code throws when it accesses `maturity.level`.
### Issue Context
- Comprehensive report shape: `{ maturityAssessment: <maturityReport>, ... }`
- ADR generator expects: `{ maturity: { level, ... }, componentScores, roadmap, ... }`
### Fix Focus Areas
- skills/enterprise-ai-adoption-audit/orchestration/skill-router.js[1348-1356]
- skills/enterprise-ai-adoption-audit/scripts/adr-generator.js[76-93]
### Suggested fix
In `SkillRouter.generateADR(report)`:
- Call `this.detectors.adr.generateADR(report.maturityAssessment, ...)` (or `this.results.maturity`) instead of `report`.
- Keep the ADR title derived from `report.maturityAssessment.maturity.level` as-is (or derive from the passed maturity report consistently).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+76
to
+93
| generateReplacements(maturityReport, options) { | ||
| const { maturity, componentScores, recommendations, roadmap } = | ||
| maturityReport; | ||
| const timestamp = new Date().toISOString().split('T')[0]; | ||
|
|
||
| return { | ||
| '\\[ADR-\\[NUMBER\\]\\]': options.adrNumber || '0001', | ||
| '\\[AI_IMPLEMENTATION_TITLE\\]': | ||
| options.title || this.generateTitle(maturity), | ||
| '\\[CURRENT_DATE\\]': timestamp, | ||
| '\\[HIGH\\/MEDIUM\\/LOW\\]': this.assessConfidence(maturityReport), | ||
|
|
||
| // Context section | ||
| '\\[AI_ADOPTION_ASSESSMENT_FINDINGS\\]': | ||
| this.generateAssessmentFindings(maturityReport), | ||
| '\\[MATURITY_LEVEL\\]': maturity.level, | ||
| '\\[GAP_ANALYSIS\\]': this.generateGapAnalysis(componentScores), | ||
| '\\[BUSINESS_IMPACT\\]': this.generateBusinessImpact(maturityReport), |
There was a problem hiding this comment.
3. Adr placeholders mismatch 🐞 Bug ✓ Correctness
ADRGenerator.generateReplacements uses placeholder tokens that do not match the ADR template (e.g., uses underscore tokens and a non-existent ADR number placeholder), so populateTemplate leaves key placeholders unreplaced and produces unusable ADR output.
Agent Prompt
### Issue description
`ADRGenerator.generateReplacements()` emits placeholder keys that don't exist in `ai-adoption-template.md`, so `populateTemplate()` fails to replace critical fields and the generated ADR retains raw placeholder text.
### Issue Context
Template uses placeholders like:
- `ADR-[NUMBER]`
- `[AI ADOPTION ASSESSMENT FINDINGS]` (with spaces)
Generator currently tries to replace:
- `\[ADR-\[NUMBER\]\]` (does not exist in template)
- `\[AI_ADOPTION_ASSESSMENT_FINDINGS\]` (underscores, does not exist in template)
### Fix Focus Areas
- skills/enterprise-ai-adoption-audit/scripts/adr-generator.js[76-93]
- skills/enterprise-ai-adoption-audit/assets/ai-adoption-template.md[1-10]
### Suggested fix
- Update replacement map keys to match the template literally (e.g. `\[NUMBER\]`, `\[AI ADOPTION ASSESSMENT FINDINGS\]`).
- Add a validation step after `populateTemplate()` to detect any remaining `\[[A-Z0-9 _/]+\]` placeholders and fail loudly or log which placeholders were not replaced.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
1
to
25
| { | ||
| "name": "copilot-repo-assessment", | ||
| "description": "Repository assessment and pack recommendation toolkit for development teams, with React and Java Spring support", | ||
| "version": "v2.0.1", | ||
| "name": "enterprise-ai-adoption-audit", | ||
| "description": "Enterprise AI adoption audit and strategy toolkit for comprehensive AI maturity assessment and Copilot integration analysis", | ||
| "version": "2.0.0", | ||
| "author": { | ||
| "name": "Matthew Van Dusen", | ||
| "email": "matt@ankhstudio.com" | ||
| }, | ||
| "license": "MIT", | ||
| "keywords": [ | ||
| "ai-adoption", | ||
| "enterprise-ai", | ||
| "copilot-integration", | ||
| "ai-maturity", | ||
| "ai-strategy", | ||
| "ai-assessment", | ||
| "ai-governance", | ||
| "architecture-decision-records", | ||
| "ai-transformation", | ||
| "copilot", | ||
| "repository-assessment", | ||
| "react", | ||
| "typescript", | ||
| "java", | ||
| "spring", | ||
| "spring-boot", | ||
| "pack-recommendations", | ||
| "frontend", | ||
| "backend", | ||
| "full-stack", | ||
| "github-copilot", | ||
| "ai-readiness", | ||
| "plugin" | ||
| ], | ||
| "homepage": "https://github.com/Ankh-Studio/copilot-eval-plugin", |
There was a problem hiding this comment.
4. Manifest hides existing skills 🐞 Bug ✓ Correctness
plugin.json now lists only skills/enterprise-ai-adoption-audit/ and renames the plugin, which makes previously documented skills like repo-assessment, recommend-workspace-pack, and artifact unavailable via the plugin manifest and inconsistent with the current repository documentation and contents.
Agent Prompt
### Issue description
`plugin.json` exposes only `skills/enterprise-ai-adoption-audit/`, but the repository README documents other skills and those skills are present. This changes plugin behavior and breaks documented usage.
### Issue Context
- README documents `repo-assessment`, `recommend-workspace-pack`, `artifact` as core skills.
- Those skill directories exist in `skills/`.
### Fix Focus Areas
- plugin.json[1-31]
- README.md[56-68]
### Suggested fix
Option A (most consistent with PR title 'Add ... skill'):
- Restore the prior plugin name/description if needed.
- Update `plugin.json.skills` to include the existing skill directories *plus* `skills/enterprise-ai-adoption-audit/`.
- Optionally update README to document the new skill.
Option B (if intentionally converting repo into a single-skill plugin):
- Update README and package.json metadata to match the new plugin identity and remove references to other skills.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add enterprise AI adoption audit skill with AI detection, Copilot analysis, maturity scoring, and ADR generation.