diff --git a/CHANGELOG.md b/CHANGELOG.md index a014f7333..888cd5b29 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,12 +14,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Whole-upstream tutorial** - Added a second tutorial showing how to vendor an entire upstream repository into a local directory with `remote:.` or `remote:/`. - **Claude sandbox starter** - Added `sbx-kits/claude` so repo-local sandbox examples now cover both OpenCode and Claude. - **Private fork migration tutorial** - Added a guarded tutorial for migrating an existing private fork or derivative repo to a repo-root `git-cross` patch with explicit backup and override review steps. +- **Repo split tutorial** - Added `docs/tutorials/split-repo-into-upstream-and-private.md`, a concise walkthrough for splitting one original repo into a clean secret-free upstream plus a private `git-cross`-managed fork. Uses a single shared rule: one `.crossignore` and one `git ls-files --ignored --exclude-from=.crossignore` selection piped to `git rm` (remove from upstream) or `rsync` (back up and keep), so the upstream and the fork can never disagree about what is private. The selection command is documented as a repeatable preview (refine `.crossignore` in a loop until it lists exactly the private files) and clarified as non-destructive — the seed removal only strips the throwaway upstream copy while private files stay in the fork as git-cross overlays. The upstream-seed removal deletes the tracked-only selection (`--cached`, without `--others`) with `rm -rf` followed by `git add -A`, instead of `git rm`: `git rm` aborts the whole batch on untracked/gitignored pathspecs (`node_modules/`, `.envrc.local`) and on submodule/embedded-repo gitlinks (`could not lookup name for submodule ...`), whereas working-tree deletion + restage handles files, directories, and gitlinks uniformly. Documented that `rm` has no `--dry-run` (preview via `xargs … echo` or confirm via `rm -rI`/`-ri` with `xargs -o`), and that `test/` matches a `test` directory at any depth while `*/test/*` does not. The backup keeps `--others` so untracked private files are still copied. Each step is labelled with the repo it runs in. ### Changed - **`.crossignore` entry syntax** - Override review behavior now uses plain non-comment `.crossignore` lines. Supported forms include basename entries such as `.env` anywhere under the patch, basename globs such as `*.env`, and directory entries such as `config/`. Full `.gitignore` semantics are still not supported. - **README workflow guidance** - Documented the current `.crossignore` behavior as a review-oriented workflow for local overlay files layered on top of upstream-managed content. ### Fixed +- **Migration tutorial backup command** - Removed the broken recommendation to call the internal Just `_crossignore_overrides` recipe during private-fork migration. The backup step now expands `.crossignore` (which is a gitignore-style *pattern* prescription, not a literal file list) against the real working tree with `git ls-files --ignored --exclude-from=.crossignore` piped into `rsync`, so globs and not-yet-present paths no longer cause `rsync --files-from` "No such file or directory" failures. Documented the safer "clean upstream seed repo first, then patch into the private repo" workflow. - **Rust override diff build failure** - Renamed a Rust loop variable that used the reserved keyword `override`, restoring Rust CLI builds and regression execution on current toolchains. - **Just override warning output** - Removed a shell-breaking semicolon from the override review warning so `just cross diff` prints the manual review commands correctly. - **P0: Sparse checkout broken on newer Git versions** - `git sparse-checkout set ` in `--no-cone` mode no longer reliably checks out directories without trailing `/`. Fixed across all three implementations (Go, Rust, Justfile.cross) by appending `/` to sparse-checkout patterns and using `git read-tree -mu HEAD` instead of bare `git checkout` (which can no-op after `--no-checkout`). diff --git a/README.md b/README.md index 352082058..341580fd9 100644 --- a/README.md +++ b/README.md @@ -173,6 +173,7 @@ This lets your main repo keep its local opinionated result while the upstream co - [Tutorial: Local Overlays And Upstream Contribution](docs/tutorials/local-overlays-and-upstream.md) - [Tutorial: Whole Upstream Into A Local Directory](docs/tutorials/whole-upstream-into-local-dir.md) - [Tutorial: Migrate A Private Fork To git-cross](docs/tutorials/migrating-private-fork-to-git-cross.md) +- [Tutorial: Split One Repo Into A Clean Upstream And A Private Overlay](docs/tutorials/split-repo-into-upstream-and-private.md) - [Sandbox Kits](sbx-kits/README.md) ## Commands diff --git a/TODO.md b/TODO.md index b2d0c2273..2c93cd34c 100644 --- a/TODO.md +++ b/TODO.md @@ -22,6 +22,8 @@ ## Completed Tasks +- [x] Fix the private-fork migration tutorial so it no longer points users at the internal `_crossignore_overrides` Just recipe, and document the cleaner "seed a public upstream, then track it from the private repo" migration path. +- [x] Add a standalone tutorial (`docs/tutorials/split-repo-into-upstream-and-private.md`) for splitting one mixed private repo into a clean upstream plus a private git-cross-managed overlay, cross-linked from README and the migration tutorial. - [x] Refactor remaining Rust commands to use `duct` for better error visibility. - [x] Complete `push` command verification in native implementations. - [x] Integrate integration tests (001-009) into a unified test runner. diff --git a/docs/tutorials/migrating-private-fork-to-git-cross.md b/docs/tutorials/migrating-private-fork-to-git-cross.md index 39f8aaa92..e1074ea00 100644 --- a/docs/tutorials/migrating-private-fork-to-git-cross.md +++ b/docs/tutorials/migrating-private-fork-to-git-cross.md @@ -39,6 +39,53 @@ Assume: - your private repo also contains local-only files such as `.env`, `.env.local`, `docker-compose.override.yml`, or private config directories - you want the repo to stay at repo root, not move upstream content into `vendor/...` +## Before Step 1: Decide Where The Upstream Comes From + +There are two valid starting points. + +### Case A: A Clean Upstream Repo Already Exists + +If you already have a real upstream repository that contains only the shareable code, continue with Step 1. + +### Case B: Your Current Private Repo Is The Only Copy + +If the current private repo is the only place where the code exists and it mixes: + +- upstream-worthy project files +- private files such as `.env` +- machine-local or company-local overrides + +create a clean upstream seed repository first, then come back and migrate the private repo with `git-cross`. + +This split has its own focused walkthrough: +`docs/tutorials/split-repo-into-upstream-and-private.md`. + +The core idea there is a single shared rule: one `.crossignore` lists the private +patterns, and `git ls-files` selects the matching files for both jobs — + +```bash +# back up / keep (rsync): include untracked too +git ls-files -z --cached --others --ignored --exclude-from=.crossignore + +# remove from upstream seed (delete from copy): tracked only +git ls-files -z --cached --ignored --exclude-from=.crossignore +``` + +- the tracked list, deleted with `rm -rf` then `git add -A`, strips those files from the new upstream seed +- the full list piped to `rsync` in your private repo *backs up and keeps* them + +Use `rm` + `git add -A` rather than `git rm` for the seed: `git rm` aborts the +whole batch on untracked pathspecs or on a submodule it cannot name. Deleting from +the working tree and restaging handles files, directories, and submodule gitlinks +uniformly. + +Do the split there, then return here at Step 1 with a clean upstream in hand. + +Important note: + +- if secrets or private files were ever committed into Git history and you plan to publish that history, do a real history rewrite first +- the split tutorial only covers the working-tree split, not secret-history cleanup + ## Step 1: Make A Safety Snapshot Before changing anything, create a backup branch or tag. @@ -132,6 +179,8 @@ If you want upstream contribution later, the cleanest pattern is to register a w If you only want to mirror the original upstream first, register the original upstream now and switch to a fork later. +If you created a clean upstream seed repository in the earlier split step, register that repository here. + Example: ```bash diff --git a/docs/tutorials/split-repo-into-upstream-and-private.md b/docs/tutorials/split-repo-into-upstream-and-private.md new file mode 100644 index 000000000..783d9023f --- /dev/null +++ b/docs/tutorials/split-repo-into-upstream-and-private.md @@ -0,0 +1,187 @@ +# Tutorial: Split One Repo Into A Clean Upstream And A Private Overlay + +Start from one **original repo** that mixes shareable code with secrets, and end +with two repos: + +| Role | Path | What it holds | +|------|------|---------------| +| **original repo** | `project/` | where you start: everything mixed together | +| **upstream repo** | `../project-upstream/` | new, clean, publishable, no secrets | +| **private fork** | `project/` (same dir as original) | the original repo, now tracking upstream via git-cross + local overlays | + +The original repo *becomes* the private fork in place. The upstream repo is a new +clean copy you publish. Each step below marks which repo you run it in. + +Use this when the original repo is the only copy of the code. If a clean upstream +already exists, use `docs/tutorials/migrating-private-fork-to-git-cross.md`. + +> Safety: this only splits the working tree, not Git history. If secrets were +> ever committed and you will publish the full history, rewrite it first with +> `git filter-repo` or BFG. Seeding a fresh snapshot repo (below) avoids that — +> the new upstream starts from one clean commit. + +## 1. Snapshot the original repo · *run in: original repo* + +```bash +git checkout -b backup/pre-split && git push origin backup/pre-split +``` + +## 2. Write `.crossignore` · *run in: original repo* + +This one file is the single source of truth. It drives what the upstream seed +drops, what you back up, and the overlay markers — so the sets cannot drift. + +```bash +cat > .crossignore <<'EOF' +.env* +*.env +docker-compose.override.yml +config/ +test/ +EOF +``` + +Patterns are gitignore-style. Match what you mean: + +- `.env` is only the literal file; `.env*` also catches `.env.local` / `.env.secrets` +- `config/` is the whole `config` tree +- `test/` matches a directory named `test` **at any depth** (recursive). Do not + use `*/test/*` — `*` never crosses `/`, so that only matches `test` one level + deep and only its immediate children + +### The shared rule + +Both jobs select files from the same `.crossignore` with `git ls-files`. The only +difference is one flag, because the two destinations handle different file sets: + +```bash +# back up (rsync): include untracked files too — rsync copies whatever exists +git ls-files -z --cached --others --ignored --exclude-from=.crossignore + +# remove from upstream (delete from copy): tracked files only +git ls-files -z --cached --ignored --exclude-from=.crossignore +``` + +The upstream side drops `--others`: untracked or gitignored files +(`node_modules/`, `.envrc.local`, …) are never part of the upstream commit +anyway, so they need no removal. + +The upstream side also deletes with `rm`, **not** `git rm`. `git rm` aborts the +whole command on anything it cannot resolve — an untracked pathspec +(`pathspec ... did not match any files`) or a submodule / embedded repo +(`could not lookup name for submodule ...`). Deleting from the working tree and +re-running `git add -A` sidesteps both and treats files, directories, and +submodule gitlinks uniformly. + +**It is a preview, so refine it in a loop.** Run the command on its own first, +read the output, edit `.crossignore`, run again — repeat until it lists exactly +the files you want to keep private. Only then pipe it anywhere. + +**Nothing is deleted from your repo.** The deletion in step 3 only strips the +*throwaway upstream copy*; your originals never leave the private fork. The split +just sorts files by role: private files stay in the fork, shareable files go +upstream. + +## 3. Create the upstream repo · *original repo → upstream repo* + +Copy the tree out, then delete the tracked files matched by `.crossignore` so +exactly the private files are dropped. + +```bash +# from the original repo +mkdir -p ../project-upstream +rsync -av --exclude .git ./ ../project-upstream/ + +# now work in the upstream repo +cd ../project-upstream +git init -b main +git add -A + +# preview: refine .crossignore until this lists exactly the private files +git ls-files --cached --ignored --exclude-from=.crossignore + +# delete the matched paths (files, dirs, and submodule gitlinks), then restage +git ls-files -z --cached --ignored --exclude-from=.crossignore | xargs -0 -r rm -rf -- +git add -A + +rm -f .crossignore # overlay marker does not belong upstream +git add -A +git commit -m "Initial upstream seed" +git remote add origin git@github.com:your-org/project.git +git push -u origin main + +cd - # back to the original repo +``` + +`rm` has no `--dry-run`. To dry-run the destructive line, print the command +instead of running it, or ask for confirmation: + +```bash +# print what would run, run nothing +git ls-files -z --cached --ignored --exclude-from=.crossignore | xargs -0 -r echo rm -rf -- + +# delete with a confirmation prompt (Linux rm -rI = one prompt; macOS rm -ri = per file) +# xargs -o reopens the terminal so the prompt can read your keypress; drop -f or it skips the prompt +git ls-files -z --cached --ignored --exclude-from=.crossignore | xargs -0 -r -o rm -ri -- +``` + +Verify no secrets survived before pushing: + +```bash +# inside ../project-upstream, before pushing +git ls-files | grep -Ei 'secret|\.env' || echo "no obvious secrets tracked" +``` + +If a private file is still tracked, your pattern missed it — fix `.crossignore` +(e.g. `*.env*` to also catch `.env.secrets`) and re-run the preview + delete. + +## 4. Back up the private files · *run in: original repo* + +Same `.crossignore`, this time with `--others` added so untracked private files +are backed up too, piped to `rsync` (only existing files are copied, so globs and +missing paths cause no error): + +```bash +mkdir -p ../private-overrides-backup +git ls-files -z --cached --others --ignored --exclude-from=.crossignore \ + | rsync -av --from0 --files-from=- ./ ../private-overrides-backup/ +``` + +## 5. Turn the original repo into the private fork · *run in: original repo* + +This is the transition: the original repo now tracks the upstream repo via +git-cross. + +```bash +git cross use upstream git@github.com:your-org/project.git +git cross patch upstream:. . # `upstream:/` is equivalent +``` + +Register a writable fork instead if you plan to contribute back. + +## 6. Restore overlays and review · *run in: private fork* + +```bash +rsync -av ../private-overrides-backup/ ./ 2>/dev/null || true +cat .crossignore # confirm it survived the patch +git cross status # shows `Override` for the root patch +git cross diff . +``` + +## 7. Commit and day-2 workflow · *run in: private fork* + +```bash +git add Crossfile .crossignore . && git commit -m "Track upstream via git-cross" +git push origin main +``` + +- pull upstream: `git cross sync .` +- review: `git cross diff .` / `git cross status` +- contribute upstream: `git cross push . --message "..."` after confirming no + private files are included + +## Related + +- `docs/tutorials/migrating-private-fork-to-git-cross.md` — clean upstream already exists +- `docs/tutorials/whole-upstream-into-local-dir.md` — vendor into `vendor/...` instead of root +- `docs/tutorials/local-overlays-and-upstream.md` — overlay a single subdirectory