Moving files from git into a store

Generated files sometimes get committed and then outgrow git: rendered movies, reduced data products, images. If the directory holding them is (or becomes) a protonfs store, protonfs import moves them out of git and into the store without ever leaving them with no verified copy.

This walkthrough uses a research repository whose products/ directory holds committed outputs: *.reduced.fits files, *.mp4 movies and *.webp images. Large *.grids.fits files sit next to them, already gitignored.

1. Make the directory a store

A store may be any directory inside the repository; it does not have to be the top level:

cd products
protonfs setup        # remote_root e.g. /my-files/myrepo/products

Commit the .protonfs/ files setup creates (config.json, ignore, include and the control .gitignore) so other clones share the store’s settings. import refuses a store whose .gitignore has uncommitted changes, so commit before importing.

2. Preview

Quote the patterns so protonfs expands them. It matches them against the files git tracks, so **/*.fits picks up the committed *.reduced.fits files and never the gitignored *.grids.fits:

protonfs import --dry-run '**/*.reduced.fits' '**/*.mp4' '**/*.webp'

The dry run lists what push would upload, every line it would add to .gitignore, and the git commands it would run. If anything is refused (a file with uncommitted changes, a git-LFS pointer stub, a file the store’s .protonfs/ignore excludes), each refusal is listed and the exit code is 1; fix them and preview again.

3. Import

protonfs import '**/*.reduced.fits' '**/*.mp4' '**/*.webp'

import pushes the files, then reads each one and checks it against what push recorded, the blob committed in HEAD and a fresh listing of Drive. Only when every file passes does it touch git: it adds one line per file to a block in products/.gitignore, runs git rm --cached on them and stages the .gitignore. The summary ends with untracked=N. If any file fails, nothing in git changes and the failures are listed; running the command again uploads only what is missing.

4. Commit

git status            # the files show as deleted from the index, still on disk
git commit -m "chore: move 130 file(s) from git into protonfs"

5. Other clones

When another clone pulls this commit, git removes the imported files from its work tree. Restore them from Drive there:

git pull
cd products && protonfs pull

Until then that clone’s push counts them as local-deleted and leaves them on Drive, even with --deletions trash.

6. History

The files are still in git history, so the repository is no smaller yet. Removing them from history is a separate step with git filter-repo, which protonfs does not run. Do it only once the import commit is pushed and every clone that needs the files has pulled them from Drive.

7. Moving a file back into git

protonfs export is the reverse, for a file that should be versioned again:

cd products
protonfs export --dry-run run01/frame0001.reduced.fits
protonfs export run01/frame0001.reduced.fits
git commit -m "chore: move 1 file(s) from protonfs into git"

It downloads the file if it was offloaded, checks it is the version Drive holds (push it first if you changed it locally), removes its line from the import block in .gitignore, stages it, and adds a line for it to .protonfs/ignore so no machine syncs it any more. The Drive copy is kept; add --trash-remote to move it to the Drive trash once git has the file.

On other clones, run protonfs push before pulling the commit: git replaces their copy of the file with the committed one.