Moving files from git into a store¶
Generated files sometimes get committed and then outgrow git: rendered movies, reduced data products, images. If the directory holding them is (or becomes) a protonfs store, protonfs import moves them out of git and into the store without ever leaving them with no verified copy.
This walkthrough uses a research repository whose products/ directory holds
committed outputs: *.reduced.fits files, *.mp4 movies and *.webp images.
Large *.grids.fits files sit next to them, already gitignored.
1. Make the directory a store¶
A store may be any directory inside the repository; it does not have to be the top level:
cd products
protonfs setup # remote_root e.g. /my-files/myrepo/products
Commit the .protonfs/ files setup creates (config.json, ignore,
include and the control .gitignore) so other clones share the store’s settings.
import refuses a store whose .gitignore has uncommitted changes, so commit
before importing.
2. Preview¶
Quote the patterns so protonfs expands them. It matches them against the files git
tracks, so **/*.fits picks up the committed *.reduced.fits files and never the
gitignored *.grids.fits:
protonfs import --dry-run '**/*.reduced.fits' '**/*.mp4' '**/*.webp'
The dry run lists what push would upload, every line it would add to .gitignore,
and the git commands it would run. If anything is refused (a file with uncommitted
changes, a git-LFS pointer stub, a file the store’s .protonfs/ignore excludes), each
refusal is listed and the exit code is 1; fix them and preview again.
3. Import¶
protonfs import '**/*.reduced.fits' '**/*.mp4' '**/*.webp'
import pushes the files, then reads each one and checks it against what push
recorded, the blob committed in HEAD and a fresh listing of Drive. Only when every
file passes does it touch git: it adds one line per file to a block in
products/.gitignore, runs git rm --cached on them and stages the
.gitignore. The summary ends with untracked=N. If any file fails, nothing in
git changes and the failures are listed; running the command again uploads only what
is missing.
4. Commit¶
git status # the files show as deleted from the index, still on disk
git commit -m "chore: move 130 file(s) from git into protonfs"
5. Other clones¶
When another clone pulls this commit, git removes the imported files from its work tree. Restore them from Drive there:
git pull
cd products && protonfs pull
Until then that clone’s push counts them as local-deleted and leaves them on
Drive, even with --deletions trash.
6. History¶
The files are still in git history, so the repository is no smaller yet. Removing them
from history is a separate step with git filter-repo, which protonfs does not run.
Do it only once the import commit is pushed and every clone that needs the files has
pulled them from Drive.
7. Moving a file back into git¶
protonfs export is the reverse, for a file that should be versioned again:
cd products
protonfs export --dry-run run01/frame0001.reduced.fits
protonfs export run01/frame0001.reduced.fits
git commit -m "chore: move 1 file(s) from protonfs into git"
It downloads the file if it was offloaded, checks it is the version Drive holds (push
it first if you changed it locally), removes its line from the import block in
.gitignore, stages it, and adds a line for it to .protonfs/ignore so no machine
syncs it any more. The Drive copy is kept; add --trash-remote to move it to the
Drive trash once git has the file.
On other clones, run protonfs push before pulling the commit: git replaces their copy of the file with the committed one.