Efficient repo status (or, jj status on a large monorepo) (or, gitstatus equivalent for jj?)

[colemickens]

Hi!

I work with nixpkgs a lot, a huge monorepo of Linux/darwin/etc packages. It's large enough that it makes any inefficiencies in tooling quickly noticeable. For example, once one uses powerlevel10k or otherwise uses gitstatus, any other shell prompt becomes painfully slow to use. I'm actually trying my hand at adding support for gitstatus(d) to starship (github) and it occurred to me that I hadn't tried jj with nixpkgs yet.

So, I gave it a shot:

cole@porty ~/code
❯ git clone https://github.com/colemickens/nixpkgs nixpkgs-jj
Cloning into 'nixpkgs-jj'...
remote: Enumerating objects: 2801144, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 2801144 (delta 0), reused 0 (delta 0), pack-reused 2801143
Receiving objects: 100% (2801144/2801144), 2.00 GiB | 30.84 MiB/s, done.
Resolving deltas: 100% (1905795/1905795), done.
Updating files: 100% (29597/29597), done.

# oof 2GB

cole@porty ~/code/nixpkgs-jj cmpkgs
❯ time jj init --git-repo=.
Initialized repo in "/home/cole/code/nixpkgs-jj/."
jj init --git-repo=.  6.78s user 1.97s system 99% cpu 8.773 total

# ~7 seconds to init/import

cole@porty ~/code/nixpkgs-jj cmpkgs 9s
❯ time git status
On branch cmpkgs
Your branch is up to date with 'origin/cmpkgs'.

nothing to commit, working tree clean
git status  0.07s user 0.26s system 114% cpu 0.288 total

## 0.07s for git status

cole@porty ~/code/nixpkgs-jj cmpkgs
❯ time jj status
Parent commit: 55cf456b5123 delve: enable for aarch64-linux
Working copy : 4be1fa553088
The working copy is clean
jj status  1.72s user 2.74s system 99% cpu 4.463 total

## 1.72s for jj status

cole@porty ~/code/nixpkgs-jj cmpkgs
❯ time jj status
Parent commit: 55cf456b5123 delve: enable for aarch64-linux
Working copy : 4be1fa553088
The working copy is clean
jj status  1.72s user 2.77s system 99% cpu 4.494 total

cole@porty ~/code/nixpkgs-jj cmpkgs
❯ time jj status
Parent commit: 55cf456b5123 delve: enable for aarch64-linux
Working copy : 4be1fa553088
The working copy is clean
jj status  1.78s user 2.83s system 99% cpu 4.620 total

# and its repeatable

Further, I think I found a bug:

cole@porty ~/code/nixpkgs-jj cmpkgs
❯ jj status
^C

cole@porty ~/code/nixpkgs-jj cmpkgs
❯ time jj status
thread 'main' panicked at 'failed to create lock file /home/cole/code/nixpkgs-jj/./.jj/working_copy/working_copy.lock: File exists (os error 17)', lib/src/lock.rs:56:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
jj status  0.11s user 0.03s system 1% cpu 9.694 total

Observations:

• jj status (1.72 seconds) is quite a bit slower than git status (0.07 seconds)
• even beyond this slowness, I presumably won't be able to leverage gitstatus as it's seems pretty specifically tied to git's state

Questions:

• how much room there is for optimization here
• thoughts about something like gitstatus for jj?
• thoughts about some sort of "repo manager" that could declaratively manage jj enlistments, maybe background optimistically sync with a remote, monitor/allow querying their status, maybe? (I imagine this as an external tool that uses jj as a library.)

[arxanas]

• We previously discussed using Watchman for the large-repo problem: Support undoing staged changes arxanas/git-branchless#46 (reply in thread)
• Git supports Watchman indirectly via the fsmonitor hook (see e.g. https://github.com/jgavris/rs-git-fsmonitor)
• The recent git-maintenance command manages enlistments and can run tasks like pre-fetch objects: https://git-scm.com/docs/git-maintenance
• On my machine, git status on nixpkgs reliably takes >1s; I wonder if you have some extra optimizations on your end induced by gitstatus?

[colemickens]

I'm not sure if git status is somehow able to use gitstatus. I'm not really able to offer an explanation - I don't think it's a particularly fast NVME and it's running under hyper-v. gitstatus does provide benchmarks that show why it feels so much more responsive when used in a shell prompt, though: https://github.com/romkatv/gitstatus/blob/master/README.md#benchmarks

Funny enough, I just ran git status in my actual checkout (rather than a new fresh one to use with jj) and it's actually even faster at a basic naive git status.

Maybe gitstatus is causing a bunch of this to stay in some sort of cache, thereby speeding up normal git operations too? I'm just not sure.

[martinvonz]

As @arxanas said, my plan has been to use watchman for large repos. That would mean that the working copy commit would always be up to date. jj commands would then not look at the files in the working copy.

You can get similar behavior already. First run jj op log and copy the operation ID of the top operation. Then run jj --at-op=<operation ID> st. When loading the repo at a specific operation ID like that, it will not look at the working copy contents. So you can try timing that command and let us know if that's faster. Instead of st, you can use diff -r @ -s to get just the short file listing.

Also, everything is slower when you run in a working copy colocated with git because JJ does automatic imports and exports on every command in that case.

I'm on mobile right now but I'll give a longer answer in a few days.

[martinvonz]

By the way, I've meant to add a global --no-commit-working-copy so you don't have to look up the operation ID from jj op log. That should be easy, and I can hopefully do that pretty soon after I get back from vacation. Once that's there, you should be able to just do something like jj st in a loop to make sure there's always a recent working copy commit. Then your prompt can call jj --no-commit-working-copy st and not have to look at the working copy files.

We may also want a flag to prevent the automatic imports and exports when running in a working copy colocated with git, or maybe that should just be implied by --no-commit-working-copy.

[colemickens]

Thanks for the initial thoughts, seems like there's definitely some easy wins I can already realize. Excited for more to come. Cheers!

[martinvonz]

On my Mac, git status takes 833.9 ms according to hyperfine, while jj status takes 1.553 s. I haven't done any profiling yet, so I don't know why it's slower. I also timed jj status --at-op=<operation ID> and that took 271.3 ms. It was also very easy to add that --no-commit-working-copy flag I mentioned. I'll send a PR for you to review once I have access to my desktop computer where my SSH key is (yes, I'll make a copy of that key for next time) :)

By the way, if you do set up a process that automatically commits the working copy, then you'll also get backups of your working copy as part of that (a bit like Dura).

[colemickens]

By the way, if you do set up a process that automatically commits the working copy, then you'll also get backups of your working copy as part of that (a bit like Dura).

Oh yes, I'm already scheming -- besides the nicety of faster status updates, I'm doing some testing right now with a jj worktree on tmpfs, backed by a git repo on "slower' storage. And super granular history too. Very excited.