Is the first problem still relevant with the new JSON parser?

[arthsmn]

After Emacs 30 introduced a new JSON parser I was wondering if the first problem is still relevant, and how faster the parsing might be.

[Ergus]

The new parser is actually very fast and it does not rely in external libraries. So, a new benchmark set could be interesting now in Emacs 30.

BTW: I don't totally understand the current benchmarks numbers I see in the Actions. Some of them seems actually confusing. How can I see the speedup difference with and without booster?

[jdtsmith]

The benchmarks provided are for the artificial case of reading from JSON vs. reading from the equivalent (translated) bytecode. That's related to but not identical to the actual speedup you might get.

You can actually run the booster without doing JSON conversion; see --disable-bytecode. If you use eglot-booster it has the option eglot-booster-io-only to enable this. If you do so maybe report your findings.

[jdtsmith]

I ran the included test with an Emacs 30 (emacs-plus on Mac M2) compiled with native-comp and the new native JS parser. Results:

% cargo test test_bytecode -- --nocapture

Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
 Running unittests src/lib.rs (/Users/jdsmith/code/rust/emacs-lsp-booster/target/debug/deps/emacs_lsp_booster-b2cd102895f6f2a3)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 0.00s

 Running unittests src/main.rs (/Users/jdsmith/code/rust/emacs-lsp-booster/target/debug/deps/emacs_lsp_booster-5ce1ba676b51f07c)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

 Running tests/app_test.rs (/Users/jdsmith/code/rust/emacs-lsp-booster/target/debug/deps/app_test-34534fccb76674ed)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

 Running tests/bytecode_test.rs (/Users/jdsmith/code/rust/emacs-lsp-booster/target/debug/deps/bytecode_test-b59d605fc84b1c71)

running 1 test
Json: 55 bytes, Bytecode: 80 bytes, ratio=1.4545454545454546
Object-type: plist
Benchmark json-parse-string 100 times: (0.002382 0 0.0)
Benchmark read & eval bytecode 100 times: (8.1e-05 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (7.4e-05 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (3.8e-05 0 0.0)
PASS!
Testing completion.json (~100KB), object type = Plist
Json: 61979 bytes, Bytecode: 45822 bytes, ratio=0.7393149292502299
Object-type: plist
Benchmark json-parse-string 100 times: (0.016266 0 0.0)
Benchmark read & eval bytecode 100 times: (0.028003 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.025971 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.041828 0 0.0)
PASS!
Testing completion2.json (~100KB), object type = Plist
Json: 63176 bytes, Bytecode: 44153 bytes, ratio=0.6988888185386856
Object-type: plist
Benchmark json-parse-string 100 times: (0.016713 0 0.0)
Benchmark read & eval bytecode 100 times: (0.027721 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.025338 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.042998 0 0.0)
PASS!
Testing completion3.json (~4KB), object type = Plist
Json: 2993 bytes, Bytecode: 2544 bytes, ratio=0.8499832943534915
Object-type: plist
Benchmark json-parse-string 100 times: (0.003003 0 0.0)
Benchmark read & eval bytecode 100 times: (0.001399 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.001303 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.001889 0 0.0)
PASS!
Testing publishDiagnostics.json (~12KB), object type = Plist
Json: 8288 bytes, Bytecode: 4226 bytes, ratio=0.5098938223938224
Object-type: plist
Benchmark json-parse-string 100 times: (0.004091 0 0.0)
Benchmark read & eval bytecode 100 times: (0.002522 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.002291 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.005612 0 0.0)
PASS!
Testing publishDiagnostics2.json (~12KB), object type = Plist
Json: 7825 bytes, Bytecode: 3858 bytes, ratio=0.49303514376996804
Object-type: plist
Benchmark json-parse-string 100 times: (0.003922 0 0.0)
Benchmark read & eval bytecode 100 times: (0.002328 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.002087 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.005279 0 0.0)
PASS!
Testing completion.json (~100KB), object type = Alist
Json: 61979 bytes, Bytecode: 47997 bytes, ratio=0.7744074605914907
Object-type: alist
Benchmark json-parse-string 100 times: (0.016326 0 0.0)
Benchmark read & eval bytecode 100 times: (0.030447 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.028194999999999998 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.047276 0 0.0)
PASS!
Testing completion2.json (~100KB), object type = Alist
Json: 63176 bytes, Bytecode: 46491 bytes, ratio=0.7358965429910093
Object-type: alist
Benchmark json-parse-string 100 times: (0.016529 0 0.0)
Benchmark read & eval bytecode 100 times: (0.029705 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.027567 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.048875999999999996 0 0.0)
PASS!
Testing completion3.json (~4KB), object type = Alist
Json: 2993 bytes, Bytecode: 2618 bytes, ratio=0.8747076511861009
Object-type: alist
Benchmark json-parse-string 100 times: (0.002946 0 0.0)
Benchmark read & eval bytecode 100 times: (0.001473 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.001381 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.001949 0 0.0)
PASS!
Testing publishDiagnostics.json (~12KB), object type = Alist
Json: 8288 bytes, Bytecode: 4505 bytes, ratio=0.5435569498069498
Object-type: alist
Benchmark json-parse-string 100 times: (0.004084 0 0.0)
Benchmark read & eval bytecode 100 times: (0.002775 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.002547 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.006064 0 0.0)
PASS!
Testing publishDiagnostics2.json (~12KB), object type = Alist
Json: 7825 bytes, Bytecode: 4113 bytes, ratio=0.5256230031948882
Object-type: alist
Benchmark json-parse-string 100 times: (0.00395 0 0.0)
Benchmark read & eval bytecode 100 times: (0.002562 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.002285 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.005166 0 0.0)
PASS!
Testing completion.json (~100KB), object type = Hashtable
Json: 61979 bytes, Bytecode: 85888 bytes, ratio=1.385759692799174
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.022452 0 0.0)
Benchmark read & eval bytecode 100 times: (0.062522 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.050462 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.074171 0 0.0)
PASS!
Testing completion2.json (~100KB), object type = Hashtable
Json: 63176 bytes, Bytecode: 87474 bytes, ratio=1.3846080790173483
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.023452999999999998 0 0.0)
Benchmark read & eval bytecode 100 times: (0.064348 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.051983 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.078378 0 0.0)
PASS!
Testing completion3.json (~4KB), object type = Hashtable
Json: 2993 bytes, Bytecode: 4258 bytes, ratio=1.422652856665553
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.003188 0 0.0)
Benchmark read & eval bytecode 100 times: (0.002742 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.002265 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.00302 0 0.0)
PASS!
Testing publishDiagnostics.json (~12KB), object type = Hashtable
Json: 8288 bytes, Bytecode: 9930 bytes, ratio=1.1981177606177607
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.004919 0 0.0)
Benchmark read & eval bytecode 100 times: (0.007054 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.005328 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.009618 0 0.0)
PASS!
Testing publishDiagnostics2.json (~12KB), object type = Hashtable
Json: 7825 bytes, Bytecode: 9153 bytes, ratio=1.1697124600638977
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.004869 0 0.0)
Benchmark read & eval bytecode 100 times: (0.006545 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.004946 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.008937 0 0.0)
PASS!
Testing huge array (100000 elements)
Json: 788891 bytes, Bytecode: 2162672 bytes, ratio=2.7414078751056863
Object-type: plist
Benchmark json-parse-string 100 times: (0.267222 0 0.0)
Benchmark read & eval bytecode 100 times: (1.284329 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (1.2987460000000002 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.559955 0 0.0)
PASS!
Testing huge map (100000 elements)
Json: 1477781 bytes, Bytecode: 5127002 bytes, ratio=3.4693922847837397
Object-type: plist
Benchmark json-parse-string 100 times: (1.676578 0 0.0)
Benchmark read & eval bytecode 100 times: (4.411016 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (4.346246 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (2.53625 0 0.0)
PASS!
test test_bytecode ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 21.19s

The plist-flavored results are most relevant for eglot. In those tests, bytecode still holds the edge for small messages (4 and 12kB), but at larger message sizes (100kB), native JSON parsing dominates by >2x on my machine. I interpret this to mean that disabling bytecode on Emacs 30 is probably a good idea if you deal with large LSP messages.

Give the test a try and report your results.

[arthsmn]

Here are my results in an AMD Ryzen 7 5700U with the nixpkgs' emacs-pgtk (30.1):

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running unittests src/lib.rs (target/debug/deps/emacs_lsp_booster-19f825df3b960817)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/emacs_lsp_booster-86400c2fb4d67bc5)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

     Running tests/app_test.rs (target/debug/deps/app_test-24e9622f6c39627b)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

     Running tests/bytecode_test.rs (target/debug/deps/bytecode_test-b129af4113d57fcb)

running 1 test
Json: 55 bytes, Bytecode: 80 bytes, ratio=1.4545454545454546
Object-type: plist
Benchmark json-parse-string 100 times: (0.004081304 0 0.0)
Benchmark read & eval bytecode 100 times: (0.000107241 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.000140502 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (4.4603e-05 0 0.0)
PASS!
Testing completion.json (~100KB), object type = Plist
Json: 61979 bytes, Bytecode: 45822 bytes, ratio=0.7393149292502299
Object-type: plist
Benchmark json-parse-string 100 times: (0.03752927599999999 0 0.0)
Benchmark read & eval bytecode 100 times: (0.042237219 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.034483812 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.06773443600000001 0 0.0)
PASS!
Testing completion2.json (~100KB), object type = Plist
Json: 63176 bytes, Bytecode: 44153 bytes, ratio=0.6988888185386856
Object-type: plist
Benchmark json-parse-string 100 times: (0.038205516 0 0.0)
Benchmark read & eval bytecode 100 times: (0.041662127 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.03330789 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.069220555 0 0.0)
PASS!
Testing completion3.json (~4KB), object type = Plist
Json: 2993 bytes, Bytecode: 2544 bytes, ratio=0.8499832943534915
Object-type: plist
Benchmark json-parse-string 100 times: (0.005571892 0 0.0)
Benchmark read & eval bytecode 100 times: (0.002245624 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.001813318 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.0026327150000000003 0 0.0)
PASS!
Testing publishDiagnostics.json (~12KB), object type = Plist
Json: 8288 bytes, Bytecode: 4226 bytes, ratio=0.5098938223938224
Object-type: plist
Benchmark json-parse-string 100 times: (0.007991330999999999 0 0.0)
Benchmark read & eval bytecode 100 times: (0.0044715499999999995 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.003733337 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.008299214000000001 0 0.0)
PASS!
Testing publishDiagnostics2.json (~12KB), object type = Plist
Json: 7825 bytes, Bytecode: 3858 bytes, ratio=0.49303514376996804
Object-type: plist
Benchmark json-parse-string 100 times: (0.007611291 0 0.0)
Benchmark read & eval bytecode 100 times: (0.004108675 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.002975924 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.0068952580000000005 0 0.0)
PASS!
Testing completion.json (~100KB), object type = Alist
Json: 61979 bytes, Bytecode: 47997 bytes, ratio=0.7744074605914907
Object-type: alist
Benchmark json-parse-string 100 times: (0.037481056000000006 0 0.0)
Benchmark read & eval bytecode 100 times: (0.044815352999999995 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.036762759 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.074234649 0 0.0)
PASS!
Testing completion2.json (~100KB), object type = Alist
Json: 63176 bytes, Bytecode: 46491 bytes, ratio=0.7358965429910093
Object-type: alist
Benchmark json-parse-string 100 times: (0.038811194 0 0.0)
Benchmark read & eval bytecode 100 times: (0.044606021 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.036133345000000004 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.076973048 0 0.0)
PASS!
Testing completion3.json (~4KB), object type = Alist
Json: 2993 bytes, Bytecode: 2618 bytes, ratio=0.8747076511861009
Object-type: alist
Benchmark json-parse-string 100 times: (0.005332196 0 0.0)
Benchmark read & eval bytecode 100 times: (0.002383852 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.0019463269999999999 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.002913719 0 0.0)
PASS!
Testing publishDiagnostics.json (~12KB), object type = Alist
Json: 8288 bytes, Bytecode: 4505 bytes, ratio=0.5435569498069498
Object-type: alist
Benchmark json-parse-string 100 times: (0.008065628 0 0.0)
Benchmark read & eval bytecode 100 times: (0.005505227 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.003534005 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.008163821 0 0.0)
PASS!
Testing publishDiagnostics2.json (~12KB), object type = Alist
Json: 7825 bytes, Bytecode: 4113 bytes, ratio=0.5256230031948882
Object-type: alist
Benchmark json-parse-string 100 times: (0.007809912 0 0.0)
Benchmark read & eval bytecode 100 times: (0.00437946 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.003140922 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.0075823670000000004 0 0.0)
PASS!
Testing completion.json (~100KB), object type = Hashtable
Json: 61979 bytes, Bytecode: 85888 bytes, ratio=1.385759692799174
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.074933217 0 0.0)
Benchmark read & eval bytecode 100 times: (0.10213267499999999 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.062896189 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.151177178 0 0.0)
PASS!
Testing completion2.json (~100KB), object type = Hashtable
Json: 63176 bytes, Bytecode: 87474 bytes, ratio=1.3846080790173483
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.079417382 0 0.0)
Benchmark read & eval bytecode 100 times: (0.10633230099999999 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.064494457 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.163300564 0 0.0)
PASS!
Testing completion3.json (~4KB), object type = Hashtable
Json: 2993 bytes, Bytecode: 4258 bytes, ratio=1.422652856665553
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.007207689 0 0.0)
Benchmark read & eval bytecode 100 times: (0.0049860799999999995 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.0033840660000000002 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.006282216 0 0.0)
PASS!
Testing publishDiagnostics.json (~12KB), object type = Hashtable
Json: 8288 bytes, Bytecode: 9930 bytes, ratio=1.1981177606177607
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.014282411 0 0.0)
Benchmark read & eval bytecode 100 times: (0.014648153 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.010076824 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.019524868 0 0.0)
PASS!
Testing publishDiagnostics2.json (~12KB), object type = Hashtable
Json: 7825 bytes, Bytecode: 9153 bytes, ratio=1.1697124600638977
Object-type: hash-table
Benchmark json-parse-string 100 times: (0.013139131 0 0.0)
Benchmark read & eval bytecode 100 times: (0.011916352 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (0.006419411 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (0.018160807 0 0.0)
PASS!
Testing huge array (100000 elements)
Json: 788891 bytes, Bytecode: 2162672 bytes, ratio=2.7414078751056863
Object-type: plist
Benchmark json-parse-string 100 times: (0.928063356 0 0.0)
Benchmark read & eval bytecode 100 times: (2.2695313059999997 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (2.1667027340000002 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (1.136876461 0 0.0)
PASS!
Testing huge map (100000 elements)
Json: 1477781 bytes, Bytecode: 5127002 bytes, ratio=3.4693922847837397
Object-type: plist
Benchmark json-parse-string 100 times: (8.204736225 0 0.0)
Benchmark read & eval bytecode 100 times: (12.454867299 0 0.0)
(Just for reference) Benchmark read bytecode only (no eval) 100 times: (11.776453621 0 0.0)
(Just for reference) Benchmark read lisp data directly 100 times: (9.664094318 0 0.0)
PASS!
test test_bytecode ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 57.11s

[jdtsmith]

Your results look similar. My guess is that JSON parsing is moderately faster than bytecode parsing once the data are in memory. This makes sense, since ELISP is a more general and complicated syntax. But small messages are dominated by their read time, which has fixed offsets associated with it. Once the parse time exceeds the read time, JSON wins. Since most people's performance problems likely stem from large messages I think disabling bytecode is the right move.

I just added some stats tracking to eglot-booster and after a bit of playing, my server's message sizes look like this:

Plenty of 100kB JSON messages, but median size is under 1kB. Probably worth disabling bytecode translation though, as the large messages "cost" more in terms of latency.