Rendering stops working for files with semantic tokens spanning more than one line

[PEZ]

Type: Bug

0. Have Calva v2.0.491 installed. (This is so that the issue can be triggered, It's not a bug in Calva.)
1. Open a .clj file with this text:

   #_[
   ]
   

   This will produce the semantic token [ 0, 0, 5, 10, 0 ], with character length 5 which spans the entire text.
2. Try to move the cursor, or scroll. The cursor doesn't move and scrolling will probably make the text in the editor disappear in chunks.

This doesn't happen in stable VS Code 1.98.2
Happens in 1.99.0-insider

Calva uses clojure-lsp as its language server, which produces tokens like these. Almost all Clojure files around has text like this and will not work with 1.99.0-insider.

VS Code version: Code - Insiders 1.99.0-insider (ea9b53f, 2025-03-18T05:04:34.769Z)
OS version: Darwin arm64 24.3.0
Modes:

System Info

Item	Value
CPUs	Apple M4 Max (16 x 2400)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	6, 5, 5
Memory (System)	64.00GB (0.17GB free)
Process Argv	--crash-reporter-id 6432db8b-e0cd-4ed4-9423-f6e4f791e7a8
Screen Reader	yes
VM	0%

Extensions (20)

Extension	Author (truncated)	Version
codesnap	adp	1.3.4
calva	bet	2.0.491
calva-spritz	bet	1.0.5
joyride	bet	0.0.44
paste-replaced	bet	1.1.14-master-dfbbac9-uk.0
magic-racket	evz	0.6.7
copilot	Git	1.287.1445
copilot-chat	Git	0.26.2025031802
gleam	gle	2.11.1
openai-chatgpt-adhoc	ope	0.0.1731981761
java	red	1.41.2025031808
intellicode-api-usage-examples	Vis	0.2.9
vscodeintellicode	Vis	1.3.2
vscode-gradle	vsc	3.16.2024111106
vscode-java-debug	vsc	0.58.2025022807
vscode-java-dependency	vsc	0.24.1
vscode-java-pack	vsc	0.29.2024091906
vscode-java-test	vsc	0.43.0
vscode-maven	vsc	0.44.2024072906
vscode-zig	zig	0.6.6

(1 theme extensions excluded)

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256197
pythonvspyt551cf:31249598
vscod805cf:30301675
py29gd2263:31024238
c4g48928:30535728
962ge761:30841072
dsvsc014:30777825
dsvsc015:30821418
h48ei257:31000450
pythontbext0:30879054
cppperfnew:30980852
pythonait:30973460
dwnewjupyter:31046869
nativerepl1:31134653
pythonrstrctxt:31093868
nativeloc1:31118317
e80f6927:31120813
iacca1:31150324
notype1:31143044
dwcopilot:31158714
5b1c1929:31184661
6074i472:31201624
dwoutputs:31242946
customenabled:31248079
hdaa2157:31222309
copilot_t_ci:31222730
f27dg485:31233432
g012b348:31231168
jda6j935:31233686
copilothoveron:31262361
pythoneinst12:31251391
bgtreat:31252123
fh1c7952:31247580
4gafe986:31261448
31787653:31262186
3e8i5726:31263299
pythonpulldiagcf:31258011
6gi0g917:31259950

[PEZ]

Related issue on Calva: BetterThanTomorrow/calva#2757

[ChrisKader]

To provide another test case:

A Lua file containing the below example will also cause the same error:

--[[
 ]]

[ChrisKader]

Appears to be related to: 082c5d8

[AndyAtTib]

Also happens in terraform files, where you have <<EOF descriptions, for example:

variable "break_rendering" {
  description = <<EOF
  breaks the rendering
  EOF
  type        = bool
  default     = true
}

This, however, with just a " description will work fine:

variable "renderings_fine" {
  description = "renders fine"
  type        = bool
  default     = true
}

Only happens in insiders, not GA (v1.98.2)

Insiders version

Version: 1.99.0-insider (user setup)
Commit: dc28988
Date: 2025-03-21T19:26:48.394Z
Electron: 34.3.2
ElectronBuildId: 11161073
Chromium: 132.0.6834.210
Node.js: 20.18.3
V8: 13.2.152.41-electron.0
OS: Windows_NT x64 10.0.26100

System information

CPUs	12th Gen Intel(R) Core(TM) i7-1255U (12 x 2611)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)
Memory (System)	15.70GB (2.15GB free)
Process Argv	--crash-reporter-id 1c8a503d-928e-47f5-b75e-87fe9dd5b4e5
Screen Reader	no
VM	0%

Enabled extensions

Extension	Author (truncated)	Version
remote-containers	ms-	0.407.0
remote-ssh	ms-	0.118.0
remote-ssh-edit	ms-	0.87.0
remote-wsl	ms-	0.88.5
vscode-remote-extensionpack	ms-	0.26.0
remote-explorer	ms-	0.4.3
remote-server	ms-	1.5.2
vscode-eslint	dba	3.0.10
copilot	Git	1.290.1455
copilot-chat	Git	0.26.2025032201
terraform	has	2.34.2025012311
git-graph	mhu	1.30.0
vscode-docker	ms-	1.29.4
signageos-vscode-sops	sig	0.9.2

A/B experimental info

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt551:31249597
vscod805cf:30301675
py29gd2263:31024238
c4g48928:30535728
962ge761:30841072
dsvsc014:30777825
dsvsc015:30821418
h48ei257:31000450
pythontbext0:30879054
cppperfnew:30980852
dwnewjupytercf:31046870
nativerepl1:31134653
pythonrstrctxt:31093868
nativeloc1:31118317
e80f6927:31120813
iacca1:31150324
dwcopilot:31158714
c3hdf307:31184662
6074i472:31201624
dwoutputs:31242946
customenabled:31248079
hdaa2157:31222309
copilot_t_ci:31222730
850i5325:31264362
g012b348:31231168
jda6j935:31233686
copilothoveron:31262361
pythoneinst12:31251391
bgtreat:31252123
fh1c7952:31247580
31787653:31262186
3e8i5726:31266632
pythonpulldiag:31258010
6gi0g917:31259950
996jf627:31264550

[fredrike]

I got the same for python & rust in insiders dc289883be5d37d5d2b2f7d30926aa42a3123437, I don't have anything dart-related installed.

I came to this message because interface freezes and drops pressed keys from time to time. I got the same set of messages in the UI: [error] [Window] Token length and text length do not match!: Error: Token length and text length do not match!

Originally posted by @peku33 in #244266

I have the same issue with a very long SQL file (2700+ lines) using the SnowFlake plugin.

Version: 1.99.0-insider (user setup)
Commit: 299b150
Date: 2025-03-24T05:03:53.092Z
Electron: 34.3.2
ElectronBuildId: 11161073
Chromium: 132.0.6834.210
Node.js: 20.18.3
V8: 13.2.152.41-electron.0
OS: Windows_NT x64 10.0.22631

[isidorn]

Sounds like a regression. Assigning to @aeschli and @hediet to double check

[hediet]

Fixed last week already with #244451!
Also verified that it happens without that fix, but not with that fix.

[PEZ]

Awesome news, @hediet! Does that mean that it works with latest Insiders already?

[isidorn]

@PEZ yes exactly! It is fixed in Insiders.

[PEZ]

Awesome. Then I can start using Insiders again! 🙏

[peku33]

It is still an issue on 88c05937becd43f3bc9457b65f2185e4cdaeae6e, same error message.

[PEZ]

I can confirm @peku33's observation. It's still an issue with latest Insiders. Though the repro is not as clear as it used to be. In fact I don't know the repro, just that some Calva users see the rendering of files stop to work with some content, and it does not happen with non-Insiders VS Code.