JavaScript heap out of memory on PNPM project

[tsteenbe]

Describe the bug

Getting JavaScript heap out of memory when running analyzer over https://github.com/nl-design-system/utrecht

To Reproduce

Steps to reproduce the behavior:

1. git clone https://github.com/nl-design-system/utrecht.git
2. mkdir utrecht-ort
3. analyze -i ../utrecht -o ./utrecht-ort
4. See error below

➜  ort analyze -i ../utrecht -o ./utrecht-ort
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
 ______________________________                                                              
/        \_______   \__    ___/       The OSS Review Toolkit, version 55.2.0-018.sha.0deb0e7,
|    |   | |       _/ |    |          built with JDK 21.0.6+7-LTS, running under Java 21.0.6.
|    |   | |    |   \ |    |          Executing 'analyze' as 'tsteenbe' on Mac OS X          
\________/ |____|___/ |____|          with 10 CPUs and a maximum of 12288 MiB of memory.     
                                                                                             
Environment variables:                                                                      
HOME = /Users/tsteenbe                                                                      
SHELL = /bin/zsh                                                                            
TERM = xterm-256color                                                                       
JAVA_HOME = /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home                  
                                                                                            
Looking for ORT configuration in the following file:
        /Users/tsteenbe/.ort/config/config.yml (does not exist)

Looking for analyzer-specific configuration in the following files and directories:
        /Volumes/Workspace/utrecht/.ort.yml (does not exist)
        /Users/tsteenbe/.ort/config/resolutions.yml (does not exist)
The following 26 package manager(s) are enabled:
        Bazel, Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, GoMod, Gradle Inspector, Maven, NPM, NuGet, PIP, Pipenv, PNPM, Poetry, Pub, SBT, SpdxDocumentFile, Stack, Swift Package Manager, Tycho, Unmanaged, Yarn, Yarn 2+
The following 2 package curation provider(s) are enabled:
        DefaultDir, DefaultFile
Analyzing project path:
        /Volumes/Workspace/utrecht
Found 1 PNPM definition file(s) at:
        package.json
Found in total 1 definition file(s) from the following 1 package manager(s):
        PNPM
22:10:50.085 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.PackageManager - PNPM failed to resolve dependencies for path 'package.json': IOException: Running 'pnpm list --json --recursive --depth Infinity --dev' in '/Volumes/Workspace/utrecht' failed with exit code 134:

<--- Last few GCs --->

[85074:0x130030000]    42636 ms: Scavenge (reduce) (interleaved) 4087.3 (4089.3) -> 4086.7 (4089.6) MB, pooled: 0 MB, 4.71 / 0.00 ms  (average mu = 0.322, current mu = 0.289) allocation failure; 
[85074:0x130030000]    42914 ms: Mark-Compact (reduce) 4087.5 (4089.6) -> 4085.2 (4089.6) MB, pooled: 0 MB, 212.29 / 0.00 ms  (+ 2342.8 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2789 ms) (average mu = 0.330

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x1050e1f5c node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 2: 0x10528cb10 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 3: 0x105449b40 v8::internal::Heap::CallGCPrologueCallbacks(v8::GCType, v8::GCCallbackFlags, v8::internal::GCTracer::Scope::ScopeId) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 4: 0x10544f5d8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_1::operator()() const [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 5: 0x105449e84 void heap::base::Stack::SetMarkerAndCallbackImpl<v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_1>(heap::base::Stack*, void*, void const*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 6: 0x104ff4028 PushAllRegistersAndIterateStack [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 7: 0x105446d6c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 8: 0x1053eb3f8 v8::internal::StackGuard::HandleInterrupts(v8::internal::StackGuard::InterruptLevel) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
 9: 0x1056f4770 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
[...skipping 18 lines...]
28: 0x104e3a3f0 Builtins_JSRunMicrotasksEntry [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
29: 0x1053bea38 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
30: 0x1053bf1a4 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
31: 0x1053e9758 v8::internal::MicrotaskQueue::PerformCheckpointInternal(v8::Isolate*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
32: 0x104ff55e4 node::InternalCallbackScope::Close() [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
33: 0x104ff5b94 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
34: 0x10500ddcc node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
35: 0x1050e80f0 node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
36: 0x1050eaac8 node::fs::AfterScanDir(uv_fs_s*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
37: 0x1050d88f0 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
38: 0x10898a14c uv__work_done [/opt/homebrew/Cellar/libuv/1.50.0/lib/libuv.1.dylib]
39: 0x10898da74 uv__async_io [/opt/homebrew/Cellar/libuv/1.50.0/lib/libuv.1.dylib]
40: 0x10899e01c uv__io_poll [/opt/homebrew/Cellar/libuv/1.50.0/lib/libuv.1.dylib]
41: 0x10898df08 uv_run [/opt/homebrew/Cellar/libuv/1.50.0/lib/libuv.1.dylib]
42: 0x104ff6488 node::SpinEventLoopInternal(node::Environment*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
43: 0x1051295e8 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
44: 0x10512933c node::NodeMainInstance::Run() [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
45: 0x1050a096c node::Start(int, char**) [/opt/homebrew/Cellar/node/23.10.0_1/bin/node]
46: 0x185dd8274 start [/usr/lib/dyld]

(Above output is limited to each 20 heading and tailing lines.)
Wrote analyzer result to '/Volumes/Workspace/utrecht-ort/analyzer-result.yml' (0.01 MiB) in 212.619875ms.
The analysis took 1m 31.749881s.
Found 1 project(s) and 0 package(s) in total (not counting excluded ones).
Applied 0 curation(s) from 0 of 2 provider(s).
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 1 error, 0 warnings, 0 hints.
There is 1 unresolved issue with a severity equal to or greater than the WARNING threshold.

Expected behavior

An analyzer-result.yml result file with no eeors

Environment

Output of the ort requirements command:

ort requirements                                
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
 ______________________________                                                              
/        \_______   \__    ___/       The OSS Review Toolkit, version 55.2.0-018.sha.0deb0e7,
|    |   | |       _/ |    |          built with JDK 21.0.6+7-LTS, running under Java 21.0.6.
|    |   | |    |   \ |    |          Executing 'requirements' as 'tsteenbe' on Mac OS X     
\________/ |____|___/ |____|          with 10 CPUs and a maximum of 12288 MiB of memory.     
                                                                                             
Environment variables:                                                                      
HOME = /Users/tsteenbe                                                                      
SHELL = /bin/zsh                                                                            
TERM = xterm-256color                                                                       
JAVA_HOME = /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home                  
                                                                                            
Looking for ORT configuration in the following file:
        /Users/tsteenbe/.ort/config/config.yml (does not exist)

Scanners:
        - Askalono: Requires 'askalono' in no specific version. Tool not found.
        - BoyterLc: Requires 'lc' in no specific version. Tool not found.
        - Licensee: Requires 'licensee' in no specific version. Tool not found.
        * ScanCode: Requires 'scancode' in version >=30.0.0. Found version 32.2.0.

PackageManagers:
        - Bazel: Requires 'bazel' in version >=7.0.0. Tool not found.
        - Bower: Requires 'bower' in version >=1.8.8. Tool not found.
        - Buildozer: Requires 'buildozer' in no specific version. Tool not found.
        * Cargo: Requires 'cargo' in no specific version. Found version 1.85.1.
        - CocoaPods: Requires 'pod' in version >=1.11.0. Tool not found.
        - Composer: Requires 'composer' in version >=1.5.0. Tool not found.
        - Go: Requires 'go' in version >=1.21.1. Tool not found.
        * Npm: Requires 'npm' in version >=6.0.0 and <11.0.0. Found version 10.9.2.
        - NuGetInspector: Requires 'nuget-inspector' in no specific version. Tool not found.
        - Pipenv: Requires 'pipenv' in version >=2018.10.9. Tool not found.
        * Pnpm: Requires 'pnpm' in version >=5.0.0 and <10.0.0. Found version 9.12.0.
        * Poetry: Requires 'poetry' in no specific version. Found version 1.8.3.
        * PythonInspector: Requires 'python-inspector' in version >=0.9.2. Found version 0.12.0.
        * Sbt: Requires 'sbt' in no specific version. Found version sbt runner version: 1.10.11.
        - Stack: Requires 'stack' in version >=2.1.1. Tool not found.
        * Swift: Requires 'swift' in no specific version. Found version 6.0.3.
        + Yarn: Requires 'yarn' in version >=1.3.0 and <1.23.0. Could not determine the version.

Other tools:
        - Conan: Requires 'conan' in version >=1.44.0 and <3.0.0. Tool not found.
        - Pub: Requires 'dart' in version >=2.10.0. Tool not found.

VersionControlSystems:
        * Git: Requires 'git' in version >=2.29.0. Found version 2.49.0.
        - GitRepo: Requires 'repo' in no specific version. Tool not found.
        - Mercurial: Requires 'hg' in no specific version. Tool not found.

Prefix legend:
        - The tool was not found in the PATH environment.
        + The tool was found in the PATH environment, but not in the required version.
        * The tool was found in the PATH environment in the required version.

ScanCode license texts not found.

Not all tools requirements were satisfied:
        ! For some tools the version could not be determined.
        ! Some tools were not found at all.

[tsteenbe]

Analyzing the sibling repository https://github.com/nl-design-system/denhaag works a bit better only getting a "PNPM failed to resolve dependencies for path 'package.json': NoSuchElementException: Key type is missing in the map." Only 1315 of 1970 dependencies are picked up see nl-design-system-denhaag-orthw.zip

➜  Workspace pnpm -v
10.7.1
➜  Workspace mkdir denhaag-orthw
➜  Workspace ort analyze -i denhaag -o ./denhaag-orthw   
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
 ______________________________                                                              
/        \_______   \__    ___/       The OSS Review Toolkit, version 55.2.0-018.sha.0deb0e7,
|    |   | |       _/ |    |          built with JDK 21.0.6+7-LTS, running under Java 21.0.6.
|    |   | |    |   \ |    |          Executing 'analyze' as 'tsteenbe' on Mac OS X          
\________/ |____|___/ |____|          with 10 CPUs and a maximum of 12288 MiB of memory.     
                                                                                             
Environment variables:                                                                      
HOME = /Users/tsteenbe                                                                      
SHELL = /bin/zsh                                                                            
TERM = xterm-256color                                                                       
JAVA_HOME = /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home                  
                                                                                            
Looking for ORT configuration in the following file:
        /Users/tsteenbe/.ort/config/config.yml (does not exist)

Looking for analyzer-specific configuration in the following files and directories:
        /Volumes/Workspace/denhaag/.ort.yml (does not exist)
        /Users/tsteenbe/.ort/config/resolutions.yml (does not exist)
The following 26 package manager(s) are enabled:
        Bazel, Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, GoMod, Gradle Inspector, Maven, NPM, NuGet, PIP, Pipenv, PNPM, Poetry, Pub, SBT, SpdxDocumentFile, Stack, Swift Package Manager, Tycho, Unmanaged, Yarn, Yarn 2+
The following 2 package curation provider(s) are enabled:
        DefaultDir, DefaultFile
Analyzing project path:
        /Volumes/Workspace/denhaag
Found 1 PNPM definition file(s) at:
        package.json
Found in total 1 definition file(s) from the following 1 package manager(s):
        PNPM
22:22:52.715 [main] WARN  org.ossreviewtoolkit.utils.common.CommandLineTool - The command is required in version >=5.0.0 and <10.0.0, but you are using version 10.7.1. This could lead to problems.
22:33:27.798 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.PackageManager - PNPM failed to resolve dependencies for path 'package.json': NoSuchElementException: Key type is missing in the map.
Wrote analyzer result to '/Volumes/Workspace/denhaag-orthw/analyzer-result.yml' (1.53 MiB) in 472.312875ms.
The analysis took 10m 44.582199s.
Found 1 project(s) and 0 package(s) in total (not counting excluded ones).
Applied 0 curation(s) from 0 of 2 provider(s).
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 1 error, 0 warnings, 0 hints.
There is 1 unresolved issue with a severity equal to or greater than the WARNING threshold.

[tsteenbe]

Tried analyzing another repository within the same project https://github.com/nl-design-system/rijkshuisstijl-community

ort analyze -i nl-design-system-rijkshuisstijl-community -o ./nl-design-system-rijkshuisstijl-community-orthw
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
 ______________________________                                                              
/        \_______   \__    ___/       The OSS Review Toolkit, version 55.2.0-018.sha.0deb0e7,
|    |   | |       _/ |    |          built with JDK 21.0.6+7-LTS, running under Java 21.0.6.
|    |   | |    |   \ |    |          Executing 'analyze' as 'tsteenbe' on Mac OS X          
\________/ |____|___/ |____|          with 10 CPUs and a maximum of 12288 MiB of memory.     
                                                                                             
Environment variables:                                                                      
HOME = /Users/tsteenbe                                                                      
SHELL = /bin/zsh                                                                            
TERM = xterm-256color                                                                       
JAVA_HOME = /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home                  
                                                                                            
Looking for ORT configuration in the following file:
        /Users/tsteenbe/.ort/config/config.yml (does not exist)

Looking for analyzer-specific configuration in the following files and directories:
        /Volumes/Workspace/nl-design-system-rijkshuisstijl-community/.ort.yml (does not exist)
        /Users/tsteenbe/.ort/config/resolutions.yml (does not exist)
The following 26 package manager(s) are enabled:
        Bazel, Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, GoMod, Gradle Inspector, Maven, NPM, NuGet, PIP, Pipenv, PNPM, Poetry, Pub, SBT, SpdxDocumentFile, Stack, Swift Package Manager, Tycho, Unmanaged, Yarn, Yarn 2+
The following 2 package curation provider(s) are enabled:
        DefaultDir, DefaultFile
Analyzing project path:
        /Volumes/Workspace/nl-design-system-rijkshuisstijl-community
Found 1 PNPM definition file(s) at:
        package.json
Found in total 1 definition file(s) from the following 1 package manager(s):
        PNPM
22:44:19.916 [main] WARN  org.ossreviewtoolkit.utils.common.CommandLineTool - The command is required in version >=5.0.0 and <10.0.0, but you are using version 10.7.1. This could lead to problems.
22:44:38.123 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.PackageManager - PNPM failed to resolve dependencies for path 'package.json': IOException: Running 'pnpm list --json --recursive --depth Infinity --prod' in '/Volumes/Workspace/nl-design-system-rijkshuisstijl-community' failed with exit code 1:
{
  "error": {
    "code": "pnpm",
    "message": "Invalid string length"
  }
}

Wrote analyzer result to '/Volumes/Workspace/nl-design-system-rijkshuisstijl-community-orthw/analyzer-result.yml' (0.00 MiB) in 202.088250ms.
The analysis took 31.349026s.
Found 1 project(s) and 0 package(s) in total (not counting excluded ones).
Applied 0 curation(s) from 0 of 2 provider(s).

Manually running pnpm list --json --recursive --depth Infinity --prod actually works and returns

[
  {
    "name": "@rijkshuisstijl-community/design-system",
    "version": "1.0.0-alpha.0",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community",
    "private": true
  },
  {
    "name": "@rijkshuisstijl-community/rhc-templates",
    "version": "1.0.2",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/apps/rhc-templates",
    "private": true
  },
  {
    "name": "@rijkshuisstijl-community/components-css",
    "version": "1.0.3",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/packages/components-css",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/components-react",
    "version": "1.0.4",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/packages/components-react",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/components-twig",
    "version": "1.3.1",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/packages/components-twig",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/font",
    "version": "1.0.1",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/packages/font",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/storybook",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/packages/storybook",
    "private": true
  },
  {
    "name": "@rijkshuisstijl-community/web-components",
    "version": "1.1.4",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/packages/web-components",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/assets",
    "version": "1.0.1",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/proprietary/assets",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/design-tokens",
    "version": "1.1.0",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/proprietary/design-tokens",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/digid-design-tokens",
    "version": "1.0.0",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/proprietary/digid-design-tokens",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/logius-design-tokens",
    "version": "1.0.0",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/proprietary/logius-design-tokens",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/mijnoverheid-design-tokens",
    "version": "1.0.0",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/proprietary/mijnoverheid-design-tokens",
    "private": false
  },
  {
    "name": "@rijkshuisstijl-community/rivm-design-tokens",
    "version": "1.0.0",
    "path": "/Volumes/Workspace/nl-design-system-rijkshuisstijl-community/proprietary/rivm-design-tokens",
    "private": false
  }
]

[tsteenbe]

On advice of @sschuberth I tried an older version of ORT to see of the rewrite by @fviernau of the ORT's npm/pnpm/yarn had any impact - first tried 43.0.0 same issue then switched to 30.0.0. Analyzing the github.com/nl-design-system/utrecht project with ORT 30.0.0 works albeit it take more 20 minutes to complete.

nl-design-system-utrecht-ort-30.0.0-analyzer-result.yml.zip

For comparison I ran Trivy which took 8 seconds to produce a cyclonedx SBOM - albeit with a lot less information in it.
trivy fs --format cyclonedx --include-dev-deps --output nl-design-system-utrecht.cyclonedx.json ../nl-design-system-utrecht => nl-design-system-utrecht.cyclonedx.json.zip

➜  nl-design-system-utrecht-orthw ort analyze -i ../nl-design-system-utrecht -o ./
 ______________________________                                                           
/        \_______   \__    ___/       The OSS Review Toolkit, version 30.0.0,             
|    |   | |       _/ |    |          built with JDK 11.0.26+4, running under Java 21.0.6.
|    |   | |    |   \ |    |          Executing 'analyze' as 'tsteenbe' on Mac OS X       
\________/ |____|___/ |____|          with 10 CPUs and a maximum of 12288 MiB of memory.  
                                                                                          
Environment variables:                                                                   
ORT_CONFIG_DIR = /Users/tsteenbe/.ort/config                                             
ORT_DATA_DIR = /Users/tsteenbe/.ort                                                      
HOME = /Users/tsteenbe                                                                   
SHELL = /bin/zsh                                                                         
TERM = xterm-256color                                                                    
JAVA_HOME = /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home               
                                                                                         
Looking for ORT configuration in the following file:
        /Users/tsteenbe/.ort/config/config.yml (does not exist)

Looking for analyzer-specific configuration in the following files and directories:
        /Volumes/Workspace/nl-design-system-utrecht/.ort.yml (does not exist)
        /Users/tsteenbe/.ort/config/resolutions.yml (does not exist)
The following 25 package manager(s) are enabled:
        Bazel, Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, GoMod, Gradle, Maven, NPM, NuGet, PIP, Pipenv, PNPM, Poetry, Pub, SBT, SpdxDocumentFile, Stack, SwiftPM, Unmanaged, Yarn, Yarn2
The following 2 package curation provider(s) are enabled:
        DefaultDir, DefaultFile
Analyzing project path:
        /Volumes/Workspace/nl-design-system-utrecht
Found 1 PNPM definition file(s) at:
        package.json
Found in total 1 definition file(s) from the following 1 package manager(s):
        PNPM
Wrote analyzer result to '/Volumes/Workspace/nl-design-system-utrecht-orthw/analyzer-result.yml' (3.34 MiB) in 1.102194917s.
The analysis took 20m 3.153053s.
Found 1 project(s) and 2919 package(s) in total (not counting excluded ones).
Applied 0 curation(s) from 0 of 2 provider(s).
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.

[tsteenbe]

I also reran https://github.com/nl-design-system/denhaag with ORT 30.0.0 and it found this time 1522 dependencies with a proper dependency tree unlike ORT 55. ORT CycloneDX search "type" : "library" results in 1515. hits.

Also ran trivy using trivy fs --format cyclonedx --include-dev-deps --output nl-design-system-denhaag.cyclonedx.json ../nl-design-system-denhaag and similar search for "type": "library" results in 1840 hits.

See also nl-design-system-denhaag-ort-30.zip for all the result files.

@fviernau Looks like ORT 30 did a better job with PNPM and we have a gap in detected dependencies compared to Trivy.

[tsteenbe]

For completeness i also analyzed https://github.com/nl-design-system/rijkshuisstijl-community with ORT 30.0.0 and it just works compared to 55.2.0 issues mentioned earlier in this issue. See for the result files nl-design-system-rijkshuisstijl-community-ort-30.zip. Again Trivy show more dependencies 2309 vs ORT's 1652.

[sschuberth]

Note that this issue now mixes various different root causes. For example, the third post (and anything that mentions "Invalid string length") is covered by #9405.

[sschuberth]

Regarding the memory issues, I wonder whether they're cause by the JSON produced by pnpm list --json --recursive --depth Infinity simply being massive, and the generated (lazily evaluated) sequence being converted to a list here.

@tsteenbe can you try out my pnpm-list-sequence branch to see if it makes a difference in terms of memory usage?

[sschuberth]

Actually, I just realized it's JavaScript, not Java, that's running out of memory:

JavaScript heap out of memory

So it again looks like an upstream PNPM bug. There seem to be several related issues.

[sschuberth]

So, I've tried to increase Node's memory via --max_old_space_size to up to 20 GiB, but still running into OOM issues. So to me, this clearly is a bug in PNPM.

@tsteenbe can you try out my pnpm-list-sequence branch to see if it makes a difference in terms of memory usage?

No need to try that out anymore, it does not help.