BUG: [CADT] Observer node OOM crashed

[fizpawiz]

No description provided.

[fizpawiz]

Here is the log from the production Observer node crash:

Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: <--- Last few GCs --->
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: [436:0x4338ae0] 1708885498 ms: Scavenge 2040.2 (2084.7) -> 2038.9 (2085.0) MB, 2.6 / 0.0 ms  (average mu = 0.219, current mu = 0.164) allocation failure
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: [436:0x4338ae0] 1708885504 ms: Scavenge 2040.5 (2085.0) -> 2039.2 (2085.5) MB, 2.4 / 0.0 ms  (average mu = 0.219, current mu = 0.164) allocation failure
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: [436:0x4338ae0] 1708886058 ms: Mark-sweep 2040.8 (2085.5) -> 2039.4 (2090.0) MB, 550.6 / 0.0 ms  (average mu = 0.154, current mu = 0.062) allocation failure scavenge might not succeed
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: <--- JS stacktrace --->
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  1: 0x99e1e0 node::Abort() [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  2: 0x8c9613 node::FatalError(char const*, char const*) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  3: 0x1069a0e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  4: 0x1069cb6 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  5: 0x1229e35  [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  6: 0x122ab6e v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  7: 0x1238ded v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  8: 0x123975a v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]:  9: 0x123ca0b v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 10: 0x11f9685 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 11: 0x11f1a2c v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 12: 0x11f1b04 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 13: 0x1470b61 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::NewInternal<v8::internal::Isolate>(v8::internal::Isolate*, int, v8::internal::AllocationType) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 14: 0x1470f13 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::AllocationType) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 15: 0x1470f7c v8::internal::Handle<v8::internal::NameDictionary> v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 16: 0x147142a v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 17: 0x15a4776 v8::internal::Runtime_AddDictionaryProperty(int, unsigned long*, v8::internal::Isolate*) [/opt/cadt/cadt]
Jan 09 23:37:20 ip-10-102-0-40 cadt[436]: 18: 0x198cdb9  [/opt/cadt/cadt]
Jan 09 23:37:38 ip-10-102-0-40 systemd[1]: cadt@ubuntu.service: Main process exited, code=dumped, status=6/ABRT
Jan 09 23:37:38 ip-10-102-0-40 systemd[1]: cadt@ubuntu.service: Failed with result 'core-dump'.

[fizpawiz]

I expect this to be resolved in 2.2.