[BUG] Nodes stuck following v6.0.2 upgrade

[web3-nodeops]

Seid version
name: sei
server_name:
version: v6.0.2
commit: f17ec67
build_tags: netgo muslc,
go: go version go1.21.4 linux/amd64
build_deps:

• cosmossdk.io/errors@v1.0.0
• filippo.io/edwards25519@v1.0.0-rc.1
• github.com/99designs/keyring@v1.2.1

Chain ID
pacific-1

Describe the bug
Nodes not syncing, stuck on block heigh 126326960

4:27AM ERR wrong Block.Header.LastResultsHash. Expected 4EDD4EBA0284AAE62A0D53469EFC4D98DAB0A409D3D651B988E73816ABB36F6C, got B433EAC981C0DC14ED3E8AC2FEB74392DF3524F3D6E7E2380A72A6DCF1D60DDE block_id={"hash":"F5F9C37F09E9F33538BF20C6C2EF69576EF603B3FAF90CED0066B73D38292BFA","parts":{"hash":"6F46883323400ADEFD1F2C020384750B3805EC4DA4127BC40A44B4FB1754F414","total":1}} height=126326961 last_commit={"block_id":{"hash":"F5F9C37F09E9F33538BF20C6C2EF69576EF603B3FAF90CED0066B73D38292BFA","parts":{"hash":"6F46883323400ADEFD1F2C020384750B3805EC4DA4127BC40A44B4FB1754F414","total":1}},"height":"126326961","round":1,"signatures":[{"block_id_flag":2,"signature":"XGszRcIyfrJGkQVafmYsvECM701Wo4614rpa1AYAeRVaGbticQn+7ntonxH6kP4cLCMhDBRcQrWbzPivWW1aCQ==","timestamp":"2025-01-15T04:22:31.068657311Z","validator_address":"BA479C5C63D95550191E3D6434BA557BD5F69508"},{"block_id_flag":2,"signature":"H5qugLoo/ma9rqVSEg8b18yTt9BuTW30JvTXg4aja+D4Ti2Yx3mR+VUm8PE5C9onRsCOtu9wqi6PBBB39mRbBA==","timestamp":"2025-01-15T04:22:31.006846943Z","validator_address":"7A657EABDE32B51E460F1893CD3C753A802AAF46"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":2,"signature":"BNyeHsLHkjSfejTK+0BplVn3U93r5T5EwblmodIR4PVaUGCKxvzxKJoYxSQA3qpWysGueoTRoTMGeIoS8ZnfCQ==","timestamp":"2025-01-15T04:22:31.010764687Z","validator_address":"7482D68A8C68FD7204273223D90864207B1558B5"},{"block_id_flag":2,"signature":"SWbCF9Ib6R4lb2iN4UmDwgYrUA7MdGFcqq/LPxbNSZNHlYAMdkgtXJ+Bbo3sFYahRvBhwm1LF8AjPOm8cyunAw==","timestamp":"2025-01-15T04:22:31.076256448Z","validator_address":"F04B5B9C48B65604A1309E0B0046B4F7CF8C7EF2"},{"block_id_flag":2,"signature":"AzyWXBzajHi3btWRujkaxcL0RaDfrLKSn/J86L3mVc3ZwxX/3+nW5oRcqmSxHV1rONUqnmvQpO49m0HIuHv3BA==","timestamp":"2025-01-15T04:22:31.073391896Z","validator_address":"3D4CEB152CA0271E5A9AF577D67C828A228C87DF"},{"block_id_flag":2,"signature":"VuaPAE/T363LYzNSpDnWDhKZrKRlOoOtmEFUgYasSf3fVn4leVOcXPoVevJZi8VP8RLendNLFJeeOSF4QD3RCQ==","timestamp":"2025-01-15T04:22:31.053699885Z","validator_address":"9F1F7B9B3584198F985ADB6AA72037731D2A3C73"},{"block_id_flag":2,"signature":"6Ynno1Ow4R9ASSUN3WQXg4v3ZRg/cojoPEvIiS3p8Me1dSbrdTFpT3b01HZElWUX5PiSkbu+UuEjlEGuSw3OCA==","timestamp":"2025-01-15T04:22:31.023409882Z","validator_address":"40F909929F7FA9D475E2C6F5A94AC9D6D97CF83D"},{"block_id_flag":2,"signature":"v5L7p++Pe3VFYF4ik4eyXNCWA0LpYzInwTxTtR5t4DtWPX4IycjR05xjPgcvnzTeObt/cLrVgss/H1tFEubdAA==","timestamp":"2025-01-15T04:22:31.104373824Z","validator_address":"24F70E6CFAE07F57FE58F8F6101ECB66E42C96BB"},{"block_id_flag":2,"signature":"xL4EB+wGjsNFei/wuokM/27cCSQCq1eXrA+ZsA1JHDDV0bYJisFcnKRTqpZlm9SZufD+gz0m80Y/GTlkeTvLCw==","timestamp":"2025-01-15T04:22:31.088752765Z","validator_address":"2F3EE9AE75D85D4A6CD93A3FDE2CA31CE7878642"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":2,"signature":"n1VXeyZ8luJKCs7U/Qe2Gas9uFkHGMSLpi1QIJ9UCIZTf3/1sqdO0Nq7WJHBSn1S6+rE4ozjRvKSb9KNG0cJBQ==","timestamp":"2025-01-15T04:22:31.052143237Z","validator_address":"B016593FC42B4A1988404F3D80AFEF984BD966E2"},{"block_id_flag":2,"signature":"+7fANZoXLBukGGgton24PyKAMBmDSv8khIKvI7X/eTOjnE/rhQDhHOiXMQVFjYRsGN46WS8Pb65z925MFILxCA==","timestamp":"2025-01-15T04:22:31.040658296Z","validator_address":"B1E0490AAC5CD08248C284378296D86810410591"},{"block_id_flag":2,"signature":"R/HV9f7gMImOoXfRaeBVY/gC1pufVE/nyIrOXbE+BX1fhl4HhuV17RxUtGAm7DZCxjQjHRoca87PJgw//ScnAA==","timestamp":"2025-01-15T04:22:31.046950681Z","validator_address":"E1CA4BCEB36E12283CCAB4A844A97E0C050F005A"},{"block_id_flag":2,"signature":"qnn/aCwfcXF0lvQFE99uwKpFhVSTo6DRkZx+G2OOz+5EHfPVPA2FBEG1e7dLYa/zoQMVesILKiiSsGAKMrCaCw==","timestamp":"2025-01-15T04:22:31.065075692Z","validator_address":"114C0350B9CE023F98518D9F22A2E2E7548DAF66"},{"block_id_flag":2,"signature":"0nrJKUcItAePLDfSlK2kD0Hp64y5zRWPfnATtYbH18bgebDLoodva3cvhn2PMIGGBuD2fejK1k6moz7w3gAHAw==","timestamp":"2025-01-15T04:22:31.128769842Z","validator_address":"6CA80EE04179FF609B07F9BBC65C6E47F20D09CA"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":2,"signature":"ntuvOyLYhZCmSIkEhokN9m0HBpND0OIMb5VX2dFD2Q5wfY/1zGfmm1juxmfA/JCgl8KLg5/84UQ8JFfG50NAAw==","timestamp":"2025-01-15T04:22:31.022004185Z","validator_address":"1AEFCC514E41F3DD21F5CE20B7BA3D142EB56256"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":2,"signature":"XftOdZUQdrgP+rU2v0MvgqBO/cZrG7LZRH20sbL3OFGz+0ZNYTHaT6COuPi38NoFoHTWcA5cLeblXSiQskqeAg==","timestamp":"2025-01-15T04:22:31.062162197Z","validator_address":"59E78E0B1133144460575337FAA2558399E43661"},{"block_id_flag":2,"signature":"NbpBMzG8UAl5F1nruB9XpaXZEg5+EF5SmSMydMev1Ka6NCdYc/0ahtze/z7f9ZHcgrexG4kHtkFhziTeS6S2DA==","timestamp":"2025-01-15T04:22:31.067715131Z","validator_address":"70D2FB9E966C1842640933D4ADC32FD56D37CD7F"},{"block_id_flag":2,"signature":"JlH3xcgKQ0gEEG9TyRiPFPuvHZgLkzixn1wLqGF4rOU6wmcp34nlecvwFHyWyXsBFKE9W3Y37lK0XWXUyTPzAQ==","timestamp":"2025-01-15T04:22:31.06073341Z","validator_address":"8E4218ECD657CCD6FDE32C0FC0C4AC3366B71E5D"},{"block_id_flag":2,"signature":"qCFGz87z4cnPWAE9Oz3VkQ3Y/2/XXN9IpJVDlXtfWx5VR7ksLujOMmMQ8XExnxvzjIitMlx84CgksG/OPe1rDQ==","timestamp":"2025-01-15T04:22:31.093815542Z","validator_address":"25ABC30B5E07AFD6CA59EAA3495AC44D6B5C3026"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":2,"signature":"iWBZ8sIqIwH2wOhpVS3y0fQOmEnqG/MhM8RIkp9E146eX6+3GiOu7bJp6LLsG5enXC6DphG8RwOX0LUqB+cdCw==","timestamp":"2025-01-15T04:22:30.990939043Z","validator_address":"922357BD8E69E6119D0630F7ABBBF7C5E3879F2F"},{"block_id_flag":2,"signature":"kubpyt3SSpSo1kxCNQiutPRNMRC/R0mvWndLtK9K8BwN0NGUBezY8eQxXmGTrtf+JqCocX7phpyhoDdRkuD/Dw==","timestamp":"2025-01-15T04:22:31.033389122Z","validator_address":"A633A4EEB521DD9F09326BF5C945BC24327C6A66"},{"block_id_flag":2,"signature":"PpgPUknHerZpboSnZoeARvzc/CECpyjcVyjJEzaL0fbu12muyiuOPq4gPloDpUfjAJtlR8O5thVDgsou6J3LBg==","timestamp":"2025-01-15T04:22:31.093017581Z","validator_address":"CCC61D60E3DD1121BBBB2EA4C0039F91FD372389"},{"block_id_flag":2,"signature":"10P/dvcpYdRO4CCnmX029SPc1rmDCKDepRY299RpSLFZ87VvyEsGKjsXLdU4/jKE24dKcP7I6gyZr9S1i0AoDw==","timestamp":"2025-01-15T04:22:31.102347671Z","validator_address":"507A1930FE00C2B4D3DAE52C7463DCDA2468A5ED"},{"block_id_flag":2,"signature":"g0z4J/zbaQvb2zUmwAoAm1gIqR9e2B4Oc9tiJxmzKzAUlaYv0YvPk4w+gO+17yH+CON4PXc3HrEx8jZfjr52Dg==","timestamp":"2025-01-15T04:22:31.029333702Z","validator_address":"8F7E88D0344F74BC17822506E7E29B220AD221DA"},{"block_id_flag":2,"signature":"Tx7h3aJX0ty8vOkCA7VnFYh7vpLICUdsIByMxVfRZ6T2IPaN76Z5rqw98iJtYziJLcrpH+NzsVzTilNPiedCAg==","timestamp":"2025-01-15T04:22:31.066226794Z","validator_address":"183388665BAC7D2537A91633FDC9ED5D6F46EF37"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":2,"signature":"2f9Gmm5iujMKn/GsTsWNn/SZZSlN9Z410u2Hwvkpjn/yenO/oJrY3gba0y0xHa4gOc5GDlvNKTPgQtlCDq/1CA==","timestamp":"2025-01-15T04:22:31.091336035Z","validator_address":"E06A77383534E7157F4BE3C873B25A62E7FF6428"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""},{"block_id_flag":2,"signature":"KeeWTK4VsOhnuWkbHlY1NkLvOgnSmkfzdMhWiPWDXNp/AQ2FHH1PBJCh5ycTMk4rWOt6Gr2BU21wsnCJCbm2CA==","timestamp":"2025-01-15T04:22:31.09338297Z","validator_address":"F878712942EE2566B849ADB75DA7993FD93AD71D"},{"block_id_flag":1,"signature":null,"timestamp":"0001-01-01T00:00:00Z","validator_address":""}]} module=blockchain
4:27AM ERR peer error err="wrong Block.Header.LastResultsHash. Expected 4EDD4EBA0284AAE62A0D53469EFC4D98DAB0A409D3D651B988E73816ABB36F6C, got B433EAC981C0DC14ED3E8AC2FEB74392DF3524F3D6E7E2380A72A6DCF1D60DDE" evicting=false module=p2p peer=0bdf3a158189084846e96632834b7311d9bf03af
4:27AM ERR peer error err="wrong Block.Header.LastResultsHash. Expected 4EDD4EBA0284AAE62A0D53469EFC4D98DAB0A409D3D651B988E73816ABB36F6C, got B433EAC981C0DC14ED3E8AC2FEB74392DF3524F3D6E7E2380A72A6DCF1D60DDE" evicting=false module=p2p peer=
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x49 pc=0x1655aa8]
goroutine 733 [running]:
github.com/tendermint/tendermint/internal/p2p.(*PeerManager).processPeerEvent(0xc000360000, {0x4307488?, 0xc000204dc0?}, {{0x0, 0x0}, {0x336be5e, 0x3}, 0x0})
/go/pkg/mod/github.com/sei-protocol/sei-tendermint@v0.4.6/internal/p2p/peermanager.go:1005 +0x128
github.com/tendermint/tendermint/internal/p2p.(*Router).routeChannel(0xc190c2c160, {0x4307488, 0xc000204dc0}, 0x40, 0xc0caf5a120, 0xc0caf5a180, {0x7f381984d7b8?, 0xc0c5ac4a30})
/go/pkg/mod/github.com/sei-protocol/sei-tendermint@v0.4.6/internal/p2p/router.go:427 +0x9ed
github.com/tendermint/tendermint/internal/p2p.(*Router).OpenChannel.func1()
/go/pkg/mod/github.com/sei-protocol/sei-tendermint@v0.4.6/internal/p2p/router.go:309 +0x8c
created by github.com/tendermint/tendermint/internal/p2p.(*Router).OpenChannel in goroutine 1
/go/pkg/mod/github.com/sei-protocol/sei-tendermint@v0.4.6/internal/p2p/router.go:300 +0x67b

To Reproduce
Steps to reproduce the behavior:
Upgrade to v6.0.2
Restart Node
See error

Expected behavior
Node syncs

[lukasz-layerzerolabs]

+1 same here. Rolling back (6.0.1 hfx 4) does not help. 6.0.2 stuck.

[blocknodehub]

I have the same problem

[xinzhongyoumeng]

+1

[angusscott]

+1

[galaxy-mario]

+1

[galaxy-mario]

@web3-nodeops if you're using docker, check: https://github.com/sei-protocol/sei-chain/pull/2036/files

you will also need to sync from scratch with a snapshot

[blocknodehub]

check the version of wasmvm to 1.5.4-sei.0.0.1

[lukasz-layerzerolabs]

1. stop the node
2. seid rollback + seid rollback --hard
3. using the image with wasmvm patch mentioned above helped us resolve the issue without having to sync from scratch.

[oleg-glagovskiy]

I am using https://github.com/alexander-sei/sei-binaries/releases/download/v6.0.2/seid-v6.0.2-linux-amd64 but it has the same problem, so it's not the docker, but issue is the same. And I am not sure how in such cae to patch wasmvm

[galaxy-mario]

@oleg-glagovskiy that binary was not built with wasm 1.5.4

[oleg-glagovskiy]

@alexander-sei can you rebuild it please with proper wasm version?

[alexander-sei]

@alexander-sei can you rebuild it please with proper wasm version?

@oleg-glagovskiy Will do 👍

[alexander-sei]

@oleg-glagovskiy @galaxy-mario please try the newly compiled v6.0.2 binary: https://github.com/alexander-sei/sei-binaries/releases/tag/v6.0.2