My v3.2.0 cluster went into a weird state, It's no...
# nebula
p
My v3.2.0 cluster went into a weird state, It's not working. Snapshot send failed, the leader changed? Does anyone know how to fix this? I20220915 162809.498340 102 NebulaSnapshotManager.cpp:67] Space 61 Part 34 start send snapshot of commitLogId 89598562 commitLogTerm 2, rate limited to 10485760, batch size is 1048576 I20220915 162809.498370 102 SnapshotManager.cpp:51] [Port: 9780, Space: 61, Part: 34] Snapshot send failed, the leader changed? I20220915 162809.498400 102 Host.cpp:355] [Port: 9780, Space: 61, Part: 34] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Send snapshot failed! I20220915 162809.498530 53 Host.cpp:337] [Port: 9780, Space: 61, Part: 115] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 32549073, firstLogId in wal = 32538198, lastLogId in wal = 32549073 I20220915 162809.498550 66 Host.cpp:337] [Port: 9780, Space: 61, Part: 124] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 97588614, firstLogId in wal = 97582907, lastLogId in wal = 97588614 I20220915 162809.498647 103 NebulaSnapshotManager.cpp:67] Space 61 Part 115 start send snapshot of commitLogId 32549062 commitLogTerm 2, rate limited to 10485760, batch size is 1048576 I20220915 162809.498672 103 SnapshotManager.cpp:51] [Port: 9780, Space: 61, Part: 115] Snapshot send failed, the leader changed? I20220915 162809.498690 103 Host.cpp:355] [Port: 9780, Space: 61, Part: 115] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Send snapshot failed! I20220915 162809.498718 104 NebulaSnapshotManager.cpp:67] Space 61 Part 124 start send snapshot of commitLogId 97588603 commitLogTerm 2, rate limited to 10485760, batch size is 1048576 I20220915 162809.498766 104 SnapshotManager.cpp:51] [Port: 9780, Space: 61, Part: 124] Snapshot send failed, the leader changed? I20220915 162809.498817 104 Host.cpp:355] [Port: 9780, Space: 61, Part: 124] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Send snapshot failed! I20220915 162809.499837 72 Host.cpp:337] [Port: 9780, Space: 61, Part: 34] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 89598573, firstLogId in wal = 89579895, lastLogId in wal = 89598573 I20220915 162809.499940 101 NebulaSnapshotManager.cpp:67] Space 61 Part 34 start send snapshot of commitLogId 89598562 commitLogTerm 2, rate limited to 10485760, batch size is 1048576 I20220915 162809.499976 101 SnapshotManager.cpp:51] [Port: 9780, Space: 61, Part: 34] Snapshot send failed, the leader changed? I20220915 162809.500000 101 Host.cpp:355] [Port: 9780, Space: 61, Part: 34] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Send snapshot failed! I20220915 162809.500094 53 Host.cpp:337] [Port: 9780, Space: 61, Part: 115] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 32549073, firstLogId in wal = 32538198, lastLogId in wal = 32549073 I20220915 162809.500183 102 NebulaSnapshotManager.cpp:67] Space 61 Part 115 start send snapshot of commitLogId 32549062 commitLogTerm 2, rate limited to 10485760, batch size is 1048576 I20220915 162809.500219 102 SnapshotManager.cpp:51] [Port: 9780, Space: 61, Part: 115] Snapshot send failed, the leader changed? I20220915 162809.500241 102 Host.cpp:355] [Port: 9780, Space: 61, Part: 115] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Send snapshot failed! I20220915 162809.500391 66 Host.cpp:337] [Port: 9780, Space: 61, Part: 124] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 97588614, firstLogId in wal = 97582907, lastLogId in wal = 97588614 I20220915 162809.500461 103 NebulaSnapshotManager.cpp:67] Space 61 Part 124 start send snapshot of commitLogId 97588603 commitLogTerm 2, rate limited to 10485760, batch size is 1048576 I20220915 162809.500497 103 SnapshotManager.cpp:51] [Port: 9780, Space: 61, Part: 124] Snapshot send failed, the leader changed? I20220915 162809.500515 103 Host.cpp:355] [Port: 9780, Space: 61, Part: 124] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Send snapshot failed! I20220915 162809.501394 72 Host.cpp:337] [Port: 9780, Space: 61, Part: 34] [Host: nebula-cluster-storaged-1.nebula-cluster-storaged-headless.cohort-search.svc.cluster.local:9780] Can't find log 1 in wal, send the snapshot, logIdToSend = 89598573, firstLogId in wal = 89579895, lastLogId in wal = 89598573 I