Dominik
11/24/2022, 4:45 PMF20221124 17:04:31.736765 14793 DiskManager.cpp:39] DataPath invalid: boost::filesystem::status: Function not implemented: "/usr/local/nebula/data/meta"
*** Check failure stack trace: ***
*** Aborted at 1669305871 (Unix time, try 'date -d @1669305871') ***
*** Signal 6 (SIGABRT) (0x39c9) received by PID 14793 (pthread TID 0x7ff1eb7bc080) (linux TID 14793) (maybe from PID 14793, UID 0) (code: -6), stack trace: ***
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x253ad41]
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x26)[0x2532246]
/usr/local/nebula/bin/nebula-metad[0x25301d7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x1297f)[0x7ff1eabdb97f]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7ff1ea816e87]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x140)[0x7ff1ea8187f0]
/usr/local/nebula/bin/nebula-metad[0x2563a49]
/usr/local/nebula/bin/nebula-metad[0x2566b43]
/usr/local/nebula/bin/nebula-metad[0x2563518]
/usr/local/nebula/bin/nebula-metad[0x25671f8]
/usr/local/nebula/bin/nebula-metad[0x1056fa4]
/usr/local/nebula/bin/nebula-metad(_ZN6nebula7kvstore11NebulaStore4initEv+0x2d9)[0x1b46319]
/usr/local/nebula/bin/nebula-metad(_Z6initKVSt6vectorIN6nebula8HostAddrESaIS1_EES1_+0x682)[0x1154562]
/usr/local/nebula/bin/nebula-metad(main+0x8e2)[0x1105532]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x7ff1ea7f9c86]
/usr/local/nebula/bin/nebula-metad(_start+0x29)[0x1142cd9]
terminate called after throwing an instance of 'apache::thrift::concurrency::SystemResourceException'
what(): pthread_create failed
*** Aborted at 1669303920 (Unix time, try 'date -d @1669303920') ***
*** Signal 6 (SIGABRT) (0x2fd1) received by PID 12241 (pthread TID 0x7f07a8f5c080) (linux TID 12241) (maybe from PID 12241, UID 0) (code: -6), stack trace: ***
/usr/local/nebula/bin/nebula-storaged(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x26443e1]
/usr/local/nebula/bin/nebula-storaged(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x26)[0x263b8e6]
/usr/local/nebula/bin/nebula-storaged[0x2639877]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x1297f)[0x7f07a837b97f]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f07a7fb6e87]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x140)[0x7f07a7fb87f0]
/usr/local/nebula/bin/nebula-storaged[0x1150f82]
/usr/local/nebula/bin/nebula-storaged(_ZN10__cxxabiv111__terminateEPFvvE+0x5)[0x2b763d5]
/usr/local/nebula/bin/nebula-storaged(_ZSt9terminatev+0x10)[0x2b76440]
/usr/local/nebula/bin/nebula-storaged(__cxa_rethrow+0x45)[0x2b765e5]
/usr/local/nebula/bin/nebula-storaged[0x1108668]
/usr/local/nebula/bin/nebula-storaged(_ZN6apache6thrift11concurrency21PriorityThreadManager12PriorityImpl5startEv+0x5d)[0x230d10d]
/usr/local/nebula/bin/nebula-storaged(_ZN6nebula7storage13StorageServer5startEv+0x19b)[0x11aa73b]
/usr/local/nebula/bin/nebula-storaged(main+0xe64)[0x1154f24]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x7f07a7f99c86]
/usr/local/nebula/bin/nebula-storaged(_start+0x29)[0x1191569]
wey
11/25/2022, 2:39 AMDataPath invalid: boost::filesystem::status: Function not implemented:
Could you please help share your information here or in the issue?
• what is the Linux version, is it ARM64 or? (uname -a
)
Thanks! if possible the mitigations are using modern versions of OS.Dominik
11/25/2022, 3:31 PMLinux / 4.15.0 #1 SMP Thu Apr 22 16:18:59 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux
Linux / 5.4.0 #1 SMP Thu Apr 22 16:18:59 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux
but I'm constantly running into this in metad:
terminate called after throwing an instance of 'std::system_error'
what(): Resource temporarily unavailable
*** Aborted at 1669568023 (Unix time, try 'date -d @1669568023') ***
*** Signal 6 (SIGABRT) (0x6660) received by PID 26208 (pthread TID 0x7fd4618e6700) (linux TID 26396) (maybe from PID 26208, UID 0) (code: -6), stack trace: ***
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x253ace1]
/usr/local/nebula/bin/nebula-metad(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x26)[0x25321e6]
/usr/local/nebula/bin/nebula-metad[0x2530177]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x1441f)[0x7fd47e3ab41f]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fd47e1e800b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12a)[0x7fd47e1c7858]
/usr/local/nebula/bin/nebula-metad[0x1101b02]
/usr/local/nebula/bin/nebula-metad(_ZN10__cxxabiv111__terminateEPFvvE+0x5)[0x2a6c3c5]
/usr/local/nebula/bin/nebula-metad(_ZSt9terminatev+0x10)[0x2a6c430]
/usr/local/nebula/bin/nebula-metad(__cxa_throw+0x43)[0x2a6c583]
/usr/local/nebula/bin/nebula-metad(_ZSt20__throw_system_errori+0x80)[0x1103ae3]
/usr/local/nebula/bin/nebula-metad(_ZNSt6thread15_M_start_threadESt10unique_ptrINS_6_StateESt14default_deleteIS1_EEPFvvE+0x28)[0x2aea238]
/usr/local/nebula/bin/nebula-metad(_ZN5folly18ThreadPoolExecutor10addThreadsEm+0x5a3)[0x2478593]
/usr/local/nebula/bin/nebula-metad(_ZN5folly18ThreadPoolExecutor19ensureActiveThreadsEv+0x84)[0x247d3a4]
/usr/local/nebula/bin/nebula-metad(_ZN5folly18ThreadPoolExecutor11addObserverESt10shared_ptrINS0_8ObserverEE+0x10f)[0x247d8bf]
/usr/local/nebula/bin/nebula-metad(_ZN8proxygen10HTTPServer14startTcpServerESt10shared_ptrIN6wangle15AcceptorFactoryEES1_IN5folly20IOThreadPoolExecutorEE+0x1f7)[0x2357927]
/usr/local/nebula/bin/nebula-metad(_ZN8proxygen10HTTPServer5startESt8functionIFvvEES1_IFvNSt15__exception_ptr13exception_ptrEEESt10shared_ptrIN6wangle15AcceptorFactoryEES8_IN5folly20IOThreadPoolExecutorEE+0xa8)[0x2358988]
/usr/local/nebula/bin/nebula-metad[0x1a462a5]
/usr/local/nebula/bin/nebula-metad[0x1a46017]
/usr/local/nebula/bin/nebula-metad[0x2aea15f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8608)[0x7fd47e39f608]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x42)[0x7fd47e2c4132]
(safe mode, symbolizer not available)
I somehow managed to start everything and connect once, tried to ADD HOSTS which was then still offline, tried to restart and since then I can't do anything reallywey
11/28/2022, 1:47 AMADD HOSTS
should be exactly the same as it was configured in storaged’s conf for its IP(or host) and Port.
The traffic between metad and storaged should be done properly to enable its ONLINE. Could you share the conf and logs of storaged and metad if it’s still offline?Dominik
11/28/2022, 2:27 AMwey
11/28/2022, 5:50 AMDominik
11/29/2022, 5:32 PMmin wu
12/06/2022, 5:21 AMDominik
12/06/2022, 5:07 PM