Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CRASH] KeyDB 6.3.4 crashed by signal: 11, si_code: 1 #864

Open
broken-ufa opened this issue Sep 11, 2024 · 0 comments
Open

[CRASH] KeyDB 6.3.4 crashed by signal: 11, si_code: 1 #864

broken-ufa opened this issue Sep 11, 2024 · 0 comments

Comments

@broken-ufa
Copy link

broken-ufa commented Sep 11, 2024 •
edited
Loading

Crash report

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
3907654:3907709:M 10 Sep 2024 17:43:09.802 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
3907654:3907709:M 10 Sep 2024 17:43:09.802 # Accessing address: 0x1
3907654:3907709:M 10 Sep 2024 17:43:09.802 # Crashed running the instruction at: 0x62f7bf5c4068

------ STACK TRACE ------
EIP:
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x62f7bf5c4068]

Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x704782042520]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x62f7bf5c4068]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](+0x1d196a) [0x62f7bf5b096a]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](readQueryFromClient(connection*)+0x39c) [0x62f7bf5a2d0c]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](connSocketEventHandler(aeEventLoop*, int, void*, int)+0x1e0) [0x62f7bf6986e0]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](ProcessEventCore+0xf8) [0x62f7bf56f338]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeProcessEvents+0x17d) [0x62f7bf5733fd]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeMain+0x3e) [0x62f7bf573a6e]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](workerThreadMain(void*)+0xfe) [0x62f7bf58be9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x704782094ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x704782126850]

------ REGISTERS ------
3907654:3907709:M 10 Sep 2024 17:43:09.810 #
RAX:4297ccfb0fc91ac9 RBX:00007047812b7380
RCX:0000704583ce43d3 RDX:0000000000000001
RDI:0000000000c91ac9 RSI:0000000000000000
RBP:00007046b85c4163 RSP:000070475affc6e0
R8 :845d0f850ffe5bcb R9 :0000704626f6f3bb
R10:0000000000000003 R11:000062f7bfb0a010
R12:0000704655081b90 R13:00000000003ba078
R14:00007045fea9f800 R15:0000704655081b90
RIP:000062f7bf5c4068 EFL:0000000000010202
CSGSFS:002b000000000033
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ef) -> 0000704700001fba
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ee) -> 0000704700000000
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ed) -> 000070475affd978
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ec) -> 0000000059b0b680
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6eb) -> 0100010100000001
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6ea) -> 000070475affc74c
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e9) -> 000000005affc838
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e8) -> 000000000000001f
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e7) -> 000062f7bf58493c
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e6) -> 0000000000000000
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e5) -> 0000000000000001
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e4) -> 000062f7bfc4b920
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e3) -> 000070447f4cd740
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e2) -> 000070447e993ac0
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e1) -> 000062f7bfc49c00
3907654:3907709:M 10 Sep 2024 17:43:09.810 # (000070475affc6e0) -> 000070475affdb38

------ INFO OUTPUT ------
# Server
redis_version:6.3.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b532cd0401cb0da4
redis_mode:cluster
os:Linux 6.5.0-27-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:3907654
process_supervised:no
run_id:3025d938de37fc657b06ca71bd393385236bc727
tcp_port:7000
server_time_usec:1725990189810847
uptime_in_seconds:427643
uptime_in_days:4
hz:200
configured_hz:200
lru_clock:14714157
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb-master2-keydb2.conf
availability_zone:
features:cluster_mget

# Clients
connected_clients:3020
cluster_connections:16
maxclients:10000
client_recent_max_input_buffer:81968
client_recent_max_output_buffer:5369
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:5
thread_0_clients:526
thread_1_clients:441
thread_2_clients:521
thread_3_clients:514
thread_4_clients:535
thread_5_clients:485

# Memory
used_memory:8545836632
used_memory_human:7.96G
used_memory_rss:9131364352
used_memory_rss_human:8.50G
used_memory_peak:9357867144
used_memory_peak_human:8.72G
used_memory_peak_perc:91.32%
used_memory_overhead:1068160545
used_memory_startup:5103768
used_memory_dataset:7477676079
used_memory_dataset_perc:87.55%
allocator_allocated:8549300992
allocator_active:9017274368
allocator_resident:9192763392
total_system_memory:16760053760
total_system_memory_human:15.61G
used_memory_lua:39936
used_memory_lua_human:39.00K
used_memory_scripts:232
used_memory_scripts_human:232B
number_of_cached_scripts:1
maxmemory:13958643712
maxmemory_human:13.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.05
allocator_frag_bytes:467973376
allocator_rss_ratio:1.02
allocator_rss_bytes:175489024
rss_overhead_ratio:0.99
rss_overhead_bytes:-61399040
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:583801504
mem_not_counted_for_evict:0
mem_replication_backlog:268435456
mem_clients_slaves:46393
mem_clients_normal:52374328
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:2518313641
rdb_bgsave_in_progress:0
rdb_last_save_time:1725562546
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:34566144
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:921173960
total_commands_processed:14952159856
instantaneous_ops_per_sec:60515
total_net_input_bytes:1900864075154
total_net_output_bytes:14500081098022
instantaneous_input_kbps:5689.22
instantaneous_output_kbps:23770.42
rejected_connections:0
sync_full:2
sync_partial_ok:0
sync_partial_err:2
expired_keys:750124171
expired_stale_perc:13.81
expired_time_cap_reached_count:3171
expire_cycle_cpu_milliseconds:7834986
evicted_keys:0
keyspace_hits:12140242071
keyspace_misses:467352731
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:6862
total_forks:1
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:530043
dump_payload_sanitizations:0
total_reads_processed:10869372781
total_writes_processed:13007580135
instantaneous_lock_contention:0
avg_lock_contention:3.781250
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:2
slave0:ip=10.65.38.205,port=7000,state=online,offset=1285841055538,lag=0
slave1:ip=10.65.60.211,port=7000,state=online,offset=1285840495003,lag=0
master_failover_state:no-failover
master_replid:54033409533fc7127f630de77883bd0aceba9a52
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1285841430090
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:268435456
repl_backlog_first_byte_offset:1285572994635
repl_backlog_histlen:268435456

# CPU
used_cpu_sys:410178.050870
used_cpu_user:591817.050714
used_cpu_sys_children:0.033826
used_cpu_user_children:0.075178
server_threads:6
long_lock_waits:11777
used_cpu_sys_main_thread:53473.541689
used_cpu_user_main_thread:82091.385327

# Modules

# Commandstats
cmdstat_lrem:calls=221,usec=2890,usec_per_call=13.08,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=9923,usec=57845,usec_per_call=5.83,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=23199227,usec=40887810,usec_per_call=1.76,rejected_calls=0,failed_calls=0
cmdstat_cluster:calls=345431834,usec=10227006479,usec_per_call=29.61,rejected_calls=209443,failed_calls=0
cmdstat_hmset:calls=23226770,usec=310812343,usec_per_call=13.38,rejected_calls=119,failed_calls=0
cmdstat_config:calls=14255,usec=1018150,usec_per_call=71.42,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=2,usec=235,usec_per_call=117.50,rejected_calls=0,failed_calls=0
cmdstat_slowlog:calls=28510,usec=47868,usec_per_call=1.68,rejected_calls=0,failed_calls=0
cmdstat_set:calls=1592112444,usec=14634545551,usec_per_call=9.19,rejected_calls=222499,failed_calls=0
cmdstat_hset:calls=19590,usec=115125,usec_per_call=5.88,rejected_calls=13,failed_calls=0
cmdstat_hgetall:calls=169701464,usec=1791784949,usec_per_call=10.56,rejected_calls=0,failed_calls=0
cmdstat_replping:calls=2,usec=0,usec_per_call=0.00,rejected_calls=34,failed_calls=0
cmdstat_eval:calls=226525,usec=7508744,usec_per_call=33.15,rejected_calls=0,failed_calls=0
cmdstat_client:calls=14255,usec=30418,usec_per_call=2.13,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=994516,usec=2301629,usec_per_call=2.31,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=817899,usec=747140,usec_per_call=0.91,rejected_calls=0,failed_calls=0
cmdstat_del:calls=359369658,usec=2393904599,usec_per_call=6.66,rejected_calls=106,failed_calls=0
cmdstat_info:calls=75131,usec=4941407,usec_per_call=65.77,rejected_calls=0,failed_calls=0
cmdstat_lpush:calls=222,usec=1832,usec_per_call=8.25,rejected_calls=0,failed_calls=0
cmdstat_get:calls=12436888906,usec=28178701149,usec_per_call=2.27,rejected_calls=83574,failed_calls=0
cmdstat_latency:calls=28510,usec=98199,usec_per_call=3.44,rejected_calls=0,failed_calls=14255

# Errorstats
errorstat_CLUSTERDOWN:count=306311
errorstat_ERR:count=14255
errorstat_LOADING:count=209477

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=12666304,expires=12665000,avg_ttl=56124314,cached_keys=12666304

# KeyDB
mvcc_depth:0

and 2 hour later

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
395091:395123:M 10 Sep 2024 19:29:49.068 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
395091:395123:M 10 Sep 2024 19:29:49.068 # Accessing address: 0x544547010303
395091:395123:M 10 Sep 2024 19:29:49.068 # Crashed running the instruction at: 0x629265e56068

------ STACK TRACE ------
EIP:
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x629265e56068]

Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x742958642520]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xbb8) [0x629265e56068]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](+0x1d196a) [0x629265e4296a]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](readQueryFromClient(connection*)+0x39c) [0x629265e34d0c]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](connSocketEventHandler(aeEventLoop*, int, void*, int)+0x1e0) [0x629265f2a6e0]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](ProcessEventCore+0xf8) [0x629265e01338]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeProcessEvents+0x17d) [0x629265e053fd]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](aeMain+0x3e) [0x629265e05a6e]
/usr/bin/keydb-server 0.0.0.0:7000 [cluster](workerThreadMain(void*)+0xfe) [0x629265e1de9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x742958694ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x742958726850]

------ REGISTERS ------
395091:395123:M 10 Sep 2024 19:29:49.073 #
RAX:50fb3b8fc077d8cf RBX:0000742957ab7380
RCX:00000000e09e2d80 RDX:0000544547010303
RDI:000000000077d8cf RSI:0000000000000000
RBP:0000742673a9ab2b RSP:0000742938dfc6e0
R8 :3f8b86a80035cc71 R9 :000074269255487b
R10:0000000000000003 R11:000062926639c010
R12:000074268facd2f0 R13:0000000000060774
R14:000074269e8d4580 R15:000074268facd2f0
RIP:0000629265e56068 EFL:0000000000010206
CSGSFS:002b000000000033
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ef) -> 0000000000000000
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ee) -> 0000742900000000
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ed) -> 0000742938dfd978
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ec) -> 00000000b6b66930
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6eb) -> 000062926635bbf9
395091:395123:M 10 Sep 2024 19:29:49.073 # (0000742938dfc6ea) -> 000000000000b000
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e9) -> 0000742938dfc838
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e8) -> 000074250000001f
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e7) -> 0000629265e1693c
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e6) -> 0000000000000000
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e5) -> 0000000000000001
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e4) -> 00006292664dd920
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e3) -> 00007425ad438a80
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e2) -> 00007425ac7733c0
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e1) -> 00006292664dbc00
395091:395123:M 10 Sep 2024 19:29:49.074 # (0000742938dfc6e0) -> 0000742938dfdb38

Aditional information

  1. OS distribution and version
root@keydb2:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy
root@keydb2:~# uname -a
Linux keydb2 6.5.0-27-generic #28~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 15 10:51:06 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

We have no idea how to reproduce.
Long time it was working well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@broken-ufa