Question about cannot cache any object

[joker-gege]

I use ats 9.2.3 as reverse proxy in Centos 7.3，It work normal before, suddenly it cannot cache any objects. when restart ats，it return to normal. I find debug message HttpCacheSM::state_cache_open_write, CACHE_EVENT_OPEN_WRITE_FAILED/TS_EVENT_CACHE_OPEN_WRITE_FAILED in diags.log.
Has anyone encountered this problem before? What caused this problem?

The diags.log information is as follows：
[Sep 23 18:32:38.940] [ET_NET 38] DEBUG: <HttpTransact.cc:7779 (build_request)> (http_trans) [15775895] [build_request] request like cacheable and conditional headers removed
[Sep 23 18:32:38.940] [ET_NET 38] DEBUG: <HttpTransact.cc:7809 (build_request)> (http_trans) [15775895] [build_request] request_sent_time: 1727087558
[Sep 23 18:32:38.940] [ET_NET 38] DEBUG: <HttpSM.cc:7415 (call_transact_and_set_next_state)> (http) [15775895] State Transition: SM_ACTION_DNS_LOOKUP -> SM_ACTION_CACHE_ISSUE_WRITE
[Sep 23 18:32:38.940] [ET_NET 38] DEBUG: <HttpSM.cc:4843 (do_cache_prepare_action)> (http_cache_write) [15775895] writing to cache with URL tslive/c56_fast_1014_720P_t10/66f140c6.ts
[Sep 23 18:32:38.940] [ET_NET 38] DEBUG: <HttpCacheSM.cc:165 (state_cache_open_write)> (http_cache) [15775895] [&HttpCacheSM::state_cache_open_write, CACHE_EVENT_OPEN_WRITE_FAILED/TS_EVENT_CACHE_OPEN_WRITE_FAILED]
[Sep 23 18:32:38.988] [ET_NET 38] DEBUG: <HttpCacheSM.cc:165 (state_cache_open_write)> (http_cache) [15775895] [&HttpCacheSM::state_cache_open_write, EVENT_INTERVAL]
[Sep 23 18:32:38.988] [ET_NET 38] DEBUG: <HttpCacheSM.cc:234 (state_cache_open_write)> (http_cache) [15775895] [state_cache_open_write] cache open write failure 1. retrying cache open write...
[Sep 23 18:32:38.988] [ET_NET 38] DEBUG: <HttpSM.cc:2717 (main_handler)> (http) [15775895] [HttpSM::main_handler, CACHE_EVENT_OPEN_WRITE_FAILED/TS_EVENT_CACHE_OPEN_WRITE_FAILED]
[Sep 23 18:32:38.988] [ET_NET 38] DEBUG: <HttpSM.cc:2524 (state_cache_open_write)> (http) [15775895] [&HttpSM : state_cache_open_write, CACHE_EVENT_OPEN_WRITE_FAILED/TS_EVENT_CACHE_OPEN_WRITE_FAILED]
[Sep 23 18:32:38.988] [ET_NET 38] DEBUG: <HttpTransact.cc:3240 (handle_cache_write_lock)> (http_trans) Next action next; nullptr
[Sep 23 18:32:38.988] [ET_NET 38] DEBUG: <HttpSM.cc:7415 (call_transact_and_set_next_state)> (http) [15775895] State Transition: SM_ACTION_CACHE_ISSUE_WRITE -> SM_ACTION_ORIGIN_SERVER_OPEN

[cmcfarlen]

Do all writes to cache fail in this condition? Also, is this reproducible or only happened once? Are a lot of clients requesting this object at once?

[joker-gege]

Yes，all writes to cache failed in this condition
I cannot reproduce this issue，when restart ats，it return normal
When this problem first occurred，a lot of clients requesting the same object at once

Thank you very much for your help
You need more details, I am happy to provide them，For example, plugins, configurations, and business scenarios

[joker-gege]

I have discovered some new clues in the diag. log, cache status is ECACHE_NOT_READY, I understand that it only happens when only ATS detects that all disks are in error. If there is an error in disk detection and the disk drops, will it continue to detect the disk in the future? If the disk is good, will it continue to load up?

[Sep 24 13:48:38.558] [ET_NET 14] DEBUG: <HttpSM.cc:7415 (call_transact_and_set_next_state)> (http) [15784295] State Transition: SM_ACTION_API_POST_REMAP -> SM_ACTION_CACHE_LOOKUP
[Sep 24 13:48:38.558] [ET_NET 14] DEBUG: <HttpSM.cc:4739 (do_cache_lookup_and_read)> (http_seq) [HttpSM::do_cache_lookup_and_read] [15784295] Issuing cache lookup for URL tslive/c56_fast_1014_720P_t10/66f140c6.ts
[Sep 24 13:48:38.558] [ET_NET 14] DEBUG: <HttpCacheSM.cc:100 (state_cache_open_read)> (http_cache) [15784295] [&HttpCacheSM::state_cache_open_read, CACHE_EVENT_OPEN_READ_FAILED/TS_EVENT_CACHE_OPEN_READ_FAILED]
[Sep 24 13:48:38.558] [ET_NET 14] DEBUG: <HttpSM.cc:2717 (main_handler)> (http) [15784295] [HttpSM::main_handler, CACHE_EVENT_OPEN_READ_FAILED/TS_EVENT_CACHE_OPEN_READ_FAILED]
[Sep 24 13:48:38.558] [ET_NET 14] DEBUG: <HttpSM.cc:2645 (state_cache_open_read)> (http) [15784295] [&HttpSM::state_cache_open_read, CACHE_EVENT_OPEN_READ_FAILED/TS_EVENT_CACHE_OPEN_READ_FAILED]
[Sep 24 13:48:38.558] [ET_NET 14] DEBUG: <HttpSM.cc:2680 (state_cache_open_read)> (http) [15784295] cache_open_read - CACHE_EVENT_OPEN_READ_FAILED with ECACHE_NOT_READY (20407)