[enhancement](cloud) clarify codes and make TTL expiration work after abnormal cache type transition #40226
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
freemandealer
changed the title
| RETURN_IF_ERROR(fs->open_file(file, &file_reader)); | ||
| Status s = fs->open_file(file, &file_reader); | ||
| if (!s.ok()) { | ||
| if (key.meta.type == FileCacheType::TTL) { |
There was a problem hiding this comment.
return first to reduce indention.
| if (key.meta.type == FileCacheType::TTL) { | |
| if (key.meta.type != FileCacheType::TTL) return s; |
|
|
||
| Status read_file_cache_version(std::string* buffer) const; | ||
|
|
||
| Status parse_filename_suffix_to_cache_type(const std::shared_ptr<LocalFileSystem>& fs, |
There was a problem hiding this comment.
Add UT for this function
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -227,7 +286,7 @@ std::string FSFileCacheStorage::get_path_in_local_cache(const UInt128Wrapper& va | |||
| } | |||
| } | |||
|
|
|||
| Status FSFileCacheStorage::rebuild_data_structure() const { | |||
| Status FSFileCacheStorage::upgrade_cache_dir_if_necessary() const { | |||
There was a problem hiding this comment.
warning: function 'upgrade_cache_dir_if_necessary' has cognitive complexity of 119 (threshold 50) [readability-function-cognitive-complexity]
Status FSFileCacheStorage::upgrade_cache_dir_if_necessary() const {
^Additional context
be/src/io/cache/fs_file_cache_storage.cpp:292: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(read_file_cache_version(&version));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:292: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(read_file_cache_version(&version));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:293: +1, including nesting penalty of 0, nesting level increased to 1
if (USE_CACHE_VERSION2 && version != "2.0") {
^be/src/io/cache/fs_file_cache_storage.cpp:293: +1
if (USE_CACHE_VERSION2 && version != "2.0") {
^be/src/io/cache/fs_file_cache_storage.cpp:297: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:301: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_it != std::filesystem::directory_iterator(); ++key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:302: +3, including nesting penalty of 2, nesting level increased to 3
if (key_it->is_directory()) {
^be/src/io/cache/fs_file_cache_storage.cpp:304: +4, including nesting penalty of 3, nesting level increased to 4
if (cache_key.size() > KEY_PREFIX_LENGTH) {
^be/src/io/cache/fs_file_cache_storage.cpp:308: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->exists(key_prefix, &exists));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:308: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(fs->exists(key_prefix, &exists));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:309: +5, including nesting penalty of 4, nesting level increased to 5
if (!exists) {
^be/src/io/cache/fs_file_cache_storage.cpp:310: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(fs->create_directory(key_prefix));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:310: +7, including nesting penalty of 6, nesting level increased to 7
RETURN_IF_ERROR(fs->create_directory(key_prefix));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:312: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->rename(key_it->path(), key_prefix / cache_key));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:312: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(fs->rename(key_it->path(), key_prefix / cache_key));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:316: +2, including nesting penalty of 1, nesting level increased to 2
if (!write_file_cache_version().ok()) {
^be/src/io/cache/fs_file_cache_storage.cpp:321: nesting level increased to 1
auto rebuild_dir = [&](std::filesystem::directory_iterator& upgrade_key_it) -> Status {
^be/src/io/cache/fs_file_cache_storage.cpp:322: +2, including nesting penalty of 1, nesting level increased to 2
for (; upgrade_key_it != std::filesystem::directory_iterator(); ++upgrade_key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:323: +3, including nesting penalty of 2, nesting level increased to 3
if (upgrade_key_it->path().filename().native().find('_') == std::string::npos) {
^be/src/io/cache/fs_file_cache_storage.cpp:324: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(fs->delete_directory(upgrade_key_it->path().native() + "_0"));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:324: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->delete_directory(upgrade_key_it->path().native() + "_0"));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:325: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:325: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:332: +1, including nesting penalty of 0, nesting level increased to 1
if constexpr (USE_CACHE_VERSION2) {
^be/src/io/cache/fs_file_cache_storage.cpp:334: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:338: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_prefix_it != std::filesystem::directory_iterator(); ++key_prefix_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:339: +3, including nesting penalty of 2, nesting level increased to 3
if (!key_prefix_it->is_directory()) {
^be/src/io/cache/fs_file_cache_storage.cpp:343: +3, including nesting penalty of 2, nesting level increased to 3
if (key_prefix_it->path().filename().native().size() != KEY_PREFIX_LENGTH) {
^be/src/io/cache/fs_file_cache_storage.cpp:346: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(fs->delete_directory(key_prefix_it->path()));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:346: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->delete_directory(key_prefix_it->path()));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:349: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:352: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:352: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:354: +1, nesting level increased to 1
} else {
^be/src/io/cache/fs_file_cache_storage.cpp:356: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:359: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:359: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^| } | ||
| return Status::OK(); | ||
| } | ||
|
|
||
| void FSFileCacheStorage::load_cache_info_into_memory(BlockFileCache* _mgr) const { |
There was a problem hiding this comment.
warning: function 'load_cache_info_into_memory' has cognitive complexity of 53 (threshold 50) [readability-function-cognitive-complexity]
void FSFileCacheStorage::load_cache_info_into_memory(BlockFileCache* _mgr) const {
^Additional context
be/src/io/cache/fs_file_cache_storage.cpp:469: nesting level increased to 1
auto add_cell_batch_func = [&]() {
^be/src/io/cache/fs_file_cache_storage.cpp:471: nesting level increased to 2
auto f = [&](const BatchLoadArgs& args) {
^be/src/io/cache/fs_file_cache_storage.cpp:473: +3, including nesting penalty of 2, nesting level increased to 3
if (_mgr->_files.contains(args.hash) && _mgr->_files[args.hash].contains(args.offset)) {
^be/src/io/cache/fs_file_cache_storage.cpp:473: +1
if (_mgr->_files.contains(args.hash) && _mgr->_files[args.hash].contains(args.offset)) {
^be/src/io/cache/fs_file_cache_storage.cpp:475: +4, including nesting penalty of 3, nesting level increased to 4
if (file_block->expiration_time() != args.ctx.expiration_time ||
^be/src/io/cache/fs_file_cache_storage.cpp:475: +1
if (file_block->expiration_time() != args.ctx.expiration_time ||
^be/src/io/cache/fs_file_cache_storage.cpp:489: +3, including nesting penalty of 2, nesting level increased to 3
if (!args.is_tmp) {
^be/src/io/cache/fs_file_cache_storage.cpp:496: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:504: nesting level increased to 1
auto scan_file_cache = [&](std::filesystem::directory_iterator& key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:506: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_it != std::filesystem::directory_iterator(); ++key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:515: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:524: +3, including nesting penalty of 2, nesting level increased to 3
for (; offset_it != std::filesystem::directory_iterator(); ++offset_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:529: +4, including nesting penalty of 3, nesting level increased to 4
if (!parse_filename_suffix_to_cache_type(fs, offset_it->path().filename().native(),
^be/src/io/cache/fs_file_cache_storage.cpp:546: +4, including nesting penalty of 3, nesting level increased to 4
if (batch_load_buffer.size() >= scan_length) {
^be/src/io/cache/fs_file_cache_storage.cpp:554: +1, including nesting penalty of 0, nesting level increased to 1
if constexpr (USE_CACHE_VERSION2) {
^be/src/io/cache/fs_file_cache_storage.cpp:556: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:560: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_prefix_it != std::filesystem::directory_iterator(); ++key_prefix_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:561: +3, including nesting penalty of 2, nesting level increased to 3
if (!key_prefix_it->is_directory()) {
^be/src/io/cache/fs_file_cache_storage.cpp:565: +3, including nesting penalty of 2, nesting level increased to 3
if (key_prefix_it->path().filename().native().size() != KEY_PREFIX_LENGTH) {
^be/src/io/cache/fs_file_cache_storage.cpp:570: +4, including nesting penalty of 3, nesting level increased to 4
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:577: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:583: +1, nesting level increased to 1
} else {
^be/src/io/cache/fs_file_cache_storage.cpp:585: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:591: +1, including nesting penalty of 0, nesting level increased to 1
if (!batch_load_buffer.empty()) {
^
| } | ||
| return Status::OK(); | ||
| } | ||
|
|
||
| void FSFileCacheStorage::load_cache_info_into_memory(BlockFileCache* _mgr) const { |
There was a problem hiding this comment.
warning: function 'load_cache_info_into_memory' has cognitive complexity of 53 (threshold 50) [readability-function-cognitive-complexity]
void FSFileCacheStorage::load_cache_info_into_memory(BlockFileCache* _mgr) const {
^Additional context
be/src/io/cache/fs_file_cache_storage.cpp:469: nesting level increased to 1
auto add_cell_batch_func = [&]() {
^be/src/io/cache/fs_file_cache_storage.cpp:471: nesting level increased to 2
auto f = [&](const BatchLoadArgs& args) {
^be/src/io/cache/fs_file_cache_storage.cpp:473: +3, including nesting penalty of 2, nesting level increased to 3
if (_mgr->_files.contains(args.hash) && _mgr->_files[args.hash].contains(args.offset)) {
^be/src/io/cache/fs_file_cache_storage.cpp:473: +1
if (_mgr->_files.contains(args.hash) && _mgr->_files[args.hash].contains(args.offset)) {
^be/src/io/cache/fs_file_cache_storage.cpp:475: +4, including nesting penalty of 3, nesting level increased to 4
if (file_block->expiration_time() != args.ctx.expiration_time ||
^be/src/io/cache/fs_file_cache_storage.cpp:475: +1
if (file_block->expiration_time() != args.ctx.expiration_time ||
^be/src/io/cache/fs_file_cache_storage.cpp:489: +3, including nesting penalty of 2, nesting level increased to 3
if (!args.is_tmp) {
^be/src/io/cache/fs_file_cache_storage.cpp:496: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:504: nesting level increased to 1
auto scan_file_cache = [&](std::filesystem::directory_iterator& key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:506: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_it != std::filesystem::directory_iterator(); ++key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:515: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:524: +3, including nesting penalty of 2, nesting level increased to 3
for (; offset_it != std::filesystem::directory_iterator(); ++offset_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:529: +4, including nesting penalty of 3, nesting level increased to 4
if (!parse_filename_suffix_to_cache_type(fs, offset_it->path().filename().native(),
^be/src/io/cache/fs_file_cache_storage.cpp:546: +4, including nesting penalty of 3, nesting level increased to 4
if (batch_load_buffer.size() >= scan_length) {
^be/src/io/cache/fs_file_cache_storage.cpp:554: +1, including nesting penalty of 0, nesting level increased to 1
if constexpr (USE_CACHE_VERSION2) {
^be/src/io/cache/fs_file_cache_storage.cpp:557: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:561: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_prefix_it != std::filesystem::directory_iterator(); ++key_prefix_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:562: +3, including nesting penalty of 2, nesting level increased to 3
if (!key_prefix_it->is_directory()) {
^be/src/io/cache/fs_file_cache_storage.cpp:566: +3, including nesting penalty of 2, nesting level increased to 3
if (key_prefix_it->path().filename().native().size() != KEY_PREFIX_LENGTH) {
^be/src/io/cache/fs_file_cache_storage.cpp:571: +4, including nesting penalty of 3, nesting level increased to 4
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:578: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:584: +1, nesting level increased to 1
} else {
^be/src/io/cache/fs_file_cache_storage.cpp:586: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:592: +1, including nesting penalty of 0, nesting level increased to 1
if (!batch_load_buffer.empty()) {
^| fs::remove_all(cache_base_path); | ||
| } | ||
| fs::create_directories(cache_base_path); | ||
| auto sp = SyncPoint::get_instance(); |
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
| auto sp = SyncPoint::get_instance(); | |
| h);auto * |
|
run buildall |
| @@ -227,7 +289,7 @@ std::string FSFileCacheStorage::get_path_in_local_cache(const UInt128Wrapper& va | |||
| } | |||
| } | |||
|
|
|||
| Status FSFileCacheStorage::rebuild_data_structure() const { | |||
| Status FSFileCacheStorage::upgrade_cache_dir_if_necessary() const { | |||
There was a problem hiding this comment.
warning: function 'upgrade_cache_dir_if_necessary' has cognitive complexity of 119 (threshold 50) [readability-function-cognitive-complexity]
Status FSFileCacheStorage::upgrade_cache_dir_if_necessary() const {
^Additional context
be/src/io/cache/fs_file_cache_storage.cpp:295: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(read_file_cache_version(&version));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:295: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(read_file_cache_version(&version));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:296: +1, including nesting penalty of 0, nesting level increased to 1
if (USE_CACHE_VERSION2 && version != "2.0") {
^be/src/io/cache/fs_file_cache_storage.cpp:296: +1
if (USE_CACHE_VERSION2 && version != "2.0") {
^be/src/io/cache/fs_file_cache_storage.cpp:300: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) {
^be/src/io/cache/fs_file_cache_storage.cpp:304: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_it != std::filesystem::directory_iterator(); ++key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:305: +3, including nesting penalty of 2, nesting level increased to 3
if (key_it->is_directory()) {
^be/src/io/cache/fs_file_cache_storage.cpp:307: +4, including nesting penalty of 3, nesting level increased to 4
if (cache_key.size() > KEY_PREFIX_LENGTH) {
^be/src/io/cache/fs_file_cache_storage.cpp:311: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->exists(key_prefix, &exists));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:311: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(fs->exists(key_prefix, &exists));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:312: +5, including nesting penalty of 4, nesting level increased to 5
if (!exists) {
^be/src/io/cache/fs_file_cache_storage.cpp:313: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(fs->create_directory(key_prefix));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:313: +7, including nesting penalty of 6, nesting level increased to 7
RETURN_IF_ERROR(fs->create_directory(key_prefix));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:315: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->rename(key_it->path(), key_prefix / cache_key));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:315: +6, including nesting penalty of 5, nesting level increased to 6
RETURN_IF_ERROR(fs->rename(key_it->path(), key_prefix / cache_key));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:319: +2, including nesting penalty of 1, nesting level increased to 2
if (!write_file_cache_version().ok()) {
^be/src/io/cache/fs_file_cache_storage.cpp:324: nesting level increased to 1
auto rebuild_dir = [&](std::filesystem::directory_iterator& upgrade_key_it) -> Status {
^be/src/io/cache/fs_file_cache_storage.cpp:325: +2, including nesting penalty of 1, nesting level increased to 2
for (; upgrade_key_it != std::filesystem::directory_iterator(); ++upgrade_key_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:326: +3, including nesting penalty of 2, nesting level increased to 3
if (upgrade_key_it->path().filename().native().find('_') == std::string::npos) {
^be/src/io/cache/fs_file_cache_storage.cpp:327: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(fs->delete_directory(upgrade_key_it->path().native() + "_0"));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:327: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->delete_directory(upgrade_key_it->path().native() + "_0"));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:328: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:328: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:335: +1, including nesting penalty of 0, nesting level increased to 1
if constexpr (USE_CACHE_VERSION2) {
^be/src/io/cache/fs_file_cache_storage.cpp:337: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:341: +2, including nesting penalty of 1, nesting level increased to 2
for (; key_prefix_it != std::filesystem::directory_iterator(); ++key_prefix_it) {
^be/src/io/cache/fs_file_cache_storage.cpp:342: +3, including nesting penalty of 2, nesting level increased to 3
if (!key_prefix_it->is_directory()) {
^be/src/io/cache/fs_file_cache_storage.cpp:346: +3, including nesting penalty of 2, nesting level increased to 3
if (key_prefix_it->path().filename().native().size() != KEY_PREFIX_LENGTH) {
^be/src/io/cache/fs_file_cache_storage.cpp:349: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(fs->delete_directory(key_prefix_it->path()));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:349: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(fs->delete_directory(key_prefix_it->path()));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:352: +3, including nesting penalty of 2, nesting level increased to 3
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:355: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:355: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/io/cache/fs_file_cache_storage.cpp:357: +1, nesting level increased to 1
} else {
^be/src/io/cache/fs_file_cache_storage.cpp:359: +2, including nesting penalty of 1, nesting level increased to 2
if (ec) [[unlikely]] {
^be/src/io/cache/fs_file_cache_storage.cpp:362: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:628: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/io/cache/fs_file_cache_storage.cpp:362: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(rebuild_dir(key_it));
^be/src/common/status.h:630: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^| @@ -4644,5 +4642,105 @@ TEST_F(BlockFileCacheTest, load_cache2) { | |||
| key1.to_string() + "_0/" + std::to_string(offset)); | |||
| } | |||
| } | |||
| */ | |||
|
|
|||
| TEST_F(BlockFileCacheTest, test_load) { | |||
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
*/
^Additional context
be/test/io/cache/block_file_cache_test.cpp:4646: 97 lines including whitespace and comments (threshold 80)
*/
^| fs::remove_all(cache_base_path); | ||
| } | ||
| fs::create_directories(cache_base_path); | ||
| auto sp = SyncPoint::get_instance(); |
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
| auto sp = SyncPoint::get_instance(); | |
| h);auto * |
TPC-H: Total hot run time: 38217 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 193318 ms |
ClickBench: Total hot run time: 32.05 s |
freemandealer
changed the title
github-actions
bot
added
the
approved
|
run buildall |
TPC-H: Total hot run time: 38186 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 193090 ms |
ClickBench: Total hot run time: 31.35 s |
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
2. read&delete retry with old format if new format failed 3. write only new format 4. transition between TTL and Others won't rename the file suffix Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
dataroaring
force-pushed
the
fix-cache-type-transition
branch
from
73c8516
to
d240c52
Compare
|
run buildall |
TPC-H: Total hot run time: 38466 ms |
TPC-DS: Total hot run time: 191794 ms |
|
TeamCity be ut coverage result: |
ClickBench: Total hot run time: 31.57 s |
|
run cloud_p0 |
… abnormal cache type transition (#40226) current TTL embeds the expiration time and type into filename and path. Maintaining both is buggy for lack of atomicity. I simplify this by using only expiration time to infer the type so that we need only expiration time. Signed-off-by: freemandealer <[email protected]>
current TTL embeds the expiration time and type into filename and path. Maintaining both is buggy for lack of atomicity. I simplify this by using only expiration time to infer the type so that we need only expiration time.
Proposed changes
Issue Number: close #xxx