-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[opt](log) refactor the log dir config #32933
Conversation
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
clang-tidy review says "All clean, LGTM! 👍" |
clang-tidy review says "All clean, LGTM! 👍" |
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 38253 ms
|
TPC-DS: Total hot run time: 182118 ms
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
|
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 38399 ms
|
TPC-DS: Total hot run time: 182219 ms
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
|
TeamCity be ut coverage result: |
run buildall |
//TODO: when dump, now could append records but can't add new query. try use better grained locks. | ||
std::unique_lock<std::mutex> l(_data_lock); // can't rehash | ||
if (_dump_type == RecordType::PerQuery) { | ||
auto path = _dir / fmt::format("query{}", to_string(query_id)); | ||
auto path = log_dir / fmt::format("query{}", to_string(query_id)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lost tracing
dir
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
TPC-H: Total hot run time: 38394 ms
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 181654 ms
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
|
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 38132 ms
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 180961 ms
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
|
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
@@ -52,6 +52,7 @@ struct ScheduleRecord { | |||
using OneQueryTraces = moodycamel::ConcurrentQueue<ScheduleRecord>; | |||
|
|||
// belongs to exec_env, for all query, if enable | |||
// curl http://{host}:{web_server_port}/api/running_pipeline_tasks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PipelineTracerContext
has nothing to do with running_pipeline_tasks. It's a experimental function WIP
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I will remove it in next PR
Refactor the config for log dir of FE and BE TLDR: - Use env variable `LOG_DIR` to set root log dir - Remove `sys_log_dir` for FE and BE Details: 1. FE 1. The root log dir is set by env variable `LOG_DIR` in `fe.conf` 2. The default value of `audit_log_dir` is same as `${LOG_DIR}/` 3. The default value of `spark_launcher_log_dir` is `${LOG_DIR}/spark_launcher_log` 4. The default value of `nereids_trace_log_dir` is `${LOG_DIR}/nereids_trace_log` 5. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir. 2. BE 1. The root log dir is set by env variable `LOG_DIR` in `be.conf` 2. Remove `pipeline_tracing_log_dir`, use `${LOG_DIR}` directly. 3. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir.
Refactor the config for log dir of FE and BE TLDR: - Use env variable `LOG_DIR` to set root log dir - Remove `sys_log_dir` for FE and BE Details: 1. FE 1. The root log dir is set by env variable `LOG_DIR` in `fe.conf` 2. The default value of `audit_log_dir` is same as `${LOG_DIR}/` 3. The default value of `spark_launcher_log_dir` is `${LOG_DIR}/spark_launcher_log` 4. The default value of `nereids_trace_log_dir` is `${LOG_DIR}/nereids_trace_log` 5. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir. 2. BE 1. The root log dir is set by env variable `LOG_DIR` in `be.conf` 2. Remove `pipeline_tracing_log_dir`, use `${LOG_DIR}` directly. 3. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir.
Refactor the config for log dir of FE and BE TLDR: - Use env variable `LOG_DIR` to set root log dir - Remove `sys_log_dir` for FE and BE Details: 1. FE 1. The root log dir is set by env variable `LOG_DIR` in `fe.conf` 2. The default value of `audit_log_dir` is same as `${LOG_DIR}/` 3. The default value of `spark_launcher_log_dir` is `${LOG_DIR}/spark_launcher_log` 4. The default value of `nereids_trace_log_dir` is `${LOG_DIR}/nereids_trace_log` 5. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir. 2. BE 1. The root log dir is set by env variable `LOG_DIR` in `be.conf` 2. Remove `pipeline_tracing_log_dir`, use `${LOG_DIR}` directly. 3. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir.
Refactor the config for log dir of FE and BE TLDR: - Use env variable `LOG_DIR` to set root log dir - Remove `sys_log_dir` for FE and BE Details: 1. FE 1. The root log dir is set by env variable `LOG_DIR` in `fe.conf` 2. The default value of `audit_log_dir` is same as `${LOG_DIR}/` 3. The default value of `spark_launcher_log_dir` is `${LOG_DIR}/spark_launcher_log` 4. The default value of `nereids_trace_log_dir` is `${LOG_DIR}/nereids_trace_log` 5. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir. 2. BE 1. The root log dir is set by env variable `LOG_DIR` in `be.conf` 2. Remove `pipeline_tracing_log_dir`, use `${LOG_DIR}` directly. 3. The origin `sys_log_dir` is deprecated, and default value is `""`. But for compatibility, if user already set `sys_log_dir` before, Doris will still use it as root log dir.
Proposed changes
Refactor the config for log dir of FE and BE
TLDR:
LOG_DIR
to set root log dirsys_log_dir
for FE and BEDetails:
FE
LOG_DIR
infe.conf
audit_log_dir
is same as${LOG_DIR}/
spark_launcher_log_dir
is${LOG_DIR}/spark_launcher_log
nereids_trace_log_dir
is${LOG_DIR}/nereids_trace_log
sys_log_dir
is deprecated, and default value is""
.But for compatibility, if user already set
sys_log_dir
before, Doris will still use it as root log dir.BE
LOG_DIR
inbe.conf
pipeline_tracing_log_dir
, use${LOG_DIR}
directly.sys_log_dir
is deprecated, and default value is""
.But for compatibility, if user already set
sys_log_dir
before, Doris will still use it as root log dir.Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...