-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature](cloud) Implement hdfs accessor #33059
Conversation
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
@@ -24,6 +24,7 @@ | |||
#include <unordered_map> | |||
#include <unordered_set> | |||
|
|||
#include "recycler/s3_accessor.h" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: 'recycler/s3_accessor.h' file not found [clang-diagnostic-error]
#include "recycler/s3_accessor.h"
^
// specific language governing permissions and limitations | ||
// under the License. | ||
|
||
#include "recycler/hdfs_accessor.h" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: 'recycler/hdfs_accessor.h' file not found [clang-diagnostic-error]
#include "recycler/hdfs_accessor.h"
^
|
||
class HDFSBuilder { | ||
public: | ||
~HDFSBuilder() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
~HDFSBuilder() {
^
HDFSBuilder() = default; | ||
|
||
// returns 0 for success otherwise error | ||
int init(const HdfsBuildConf& conf) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: method 'init' can be made static [readability-convert-member-functions-to-static]
int init(const HdfsBuildConf& conf) { | |
static int init(const HdfsBuildConf& conf) { |
} | ||
|
||
// returns 0 for success otherwise error | ||
int init_hdfs_builder() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: method 'init_hdfs_builder' can be made static [readability-convert-member-functions-to-static]
int init_hdfs_builder() { | |
static int init_hdfs_builder() { |
} | ||
|
||
// returns 0 for success otherwise error | ||
int check_krb_params(std::string_view hdfs_kerberos_principal, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: method 'check_krb_params' can be made static [readability-convert-member-functions-to-static]
int check_krb_params(std::string_view hdfs_kerberos_principal, | |
static int check_krb_params(std::string_view hdfs_kerberos_principal, |
#ifdef USE_HADOOP_HDFS | ||
#include <hadoop_hdfs/hdfs.h> // IWYU pragma: export | ||
#else | ||
#include <hdfs/hdfs.h> // IWYU pragma: export |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: 'hdfs/hdfs.h' file not found [clang-diagnostic-error]
#include <hdfs/hdfs.h> // IWYU pragma: export
^
#include <string> | ||
#include <vector> | ||
|
||
#include "recycler/obj_store_accessor.h" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: 'recycler/obj_store_accessor.h' file not found [clang-diagnostic-error]
#include "recycler/obj_store_accessor.h"
^
5088dc9
to
2e53a91
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
// specific language governing permissions and limitations | ||
// under the License. | ||
|
||
#include "recycler/hdfs_accessor.h" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: 'recycler/hdfs_accessor.h' file not found [clang-diagnostic-error]
#include "recycler/hdfs_accessor.h"
^
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
namespace doris::cloud { | ||
|
||
#define HdfsAccessorTest DISABLED_HdfsAccessorTest | ||
TEST(HdfsAccessorTest, normal) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
TEST(HdfsAccessorTest, normal) {
^
Additional context
cloud/test/hdfs_accessor_test.cpp:47: 92 lines including whitespace and comments (threshold 80)
TEST(HdfsAccessorTest, normal) {
^
|
run buildall |
run compile |
cbfd416
to
c72c61a
Compare
run buildall |
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 38983 ms
|
TPC-DS: Total hot run time: 181489 ms
|
ClickBench: Total hot run time: 31.79 s
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Proposed changes
Implement hdfs accessor
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...