Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Subtask] Accessing HDFS from outside the graviton-ci-hive Docker #396

Closed
Tracked by #244
xunliu opened this issue Sep 14, 2023 · 2 comments · Fixed by #398
Closed
Tracked by #244

[Subtask] Accessing HDFS from outside the graviton-ci-hive Docker #396

xunliu opened this issue Sep 14, 2023 · 2 comments · Fixed by #398
Assignees
@xunliu
Milestone
Graviton v0.2.0

Comments

@xunliu
Copy link
Member

xunliu commented Sep 14, 2023 •
edited
Loading

Describe the subtask

Currently, HDFS uses localhost in the graviton-ci-hive Docker. localhost is a loopback IP address in the LINUX,
The application (Trino) can't use this IP to access HDFS in the graviton-ci-hive Docker container.

We need to change these configurations:

  • Modify fs.defaultFS to hdfs://0.0.0.0:9000 in the /usr/local/hadoop/etc/hadoop/core-site.xml file.
  • Export 9000 port in the Dockerfile, Trino needs to access HDFS.

Optimized

  • Remove 7180 port, It's an unused port. It's look like Cloudera manager port
  • Reorder port list in the Dockerfile, Docker run command, and docs.
  • Removed special tag name for docker run command, automatically using latest.

Parent issue

#244
@xunliu xunliu added this to the Graviton v0.2.0 milestone Sep 14, 2023
@xunliu xunliu self-assigned this Sep 14, 2023
@xunliu xunliu mentioned this issue Sep 13, 2023
Closed
19 tasks
@xunliu
Copy link
Member Author

xunliu commented Sep 14, 2023

@diqiu50 I created this issue to fix the problem with Trino not being able to connect HDFS.
Please help me review the description of this issue. Thanks.

@diqiu50
Copy link
Contributor

diqiu50 commented Sep 14, 2023

i think that's ok.

@xunliu xunliu changed the title [Subtask] Accessing HDFS from outside the graviton-ci-hive Docker container [Subtask] Accessing HDFS from outside the graviton-ci-hive Docker Sep 14, 2023
@xunliu xunliu mentioned this issue Sep 14, 2023
Merged
jerryshao pushed a commit that referenced this issue Sep 18, 2023
@xunliu
[#396] feat(CI): Accessing HDFS from outside the graviton-ci-hive Doc…
55aa215 
…ker (#398)

### What changes were proposed in this pull request?

We need to change these configurations:
+ Modify `fs.defaultFS` to `hdfs://0.0.0.0:9000` in the
`/usr/local/hadoop/etc/hadoop/core-site.xml` file.
+ Export 9000 port in the Dockerfile, Trino needs to access HDFS.

#### Optimized
+ Remove `7180` port, It's an unused port. It's look like [Cloudera
manager
port](https://community.cloudera.com/t5/Support-Questions/Localhost-7180-Connection-Refused-Cloudera-Manager/td-p/55285)
+ Reorder port list in the Dockerfile, Docker run command, and docs.
+ Removed special tag name for docker run command, automatically using
`latest`.

### Why are the changes needed?

Currently, HDFS uses `localhost` in the graviton-ci-hive Docker.
`localhost` is a loopback IP address in the LINUX,
The application (Trino) can't use this IP to access HDFS in the
graviton-ci-hive Docker container.

Fix: #396 

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

CI passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xunliu xunliu

Labels
None yet
Projects
None yet
Milestone
Graviton v0.2.0
Development

Successfully merging a pull request may close this issue.

[#396] feat(CI): Accessing HDFS from outside the graviton-ci-hive Docker
2 participants
@xunliu @diqiu50