[#396] feat(CI): Accessing HDFS from outside the graviton-ci-hive Docker

.github/workflows/docker-image.yml

@@ -24,10 +24,10 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
-      - name: Check release token
+      - name: Check publish Docker token
         run: |
           if [[ "${secrets_token}" != "${input_token}" ]]; then
-            echo "The token you entered is incorrect. Please try again."
+            echo "You have entered an incorrect token. Please re-enter it."
             exit 1
           fi
 

.github/workflows/integration-test.yml

@@ -10,7 +10,7 @@ on:
 
 env:
   HIVE_IMAGE_NAME: datastrato/graviton-ci-hive
-  HIVE_IMAGE_TAG_NAME: 0.1.1
+  HIVE_IMAGE_TAG_NAME: 0.1.2
 
 concurrency:
   group: ${{ github.worklfow }}-${{ github.event.pull_request.number || github.ref }}
@@ -47,7 +47,7 @@ jobs:
 
       - name: Run AMD64 container
         run: |
-          docker run --rm --name ${DOCKER_RUN_NAME} --platform ${PLATFORM} -d -p 8088:8088 -p 50070:50070 -p 50075:50075 -p 10000:10000 -p 10002:10002 -p 8888:8888 -p 9083:9083 -p 8022:22 ${HIVE_IMAGE_NAME}:${HIVE_IMAGE_TAG_NAME}
+          docker run --rm --name ${DOCKER_RUN_NAME} --platform ${PLATFORM} -d -p 8022:22 -p 8088:8088 -p 9000:9000 -p 9083:9083 -p 10000:10000 -p 10002:10002 -p 50070:50070 -p 50075:50075 ${HIVE_IMAGE_NAME}:${HIVE_IMAGE_TAG_NAME}
           docker ps -a
 
       - name: Setup Gradle
@@ -76,7 +76,7 @@ jobs:
             docker stop ${DOCKER_RUN_NAME}
             sleep 3
             docker ps -a
-            docker run --rm --name ${DOCKER_RUN_NAME} --platform ${PLATFORM} -d -p 8088:8088 -p 50070:50070 -p 50075:50075 -p 10000:10000 -p 10002:10002 -p 8888:8888 -p 9083:9083 -p 8022:22 ${HIVE_IMAGE_NAME}:${HIVE_IMAGE_TAG_NAME}
+            docker run --rm --name ${DOCKER_RUN_NAME} --platform ${PLATFORM} -d -p 8022:22 -p 8088:8088 -p 9000:9000 -p 9083:9083 -p 10000:10000 -p 10002:10002 -p 50070:50070 -p 50075:50075 ${HIVE_IMAGE_NAME}:${HIVE_IMAGE_TAG_NAME}
             sleep 60
 
       - name: Integration Test

dev/docker/hive/Dockerfile

@@ -154,7 +154,7 @@ RUN rm -rf /tmp/packages
 
 ################################################################################
 # expose port
-EXPOSE 8088 50070 50075 10002 10000 8888 9083 7180 22
+EXPOSE 22 8088 9000 9083 10000 10002 50070 50075
 
 ################################################################################
 # create startup script and set ENTRYPOINT

dev/docker/hive/README.md

@@ -2,23 +2,49 @@
   Copyright 2023 Datastrato.
   This software is licensed under the Apache License version 2.
 -->
-# hadoop2
-Build docker image that includes Hadoop2, Hive2
+# Hadoop and Hive Docker image
+This Docker image is used to support Graviton integration testing.
+It includes Hadoop-2.x and Hive-2.x, you can use this Docker image to test the Graviton catalog-hive module.
 
-Build Image
-===========
+## Build Docker image
 ```
 ./build-docker.sh --platform [all|linux/amd64|linux/arm64] --image {image_name} --tag {tag_name}
 ```
 
-Run container
-=============
+## Run container
 ```
-docker run --rm -d -p 8088:8088 -p 50070:50070 -p 50075:50075 -p 10000:10000 -p 10002:10002 -p 8888:8888 -p 9083:9083 -p 8022:22 datastrato/graviton-ci-hive:0.1.0
+docker run --rm -d -p 8022:22 -p 8088:8088 -p 9000:9000 -p 9083:9083 -p 10000:10000 -p 10002:10002 -p 50070:50070 -p 50075:50075 datastrato/graviton-ci-hive
 ```
 
-Login to the server
-=============
+## Login Docker container
 ```
 ssh -p 8022 datastrato@localhost (password: ds123, this is a sudo user)
-```
+```
+
+# Docker hub repository
+- [datastrato/graviton-ci-hive](https://hub.docker.com/r/datastrato/graviton-ci-hive)
+
+## Version change history
+### 0.1.0
+- Docker image `datastrato/graviton-ci-hive:0.1.0`
+- `hadoop-2.7.3`
+- `hive-2.3.9`
+- Expose ports:
+  - `22` SSH
+  - `9000` HDFS defaultFS
+  - `50070` HDFS NameNode
+  - `50075` HDFS DataNode
+  - `8088` YARN Service
+  - `9083` Hive Metastore
+  - `10000` HiveServer2
+  - `10002` HiveServer2 HTTP
+
+### 0.1.1
+- base on `datastrato/graviton-ci-hive:0.1.0`
+- Modify HDFS/YARN/HIVE `MaxPermSize` from `8GB` to `128MB`
+- Modify `HADOOP_HEAPSIZE` from `8192` to `128
+
+### 0.1.2
+- base on `datastrato/graviton-ci-hive:0.1.1` 
+- Modify `fs.defaultFS` from `local` to `0.0.0.0` in the `core-site.xml` file.
+- Expose `9000` port int the `Dockerfile` file.

dev/docker/hive/core-site.xml

@@ -1,7 +1,7 @@
 <configuration>
   <property>
     <name>fs.defaultFS</name>
-    <value>hdfs://localhost:9000</value>
+    <value>hdfs://0.0.0.0:9000</value>
   </property>
 
   <property>

docs/integration-test.md

@@ -27,9 +27,9 @@ Before running the tests, make sure Docker is installed.
 Then, execute blow steps: 
 1. Execute the `./gradlew clean build` command to build Graviton project.
 2. The Graviton project is compiled and packaged in the `distribution` directory using the `./gradlew compileDistribution` command.
-3. Run a hive docker test environment container in the local using the `docker run --rm -d -p 8088:8088 -p 50070:50070 -p 50075:50075 -p 10000:10000 -p 10002:10002 -p 8888:8888 -p 9083:9083 -p 8022:22 datastrato/graviton-ci-hive:0.1.0` command.
+3. Run a hive docker test environment container in the local using the `docker run --rm -d -p 8022:22 -p 8088:8088 -p 9000:9000 -p 9083:9083 -p 10000:10000 -p 10002:10002 -p 50070:50070 -p 50075:50075 datastrato/graviton-ci-hive` command.
 4. The integration test cases in the `integration-test` module are executed using the `./gradlew integrationTest` command.
 
 Additionally, the Graviton Server and third-party data source Docker runtime environments will use certain ports. Ensure that these ports are not already in use:
-- Graviton Server: Port `8088`
-- Hive Docker runtime environment: Ports `50070`, `50075`, `10002`, `10000`, `8888`, `9083`, `7180`, and `22`
+- Graviton Server: Port `8090`
+- Hive Docker runtime environment: Ports is `22`, `7180`, `8088`, `8888`, `9000`, `9083`, `10000`, `10002`, `50070`, and ` 50075`

docs/publish-docker-images.md

@@ -21,7 +21,7 @@ We use Github Actions to publish the docker images to the DockerHub repository.
 
 4. Input the `tag name`, for example: `0.1.0`, Then build and push the docker image name is `datastrato/{image-name}:0.1.0`.
 
-   + Currently, we only support publishing the `hive` docker image, for example: `datastrato/graviton-ci-hive:0.1.0`.
+   + Currently, we only support publishing the `hive` docker image, for example: `datastrato/graviton-ci-hive`.
    + We will support publishing other data sources, for example, `Iecberg` docker image in the future.
 
 5. You must enter the correct `publish docker token` before you can execute run `Publish Docker Image` workflow.
@@ -30,6 +30,4 @@ We use Github Actions to publish the docker images to the DockerHub repository.
 [<img src="assets/publish-docker-image.png" width="400"/>](assets/publish-docker-image.png)
 
 ## The version of the Data source in the Docker image
-| Docker image name | Docker image tag | Data source version      |
-|-------------------|------------------|--------------------------|
-| graviton-ci-hive  | 0.1.0            | hadoop-2.7.3, hive-2.3.9 |
+- [graviton-ci-hive](../dev/docker/hive/README.md)