-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add an easy to use local demo of Motion S3 connector
Create a local demo of Motion s3 connector that focuses on ease of use by relaxing the Filecoin interaction in favour of purely storing data onto the local disk. This is to unblock users who are most interested in the Motion S3 integration part. Add README with example usage.
- Loading branch information
Showing
2 changed files
with
103 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,67 @@ | ||
# Motion S3 Connector: Local Storage | ||
|
||
## Overview | ||
This directory contains the setup for a local instance of the Motion and Motion S3 connector. It demonstrates the use of local disk storage for data, primarily stored in `$HOME/.motion`. This path can be customized using the `MOTION_HOME` environment variable. | ||
|
||
This setup is intended for demonstration purposes only and does not connect with the Filecoin network. It highlights the utilization of the DeStor REST API as a backend for the AWS S3 API. | ||
|
||
## Prerequisites | ||
- Docker and Docker Compose installed | ||
- AWS CLI installed | ||
|
||
## Configuration | ||
- **Default Storage Directory:** `$HOME/.motion` (modifiable via `MOTION_HOME`) | ||
- **Credentials Configuration:** Edit [`authdata.json`](../authdata.json) or set `S3AUTH_CONFIG` (details in [CloudServer documentation](https://github.com/filecoin-project/motion-cloudserver/blob/378f6264f945eddd701f197280f3d94b2a3db1a9/docs/GETTING_STARTED.rst#setting-your-own-access-and-secret-key-pairs)) | ||
- **AWS CLI Endpoint URL:** `http://localhost:8000` | ||
|
||
## Quick Start | ||
1. **Prepare Storage Directory:** Ensure `MOTION_HOME` exists (default: `$HOME/.motion`). | ||
2. **Launch Services:** Execute `docker compose up`. | ||
3. **Set AWS CLI Credentials:** | ||
``` | ||
export AWS_ACCESS_KEY_ID='accessKey1' | ||
export AWS_SECRET_ACCESS_KEY='verySecretKey1' | ||
export AWS_DEFAULT_REGION='location-motion-v1' | ||
``` | ||
4. **Interact Using AWS CLI:** Use `--endpoint-url http://localhost:8000` for AWS CLI commands. | ||
|
||
To halt the services, press `Ctrl + C`. | ||
|
||
*Note: Use `MOTION_VERSION` and `MOTION_CLOUDSERVER_VERSION` environment variables to specify container versions.* | ||
|
||
## Example Usage | ||
Here's a basic example to demonstrate interaction with the Motion S3 Connector: | ||
```shell | ||
# Ensure MOTION_HOME directory | ||
mkdir -p $HOME/.motion | ||
|
||
# Start services in the background | ||
docker compose up -d | ||
|
||
# Configure AWS CLI credentials | ||
export AWS_ACCESS_KEY_ID='accessKey1' AWS_SECRET_ACCESS_KEY='verySecretKey1' AWS_DEFAULT_REGION='location-motion-v1' | ||
|
||
# Create a test file | ||
echo 'Hello Motion S3 connector!' > upload.txt | ||
|
||
# Create a new S3 bucket | ||
aws --endpoint-url http://localhost:8000 s3 mb s3://test-bucket | ||
|
||
# Upload a file to the bucket | ||
aws --endpoint-url http://localhost:8000 s3 cp upload.txt s3://test-bucket | ||
|
||
# List files in the bucket | ||
aws --endpoint-url http://localhost:8000 s3 ls s3://test-bucket | ||
|
||
# Download the file | ||
aws --endpoint-url http://localhost:8000 s3 cp s3://test-bucket/upload.txt download.txt | ||
|
||
# Remove the file from the bucket | ||
aws --endpoint-url http://localhost:8000 s3 rm s3://test-bucket/upload.txt | ||
|
||
# Delete the bucket | ||
aws --endpoint-url http://localhost:8000 s3 rb s3://test-bucket | ||
|
||
# Stop the services | ||
docker compose down | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
version: '3.8' | ||
|
||
services: | ||
motion: | ||
platform: linux/amd64 | ||
image: ghcr.io/filecoin-project/motion:${MOTION_VERSION:-v0.2.3} | ||
ports: | ||
- 40080:40080 | ||
environment: | ||
- MOTION_STORE_DIR=/usr/src/app/storage | ||
volumes: | ||
- motion-singularity-volume:/usr/src/app/storage | ||
|
||
cloudserver: | ||
platform: linux/amd64 | ||
image: ghcr.io/filecoin-project/motion-cloudserver:${MOTION_CLOUDSERVER_VERSION:-0.1.1} | ||
environment: | ||
REMOTE_MANAGEMENT_DISABLE: 1 | ||
S3DATA: 'multiple' | ||
MOTION_HOST: 'motion' | ||
ports: | ||
- "8000:8000" | ||
volumes: | ||
- ${MOTION_HOME:-$HOME/.motion}/zenko/localData:/usr/src/app/localData | ||
- ${MOTION_HOME:-$HOME/.motion}/zenko/localMetadata:/usr/src/app/localMetadata | ||
- ${S3AUTH_CONFIG:-$PWD/../authdata.json}:/usr/src/app/conf/authdata.json | ||
depends_on: | ||
- motion | ||
|
||
volumes: | ||
motion-singularity-volume: | ||
driver: local | ||
driver_opts: | ||
type: 'none' | ||
o: 'bind' | ||
device: ${MOTION_HOME:-$HOME/.motion} |