Skip to content

Commit

Permalink
Add an easy to use local demo of Motion S3 connector
Browse files Browse the repository at this point in the history
Create a local demo of Motion s3 connector that focuses on ease of use
by relaxing the Filecoin interaction in favour of purely storing data
onto the local disk. This is to unblock users who are most interested in
the Motion S3 integration part.

Add README with example usage.
  • Loading branch information
masih committed Nov 15, 2023
1 parent c83d9f6 commit b98706e
Show file tree
Hide file tree
Showing 2 changed files with 103 additions and 0 deletions.
67 changes: 67 additions & 0 deletions local/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Motion S3 Connector: Local Storage

## Overview
This directory contains the setup for a local instance of the Motion and Motion S3 connector. It demonstrates the use of local disk storage for data, primarily stored in `$HOME/.motion`. This path can be customized using the `MOTION_HOME` environment variable.

This setup is intended for demonstration purposes only and does not connect with the Filecoin network. It highlights the utilization of the DeStor REST API as a backend for the AWS S3 API.

## Prerequisites
- Docker and Docker Compose installed
- AWS CLI installed

## Configuration
- **Default Storage Directory:** `$HOME/.motion` (modifiable via `MOTION_HOME`)
- **Credentials Configuration:** Edit [`authdata.json`](../authdata.json) or set `S3AUTH_CONFIG` (details in [CloudServer documentation](https://github.com/filecoin-project/motion-cloudserver/blob/378f6264f945eddd701f197280f3d94b2a3db1a9/docs/GETTING_STARTED.rst#setting-your-own-access-and-secret-key-pairs))
- **AWS CLI Endpoint URL:** `http://localhost:8000`

## Quick Start
1. **Prepare Storage Directory:** Ensure `MOTION_HOME` exists (default: `$HOME/.motion`).
2. **Launch Services:** Execute `docker compose up`.
3. **Set AWS CLI Credentials:**
```
export AWS_ACCESS_KEY_ID='accessKey1'
export AWS_SECRET_ACCESS_KEY='verySecretKey1'
export AWS_DEFAULT_REGION='location-motion-v1'
```
4. **Interact Using AWS CLI:** Use `--endpoint-url http://localhost:8000` for AWS CLI commands.

To halt the services, press `Ctrl + C`.

*Note: Use `MOTION_VERSION` and `MOTION_CLOUDSERVER_VERSION` environment variables to specify container versions.*

## Example Usage
Here's a basic example to demonstrate interaction with the Motion S3 Connector:
```shell
# Ensure MOTION_HOME directory
mkdir -p $HOME/.motion

# Start services in the background
docker compose up -d

# Configure AWS CLI credentials
export AWS_ACCESS_KEY_ID='accessKey1' AWS_SECRET_ACCESS_KEY='verySecretKey1' AWS_DEFAULT_REGION='location-motion-v1'

# Create a test file
echo 'Hello Motion S3 connector!' > upload.txt

# Create a new S3 bucket
aws --endpoint-url http://localhost:8000 s3 mb s3://test-bucket

# Upload a file to the bucket
aws --endpoint-url http://localhost:8000 s3 cp upload.txt s3://test-bucket

# List files in the bucket
aws --endpoint-url http://localhost:8000 s3 ls s3://test-bucket

# Download the file
aws --endpoint-url http://localhost:8000 s3 cp s3://test-bucket/upload.txt download.txt

# Remove the file from the bucket
aws --endpoint-url http://localhost:8000 s3 rm s3://test-bucket/upload.txt

# Delete the bucket
aws --endpoint-url http://localhost:8000 s3 rb s3://test-bucket

# Stop the services
docker compose down
```
36 changes: 36 additions & 0 deletions local/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
version: '3.8'

services:
motion:
platform: linux/amd64
image: ghcr.io/filecoin-project/motion:${MOTION_VERSION:-v0.2.3}
ports:
- 40080:40080
environment:
- MOTION_STORE_DIR=/usr/src/app/storage
volumes:
- motion-singularity-volume:/usr/src/app/storage

cloudserver:
platform: linux/amd64
image: ghcr.io/filecoin-project/motion-cloudserver:${MOTION_CLOUDSERVER_VERSION:-0.1.1}
environment:
REMOTE_MANAGEMENT_DISABLE: 1
S3DATA: 'multiple'
MOTION_HOST: 'motion'
ports:
- "8000:8000"
volumes:
- ${MOTION_HOME:-$HOME/.motion}/zenko/localData:/usr/src/app/localData
- ${MOTION_HOME:-$HOME/.motion}/zenko/localMetadata:/usr/src/app/localMetadata
- ${S3AUTH_CONFIG:-$PWD/../authdata.json}:/usr/src/app/conf/authdata.json
depends_on:
- motion

volumes:
motion-singularity-volume:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: ${MOTION_HOME:-$HOME/.motion}

0 comments on commit b98706e

Please sign in to comment.