diff --git a/.gitignore b/.gitignore index 455a32b..282c584 100644 --- a/.gitignore +++ b/.gitignore @@ -39,3 +39,5 @@ e2e/cert.pem e2e/key.pem dist/ + +tapes/frames/ diff --git a/Makefile b/Makefile index a92a2fd..4dcf0d9 100644 --- a/Makefile +++ b/Makefile @@ -9,37 +9,36 @@ mock: docker-dev: cd docker/compose ;\ docker compose up -d - docker ps .PHONY: docker-infra docker-infra: cd docker/compose ;\ docker compose -f docker-compose.infra.yml -f docker-compose.nginx.yml up -d - docker ps .PHONY: docker-infra-clean docker-infra-clean: cd docker/compose ;\ docker compose -f docker-compose.infra.yml -f docker-compose.nginx.yml down -v - docker ps .PHONY: docker-dev-build docker-dev-build: cd docker/compose ;\ docker compose up -d --build - docker ps .PHONY: docker-dev-build-svc docker-dev-build-svc: cd docker/compose ;\ docker compose up -d --no-deps --build $$SVC - docker ps .PHONY: docker-dev-clean docker-dev-clean: cd docker/compose ;\ docker compose down -v +.PHONY: docker-ps +docker-ps: + docker ps --format="table {{.Image}}\t{{.Status}}\t{{.Names}}" + .PHONY: goimports goimports: goimports -w . diff --git a/README.md b/README.md index 005e934..9baf32b 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,14 @@ WIP -This is a final project for Yandex Practicum's Go Course. +## Overview -Vidi is a media server that supports: - - simple user registration - - video uploading by registered users - - On-demand streaming of uploaded videos using MPEG-DASH +Vidi is a video service (or platform). Features include: + - simple user registration and login + - mp4 files upload (with h.264 AVC and mp4a AAC codecs) + - resuming of interrupted upload + - upload quotas per user + - on-demand streaming of uploaded videos with MPEG-DASH Uploaded mp4 files are pre-processed, so they could be streamed to dash clients. Preprocessing includes: - Segmentation (using awesome [Eyevinn/mp4ff](https://github.com/Eyevinn/mp4ff) package) @@ -17,18 +19,60 @@ Uploaded mp4 files are pre-processed, so they could be streamed to dash clients. Also project uses: - PostgreSQL for video object storage and user storage -- S3 compatible storage for video data (compose project uses minio) +- S3 compatible storage for video content - Redis for media sessions storage +This is my pet project and is very far from being production-ready. Nice and shiny web UI is not even planned. + +## Architecture + +Vidi consists of several services + +### Userapi + +Simple user registration and login service. Made with `labstack/echo`. + +### Videoapi + +Main API that implements the most of the business logic, like video object CRUD operations. It is also an entry point for upload and watch flows. + +Video API has service-side (for back2back comms) and user-side (user->api comms) parts. User-side is served as http with `labstack/echo` and as grpc. Service-side has only grpc. + +Both Userapi and Videoapi are considered as API-domain. + +### Uploader + +This service is responsible for media content upload. It uses upload sessions created by videoapi to identify and validate upload requests. Made with `valyala/fasthttp`. + +### Streamer + +This service serves DASH segments to users. It uses watch sessions created by videoapi to identify and validate download requests. Made with `valyala/fasthttp`. + +### Processor + +This is worker-style service that processes uploaded videos to DASH-format. Uses `Eyevinn/mp4ff` in its core. + +Processor, uploader and streamer are considered as Media-domain. + +### Vidit + +This is client-side tool, that provides TUI to interact with Vidi. It supports almost every operation, excluding video playback: at the moment it can generate watch url which can be used with DASH reference player (included in compose project). Made with `charmbracelet/bubbletea`. + ## Development +`./docker` dir contains reference all-in-one deployment with docker-compose. Nginx is used as gateway. It also has simple web page with DASH reference client to test video playback. Minio is used as self-hosted s3. + ### Docker ```bash +# generate certificates +make tls # start compose project -make docker-dev +make docker-dev +# or +make docker-dev-build # to rebuild images -# stop compose project and cleanup volumes +# stop compose project and remove volumes make docker-dev-clean ``` @@ -38,11 +82,13 @@ make docker-dev-clean # run unit tests make unittests -# run unit and end2end tests -# docker compose project should be stopped (it will be ran by the command) +# run unit and e2e tests +# docker compose project should be stopped (it will be started by the command) make test-all ``` -### Happy Path +### Showcase + +Showcase of project startup and some features -User interaction flow is described [here](./docs/happy_path.md). +![showcase](tapes/showcase.gif) diff --git a/docs/happy_path.md b/docs/happy_path.md deleted file mode 100644 index 12acbe3..0000000 --- a/docs/happy_path.md +++ /dev/null @@ -1,93 +0,0 @@ -## Happy path - -This docu describes user interaction flow with vidi platform. To follow these steps you should have docker-compose project started with all containers running. - -### User registration - -Register user with following call: -```shell -curl -v -XPOST -H "Content-Type: application/json" \ - http://localhost:8080/api/users/register \ - -d '{"username":"johnS","password":"qwerty123"}' -``` - -Response should be `200 OK` with authentication cookie. - -To double-check that user is created you can call login with same credentials: - -```shell -curl -v -XPOST -H "Content-Type: application/json" \ - http://localhost:8081/api/users/login \ - -d '{"username":"johnS","password":"qwerty123"}' -``` - -Response should also be `200 OK` with authentication cookie. Both cookies will work until expiration time, which is 12 hours by default. - -From this point you can make authenticated requests using received cookie. For convenience, you can make an alias: - -```shell -alias curl_auth='curl --cookie "vidiSessID=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJRT0RBUW5xM1NkNjg2bmtsQ1RHZkhnIiwibmFtZSI6ImpvaG5TIiwiZXhwIjoxNzA1MTg2MjgyfQ.KnwB89hW9tQRvW18e_Xno45nvc_7aMcJimJ2ehVuMIM"' -``` - -### Video upload - -#### Prepare file - -At this time, vidi supports only mp4 files with `AVC (h.264)` video codec and `mp4a` audio codec. - -To check if your file is supported, you can use vidi cli: -```shell -go run cmd/vidi/vidi.go mp4 dump -f ./testfiles/test_seq_h264_high.mp4 -``` -At the end of output should be message `Codecs are supported!`. - -#### Upload video - -First, create video object using video API call: -```shell -curl_auth -v -XPOST http://localhost:8080/api/video/user/ -``` -```shell -{"id":"PMO7_KJTRO2xntQyQdfRAw","status":"created","created_at":"2024-01-15 22:16:06.318666558 +0000 UTC m=+59.538474225","upload_url":"http://localhost:8080/upload/4F0lCtrmSl27B2bRFQNvDg"} -``` -You should receive `201 Created` and video object parameters. Most important one is `upload_url`, which should be used to upload mp4 video file. This url is unique and valid for a short period of time (default upload session time is 5 min). - -Now you can actually upload video: - -```shell -curl -v -XPOST http://localhost:8080/upload/4F0lCtrmSl27B2bRFQNvDg -H "Content-Type: video/mp4" --data-binary "@./testfiles/test_seq_h264_high.mp4" -``` - -Successful upload response should be `204 No Content`. You can check video status by using video ID from video creation response. Status could be `uploaded` or `ready` if video is very short and was already processed. - -```shell -curl_auth -v -XGET http://localhost:8080/api/video/user/PMO7_KJTRO2xntQyQdfRAw -``` -```shell -{"id":"PMO7_KJTRO2xntQyQdfRAw","status":"ready","created_at":"0001-01-01 00:00:00 +0000 UTC"} -``` - -#### Watch video - -If video is `ready`, you can get watch URL by calling: -```shell -curl_auth -v -XPOST http://localhost:8080/api/video/user/PMO7_KJTRO2xntQyQdfRAw/watch -``` -```shell -{"watch_url":"http://localhost:8080/watch/bsz-zpKZTFqOXgfLYSSgmg/manifest.mpd"} -``` - -Response will contain generated unique URL that can be fed to any player that understands MPEG-DASH. - -Compose project provides dash.js player which can be used to test if video can actually be played. Open http://localhost:8080/ in Chrome, paste `watch_url` into the input bar at the top and hit play. Video should start playing. - -> (Relatively) latest chrome should play video with no problems. There's some issues with Safari, do not use it for now. Other browsers were not tested. - -Watch URL will be valid as long as session is not expired (which is 5 min after last segment request by default). If session is expired, watch URL will return 404. To watch video again, you should again request watch URL from video API. - -#### Delete video - -If video is no longer needed you may delete it using: -```shell -curl_auth -v -XDELETE http://localhost:8080/api/video/user/PMO7_KJTRO2xntQyQdfRAw -``` diff --git a/tapes/showcase.gif b/tapes/showcase.gif new file mode 100644 index 0000000..16832e6 Binary files /dev/null and b/tapes/showcase.gif differ diff --git a/tapes/showcase.tape b/tapes/showcase.tape new file mode 100644 index 0000000..b69d302 --- /dev/null +++ b/tapes/showcase.tape @@ -0,0 +1,126 @@ +Output tapes/showcase.gif +Output tapes/frames/ + +Set Theme Atom +Set Shell zsh +Set Width 1600 +Set Height 900 +Set TypingSpeed 250ms + +Set Margin 10 +Set MarginFill "#6B50FF" +Set BorderRadius 10 +Set Framerate 24 + +Set PlaybackSpeed 1.0 + +# prepare +Enter@2 +Type "echo 'Generating certificates'" +Enter@1 + +Type "make tls" +Sleep 1 +Enter@4 +Enter +Type "echo 'Starting compose project'" +Enter@1 + +# images better be pre-built +Type "make docker-dev" +Sleep 1 +Enter@5 +Type "make docker-ps" +Enter@5 + +# start vidit +Type "clear" +Enter@1 +Type "echo 'Starting vidit tool'" +Enter@1 +Type "echo '-d sets starting directory for file picker'" +Enter@1 +Type "go run cmd/vidit/main.go -d $(pwd)/testfiles" +Enter@5 + +# config screen +Type "http://localhost" +Enter@3 + +# new user screen +# select register +Down@1 +Enter@2 + +# enter creds +Type "JaneDoe" +Enter@2 +Type "password" +Enter@2 +Left@1 +Enter@5 + +# show empty videos screen +Enter@3 +Backspace@1 + +# show quotas screen +Down@1 +Down@1 +Enter@1 +Down@1 +Down@3 +Backspace@2 + +# goto uploads screen +Down@2 +Enter@2 +# select file +Down@1 +Enter@1 + +# enter name +Type "test video" +Sleep 1 +Enter@1 +# confirm +Left@1 +Enter@7 +Enter@2 + +# show quotas screen +Down@1 +Down@1 +Enter@1 +Down@1 +Down@3 +Backspace@2 + +# show videos screen +Enter@2 +Type@6 w +Backspace@5 +Ctrl+c +Sleep 3 + +# show state +Type "echo 'Vidit persists its state into home dir'" +Enter@1 +Type "ls $HOME/.vidi" +Enter@2 + +Type "cat $HOME/.vidi/state.json" +Enter@2 + +Type "echo 'Next run will reuse endpoint config and user info'" +Enter@1 + +# start again +Type "go run cmd/vidit/main.go" +Enter@5 + +# show videos screen +Enter@3 +Backspace@5 +Ctrl+c +Sleep 3