Skip to content

Latest commit

 

History

History
46 lines (36 loc) · 1.71 KB

readme.md

File metadata and controls

46 lines (36 loc) · 1.71 KB
Raw

µ-batching

Overview

A simple microbatching system in Go, designed to process jobs efficiently in batches. It includes a queue for job management and a batch processing mechanism. The system also provides a REST API to submit jobs, check job status, and retrieve information about queued and processed jobs.

Features

  • Job Submission: Submit jobs for processing via a REST API.
  • Job Queueing: Jobs are queued before being batch processed.
  • Batch Processing: Jobs are processed in configurable batch sizes and intervals.
  • Job Status Tracking: Track the status of submitted jobs, whether they're queued or processed.
  • Job ID Retrieval: Retrieve the IDs of all jobs currently in the queue.
  • Processed Job Retrieval: Retrieve all processed jobs.

Getting Started

Prerequisites

Go (version 1.15 or later)

Installation & hey, just run it

Clone the repository to your local machine:

git clone https://github.com/1x-eng/mbat.git
cd mbat
touch ./.env # edit the config as you see fit

# build and run the executable
go build -o ./bin/mbat ./cmd/
./bin/mbat

API Endpoints

  • Submit Job: POST /job/submit
  • Job Status: GET /job/status/:jobID
  • Queued Jobs: GET /jobs/queue
  • Processed Jobs: GET /jobs/processed

Swagger / OpenAPI

  • µ-batching uses swaggo cli to generate swagger specs. Install swaggo cli using - go install github.com/swaggo/swag/cmd/swag@latest
  • To generate or refresh swagger specs, swag init -g ./pkg/server/handlers.go
  • µ-batching also uses fiber-swagger to render swaghger UI.
  • Once the project is built and run, visit <api>/swagger to view swagger UI.

TODO

  • Integrate github actions / promote CI/CD and abstract swaggo / fiber-swagger
  • A whole lot of tests :P