An implementation of the MapReduce model in Golang.
The project is an attempt on Lab1 of 6.824 Distributed Systems by MIT.
Any MapReduce application comprises of two basic components :
-
Worker
: Component that sub-tasks are assigned to -
Master
: Component responsible for managing all the workers and ensures completion of assigned task
This repository provides a CLI tool mapReducer
to launch these components as required.
-
Clone the repository
git clone [email protected]:dafaqdhruv/Map-Reduce.git
-
Build
mapReducer
for the required taskmake build plugin=plugin/xyz.go
where
plugin/xyz.go
is the file containing requiredmap()
andreduce()
methods for the task.
-
Launch task coordinator with inputs files
./mapReducer mrcoordinator [input-files...]
-
Launch a single worker with plugin
./mapReducer mrworkers xyz.so
-
Launch multiple workers
./mapReducer mrworkers xyz.so & ./mapReducer mrworkers xyz.so & ./mapReducer mrworkers xyz.so