Installation
Create a /tmp/config.conf. Can be created in another folder to configure this you must run consumer and monitor jar using[-p] /custom/folder/config.conf.
config.conf must contain content below:
# Destination folder for the files events
cluster.database=/apps/fs/db/
kafka.producer.stream=/fsmonitor
kafka.consumer.stream=/fsmonitor_replica
kafka.consumer.group.id=fsmon
kafka.common.enable.auto.commit=false
kafka.common.auto.offset.reset=latest
kafka.common.auto.create.topics.enable=true
kafka.common.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
kafka.common.key.serializer=org.apache.kafka.common.serialization.StringSerializer
kafka.common.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
kafka.common.value.serializer=org.apache.kafka.common.serialization.StringSerializer
# MapR to S3 Replication configuration
s3.consumer.stream=/fsmonitor_replica
s3.consumer.group.id=s3Replicator
s3.bucket=mapr-sync
s3.directories=vol1:/mapr/my.cluster.com/volumes/vol1,vol2:/mapr/my.cluster.com/volumes/vol2
The File system Monitor and Consumer are using MapR Streams to exchange event information. So you have to create new stream that will contains a topic by volume.
The stream name is configured in the configuration file:
# FS Monitor sends data to the following topics : /fsmonitor/change_<volume_name>
kafka.producer.stream=/fsmonitor
# Consumer received data from the following topic : /fsmonitor_replica/change_<volume_name>
kafka.consumer.stream=/fsmonitor_replica
Create the streams on source and target cluster using the following command:
maprcli stream create -path /fsmonitor -produceperm p -consumeperm p -topicperm p
On each node of the source cluster, with the NFS gateway run the following command:
java -cp /home/mapr/monitor-1.0-SNAPSHOT-jar-with-dependencies.jar:$(mapr classpath) com.mapr.fs.application.Monitor -p [path to config file]
Note:
On one node of the monitoring cluster run the UI using this command:
java -jar mapr-monitor-fs-ui-1.0-SNAPSHOT.jar -p [path to config file]
Note: you can select the listening port using the property
server.portin application.property file in resource folder, by default it 3000
Now that you have the Monitor (producer) and UI started you can configure the monitor to select the source.
Go to the Admin page:
http://[SERVER]:3000/
Enter a new volume name (example: source1)
Enter a path to volume (example: /mapr/my.cluster.com/fs-sync/source/source1) and click add
Result
On one node of the destination cluster:
java -cp "`mapr classpath`:/home/mapr/consumer-1.0-SNAPSHOT-jar-with-dependencies.jar" com.mapr.fs.application.Consumer -p [path to config file]
Note:
On one node of the destination cluster run the UI using this command:
java -jar mapr-consumer-fs-ui-1.0-SNAPSHOT.jar -p [path to config file]
Note: you can select the listening port using the property
server.portin application.property file in resource folder, by default it 8080
Now that you have the Monitor (producer), Consumer and UI started you can configure the consumer to select the source.
Go to the Admin page:
http://[SERVER]:8080/
Enter a new cluster name, click add
Click on the cluster name
Enter a "Volume name" that should be the same as source1
Enter a "Path to volume" for example /mapr/demo.mapr.com/fs-sync/destination/, click add
Result
Also, you can see consumed files status, for this click on the Monitoring link
Select volume
Result
Currently you should start s3 jar where monitor starts. Then you should start the S3 UI. On the UI you should specify volume name bucket name(should already exist) and Path, there Path should be the SAME as on monitor. Then specify checkboxes and click start. Then you can play with checkboxes. All should work in runtime.
If you want to reset the prototype you have to:
- stop the services
- delete content from source and destination folder
- delete the stream
- delete the tables (
rm -Rf /mapr/demo.mapr.com/apps/fs/db/*)
then
- create the stream
- start the services
- go in the UI to register a new source