From f327fc2aa916e26d8eddac438c5adfb8cdef8abe Mon Sep 17 00:00:00 2001 From: Tugdual Grall Date: Mon, 18 Dec 2017 12:16:48 +0100 Subject: [PATCH] Update introduction and architecture - issue #96 --- doc/tutorials/001-introduction.md | 22 ++-- doc/tutorials/002-mapr-music-architecture.md | 130 ++++++++----------- 2 files changed, 65 insertions(+), 87 deletions(-) diff --git a/doc/tutorials/001-introduction.md b/doc/tutorials/001-introduction.md index a5b5219f..8f146974 100644 --- a/doc/tutorials/001-introduction.md +++ b/doc/tutorials/001-introduction.md @@ -1,7 +1,7 @@ # Introduction MapR Music Application uses the [MusicBrainz](https://musicbrainz.org/) database to explain how to develop an -application with MapR Converged Data Platform. +application with MapR Converged Data Platform with a focus on MapR-DB JSON. ## Contents @@ -34,35 +34,31 @@ Contains Wildfly configuration files, which are used while building MapR Music D * [data-generator](https://github.com/mapr-demos/mapr-music/tree/master/data-generator) -Utility application which allows to convert [MusicBrainz](https://musicbrainz.org/) database dump into MapR Music Data -Set. +Utility application which allows to convert [MusicBrainz](https://musicbrainz.org/) database dump into MapR Music Dataset. * [dataset](https://github.com/mapr-demos/mapr-music/tree/master/dataset) -Sample, ready to use MapR Music dataset. +Sample, ready to use MapR Music dataset, generated with the script described above. * [doc](https://github.com/mapr-demos/mapr-music/tree/master/doc) -Application's documentation. +Application's documentation and tutorial. * [elasticsearch-service](https://github.com/mapr-demos/mapr-music/tree/master/elasticsearch-service) -MapR Music Elastic Search Service, which listens changelogs and publishes the changes to the ElasticSearch. +MapR Music Elasticsearch Service, which listens albums and artists table events and publishes the changes to the Elasticsearch. This service use MapR-DB Change Data Capture (CDC) feature. * [mapr-rest](https://github.com/mapr-demos/mapr-music/tree/master/mapr-rest) -MapR Music REST Service, built on top of MapR-DB. +MapR Music REST Service, built on top of MapR-DB. This Web application developped in Java use [JAX-RS](https://github.com/jax-rs) and [OJAI](https://maprdocs.mapr.com/home/MapR-DB/JSON_DB/develop-apps-jsonDB.html) to expose CRUD operations at the top of MapR-DB tables. * [mapr-ui](https://github.com/mapr-demos/mapr-music/tree/master/mapr-ui) -Angular app, which provides user interface. +[Angular](https://angular.io/) application that consume the REST APIs to allow user to browse, edit and create albums and artists. * [recommendation-engine](https://github.com/mapr-demos/mapr-music/tree/master/recommendation-engine) -Recommendation Engine, built using Spark MLlib's Alternating Least Squares algorithm, which allows to make -Albums/Artists recommendations. +Recommendation Engine, built using Spark MLlib's Alternating Least Squares (ALS) algorithm, to create Albums/Artists recommendations. This service uses the [MapR-DB OJAI Connector for Apache Spark](https://maprdocs.mapr.com/home/Spark/NativeSparkConnectorJSON.html) to read albums, artists ratings, and writes the recommendations back into MapR-DB. Moreover, repository contains [Dockerfile](https://github.com/mapr-demos/mapr-music/blob/master/Dockerfile), which allows -you to build Mapr Music Docker image, which contains properly configured Wildfly application server, -ElasticSearch and MapR Music components. - +you to build Mapr Music Docker image based on [MapR PACC](https://mapr.com/products/persistent-application-client-container/), download,configure and start: Wildfly, Elasticsearh and deploy MapR Music application. diff --git a/doc/tutorials/002-mapr-music-architecture.md b/doc/tutorials/002-mapr-music-architecture.md index d03350e1..633a9498 100644 --- a/doc/tutorials/002-mapr-music-architecture.md +++ b/doc/tutorials/002-mapr-music-architecture.md @@ -1,27 +1,21 @@ # MapR Music Architecture + +![Mapr Music Architeture](https://raw.githubusercontent.com/mapr-demos/mapr-music/master/doc/tutorials/images/music-app-architecture.png) + + MapR Music Application consists of the following modules: -1. [MapR Music REST Service](https://github.com/mapr-demos/mapr-music/tree/master/mapr-rest) +1. [MapR Music REST Service](https://github.com/mapr-demos/mapr-music/tree/master/mapr-rest): built on top of MapR-DB. This Web application developped in Java use [JAX-RS](https://github.com/jax-rs) and [OJAI](https://maprdocs.mapr.com/home/MapR-DB/JSON_DB/develop-apps-jsonDB.html) to expose CRUD operations at the top of MapR-DB tables. -MapR Music REST Service, built on top of MapR-DB. -2. [MapR Music UI](https://github.com/mapr-demos/mapr-music/tree/master/mapr-ui) - -Angular app, which provides user interface. +2. [MapR Music UI](https://github.com/mapr-demos/mapr-music/tree/master/mapr-ui): [Angular](https://angular.io/) application that consume the REST APIs to allow user to browse, edit and create albums and artists. -3. [Elastic Search Service](https://github.com/mapr-demos/mapr-music/tree/master/elasticsearch-service) - -MapR Music Elastic Search Service, which listens changelogs and publishes the changes to the ElasticSearch. -4. [Recommendation Engine](https://github.com/mapr-demos/mapr-music/tree/master/recommendation-engine) +3. [Elastic Search Service](https://github.com/mapr-demos/mapr-music/tree/master/elasticsearch-service): which listens albums and artists table events and publishes the changes to the Elasticsearch. This service use MapR-DB Change Data Capture (CDC) feature. -Recommendation Engine, built using Spark MLlib's Alternating Least Squares algorithm, which allows to make -Albums/Artists recommendations. +4. [Recommendation Engine](https://github.com/mapr-demos/mapr-music/tree/master/recommendation-engine): this service uses the [MapR-DB OJAI Connector for Apache Spark](https://maprdocs.mapr.com/home/Spark/NativeSparkConnectorJSON.html) to read albums, artists ratings, and writes the recommendations back into MapR-DB. -5. [Data Generator](https://github.com/mapr-demos/mapr-music/tree/master/data-generator) - -Utility application which allows to convert [MusicBrainz](https://musicbrainz.org/) database dump into MapR Music Data -Set. +5. [Data Generator](https://github.com/mapr-demos/mapr-music/tree/master/data-generator): utility application which allows to convert [MusicBrainz](https://musicbrainz.org/) database dump into MapR Music Dataset. ## MapR Music REST Service Architecture @@ -31,30 +25,23 @@ is widely known by most architects, designers, and developers. Album service will have three tiers, organized into horizontal layers, each layer performing a specific role within the application: -* Presentation Layer -Communicates with other tiers and provides results to the browser or other clients. +* Presentation Layer: communicates with other tiers and provides results to the browser or other clients. -* Business Layer +* Business Layer: performs business logic on domains. -Performs business logic on domains. +* Data Layer: includes the data persistence mechanisms. -* Data Layer +## Data layer -Includes the data persistence mechanisms. +Albums data are stored in +[MapR-DB JSON Tables](https://docstage.mapr.com/public/beta/60/MapR-DB/JSON_DB/json_tables.html). The application uses the [MapR OJAI library](https://maprdocs.mapr.com/apidocs/60/OJAI/index.html) to insert, update, delete and query documents. -## Data layer +The first step to develop MapR-DB Application in Java, you must add the MapR OJAI Driver library to your Maven project `pom.xml` file: -Albums data will be stored in -[MapR-DB JSON Tables](https://docstage.mapr.com/public/beta/60/MapR-DB/JSON_DB/json_tables.html) so we need to implement -persistence mechanism, which accesses -[MapR-DB](https://docstage.mapr.com/public/beta/60/MapR-DB/developing_client_applications_for_mapr_db.html) and -stores/queries data to/from it. This can be accomplished by using -[OJAI Connection and Driver interfaces](https://docstage.mapr.com/public/beta/OJAI/index.html). Define OJAI Driver -dependency in your `pom.xml`: ``` - 6.0.0-mapr-beta + 6.0.0-mapr ... @@ -146,15 +133,14 @@ public class Album { } ``` -You can notice `@MaprDbTable` annotation. It's custom annotation used to define MapR-DB JSON Table path so -implementation of DAO will know which table to query. +You can notice [`@MaprDbTable`](https://github.com/mapr-demos/mapr-music/blob/master/mapr-rest/src/main/java/com/mapr/music/annotation/MaprDbTable.java) annotation; it's custom annotation used to define MapR-DB JSON Table path so implementation of DAO will know which table to query. OJAI Driver allows to create OJAI connection to the cluster and access the OJAI Document Store in the following way: -``` +```java ... // Create an OJAI connection to MapR cluster - Connection connection = DriverManager.getConnection(CONNECTION_URL); + Connection connection = DriverManager.getConnection("ojai:mapr:"); // Get an instance of OJAI DocumentStore final DocumentStore store = connection.getStore(tablePath); @@ -174,50 +160,51 @@ OJAI Driver allows to create OJAI connection to the cluster and access the OJAI ... ``` -This approach is used by by -[MaprDbDao](https://github.com/mapr-demos/mapr-music/blob/master/mapr-rest/src/main/java/com/mapr/music/dao/MaprDbDao.java) -class, which implements common operations. +This approach is used by by [MaprDbDao](https://github.com/mapr-demos/mapr-music/blob/master/mapr-rest/src/main/java/com/mapr/music/dao/MaprDbDao.java) +class, which implements common operations like find with various parameters, insert, update and delete documents. OJAI [Connection](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/Connection.html) interface along with -[DocumentStore](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html) interface defines all -key methods to interact with MapR-DB. +[DocumentStore](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html) interface defines all key methods to interact with MapR-DB. + + +The [MaprDbDao](https://github.com/mapr-demos/mapr-music/blob/master/mapr-rest/src/main/java/com/mapr/music/dao/MaprDbDao.java) class is a wrapper to the various methods used to inreact with MapR-DB JSON, here the list of the principal DocumentStore methods used by this classs. #### Query the Documents in the DocumentStore -Get all Documents using -[DocumentStore#find()](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#find--) method. You -can specify conditions using [DocumentStore#find(QueryCondition condition)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#find-org.ojai.store.QueryCondition-) -or get single Document by it's identifier using [DocumentStore#findById(String id)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#findById-java.lang.String-) -method. +Here the list of methods to query the DocumentStore: + +* [DocumentStore#find()](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#find--) to get all the documents. +* [DocumentStore#find(QueryCondition condition)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#find-org.ojai.store.QueryCondition-) to get documents using a condition (_where clause_) +* [DocumentStore#findById(String id)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#findById-java.lang.String-) to get a single document using its `_id`. #### Use projection by specifying the list of fields that should be returned in the read document. Here listed some methods that allow to use projection: -* [DocumentStore#find(FieldPath... fieldPaths)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#find-org.ojai.FieldPath...-) -* [DocumentStore#find(String... fieldPaths)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#find-java.lang.String...-) -* [DocumentStore#find(QueryCondition condition, String... fieldPaths)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#find-org.ojai.store.QueryCondition-java.lang.String...-) -* [DocumentStore#findById(String id, FieldPath... fieldPaths)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#findById-java.lang.String-org.ojai.FieldPath...-) +* [DocumentStore#find(FieldPath... fieldPaths)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#find-org.ojai.FieldPath...-) +* [DocumentStore#find(String... fieldPaths)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#find-java.lang.String...-) +* [DocumentStore#find(QueryCondition condition, String... fieldPaths)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#find-org.ojai.store.QueryCondition-java.lang.String...-) +* [DocumentStore#findById(String id, FieldPath... fieldPaths)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#findById-java.lang.String-org.ojai.FieldPath...-) #### Create Documents and insert them into DocumentStore There are several way to create new OJAI Document using -[Connection](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/Connection.html) interface: -* [Connection#newDocument()](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/Connection.html#newDocument--) - creates and returns a new, empty instance of an OJAI Document. -* [Connection#newDocument(String jsonString)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/Connection.html#newDocument-java.lang.String-) - returns a new instance of OJAI Document parsed from the specified JSON string. -* [Connection#newDocument(Object bean))](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/Connection.html#newDocument-java.lang.Object-) - returns a new instance of Document built from the specified Java bean. -* [Connection#newDocument(Map map)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/Connection.html#newDocument-java.util.Map-) - returns a new instance of Document constructed from the specified Map. +[Connection](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/Connection.html) interface: +* [Connection#newDocument()](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/Connection.html#newDocument--) - creates and returns a new, empty instance of an OJAI Document. +* [Connection#newDocument(String jsonString)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/Connection.html#newDocument-java.lang.String-) - returns a new instance of OJAI Document parsed from the specified JSON string. +* [Connection#newDocument(Object bean))](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/Connection.html#newDocument-java.lang.Object-) - returns a new instance of Document built from the specified Java bean. +* [Connection#newDocument(Map map)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/Connection.html#newDocument-java.util.Map-) - returns a new instance of Document constructed from the specified Map. Created OJAI Document can be inserted in DocumentStore using -[DocumentStore#insert(Document doc)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#insert-org.ojai.Document-) +[DocumentStore#insert(Document doc)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#insert-org.ojai.Document-) method. #### Update Documents In order to update the Document we should construct corresponding -[DocumentMutation](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentMutation.html) object. This +[DocumentMutation](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentMutation.html) object. This interface defines the APIs to perform mutation of a Document already stored in a DocumentStore. After that you can update the Document using following methods: -* [DocumentStore#update(String _id, DocumentMutation mutation)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#update-java.lang.String-org.ojai.store.DocumentMutation-) -* [DocumentStore#update(Value _id, DocumentMutation mutation)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#update-org.ojai.Value-org.ojai.store.DocumentMutation-) +* [DocumentStore#update(String _id, DocumentMutation mutation)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#update-java.lang.String-org.ojai.store.DocumentMutation-) +* [DocumentStore#update(Value _id, DocumentMutation mutation)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#update-org.ojai.Value-org.ojai.store.DocumentMutation-) For instance: ``` @@ -250,19 +237,15 @@ For instance: #### Delete Documents Document can be deleted using -[DocumentStore#delete(String _id)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#delete-java.lang.String-) -method. Also, it's possible to delete the set of Documents that match some condition using -[DocumentStore#checkAndDelete(String _id, QueryCondition condition)](https://docstage.mapr.com/public/beta/OJAI/org/ojai/store/DocumentStore.html#checkAndDelete-java.lang.String-org.ojai.store.QueryCondition-) +[DocumentStore#delete(String _id)](https://maprdocs.mapr.com/apidocs/60/OJAI/org/ojai/store/DocumentStore.html#delete-java.lang.String-) method. -For more information about OJAI Driver refer [OJAI API library javadoc](https://docstage.mapr.com/public/beta/OJAI/index.html). +For more information about OJAI Driver refer [OJAI Javadoc](https://maprdocs.mapr.com/apidocs/60/OJAI/index.html). ## Business Layer -Business layer is responsible of performing business logic. It provides data to the presentation layer in a suitable -form using DTO pattern, implements pagination logic, communicates with Data Layer in order to fetch and persist the -data. Source code of all interfaces and actual implementation classes can be found at +Business layer is responsible of performing business logic. It provides data to the presentation layer in a suitable form using [DTO (Data Transfer Object)](https://en.wikipedia.org/wiki/Data_transfer_object) pattern, implementing the pagination logic, communicates with Data Layer in order to fetch and persist the data. Source code of all interfaces and actual implementation classes can be found in the [service package](https://github.com/mapr-demos/mapr-music/tree/master/mapr-rest/src/main/java/com/mapr/music/service). ## Presentation Layer @@ -280,11 +263,11 @@ Presentation Layer of Album Service is represented by REST API endpoints. GET /api/1.0/albums - Get list of albums, which is represented by page. Supports the following query options: + Get list of albums by page. Supports the following query parameters:
    -
  • 'page' - specifies page number
    • -
  • 'per_page' - specifies number of albums per page
    • -
  • 'sort' - sort string, which contains 'ASC' or 'DESC' sort type and comma separated list of sort fields
    • +
  • page - specifies the page number
    • +
  • per_page - specifies the number of albums per page
    • +
  • sort - sort string, which contains 'ASC' or 'DESC' sort type and comma separated list of sort fields
curl -X GET "http://localhost:8080/mapr-music-rest/api/1.0/albums?per_page=2&page=1548&sort=ASC,released_date,name" @@ -292,19 +275,19 @@ Presentation Layer of Album Service is represented by REST API endpoints. GET /api/1.0/albums/{id} - Get single album by it's identifier + Get single album using identifier (_id) curl -X GET http://localhost:8080/mapr-music-rest/api/1.0/albums/1 DELETE /api/1.0/albums/{id} - Delete single album by it's identifier + Delete single album using identifier (_id) curl -u jdoe:music -X DELETE http://localhost:8080/mapr-music-rest/api/1.0/albums/1 PUT /api/1.0/albums/{id} - Update single album by it's identifier + Update single album using identifier (_id) curl -u jdoe:music -d '{"name":"NEW NAME"}' -H "Content-Type: application/json" -X PUT http://localhost:8080/mapr-music-rest/api/1.0/albums/1 @@ -320,9 +303,8 @@ Note: to get detailed endpoints description, follow [API Reference](http://local Note: all modification operations require user authorization. -Presentation layer implemented using JAX-RS annotations. It communicates with business layer to get the model data in -a suitable form and sends it as response to the client's request. Here is code snippet of Album endpoint: -``` +Presentation layer is implemented using JAX-RS annotations. It communicates with business layer to get the model data in a suitable form and sends it as response to the client's request. Here is code snippet of Album endpoint: +``` java /** * Endpoint for accessing 'Album' resources. */