v5.5.0

Version Summary

Version Number : v5.5.0 [tag: v5.5.0]
Date: 03-April-2024

Projects / Repos Included in this release: IUDX Data Ingestion Server, IUDX Deployment and installation, IUDX's Documentation

Release Summary :

India Urban Data Exchange (IUDX) | Data Ingestion Server v5.5.0 is released with enhanced features.

Highlights:

• Updated Software Client libraries
• Updated Software Testing
  • Updated Postman Collection
  • Included REST Assured based integration testing
  • Improved test coverage
• Software License:
  • With this release, all IUDX Softwares will be distributed under Apache 2.0 License

Test Reports:

Please find the release test details and reports here

API Docs

The api docs can be found here.

How to use the Release:

External Dependencies Installation

The Data ingestion pipeline connects to external dependencies namely

• RabbitMQ

Make configuration

Make a config file based on the template in ./configs/config-example.json

• Generate a certificate using Lets Encrypt or other methods
• Make a Java Keystore File and mention its path and password in the appropriate sections
• Modify the database url and associated credentials in the appropriate sections

Docker based

1. Install docker and docker-compose
2. Clone this repo
3. Build the images
   ./docker/build.sh
4. Modify the docker-compose.yml file to map the config file you just created
5. Start the server in production (prod) or development (dev) mode using docker-compose
   docker-compose up prod

Maven based

1. Install java 11 and maven
2. Use the maven exec plugin based starter to start the server
   mvn clean compile exec:java@data-ingestion-server

JAR based

1. Install java 11 and maven
2. Set Environment variables

export DI_URL=https://<rs-domain-name>
export LOG_LEVEL=INFO

3. Use maven to package the application as a JAR
   mvn clean package -Dmaven.test.skip=true
4. 2 JAR files would be generated in the target/ directory
   • iudx.data.ingestion.server-cluster-0.0.1-SNAPSHOT-fat.jar - clustered vert.x containing micrometer metrics
   • iudx.data.ingestion.server-dev-0.0.1-SNAPSHOT-fat.jar - non-clustered vert.x and does not contain micrometer metrics

Running the clustered JAR

Note: The clustered JAR requires Zookeeper to be installed. Refer here to learn more about how to set up Zookeeper. Additionally, the zookeepers key in the config being used needs to be updated with the IP address/domain of the system running Zookeeper.
The JAR requires 3 runtime arguments when running:

• --config/-c : path to the config file
• --hostname/-i : the hostname for clustering
• --modules/-m : comma separated list of module names to deploy
  e.g. java -jar ./fatjar.jar --host $(hostname) -c configs/config.json -m iudx.data.ingestion.server.ApiServerVerticle
  Use the --help/-h argument for more information. You may additionally append an DI_JAVA_OPTS environment variable containing any Java options to pass to the application.
  e.g.

$ export RS_JAVA_OPTS="-Xmx4096m"
$ java $RS_JAVA_OPTS -jar target/iudx.data.ingestion.server-cluster-0.0.1-SNAPSHOT-fat.jar ...

Running the non-clustered JAR

The JAR requires 1 runtime argument when running:

• --config/-c : path to the config file
  e.g. java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -jar target/iudx.data.ingestion.server-dev-0.0.1-SNAPSHOT-fat.jar -c configs/config.json
  Use the --help/-h argument for more information. You may additionally append an RS_JAVA_OPTS environment variable containing any Java options to pass to the application.
  e.g.

$ export RS_JAVA_OPTS="-Xmx1024m"
$ java $RS_JAVA_OPTS -jar target/iudx.data.ingestion.server-dev-0.0.1-SNAPSHOT-fat.jar ...

License

With this release, all IUDX Softwares will be distributed under Apache 2.0 License

What's Changed

• Jenkins CI: Updated image tag to 5.5.0-alpha by @karun-singh in #87
• Software Updates by @Kranthi-Guribilli in #88
• License information update by @sushanthakumar in #89

New Contributors

• @Kranthi-Guribilli made their first contribution in #88
• @sushanthakumar made their first contribution in #89

Full Changelog: v5.0.0...v5.5.0

Data Ingestion Server v5.0.0

What's new?

• Updated the ID structure to Universally Unique IDentifiers (UUIDs) for improved security.

Testing

• Added PMD and Checkstyle to the server for improved code quality analysis

Data Ingestion Server v4.5.0

Enhancement

• Improved and enhanced auditing service workflow for better QOS
• Miscellaneous library upgrades, and bug fixes

Data Ingestion Server v4.0.0

New Features

• Software Updates
  • With release v4.0.0 all the client libraries are updated to latest versions
• Enhanced Code Coverage
  • IUDX API server are now unit tested in an automated fashion with a Code Coverage > 90%
• Enhanced Validation and cache performance to improve query time

Data Ingestion Server v3.5.0

New Features

• DI server is an external server which allows Providers to push (as opposed to connectors pulling the data) to IUDX Resource Server. This helps to onboard data in scenarios where APIs to pull data are non-existing.
• Implements /ingestion and /entities (POST Query) from the specification

What's Changed

• Updated .gitignore by @swaminathanvasanth in #1
• Data Ingestion Server Stub Code by @swaminathanvasanth in #2
• Added functionality for publish data RMQ by @gautamv95 in #3
• Api-Verticle by @gopal-mahajan in #5
• Format code for RMQ, and add Promise by @gautamv95 in #4
• refactoring by @kailash in #6
• complete response format by @kailash in #7
• Ingest data RMQ by @gautamv95 in #8
• /ingestion added. by @gopal-mahajan in #9
• Update AuthHandler.java by @kailash in #10
• ApiServerVerticle Refactored by @gopal-mahajan in #11
• Add testcases for databroker verticle by @gautamv95 in #12
• Router for api added. by @gopal-mahajan in #13
• Updated Readme by @gopal-mahajan in #14
• Update README.md by @swaminathanvasanth in #15
• catalogue urn by @kailash in #19
• Deployment and configuration through docker and docker compose by @swaminathanvasanth in #20
• Audit Service Integration. by @gopal-mahajan in #21
• Updates by @swaminathanvasanth in #22
• Integration with Jenkins CI pipeline by @karun-singh in #23
• Remove sensitive logs, irrelevant info logs by @abhi4578 in #24
• Correct metrics & add curator dep. for clustering by @abhi4578 in #25
• Jenkins CI improvements by @karun-singh in #26
• Cleanup fixes 3.5 by @abhi4578 in #27
• Jenkins CI update: Added cleanup of unit test reports after stage completion by @karun-singh in #28
• Graceful shutdown fix by @pranavrd in #31
• non-root img for DI server by @hackcoderr in #29
• Update README.md for jar based depl by @pranavrd in #30
• readme/setup by @kailash in #32
• added diagram by @kailash in #33

New Contributors

• @swaminathanvasanth made their first contribution in #1
• @gautamv95 made their first contribution in #3
• @gopal-mahajan made their first contribution in #5
• @kailash made their first contribution in #6
• @karun-singh made their first contribution in #23
• @abhi4578 made their first contribution in #24
• @pranavrd made their first contribution in #31
• @hackcoderr made their first contribution in #29

Full Changelog: https://github.com/datakaveri/iudx-data-ingestion-server/commits/v3.5.0