Query on summary-rollup-syncts Druid data source

[spsac08]

Hi Team,

We are trying to create reports for Learn related data on the top of summary-rollup-syncts Druid data source,
And we have a supervisor running for this data source in Druid,
We wanted to know how do we enable this data source in druid and what type of data to be pushed.

@sowmya-dixit Can you please help us on this ?

cc @aks30, @kiranharidas187, @vijiurs,@Shakthieshwari

[spsac08]

Hi Team,
I need help on this query ( Understading ), can anyone please assist me ?

[spsac08]

@anandp504 We have a supervisor running in Druid and we added injection spec to it.
Our query is just what type of data we need to push to Kafka so that we can create this data source,
if any data source is dependent on this.

[sowmya-dixit]

@spsac08 These summary-rollup-syncts data are derived summary events generated out of raw telemetry events which are generated by user interactions from UI. Complete data pipeline and sunbird data products set up needs to be available to generate these kind of summary data. Please provide more details on your requirement of enabling this data source.

[spsac08]

@sowmya-dixit Thank you, and can you please provide us which other data pipeline needed and sunbird data product is required?
can you please give us some git repo links to generate the summary rollup syncts datasource.

[sowmya-dixit]

@spsac08 Please share us more details on your requirement like - are you trying to set this up in your own separate env? If yes does it already have telemetry events getting generated? What other set ups are already available related to Obsrv?
Please find more details in below docs which also has git repo details,
Data pipeline - https://obsrv.sunbird.org/learn/product-and-developer-guide/data-pipeline
Summary Data Product - https://obsrv.sunbird.org/learn/product-and-developer-guide/summarisers

[spsac08]

@sowmya-dixit Yes, We are trying to set it up in own separate env, and we are also using telemetry services, and data getting generated and Flink jobs and Druid Clusters with ingestion Tasks

[spsac08]

Hi @sowmya-dixit,
All our environment stepup like Flink Job, Data Products and Analytics stepup is done and completed.
We need to know what type of data to submit, as in course etc. And where it will be stored in places like Kafka, mongo db, Casandra etc..

[spsac08]

@sowmya-dixit can you please help us with this, How do we check the Workflow Summarizer - Data Product is running or not?

[sowmya-dixit]

@spsac08
Please check if crontab is setup for WFS, it will be run using this script in spark server - /mount/data/analytics/scripts/run-job.sh wfs
If not please setup this script to run everyday. It can also be run manually once to check if it runs fine and execution logs can be found under - /mount/data/analytics/logs/data-products/<CurrentDate>-job-execution.log file.

[spsac08]

Thank you @sowmya-dixit

[spsac08]

we can see data in druid, Thank you very much @sowmya-dixit, the last request is if there is any document to understand the data can you please point out where will get it?

[sowmya-dixit]

@spsac08 - You can find more details on the summary data in this document - https://github.com/project-sunbird/sunbird-core-dataproducts/wiki/%5BData-Product%5D-Workflow-Summarizer#output