Search crawl framework

[lmichaud]

To support the release of the prototype, there is a need to define a search crawl between HC, TC, and CFIA data in order to feed the LucidWords indexing engine.

[lmichaud]

Decisions captured during meeting of November 1st. Topic was around discussing crawl frequency. options, and next steps.

CSV crawl will be daily (evening preferably).. LucidWords can crawl every 5 minutes but if on-demand is supported, this can be pushed on a daily schedule
Preference is to use on-demand crawl push - will require updates to RSAM system, and TC DB if they require it.
will require IP whiltelist to avoid network issues to support IO calls
Mario working with TC team to capture vehicle recalls (.net environment)
Many of the data requirements can be supported at the crawl level (e.g. relationships between values for filters and search meta can be defined using common syntax in crawl - such as comma separated values)
another example shared (food allergies), hyphenated category names can automatically generate sub-relationships

Tasks:

• Cristian/Mario to share guidance on how to enable on-demand indexing with LucidWorks (with Sylvie Rossignol)
• Louis to validate feedback and confirm categorization (excel) with Miguel
• once validated, Miguel to share metadata lists (type) and mapping with Cristian
• Cristian/Mario to share IP range with Sylvie for IO calls
• Mario to open thread on the RSA Mobile GitHub channel to define search approach (adaptable for the app)... to start dialogue with HC apps team

[alamarch]

@RickyHGitHub see above.

[lmichaud]

Here's my rendition of the search data indexing framework that we discussed yesterday. This should give us a common understanding of the mechanics supporting the new RSA search landing page (indexing included by lucidworks).

@masterbee can you validate that the wireframe is accurate? I'm also sharing a link to the powerpoint source in case someone wants to play around with the illustration.

https://docs.google.com/presentation/d/1vfMOdB_owRWCVAGwusb9NqNXRkLbI1wAMU7QqwoqmQk/edit#slide=id.p1

Original from brainstorm session:

[lbelmore]

Thanks Louis!
Here are some of the action items from our last meeting:

• @masterbee to review and validate the framework diagram that Louis created - Posted above

• @sylros to coordinate with her team on implementation and next steps on the dev side

• Lisa/Louis to review and validate the crawl output http://io.canada.ca/hc-sc/rsa/latest.xml

• @masterbee @sylros to coordinate technical meeting for next week to finalize details for key/token implementation for on demand pushes