Modular plugin-style document type definitions

[charlesreid1]

Organize things in a modular plugin way, have one file per "collection type" (github issues, github markdown, google drive, groups.io emails).

Current schema:

Search object:
  add_drive_file()
  add_issue()
  add_markdown()
  update_index_gdocs()
  update_index_issues()
  update_index_markdown()
  open_index()
  create_search_result()
  get_document_total_count()

The add_* and update_* methods need to be split into separate importable files. The way these files are written depends 100% on the schema, so they aren't independent of the Search object, but being in separate files makes them easier to edit, duplicate, and modify as the schema changes.

New schema:

Search object:
  open_index()
  create_search_result()
  get_document_total_count()

centillion_gdocs.py:
  add_drive_file()
  update_index_gdocs()

centillion_ghissue.py:
  add_issue()
  update_index_issues()

centillion_markdown.py:
  add_markdown()
  update_index_markdown()

Documentation: one page for centillion_gdocs, one page for centillion_ghissue, etc.

Portability: ultimately there is no way to make centillion drop-in, because everything depends so much on the schema. The schema is what makes your search engine what it is, what makes it useful. Your schema determines your frontend (what info is presented from each search record) and your backend (how documents in collections are turned into search index records).

[charlesreid1]

what if each class had their own definition of columns to add to the schema?

then you could say, register schema X, schema Y, and schema Z with the search index, and the search index would determine what fields X/Y/Z each add, and put them all together into a super-schema that includes all fields.

[charlesreid1]

Two approaches:

1. add each new type of document into the schema by having each field of document type X be prefixed, e.g., markdown_title, markdown_url, ... - this offers logical separation but makes it really hard to search on fields. this approach is out.

2. each document has a schema specified - what columns and what types. the search index schema is similarly defined (yaml file?). there is a validation step that sets up the index, then checks each specified field and type to ensure it is safe.