Skip to content

Latest commit

 

History

History
67 lines (48 loc) · 2.64 KB

README.md

File metadata and controls

67 lines (48 loc) · 2.64 KB
Raw

teraslice state storage - currently experimental

Module to save state for teraslice processes. Currently uses elasticsearch for state persistance and lru-caching to reduce network calls. Requires job context for connection details and config object which is passed from the _op in the teraslice job.

Intallation and Usage

Installation

Installation TBD

Usage

const StateStorage = require('teraslice-state-storage');
const stateStorage = new StateStorage(context, config);
stateStorage.count();
stateStorage.mset(docArray);

Configuration Options:

  • cache_limit - max number of items to store in the cache (not memory size)
    positive integer, default: 1000000

  • max_age - length of time before a record expires in milliseconds
    positive integer, default: 24 hours

  • connection - elasticsearch connection
    string, default: 'default'

  • index - name of elasticsearch index
    string, default: 'index'

  • type - type of the elasticsearch data string, default: 'type'

  • chunk_size - how many docs to send in the elasticsearch mget request at a time
    postitive integer, default is 2500

  • concurrency - number of cuncurrent requests to elasticsearch
    positive integer, default: 100

  • source_fields - fields to retreive from elasticsearch
    array of fields, defaults to all fields

  • id_field - specifies the field to use as the key for caching and retrieving docs from elasticsearch
    string, default: 'id'

  • persist - If set to true will save state in storage for mset, doest not apply to set.
    boolean, default: false

  • persist_field - If persist is true this option is the name of the key field that will be the key in the es bulk update, this may be the same as the id_field but not necessarily.
    string, default: 'key'

Functions:

  • get - expects an object with the id_field as a property and returns an object with an id and saved fields
    example: get(doc)

  • mget - expects an array of objects and returns an object. Returns a promise and the data returned is an object.
    example: mget(docArray)

  • set - expects and object, boolean is optional. Updates state with an option to save state in storage
    Returns a promise.
    example: set(doc)
    example: set(doc, true) - saves the state in storage

  • mset - expects an array of objects boolean is optional. Updates state with an option to save state in storage.
    Returns a promise.
    example: mset(docArray)
    example: mset(docArray, true) - saves the state in storage

  • count - returns the number docs currently saved in the cache
    example: count()