diff --git a/.nojekyll b/.nojekyll new file mode 100644 index 00000000..e69de29b diff --git a/404.html b/404.html new file mode 100644 index 00000000..f2769466 --- /dev/null +++ b/404.html @@ -0,0 +1,2497 @@ + + + +
+ + + + + + + + + + + + + + + + +DP³'s has HTTP API which you can use to post datapoints and to read data stored in DP³.
+As the API is made using FastAPI, there is also an interactive documentation available at /docs
endpoint.
There are several API endpoints:
+GET /
: check if API is running (just returns It works!
message)POST /datapoints
: insert datapoints into DP³GET /entity/<entity_type>
: list current snapshots of all entities of given typeGET /entity/<entity_type>/<entity_id>
: get data of entity with given entity idGET /entity/<entity_type>/<entity_id>/get/<attr_id>
: get attribute valueGET /entity/<entity_type>/<entity_id>/set/<attr_id>
: set attribute valueGET /entity/<entity_type>/_/distinct/<attr_id>
: get distinct attribute values and their counts based on latest snapshotsDELETE /entity/<entity_type>/<entity_id>
: delete entity data for given idPOST /entity/<entity_type>/<entity_id>/ttl
: extend TTLs of the specified entityGET /entities
: list entity configurationGET /control/<action>
: send a pre-defined action into execution queue.GET /telemetry/sources_validity
: get information about the validity of the data sourcesHealth check.
+GET /
200 OK
:
{
+ "detail": "It works!"
+}
POST /datapoints
All data are written to DP³ in the form of datapoints. A datapoint sets a value of a given attribute of given entity.
+It is a JSON-encoded object with the set of keys defined in the table below. Presence of some keys depends on the primary type of the attribute (plain/observations/timseries).
+Payload to this endpoint is JSON array of datapoints. For example:
+ +Key | +Description | +Data-type | +Required? | +Plain | +Observations | +Timeseries | +
---|---|---|---|---|---|---|
type |
+Entity type | +string | +mandatory | +✔ | +✔ | +✔ | +
id |
+Entity identification | +string | +mandatory | +✔ | +✔ | +✔ | +
attr |
+Attribute name | +string | +mandatory | +✔ | +✔ | +✔ | +
v |
+The value to set, depends on attr. type and data-type, see below | +-- | +mandatory | +✔ | +✔ | +✔ | +
t1 |
+Start time of the observation interval | +string (RFC 3339 format) | +mandatory | +-- | +✔ | +✔ | +
t2 |
+End time of the observation interval | +string (RFC 3339 format) | +optional, default=t1 |
+-- | +✔ | +✔ | +
c |
+Confidence | +float (0.0-1.0) | +optional, default=1.0 | +-- | +✔ | +✔ | +
src |
+Identification of the information source | +string | +optional, default="" | +✔ | +✔ | +✔ | +
More details depends on the particular type of the attribute.
+{
+ "type": "ip",
+ "id": "192.168.0.1",
+ "attr": "note",
+ "v": "My home router",
+ "src": "web_gui"
+}
+
{
+ "type": "ip",
+ "id": "192.168.0.1",
+ "attr": "open_ports",
+ "v": [22, 80, 443],
+ "t1": "2022-08-01T12:00:00",
+ "t2": "2022-08-01T12:10:00",
+ "src": "open_ports_module"
+}
+
regular
:
{
+ ...
+ "t1": "2022-08-01T12:00:00",
+ "t2": "2022-08-01T12:20:00", // assuming time_step = 5 min
+ "v": {
+ "a": [1, 3, 0, 2]
+ }
+}
+
irregular
: timestamps must always be present
{
+ ...
+ "t1": "2022-08-01T12:00:00",
+ "t2": "2022-08-01T12:05:00",
+ "v": {
+ "time": ["2022-08-01T12:00:00", "2022-08-01T12:01:10", "2022-08-01T12:01:15", "2022-08-01T12:03:30"],
+ "x": [0.5, 0.8, 1.2, 0.7],
+ "y": [-1, 3, 0, 0]
+ }
+}
+
irregular_interval
:
{
+ ...
+ "t1": "2022-08-01T12:00:00",
+ "t2": "2022-08-01T12:05:00",
+ "v": {
+ "time_first": ["2022-08-01T12:00:00", "2022-08-01T12:01:10", "2022-08-01T12:01:15", "2022-08-01T12:03:30"],
+ "time_last": ["2022-08-01T12:01:00", "2022-08-01T12:01:15", "2022-08-01T12:03:00", "2022-08-01T12:03:40"],
+ "x": [0.5, 0.8, 1.2, 0.7],
+ "y": [-1, 3, 0, 0]
+ }
+}
+
Can be represented using both plain attributes and observations. The difference will be only +in time specification. Two examples using observations:
+no data - link<mac>
: just the eid is sent
{
+ "type": "ip",
+ "id": "192.168.0.1",
+ "attr": "mac_addrs",
+ "v": "AA:AA:AA:AA:AA",
+ "t1": "2022-08-01T12:00:00",
+ "t2": "2022-08-01T12:10:00"
+}
+
with additional data - link<ip, int>
: The eid and the data are sent as a dictionary.
{
+ "type": "ip",
+ "id": "192.168.0.1",
+ "attr": "ip_dep",
+ "v": {"eid": "192.168.0.2", "data": 22},
+ "t1": "2022-08-01T12:00:00",
+ "t2": "2022-08-01T12:10:00"
+}
+
200 OK
:
400 Bad request
:
Returns some validation error message, for example:
+1 validation error for DataPointObservations_some_field
+v -> some_embedded_dict_field
+ field required (type=value_error.missing)
+
List latest snapshots of all ids present in database under entity type.
+Contains only latest snapshot.
+Uses pagination.
+GET /entity/<entity_type>
Optional query parameters:
+Get data of entity type's eid.
+Contains all snapshots and master record. Snapshots are ordered by ascending creation time.
+GET /entity/<entity_type>/<entity_id>
Optional query parameters:
+Get attribute value
+Value is either of:
+GET /entity/<entity_type>/<entity_id>/get/<attr_id>
Optional query parameters:
+Set current value of attribute
+Internally just creates datapoint for specified attribute and value.
+This endpoint is meant for editable
plain attributes -- for direct user edit on DP3 web UI.
POST /entity/<entity_type>/<entity_id>/set/<attr_id>
Required request body:
+ +Gets distinct attribute values and their counts based on latest snapshots
+Useful for displaying <select>
enumeration fields.
Works for all plain and observation data types except dict
and json
.
GET /entity/<entity_type>/_/distinct/<attr_id>
Delete master record and snapshots with the specified etype
and eid
.
Raw datapoints are not deleted,
+and the entity can be restored by sending new datapoints with the same etype
and eid
.
DELETE /entity/<entity_type>/<entity_id>
Extend TTLs of the specified entity.
+Raw datapoints are not deleted,
+and the entity can be restored by sending new datapoints with the same etype
and eid
.
POST /entity/<entity_type>/<entity_id>/ttl
The request body must be a dictionary of TTLs to extend, with string keys to identify the type of TTL. +The values must be UTC timestamps, for example:
+ +TTLs of the same name will be extended, and you add as many TTL names as you want.
+List entity types
+Returns dictionary containing all entity types configured -- their simplified configuration and current state information.
+GET /entities
{
+ "<entity_id>": {
+ "id": "<entity_id>",
+ "name": "<entity_spec.name>",
+ "attribs": "<MODEL_SPEC.attribs(e_id)>",
+ "eid_estimate_count": "<DB.estimate_count_eids(e_id)>"
+ },
+ ...
+}
+
Execute Action - Sends the given action into execution queue.
+You can see the enabled actions in /config/control.yml
, available are:
make_snapshots
- Makes an out-of-order snapshot of all entitiesrefresh_on_entity_creation
- Re-runs the on_entity_creation
callback for selected etype
refresh_module_config
- Re-runs the load_config
for selected module and will refresh the values derived by the module when configured to do soYou can learn more about the actions in the Actions section of the Control
configuration documentation.
GET /control/<action>
Returns information about the validity of the data sources, i.e. when the last datapoint was received from each source.
+GET /telemetry/sources_validity
DP³ is generic platform for data processing. +It's currently used in systems for management of network devices in CESNET, +but during development we focused on making DP³ as universal as possible.
+This page describes the high-level architecture of DP³ and the individual components.
+The base unit of data that DP³ uses is called a data-point, which looks like this:
+{
+ "type": "ip", // (1)!
+ "id": "192.168.0.1", // (2)!
+ "attr": "open_ports", // (3)!
+ "v": [22, 80, 443], // (4)!
+ "t1": "2022-08-01T12:00:00", // (5)!
+ "t2": "2022-08-01T12:10:00",
+ "src": "open_ports_module" // (6)!
+}
+
type
.id
. attr
field specifies the attribute of the data-point.v
field.t1
and t2
field. src
field.This example shows an example of an observations data-point (given it has a validity interval), +to learn more about the different types of data-points, please see the API documentation.
+The DP³ architecture as shown in the figure above consists of several components, +where the DP³ provided components are shown in blue:
+The application-specific components, shown in yellow-orange, are as following:
+yml
files determines the entities and their attributes,
+ together with the specifics of platform behavior on these entities.
+ For details of entity configuration, please see the database entities configuration page./scripts/dummy_sender.py
.
+ Feel free to check out the secondary module API documentation, or you can also
+ refer to the CallbackRegistrar code reference or
+ check out the test modules in /modules/
or /tests/modules/
.This section describes the data flow within the platform.
+ +The above figure shows a zoomed in view of the worker-process from the architecture figure.
+Incoming Tasks, which carry data-points from the API,
+are passed to secondary module callbacks configured on new data point, or around entity creation.
+These modules may create additional data points or perform any other action.
+When all registered callbacks are processed, the resulting data is written to two collections:
+The data-point (DP) history collection, where the raw data-points are stored until archivation,
+and the profile history collection, where a document is stored for each entity id with the relevant history.
+You can find these collections in the database under the names {entity}#raw
and {entity}#master
.
DP³ periodically creates new profile snapshots, triggered by the Scheduler.
+Snapshots take the profile history, and compute the current value of the profile,
+reducing each attribute history to a single value.
+The snapshot creation frequency is configurable.
+Snapshots are created on a per-entity basis, but all linked entities are processed at the same time.
+This means that when snapshots are created, the registered snapshot callbacks
+can access any linked entities for their data correlation needs.
+After all the correlation callbacks are called, the snapshot is written to the profile snapshot collection,
+for which it can be accessed via the API. The collection is accessible under the name {entity}#snapshots
.