A correlation engine analyzes and identifies relationships and patterns among data sets.
For example, a TV or radio station may have playout data indicating that a broadcast event for a pizza commercial is scheduled to take place at dinner time. If the broadcasts were processed by aiWARE, the resulting data set can be examined for relevant dates and times and matching analysis can be performed to ensure the aired spots are consistent with the playout logs. This allows efficient lift, or the measure of success of a campaign.
When a correlation engine runs a task, it performs the following workflow:
- A task payload containing a Temporal Data Object (TDO) ID and schema is sent to the engine.
- The input TDO is analyzed and fields identified that match those specified in the schema.
- Structured data objects (SDOs) are extracted that conform to the schema. Based on the matching TDO attributes, the engine uses one of the following methods to collect relevant SDO data:
- Option 1: Query aiWARE's datastore to search for matching SDO data.
- Option 2: Call external APIs to fetch matching data.
The result is a new asset in AION format.
Correlation engine components
| Component | Description |
| Schema | A JSON schema that describes the required and optional fields of an SDO. The schema is used to validate against the data that comes into the system and to define what to write to the search index. |
| Ingestion adapter | An engine that pulls data from an external data source and saves it to the search index as an SDO through the aiWARE API. Adapters are used with correlation engines that access SDO content stored in aiWARE. |
| External data API | Third-party API or a data source that an ingestion adapter or correlation engine can retrieve data from. Correlation engines that access data stored outside of aiWARE can use an external data API to pull in and correlate data. |
| Search API | API used to search the aiWARE SDO index for data that has been ingested through an ingestion adapter. This API is built into correlation engines that access SDO content stored in aiWARE. |
| Create Asset API | API to write an SDO asset that contains the correlation results. |
Data residency
Correlation engines share a common setup to perform task processing, but based on where the structured data (SDO) data lives, one of two options are chosen to set up the engine to fetch SDO content.
- Option 1: Data residing externally: Data that lives in an external source can be ingested directly as an SDO and correlated to a TDO in real time. This option may use third-party APIs to pull in data that conforms to your schema and then automatically performs correlation upon ingestion. For organizations with different security requirements, this option offers the flexibility to store data in their own secure environment. It also works well when accessing pre-existing datasets that are too large to conveniently pull down into aiWARE.
- Option 2: Data residing in aiWARE: If your engine will be accessing data thats stored in aiWARE, your engine can query the search index for SDO data to correlate against the TDO. With this option, you'll either use aiWARE APIs or create an adapter to pull in data from an external source and save it to the index as an SDO.
In the example of correlation logic below, written in Node.js, SDO data to correlate is obtained by search. The search is based in part on start and stop times contained in the passed-in TDO.
Available correlation engines
See the list of available correlation engines here, under our data engine class.
You can also build your own.
