Source Types

Ingestion adapters are configured through a system of Sources and Source Types. This system provides a structured way to manage where aiWARE ingests content from and how that ingestion is handled.

• Source Type: A Source Type is a template that defines a specific kind of content source. Each Source Type is associated with a specific adapter and has a defined schema that dictates the configuration information required to connect to that type of source. For example, the Podcast source type is linked to the Podcast Adapter, and its schema would require an RSS feed URL.

  • Is Live: A Source Type is considered “Live” if it is configured to expect a continuous stream of information rather than file-based or periodic batches (e.g., closed files)

  • Requires Scan: A source type requires scanning if it relies on an additional step before the actual ingestion occurs. Typically this is as simple as discovering and retrieving the list of files in an S3 bucket or fetching the feed entries in an RSS feed.

• Source: A Source is a specific, configured instance of a Source Type. It represents a single, unique ingestion point and holds the actual connection details (like a URL, API endpoint, or bucket name) and any necessary credentials. You might have one "S3" Source Type, but you would create a separate Source for each individual S3 bucket you want to ingest from.

• Adapter Engine: An adapter engine is an engine that is purpose built to utilize the schema information provided by a source type in order to perform the actual ingestion of content into the aiWARE system. An engine is linked to a source type by declaring the supported source type IDs in the supportedSourceTypes field

This structure allows you to create many specific Sources from a limited number of Source Types, making it possible to manage numerous ingestion points in an organized and scalable way. Sources are created and managed within the Data Center utility in aiWARE. For adapters requiring more complex configurations, creating or updating a Source may require using the GraphQL API in addition to or instead of the Data Center interface.

Source Types with Descriptions