Webhooks

All engines in aiWARE are packaged as executable Docker containers. Each engine is a process that provides two webhooks so the Engine Toolkit can invoke the process data.

Each webhook provides unique functionality, and is therefore triggered by a unique HTTP request. An engine webhook can pass data as a JSON object, which makes engines widely compatible with all the programming languages.

The ready and process webhooks are provided by an engine. The heartbeat and result webhooks are provided by the Engine Toolkit.

Engine webhooks

Engines implement two webhooks: one for ready requests and one for process requests. The Engine Toolkit uses the following environment variables to discover the webhooks as provided by the core engine:

ENV VERITONE_WEBHOOK_READY="http://0.0.0.0:8888/ready" ENV VERITONE_WEBHOOK_PROCESS="http://0.0.0.0:8888/process"

The webhook endpoints are configurable and are specified using environment variables in the Docker file. Webhooks are expected to return a 200 OK successful response, otherwise the engine toolkit retries the operation by making the same requests again.

Request

The Engine Toolkit invokes the webhook with the following input data in its POST payload

Ready webhook

The ready webhook determines if the engine is ready to start doing work or not.

GET /ready

The webhook should reply with a 503 Service Unavailable status until the engine is ready to receive work, at which point it should reply to this webhook with a simple 200 OK response.

If the webhook replies with 500 http status, then the Engine Toolkit will shut down.

Process webhook

The process webhook is used to perform some processing on a file (such as the frame from a video).

POST /process

The body of the request is a multipart form (where Content-Type is multipart/form-data) containing everything the engine needs to do its work.

The following fields will be posted to the Process webhook:

• internalJobId - (string) The internal job ID
• internalTaskId - (string) The internal task ID
• chunkMimeType - (string) The MIME type of the chunk (for example, image/jpg)
• startOffsetMS - (int) The start time of the chunk (for example, the timestamp of when a frame was extracted from a video)
• endOffsetMS - (int) The end time of the chunk (see startOffsetMS)
• cacheURI - (string) - (string) Stream engines should use this to retrieve the input stream
• veritoneApiBaseUrl - (string) URI to AI Data if this edge is connected to a core
• token - (string) aiWARE token to use with AI Data
• payload - (string) task payload as a JSON object
• chunk-metadata-xxxx - (string) The input chunk may have metadata as intended by the parent task, which is typically sent by the caller to the parent task. The Engine Toolkit will make these available to the engine task with the prefix. chunk-metadata- and stringified value
• inputFolderID - (string) The ID of the input folder for this chunk

If the engine is a chunk, and the chunk is an image or video:

• width - (int) The width of the chunk
• height - (int) The height of the chunk

The following advanced fields are also included:

• cacheURI - (URI) URL of the chunk source file
• veritoneApiBaseUrl - (string) The root URL for Veritone platform API requests
• token - (string) The token to use when making low level API requests
• payload - (string) JSON string containing the entire task payload

If the engine leverages a library, the following fields will be included:

• libraryId - (string) ID of the library related to this task
• libraryEngineModelId - (string) ID of the library engine model related to this task
• libraryModelDirectory - (string) Directory the engine can use to store library engine models, either after a training job is done or when in matching mode, and the engine would like to cache the model for quick access in subsequent tasks.

File system related fields:

• taskTmpDir - (string) Temporary directory that serves as a scratchpad for the engine to store temporary data related to the task. This directory will be purged after a certain time (3 days default). For any permanent data, use other means. This directory should already exist when the engine receives its chunk input.
• engineDir - (string) Directory the engine can use to store shared assets across different instances.

Processing control fields:

• heartbeatWebhook - (URI) The heartbeat webhook provided by the Engine Toolkit. Engines with async processing for the /process webhook, such as stream or batch engines, should submit heartbeats with progress information.
• resultWebhook - (URI) The result webhook provided by the Engine Toolkit. The engine should submit results of the processing as soon as it can.
• externalCallbackWebhook (URI) Engines may pass this callback URI to an external entity performing the real processing.
• maxTTL - (int) The maximum time that Engine Toolkit can wait for results from the engine.

If the AIWARE_CONTROLLER environment variable is exposed to the engine, then this will be set:

• controllerToken - (string) if AIWARE_CONTROLLER is exposed to engine, this will require an edgeToken to use

Chunk metadata

Input chunk metadata can be made available to the engine as part of the fields for the /process webhook. The names for the metadata fields are prefixed with chunk-metadata- to minimize name collision with other data.

The values will be strings. Therefore the engine should perform type conversion as needed.

Process webhook response

The handler for the Process webhook should return the results by writing a JSON response in AION if applicable. If the engine returns content in different format such as TTML for transcripts, then the engine must set the MIME type and send the response.

[Note] Most languages and frameworks have very easy ways of consuming HTTP endpoints and writing JSON responses. It is recommended that you use existing libraries where possible.

If the engine cannot return the result synchronously (in the same HTTP request as a response), then the engine must return a JSON payload as defined below:

• estimatedProcessingTimeInSeconds - (int) Amount of seconds that the engine estimates to produce some results.

The suggested value for estimatedProcessingTimeInSeconds is maxTTL - or more if maxTTL is insufficient. For example, if the engine is to call out to external entity for processing and thus may require more than hours to process.

At that point the engine should use the supplied heartbeatWebhook and resultWebhook endpoints.

Example webhook response: faces

The following JSON is an example showing some faces that were found in the image.

{
    "series": [{
        "startTimeMs": 1000,
        "stopTimeMs": 2000,
        "object": {
            "type": "face",
            "confidence": 0.95,
            "boundingPoly": [
                {"x":0.3,"y":0.1}, {"x":0.5,"y":0.1},
                {"x":0.5,"y":0.9}, {"x":0.3,"y":0.9}
            ]
        }
    }, {
        "startTimeMs": 5000,
        "stopTimeMs": 6000,
        "object": {
            "type": "face",
            "confidence": 0.95,
            "boundingPoly": [
                {"x":0,"y":0}, {"x":1,"y":0},
                {"x":1,"y":1}, {"x":0,"y":1}
            ]
        }
    }]
}

• series - (array) List of items found
• series[].startTimeMs - (int) The start time of the chunk
• series[].stopTimeMs - (int) The end time of the chunk
• series[].object - (object) An object describing what was found
• series[].object.type - (string) The type of the object
• series[].object.confidence - (number) A number 0-1 of how confident the engine is about this object
• series[].object.boundingPoly - (array) Array of points that describe the region within a larger image.

Ignoring chunks

If your engine is not going to process a chunk, the Process webhook should return a 204 No Content response.

The Engine Toolkit will report the chunk as ignored.

Failed responses

If the chunk cannot be processed, the webhook should return a non-200 response code (e.g. 500) and a meaningful error should be written as the response.

[Note] There is no need to return a JSON body on failures, plain text is fine.

Engine Toolkit webhooks

The Engine Toolkit listens for a response from an engine using the heartbeat webhook and posts the payload to the process.

Heartbeat webhook

Engines performing the processing asynchronously such as stream or batch engines should submit heartbeats to the Engine Toolkit on a regular interval (suggested: once every minute). The webhook URI can be discovered in the heartbeatWebhook field of the original /process request.

Request

The POST request to the heartbeatWebhook should have the following data as a JSON payload:

• status - (string) Required. Possible values: running, complete, failed.
• bytesRead - (int64) Optional
• bytesWritten - (int64) Optional
• messagesWritten - (int64) Optional
• infoMs - (string) Optional. Informational message about the running status of the engine.
• failureReason (enum)| Required when status is set to failed. Possible values:
  internal_error
  external_error
  unknown
  url_not_found
  url_not_allowed
  url_timeout
  url_connection_refused
  url_error
  invalid_data
  rate_limited
  api_not_allowed
  api_authentication_error
  api_not_found
  api_error
  file_write_error
  stream_read_error
  system_dependency_missing
  system_error
  heartbeat_timeout
  other
• failureMsg (string) Optional: supplemental info for failureReason

Response

Engine Toolkit will return with 204 No Content

Result webhook

Whenever the engine has a complete unit of processed work for the task, it should submit the result back to Engine Toolkit for persistence. The webhook is given to the engine in the resultWebhook field of the /process request.

This is useful in the following scenarios:

• Result takes a long time to process
• Result is non-AION
• Result requires chunk metadata
• Result requires a specific output folder

Request

The POST request to the resultWebhook should have the following data as a JSON payload:

• chunkContext - (string) Optional. This is the value of the chunkContext as passed into the request of the /process. It should be present if the engine is a chunk engine.
• startOffsetMs - (int64) Optional. Offset of the start of the chunk from the beginning of the input stream for which this result was produced.
• endOffsetMs - (int64) Optional. Offset of the end of the chunk from the beginning of the input stream for which this result was produced.
• output - (JSON) Required. AION of the result. See Engine output standard for engine AION output format.

If the engine wants control over which outputFolder gets the chunk, it should pass in:

• outputFolderID - (string) The ID of the output folder to send the chunk. If this is not specified, all output folders will get the chunk.

If the engine wants to specify additional metadata (all metadata on the incoming chunk is copied to output), the following must be added:

• chunk-metadata-xxx - (string) xxx will be the key for the metadata with the value specified as a string.

If the engine wants to send binary or non-AION standard, then the engine must set multipart to true and use multi-part uploads. If more than one file is uploaded, multiple chunks will be created.

• multipart - (bool) Optional. Default is false, set to true to enable chunks sent as multipart.

For example:

POST <http://localhost:1234/result/task1234>
{ 
  "index" : 1,
  "startOffsetMs": 0,
  "endOffsetMs": 1200,
  "output" : {"series": [
    {
      "startTimeMs": 0,
      "stopTimeMs": 300,
      "words": [
        {
          "word": "this"
        }
      ]
    },
    {
      "startTimeMs": 300,
      "stopTimeMs": 500,
      "words": [
        {
          "word": "is"
        }
      ]
    },
    {
      "startTimeMs": 500,
      "stopTimeMs": 800,
      "words": [
        {
          "word": "a"
        }
      ]
    },
    {
      "startTimeMs": 800,
      "stopTimeMs": 1200,
      "words": [
        {
          "word": "sentence"
        }
      ]
    }
  ]}
}

Response

Chunk engines that produce a JSON response for AION will continue to do so.

For stream and batch engines with async or stand-alone processing mode, it is required that the following be returned in the JSON response. The suggested value for estimatedProcessingTimeInSeconds is maxTTL - or more if maxTTL is insufficient. For example, if the engine is to call out to external entity for processing and thus may require more than hours to process.

Testing your webhooks

Webhooks are HTTP endpoints, so you can test them by HTTP requests directly to your own code.

To manually test the webhooks, access the built-in Engine Toolkit Test Console, a web-based tool that lets you simulate the HTTP requests that your engine receives in production. You can upload your own file to process and use the web form to tune the parameters that your engine expects to support.

To access the test console, run your Docker container with the docker run command. Set the environment variable VERITONE_TESTMODE=true and expose port 9090 with the -p 9090:9090 argument:

-e "VERITONE_TESTMODE=true" -p 9090:9090

For example, you might run:

docker build -f Dockerfile -t your-engine .
docker run -e "VERITONE_TESTMODE=true" -p 9090:9090 -p 8080:8080 --name your-engine -t your-engine

Once the container is running, open a browser at http://localhost:9090/ to access the Engine Toolkit Test Console and use it to interact with your engine.

[Warn] Don't set the VERITONE_TESTMODE environment variable in your Dockerfile because there's a risk it could be deployed to production in test mode, which prevents your engine from working.

• Read more about the Veritone Engine Toolkit
• Deploy an engine
• Download Docker
• For an overview of building an engine, see Writing engines.

• Engine webhooks
• Engine Toolkit webhooks