SI2 Audio/Video Chunk Creator

The SI2 adapter takes audio or video input and outputs chunks of that media. It also takes images or text data and outputs them as raw chunks, so the next engine task uses the same data. Additionally, this adapter takes video as input and outputs the audio track or frames as images.

Use cases

If you have a URL for an image and need an image chunk:

engineId: "8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440"
payload: {
  url: 
  ffmpegTemplate: "rawchunk"
}

This will copy the file from the URL and produce as output a "chunk" that is the file. This will not change the type of the file (i.e. if the URL is for a TIFF, then the output will be a TIFF).

If you have a URL for audio and need:

An audio chunk:

payload: {
  url: 
  ffmpegTemplate: "audio"
  customFFMPEGProperties: { chunkSizeInSeconds: "300" }
}

If you have a URL for a video and need:

An image chunk:

engineId: "8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440"
payload: {
  url: 
  ffmpegTemplate: "frame"
  customFFMPEGProperties: { framesPerSecond: "1" }
}

This will take any video (that ffmpeg supports) and produce multiple image/jpeg files.

The value of framesPerSecond is a fraction. For example, you can request one frame every 2 seconds with:

framesPerSecond : 0.5

The default value of framesPerSecond is 1.0, or one frame every second.

The output file type can be changed by supplying the desired type, like customFFMPEGProperties: { outputFormat: "png" }; however, that should rarely be necessary, as all downstream engines accept image/jpeg as input.

An audio chunk:

engineId: "8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440"
payload: {
  url: 
  ffmpegTemplate: "audio"
  customFFMPEGProperties: { chunkSizeInSeconds: "300" }
}

This will take any video (that ffmpeg supports) and produce multiple audio/wav files in 5 minute durations each.

The produced files will have a duration specified by chunkSizeInSeconds if the video is longer than the specified chunk size. If the video is shorter than the chunk size, then only one file will be produced. The default chunk length is 5 minutes for files, and 15 minutes for streams.

The output file type can be changed by supplying the type like customFFMPEGProperties: { outputFormat: "mp3" }, however this shouldn't be necessary as all cognition engines (except one) accept audio/wav as input. The one exception is "Lip Voice Movement Correlation," which only accepts audio/webm.

A video chunk:

engineId: "8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440"
payload: {
  url: 
  ffmpegTemplate: "video"
  customFFMPEGProperties: { chunkSizeInSeconds: "300" }
}

This will take any video (that ffmpeg supports) and produce multiple video/mp4 files.

The produced files will have a duration specified by chunkSizeInSeconds if the video is longer than the specified chunk size. If the video is shorter than the chunk size, then only one file will be produced. The default chunk length is 2 minutes.

The output file type can be changed by supplying the desired type, like customFFMPEGProperties: { outputFormat: "wmv" }, however this shouldn't be necessary as all cognition engines accept video/mp4 as input.

Other uses

This adapter is extremely powerful in that it is basically a wrapper around ffmpeg, and by using ffmpegTemplate: "custom" you can provide any parameters to ffmpeg you want and make it do almost anything.

For example, to convert a media stream or chunk to mp3, you could use:

engineId: "8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440"
payload: {
  url: 
  ffmpegTemplate: "custom"
  customFFMPEGTemplate: "-map 0 -f "
  customFFMPEGProperties: { "outputFormat": "mp3" }
}

If you have any stream and need a chunk:

In the above situations, there was a URL parameter in the payload. This ingestor also works with a stream input instead of a URL. For instance, if you have a Webstream Adapter (WSA) pulling a stream from a URL, you can connect the output of the WSA to the input of the Stream Ingestor 2 (SI2), and then use any of the snippets above.

Example: If you have a WSA pulling a stream from a source video URL file, and you want to run both image and audio cognition on it, you can use two different SI2 engines connected to the output of the WSA, one with ffmpegTemplate:"frame" and the other with ffmpegTemplate:"audio", but neither with a URL value.

If you have a URL for a text file:

payload: {
  url: 
  ffmpegTemplate: "rawchunk"
}

This will copy the text file from the URL and produce as output a "chunk" that is the file. This will not change the type of the file or alter it in any way.

• Adapter engine reference list
• An overview of adapters

• Developer's reference
• Engines
• Writing engines
• Adapters
• Adapter engine references
• SI2 Audio/Video Chunk