To recognize content, cognitive engines need to be trained. Training usually involves showing the engine various examples of the objects you want the engine to detect.
This topic shows how to:
- Train a face detection engine to recognize Yoko Ono in images
- Test the engine to see if it can recognize Yoko Ono in a group photo
Steps
1. Create an empty library.
Log in to Veritone, then go to the Veritone GraphQL Sandbox and run this mutation to create an empty library object named Rock Stars in your organization.
mutation CreateLibrary {
createLibrary(input: {name: "Rock Stars", libraryTypeId: "people"}) {
id
}
}
GraphQL returns a response like this.
{
"data": {
"createLibrary": {
"id": "23002de2-5af8-4f92-9357-9558f8362399"
}
}
}
Save the id.
2. Create an empty entity in that library.
Paste this mutation, replace the libraryid with the id from the previous step, and then run the mutation to create an entity object named Yoko. An entity object is a container that stores various examples of the entity.
mutation createEntity {
createEntity(input: {
libraryId: "b370ecab-88c0-4e2b-ae9c-7739a7ed16c7",
name: "Yoko"}) {
id
}
}
The response looks like this.
{
"data": {
"createEntity": {
"id": "37f7ec3f-5e30-4d30-a150-ae5a9f0cb501"
}
}
}
Save the id.
3. Add content (assets) to the entity
Paste this mutation, replace the entityId with the id from the previous step and the url with the URL of an image to use for training, then run the mutation to add the image to your entity.
mutation createEntityIdentifier {
createEntityIdentifier(input: {
entityId:"7c7b08e6-6b72-48af-b768-043f023e15cc",
identifierTypeId: "face",
contentType:"image/jpeg",
storeReference:true,
url:"https://i.pinimg.com/originals/a0/cc/e0/a0cce07f699ef0cd36cba57ce4cd9c51.jpg"
}) {
id
}
}
[Note] It's important to set the storeReference flag to true. This allows the image to be specified by reference, rather than by value. If the flag's default value of false is used, the mutation will fail unless it is POSTed in conjunction with an actual file upload -- something not currently possible in the GraphQL Sandbox. To perform a simultaneous POST mutation/upload, use cURL, Postman, or custom code.
When training an engine, it helps to have more than one exemplar in the entity collection. To add more exemplars, run the above mutation multiple times, substituting a new url value each time.
To see how the library looks, go to Veritone Library, select Rock Stars, and then select Yoko.
4. Publish the library
Paste this mutation, replace id with the id of the library, and run the mutation to publish the library and begin training all of the engines that can analyze the contentType that are listed in the createEntityIdentifier mutation.
mutation publishLibrary {
publishLibrary(id: "b370ecab-88c0-4e2b-ae9c-7739a7ed16c7") {
name
entities {
records {
name
id
identifiers {
records {
url
}
}
}
}
}
}
[Note] When a library is published, it trains all the engines that can analyze the contentType that were listed in the createEntityIdentifier mutation. Publishing a library also creates or updates the engineModel objects containing the engines' learnings.
5. Poll the status of training tasks
Paste this query, replace the id with the id of the library, and run the mutation to see the training status of the various engineModel objects created during training.
query GetLibraryInfo {
library(id:"b370ecab-88c0-4e2b-ae9c-7739a7ed16c7") {
engineModels {
records {
id
trainStatus
engine {
name
id
}
}
}
}
}
The response looks like this.
{
"data": {
"library": {
"engineModels": {
"records": [
{
"id": "80960853-97d9-45a0-a339-978609359d42",
"trainStatus": "complete",
"engine": {
"name": "Face Recognition - L - V2F",
"id": "b74d4058-90f6-453a-9636-5982e34abe0c"
}
},
{
"id": "0cf73642-3c92-41b0-812e-4c8fe8129e67",
"trainStatus": "complete",
"engine": {
"name": "Face Recognition - F - V2F",
"id": "434db220-b38f-4bb6-bf65-02a4469559ac"
}
},
{
"id": "dc0753c6-7180-4e87-9366-016b0bc91d8c",
"trainStatus": "complete",
"engine": {
"name": "Face Recognition - C",
"id": "799155d9-7d48-4b4c-b98f-501e2894abb0"
}
},
{
"id": "5d91fe57-8b59-43fb-a7c2-d2934bfbf0eb",
"trainStatus": "complete",
"engine": {
"name": "Face Recognition - B",
"id": "3f115b93-97be-46f0-b0f2-7460db15ec34"
}
},
{
"id": "9a30d2c3-5743-48b8-b39a-3c6618f5dd40",
"trainStatus": "complete",
"engine": {
"name": "Face Recognition - M - V2F",
"id": "fa7e75da-5955-476a-b20c-28e286fdfd8e"
}
},
{
"id": "2c3a1855-7a29-457b-b8cc-3e4a869a130e",
"trainStatus": "complete",
"engine": {
"name": "Face Recognition - I - V2F",
"id": "dcef5300-5cc1-4fe3-bd8f-5c4d3a09b281"
}
},
{
"id": "2326f4d9-f501-4e75-988c-fa081b591d8b",
"trainStatus": "complete",
"engine": {
"name": "Face Similarity - A - V2F",
"id": "bab908d5-1eb0-4b94-9b0c-5c4bb6a81d78"
}
}
]
}
}
}
}
[Note] Run the query as many times as needed until all trainStatus fields are shown as "complete". Do not attempt to run a job on a trained engine if the status is not yet shown as "complete". The wait time for training depends on many factors, but is generally seconds, not minutes or hours.
6. Test the trained engine
Once all the trainStatus fields return "complete", paste this mutation, update the libraryId and libraryEngineModelId, and run the mutation. The mutation uses the library you trained to run a face recognition job on a group photo of Yoko Ono.
[Note] When target is used at the beginning of the mutation instead of targetId, aiWARE will create a TDO (Temporal Data Object) for the job so a TDO doesn't have to pass to the job.
mutation runFacialRecognitionImage {
createJob(input: {
target: {
startDateTime: 1548432520
stopDateTime: 1548436341
name: "Grammys pic by Gruen"
}
clusterId: "prd5-21xbyq0x-4h0s-o685-snas-oovhdai552v9"
tasks: [
{
engineId: "9e611ad7-2d3b-48f6-a51b-0a1ba40fe255"
payload: {
url: "https://northerntransmissions.com/wp-content/uploads/2013/02/07928_184147_johnyokolennonPAPHOTOS_01.jpg"
}
ioFolders: [
{
referenceId: "wsaOutput"
mode: stream
type: output
}
]
executionPreferences: {
priority: -5
}
},
{
engineId: "75fc943b-b5b0-4fe1-bcb6-9a7e1884257a" # SI2 Asset Creator, needed for adding image as primary asset
payload: {
setAsPrimary: true
}
ioFolders: [
{
referenceId: "assetCreatorInput"
mode: stream
type: input
}
]
executionPreferences: {
parentCompleteBeforeStarting: true
priority: -5
}
},
{
engineId: "8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440"
payload: {
ffmpegTemplate: "rawchunk"
}
ioFolders: [
{
referenceId: "chunkImageInput"
mode: stream
type: input
},
{
referenceId: "chunkImageOutput"
mode: chunk
type: output
}
]
executionPreferences: {
parentCompleteBeforeStarting: true
priority: -5
}
},
{
engineId: "df6e189f-8947-4c73-a30b-f786defc60e8" # Facebox Recognize (library training), requires image input
payload: {
libraryId: "3659d77f-0a73-4ede-ab6c-ea51500c84cf" # Library containing trained faces
libraryEngineModelId: "7b282445-7d52-4cfa-a27a-030040076191" # Library Engine Model representing the training data
}
ioFolders: [
{
referenceId: "faceInput"
mode: chunk
type: input
},
{
referenceId: "faceOutput"
mode: chunk
type: output
}
]
executionPreferences: {
maxEngines: 10
parentCompleteBeforeStarting: true
priority: -5
}
},
{
engineId: "8eccf9cc-6b6d-4d7d-8cb3-7ebf4950c5f3"
ioFolders: [
{
referenceId: "owInput"
mode: chunk
type: input
}
]
executionPreferences: {
parentCompleteBeforeStarting: true
priority: -10
}
}
]
routes: [
{
parentIoFolderReferenceId: "wsaOutput"
childIoFolderReferenceId: "assetCreatorInput"
},
{
parentIoFolderReferenceId: "wsaOutput"
childIoFolderReferenceId: "chunkImageInput"
},
{
parentIoFolderReferenceId: "chunkImageOutput"
childIoFolderReferenceId: "faceInput"
},
{
parentIoFolderReferenceId: "faceOutput"
childIoFolderReferenceId: "owInput"
}
]
}) {
id
targetId
clusterId
tasks {
records{
id
engineId
payload
taskPayload
status
output
ioFolders {
referenceId
type
mode
}
}
}
routes {
parentIoFolderReferenceId
childIoFolderReferenceId
}
}
}
In the response, copy the id, which is the job ID. It looks like "22072921_UFGlQ3BfYU".
Paste this query, replace the id and jobId with the job ID, then run the query to check the results of the job and the engine.
query PollJob {
job(id: "19104217_A517gxCHlk") {
status
targetId
tasks {
count
records {
status
engine {name}
taskOutput
startedDateTime
completedDateTime
}
}
}
engineResults(jobId: "19104217_A517gxCHlk") {
records {
assetId
tdoId
jsondata
}
}
}
The response contains jsondata that includes an array of results, each with an associated confidence value. Every face in the image will be detected, but most will have a confidence of zero and no associated label. If Yoko Ono is detected, one of the hits will give a strong confidence value, an entityId that matches Yoko's entity ID, and a label of Yoko. For example:
"startTimeMs": 0,
"stopTimeMs": 0,
"entityId": "ea569a53-ca59-42dc-a99d-b21eb29cda69",
"libraryId": "3659d77f-0a73-4ede-ab6c-ea51500c84cf",
"object": {
"label": "Yoko",
"type": "face",
"entityId": "ea569a53-ca59-42dc-a99d-b21eb29cda69",
"libraryId": "3659d77f-0a73-4ede-ab6c-ea51500c84cf",
"confidence": 1,
"emotions": [
{
"emotion": "CALM",
"emotionConfidence": 0.6895350646972657
This result tells us that the engine is 100% confident that the face at the bounding polygon in question is Yoko Ono. Going to the uri in your browser will download the image, which looks like this:
