Create and run a job that sends Pub/Sub status notifications

This document explains how to create a Batch job that sends Pub/Sub notifications. You can use Pub/Sub to get notifications when a job or task state changes, or when a job or task enters a specific state. For more information, see Monitor jobs using notifications.

Before you begin

Required roles

  • To get the permissions that you need to create and run a job that sends notifications, ask your administrator to grant you the following IAM roles:

    For more information about granting roles, see Manage access.

    You might also be able to get the required permissions through custom roles or other predefined roles.

  • Unless you are using the default configuration for the job's service account, ensure that it has the necessary permissions.

    To ensure that the job's service account has the necessary permissions to publish Pub/Sub notifications, ask your administrator to grant the job's service account the Pub/Sub Publisher (roles/pubsub.publisher) IAM role on your Pub/Sub topic.

  • If you want a job to publish notifications to a Pub/Sub topic that is in a different project than the job, then the Batch service agent for the job's project must be granted permission to publish to that topic.

    To ensure that the Batch service agent for the job's project has the necessary permissions to publish Pub/Sub notifications to a Pub/Sub topic in another project, ask your administrator to grant the Batch service agent for the job's project the Pub/Sub Publisher (roles/pubsub.publisher) IAM role on the Pub/Sub topic.

Create and run a job that sends notifications

You can create a Batch job that sends Pub/Sub notifications by doing the following:

Use the Google Cloud CLI or REST API to create a job that includes the notifications field and one or more jobNotification objects in the main body of the JSON file:

  "notifications": [
    {
      "pubsubTopic": "projects/PROJECT_ID/topics/TOPIC_ID",
      "message": {
        ATTRIBUTES
      }
    }
  ]

Replace the following:

  • PROJECT_ID: the project ID of the project that contains the Pub/Sub topic.
  • TOPIC_ID: the Pub/Sub topic ID of the topic you created when you enabled Pub/Sub notifications.

  • ATTRIBUTES: specify one or more of the following attributes, which each let you receive notifications about the state of the job or all of its tasks.

    • For notifications about all job state changes, specify the following:

      "type": "JOB_STATE_CHANGED"

    • For notifications about a specific job state change, specify the following:

      "type": "JOB_STATE_CHANGED",
"newJobState": "JOB_STATE"

      Replace JOB_STATE with one of the following job states:

      • QUEUED
      • SCHEDULED
      • RUNNING
      • SUCCEEDED
      • FAILED

      For more information about job states, see Job lifecycle.

    • For notifications about all task state changes, specify the following:

      "type": "TASK_STATE_CHANGED"

    • For notifications about specific task state changes, specify the following:

      "type": "TASK_STATE_CHANGED",
"newTaskState": "TASK_STATE"

      Replace TASK_STATE with one of the following task states:

      • PENDING
      • ASSIGNED
      • RUNNING
      • SUCCEEDED
      • FAILED

      For more information about task states, see Job lifecycle.

For example, suppose you want to receive notifications about all job state changes and any time a task fails. To do so, you can have a JSON configuration file that is similar to the following:

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "script": {
              "text": "echo Hello World! This is task $BATCH_TASK_INDEX."
            }
          }
        ]
      },
      "taskCount": 3,
    }
  ],
  "logsPolicy": {
      "destination": "CLOUD_LOGGING"
  },
  "notifications": [
    {
      "pubsubTopic": "projects/PROJECT_ID/topics/TOPIC_ID",
      "message": {
        "type": "JOB_STATE_CHANGED"
      }
    },
    {
      "pubsubTopic": "projects/PROJECT_ID/topics/TOPIC_ID",
      "message": {
        "type": "TASK_STATE_CHANGED",
        "newTaskState": "FAILED"
      }
    }
  ]
}

After the job starts running, you can use its notifications. For example, if the Pub/Sub topic for your job has a subscription that streams notifications to BigQuery, you can analyze the Pub/Sub notifications in BigQuery.

What's next