View logs for your Ray cluster on Vertex AI

When you perform tasks with your Ray cluster on Vertex AI, tracking logs are automatically generated and stored in both Cloud Logging and open source Ray. This section describes how to access the generated logs through the Google Cloud console.

Before you begin, make sure to read the Ray on Vertex AI overview and set up all the prerequisite tools you need.

Cloud Logging console

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Logs Explorer:

    Go to Logs Explorer

  2. Select an existing Google Cloud project, folder, or organization.

  3. To display all Ray logs, enter the following query into the query-editor field, and then click Run query:

    resource.labels.task_name="ray-cluster-logs"

  4. To narrow down the logs to a specific Ray cluster, add the following line to the query and then click Run query:

    labels."ml.googleapis.com/ray_cluster_id"=CLUSTER_NAME

    where CLUSTER_NAME is the name for your Ray cluster.

  5. To further narrow down the logs to a specific log file like raylet.out, click the name of the log under Log fields -> Log name.

  6. You can group similar log entries together:

    1. In the Query results, click a log entry to expand the log.

    2. In the jsonPayload, click the tailed_path value. A drop-down menu appears.

    3. Click Show matching entries.

Head node shell

You can view the open source Ray log files through the interactive shell on the Ray head node:

  1. In the Google Cloud console, go to the Ray on Vertex AI page.

    Go to the Ray on Vertex AI page

  2. In the row for the cluster you created, click more actions menu.

  3. Select Head node interactive shell. The shell opens in another tab.

  4. Navigate to the /var/log-storage/ directory:

    cd /var/log-storage/

  5. Access the Ray logs under the session_DATE/logs/ directory where DATE is the date you created the Ray cluster on Vertex AI.

What's next