Extensions API

Extensions are tools for large language models (LLMs) to access external data, run computations, and perform other operations. They can process real-time data and perform real-world actions.

Vertex AI provides the Extension API that can register, manage, and execute extensions. Vertex AI also provides a set of prebuilt extensions from the Extension API including the code interpreter extension and Vertex AI Search extension.

Limitations

The Extension API is only available in the us-central1 region.

Example syntax

Syntax to create an extension resource.

curl

curl -X POST \

-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \

http://${LOCATION}-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/${LOCATION}/extensions:import \
-d '{
  "displayName": "...",
  "description": "...",
  "manifest": {
    ...
    "apiSpec": {
      ...
    },
    "authConfig": {
      ...
    }
    ...
  }
}'

Python

from vertexai.preview import extensions

extensions.Extension.create(
  manifest: Union[JsonDict, ExtensionManifest],
  display_name: Optional[str] = None,
  description: Optional[str] = None,
  runtime_config: Optional[Union[JsonDict, RuntimeConfig]] = None
)

Parameter list

Request body

Parameters

displayName

Optional: string

The display name of the extension that is displayed to users from the API and UI. This should be a UTF-8 string up to 128 characters.

description

Optional: string

The description of the extension that is displayed to users from the API and UI. This should be a UTF-8 string up to 1MB.

manifest

JsonDict|ExtensionManifest

The manifest of the extension.

runtimeConfig

Optional: JsonDict|RuntimeConfig

The runtime configuration that controls the runtime behavior of the extension.

For the code interpreter extension, the format is the following:

  "runtimeConfig": {
    "codeInterpreterRuntimeConfig": {
        "fileInputGcsBucket": string,
        "fileOutputGcsBucket": string
    }
  }

For the Vertex AI Search extension, the format is the following: 

  "runtimeConfig": {
    "vertexAiSearchRuntimeConfig": {
      "servingConfigName": string,
    }
  }

manifest

The manifest of the extension.

Parameters

name

string

The name of the extension that is used by the LLM for reasoning. This should be a UTF-8 string up to 128 characters.

description

string

The natural language description of the extension's usage. The description is shown to the LLM to help it perform reasoning.This should be a UTF-8 string up to 1MB.

apiSpec

ApiSpec

The API specification shown to the LLM for reasoning. You should provide a meaningful and informative description. apiSpec contains the reference to the Cloud Storage URI storing the OpenAPI yaml file. 

  "apiSpec": {
    "openApiGcsUri": string
  }

authConfig

JsonDict|AuthConfig

The type of auth supported by this extension.

An extension import request must contain an authentication configuration.

"authConfig": {
  "authType": "GOOGLE_SERVICE_ACCOUNT_AUTH",
  "googleServiceAccountConfig": {
    "serviceAccount": string
  },
}

apiSpec

The API specification shown to the LLM for reasoning.

Parameters

openApiGcsUri

string

Cloud Storage URI of the OpenAPI YAML file describing the extension API, such as gs://vertex-extension-public/code_interpreter.yaml

authConfig

The type of authentication supported by this extension.

Parameters

authType

string

Authentication method. Supported values: GOOGLE_SERVICE_ACCOUNT_AUTH.

googleServiceAccountConfig

The code interpreter extension and Vertex AI Search extension only support Google Service Account authentication, where Vertex AI uses the Vertex AI Extension Service Agent to access the APIs.

Parameters

serviceAccount

Optional: string

The service account that the extension execution runs as. If the service account is specified, the iam.serviceAccounts.getAccessToken permission should be granted to the Vertex AI Extension Service Agent on the specified service account. If not specified, the Vertex AI Extension Service Agent is used to execute the extension.

runtimeConfig

The runtimeConfig object contains additional configurations used when running the extension.

Code interpreter extension

Parameters

fileInputGcsBucket

Optional: string

The Cloud Storage bucket for the file input to the extension. The Vertex Extension Custom Code Service Agent should be granted the roles/storage.objectViewer permission to this bucket. If not specified, the extension only accepts file contents from the request body and rejects Cloud Storage file inputs.

fileOutputGcsBucket

Optional: string

The Cloud Storage bucket for the file output from the extension. The Vertex Extension Custom Code Service Agent should be granted the roles/storage.objectUser permission to this bucket. If not specified, the file content is output in the response body.

Vertex AI Search extension

Parameters

servingConfigName

string

Vertex AI Search serving config name to specify which Vertex AI Search resource the extension uses. Format:

projects/{project}/locations/{location}/collections/{collection}/engines/{engine}/servingConfigs/{serving_config}

Execute extension

Parameters

operation_id

string

The selected ID of the operation to be executed in this extension.

operation_params

Optional: JsonDict|Struct

Request parameters that are used for executing this operation. The JSON should be in the form of a map with parameter name as the key and the actual parameter value as the value. For example, to set a parameter called query to the string "What is Vertex AI?", you can use {"query": "What is Vertex AI?"}.

Examples

Import a code interpreter extension

Create or register an extension resource.

This example shows how to import a code interpreter extension.

REST

Before using any of the request data, make the following replacements:

  • PROJECT_ID: Your project ID.
  • LOCATION: The region to process the request.
  • DISPLAY_NAME: The display name of the extension that is displayed to users from the API and UI. This should be a UTF-8 string up to 128 characters.
  • DESCRIPTION: The description of the extension that is displayed to users from the API and UI. This should be a UTF-8 string up to 1MB.
  • MANIFEST_NAME: The name of the extension that is used by the LLM for reasoning. This should be a UTF-8 string up to 128 characters
  • MANIFEST_DESCRIPTION: The natural language description shown to the LLM. It should describe the usage of the extension, and is essential for the LLM to perform reasoning. This should be a UTF-8 string up to 1MB.
  • GCS_URI: Cloud Storage URI of the OpenAPI YAML file describing the extension API.
  • AUTH_TYPE: Authentication method. Supported values: GOOGLE_SERVICE_ACCOUNT_AUTH.

HTTP method and URL: 

POST http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions:import

Request JSON body: 

{
  "displayName": "DISPLAY_NAME",
  "description": "DESCRIPTION",
  "manifest": {
    "name": "NAME",
    "description": "MANIFEST_DESCRIPTION",
    "apiSpec": {
      "openApiGcsUri": "GCS_URI",
    },
    "authConfig": {
      "authType": "AUTH_TYPE",
      "googleServiceAccountConfig": {}
    }
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions:import"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions:import" | Select-Object -Expand Content

Python

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Python API reference documentation. 

import vertexai
from vertexai.preview import extensions

# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"

vertexai.init(project=project_id, location="us-central1")

extension = extensions.Extension.create(
    display_name="Code Interpreter",
    description="This extension generates and executes code in the specified language",
    manifest={
        "name": "code_interpreter_tool",
        "description": "Google Code Interpreter Extension",
        "api_spec": {
            "open_api_gcs_uri": "gs://vertex-extension-public/code_interpreter.yaml"
        },
        "auth_config": {
            "google_service_account_config": {},
            "auth_type": "GOOGLE_SERVICE_ACCOUNT_AUTH",
        },
    },
)

Import with a runtime configuration

Create or register an extension resource.

This example shows to import a Vertex AI Search extension by specifying RuntimeConfig.

REST

Before using any of the request data, make the following replacements:

  • PROJECT_ID: Your project ID.
  • LOCATION: The region to process the request.
  • DISPLAY_NAME: The display name of the extension that is displayed to users from the API and UI. This should be a UTF-8 string up to 128 characters.
  • DESCRIPTION: The description of the extension that is displayed to users from the API and UI. This should be a UTF-8 string up to 1MB.
  • MANIFEST_NAME: The name of the extension that is used by the LLM for reasoning. This should be a UTF-8 string up to 128 characters
  • MANIFEST_DESCRIPTION: The natural language description shown to the LLM. It should describe the usage of the extension, and is essential for the LLM to perform reasoning. This should be a UTF-8 string up to 1MB.
  • GCS_URI: Cloud Storage URI of the OpenAPI YAML file describing the extension API.
  • AUTH_TYPE: Authentication method. Supported values: GOOGLE_SERVICE_ACCOUNT_AUTH.
  • SERVING_CONFIG_NAME: Vertex AI Search serving config name to specify which Vertex AI Search resource the extension uses. Format: projects/{project}/locations/{location}/collections/{collection}/engines/{engine}/servingConfigs/{serving_config}

HTTP method and URL: 

POST http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions:import

Request JSON body: 

{
  "displayName": "DISPLAY_NAME",
  "description": "DESCRIPTION",
  "manifest": {
    "name": "NAME",
    "description": "MANIFEST_DESCRIPTION",
    "apiSpec": {
      "openApiGcsUri": "GCS_URI",
    },
    "authConfig": {
      "authType": "AUTH_TYPE",
      "googleServiceAccountConfig": {}
    },
    runtime_config={
      "vertex_ai_search_runtime_config": {
          "serving_config_name": SERVING_CONFIG_NAME,
      }
    }
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions:import"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions:import" | Select-Object -Expand Content

Python

import vertexai
from vertexai.preview import extensions

vertexai.init(project=PROJECT_ID, location=LOCATION)

extension_vertex_ai_search = extensions.Extension.create(
    display_name = "vertex_ai_search",
    description = "This extension search from provided datastore",
    manifest = {
        "name": "vertex_ai_search",
        "description": "Google Vertex AI Search Extension",
        "api_spec": {
            "open_api_gcs_uri": "gs://vertex-extension-public/vertex_ai_search.yaml"
        },
        "auth_config": {
            "google_service_account_config": {},
            "auth_type": "GOOGLE_SERVICE_ACCOUNT_AUTH",
        },
    },
    runtime_config={
        "vertex_ai_search_runtime_config": {
            "serving_config_name": SERVING_CONFIG_NAME,
        }
    }
)

Execute an extension

To execute an extension, directly call the extension and provide the execution parameters in the request.

This example executes the code interpreter extension generate_and_execute to get the answer for the query find the max value in the list: [1,2,3,4,-5].

REST

Before using any of the request data, make the following replacements:

  • PROJECT_ID: Your project ID.
  • LOCATION: The region to process the request.
  • EXTENSION_ID: The ID of the extension.
  • OPERATION_ID: The selected ID of the operation to be executed in this extension.
  • QUERY: The Request parameters to execute the operation in a Key-Value format, {"query": "What is Vertex AI?"}.

HTTP method and URL: 

POST http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID:execute

Request JSON body: 

{
  "operation_id": "OPERATION_ID",
  "operation_params": {
    "query": "QUERY",
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID:execute"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID:execute" | Select-Object -Expand Content

Python

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Python API reference documentation. 

import vertexai
from vertexai.preview import extensions

# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"
# extension_id = "EXTENSION_ID"

vertexai.init(project=project_id, location="us-central1")

extension = extensions.Extension(extension_id)
response = extension.execute(
    operation_id="generate_and_execute",
    operation_params={"query": "find the max value in the list: [1,2,3,4,-5]"},
)
print(response)

List Extensions

List extensions in a project.

REST

Before using any of the request data, make the following replacements:

  • PROJECT_ID: Your project ID.
  • LOCATION: The region to process the request.
  • EXTENSION_ID: The ID of the extension.

HTTP method and URL: 

GET http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions

To send your request, choose one of these options:

curl

Execute the following command: 

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions"

PowerShell

Execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions" | Select-Object -Expand Content

Python

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Python API reference documentation. 

import vertexai
from vertexai.preview import extensions

# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"

vertexai.init(project=project_id, location="us-central1")

extensions = extensions.Extension.list()
print(extensions)

Get an extension

Get details of an extension.

REST

Before using any of the request data, make the following replacements:

  • PROJECT_ID: Your project ID.
  • LOCATION: The region to process the request.
  • EXTENSION_ID: The ID of the extension.

HTTP method and URL: 

GET http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID

To send your request, choose one of these options:

curl

Execute the following command: 

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID"

PowerShell

Execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID" | Select-Object -Expand Content

Python

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Python API reference documentation. 

import vertexai
from vertexai.preview import extensions

# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"
# extension_id = "EXTENSION_ID"

vertexai.init(project=project_id, location="us-central1")

extension = extensions.Extension(extension_id)
print(extension)

Update an extension

Update an extension.

REST

Before using any of the request data, make the following replacements:

  • PROJECT_ID: Your project ID.
  • LOCATION: The region to process the request.
  • EXTENSION_ID: The ID of the extension.
  • UPDATE_MASK: The parameter to updated. Accepted values, displayName, description, or toolUseExamples.

HTTP method and URL: 

PATCH http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID?update_mask="UPDATE_MASK"

Request JSON body: 

{
  "description": "UPDATE_MASK",
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X PATCH \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID?update_mask="UPDATE_MASK""

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method PATCH `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID?update_mask="UPDATE_MASK"" | Select-Object -Expand Content

Delete Extension

Delete an extension.

This example deletes the extension associated with the extension ID.

REST

Before using any of the request data, make the following replacements:

  • PROJECT_ID: Your project ID.
  • LOCATION: The region to process the request.
  • EXTENSION_ID: The ID of the extension.

HTTP method and URL: 

DELETE http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID

To send your request, choose one of these options:

curl

Execute the following command: 

curl -X DELETE \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID"

PowerShell

Execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method DELETE `
    -Headers $headers `
    -Uri "http://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/extensions/EXTENSION_ID" | Select-Object -Expand Content

Python

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Python API reference documentation. 

import vertexai
from vertexai.preview import extensions

# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"
# extension_id = "EXTENSION_ID"

vertexai.init(project=project_id, location="us-central1")

extension = extensions.Extension(extension_id)
extension.delete()

What's next