Apache Kafka for BigQuery pricing

Preview notice

Apache Kafka for BigQuery is not free to use during the Preview.

Pricing

The cost of operating an Apache Kafka for BigQuery consists of charges related to running Kafka clusters and Private Service Connect (PSC) fees for access to the cluster.

Apache Kafka for BigQuery cluster charges

Cluster charges will be billed using three SKUs in the region where you run the cluster:

• Compute, covering vCPU and RAM
• Storage, including local storage of data on brokers (Persistent Disk) and long term storage.
• Networking, covering charges for inter-zone data transfer for replication of data within a cluster.

Understanding Apache Kafka for BigQuery slots

The vCPU and RAM usage is reported in terms of abstract units called slots. A simple way to think about this unit is that a single vCPU with 4GB of RAM corresponds to 1.5 slots. This means for example, that a 6 vCPU cluster with 4GiB of RAM per vCPU would cost 6 x (1 vCPU + 4GiB) * 1.5 slots * 6 cents / hour / slot = 54 cents / hour in us-central1.

You can, however, use different amounts of RAM per vCPU in your clusters. The conversion factors from compute resources to slots are

This means that when you use 1 GiB of RAM for an hour, you pay for 0.125 slots for that hour.

Note that an Apache Kafka for BigQuery slot is a unit analogous to but distinct from BigQuery slots. The two have the same price to simplify comparison.

Understanding the storage costs

The local storage is provisioned automatically. As a rough guide, you should expect to pay for 100GB of local storage for every vCPU you provision.

The cost of persistent storage is determined by how much data you retain in your topics. You do not pay for provisioning persistent storage capacity. You are billed for only a single replica of data in persistent storage. This is independent of the cluster configuration. For example, if you publish 1GiB of data and retain it for 1 day, you will pay for 1GiB-day of persistent storage.

In contrast, local storage is used by every replica of the data. So if you publish 1 GiB of data you will typically use 3 GiB of local disk space. Persistent storage charges are based on the total volume of data stored, while local storage charges are based on the volume of provisioned storage. Therefore, the actual volume of data stored in local storage does not affect what you pay for local storage.

PSC charges

Apache Kafka for BigQuery service sets up all PSC endpoints for your service. These endpoints are resources in a project you own and will be billed separately, as PSC charges. The PSC charges include

• Hourly charge for each endpoint (minimum of four per cluster)
• Consumer data processing charges

You should expect to pay $0.01/hour for each endpoint and $0.004-$0.01/GiB for the data published by producers and read by the consumers. Please see the subsection on "Using a Private Service Connect endpoint (forwarding rule) to access a published service" in the "Private Service Connect" section of the VPC pricing guide for the most up-to-date information on the pricing site.

Total cost of ownership

The total cost of running an Apache Kafka for BigQuery cluster will depend on the size of the cluster, the volume of data you publish and consume, as well as the duration for which you store the data. In addition, an important parameter to consider is the average utilization of your cluster. The utilization determines the balance of data transfer charges and compute charges. A rough guide, you can assume that a single vCPU can handle 20MiBps of publish traffic and 80MiBps of consumer traffic.

For example, consider the estimates of the total cost of a 100MB/s workload on Apache Kafka for BigQuery and Pub/Sub in us-central1, assuming 40% average CPU utilization and 24 hour message storage. For comparison we also provide an estimate of what it would take to serve the same workload by running an open source Apache Kafka cluster on Compute Engine using N2 CPUs and local replica followers for consumers.

Note that the relatively high change in the price with fanout is due to the PSC data transfer charges.