Skip to main content
Skip to main content
Edit this page

Monitoring ClickPipes

In addition to in-console monitoring, ClickPipes exposes metrics to a Prometheus-compatible endpoint for scraping. These metrics are published with other ClickHouse Cloud service metrics, and allow you to integrate ClickPipes monitoring with your existing observability stack (e.g., Grafana, Datadog).

Metric labels​

ClickPipes metrics use the standard service-level labels (clickhouse_org, clickhouse_service, clickhouse_service_name) plus the following ClickPipes-specific labels:

LabelDescription
clickpipe_idThe unique identifier of the ClickPipe
clickpipe_nameThe name of the ClickPipe
clickpipe_sourceThe source type (e.g., kafka, postgres, s3)
clickpipe_stateThe state of the ClickPipe (e.g., Stopped, Provisioning, Running)

Sample response​

The following shows what ClickPipes metrics look like in a Prometheus scrape response for a Kafka ClickPipe. The set of exposed metrics varies by pipe type — see Available metrics for which metrics apply to which pipe types:

# HELP ClickPipes_Info Always equal to 1. Label "clickpipe_state" contains the current state of the pipe: Stopped/Provisioning/Running/Paused/Failed
# TYPE ClickPipes_Info gauge
ClickPipes_Info{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent",clickpipe_state="Running"} 1

# HELP ClickPipes_FetchedEvents_Total Total number of records fetched from the source.
# TYPE ClickPipes_FetchedEvents_Total counter
ClickPipes_FetchedEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5535376

# HELP ClickPipes_SentEvents_Total Total number of records sent to ClickHouse
# TYPE ClickPipes_SentEvents_Total counter
ClickPipes_SentEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5534250

# HELP ClickPipes_FetchedBytes_Total Total uncompressed bytes fetched from the source.
# TYPE ClickPipes_FetchedBytes_Total counter
ClickPipes_FetchedBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 873286202

# HELP ClickPipes_SentBytes_Total Total uncompressed bytes sent to ClickHouse.
# TYPE ClickPipes_SentBytes_Total counter
ClickPipes_SentBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 477187967

# HELP ClickPipes_Errors_Total Total errors ingesting data.
# TYPE ClickPipes_Errors_Total counter
ClickPipes_Errors_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 0

# HELP ClickPipes_Latency Time in milliseconds between when the record was produced to when it was written to ClickHouse.
# TYPE ClickPipes_Latency gauge
ClickPipes_Latency{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 340

Available metrics​

Not every metric is emitted by every pipe type. In particular, CDC ClickPipes expose a different subset of metrics than streaming and object storage ClickPipes.

  • Streaming, object storage — Kafka, Kinesis, Pub/Sub, S3, and GCS
  • CDC — Postgres, MySQL, MongoDB and BigQuery
  • All — every pipe type

Data transfer​

MetricTypeAvailable forDescription
ClickPipes_FetchedBytes_TotalCounterStreaming, object storageTotal uncompressed bytes fetched from the source.
ClickPipes_FetchedBytesCompressed_TotalCounterAllTotal bytes fetched from the source, as read over the wire. If the source data is uncompressed, equivalent to ClickPipes_FetchedBytes_Total. For CDC ClickPipes, this excludes bytes fetched from initial load, which is emitted separately to ClickPipes_FetchedBytesInitialLoad_Total.
ClickPipes_FetchedBytesInitialLoad_TotalCounterCDCTotal bytes fetched during the initial load phase, including resyncs.
ClickPipes_SentBytes_TotalCounterStreaming, object storageTotal uncompressed bytes sent to ClickHouse.
ClickPipes_SentBytesCompressed_TotalCounterStreaming, object storageTotal compressed bytes sent to ClickHouse.

Events and records​

MetricTypeAvailable forDescription
ClickPipes_FetchedEvents_TotalCounterStreaming, object storageTotal number of records fetched from the source.
ClickPipes_SentEvents_TotalCounterStreaming, object storageTotal number of records sent to ClickHouse.
ClickPipes_FetchedEvents_PerTableGaugeCDCNumber of records fetched per destination table and event type. Uses additional destination_table and event_type labels.

Errors​

MetricTypeAvailable forDescription
ClickPipes_Errors_TotalCounterAllTotal errors encountered during ingestion.

Latency​

MetricTypeAvailable forDescription
ClickPipes_LatencyGauge (avg)Streaming, object storageTime in milliseconds between when a record was produced at the source and when it was written to ClickHouse.
ClickPipes_SourceReplicationLatency_MiBGauge (avg)CDC (Postgres only)Replication slot lag at the source in MiB. Only emitted by Postgres ClickPipes.

Resource utilization​

ClickPipes_Replica_* metrics report resource utilization for streaming and object storage ClickPipes. Each ClickPipe has dedicated compute, so these values reflect utilization for individual ClickPipes.

MetricTypeAvailable forDescription
ClickPipes_Replica_CPUUsageGauge (avg)Streaming, object storageAverage CPU usage of the ingestion replicas, in millicores.
ClickPipes_Replica_CPULimitGauge (last)Streaming, object storageConfigured CPU limit of the ingestion replicas, in millicores.
ClickPipes_Replica_MemoryUsageGauge (avg)Streaming, object storageAverage memory usage of the ingestion replicas, in bytes.
ClickPipes_Replica_MemoryLimitGauge (last)Streaming, object storageConfigured memory limit of the ingestion replicas, in bytes.

ClickPipes_CDC_* metrics report resource utilization for CDC ClickPipes. All CDC ClickPipes in the same service share compute, so these values reflect the combined usage of every CDC ClickPipe in the service.

MetricTypeAvailable forDescription
ClickPipes_CDC_CPUUsageGauge (avg)CDCAverage CPU usage, in millicores. Shared across all CDC ClickPipes in the service.
ClickPipes_CDC_CPULimitGauge (last)CDCConfigured CPU limit, in millicores. Shared across all CDC ClickPipes in the service.
ClickPipes_CDC_MemoryUsageGauge (avg)CDCAverage memory usage, in bytes. Shared across all CDC ClickPipes in the service.
ClickPipes_CDC_MemoryLimitGauge (last)CDCConfigured memory limit, in bytes. Shared across all CDC ClickPipes in the service.
ClickPipes_CDC_NetworkReceiveBytesGaugeCDCInbound network bandwidth, in bytes (delta per scrape window). Shared across all CDC ClickPipes in the service.

State and progress​

MetricTypeAvailable forDescription
ClickPipes_InfoGaugeAllAlways equal to 1. Use the clickpipe_state label to filter or alert on pipe state.
ClickPipes_LastFetchedBatchIdGaugeCDCID of the last batch fetched from the source.
ClickPipes_LastSentBatchIdGaugeCDCID of the last batch sent to ClickHouse.
  • Prometheus endpoint — Endpoint reference, authentication, and setup for external tools (e.g., Grafana, Datadog)
  • Error reporting — Where record and system errors are stored