Resource recommendations

This section describes the compute and disk requirements that are recommended for Promscale, based on the ingest rate and retention.

Disk size per day: disk consumption per day based on the ingest rate.

Uncompressed buffer: disk consumed by uncompressed data before compression.

Total disk size: size of disk required to store the data based on the ingest rate and retention.

note

You can calculate the total disk size based on retention and ingest rates with this formula:

Total disk size = (Disk size per day based on the ingest rate * Retention in days) + Uncompressed buffer based on the ingest rate.

Metrics

Resource recommendation for ingestion using Prometheus remote-write.

Prometheus remote write

For optimal performance of remote_write to Promscale, use this Prometheus remote_write configuration:

remote_write:
  remote_timeout: 100s
  queue_config:
    capapcity: 100000
    max_samples_per_second: 10000
    batch_send_deadline: 30s
    min_shards: 20
    max_shards: 20
    min_backoff: 100ms
    max_backoff: 10s

Compute recommendations for the Promscale connector and TimescaleDB are:

Ingestion RateConnector CPUConnector MemoryDB CPUDB MemoryDB connections
10k samples/sec0.5250 MB14 GB1
50k samples/sec2700 MB416 GB2
100k samples/sec42 GB832 GB4
200k samples/sec84.5 GB1664 GB8

Disk recommendations for TimescaleDB are:

The default chunk interval is 8h

Ingest rateRetentionDisk size per dayUncompressed bufferTotal disk sizeWAL size
10k samples/sec90 days~2 GB~21 GB~200 GB1.25 GB
50k samples/sec90 days~10 GB~105 GB~1 TB-
100k samples/sec90 days~20 GB~210 GB~2 TB-
200k samples/sec90 days~40 GB~420 GB~4 TB-

Traces

Resource recommendation for ingestion through OTLP (OpenTelemetry Line Procotol) gRPC endpoint.

OpenTelemetry Line Protocol

We recommend using the OpenTelemetry collector for ingesting the spans to Promscale. Use this configuration with the OTLP exporter and batch processor to help with retries for failed writes, and to batch the write requests to Promscale:

exporters:
  logging:
  otlp:
    endpoint: "<PROMSCALE_HOST>:<gRPC_PORT>"
    tls:
      insecure: true
    sending_queue:
      queue_size: 1000000
    timeout: 10s
processors:
  batch:
    send_batch_size: 4000
    send_batch_max_size: 4000
    timeout: 10s

Where:

  • <PROMSCALE_HOST>: hostname of Promscale
  • <gRPC_PORT>: gRPC port of Promscale. The default port is 9202.

Compute recommendations for the Promscale connector and TimescaleDB are:

Ingestion RateConnector CPUConnector MemoryDB CPUDB MemoryDB connections
10k spans/sec = ~2.5 MB/sec0.54 GB24 GB4
50k spans/sec = ~5 MB/sec28 MB48 GB8
100k spans/sec = ~10 MB/sec416 GB816 GB16
200k spans/sec = ~20 MB/sec816 GB1616 GB32

Disk recommendations for TimescaleDB are:

The default chunk interval is 1h

Ingest rateRetentionDisk size per dayUncompressed bufferTotal disk sizeWAL size
10k spans/sec = ~2.5 MB/sec30 days~30 GB~35 GB~935 GB20 GB
50k spans/sec = ~5 MB/sec30 days~60 GB~175 GB~2 TB-
100k spans/sec= ~10 MB/sec30 days~150 GB~350 GB~5 TB-
200k spans/sec= ~20 MB/sec30 days~300 GB~700 GB~10 TB-

Database Configuration

To set the most common parameters to optimal values based on your system, run timescaledb-tune. It accounts for memory, CPU, and PostgreSQL version. For more information, see configuration. However, there are a few other PostgreSQL parameters worth tuning:

  • checkpoint_timeout=15min - when a lot of data is ingested, increase the checkpoint timeout to reduce the input/output pressure.
  • bgwriter_delay=10ms - the background writer needs to be active to reduce delays.
  • bgwriter_lru_maxpages=100000 - increase the number of pages a background writer handles to make it more efficient.
  • max_wal_size - set it to a high enough value so that the checkpoint is triggered by the timeout setting and not when the maximum_wal_size is reached.
  • synchronous_commit=off - this does not cause data corruption or inconsistency. However, in case of a crash, some of the last data points may be lost. For a monitoring observability use case, it's a reasonable tradeoff to increase ingest performance.

important

Make sure that the maximum latency between the Promscale connector and the database is no more than 100ms.

Found an issue on this page?

Report an issue!

Keywords

Related Content