Metrics

Metrics describe observations, not an authoritative count of jobs in a queue. They are buffered in memory, flushed every 60 seconds by default, and read from PostgreSQL. An abrupt worker exit can lose observations that have not been flushed. Graceful shutdown flushes metrics after final job updates.

Meaning and units

• Counters sum observed increments within the requested window.

• Gauges select the most recently sampled value, including across workers.

• Histograms summarize count, sum, minimum and maximum; averages are weighted by observation count. These summaries do not support percentiles.

• Job state and throughput counters count accepted, committed job update events, including retries. They do not measure queue occupancy, unique jobs, or every state transition: claiming, manual cancellation and recovery use other paths.

• Job and queue execution times are in seconds and measure the final attempt for updates containing both start and completion timestamps.

• Workflow counters describe committed scheduler transitions. Workflow duration is the elapsed time from creation to the scheduler’s terminal update, including dependency waits and scheduler delay. Producer submissions are not counted.

• Table sizes are gauges in bytes, sampled by the leader only.

Custom observations can specify unit on increment_counter(), record_gauge() and record_histogram_value(). A key’s type and unit must remain consistent. Keys may contain up to 1,024 UTF-8 bytes; values must be finite numbers. Invalid observations are dropped with a rate-limited warning.

Ranges, gaps and freshness

GET /api/v1/metrics/{prefix} accepts a resolution and a separate range in seconds. For example, ?resolution=5min&range=86400 requests 24 hours at five-minute resolution. worker_id optionally limits the observations to one worker. A prefix matches the exact key and descendants separated by :.

The response contains start, end, resolution, generated_at and a series mapping. Each series contains type, unit, aggregation, sampled_at, summary and chronological data points. Each point contains its bucket timestamp, last observation’s sampled_at, and value. summary uses the same reducer as the metric type over the returned points. An empty series has a null summary and null sample time.

Windows are UTC-aligned and half-open: [start, end). The default includes the current, potentially incomplete bucket. Alternatively, pass timezone-aware ISO start and end timestamps aligned to the chosen resolution. limit is also supported as a number of time buckets, not a number of populated samples; it cannot be combined with start or range. Invalid windows return HTTP 422. Requests are limited to 1,000 buckets and 100 matching series. Read results may be cached for ten seconds.

Missing buckets mean no observation, not zero. Histograms and totals cover only observed samples; sample timestamps show how recently a metric was observed. The dashboard leaves gaps in charts and displays an unavailable value where no summary exists. A success rate requires observations for both outcomes; an absent failure series is not evidence of zero failures.

Retention and resource use

Default retention is elapsed time, rather than a count of populated samples:

Resolution	Retention
1 minute	1 hour
5 minutes	24 hours
1 hour	7 days
1 day	90 days

Expired samples are excluded from reads even before the Pruner removes them. Storage groups at most 12 buckets into a chunk; pruning removes fully expired chunks, so a boundary chunk can temporarily contain older samples. Each pass removes at most 10,000 chunks. Unreferenced metric definitions are also pruned.

Workers retain current chunks and pending writes instead of reading all workers’ history. Only changed chunks are written; retries replace the same worker-session snapshot without counting it again. Reusing a worker ID preserves previous sessions. More frequent worker restarts create additional chunks.

The in-memory buffer defaults to 10,000 bucket snapshots. If persistence is unavailable, dirty observations are retried with backoff, within retention and this memory bound. New buckets exceeding the bound are dropped with a warning. The API bounds its local read cache to 32 windows and 50,000 points.

Upgrading

Metrics migration 2 clears existing metric history and replaces its storage layout. Previous timestamps, gauges and sparse retention cannot be translated reliably into the new contract. Jobs, queues and workflows are unaffected. Stop workers and API processes before migrating, then restart them together; the old collector and API are incompatible with the new schema. Downgrading also clears metric history.

Reference

Metrics plugin for collecting and aggregating job and queue metrics.

class chancy.plugins.metrics.Metrics(*, sync_interval=60, max_points_per_resolution=None, maximum_metric_age=7776000, collection_interval=30, max_buffered_points=10000)

Bases: Plugin

A plugin that collects and aggregates various metrics from jobs and queues.

The plugin maintains time-series data for various metrics, with automatic aggregation and pruning to provide useful historical data.

Metrics from all workers can be queried together or filtered by worker.

Note

This plugin is enabled by default, you only need to provide it in the list of plugins to customize its arguments or if no_default_plugins is set to True.

Enable the plugin by adding it to the list of plugins in the Chancy constructor:

from chancy import Chancy
from chancy.plugins.metrics import Metrics

async with Chancy(..., plugins=[Metrics()]) as chancy:
    ...

Data points are aggregated at different resolutions (1 minute, 5 minutes, 1 hour, 1 day). Counters total observed increments, gauges show the latest value, and histograms summarize observations with a weighted average.

Retention is based on elapsed time, including periods with no observations. Missing observations are not treated as zero.

Default Retention Policy:

Resolution	Time Buckets	Total Time Period
1 minute	60	1 hour
5 minutes	288	1 day (24 hours)
1 hour	168	1 week (7 days)
1 day	90	90 days (3 months)

Job counters count saved job updates, including retries, rather than the number of jobs currently in a queue. Job execution times are in seconds and measure the final attempt. Metrics become available after the next flush, every 60 seconds by default; an abrupt worker exit can lose recent observations that have not yet been saved.

Note

While you can use this plugin to record your own arbitrary metrics, it’s not designed as a general-purpose monitoring solution. For more advanced monitoring and visualization, consider using a dedicated monitoring tool like Prometheus or Grafana.

Parameters:

• sync_interval (int)

• max_points_per_resolution (dict[Literal['1min', '5min', '1hour', '1day'], int] | None)

• maximum_metric_age (int)

• collection_interval (int)

• max_buffered_points (int)

api_plugin()

If this plugin has an associated API component, returns the import string for the plugin.

async cleanup(chancy)

Clean up old metrics data.

Called automatically by the Pruner plugin, or may be manually invoked.

Expired data is excluded from queries even before cleanup runs.

async flush(chancy)

Save recorded metrics so they can be queried.

Workers call this periodically and during graceful shutdown. Call it explicitly when you need pending observations saved sooner. If saving fails, pending observations are kept so this method can be retried.

Parameters:

chancy – The Chancy application instance.

static get_identifier()

Returns a unique identifier for this plugin.

This identifier should be unique across all active plugins. If a custom plugin provides compatible functionality to a built-in plugin, it may use the same identifier as the built-in plugin.

async get_metrics(chancy, metric_prefix=None, worker_id=None, *, resolution='5min', start=None, end=None, range_seconds=None, limit=None)

Get metrics matching the given prefix and time range.

Metrics from all workers are combined unless worker_id is provided. Counters sum observed increments, gauges return the most recent value, and histogram averages are weighted by the number of observations. Missing data points are omitted rather than filled with zeros.

For example, to get the default queue’s metrics for the last 24 hours:

result = await metrics.get_metrics(
    chancy,
    metric_prefix="queue:default",
    resolution="5min",
    range_seconds=86400,
)
for key, metric in result["series"].items():
    print(key, metric.summary, metric.unit)

Parameters:

• chancy – The Chancy application instance.

• metric_prefix – Optional metric key or prefix. queue:default matches that key and its :-separated descendants.

• worker_id – Optional worker ID to filter by.

• resolution – Size of each time bucket: 1min, 5min, 1hour or 1day. Defaults to 5min.

• start – Optional inclusive start time, with a timezone and aligned to the chosen resolution. Cannot be combined with range_seconds or limit.

• end – Optional exclusive end time, with a timezone and aligned to the chosen resolution. Defaults to the end of the current, potentially incomplete bucket.

• range_seconds – Length of the requested time range, in seconds. Must be a multiple of the resolution and within retention. Defaults to 24 hours or the retention period, whichever is shorter.

• limit – Alternative way to specify the range as a number of time buckets, including buckets with no observations. Cannot be combined with start or range_seconds.

Returns:

A dictionary containing start, end, resolution, generated_at and a series mapping of keys to metrics. Only saved observations are returned; results may be up to ten seconds behind newly saved data.

Raises:

ValueError – If the window is invalid, exceeds retention or 1,000 time buckets, or matches more than 100 metric keys.

get_tables()

Get the names of all tables this plugin is responsible for.

async increment_counter(metric_key, value, *, unit='count')

Increment a counter metric.

Counter metrics accumulate values over time periods.

Parameters:

• metric_key (str) – The unique key for the metric

• value (float) – The value to increment the counter by

• unit (str) – Unit of the increments. Defaults to count; use the same unit for every observation of this metric key.

async list_metrics(chancy)

Get the keys of all available metrics, in alphabetical order.

Parameters:

chancy – The Chancy application instance.

Returns:

A list of metric keys.

migrate_key()

Get the unique identifier for this plugin’s migrations.

migrate_package()

Get the package that contains the migrations for the metrics plugin.

async on_job_updated(*, worker, job)

Called for each accepted job update after its transaction commits.

This method cannot modify the saved update. Updates rejected because their claims no longer match or their jobs no longer exist do not invoke this hook. Accepted updates retain their original batch order, including repeated updates to the same job.

Parameters:

• worker – The worker saving the update.

• job (QueuedJob) – The accepted update that was committed.

async on_worker_started(*, worker)

Called when a worker starts, before plugin tasks begin running.

Use this hook to register event handlers or prepare resources needed by other plugins. It is called for plugins with worker scope.

Parameters:

worker – The worker being started.

async on_worker_stopped(*, worker)

Called during shutdown, after the worker saves its final job updates.

Use this hook to save remaining plugin data and release resources. Background tasks have already stopped. If shutdown fails and is retried, this hook may be called again, so repeated calls must be safe.

Parameters:

worker – The worker being stopped.

async record_gauge(metric_key, value, *, unit='number')

Record a gauge metric which represents a point-in-time value.

Gauge metrics record the most recent value in each time bucket.

Parameters:

• metric_key (str) – The unique key for the metric

• value (float) – The value to record

• unit (str) – Unit of the value. Defaults to number; use the same unit for every observation of this metric key.

async record_histogram_value(metric_key, value, *, unit='number')

Record a value to a histogram metric.

Histogram metrics track statistics (min, max, avg, count) for values over time periods.

Parameters:

• metric_key (str) – The unique key for the metric

• value (float) – The value to record

• unit (str) – Unit of the observations, such as seconds. Defaults to number; use the same unit for this metric key.

retention(resolution)

Get the retention period for a resolution, in seconds.

The configured bucket count and maximum_metric_age both limit how long observations remain available.

async run(worker, chancy)

Run the metrics plugin.

Periodically save recorded metrics and collect database table sizes. Workers start this automatically when the plugin is enabled.

window(resolution='5min', *, start=None, end=None, range_seconds=None, limit=None)

Get the start and end times for a metric query.

Times must include a timezone and align to the chosen resolution. The start is inclusive and the end is exclusive. By default the range ends at the end of the current, potentially incomplete bucket.

Accepts the same window options as get_metrics().

Returns:

A (start, end) pair.

Raises:

ValueError – If the requested window is invalid.

class chancy.plugins.metrics.metrics.Metric(type, unit, data)

A metric’s observations and summary for a requested time range.

Parameters:

• type (Literal['counter', 'gauge', 'histogram']) – counter, gauge or histogram.

• unit (str) – Unit of the recorded values, such as count or seconds.

• data (list[MetricPoint]) – Observed data points in chronological order.

property aggregation

How values are summarized: sum, last or summary.

property sampled_at

Time of the latest observation, or None if no data is available.

property summary: float | dict[str, float | int] | None

Summarize observations in this metric’s time range.

Returns the counter total, most recent gauge value, or histogram statistics (count, sum, avg, min and max). Returns None when there are no observations.

class chancy.plugins.metrics.metrics.MetricPoint(timestamp, value, sampled_at)

An observed value for one time bucket.

Parameters:

• timestamp (datetime) – Start of the time bucket.

• value (float | dict[str, float | int]) – A counter total, gauge value, or histogram statistics.

• sampled_at (datetime) – Time of the most recent observation in this bucket.