Repository navigation
Releases: prometheus/prometheus
Releases ยท prometheus/prometheus
Release list
3.15.1 / 2026-10-21
Immutable
release. Only release title and notes can be modified.
- [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
- [SECURITY] Build with Go 1.27.3 to fix vulnerabilities in
net/http,net/textproto,crypto/tlsandhtml/template. - [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
3.13.5 / 2026-10-21
Immutable
release. Only release title and notes can be modified.
- [SECURITY] Container images: Update CA certificates to distrust "emSign ECC Root CA - G3" and "Comodo AAA Services root" following the compromise of eMudhra intermediate CA private keys.
- [SECURITY] Build with Go 1.26.10 to fix vulnerabilities in
net/http,net/textproto,crypto/tlsandhtml/template. - [SECURITY] Bump golang.org/x/net to v0.60.0 to fix HTTP/2 vulnerabilities (GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6617).
- [SECURITY] Bump OpenTelemetry to v1.45.0 to fix exporter endpoint URLs leaking into info logs (GO-2026-6505).
- [CHANGE] Building Prometheus now requires Go 1.26 or later.
3.13.4 / 2026-09-29
Immutable
release. Only release title and notes can be modified.
- [SECURITY] Bump google.golang.org/grpc to v1.83.1 to fix HTTP/2 DATA frame fragmentation memory exhaustion (GO-2026-6348). #19834
- [SECURITY] UI: Update vulnerable npm dependencies. #19834
- [BUGFIX] Agent: Ignore unknown WAL record types to allow rolling back from newer versions. #19814
- [BUGFIX] Federation: Fix corruption of float native histograms. #19679
- [BUGFIX] Scrape: Fix failing scrapes of protobuf float histograms with zero sample, zero and bucket counts, such as the result of recording
rate(x[1m])over a constant histogram. #19682
3.15.0 / 2026-09-24
Immutable
release. Only release title and notes can be modified.
- [CHANGE] PromQL: A range query whose
endwas not aligned tostepcaused subqueries inside it to evaluate past the parent's last actual step, inflatingpeakSamplesin the query stats and against thequery.max-sampleslimit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in #18081. #18598 - [CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. #19454
- [CHANGE] Logging: Deprecate
--log.level; useruntime.log_levelconfiguration to supply the default level. #19511 - [FEATURE] Configuration: Allow changing the process log level through
runtime.log_levelon configuration reload. #19511 - [FEATURE] Prometheus: Add
--auto-gomemlimit.refresh-intervalflag to periodically re-detect the container or system memory limit and updateGOMEMLIMITat runtime. #18843 - [FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. #12024. #18091
- [FEATURE] scrape: Implement OM2.0 scrape format. #18606
- [ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. #18001
- [ENHANCEMENT] Docker SD: Add labels
__meta_docker_container_imageand__meta_docker_container_image_id. #19386 - [ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. #19500
- [ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. #19522
- [ENHANCEMENT] PromQL/TSDB: The
--enable-feature=st-storageflag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to passxor2-encodingandhistograms-st-encodingalongside it. #19518 - [ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding
session_nameandtagsfields for STS AssumeRole sessions. The previously undocumentedservice_namefield is now also documented. #19569 - [ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag
zstd-scrape. #19502 - [ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding.
--enable-feature=xor2-encodingis deprecated; usestorage.tsdb.chunk_encoding.floats: xor2instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461 - [ENHANCEMENT] TSDB: add
prometheus_tsdb_head_appenders_created_totalmetric. #19411 - [ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. #19410
- [ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. #19384
- [ENHANCEMENT] scrape: Enable start time synthesis for summary
_countand_sumseries in scrape appender v2. #19323 - [ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. #19295
- [ENHANCEMENT] storage/remote: Add undocumented failed_request_logging config field to debug log remote write V2 requests on send errors. #19249
- [ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. #18049
- [ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. #19332
- [ENHANCEMENT] TSDB: Add
prometheus_tsdb_head_series_pending_commit_underflow_totalto report pending-sample reservation underflows. #19470 - [PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. #19504
- [PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by
request_concurrency. #19506 - [PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. #19585
- [PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. #19503
- [PERF] Remote write: Reuse OTLP converter scratch state between requests. #19388
- [PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. #19446
- [BUGFIX] PromQL: Fix
info()enrichment for composite expressions with mixed@/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. #19387 - [BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. #19557
- [BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. #19413
- [BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. #19272
- [BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. #19470
- [BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. #19470
- [BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. #19664
- [BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. #19460
- [BUGFIX] TSDB: fix default block reload interval for custom options. #19368
- [BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. #19194
- [BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. #19435
- [BUGFIX] AWS SD: Reject non-positive
request_concurrencyinstead of hanging service discovery indefinitely. #19524 - [BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. #19302
- [BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. #19324
- [BUGFIX] Discovery: delete the stale
prometheus_sd_last_update_timestamp_secondsseries for a config that is removed on reload. #19131 - [BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. #19661
- [BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. #19438
- [BUGFIX] Metadata will not affect the number of Remote Write v2 shards. #19218
- [BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. #19444
- [BUGFIX] Native histograms:
DetectResetno longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogramrate()/increase()undercount. #19367 - [BUGFIX] Never skip histogram buckets for histogram_stddev and histogram_stdvar functions. #19521
- [BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. #19343
- [BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g.
.*\|(foo)\|.*). #19516 - [BUGFIX] PromQL: Fix a panic in range selectors using the experimental
anchoredorsmoothedmodifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. #19431 - [BUGFIX] PromQL: Fix empty results when a subquery with
@is used as the matrix argument of a call that is not step-invariant (for examplequantile_over_time(scalar(x), metric[...:...] @ T)). #19187 - [BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. #18810
- [BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. #19403
- [BUGFIX] PromQL: Reject duration-expression offsets and
@ start()/@ end()before range selectors, matching the existing rejection of literal offsets and@ <timestamp>. #19406 - [BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of
histogram_quantileandhistogram_fraction. #19330 - [BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. #19266
- [BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. #19433
- [BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. #19328
- [BUGFIX] Scrape: JSON log formatter correctly format scrape target info. #19472
- [BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. #19450
- [BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping
firstOOOChunkIDmodulo 2^23 instead of growing unbounded. #19216 - [BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326
- [BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. #19333
- [BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. #19584
- [BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. #19512
- [BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. #19396
- [BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields #19422. #19422
- [BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. #19418
- [BUGFIX] discovery/kubernetes: Populate
__meta_kubernetes_service_loadbalancer_ipfromstatus.loadBalancer.ingress, falling back to deprecatedspec.loadBalancerIP. #19404 - [BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. #19312
- [BUGFIX] promtool: Fixed
tsdb dumpsilently dropping native histogram samples....
3.15.0-rc.1 / 2026-09-21
Immutable
release. Only release title and notes can be modified.
- [BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. #19700
3.15.0-rc.0 / 2026-09-09
Immutable
release. Only release title and notes can be modified.
- [CHANGE] PromQL: A range query whose
endwas not aligned tostepcaused subqueries inside it to evaluate past the parent's last actual step, inflatingpeakSamplesin the query stats and against thequery.max-sampleslimit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in #18081. #18598 - [CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. #19454
- [CHANGE] Logging: Deprecate
--log.level; useruntime.log_levelconfiguration to supply the default level. #19511 - [FEATURE] Configuration: Allow changing the process log level through
runtime.log_levelon configuration reload. #19511 - [FEATURE] Prometheus: Add
--auto-gomemlimit.refresh-intervalflag to periodically re-detect the container or system memory limit and updateGOMEMLIMITat runtime. #18843 - [FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. #12024. #18091
- [FEATURE] scrape: Implement OM2.0 scrape format. #18606
- [ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. #18001
- [ENHANCEMENT] Docker SD: Add labels
__meta_docker_container_imageand__meta_docker_container_image_id. #19386 - [ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. #19500
- [ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. #19522
- [ENHANCEMENT] PromQL/TSDB: The
--enable-feature=st-storageflag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to passxor2-encodingandhistograms-st-encodingalongside it. #19518 - [ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding
session_nameandtagsfields for STS AssumeRole sessions. The previously undocumentedservice_namefield is now also documented. #19569 - [ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag
zstd-scrape. #19502 - [ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding.
--enable-feature=xor2-encodingis deprecated; usestorage.tsdb.chunk_encoding.floats: xor2instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. #19461 - [ENHANCEMENT] TSDB: add
prometheus_tsdb_head_appenders_created_totalmetric. #19411 - [ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. #19410
- [ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. #19384
- [ENHANCEMENT] scrape: Enable start time synthesis for summary
_countand_sumseries in scrape appender v2. #19323 - [ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. #19295
- [ENHANCEMENT] storage/remote: Add undocumented failed_request_logging config field to debug log remote write V2 requests on send errors. #19249
- [ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. #18049
- [ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. #19332
- [ENHANCEMENT] TSDB: Add
prometheus_tsdb_head_series_pending_commit_underflow_totalto report pending-sample reservation underflows. #19470 - [PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. #19504
- [PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by
request_concurrency. #19506 - [PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. #19585
- [PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. #19503
- [PERF] Remote write: Reuse OTLP converter scratch state between requests. #19388
- [PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. #19446
- [BUGFIX] PromQL: Fix
info()enrichment for composite expressions with mixed@/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. #19387 - [BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. #19557
- [BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. #19413
- [BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. #19272
- [BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. #19470
- [BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. #19470
- [BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. #19664
- [BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. #19460
- [BUGFIX] TSDB: fix default block reload interval for custom options. #19368
- [BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. #19194
- [BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. #19435
- [BUGFIX] AWS SD: Reject non-positive
request_concurrencyinstead of hanging service discovery indefinitely. #19524 - [BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. #19302
- [BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. #19324
- [BUGFIX] Discovery: delete the stale
prometheus_sd_last_update_timestamp_secondsseries for a config that is removed on reload. #19131 - [BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. #19661
- [BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. #19438
- [BUGFIX] Metadata will not affect the number of Remote Write v2 shards. #19218
- [BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. #19444
- [BUGFIX] Native histograms:
DetectResetno longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogramrate()/increase()undercount. #19367 - [BUGFIX] Never skip histogram buckets for histogram_stddev and histogram_stdvar functions. #19521
- [BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. #19343
- [BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g.
.*\|(foo)\|.*). #19516 - [BUGFIX] PromQL: Fix a panic in range selectors using the experimental
anchoredorsmoothedmodifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. #19431 - [BUGFIX] PromQL: Fix empty results when a subquery with
@is used as the matrix argument of a call that is not step-invariant (for examplequantile_over_time(scalar(x), metric[...:...] @ T)). #19187 - [BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. #18810
- [BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. #19403
- [BUGFIX] PromQL: Reject duration-expression offsets and
@ start()/@ end()before range selectors, matching the existing rejection of literal offsets and@ <timestamp>. #19406 - [BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of
histogram_quantileandhistogram_fraction. #19330 - [BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. #19266
- [BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. #19433
- [BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. #19328
- [BUGFIX] Scrape: JSON log formatter correctly format scrape target info. #19472
- [BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. #19450
- [BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping
firstOOOChunkIDmodulo 2^23 instead of growing unbounded. #19216 - [BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. #19326
- [BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. #19333
- [BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. #19584
- [BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. #19512
- [BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. #19396
- [BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields #19422. #19422
- [BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. #19418
- [BUGFIX] discovery/kubernetes: Populate
__meta_kubernetes_service_loadbalancer_ipfromstatus.loadBalancer.ingress, falling back to deprecatedspec.loadBalancerIP. #19404 - [BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. #19312
- [BUGFIX] promtool: Fixed
tsdb dumpsilently dropping native histogram samples....
3.13.3 / 2026-09-07
Immutable
release. Only release title and notes can be modified.
- [SECURITY] Bump github.com/klauspost/compress to v1.18.7 (GO-2026-5841) and golang.org/x/crypto to v0.55.0 (GO-2026-6303).
- [BUGFIX] Docker SD: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
- [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
- [BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
- [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
- [BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
- [BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
- [BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
3.14.0 / 2026-08-17
Immutable
release. Only release title and notes can be modified.
- [CHANGE] API: Deprecate the
statsquery parameter of/api/v1/queryand/api/v1/query_rangefor values other thantrueandall. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 - [CHANGE] API:
/api/v1/status/confignow correctly showsseparator: ""andreplacement: ""in relabel configs when explicitly set to empty, instead of omitting them. #18653 - [CHANGE] Discovery/Hetzner: Drop the
__meta_hetzner_datacenterlabel forhcloudtargets, following its removal from the Hetzner Cloud API. #19269 - [CHANGE] PromQL: Enable duration expressions by default. The
promql-duration-exprfeature flag is now a no-op. #19033 - [CHANGE] PromQL: Promote
first_over_timeto stable. It no longer requires thepromql-experimental-functionsfeature flag. #19093 - [FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (
oci_sd_configs). #18919 - [FEATURE] PromQL: Add experimental
start_timestamp(instant-vector)function returning the start timestamp of each sample in the given vector. Requires theuse-start-timestampsfeature flag. #19089 - [FEATURE] PromQL: Allow
rate()andincrease()to use start timestamps as an alternative for rate extrapolation. Hidden behind theuse-start-timestampsfeature flag. #18619 - [FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the
histograms-st-encodingfeature flag. #18609 - [ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g.
k8s.pod.nameandk8s_pod_nameboth becomek8s_pod_name), and expose theprometheus_api_otlp_translation_warnings_totalcounter labelled bycategoryto track such warnings. #18957 - [ENHANCEMENT] Promtool: Add
--remote-write.pathflag topush metricsfor backends that use a non-default remote-write endpoint. #19086 - [ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903
- [ENHANCEMENT] TSDB: Add
prometheus_tsdb_head_native_histogram_seriesandprometheus_tsdb_head_native_histogram_bucketsgauges tracking the number of native histogram series and buckets in the head. #19170 - [ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (
step(),range(),min_of(),max_of()) in range selectors and subqueries. #18625 - [ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706
- [ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181
- [PERF] Speed up regex label matchers matching a set of literal values (e.g.
{job=~"foo|bar|baz"}). #18833 - [PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470
- [PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143
- [PERF] Scrape: Reduce native histogram scrape parsing allocations by ~49%. #19282
- [PERF] TSDB: Speed up queries on series with many in-memory chunks. #18300
- [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
- [BUGFIX] Discovery/AWS: Stop
promtool check configfrom making AWS metadata service (IMDS) network calls when theregionfield is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037 - [BUGFIX] Discovery/Docker: Set a request timeout for
docker_sdanddockerswarm_sdonunix,npipe, andtcphosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237 - [BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
- [BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778
- [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
- [BUGFIX] PromQL: Fix
mad_over_timereturning 0 instead ofNaNwhen the range contains aNaNsample. #19040 - [BUGFIX] Promtool: Accept
--enable-feature=promql-binop-fill-modifiersincheck rules, which previously rejected validfill()/fill_left()/fill_right()expressions. #19153 - [BUGFIX] Remote write: Respect the
AZURE_FEDERATED_TOKEN_FILEenvironment variable for workload identity authentication instead of hardcoding the token file path. #18973 - [BUGFIX] Rules: Clean up stale
rule_group_last_rule_duration_sum_secondsandrule_group_last_restore_duration_secondsseries when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing/metricscardinality over time. #19107 - [BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
- [BUGFIX] TSDB: Fix silent data loss and potential crash loop when
stale_series_compaction_thresholdis used in the config file. #19016 - [BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted. #19016
- [BUGFIX] TSDB: Fix
prometheus_tsdb_head_stale_seriesover-counting and early eviction of series that change between float, integer histogram, and float histogram sample types. #19183 - [BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
- [BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
- [BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
- [BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart. #19202
- [BUGFIX] TSDB: Surface query errors that were previously silently discarded. #19120
- [BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with
--enable-feature=xor2-encodingcould silently revert to XOR after compaction. #19145 - [BUGFIX] UI: Show the delete-series form on the TSDB Status page when
--web.enable-admin-apiis enabled. #19025
3.14.0-rc.0 / 2026-08-06
Immutable
release. Only release title and notes can be modified.
- [CHANGE] API: Deprecate the
statsquery parameter of/api/v1/queryand/api/v1/query_rangefor values other thantrueandall. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 - [CHANGE] API:
/api/v1/status/confignow correctly showsseparator: ""andreplacement: ""in relabel configs when explicitly set to empty, instead of omitting them. #18653 - [CHANGE] Discovery/Hetzner: Drop the
__meta_hetzner_datacenterlabel forhcloudtargets, following its removal from the Hetzner Cloud API. #19269 - [CHANGE] PromQL: Enable duration expressions by default. The
promql-duration-exprfeature flag is now a no-op. #19033 - [CHANGE] PromQL: Promote
first_over_timeto stable. It no longer requires thepromql-experimental-functionsfeature flag. #19093 - [FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (
oci_sd_configs). #18919 - [FEATURE] PromQL: Add experimental
start_timestamp(instant-vector)function returning the start timestamp of each sample in the given vector. Requires theuse-start-timestampsfeature flag. #19089 - [FEATURE] PromQL: Allow
rate()andincrease()to use start timestamps as an alternative for rate extrapolation. Hidden behind theuse-start-timestampsfeature flag. #18619 - [FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the
histograms-st-encodingfeature flag. #18609 - [ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g.
k8s.pod.nameandk8s_pod_nameboth becomek8s_pod_name), and expose theprometheus_api_otlp_translation_warnings_totalcounter labelled bycategoryto track such warnings. #18957 - [ENHANCEMENT] Promtool: Add
--remote-write.pathflag topush metricsfor backends that use a non-default remote-write endpoint. #19086 - [ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903
- [ENHANCEMENT] TSDB: Add
prometheus_tsdb_head_native_histogram_seriesandprometheus_tsdb_head_native_histogram_bucketsgauges tracking the number of native histogram series and buckets in the head. #19170 - [ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (
step(),range(),min_of(),max_of()) in range selectors and subqueries. #18625 - [ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706
- [ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181
- [PERF] Speed up regex label matchers matching a set of literal values (e.g.
{job=~"foo|bar|baz"}). #18833 - [PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470
- [PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143
- [PERF] Scrape: Reduce native histogram scrape parsing allocations by ~49%. #19282
- [PERF] TSDB: Speed up queries on series with many in-memory chunks. #18300
- [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
- [BUGFIX] Discovery/AWS: Stop
promtool check configfrom making AWS metadata service (IMDS) network calls when theregionfield is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037 - [BUGFIX] Discovery/Docker: Set a request timeout for
docker_sdanddockerswarm_sdonunix,npipe, andtcphosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237 - [BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
- [BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778
- [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
- [BUGFIX] PromQL: Fix
mad_over_timereturning 0 instead ofNaNwhen the range contains aNaNsample. #19040 - [BUGFIX] Promtool: Accept
--enable-feature=promql-binop-fill-modifiersincheck rules, which previously rejected validfill()/fill_left()/fill_right()expressions. #19153 - [BUGFIX] Remote write: Respect the
AZURE_FEDERATED_TOKEN_FILEenvironment variable for workload identity authentication instead of hardcoding the token file path. #18973 - [BUGFIX] Rules: Clean up stale
rule_group_last_rule_duration_sum_secondsandrule_group_last_restore_duration_secondsseries when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing/metricscardinality over time. #19107 - [BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
- [BUGFIX] TSDB: Fix silent data loss and potential crash loop when
stale_series_compaction_thresholdis used in the config file. #19016 - [BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted. #19016
- [BUGFIX] TSDB: Fix
prometheus_tsdb_head_stale_seriesover-counting and early eviction of series that change between float, integer histogram, and float histogram sample types. #19183 - [BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
- [BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
- [BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
- [BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart. #19202
- [BUGFIX] TSDB: Surface query errors that were previously silently discarded. #19120
- [BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with
--enable-feature=xor2-encodingcould silently revert to XOR after compaction. #19145 - [BUGFIX] UI: Show the delete-series form on the TSDB Status page when
--web.enable-admin-apiis enabled. #19025
3.13.2 / 2026-07-29
Immutable
release. Only release title and notes can be modified.
What's Changed
- [SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama
- [BUGFIX] PromQL: Preallocate the active query tracker file to avoid SIGBUS crashes when the data disk is full. #19289 by @akshajrawat
Full Changelog: v3.13.1...v3.13.2