[METRICS SDK] Async instruments don't drop unreported attribute sets under Cumulative temporality

[kyusic]

Describe your environment

• opentelemetry-cpp: v1.27.0
• Affected instruments: ObservableCounter, ObservableGauge, ObservableUpDownCounter (all async instruments share AsyncMetricStorage + TemporalMetricStorage in v1.27.0)
• Affected temporality: Cumulative (Delta works as expected)
• Verification environment for the reproducer below:
  • Ubuntu 24.04 in Docker (ubuntu:24.04, kernel 6.8.0 aarch64)
  • g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
  • cmake 3.28.3
  • opentelemetry-cpp built at tag v1.27.0 with ostream exporter only (-DBUILD_TESTING=OFF -DWITH_EXAMPLES=OFF -DWITH_OTLP_GRPC=OFF -DWITH_OTLP_HTTP=OFF -DWITH_OTLP_FILE=OFF -DWITH_PROMETHEUS=OFF)

Steps to reproduce

The reproducer below registers an Int64ObservableGauge and an Int64ObservableUpDownCounter whose callbacks report two attribute sets {key=A} and {key=B} for the first three collection cycles, then stop reporting {key=B} from cycle 4 onward. The default cumulative temporality of OStreamMetricExporter is used.

// repro.cc
#include <atomic>
#include <chrono>
#include <map>
#include <memory>
#include <string>
#include <thread>

#include "opentelemetry/exporters/ostream/metric_exporter_factory.h"
#include "opentelemetry/metrics/provider.h"
#include "opentelemetry/sdk/metrics/export/periodic_exporting_metric_reader_factory.h"
#include "opentelemetry/sdk/metrics/export/periodic_exporting_metric_reader_options.h"
#include "opentelemetry/sdk/metrics/meter_context_factory.h"
#include "opentelemetry/sdk/metrics/meter_provider_factory.h"
#include "opentelemetry/sdk/metrics/provider.h"

namespace metrics_api   = opentelemetry::metrics;
namespace metrics_sdk   = opentelemetry::sdk::metrics;
namespace exportmetrics = opentelemetry::exporter::metrics;

namespace {

std::atomic<int> gauge_cycle{0};
std::atomic<int> updown_cycle{0};

void GaugeCallback(opentelemetry::metrics::ObserverResult result, void*) {
  auto observer = opentelemetry::nostd::get<
      opentelemetry::nostd::shared_ptr<
          opentelemetry::metrics::ObserverResultT<int64_t>>>(result);
  const int n = gauge_cycle.fetch_add(1);
  observer->Observe(100 + n, std::map<std::string, std::string>{{"key", "A"}});
  if (n < 3) {
    observer->Observe(200 + n, std::map<std::string, std::string>{{"key", "B"}});
  }
}

void UpDownCallback(opentelemetry::metrics::ObserverResult result, void*) {
  auto observer = opentelemetry::nostd::get<
      opentelemetry::nostd::shared_ptr<
          opentelemetry::metrics::ObserverResultT<int64_t>>>(result);
  const int n = updown_cycle.fetch_add(1);
  observer->Observe(10 + n, std::map<std::string, std::string>{{"key", "A"}});
  if (n < 3) {
    observer->Observe(20 + n, std::map<std::string, std::string>{{"key", "B"}});
  }
}

}  // namespace

int main() {
  auto exporter = exportmetrics::OStreamMetricExporterFactory::Create();

  metrics_sdk::PeriodicExportingMetricReaderOptions options;
  options.export_interval_millis = std::chrono::milliseconds(1000);
  options.export_timeout_millis  = std::chrono::milliseconds(500);
  auto reader = metrics_sdk::PeriodicExportingMetricReaderFactory::Create(
      std::move(exporter), options);

  auto context = metrics_sdk::MeterContextFactory::Create();
  context->AddMetricReader(std::move(reader));
  auto sdk_provider = metrics_sdk::MeterProviderFactory::Create(std::move(context));
  std::shared_ptr<metrics_api::MeterProvider> api_provider(sdk_provider.release());
  metrics_api::Provider::SetMeterProvider(api_provider);

  auto meter  = api_provider->GetMeter("repro", "1.0.0");
  auto gauge  = meter->CreateInt64ObservableGauge("test_gauge");
  auto updown = meter->CreateInt64ObservableUpDownCounter("test_updown_counter");
  gauge->AddCallback(GaugeCallback, nullptr);
  updown->AddCallback(UpDownCallback, nullptr);

  std::this_thread::sleep_for(std::chrono::seconds(8));

  gauge->RemoveCallback(GaugeCallback, nullptr);
  updown->RemoveCallback(UpDownCallback, nullptr);
  metrics_sdk::Provider::SetMeterProvider(nullptr);
  return 0;
}

Build against an installed opentelemetry-cpp v1.27.0:

# CMakeLists.txt
cmake_minimum_required(VERSION 3.20)
project(otel_repro CXX)
set(CMAKE_CXX_STANDARD 17)
find_package(opentelemetry-cpp CONFIG REQUIRED)
add_executable(repro repro.cc)
target_link_libraries(repro PRIVATE
  opentelemetry-cpp::ostream_metrics_exporter
  opentelemetry-cpp::metrics
  opentelemetry-cpp::api)

What is the expected behavior?

Per the SDK specification, Observations inside asynchronous callbacks:

The implementation SHOULD NOT produce aggregated metric data for a previously-observed attribute set which is not observed during a successful callback.

The .NET docs, Memory management, state the same intended behavior:

If the callback stops reporting a particular attribute set, it will be omitted from subsequent exports.

After the callback stops reporting {key=B} at cycle 4, no further data point for {key=B} should appear in any subsequent export.

What is the actual behavior?

{key=B} continues to be exported indefinitely after the callback stops reporting it, with its last observed value frozen in place.

Excerpts from the captured ostream exporter output (cycle ends marked):

Cycle 3 (last cycle in which the callback reports {key=B}):

  instrument name : test_updown_counter
  type            : SumPointData
  value           : 22
  attributes      :
        key: B
  type            : SumPointData
  value           : 12
  attributes      :
        key: A
  ...
  instrument name : test_gauge
  type            : LastValuePointData
  timestamp       : 1779971305460467084
  value           : 202
  attributes      :
        key: B
  type            : LastValuePointData
  timestamp       : 1779971305460472459
  value           : 102
  attributes      :
        key: A

Cycle 4 (callback has stopped reporting {key=B}):

  instrument name : test_updown_counter
  type            : SumPointData
  value           : 22                            ← STUCK at cycle-3 value
  attributes      :
        key: B
  type            : SumPointData
  value           : 13
  attributes      :
        key: A
  ...
  instrument name : test_gauge
  type            : LastValuePointData
  timestamp       : 1779971305460467084          ← STUCK at cycle-3 timestamp
  value           : 202                          ← STUCK at cycle-3 value
  attributes      :
        key: B
  type            : LastValuePointData
  timestamp       : 1779971306464486905
  value           : 103
  attributes      :
        key: A

The same {key=B} data point continues to appear in every subsequent collection cycle. For ObservableGauge the LastValue point's timestamp also stays frozen at the cycle-3 observation time, so the data point is structurally identical from cycle 4 onward — yet it is still emitted.

The Delta temporality path works correctly because it only emits what arrived in the current cycle's delta hash map.

Root cause analysis (v1.27.0 source)

Data flow for an async instrument under cumulative temporality:

1. AsyncMetricStorage::Record() is called with the current callback's measurements. For each attribute set, the absolute value is diffed against cumulative_hash_map_ to produce a delta which is stored in delta_hash_map_. cumulative_hash_map_ itself never has entries removed.
2. AsyncMetricStorage::Collect() moves the current delta_hash_map_ into TemporalMetricStorage::buildMetrics() and resets delta_hash_map_ to empty.
3. TemporalMetricStorage::buildMetrics() merges the current cycle's deltas into a per-collector cumulative map last_reported_metrics_[collector].attributes_map, then emits the entire map.

The problem is at temporal_metric_storage.cc#L131-L153 and #L175-L183:

if (aggregation_temporarily == AggregationTemporality::kCumulative)
{
  // merge current delta to previous cumulative
  last_aggr_hashmap->GetAllEntries(
      [&merged_metrics, this](const MetricAttributes &attributes, Aggregation &aggregation) {
        auto agg = merged_metrics->Get(attributes);
        if (agg) {
          merged_metrics->Set(attributes, agg->Merge(aggregation));
        } else {
          auto def_agg = DefaultAggregation::CreateAggregation(
              aggregation_type_, instrument_descriptor_, aggregation_config_);
          merged_metrics->Set(attributes, def_agg->Merge(aggregation));
        }
        return true;
      });
}
...
last_reported_metrics_[collector] = LastReportedMetrics{std::move(merged_metrics), collection_ts};
...
result_to_export->GetAllEntries(/* emit every key in last_reported_metrics_ */);

Every key ever inserted into last_aggr_hashmap is unconditionally carried into merged_metrics. There is no filtering against the current cycle's delta (i.e. the keys actually observed during this callback), so once a key appears it is emitted on every subsequent collection.

The Delta path skips this merge and emits only the keys present in the current cycle's delta_metrics, which is why Delta is unaffected.

This bug applies equally to all async instrument families since Meter::Create*Observable*() all go through RegisterAsyncMetricStorage():

Instrument creator	Storage
CreateInt64/DoubleObservableCounter	AsyncMetricStorage (affected)
CreateInt64/DoubleObservableGauge	AsyncMetricStorage (affected)
CreateInt64/DoubleObservableUpDownCounter	AsyncMetricStorage (affected)
CreateInt64/DoubleCounter / Histogram / UpDownCounter / sync Gauge	SyncMetricStorage (correct — spec MUST keep all attribute sets for sync cumulative)

Why this matters

In dynamic environments (Kubernetes pods, per-shard / per-tenant metrics, ephemeral resources), attribute sets naturally come and go. With this bug:

• Cardinality grows unboundedly in the backend even when the source no longer reports those series.
• Stale gauge values mislead alerts and dashboards (a deleted shard's "queue depth" stays pinned at its last value forever; for ObservableGauge even the data point's own timestamp stays frozen).
• The SDK and downstream pipeline accumulate memory over the process lifetime.

The only current workaround is to give up cumulative temporality for async instruments, which forces unrelated changes on the export pipeline.

Cross-references — sister SDKs have already fixed this

• opentelemetry-dotnet: issue #5950, fix PR #6883 (merged 2026-02-27). The PR cites the SDK spec clause for asynchronous instruments and changes the cumulative snapshot path so that metric points not reported in the current callback cycle are skipped.
• opentelemetry-rust: issue #2213, fix PR #2618 (merged 2025-02-05). Changes ObservableGauge to collect data points only since the previous collection, achieving the same spec-compliant behavior.

opentelemetry-cpp appears to be the remaining major SDK with this spec violation in the async-cumulative path.

Previously reported in this repo

Issue #2199 "measurement residual in SDK" (2023-06) reported the same symptom on v1.9.0. It was concluded at the time that the behavior matched other language SDKs and was therefore intended, and the issue went stale.

That premise no longer holds: since 2025, both opentelemetry-rust (#2618) and opentelemetry-dotnet (#6883) have treated this as a spec violation and shipped fixes. This issue asks to revisit the conclusion of #2199 and bring opentelemetry-cpp in line.