v0.8.0 throws std::bad_alloc exception

[lxh-ob]

Hello Together,

We sometimes get an std::bad_alloc exception when the OS fails to allocate memory.

the coredump backtrace is below.

Is there any ways to prevent process coredump？

Or, would you please catch the exeception and ignore it? It would be better to keep the program running continuously rather than terminating it due to metric issues

The version is v0.8.0. And it seems that other versions will also may trigger the exception.

?? ??:0
?? ??:0
?? ??:0
__gnu_cxx::__verbose_terminate_handler() [clone .cold] at vterminate.o:?
__cxxabiv1::__terminate(void (*)()) at ??:?
std::terminate() at ??:?
__cxa_rethrow at ??:?
prometheus::ClientMetric* std::__uninitialized_copy<false>::__uninit_copy<__gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, prometheus::ClientMetric*>(__gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, __gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, prometheus::ClientMetric*) at ??:?
prometheus::ClientMetric* std::uninitialized_copy<__gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, prometheus::ClientMetric*>(__gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, __gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, prometheus::ClientMetric*) at ??:?
prometheus::ClientMetric* std::__uninitialized_copy_a<__gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, prometheus::ClientMetric*, prometheus::ClientMetric>(__gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, __gnu_cxx::__normal_iterator<prometheus::ClientMetric const*, std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > >, prometheus::ClientMetric*, std::allocator<prometheus::ClientMetric>&) at ??:?
std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> >::vector(std::vector<prometheus::ClientMetric, std::allocator<prometheus::ClientMetric> > const&) at ??:?
prometheus::MetricFamily::MetricFamily(prometheus::MetricFamily const&) at ??:?
prometheus::Family<prometheus::Counter>::Collect() at ??:?
void prometheus::(anonymous namespace)::CollectAll<std::vector<std::unique_ptr<prometheus::Family<prometheus::Counter>, std::default_delete<prometheus::Family<prometheus::Counter> > >, std::allocator<std::unique_ptr<prometheus::Family<prometheus::Counter>, std::default_delete<prometheus::Family<prometheus::Counter> > > > > >(std::vector<prometheus::MetricFamily, std::allocator<prometheus::MetricFamily> >&, std::vector<std::unique_ptr<prometheus::Family<prometheus::Counter>, std::default_delete<prometheus::Family<prometheus::Counter> > >, std::allocator<std::unique_ptr<prometheus::Family<prometheus::Counter>, std::default_delete<prometheus::Family<prometheus::Counter> > > > > const&) at registry.cc:?
prometheus::Registry::Collect() at ??:?
prometheus::detail::MetricsHandler::CollectMetrics() const at ??:?
prometheus::detail::MetricsHandler::handleGet(CivetServer*, mg_connection*) at ??:?
CivetServer::requestHandler(mg_connection*, void*) at ??:?
handle_request at civetweb.c:?
process_new_connection at civetweb.c:?
worker_thread_run at civetweb.c:?
worker_thread at civetweb.c:?
?? ??:0

[FujiwaraNaoto]

Hi!. I’m new to this community.
I have tried to research this core-dump. And I have reached conclusion.

conclusion

The direct cause is a single line in core/src/family.cc:104:

return {family};

{family} invokes the std::initializer_list constructor of std::vector<MetricFamily>.
Elements of an initializer_list are always const, so family cannot be moved — it is
copied via MetricFamily(const MetricFamily&). That copy duplicates the nested
std::vector<ClientMetric> member (and each ClientMetric's std::vector<Label>), which
allocates heap memory. Under memory pressure this allocation throws std::bad_alloc.

Minimal fix:

return {std::move(family)};

For broader resilience against bad_alloc, also consider wrapping the collection path
(the HTTP handler) in a try/catch (const std::bad_alloc&) and returning a 5xx response.

How the Backtrace Pins It Down

Signal 1 — const& proves copy, not move

Two frames carry decisive type signatures:

prometheus::MetricFamily::MetricFamily(prometheus::MetricFamily const&)
std::vector<...ClientMetric...>::vector(std::vector<...ClientMetric...> const&)

Both take const&:

Signature	Meaning
MetricFamily(const MetricFamily&)	copy constructor
MetricFamily(MetricFamily&&)	move constructor

So the operation is a copy, confirmed. A move would show && and would never enter
std::uninitialized_copy (a copy-only routine) at all. The innermost frame reinforces this:
its source iterator type is prometheus::ClientMetric const* — reading from a const
source, i.e. copying.

Signal 2 — the caller of the MetricFamily copy ctor is Family::Collect()

The frame immediately below MetricFamily::MetricFamily(const&) — its caller — is:

prometheus::Family<prometheus::Counter>::Collect()

Therefore the MetricFamily copy is constructed directly inside Family<Counter>::Collect().
This narrows the search to "where in Collect() is a MetricFamily copy-constructed?"

Next Step

Further investigation may still be required, but I would like to try creating a PR for this fix.