Error EIO: [Errno 110] Connection timed out

[stumbaumr]

Hi,
I activated a while ago the telemetry on my three node ceph cluster and just got out of bed at 1 am because of a lot of SMS messages being sent to my mobile...

root@proxmox01:~# ceph health detail
HEALTH_ERR Module 'telemetry' has failed: HTTPSConnectionPool(host='telemetry.ceph.com', port=443): Max retries exceeded with url: /report (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9949274550>: Failed to establish a new connection: [Errno 110] Connection timed out',))

No idea if this is intended that the whole cluster has an ERR health status now just because sending telemetry fails.

I think it is better to turn telemetry off for a while...

Upsi, can't turn it off...

root@proxmox01:~# ceph telemetry off
Error EIO: Module 'telemetry' has experienced an error and cannot handle commands: HTTPSConnectionPool(host='telemetry.ceph.com', port=443): Max retries exceeded with url: /report (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9949274550>: Failed to establish a new connection: [Errno 110] Connection timed out',))
root@proxmox01:~#

[dmick]

We moved the telemetry server to a different location today; I would expect maybe a few failures, but it should be working again now?

I agree that "entire cluster health error" is maybe not advisable, and that it ought to be disableable regardless of past errors (or else the error message should be cleaned up). Both of those, however, are Ceph issues, not with this repo (which is only the server-side scripting).

[dmick]

apologies, anyway, for waking you.

[stumbaumr]

Good morning!

So should I close the issue here and reopen in ceph/ceph?

Mmmmh, there is no issue tab and no hint in the Readme on how/where to open issues...

[dmick]

ceph issues can be filed at tracker.ceph.com

[stumbaumr]

Just got above error again.
That is it for a while of telemetry data from me.

[dmick]

Sorry things failed. Did you ever file an issue at tracker.ceph.com? If so, has it gotten any attention?