[feature request] Support disabling self-diagnostics beyond start-up

[martincostello]

Package

OpenTelemetry

Is your feature request related to a problem?

I was looking through profiles for an OTel instrumented application of mine, and noticed that code related to self-diagnostics was consistently showing up in the top 5 allocation samples:

This is essentially because SelfDiagnosticsConfigRefresher is running in the background every 10 seconds for the lifetime of the application to see if OTEL_DIAGNOSTICS.json appears/changes to update the configuration:

opentelemetry-dotnet/src/OpenTelemetry/Internal/SelfDiagnosticsConfigRefresher.cs

Lines 130 to 138 in bfa5afc

	private async Task Worker(CancellationToken cancellationToken)
	{
	await Task.Delay(ConfigurationUpdatePeriodMilliSeconds, cancellationToken).ConfigureAwait(false);
	while (!cancellationToken.IsCancellationRequested)
	{
	this.UpdateMemoryMappedFileFromConfiguration();
	await Task.Delay(ConfigurationUpdatePeriodMilliSeconds, cancellationToken).ConfigureAwait(false);
	}
	}

In a production, especially containerised, environment this seems wasteful - if the file didn't exist on start-up, it is unlikely to ever appear, and if it does exist, it's unlikely to be updated.

A way to opt-out of this always-on functionality would provide an efficiency, albeit small, improvement for production environments or when an application is not being actively troubleshooted.

What is the expected behavior?

An opt-in (potentially with intelligent defaults) mechanism to allow the background work to be disabled by the user.

Possible intelligent defaults could include:

• Only refresh for updates if the file existed on start-up
• Only refresh for updates by default in debug
• Only refresh in ASP.NET Core applications if the DOTNET_ENVIRONMENT is Development

Which alternative solutions or features have you considered?

None.

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[rajkumar-rangaraj]

I would consider self-diagnostics a critical capability. I have personally used this to troubleshoot an issue in a production containerized app by exec-ing into the container, dropping an OTEL_DIAGNOSTICS.json, and finding the root cause live without a redeploy.

The allocation overhead here is minimal, and the trade-off of being able to dynamically light up SDK-internal diagnostics in a running production process far outweighs that cost. I would be opposed to defaults that disable this in production. That is exactly where you need it most.

[martincostello]

The "potentially with intelligent defaults" part is just a suggestion.

Mainly it would just be good to have a way to turn it off if I want to. If I later know I need to debug something, then I'd remove the override. Containers are more likely to be immutable, compared to say an EC2 instance you can SSH/RDP into, so I can definitely see there's scenarios where you might want to create the file and have it "just work".

[martincostello]

Something I also considered but didn't end up writing in the original description, is that in the context of an ASP.NET Core app it could be improved by using a file-watching configuration source, but unfortunately it's the wrong layer to be able to use that.