pageserver deploy: .../location_config: heatmap upload fails with noisy WARN message

[problame]

Problem

During pageserver deploys, /location_config calls that request heatmap flush (flush_ms=Some(...)) create noise WARN messages because the secondary_controller shuts down before the mgmt API

2024-10-29T16:20:45.308290Z  WARN request{method=PUT path=/v1/tenant/TENANT_ID/location_config request_id=8dde6bcd-7e0d-4bc3-99cc-6757ffcfba98}: Failed to flush heatmap: Tenant TENANT_ID is not active

Code

neon/pageserver/src/http/routes.rs

Lines 1594 to 1606 in 34b6bd4

	if let Some(_flush_ms) = flush {
	match state
	.secondary_controller
	.upload_tenant(tenant_shard_id)
	.await
	{
	Ok(()) => {
	tracing::info!("Uploaded heatmap during flush");
	}
	Err(e) => {
	tracing::warn!("Failed to flush heatmap: {e}");
	}
	}

Solution

Introduce distinguished error and log at INFO level.

We can't do much better than this right now because we keep the whole mgmt API online during PS shutdown.
We could improve this fundamentally with a middleware that takes down most of the mgmt API early, except the status & healthcheck endpoints.
But such a change is bigger and should go through the RFC process.

[jcsp]

In #9574 we are hitting issue while calling into the specific heatmap upload API, whereas this ticket is in the implicit upload during transition to AttachedStale -- let's check we aren't accidentally doing both in the controller's live migration code.