Logging & observability

#Logs, metrics and traces: different questions

“Payments are slow” is not a diagnosis. You need impact, the delayed stage and failure details. Observability is the ability to infer internal system behavior from its signals, rather than merely installing a dashboard.

Connect signals with service, environment, version and trace identity. Sampling means not every trace is stored, and no error log does not prove success. Telemetry needs limits, redaction and export-failure monitoring too.

#Structured logging, LoggerMessage and Serilog

A flattened log string makes questions such as “which orders exceeded this amount?” require parsing. A message template preserves named properties when the provider retains structure. Keep property names stable across requests.

logger.LogInformation($"Order {orderId} paid {amount}");

logger.LogInformation("Order {OrderId} paid {Amount}", orderId, amount);

using Microsoft.Extensions.Logging;

public static partial class OrderLog
{
    [LoggerMessage(EventId = 1001, Level = LogLevel.Information,
        Message = "Order {OrderId} paid {Amount}")]
    public static partial void Paid(
        ILogger logger, Guid orderId, decimal amount);     // (1)
}
// At the call site: OrderLog.Paid(logger, orderId, amount);

// Packages: Serilog.AspNetCore and Serilog.Sinks.Console
using Serilog;

builder.Host.UseSerilog((context, configuration) => configuration
    .MinimumLevel.Information()
    .Enrich.FromLogContext()
    .WriteTo.Console());                                  // (2)

Do not log tokens, passwords, card data or entire request bodies by default. Avoid logging one exception at every layer; record it at the boundary responsible for its outcome. See source-generated logging.

#Log levels, scopes and correlation

W3C traceparent contains version, trace ID, parent span ID and flags. Each span has its own ID, while related spans share a trace ID. A business correlation ID may span retries or days and need not equal a trace ID.

using System.Diagnostics;

using var scope = logger.BeginScope(new Dictionary<string, object?>
{
    ["OrderId"] = orderId,
    ["TraceId"] = Activity.Current?.TraceId.ToString() // (1)
});
logger.LogWarning("Payment confirmation is delayed"); // (2)

ASP.NET Core and HttpClient support propagation, but broker messages need context injection and extraction through instrumentation or a framework. Inbound correlation headers are not identity evidence: validate format and size, and never put secrets in baggage.

#OpenTelemetry instrumentation and OTLP

Instrument application behavior independently of a specific backend. In .NET, ActivitySource produces spans and Meter produces measurements. The OpenTelemetry SDK selects sources, processes signals, samples and exports. OTLP is a transport protocol, rather than a database or dashboard.

// Packages: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol,
// OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http,
// OpenTelemetry.Instrumentation.Runtime
using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("orders-api")) // (1)
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("Shop.Checkout")
        .AddOtlpExporter())                                         // (2)
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("Shop.Orders")
        .AddOtlpExporter());                                        // (3)
builder.Logging.AddOpenTelemetry(logging => logging.AddOtlpExporter()); // (4)

Configure OTLP endpoint, protocol, TLS and authentication for the collector, commonly using OTEL_EXPORTER_OTLP_ENDPOINT. A collector can batch, redact and route telemetry. Bound queues and monitor dropped data; exporting does not prove storage, and unintended duplicate pipelines increase cost. See .NET OpenTelemetry guidance.

#Custom spans and bounded-cardinality metrics

An HTTP span measures a request but may not explain an inventory-reservation delay. Add spans around meaningful business steps and aggregate metrics for their durations. Never use order IDs or emails as metric labels: each new value creates another time series.

using System.Diagnostics;
using System.Diagnostics.Metrics;

public static class OrderTelemetry
{
    public static readonly ActivitySource Source = new("Shop.Checkout");
    public static readonly Meter Meter = new("Shop.Orders");
    public static readonly Counter<long> Completed =
        Meter.CreateCounter<long>("orders.completed");           // (1)
    public static readonly Histogram<double> Duration =
        Meter.CreateHistogram<double>("orders.duration", "s"); // (2)
}

// Inside an async service method; ReserveAsync is an application operation.
using var activity = OrderTelemetry.Source.StartActivity("reserve-stock");
long started = Stopwatch.GetTimestamp();
try
{
    await ReserveAsync(ct);
    OrderTelemetry.Completed.Add(1);
}
catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error, ex.GetType().Name); // (3)
    throw;
}
finally
{
    OrderTelemetry.Duration.Record(
        Stopwatch.GetElapsedTime(started).TotalSeconds);          // (4)
}

Keep ActivitySource and Meter long-lived. Bounded operation and outcome tags suit metrics; identifiers belong in appropriately protected logs or traces. Head sampling is cheaper, while collector tail sampling can select slow or failed traces at the cost of buffering and processing.

#RED, USE, histograms and percentiles

RED describes services: Rate, Errors and Duration. USE describes resources: Utilization, Saturation and Errors. Utilization differs from saturation: queued CPU work or connection-pool waiters can reveal a bottleneck before requests fail.

A p95 means approximately 95% of observations in the window are at or below that value, rather than necessarily 95% of users. Averages hide tail latency. Do not average instance p95 values; aggregate compatible distributions before computing the percentile.

Requests: 95 at 100 ms, 5 at 2000 ms                       // (1)
Mean: (95 * 100 + 5 * 2000) / 100 = 195 ms
Tail: five requests still take two seconds
SLO threshold: count requests <= 300 ms / eligible requests // (2)

#SLIs, SLOs, error budgets and alerts

An SLI measures service experience, such as successful checkouts among eligible requests. An SLO sets a target over a window; an SLA is a contractual commitment with possible consequences. Define success, exclusions and measurement source: an HTTP 200 can still contain a failed business operation.

SLO: 99.9% successful eligible requests over 30 days
Budget fraction: 1 - 0.999 = 0.001                         // (1)
At 1,000,000 eligible requests: 1,000 failures allowed
Observed error ratio: 1% => burn rate = 0.01 / 0.001 = 10  // (2)

Page for actionable symptoms: user failures, latency, oldest-message age and sustained rapid budget burn. Combine short and long windows to confirm impact and reduce noise. Each alert needs an owner, runbook, dashboard and action. High CPU alone often belongs in investigation or capacity planning rather than automatic paging.

The budget informs priorities: heavy consumption favors reliability work and lower deployment risk under an agreed policy, rather than an automatic ban on all feature development.

#Liveness, readiness and startup checks

Liveness asks whether the process needs restarting; readiness asks whether it can accept traffic now. An orchestrator's startup probe can allow initialization time before normal probes apply. Restarting cannot repair an unavailable external database; place that dependency in readiness if it is essential.

using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;

builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
    .AddCheck<DatabaseReadiness>("database", tags: ["ready"]); // (1)

// After builder.Build():
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = registration => registration.Tags.Contains("live") // (2)
});
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = registration => registration.Tags.Contains("ready")
});

// Application check; IOrderStore is an injected database adapter.
public sealed class DatabaseReadiness(IOrderStore store) : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken ct = default)
    {
        return await store.CanConnectAsync(ct)
            ? HealthCheckResult.Healthy()
            : HealthCheckResult.Unhealthy("Database unavailable"); // (3)
    }
}

Do not make every optional dependency a readiness requirement; that can remove all useful degraded service. Bound timeouts, probe frequency and thresholds so health checks do not overload dependencies. See health-check guidance.

#Production debugging and .NET diagnostics

Start with impact rather than a tool. When users are failing now, a safe mitigation can matter more than completing the investigation first.

1. Establish affected users, failure rate, timing and critical paths; choose evidence-based rollback, feature isolation or dependency mitigation.
2. Use metrics to narrow endpoint, region and version, comparing deployments and traffic changes.
3. Follow an affected trace's critical path and read related logs using its trace ID.
4. Test one hypothesis with suitable evidence: CPU samples, thread stacks or heap retention.
5. Reproduce, add a regression test, roll out a fix gradually and watch the same SLI.
6. Record a blameless timeline, contributing factors and owned, dated follow-up actions.

# Tools must be installed; replace 1234 with the target process ID.
dotnet-counters monitor --process-id 1234 --counters System.Runtime
dotnet-trace collect --process-id 1234 --duration 00:00:30
dotnet-dump collect --process-id 1234 --output process.dmp
dotnet-dump analyze process.dmp

Bound collection duration, disk use and operational impact. Dumps can contain tokens and personal data. A growing heap is not automatically a leak; inspect retention and roots over time. A high thread count alone does not justify increasing ThreadPool minimums. See counters, trace collection and dump analysis.