HTTP & networking

#From typing a URL to rendering a page

When a page is slow, “the network is slow” is too broad a diagnosis. Separate DNS, connection establishment, server processing and rendering. This is a typical TCP-based HTTPS journey; connection reuse and caches can skip stages, and HTTP/3 uses QUIC instead of TCP.

1. The browser parses the scheme, host, port and path, checking caches and policies such as HSTS and any service worker.
2. DNS checks local caches and a recursive resolver. Without a cached answer, the resolver may ask a root server about the TLD, the TLD about the authoritative nameserver, and that server for the record, following aliases. This chain is not repeated on every visit.
3. TCP exchanges SYN, SYN-ACK and ACK. The client generally uses an ephemeral port and HTTPS usually targets port 443.
4. In TLS 1.3, ClientHello offers capabilities and a key share. The server returns its selection, key share, certificate and authentication messages. The client validates hostname and trust, and both derive encryption keys. ALPN negotiates the HTTP protocol.
5. The client sends an encrypted HTTP request and a server or proxy returns status, headers and body.
6. The browser parses HTML and CSS, builds DOM and CSSOM, fetches resources, executes JavaScript, and performs layout and paint. Some work overlaps rather than waiting for every resource.

#Layers, TCP, UDP and sockets

Layers separate responsibilities and explain why successful ping does not prove HTTPS works. OSI names seven layers: Physical, Data link, Network, Transport, Session, Presentation and Application. A practical TCP/IP model commonly groups these into Link, Internet, Transport and Application. TLS does not need a forced one-to-one layer mapping.

A port is a logical transport endpoint number; a socket is an OS communication abstraction. A TCP connection is distinguished by source IP, source port, destination IP and destination port together with the protocol. One server can serve many connections on 443 because their client endpoints differ.

One TCP read need not equal one application message; applications need framing. UDP is not always faster; QUIC adds reliable streams and congestion control above it. DNS can also use TCP, so do not reduce a protocol to one port and transport combination.

#HTTP requests, methods and status codes

HTTP semantics express intent rather than URL shape. GET and HEAD are safe and idempotent. PUT and DELETE are idempotent in intended effect, not necessarily response. POST is not inherently idempotent; PATCH depends on its operation and design. Safe means no requested business-state change, not an absence of logging or metrics.

This raw HTTP/1.1 example illustrates the format; actual wire data needs CRLF and HTTPS transport for a protected endpoint. Content-Type describes the sent body; Accept describes acceptable response media types.

GET /products/42 HTTP/1.1
Host: store.example.com
Accept: application/json
If-None-Match: "product-v7"

HTTP/1.1 304 Not Modified
ETag: "product-v7"
Cache-Control: public, max-age=60

The client already has the representation matching the ETag, so the server returns 304 without a body. Header names are case-insensitive. Do not blindly retry a POST after timeout: it may have completed while its response was lost. Use an explicitly defined idempotency-key contract when retry must avoid duplicate effects.

#HTTP/1.1, HTTP/2 and HTTP/3 in Kestrel

Loading many resources on one connection benefits from multiplexing. HTTP/2 removes HTTP-level serialization, but a lost TCP packet can still stall every stream until recovery. HTTP/3 carries streams over QUIC, so loss in one stream need not prevent delivery on another.

This complete Program.cs targets an ASP.NET Core 10 web project with a configured default HTTPS certificate. HTTP/3 additionally needs a supported platform and MsQuic, plus an open UDP port.

using Microsoft.AspNetCore.Server.Kestrel.Core;

var builder = WebApplication.CreateBuilder(args);
builder.WebHost.ConfigureKestrel(options =>
{
    options.ListenAnyIP(5001, listen =>
    {
        listen.Protocols = HttpProtocols.Http1AndHttp2AndHttp3; // (1)
        listen.UseHttps(); // (2)
    });
});
var app = builder.Build();
app.MapGet("/", (HttpContext context) => context.Request.Protocol); // (3)
app.Run();

HTTP semantics remain consistent across versions. HTTP/3 is not universally faster, and some networks block UDP, so fallback matters. See the HTTP/3 specification and Kestrel requirements.

#Keep-alive, connection pooling and socket exhaustion

TCP and TLS handshakes cost time. HTTP/1.1 reuses connections by default, while HTTP/2 can share a connection across streams. HTTP keep-alive differs from TCP keepalive probes used to detect dead idle connections. Creating a new HttpClient and handler per request wastes pools and can exhaust ephemeral ports under load.

This registration assumes a .NET 10 application with DI and Microsoft.Extensions.Http. Resolve clients through the factory; do not retain a factory-created client indefinitely if relying on handler rotation for DNS refresh.

builder.Services.AddHttpClient("catalog", client =>
{
    client.BaseAddress = new Uri("https://catalog.example.com");
    client.Timeout = TimeSpan.FromSeconds(10); // (1)
}).ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(5), // (2)
    MaxConnectionsPerServer = 20 // (3)
});

// Inside a service receiving IHttpClientFactory factory:
using var client = factory.CreateClient("catalog");
using var response = await client.GetAsync("/products/42", cancellationToken);
response.EnsureSuccessStatusCode();

A long-lived HttpClient with a suitable SocketsHttpHandler connection lifetime is another valid pattern. Complete or dispose responses, especially streaming ones, to release resources. Keep retries bounded and safe for side effects rather than treating every timeout as automatically retryable. See HttpClient guidelines.

#Cookies, tokens and CORS

A cookie is a browser storage and transport mechanism; a token represents identity or authority and can itself be stored in a cookie. They are not mutually exclusive alternatives. Secure restricts cookie transmission to HTTPS, HttpOnly prevents JavaScript reads, and SameSite constrains cross-site sending. HttpOnly does not stop XSS from issuing requests with the user's privileges.

This raw response header uses a placeholder session value; the actual value must come from authentication infrastructure.

Set-Cookie: session=SESSION_VALUE; Path=/; Secure; HttpOnly; SameSite=Lax

Modern browsers require Secure with SameSite=None, and third-party-cookie policies may still prevent sending. Cookie authentication needs appropriate CSRF defenses. An explicit bearer Authorization header is not automatically attached in the same way, but JavaScript-readable token storage increases XSS exposure.

CORS is a browser policy governing cross-origin response access. An origin is scheme, host and port, which differs from a site. This ASP.NET Core registration and middleware belong after routing and before authorization.

builder.Services.AddCors(options => options.AddPolicy("store", policy =>
    policy.WithOrigins("https://shop.example.com") // (1)
        .WithMethods("GET", "POST")
        .WithHeaders("Content-Type")
        .AllowCredentials())); // (2)

// After builder.Build() and routing:
app.UseCors("store");

A preflight is usually an OPTIONS request before a non-simple request. CORS is not authentication or a firewall: curl and server clients are not constrained by it, and some cross-origin requests reach the server even when the browser cannot read their responses. See the CORS protocol.

#Caching headers and compression

Correct caching reduces latency and server load; incorrect caching can expose user data. max-age defines freshness, public allows shared caching, and private restricts storage to a private user cache. no-cache requires revalidation before reuse rather than forbidding storage; no-store forbids storage. ETag with If-None-Match can produce a bodyless 304.

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: public, max-age=60
ETag: "catalog-v8"
Vary: Accept-Encoding
Content-Encoding: br

[Brotli-compressed representation bytes]

This schematic response describes a public catalog; the shown body describes bytes rather than sendable JSON. Vary identifies request headers that affect representation selection and cache keys. Accept-Encoding negotiates compression; br or gzip saves bandwidth at CPU cost, while already compressed images often gain little.

no-store does not erase previously cached copies. Invalidation and browser/CDN policy need a coherent design. See the HTTP caching specification.

#Proxies, load balancers and CDNs

A forward proxy represents clients on outbound requests; a reverse proxy represents servers to clients. A load balancer distributes connections or requests; a CDN brings delivery closer to users and may provide caching and edge execution.

TLS termination creates a trust boundary between proxy and application. Configure trusted forwarded headers, body limits, timeouts and health checks. CDN caching helps static assets, but cookies, Authorization and queries need deliberate cache-key handling to prevent leakage. Sticky sessions do not replace shared state, and a load balancer cannot guarantee a payment executes exactly once.

#DNS records, TTL, IPv4, IPv6 and NAT

DNS maps names to information, not simply URLs to IPs. TTL bounds record caching; it is neither connection lifetime nor a guarantee that every client switches simultaneously.

IPv4 uses 32-bit addresses; IPv6 uses 128 bits. Private IPv4 ranges are 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Loopback such as 127.0.0.1 differs from private LAN space. IPv6 includes loopback ::1, unique-local space fc00::/7 and link-local space fe80::/10.

NAT translates addresses; PAT shares a public IP using ports. Neither is encryption or a replacement for a firewall. IPv6 still requires access policies. Test both address families in dual-stack environments; a broken AAAA path can delay requests even when A works. Lowering TTL before failover helps but does not close existing pooled connections.

#WebSockets, SSE and diagnostic tools

Repeated polling wastes requests when nothing changes. WebSockets provide a long-lived bidirectional message channel; SSE provides server-to-client text events using text/event-stream, with browser reconnection through EventSource. WebSockets are not universally restricted to HTTP/1.1 Upgrade; newer HTTP versions have mechanisms where supported by the stack.

ASP.NET Core 10 provides TypedResults.ServerSentEvents. For any stream, configure cancellation, backpressure, heartbeats, proxy buffering and idle timeouts. Reconnection needs event IDs and a replay policy when lost events are unacceptable.

These Bash commands use illustrative endpoints, not a service running here. Flags such as HTTP/3 require a suitably built curl; inspect curl --version.

curl -I https://store.example.com/products/42       # (1)
curl -v --http2 https://store.example.com/products/42 # (2)
curl --compressed https://store.example.com/products/42
curl -N -H 'Accept: text/event-stream' \
  https://store.example.com/orders/42/events        # (3)
curl --resolve store.example.com:443:192.0.2.10 \
  https://store.example.com/products/42             # (4)

curl is not a browser: success does not validate browser CORS or cookie behavior. Diagnose DNS, TLS, HTTP and application behavior separately.