Understanding the Costs of Default Load Balancing in Service Mesh Architecture

If you’re managing a multi-availability zone (AZ) Kubernetes cluster with a service mesh deployed, you may face an often-overlooked cost that isn't directly reflected on your dashboards.

The Default Setting Challenge

Kubernetes Services and Istio utilize a default load balancing strategy that randomly distributes traffic among all healthy endpoints, ignoring which availability zone the pods reside in. While this method may be effective for single-AZ setups, it quickly becomes disadvantageous in a multi-AZ scenario. For instance, a pod located in us-east-1a has a significant chance of communicating with a pod situated in a different AZ—roughly a two-thirds chance when operating across three AZs. This results in several inter-AZ hops for a single user request, leading to elevated latency and increased operational costs.

AWS exacerbates this issue by allowing cross-zone load balancing on Network Load Balancers (NLBs), which can route connections to targets irrespective of proximity. As a result, incoming connections can be diverted to a target in another AZ, potentially overlooking a ready target in the same AZ.

Identifying the Actual Costs

The ramifications of this default configuration manifest in two primary forms. First is latency. In a typical mid-sized production setting that processes around 3,500 requests per second (RPS) across three AZs, data tracing reveals that requests within the same AZ average latency of 15-18 milliseconds at the p50 level. Meanwhile, requests crossing AZ boundaries experience a significantly slower response, peaking around 25-30 milliseconds—40% to 65% slower overall. Since about two-thirds of requests cross AZs, the overall response time sits at approximately 24 milliseconds instead of the ideal 17 milliseconds, a difference that compounds in requests with multiple service interactions.

Second, there’s financial impact. AWS charges $0.01 per GB for traffic moving between AZs, affecting costs more than expected. Internally generated traffic, like service-to-service requests, often eclipses external usage, sometimes generating five to ten times the number of requests. Within the analyzed environment, cross-AZ traffic added up to around $600 monthly before considering other factors like Aurora cross-AZ replication. Each unique traffic mix will yield different financial impacts, so scrutinize your AWS Cost Explorer reports to adjust for your consumption accurately.

Implementing Locality-Aware Load Balancing

Addressing these issues requires employing Istio’s support for locality-aware load balancing. While a straightforward approach might suggest directing all traffic to the local AZ, this can strip away the benefits of fault tolerance during deployment issues or outages. A more effective strategy includes a weighted distribution that allocates 80% of traffic to the local AZ while spreading 10% to each of the other two AZs, combined with outlier detection for automatic removal of unhealthy endpoints.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: checkout-service-locality-lb
spec:
  host: checkout-service.prod.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
          - from: us-east-1a/*
            to:
              "us-east-1a/*": 80
              "us-east-1b/*": 10
              "us-east-1c/*": 10
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

To effectively realize this strategy, three conditions must be met:

1. Even distribution of pods across AZs: An imbalance, where one AZ holds a disproportionate share of pods, will skew traffic distributions. Implement topologySpreadConstraints with maxSkew: 1 to ensure balanced pod counts.
2. Spread ingress gateway pods across all AZs: Locality-aware routing will fail to optimize if requests funnel into ingress pods concentrated in a single AZ.
3. Disable cross-zone load balancing on NLB: Use the annotation service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false" to prevent the load balancer from reintroducing imbalance to your traffic flow.

Assessing the Results and Managing Complexity

Adopting locality-aware load balancing not only reduces latency but also mitigates the risk associated with losing an AZ. Rather than forcing the remainder to absorb a sudden influx of load, an 80/10/10 distribution allows for smoother adjustments during an outage, while outlier detection can proactively manage faults.

However, implementing this setup comes with increased operational complexity. Traffic distribution becomes less transparent; instead of a straightforward balance, engineers may need to understand and document the locality policy to avoid confusion during troubleshooting. Proper documentation can prevent issues arising from misinterpretations of traffic distribution metrics.

Pre-Implementation Recommendations

Before rolling out this configuration, it’s wise to validate it in a staging environment under realistic loads. A performance test using tools like k6 or Locust will help identify imbalances caused by uneven pod distribution. Begin with canary deployments on lower-traffic services and monitor key performance metrics such as error rates and p95 latencies before scaling to critical user-facing applications.

Locality-aware load balancing isn’t a novel concept; it’s a feature within Istio that remains underutilized. However, its benefits become apparent when users scrutinize the AWS CHARGES or during a critical AZ failure scenario. Addressing these misalignments can significantly enhance the efficiency of a multi-AZ cluster.