TSVWG Z. Li Internet-Draft Z. Du Intended status: Standards Track China Mobile Expires: 5 January 2027 J. Wang W. Cheng G. Zhang Centec X. Sun Inesa C. Zhao SAIA 4 July 2026 Transport Considerations for Large-Scale Distributed Inference Networks draft-li-tsvwg-inference-transport-00 Abstract Large-scale distributed inference systems generate traffic patterns that differ from both traditional data center workloads and distributed training workloads. Disaggregated prefill/decode serving transfers key-value cache state between server pools, and expert- parallel architectures generate all-to-all traffic among expert groups. These flows are typically carried over a small number of RDMA connections, producing low-entropy traffic that is prone to uneven link utilization under Equal-Cost Multipath (ECMP) forwarding. This document specifies transport considerations for such networks, covering path load awareness, path steering through ECMP entropy variation, ordering tolerance at the receiver, and differentiated reliability for data with different loss sensitivity. The discussion builds on existing IETF building blocks; this document does not define new protocol elements. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Li, et al. Expires 5 January 2027 [Page 1] Internet-Draft Inference Transport July 2026 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 5 January 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 4 4. Transport Considerations . . . . . . . . . . . . . . . . . . 4 4.1. Path Load Awareness . . . . . . . . . . . . . . . . . . . 4 4.2. Path Steering Using ECMP Entropy . . . . . . . . . . . . 5 4.3. Ordering Considerations . . . . . . . . . . . . . . . . . 5 4.4. Differentiated Reliability . . . . . . . . . . . . . . . 6 5. Deployment Considerations . . . . . . . . . . . . . . . . . . 6 6. Security Considerations . . . . . . . . . . . . . . . . . . . 6 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 8. Normative References . . . . . . . . . . . . . . . . . . . . 7 9. Informative References . . . . . . . . . . . . . . . . . . . 8 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 8 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8 1. Introduction Early inference serving deployed models on single servers or small clusters, with modest demands on the interconnection network. Current large-scale inference systems are different in several respects. Disaggregated serving separates the prefill phase (processing the input prompt) from the decode phase (generating output tokens) onto distinct server pools. The key-value (KV) cache Li, et al. Expires 5 January 2027 [Page 2] Internet-Draft Inference Transport July 2026 computed during prefill is transferred over the network to the decode pool, and the latency of this transfer directly affects time-to- first-token. Mixture-of-experts models deploy expert-parallel (EP) groups across many servers. Token routing between experts generates all-to-all communication whose scale grows with the EP group size, regularly crossing leaf and spine tiers of the data center fabric. This traffic is typically carried by RDMA transports such as RoCEv2 over a small number of connections between any given pair of endpoints. The resulting flows are large and few -- low-entropy traffic from the perspective of flow-based load balancing. With Equal-Cost Multipath (ECMP) forwarding, the hash function maps each flow to one path; with few flows, multiple large flows can hash onto the same link while parallel links remain idle. The operational issues of low-entropy traffic with flow-based load distribution are described in [RFC7424]. In inference fabrics, such collisions translate into jitter and increased tail latency for KV cache transfer and all-to-all exchanges. This document specifies transport-layer considerations for these networks: how endpoints can become aware of per-path load, how traffic can be steered across paths using existing ECMP mechanisms without network upgrades, what ordering tolerance is required at receivers, and how reliability can be differentiated for data with different loss sensitivity. The intent is to document the considerations and map them to existing IETF building blocks. This document does not define new protocol elements or data plane behavior. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 2. Terminology Prefill: The inference phase that processes the input prompt and produces the initial KV cache. Decode: The inference phase that generates output tokens incrementally, consuming and extending the KV cache. KV Cache: Intermediate attention state (keys and values) produced during inference. In disaggregated serving, the KV cache is transferred from prefill servers to decode servers. Li, et al. Expires 5 January 2027 [Page 3] Internet-Draft Inference Transport July 2026 Expert Parallelism (EP): A parallelization strategy for mixture-of- experts models in which experts are distributed across servers, requiring all-to-all token exchange. Message: An application-level unit of transfer (e.g., one RDMA operation). A message comprises one or more packets. Entropy: Variability in the packet header fields used by ECMP hashing to select among equal-cost paths, as discussed in [RFC7424]. 3. Problem Statement Several operational approaches are deployed today to mitigate uneven link utilization caused by low-entropy RDMA traffic. Each involves trade-offs. Increasing the number of connections: spreading traffic over more RDMA connections (queue pairs) increases entropy and reduces the probability that large flows collide on a single link; however, additional queue pairs consume NIC resources and add scheduling overhead. Placement affinity: scheduling communicating workers under the same top-of-rack switch or leaf reduces the volume of traffic crossing the tiers where collisions occur; this reduces exposure but does not change the load-distribution behavior itself. Per-hop dynamic load balancing: switches can forward packets of the same flow across different links based on real-time link load; this achieves fine-grained balance but introduces packet reordering that receivers must tolerate. A complementary approach is for endpoints to steer traffic across paths using mechanisms that ECMP fabrics already support, informed by an endpoint view of per-path load. The remainder of this document discusses the considerations for this approach. 4. Transport Considerations 4.1. Path Load Awareness Endpoint-driven path steering benefits from knowledge of the relative load of the candidate paths. Two sources of this knowledge are available with existing building blocks. On-path telemetry: in networks where devices support In situ Operations, Administration, and Maintenance (IOAM) [RFC9197], probe or data packets can collect per-hop information along their forwarding path, including transit delay and queue depth; the export of collected data is described in [RFC9326]. Endpoint estimation: where on-path support is not available, endpoints can estimate relative path load from end-to-end measurements such as round-trip time and delivery rate per path, in the manner familiar from delay-based congestion control. Li, et al. Expires 5 January 2027 [Page 4] Internet-Draft Inference Transport July 2026 A sender MAY combine both sources where available. Load information is advisory input to path steering and MUST NOT be interpreted as a congestion signal in the sense of [RFC3168]; existing congestion control behavior is unchanged. 4.2. Path Steering Using ECMP Entropy ECMP path selection is a function of packet header fields. For RoCEv2 traffic, the UDP source port is commonly included in the hash input, and varying it changes the selected path without any change to network devices; the fabric continues to perform standard flow-based ECMP. The use of header entropy for load distribution is discussed in [RFC7424], and an analogous technique using the IPv6 Flow Label is described in [RFC6438]. A sender that observes uneven path load MAY change the entropy value (e.g., the UDP source port) used for subsequent traffic on a connection, causing that traffic to be hashed onto a different path. Senders SHOULD rate-limit such changes; frequent repathing can itself induce load oscillation across the fabric. 4.3. Ordering Considerations Changing the path of in-flight traffic reorders packets across the change. The disruption can be confined by aligning path changes with application-level message boundaries: all packets of a given message SHOULD carry the same entropy value, so that each message traverses a single path and arrives in order within itself; the entropy value MAY differ between messages, distributing successive messages across paths. With this alignment, the receiver observes reordering only between messages, not within a message. Receivers of multipath traffic MUST tolerate inter-message arrival reordering. For transports where each message is independently placed in receiver memory (as with RDMA operations carrying explicit placement information), inter-message reordering does not require reassembly buffering. The message size determines the granularity of load distribution. Smaller messages distribute load more evenly but increase per-message overhead; larger messages reduce overhead but coarsen the distribution. A sender MAY adjust message sizing based on observed path balance, preferring larger messages on lightly loaded paths. Li, et al. Expires 5 January 2027 [Page 5] Internet-Draft Inference Transport July 2026 4.4. Differentiated Reliability Inference traffic is not uniformly sensitive to loss. The sensitivity of model state to perturbation varies with position in the model; loss affecting early-layer data propagates through all subsequent computation, while loss affecting late-layer data has more bounded effect on output quality. This creates an opportunity for differentiated reliability, for which the IETF has established precedents: partial reliability in SCTP [RFC3758] allows a sender to abandon delivery of selected data, and the QUIC DATAGRAM extension [RFC9221] provides unreliable delivery within a reliable connection. When the application indicates the loss sensitivity of the data it submits (for example, by model layer), the transport MAY apply full retransmission to loss-sensitive data and bounded or no retransmission to loss-tolerant data, particularly under high path load. The mapping from data category to reliability level is an application policy decision; it SHOULD be set so that service quality objectives (such as response accuracy and token latency) are preserved. How the application communicates this indication to the transport is a local interface matter outside the scope of this document. 5. Deployment Considerations The path steering requires only standard ECMP in the fabric and is therefore deployable incrementally: endpoints that implement it coexist with endpoints that do not. On-path telemetry is an optimization, not a dependency; endpoint estimation suffices where IOAM support is absent. Path steering and per-hop dynamic load balancing should not operate on the same traffic simultaneously without coordination, as independent repathing decisions at both the endpoint and the fabric can interact unpredictably. Differentiated reliability should be introduced conservatively, with loss-tolerant treatment applied only where its effect on inference quality has been validated for the model in use. 6. Security Considerations Path load information, whether collected via IOAM or estimated at endpoints, reveals aspects of fabric topology and utilization. The security considerations of [RFC9197] and [RFC9326] apply to telemetry collection and export; access to collected data SHOULD be restricted to authorized components. Entropy-based path steering uses header fields that are part of normal traffic; it does not introduce new spoofing surface beyond that of the underlying transport. However, an endpoint that repaths Li, et al. Expires 5 January 2027 [Page 6] Internet-Draft Inference Transport July 2026 aggressively can concentrate load deliberately; fabrics serving multiple tenants SHOULD apply the usual per-tenant rate and resource isolation. Differentiated reliability relies on application indications of loss sensitivity. A compromised or misconfigured application could mark loss-sensitive data as tolerant, degrading its own service quality; this is contained within the application's own traffic. 7. IANA Considerations This document has no IANA actions. 8. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, . [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. Conrad, "Stream Control Transmission Protocol (SCTP) Partial Reliability Extension", RFC 3758, DOI 10.17487/RFC3758, May 2004, . [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label for Equal Cost Multipath Routing and Link Aggregation in Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, . [RFC7424] Krishnan, R., Yong, L., Ghanwani, A., So, N., and B. Khasnabish, "Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks", RFC 7424, DOI 10.17487/RFC7424, January 2015, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . Li, et al. Expires 5 January 2027 [Page 7] Internet-Draft Inference Transport July 2026 [RFC9197] Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197, May 2022, . [RFC9221] Pauly, T., Kinnear, E., and D. Schinazi, "An Unreliable Datagram Extension to QUIC", RFC 9221, DOI 10.17487/RFC9221, March 2022, . 9. Informative References [RFC9326] Song, H., Gafni, B., Brockners, F., Bhandari, S., Mizrahi, T., Sivakolundu, R., Li, Z., and T. Zhou, "In Situ Operations, Administration, and Maintenance (IOAM) Direct Exporting", RFC 9326, DOI 10.17487/RFC9326, November 2022, . Acknowledgements The authors acknowledge ongoing IETF discussion of AI workload networking, including problem statements on training-network load balancing and congestion, which provides context for the inference- specific considerations in this document. Authors' Addresses Zhiqiang Li China Mobile Beijing 100053 China Email: lizhiqiangyjy@chinamobile.com Zongpeng Du China Mobile Beijing 100053 China Email: duzongpeng@chinamobile.com Junjie Wang Centec Shanghai 201203 China Li, et al. Expires 5 January 2027 [Page 8] Internet-Draft Inference Transport July 2026 Email: wangjj@centec.com Wei Cheng Centec Shanghai 201203 China Email: chengw@centec.com Guoying Zhang Centec Shanghai 201203 China Email: zhanggy@centec.com Xun Sun Inesa Shanghai 200030 China Email: sunxun@inesa.com Chunhao Zhao SAIA Shanghai 200125 China Email: chunhao.zhao@sh-aia.com Li, et al. Expires 5 January 2027 [Page 9]