nmop Y. Fu Internet-Draft Q. Sun Intended status: Standards Track X. Song Expires: 7 January 2027 C. Xie China Telecom 6 July 2026 Token Operation Problem Statement draft-fu-nmop-tokenops-probelem-statement-00 Abstract Distributed LLM inference relies heavily on high-performance networking to synchronize states across accelerators (e.g., GPUs) and nodes. Unlike traditional web services, inference workloads particularly those involving Mixture-of-Experts (MoE) models and long-context windows exhibit unique traffic patterns characterized by massive east-west traffic and strict latency constraints. Current network infrastructures and scheduling methods often treat compute resources and network paths independently, leading to suboptimal performance and degraded Quality of Experience (QoE). This document elaborates on these issues to guide potential protocol enhancements within the IETF. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 7 January 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. Fu, et al. Expires 7 January 2027 [Page 1] Internet-Draft Token Operation July 2026 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 1.2. Definition and Terminology . . . . . . . . . . . . . . . 4 2. Architecture for Distrubuted Inference . . . . . . . . . . . 5 3. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 7 3.1. Amplified Inter-GPU Communication Tail Latency Degrades Overall Inference Efficiency . . . . . . . . . . . . . . 7 3.2. Disaggregated Prefill-Decode Deployment Causes Emerging KV Cache Transmission Bottlenecks . . . . . . . . . . . . . 7 3.3. Decoupled Modeling and Network Scheduling Leads to Suboptimal Inference Service Experience . . . . . . . . . 8 3.4. Isolated Inference and Network Metrics Impair Fault Localization and Experience Assurance . . . . . . . . . . 9 4. Security Considerations . . . . . . . . . . . . . . . . . . . 9 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 6. Normative References . . . . . . . . . . . . . . . . . . . . 9 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 10 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 1. Introduction Prior to 2023, the artificial intelligence (AI) industry was predominantly focused on model training. The primary objective was to build larger models by aggregating massive computational resources, where network infrastructure served as a peripheral utility to interconnect GPU clusters. During this era, the optimization goals for network services were strictly confined to maximizing collective communication efficiency and cluster scalability to ensure high utilization of training compute resources. Fu, et al. Expires 7 January 2027 [Page 2] Internet-Draft Token Operation July 2026 However, by 2026, AI deployment has entered a phase of large-scale commercialization. The AI operation has decisively shifted from training to inference. The sustainable, stable, low-cost token generation capability directly determines the commercial viability of AI services. Consequently, the network is no longer merely a support facility for compute; it has evolved into the critical backbone of the token production pipeline. The network directly dictates key business metrics, including Time to First Token (TTFT) etc., and overall operational expenditure. The transition to inference-centric AI is accompanied by a fundamental transformation in application architecture, specifically the emergence of Agentic AI. This shift introduces novel traffic patterns that legacy network designs fail to accommodate. • Traditional Chatbots, characterized by stateless, single-turn interactions with short contexts. These workloads are compute- intensive but network-light, as the volume of data transferred is minimal relative to the computation performed. • Agentic AI, characterized by long-lived sessions, multi-turn interactions, extensive tool calling, and high reuse rates of KV- Cache (Key-Value Cache). In Agentic workflows, agents frequently reference historical context to perform complex reasoning. This dynamic shifts the network bottleneck: "computation decreases while data movement increases." Since the KV-Cache constitutes a significant portion of the session state and must persist across interactions, the network becomes the primary conduit for state retrieval and sharing. In this context, the network is no longer an insignificant "pipe" but the dominant factor determining inference performance and latency.. The split between training-era and inference-era network positioning creates incompatible design requirements for token operation traffic steering • Network for Training: The network acts as supporting infrastructure exclusively for interconnecting GPU compute clusters. Its sole design objective is to eliminate communication bottlenecks so that GPU arithmetic units can operate at full utilization. Traffic decisions prioritize bulk synchronous collective communication, batch-level load balancing, and long-timescale cluster resource planning. Token generation, if present, is a minor side workload without dedicated network optimization logic. Fu, et al. Expires 7 January 2027 [Page 3] Internet-Draft Token Operation July 2026 • Network for Inference & Agent: The network becomes the core backbone of the end-to-end token production pipeline. All critical commercial metrics，such as TTFT etc., directly depend on dynamic network scheduling of KV-cache state, agent context, and sequential token streams. Network decisions must jointly evaluate real-time compute capacity, cross-node state transfer overhead, path latency, session affinity, and persistent cache locality to select optimal service instances for each agent request. To address these issues, traffic steering mechanisms must evolve to consider Token-specific metrics. The process of selecting service instances and routing traffic based on the state of KV-Caches, the locality of context, and the specific requirements of the inference phase (prefill vs. decode) is essential. This draft defines this emerging requirement as Token-Aware Traffic Steering (TATS), a necessary evolution to support the next generation of AI infrastructure. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 1.2. Definition and Terminology • AI：Artificial intelligence • KV-Cache：Key-Value Cache • AI Gateway：AI GW • QoE：Quality of Experience • QoS：Quality of Service • TTFT：Time To First Token • TE：Traffic Engineering • TPOT：Traffic Engineering Fu, et al. Expires 7 January 2027 [Page 4] Internet-Draft Token Operation July 2026 2. Architecture for Distrubuted Inference The distrubuted inference network architecture characterizes three distinct scenorios of communication interactions, which differ in traffic characteristics, performance requirements. Scenorio 1: Client-to-Ingress Service Communication. This scenorio carries the service traffic between end-users and the inference service ingress. Its performance evaluation focuses primarily on user-perceived latency, request success rate, Time-To- First-Token (TTFT), and the stability of streaming output (e.g., token inter-arrival jitter). Consequently, the network path should exhibit low latency and high availability to ensure a satisfactory Quality of Experience (QoE). Scenorio 2: Intra-Cluster High-Frequency Communication. This scenorio refers to high-bandwidth, low-latency interactions within the inference cluster. Traffic patterns include inter-GPU tensor synchronization, pipeline stage handovers, Mixture-of-Experts (MoE) routing, and KV Cache read/write operations as well as live migration. Due to the fine-grained nature of these transactions, this scenorio imposes stringent requirements on the fabric, mandating minimal latency, high bandwidth, and ultra-low jitter. Scenorio 3: Cross-Cluster, Cross-Domain, and Cloud-Edge Coordination Communication. This scenorio refers to communication across clusters, geographical regions, and cloud-edge infrastructures. Use cases include task scheduling across heterogeneous compute centers, cross-site KV Cache migration, disaster recovery mechanisms for model services, and edge- localized inference offloading. Unlike intra-cluster communication, which prioritizes raw throughput, the efficiency of this communication class relies heavily on the capabilities of the carrier network, specifically regarding intelligent path selection, Traffic Engineering (TE), and robust Quality of Service (QoS) assurance mechanism. Fu, et al. Expires 7 January 2027 [Page 5] Internet-Draft Token Operation July 2026 +-------------+ | Client | | | +-------------+ │ +-------------v---------------+ | AI GW | | | +-----------------------------+ | | +-----------------------v-------------------------+ ( ISP Network ) ( ) ( ) ( +----------+ +---------+ +------+ ) ( | Routing | | Traffic | | SLA | ) ( |Forwarding| | Steering| | | ) ( +---------+ +---------+ +------+ ) ( ) ( ) ( ) +--------------------------------------------------+ | | | | | | +-------------v-------------+ +---------------v------------+ | | | | | CLoud 1 | | Cloud 2 | | | | | | +--------------------+ | | +---------------------+ | | | | | | | | | | | KV Cache | | | | KV Cache | | | | | | | | | | | +-----------^--------+ | | +----------^----------+ | | | | | | | | | | | | | | +---------+-------+----<----->----+--------+--------+ | | | | | | | | | | | | | | | | | | | | | |+---v---+ +---v--+ +--v--+ | | +---v---+ +--v---+ +--v--+ | ||Prefill| |Decode| | GPU | | | |Prefill| |Decode| | GPU | | |+-------+ +------+ +-----+ | | +-------+ +------+ +-----+ | | | | | +---------------------------+ +----------------------------+ Fu, et al. Expires 7 January 2027 [Page 6] Internet-Draft Token Operation July 2026 3. Problem Statement As inference architectures evolve towards tensor/pipeline/expert parallelism, prefill/decode disaggregation, and geo-distributed serving, the network has transitioned from a passive transport medium to a critical performance bottleneck. This section outlines four specific problems: amplified tail latency in GPU clusters, KV Cache transfer bottlenecks, the disconnect between compute and network scheduling, and the lack of observability correlation between application-level metrics and network telemetry. These problems motivate the need for enhanced protocols and metrics to support AI- driven traffic engineering. 3.1. Amplified Inter-GPU Communication Tail Latency Degrades Overall Inference Efficiency Modern large model inference relies on tensor parallelism, pipeline parallelism, and MoE expert parallelism, which require collaborative computation across multiple GPU nodes and network devices. * The Problem: Distributed training and inference rely heavily on collective communication. In these patterns, the straggler effect is pronounced; a delay or jitter on a single node or link forces all other participating nodes to wait. Consequently, network tail latency is amplified, directly degrading the token generation rate. Existing network mechanisms lack dedicated optimization for fine-grained, synchronous AI collective communication flows. Conventional cluster network designs focus on average latency rather than tail latency metrics, and static task scheduling fails to adapt to dynamic network jitter and congestion. No standardized metric system exists for characterizing inference- specific network performance, nor are there defined telemetry indicators for tail latency, jitter, and collective communication completion status targeted at GPU cluster inference workloads. 3.2. Disaggregated Prefill-Decode Deployment Causes Emerging KV Cache Transmission Bottlenecks To optimize resource utilization, modern inference architectures adopt separated Prefill and Decode deployment modes. In this model, the compute-intensive Prefill phase and the memory-bandwidth- intensive Decode phase are scheduled on different sets of resources. While this improves compute efficiency, it introduces a heavy dependency on network bandwidth. * The Problem: with the rapid growth of long-context inference, multi-turn dialogue interactions, and concurrent inference requests, the volume of cross-node KV Cache transmission increases Fu, et al. Expires 7 January 2027 [Page 7] Internet-Draft Token Operation July 2026 exponentially, introducing severe bandwidth pressure and queuing latency. Traditional network scheduling and task placement strategies ignore session-level context continuity and KV Cache transmission requirements. Current task scheduling policies prioritize GPU resource occupancy while neglecting end-to-end network path quality between Prefill and Decode nodes. Existing schedulers often select Decode nodes based solely on GPU availability, ignoring the network "distance" or available bandwidth required to transfer the KV Cache, leading to increased Time-To-First-Token (TTFT) and inter-token latency. There is no standardized mechanism for exposing application-layer KV Cache transmission demands, including context length, session stickiness, latency tolerance, and cache migration requirements, to the network plane. The absence of unified priority marking and QoS guarantee mechanisms for KV Cache flows further exacerbates transmission instability in large-scale concurrent scenarios. 3.3. Decoupled Modeling and Network Scheduling Leads to Suboptimal Inference Service Experience Current orchestration systems typically treat compute/model scheduling and network routing as independent silos. * The Problem: the distributed inference scheduling systems primarily optimize computing/modeling resource indicators, including GPU idle rate, memory footprint, and model loading status, while neglecting network path quality between users and computing nodes. A computing node with sufficient idle resources may still deliver poor user experience due to high end-to-end latency, persistent packet loss, cross-domain transmission instability, or link congestion. The decoupling of computing/ modeling scheduling and network scheduling results in a common paradox: idle computing/modeling resources cannot be fully utilized due to network constraints, while high-quality network paths cannot be matched with latency-sensitive inference tasks. No standardized compute-network joint scheduling model is available for AI inference services. The existing IETF framework lacks explicit definitions for inference-specific joint metrics, maybe CATS (Compute-Aware Traffic Scheduling) [I-D.ietf-cats-metric-definition] related? There is no unified interface for computing nodes to expose real-time computational load status or for network devices to feedback path quality indicators, making it impossible to implement fine-grained, experience-oriented collaborative scheduling for agent collaboration, real-time dialogue, and multimedia inference services. Fu, et al. Expires 7 January 2027 [Page 8] Internet-Draft Token Operation July 2026 3.4. Isolated Inference and Network Metrics Impair Fault Localization and Experience Assurance In current industrial deployment, inference service metrics and network performance metrics are collected and maintained by independent platforms without unified correlation and association. * The Problem: user-facing experience indicators (e.g., Time-To- First-Token (TTFT), Time-Per-Token (TPOT), end-to-end inference latency, request success rate) are decoupled from underlying network metrics (e.g., packet loss, jitter, retransmission rate, link congestion). When service degradation occurs, operators cannot accurately locate root causes among model computation overhead, task queuing delay, KV Cache transmission latency, or network link anomalies. The lack of a cross-layer unified observability system breaks the end-to-end inference service link. No standardized flow identification, marker fields, or telemetry reporting formats are defined for AI inference traffic. There is no unified specification for associating request IDs, session IDs, model instances, GPU node information, and network path telemetry data, which hinders automated fault diagnosis, service quality auditing, and refined experience assurance for large-scale intelligent agent and LLM inference services. 4. Security Considerations TBD 5. IANA Considerations This document has no IANA actions. 6. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . Fu, et al. Expires 7 January 2027 [Page 9] Internet-Draft Token Operation July 2026 [I-D.ietf-cats-metric-definition] Yao, K., Li, C., Contreras, L. M., Ros-Giralt, J., and G. Zeng, "CATS Metrics Definition", Work in Progress, Internet-Draft, draft-ietf-cats-metric-definition-10, 22 June 2026, . Acknowledgements TBD Authors' Addresses Yu Fu China Telecom Beijing China Email: fuy44@chinatelecom.cn Qiong Sun China Telecom Beijing China Email: sunqiong@chinatelecom.cn Xin Song China Telecom Beijing China Email: songx18@chinatelecom.cn Chongfeng Xie China Telecom Beijing China Email: xiechf@chinatelecom.cn Fu, et al. Expires 7 January 2027 [Page 10]