Enabling edge AI: building the network for distributed inference

Wherever it runs, most edge AI involves inference: applying new data to a trained model to generate an output in real-world use. As AI moves from model development to widespread adoption, inference is becoming an increasingly dominant workload, driving more processing to the edge.

Why are inference workloads moving to the edge?

Four factors are driving this infrastructure shift:

• Latency: Many emerging AI workloads can’t wait. Applications on mobile devices, factory production lines and vehicles may need on-the-spot decisions in milliseconds.
• Efficiency: Sending every raw input or video stream to the cloud is costly and often unnecessary. Processing data closer to the source reduces bandwidth and backhaul demand.
• Reliability: Reducing the number of hops by moving workloads to the edge can improve reliability.
• Compliance: Processing sensitive data locally can keep it within a specific site or jurisdiction, helping regulated organisations meet privacy, security and data-residency rules.

In short, time-sensitive processing happens locally, while central data centres handle more compute-intensive workloads. That means faster responses, improved efficiency and greater control over sensitive information.

So edge AI complements, rather than replaces, centralised data centres, redistributing AI workloads and reshaping network demand.

What’s accelerating edge AI adoption?

Computer vision, robotics, autonomous systems and predictive maintenance are all driving demand for local, real-time data processing. As AI becomes part of everyday life, it’s being embedded across more devices, locations and real-world operations.

At the same time, evolving regulations governing both data at rest and data in motion are encouraging organisations to keep data local.

Meanwhile, the volume of data generated at the edge continues to grow. An estimated 21.1bn connected IoT devices were active worldwide in 2025. That figure could reach 39bn by 2030.

Advances in AI chips and model optimisation are also fuelling adoption. With the rise of small language models designed for specific edge use cases, AI inference is becoming practical on ever-smaller, resource-constrained devices.

And investment is following suit. IDC expects worldwide edge computing spending to reach $380bn by 2028, with AI among the fastest-growing workloads.

Overall, more applications demanding on-the-spot responses, more devices, more capable hardware, and new edge-capable models are all pushing AI beyond the data centre.

Types of edge AI architectures

Edge AI spans a range of compute, from endpoint devices and local gateways to on-site infrastructure. Where each workload runs depends on latency, compute, bandwidth, resilience and data-governance requirements.

On-device edge

The model runs on hardware such as a phone, smart camera, vehicle or industrial controller. Inference can continue without connectivity, avoiding network transit time. However, compute, memory, power and thermal limits constrain the models you can run.

Gateway edge

Several devices send data to a nearby gateway for inference. In a factory, for example, multiple cameras could connect to a local server running computer-vision models. This pools compute without sending every video stream off-site.

On-site edge

AI runs on infrastructure within a factory, hospital, retail outlet or other business site. You get more compute for demanding workloads while retaining selected data on your premises.

Network edge

Compute is hosted within the service provider’s network or in a particular metro area, closer to users and connected devices. For example, StonesThro is deploying edge AI cloud nodes at mobile infrastructure sites across the UK.

Hybrid edge-cloud

Hybrid architectures split workloads and data flows between edge and central infrastructure. The edge handles time-sensitive inference, while central platforms provide greater compute, model training, fleet management and long-term storage.

So edge AI is a continuum, not a choice between edge and cloud. Move the workload, and you change the traffic.

How are AI workloads reshaping network traffic?

AI is changing not just the volume of network traffic but its shape and direction. Several shifts are already emerging.

Inference is rising

Traffic measured across two service provider networks by Cisco shows AI inference volumes growing fourfold over eight months. While the absolute volume remains small, Cisco projects that AI inference traffic could account for around 25% of network traffic by 2035.

Lasting longer

Cisco also found that inference flows last roughly twice as long as typical web transactions. Responses stream token by token, creating smoother, more sustained flows rather than short bursts. And longer sessions have knock-on effects for capacity planning, load balancing, firewalls and intrusion detection.

Turning upstream

For decades, networks have been designed on the basis that downloads dominate. But more data-intensive inputs are starting to weaken that imbalance.

For example, Ericsson reports uplink growth is already outpacing downlink for many mobile service providers. Its scenario modelling suggests additional AI use could make mobile uplink traffic three times higher in 2031 than in 2025.

Multiplying interactions

Agentic AI could shift traffic growth up another gear. One AI agent may call several models, tools and APIs as it completes a task, operating 24/7 at machine speed. For example, in one controlled test, an agent-led task generated 450% more traffic than the equivalent human-led process, with inference accounting for 70% of the additional traffic.

Redistributing traffic

Distributed inference creates a complex mix of flows between edge systems, data centres and cloud platforms:

• Inference outputs travelling to applications and control systems
• Models and software updates moving out to edge systems
• Increasing agentic AI traffic generated both centrally and locally
• Workload traffic passing directly between distributed edge sites
• Operational data returning for storage, analysis or retraining
• Telemetry flowing back for monitoring and management

So the immediate challenge isn’t simply a flood of extra data. It’s managing more distributed, sustained, upstream-heavy and increasingly business-critical flows.

How should networks adapt for edge AI?

To support edge AI, networks must provide the performance, resilience and control required across the complete inference path. Here are some factors to consider:

• Latency and location: An industrial control system demanding a response within milliseconds may need on-device or on-site inference. Less time-sensitive applications can often run at regional edge locations. However, moving inference closer doesn’t automatically guarantee a faster response. Model processing currently accounts for most of the latency in many LLM workloads, although network performance will matter more as inference accelerates.
• Capacity and scalability: Edge processing may reduce raw-data backhaul, but it creates sustained upstream, cross-site and synchronisation flows. Capacity needs to scale in both directions and across more locations.
• Resilience and diversity: Local inference can keep critical functions running when WAN connectivity fails. However, model updates, monitoring and cross-site coordination may stop. Diverse routes, rapid failover and defined fallback behaviour can protect the wider service.
• Visibility and fault detection: A slow response could be due to a fault on the device, the network, the data pipeline or the cloud platform. Correlating network, infrastructure and application telemetry makes troubleshooting easier.
• Encryption and security: While edge AI can keep sensitive data local, more devices, interfaces and sites can increase attack surfaces. Encryption, segmentation and consistent policy enforcement are essential to protect data in transit.

Ultimately, edge AI must perform securely and reliably across every node, route and location. So how do you maintain that performance at scale?

Scaling edge AI across the UK

Connectivity underpins the entire distributed inference architecture, carrying data, models and outputs between edge nodes, data centres and clouds. To scale, you need resilient, high capacity connectivity built for growing upstream, downstream and cross-site traffic.

However, not all edge AI traffic is created equal. An on-device inferencing application has very different network demands from a distributed GPUaaS platform or sovereign AI cloud.

Some use cases depend on dense metro connectivity, while others require high capacity, low latency links between data centres, edge locations and the cloud. Designing the right solution for each one is key.

At Neos Networks, we’re using our high capacity, UK-wide fibre network to help hyperscalers, neoscalers and microscalers extend edge AI networks across the UK.

Customer story: enabling edge AI with StonesThro

We’re helping pioneering UK microscaler StonesThro connect and scale its edge AI platform. Working with Cornerstone, the UK’s leading mobile infrastructure services provider, StonesThro is deploying sovereign edge AI cloud nodes at Cornerstone sites across the UK.

Learn more about StonesThro’s edge AI deployment

Build your edge AI network

If you need resilient, high capacity connectivity to support distributed AI inference across the UK, get in touch. We’ll help you explore the right connectivity for your architecture.