AI Inference: Edge vs Cloud

The placement spectrum

What drives the choice

Five axes:

1. Model capability needed. The largest models (hundreds of billions of parameters) only run on centralized GPU clusters. Phones can run small distilled models; quality is task-dependent.
2. Latency budget. Voice assistants need <100ms wake response. Real-time translation needs <200ms. Chat is fine at 500ms-2s TTFT. Tighter budgets push inference closer to the user.
3. Privacy / data sensitivity. If inputs cannot leave the device (medical, financial, certain enterprise), on-device is mandatory.
4. Input data size. Small text: cheap to send anywhere. Video frames at 30 fps: expensive to send; do inference locally.
5. Cost structure. Per-call cloud APIs are expensive at scale; per-device or per-region capacity flips that for high-volume workloads.

The bandwidth equation

For text, bandwidth is trivial. For multimodal workloads:

• A 720p video frame is ~300 KB compressed. At 30 fps that's 9 MB/s per camera.
• An hour of HD video is several GB.
• Round-trip-time to a regional cloud is 30-100 ms; to a centralized cloud is 50-200 ms.

For real-time video AI (security cameras, retail analytics, vehicle perception), sending raw streams to the cloud is bandwidth-prohibitive. Edge inference produces a small structured output (object boxes, classifications) that travels cheaply.

On-device LLM constraints

Modern phones with dedicated NPUs can run quantized small LLMs at acceptable speeds. The constraints:

• RAM. A 4-bit-quantized 7B model is ~3.5 GB. Phones with 8 GB RAM can host one with reserve for the OS. Below that, smaller models only.
• Memory bandwidth. Decode is memory-bandwidth-bound. Phone memory bandwidth is 50-100 GB/s; data center HBM is 3000+ GB/s. Per-token decode is correspondingly slower.
• Battery. Sustained inference is power-hungry. Continuous LLM streaming drains a battery quickly.
• Thermal. Phones throttle when hot. Heavy inference for minutes degrades throughput.

For short bursts (a few seconds of generation), on-device is fine. For long-running agentic workloads, the phone is the wrong host.

The hybrid pattern

The most common production architecture is hybrid: small tasks on device, large tasks in cloud. Examples:

• Wake-word and intent classification on device. Full conversation in cloud.
• Real-time camera object detection on edge. Cloud summarization for analytics.
• Local first-pass query understanding. Cloud retrieval + generation for the answer.
• On-device draft model. Cloud verifies via speculative decoding.

The data flow choreography is the engineering challenge — caching, fallback, partial results.

Edge model deployment

For edge / on-device, getting the model there is a network problem. Approaches:

• Bundle in app binary. Simple but bloats download size and updates require app re-release.
• Download on first run. Smaller initial binary; one-time bandwidth hit per device.
• Differential updates. Model patches, not full re-downloads, when weights change.
• Streaming layer-by-layer load. Begin inference before the whole model downloads. Rare in practice.

For CDN-edge deployment, models are stored at the edge POP itself, replicated across POPs by the CDN's distribution mechanism.

Cost model differences

For high-volume workloads, the unit economics of on-device or self-hosted often beat per-call APIs. For variable or low-volume workloads, per-call cloud APIs are cheaper because they amortize the capacity across many customers.

Offline capability

If the application must function without internet, on-device is mandatory. Offline use cases include in-vehicle assistants where cellular coverage is spotty, field workers in remote areas, secure facilities, and applications where users frequently lose connectivity.

Frequently Asked Questions

What is the difference between edge and cloud inference?

Cloud inference runs models in centralized data centers — large GPUs, big models, no per-user hardware. Edge inference runs models closer to users: on the device itself (phone, laptop), on a nearby edge POP, or on a regional data center. The tradeoffs are model size (cloud can run much larger models), latency (edge is closer), bandwidth (edge avoids sending data to the cloud), and cost structure (cloud is per-call, edge is per-device or per-region capacity).

What models can run on a phone?

As of the mid-2020s, models in the 1B-8B parameter range run on flagship phones using quantization (int4 or int8). Apple, Google, and Samsung have integrated dedicated NPUs that accelerate this. Models above 10B parameters generally need more RAM and bandwidth than current phones provide. Specialized small models for narrow tasks (intent classification, simple summarization) work well; general-purpose chat is constrained.

When does on-device inference make sense?

When latency matters (sub-100ms targets), when data should not leave the device (privacy-sensitive inputs), when the application must work offline, or when per-call cloud costs would dominate at scale. Voice assistants doing wake-word and intent detection, on-device translation, photo organization, and predictive text are common cases.

What is edge AI in the CDN sense?

CDN-style edge AI runs inference at the CDN POPs — close to users, far from origin data centers. Used for low-latency tasks like content classification, personalization, or running small models on per-request data. Distinct from on-device edge: the CDN edge is server infrastructure, just placed near users instead of centralized.

How does network bandwidth affect the choice?

For text-only LLM workloads, network bandwidth is rarely the bottleneck — token streams are small. For multimodal inputs (images, video, audio), sending data to cloud inference can be the dominant cost. Edge inference handles the data locally and sends only the inference result, which is much smaller. Real-time video AI (object detection, gesture recognition) typically belongs at the edge for this reason.