Ship On Device Machine Learning in 90 Days for Developers

Home Ship On Device Machine Learning in 90 Days for Developers
Engineer testing an anonymous edge computing board
Artificial Intelligence

Ship On Device Machine Learning in 90 Days for Developers

By, Amy S
  • 5 Sep, 2026
  • 1 Views
  • 0 Comment

On-device machine learning means running a model directly on a phone, wearable, or embedded chip instead of sending data to a server for inference. It cuts latency to milliseconds, works without a network connection, and keeps sensitive data off third-party servers. The trade-off is real: you’re bound by the device’s compute, memory, and battery budget, so it fits mobile apps, privacy-sensitive tools, and latency-critical IoT far better than heavy generative workloads.

TL;DR:

  • On-device inference is feasible on many smartphones and microcontrollers with quantized models, but on-device training remains limited to flagship devices with ample RAM.
  • Privacy, latency, and offline operation are primary drivers for on-device ML, with energy limits forcing trades between model size, accuracy, and inference frequency.
  • Optimization techniques like quantization, pruning, knowledge distillation, and parameter-efficient fine-tuning must be applied sequentially and validated carefully to avoid accuracy regressions.
  • Runtimes such as ExecuTorch, LiteRT, and LocalAI support different hardware and framework needs, with hardware-aware profiling essential to ensure models run efficiently on target devices.
  • The future trend points toward smaller, dedicated generative AI models optimized for edge hardware, with convergence in deployment pipelines and increased adoption of federated and continual learning methods.

Table of Contents

What On-Device ML Actually Covers

Most people use “on-device ML” as a catch-all, but it splits into two genuinely different problems. Edge inference means a model was trained elsewhere and now runs locally to make predictions. On-device learning means the model itself updates on the device, whether through fine-tuning, continual learning, or federated updates. A survey framing on-device learning as resource-constrained learning makes the distinction explicit: compute and memory ceilings dictate which algorithms are even mathematically viable, not just which ones are convenient.

Inference-only deployment is the default for good reason. It’s simpler, more predictable, and works on almost any modern phone or single-board computer. On-device training is a different animal entirely, since gradient computation and optimizer state consume memory that inference never touches. A taxonomy of edge inference versus edge learning treats these as separate technical categories requiring different tooling, and that separation matters when you’re scoping a project.

Here’s where each pattern actually shows up in production:

  • Mobile assistants and keyboard prediction run small language models locally for next-word suggestions and voice commands without a round trip to a server.
  • On-phone computer vision powers real-time object detection, document scanning, and photo classification, often using architectures purpose-built for lightweight edge object detection.
  • IoT anomaly detection runs compact models on sensors or gateways to flag equipment faults before a network call would even complete.
  • Wearables use micro-models for step counting, heart rate anomaly flags, and fall detection, where battery life outranks model sophistication every time.
  • Local large language models are the newest addition, letting privacy-conscious apps run chat or summarization features entirely offline.

A rough feasibility check before you commit to an approach: inference is realistic on many smartphones and on various microcontrollers running quantized models of modest size. On-device fine-tuning is realistic mainly on flagship phones and tablets with several gigabytes of RAM to spare, and full from-scratch training almost never makes sense outside of research prototypes.

Why Choose On-Device: Privacy, Latency, Bandwidth, and Energy

Four forces push teams toward local inference, and each one shapes architecture decisions differently.

Privacy by design tops the list for a reason. When a model runs locally, raw data (a photo, a voice clip, a health metric) never leaves the device. Engineers at Qualcomm have pointed to this as a driving force behind running generative AI at the edge, since it avoids shipping sensitive user data to third-party clouds entirely. For health apps, financial tools, or anything under strict data residency rules, this isn’t a nice bonus, it’s frequently the deciding factor.

Latency is the second driver, and it’s the one users actually feel. A cloud round trip for a camera app or AR feature typically costs 100 to 300 milliseconds just in network time, before the model even runs. Local inference collapses that to single-digit milliseconds, which is the difference between an AR filter that tracks a face smoothly and one that visibly lags a frame behind.

Bandwidth and offline capability matter most outside dense urban coverage. A logistics app scanning barcodes in a warehouse basement, or a construction site tool logging equipment faults, can’t depend on a live connection. Local models keep working when the signal doesn’t.

Energy and thermal limits are the constraint that pushes back hardest against the other three. A phone’s battery and thermal envelope cap how large and how frequently a model can run before it visibly drains the battery or throttles the chip. This is why product teams routinely trade a few points of accuracy for a model that’s a third the size: a slightly less accurate model that runs constantly beats a sharper one that kills battery life in an afternoon.

Core Optimization Techniques for Shrinking Models

Four techniques do almost all the heavy lifting in on-device optimization, and the order you apply them in matters as much as the techniques themselves.

  1. Quantization reduces numeric precision, typically from 32-bit floats to 8-bit integers, shrinking model size roughly fourfold and speeding up inference on hardware with integer math support. Post-training quantization is fast to apply and usually costs a small accuracy drop, often under one or two percentage points on well-behaved models. Quantization-aware training bakes the precision loss into the training loop itself, which recovers most of that gap at the cost of a longer training cycle.
  2. Pruning removes weights or entire structures (channels, layers, attention heads) that contribute little to output quality. Unstructured pruning zeroes out individual weights and can hit high sparsity ratios, but it needs specialized sparse-matrix runtimes to actually speed things up. Structured pruning removes whole channels or filters, which is less aggressive on sparsity but yields real speedups on standard hardware without special kernels.
  3. Knowledge distillation trains a small “student” model to mimic a larger “teacher” model’s outputs, often achieving much of the teacher’s accuracy in a fraction of the parameter count. It’s particularly effective when you have a strong cloud model already in production and need a lightweight version for the device.
  4. Parameter-efficient fine-tuning, using methods like LoRA or lightweight adapters, lets you personalize a base model for an individual user without retraining the whole network. Only a small adapter layer updates, which keeps memory and compute costs low enough for on-device personalization to actually be feasible.

The real skill isn’t picking one technique, it’s sequencing and validating them together. Academic guidance on this front is consistent: start with post-training quantization, measure the actual impact, then layer in pruning or distillation only if the accuracy budget still allows it. Track every step’s effect on accuracy, latency, and model size in your continuous integration pipeline, the same way you’d track unit test coverage, borrowing practices from a broader AI testing framework for applications.

Pro Tip: Never combine quantization, pruning, and distillation in one giant experiment. Apply and measure one technique at a time, because stacking them blind makes it impossible to tell which change actually caused a regression when accuracy drops.

Which Runtimes and Toolchains Handle On-Device Deployment?

Every production on-device pipeline needs a runtime that handles three jobs: converting the model into a device-friendly format, planning memory ahead of time, and delegating operations to the right hardware (CPU, GPU, or NPU). The tool you pick depends heavily on which training framework you already use.

ExecuTorch and PyTorch AOT give PyTorch users a native export path. ExecuTorch documents a full export, compile, and runtime flow that produces a compact .pte file, plans memory ahead of time, and partitions operations across hardware backends from microcontrollers up to full smartphones. The torch.compiler AOT compilation tools sit underneath this, pre-optimizing the computation graph for the target architecture before a single inference runs, which cuts runtime overhead compared to interpreting the graph on the fly.

LiteRT is Google’s evolution of TensorFlow Lite (TFLite), rebuilt to be a high-performance runtime for on-device ML with dedicated tooling for generative AI workloads through LiteRT-LM. It supports efficient model conversion and hardware-aware optimization across phones, wearables, and even browser-based deployments, which makes it the default choice for teams already inside the TensorFlow ecosystem.

LocalAI and Foundry Local solve a different problem: local serving with a familiar API surface. LocalAI offers a small, composable core with OpenAI-compatible endpoints, so developers can prototype against a familiar API and then ship the same interface in-app. Foundry Local extends this with automatic hardware acceleration and a curated model catalog, handling model acquisition and falling back to smaller variants automatically depending on the device it lands on.

Choosing between them comes down to three questions:

  • Which training framework did the model come from, PyTorch or TensorFlow?
  • How much binary footprint can the target app tolerate, since a full runtime can add several megabytes?
  • Does the target hardware have NPU or GPU delegate support the runtime can actually use?

Hardware Patterns: CPUs, GPUs, NPUs, and Memory Budgeting

Device hardware falls into four rough classes, and each one caps what’s realistic. Microcontrollers (MCUs) offer kilobytes to a few megabytes of RAM and no floating-point acceleration, so only heavily quantized models under a megabyte or two make sense. Mobile system-on-chips (SoCs) bring gigabytes of RAM and dedicated ML accelerators, opening the door to real-time vision and mid-sized language models. Embedded GPUs handle parallel workloads like image processing well but draw more power than a dedicated accelerator. Neural processing units (NPUs) are purpose-built for matrix math at low power, and they’re increasingly standard in flagship phones and even some laptops.

Getting a model onto an NPU isn’t automatic. AOT compilation partitions the computation graph so operators the NPU supports run there, while unsupported operators fall back to the CPU or GPU. This partitioning step is exactly what runtimes like ExecuTorch handle, and it’s why AOT compilation is treated as a distinct workflow rather than an afterthought in production pipelines.

Memory planning is the other half of hardware-aware deployment. A model’s peak memory usage during inference, not just its file size on disk, determines whether it fits on a target device without crashing other apps. Runtimes that plan memory ahead of time can reuse buffers across layers, which often cuts peak memory usage significantly compared to naive execution.

Before optimizing anything, gather these profiling numbers on real hardware:

  • Peak memory usage during a full inference pass, not just model file size.
  • CPU, GPU, or NPU utilization per layer, to find the actual bottleneck.
  • Sustained latency over a multi-minute run, not just a single cold-start measurement.
  • Power draw per inference, ideally measured with a hardware power monitor rather than estimated.

Pro Tip: Run your latency benchmark for at least five minutes continuously, not five seconds. Thermal throttling on phones often doesn’t show up until the chip has been under sustained load, and a single fast measurement will hide it completely.

The Deployment Pipeline: Train, Optimize, Compile, Deploy

A reliable on-device pipeline has four checkpoints, and skipping any one of them is where most teams get burned in production.

  1. Train in the cloud with device constraints defined upfront. Decide your target memory and latency budget before training starts, not after. A model architecture chosen without hardware constraints in mind almost always needs a costly redesign later.
  2. Run optimization steps inside CI, not as a manual one-off. Quantization, pruning, and distillation should be automated pipeline stages that log accuracy, size, and latency deltas on every model version, mirroring how a mature testing framework for AI-driven applications tracks regressions over time.
  3. Export and compile ahead of time, then test on the actual target hardware. A model that benchmarks well on a development laptop can behave very differently on a mid-range phone with a smaller cache and slower memory bus. Test on the cheapest device in your supported range, not the newest one.
  4. Validate offline behavior, rollback paths, and over-the-air updates. Confirm the app degrades gracefully with no connectivity, and that a bad model push can be rolled back remotely without forcing a full app update through app store review.

Treat each checkpoint as a gate, not a suggestion. A model that fails the hardware test in step three has no business reaching step four.

How Do You Benchmark On-Device Models Properly?

Six metrics matter for on-device benchmarking, and most teams only track one or two of them, usually accuracy and average latency, which hides the problems that actually cause bad reviews.

  • Latency at the p95 and p99 percentile, not just the average, since tail latency is what users notice as stuttering.
  • Throughput, or inferences per second, particularly for streaming use cases like continuous video analysis.
  • Peak memory usage, measured during actual inference rather than estimated from model file size.
  • Energy per inference, measured with a power monitor where possible, since this determines real-world battery impact.
  • Model size on disk, which affects app download size and update friction.
  • Accuracy, measured against a held-out set that reflects the real distribution of on-device inputs, not the original training distribution.

Benchmarking rigor is where most teams cut corners. Measuring only a cold start hides the steady-state behavior users actually experience, and testing on a single high-end device hides how the model performs on the mid-range hardware most of your users actually own. Research on edge ML methodology has flagged this exact gap: repeated short benchmark runs frequently miss thermal throttling and long-run degradation that only shows up under sustained real-world use.

Interpreting the results means picking the right trade-off for your product, not chasing the smallest possible model. A camera app that runs detection once per shutter press can tolerate a heavier, more accurate model than one that runs detection thirty times per second on a live viewfinder. Set your acceptable accuracy floor first, then optimize size and speed within that floor, not the other way around.

A Practical Starter Checklist for Your First On-Device Project

Before writing a line of model code, nail down five decisions that will otherwise cost you a rebuild later.

  1. Define the product impact and hardware target first. Know the cheapest device in your supported range and its RAM, CPU, and NPU capabilities before choosing a model architecture.
  2. Pick a baseline model and set an accuracy budget you can defend. Decide how much accuracy you’re willing to trade for size and speed, in writing, before you start optimizing.
  3. Select the runtime and toolchain that match your training framework. PyTorch teams typically lean toward ExecuTorch; TensorFlow teams typically lean toward LiteRT.
  4. Build benchmarks and CI checks around every optimization step. Automate accuracy, latency, and size tracking so no change ships without visibility into its trade-offs.
  5. Plan over-the-air updates and monitoring before launch, not after. Assign clear ownership: an ML engineer handles model quality, a mobile engineer handles integration, and an SRE or DevOps engineer owns the OTA and monitoring pipeline.

Pro Tip: Assign one person to own the rollback plan before the first model ships, not after the first bad update goes out. Knowing exactly how a bad model gets pulled from production is the single most-skipped step in on-device projects.

What a 90-Day Mobile Computer Vision Pilot Actually Looks Like

Digital Fractal’s 90-day mobile computer vision playbook breaks a pilot into three phases: a two-week feasibility and hardware audit, a five-week model build and optimization cycle, and a final phase focused on field testing and rollback validation before wider rollout.

The lessons that repeat across pilots like this one:

  • Instrumentation has to ship with the model, not after it. Teams that added logging for latency and confidence scores from day one caught dataset drift weeks before it affected accuracy metrics.
  • A rollback plan needs to exist before launch day, not get improvised during an incident. Pilots that defined a clear “revert to previous model version” path recovered from bad updates in hours instead of days.
  • Dataset drift shows up fastest in edge cases the original training data underrepresented, which is why field data collection during the pilot matters more than expanding the initial training set.
  • Hardware variance across mid-range devices causes more real-world failures than model accuracy does. Testing exclusively on flagship phones during development is the single most common blind spot.

How Do You Manage Data and Incremental Learning on Device?

On-device data management usually means deciding what to keep, what to discard, and what never leaves the device at all. Most production systems store only aggregated statistics or model updates locally, discarding raw sensor data or images after a short retention window, both to save storage and to minimize privacy exposure.

Incremental learning, sometimes called continual learning, lets a model adapt to a user’s patterns over time without a full retrain. A keyboard app learning a user’s vocabulary or a wearable adjusting its activity baseline are both light forms of this. The catch is catastrophic forgetting, where a model adapting to new data quietly loses accuracy on the patterns it learned originally. Parameter-efficient fine-tuning methods like LoRA adapters help here, since only a small adapter layer updates while the base model’s original weights stay frozen and protected.

Federated learning extends this idea across a device fleet: instead of raw data ever leaving individual devices, only model updates get aggregated centrally. This lets a company improve a shared model without ever centralizing individual users’ data, which is a meaningfully different privacy posture than uploading raw logs to a server. It comes at a real engineering cost though, since coordinating updates across thousands of heterogeneous devices with intermittent connectivity is a distributed systems problem as much as a machine learning one.

Federated learning updates across device fleet

Power Management Strategies for On-Device ML

Battery drain is the fastest way an on-device feature gets uninstalled, so power management deserves the same design attention as accuracy. The most direct lever is inference frequency: running a model continuously at thirty frames per second costs dramatically more energy than running it on a triggered basis, say once per second or only when a motion sensor flags activity worth analyzing.

Hardware delegation matters just as much as model size. Running inference on a dedicated NPU instead of the general-purpose CPU can cut energy per inference substantially, since NPUs are purpose-built for the matrix multiplications that dominate neural network workloads. A model that runs fine on a CPU during development can behave completely differently once delegated properly to an NPU in production.

Batching and duty-cycling help too. Rather than processing each frame the instant it arrives, some applications buffer several inputs and process them together, then let the chip idle between batches. This trades a small amount of latency for a real reduction in average power draw, which is usually a good trade for background tasks that don’t need instant results.

Thermal throttling is the failure mode that catches teams off guard. A chip running a model too aggressively for too long will throttle its own clock speed to avoid overheating, silently degrading latency in a way that a five-second benchmark will never catch. Building in a lower-power fallback mode, one that shifts to a smaller model or reduced inference frequency when the device reports thermal pressure, keeps the user experience stable instead of letting it silently degrade.

What Are the Real Limitations of On-Device Machine Learning?

The honest limitations rarely show up in a marketing deck. Model capacity is the first wall: a device with a few gigabytes of RAM simply cannot run the same scale of model a cloud server can, so on-device systems accept a real accuracy ceiling compared to their cloud counterparts, especially for tasks needing broad world knowledge or complex reasoning.

Fragmentation is the second wall, and it’s the one that eats the most engineering time. The Android and iOS device landscape spans years of chip generations, memory configurations, and NPU capabilities, which means a model tuned for one flagship phone may underperform badly on a three-year-old mid-range device still running the same app. Testing across that spread is expensive and easy to underinvest in.

Update logistics compound both problems. Pushing a new model version to millions of devices requires careful versioning, bandwidth-conscious download sizes, and a fallback path for devices that fail to update. Unlike a cloud model update, which rolls out instantly to every user, an on-device model update depends on users actually installing it, which can leave a meaningful share of your user base on a stale, potentially buggy model for months.

Debugging is harder too. A cloud model that misbehaves can be inspected with full logs and reproduced on demand. A model misbehaving on a user’s device, three model versions behind, on hardware you don’t have in your test lab, is a genuinely difficult problem to diagnose from a bug report alone.

Beyond Privacy: Model Tampering and Adversarial Risk

Shipping a model onto a device you don’t control introduces security risks that a cloud-hosted model never faces, because the model file itself becomes something an attacker can extract, inspect, and modify.

Model extraction and reverse engineering is the first concern. Once a model ships inside an app package, a sufficiently motivated attacker can extract the weights and either steal the intellectual property or study the architecture for weaknesses. Obfuscating the model file and encrypting it at rest on the device, decrypting only into protected memory at load time, raises the bar considerably without eliminating the risk entirely.

Model tampering is the second concern. If an attacker can modify the model file on a rooted or jailbroken device, they can alter its behavior, for example, causing a fraud detection model to systematically miss certain patterns. Checksums and signature verification at load time catch a tampered file before it ever runs.

Adversarial inputs are the third, and the least intuitive. Carefully crafted inputs, sometimes imperceptibly different from a normal input to a human eye, can cause a model to misclassify with high confidence. A vision model on a security camera or a fraud detector processing transaction images both face this risk in production, and defending against it usually means adversarial training on top of the standard training pipeline, plus input validation that flags statistically unusual inputs before they reach the model.

None of these risks disappear by moving to the cloud, but they’re distinct enough from the classic privacy conversation that teams often plan for one and miss the other entirely.

Where Is On-Device ML Heading Next?

The clearest trend right now is the move from inference-only deployment toward genuine on-device generative AI. LiteRT’s dedicated LM tooling and the broader push toward running generative models at the edge both point the same direction: smaller, more efficient language models designed from the start for phone and wearable hardware, not just shrunk-down versions of server models.

Hardware is catching up fast too. NPUs are becoming standard even in mid-range phones, and dedicated ML silicon is starting to appear in categories that never had it before, laptops and even some IoT gateways. This closes the performance gap between flagship and mid-tier devices faster than software optimization alone ever could.

Toolchains are consolidating around the export, compile, and runtime pattern that ExecuTorch and LiteRT both represent, rather than the fragmented, framework-specific approaches common a few years ago. Expect more convergence here, not less, as teams increasingly want a single pipeline that targets multiple hardware backends from one model definition. Federated and continual learning techniques are also maturing past research prototypes into production use, particularly for personalization features that need to adapt without ever centralizing user data.

When Should You Actually Choose On-Device Over the Cloud?

On-device wins clearly when privacy, latency, or offline reliability are non-negotiable for the product. If a feature needs to work in a basement with no signal, or if sending user data anywhere would break trust or compliance, the decision is easy.

Cloud or hybrid approaches still win for anything needing heavy compute or frequent retraining on fresh data across a whole user base. Trying to force a large, constantly updated model onto a device is usually a losing fight against the hardware.

The smartest path for most teams isn’t picking one side forever. It’s staging a small pilot on a narrow, well-defined feature, measuring the real trade-offs on actual target hardware, and scaling from there. If you’re unsure whether to invest in on-device tooling now, the cloud versus edge trade-off usually resolves itself once you profile your specific latency and privacy requirements rather than debating it in the abstract.

— Souhail

How Digital Fractal Helps You Ship On-Device ML Faster

Building the pipeline described above from scratch, quantization, runtime selection, hardware profiling, OTA rollback, takes most in-house teams months of trial and error before the first model reaches a real device. Digital Fractal compresses that timeline with a structured AI Readiness Audit that maps your existing hardware targets, data pipeline, and team capabilities against a realistic 90-day path to a working on-device pilot, instead of a generic consulting engagement that never gets specific about your actual constraints.

Digitalfractal

The audit identifies exactly where automation and optimization opportunities exist in your current mobile or IoT stack, then hands you a concrete plan built around your hardware, not a one-size-fits-all template. For teams that want the pilot itself run for them rather than staffed internally, the 90-day mobile computer vision pilot gives you a consultant-led path to a shipped feature without hiring a full ML team first. Start with the AI Readiness Audit to see where your hardware and data actually stand before committing engineering time to a build.

Sources

For deeper technical grounding, the algorithms and learning theory survey covers the theory behind resource-constrained learning. The edge ML taxonomy maps inference versus learning techniques in detail. For implementation, consult ExecuTorch’s documentation, LiteRT’s developer docs, and LocalAI’s overview for runtime specifics.

Tags:

7 Automation Use Cases for Logistics Teams That Deliver ROI in 90 Days

AGV moving totes through warehouse aisles