Machine learning on Android with ML Kit
ML Kit gives Android developers a practical way to add barcode scanning, image recognition, object detection, and text recognition without building a complete inference stack. Using machine learning models with ML Kit, predictions can run on the phone, so a captured image does not need to travel to a remote server. That reduces latency, keeps sensitive images on the device, and allows core features to work when reception is patchy.
ML Kit is an inference tool rather than a model-training environment. A project may use a ready-made ML Kit vision API or package a custom TensorFlow Lite model through ML Kit’s LiteRT inference support. Training happens separately, while the surrounding Android code determines whether the model feels useful. Permission handling, camera frames, confidence thresholds, loading states, and fallback messages all contribute to the final experience.
Choose a task before choosing a model
Start with a decision the app must make, not with a model class that happens to be available. Consider three Australian scenarios: a Sydney retail worker checking shelf stock, a document tool reading customer forms, and a community event scanner reading barcodes. These jobs suit different APIs, inputs, and acceptance tests. A task such as “recognise a product on a bench” also needs a defined answer because a broad label like “food” may be too vague for stocktaking.
For a Melbourne café, image labels could suggest whether a photographed tray contains a cup, bowl, or pastry, while a staff member confirms the final count. Such an example highlights the need for domain-specific labels and thresholds. Generic ML Kit models provide a useful starting point, but custom training data and a clear user decision are usually required for dependable business results. Keep human confirmation available when an incorrect prediction would be costly.
Configure the Android project
Add only the ML Kit component needed by the app, then pin the current stable version in the app module’s Gradle build file. For example, an image-labelling app can start with:
dependencies {
implementation("com.google.mlkit:image-labeling:17.0.1")
}
Replace the version with the current stable release for your project and check the component’s release notes before upgrading. Model APIs, minimum Android versions, and artefact names can change. For an Australian Google Play release, check the final bundle size and test the same build on the lower-cost and older phones in your audience.
Create the detector once, normally in a ViewModel or another lifecycle-aware component, and close it when its owner is destroyed. Configure the confidence threshold to filter weak results:
val options = ImageLabelingOptions.Builder()
.setConfidenceThreshold(0.65f)
.build()
val labeler = ImageLabeling.getClient(options)
Treat 0.65 as an experiment rather than a universal constant. Record predictions on real Australian shopfronts, kitchens, and homes, then adjust the value against the errors users would actually make.
Compare model delivery options
Three delivery routes cover most Android projects. The appropriate choice depends on model size, update frequency, connectivity, and whether the app needs fixed or domain-specific classes.
| Approach | How the model is obtained | Strongest use | Main trade-off |
|---|---|---|---|
| Bundled ML Kit API | Model ships inside the app | Offline barcode, text, label, or object tasks | Larger download and fixed task classes |
| Unbundled ML Kit API | Model is fetched and cached on demand | Large models that do not need to ship in the APK | Network-dependent first use and loading states |
| Custom LiteRT model | Developer supplies the model and labels | Specialist classification or detection | More validation, preprocessing, and maintenance work |
Bundled APIs give the quickest and most predictable first launch. Unbundled APIs can reduce the initial download, but users on regional NSW mobile connections may see an empty screen while a model downloads; progress and retry states are essential. A custom model provides control over recognised classes and thresholds, yet the team owns label quality, input resizing, conversion, and regression testing.
The choice also affects updates. A bundled model changes only when the app updates, whereas a downloaded model can evolve more independently. ML Kit still performs inference; it does not remove the need to train, validate, or monitor the model supplying those predictions.
Connect camera frames or stored media
CameraX’s ImageAnalysis use case is a common way to supply frames to ML Kit. It provides a repeatable stream that works well with a camera preview, permission flow, and analysis lifecycle. Convert each frame to an InputImage, pass it to the detector, and close every ImageProxy exactly once.
fun analyze(imageProxy: ImageProxy) {
val mediaImage = imageProxy.image
if (mediaImage == null) {
imageProxy.close()
return
}
val input = InputImage.fromMediaImage(
mediaImage,
imageProxy.imageInfo.rotationDegrees
)
labeler.process(input)
.addOnSuccessListener { labels -> renderLabels(labels) }
.addOnFailureListener { error -> showModelError(error) }
.addOnCompleteListener { imageProxy.close() }
}
Real-time analysis needs back-pressure because inference can outpace the camera. Use ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST when only the newest frame matters, and keep heavy rendering away from the main thread. For a still image, call the image-labelling API directly. A scanned receipt may need text recognition as its first stage, with barcode scanning added only if the workflow requires it.
Declare android.permission.CAMERA, request it at runtime, and design a useful denial state. A friendly Australian message such as “No worries—scan it again” can accompany a retry action while explaining why camera access is needed. For stored photos, check orientation and avoid repeatedly decoding full-resolution images on the UI thread.
Plan, present, and validate predictions
Show the prediction in the form of the user’s task. If the goal is stock counting, display the recognised object, confidence, and an action to correct it; a raw API label is rarely enough. Recognised results should be visually distinct from suggestions, and a low-confidence result must not appear as a firm answer.
For an app with scanning, history, and settings, the bottom navigation patterns described here can keep machine-learning tools separate from ordinary navigation on compact phones. This helps users check a quick scan result while standing at a crowded counter.
Model selection checklist
- Define the exact decision and acceptable error cost
- Test representative lighting, angles, objects, and languages
- Measure inference time and memory on lower-cost Android devices
- Set a confidence threshold from collected test results
Release checklist
- Handle camera denial, model download failure, and no-result states
- Verify orientation and permissions on physical devices
- Add model-version logging for regression investigation
- Review stored-image privacy and deletion behaviour
A release is ready when the prediction, its confidence, and the next available action all make sense to the person using it, even when the model returns nothing useful.