Find hand points using Hand Gesture Recognition feature by Huawei ML Kit in Android (Kotlin)

Introduction

In this article, we can learn how to find the hand key points using Huawei ML Kit of Hand Gesture Recognition feature. This service provides two capabilities: hand keypoint detection and hand gesture recognition. The hand keypoint detection capability can detect 21 hand keypoints (including fingertips, knuckles, and wrists) and return the positions of the key points. The hand gesture recognition capability can detect and return the positions of all rectangle areas of the hand from images and videos, and the type and confidence of a gesture. This capability can recognize 14 gestures, including the thumbs-up/down, OK sign, fist, finger heart, and number gestures from 1 to 9. Both capabilities support detection from static images and real-time camera streams.

Use Cases

Hand keypoint detection is widely used in daily life. For example, after integrating this capability, users can convert the detected hand keypoints into a 2D model, and synchronize the model to the character model, to produce a vivid 2D animation. In addition, when shooting a short video, special effects can be generated based on dynamic hand trajectories. This allows users to play finger games, thereby making the video shooting process more creative and interactive. Hand gesture recognition enables your app to call various commands by recognizing users’ gestures. Users can control their smart home appliances without touching them. In this way, this capability makes the human-machine interaction more efficient.

Requirements

1. Any operating system (MacOS, Linux and Windows).

2. Must have a Huawei phone with HMS 4.0.0.300 or later.

3. Must have a laptop or desktop with Android Studio, Jdk 1.8, SDK platform 26 and Gradle 4.6 and above installed.

4. Minimum API Level 21 is required.

5. Required EMUI 9.0.0 and later version devices.

How to integrate HMS Dependencies

1. First register as Huawei developer and complete identity verification in Huawei developers website, refer to register a Huawei ID.

2. Create a project in android studio, refer Creating an Android Studio Project.

3. Generate a SHA-256 certificate fingerprint.

4. To generate SHA-256 certificate fingerprint. On right-upper corner of android project click Gradle, choose Project Name > Tasks > android, and then click signingReport, as follows.

Note: Project Name depends on the user created name.

5. Create an App in AppGallery Connect.

6. Download the agconnect-services.json file from App information, copy and paste in android Project under app directory, as follows.

7. Enter SHA-256 certificate fingerprint and click Save button, as follows.

Note: Above steps from Step 1 to 7 is common for all Huawei Kits.

8. Click Manage APIs tab and enable ML Kit.

9. Add the below maven URL in build.gradle(Project) file under the repositories of buildscript, dependencies and allprojects, refer Add Configuration.

10. Add the below plugin and dependencies in build.gradle(Module) file.

11. Now Sync the gradle.

12. Add the required permission to the AndroidManifest.xml file.

Let us move to development

I have created a project on Android studio with empty activity let us start coding.

In the MainActivity.kt we can find the business logic for buttons.

In the LiveHandKeyPointAnalyse.kt we can find the business logic for live analysis.

Create LensEnginePreview.kt class to find the business logic for lens engine view.

Create HandKeypointGraphic.kt class to find the business logic for hand key point.

Create GraphicOverlay.kt class to find the business logic for graphic overlay.

In the StaticHandKeyPointAnalyse.kt we can find the business logic static hand key point analyses.

In the activity_main.xml we can create the UI screen.

In the activity_live_hand_key_point_analyse.xml we can create the UI screen.

In the activity_static_hand_key_point_analyse.xml we can create the UI screen.

Demo

Tips and Tricks

1. Make sure you are already registered as Huawei developer.

2. Set minSDK version to 21 or later, otherwise you will get AndriodManifest merge issue.

3. Make sure you have added the agconnect-services.json file to app folder.

4. Make sure you have added SHA-256 fingerprint without fail.

5. Make sure all the dependencies are added properly.

Conclusion

In this article, we have learned how to find the hand key points using Huawei ML Kit of Hand Gesture Recognition feature. This service provides two capabilities: hand keypoint detection and hand gesture recognition. The hand keypoint detection capability can detect 21 hand keypoints (including fingertips, knuckles, and wrists) and return the positions of the key points. The hand gesture recognition capability can detect and return the positions of all rectangle areas of the hand from images and videos, and the type and confidence of a gesture. This capability can recognize 14 gestures, including the thumbs-up/down, OK sign, fist, finger heart, and number gestures from 1 to 9. Both capabilities support detection from static images and real-time camera streams.

I hope you have read this article. If you found it is helpful, please provide likes and comments.

Reference

ML Kit — Hand Gesture Recognition

ML Kit — Training Video