Introduction
In today’s fast-paced digital landscape, intuitive interaction is the key to enhancing user experiences. VisualGestures.js redefines how we engage with machines, offering a seamless, touchless interface controlled by hand and finger movements in the air. Whether it’s improving productivity or enabling more fluid interaction, our tool brings next-level innovation to any workspace or project. This revolutionary tool is designed to meet the evolving needs of modern users and businesses alike. Imagine controlling your cursor with a simple hand movement — VisualGestures.js makes it possible. Ready to explore how this innovative tool can reshape user experiences and boost productivity? Let’s explore its limitless possibilities!

Learning Objectives

• To explore how touchless gesture interfaces can revolutionize user interactions across various industries.
• To understand how to effectively set up, run, and integrate VisualGestures.js in your local environment.
• To dive into real-world applications where gesture control significantly enhances productivity and user experience.
• To utilize advanced debugging tools for optimizing gesture-based interactions in real-time scenarios.
• To grasp the seamless compatibility and scalability of VisualGestures.js across diverse platforms and environments.

Table of contents

1. Why VisualGestures.js?
2. Getting started
3. Running the code locally
4. Comprehensive Ecosystem
5. Debugging Panel
6. Compatibility
7. Use-cases across industries
8. Conclusion
9. Maintainers
10. Citation
11. Key Takeaways

Why VisualGestures.js?

VisualGestures.js is an open-source TypeScript package that empowers users to effortlessly control the cursor, including actions such as hover, click, drag, and drop, through precise finger movements in the air.

1. Exceptional User Experience: Seamless, immersive interaction with intuitive gestures
Offers a seamless, touch-free interface with intuitive hand gestures, delivering an immersive and engaging experience ideal for gaming, virtual reality, and creative industries.
2. Novel Algorithm: Redefines gesture control with fluid and natural interactions, setting new standards
At its core, the innovative FKR algorithm precisely recognizes subtle hand movements, delivering fluid, accurate gesture control for natural, responsive interactions that elevate digital engagement.
3. Scalable and extensible: Designed to expand across multiple applications
Scalable and adaptable, suitable for various industries — from factory machinery control to automotive displays — offering flexible solutions for public kiosks and high-tech environments, enhancing enterprise interaction models.
4. Portable and Offline: Developed in TypeScript, works anywhere with full offline functionality
Developed in TypeScript, the solution is portable and reliable, offering full offline functionality, making it ideal for remote or restricted environments with consistent performance, regardless of location.
5. Smart Debugging: Advanced debugging capabilities for real-time insights and optimization.
Includes advanced debugging tools that deliver real-time performance insights, enabling quick optimization and efficient troubleshooting for seamless implementation by developers and end-users.

Getting started

1. Install our npm package

npm install @learn-hunger/visual-gestures

2. Integrate into your existing website

import { VisualGestures } from "@learn-hunger/visual-gestures/dist/";

/**
 *create instance of visual-gestures
 *which accepts optional parameters of container and the landmark to be used as pointer
 *[Default body and landmark 8 is used respectively]
 */
const vg = new VisualGestures();

// get hand landmarks from mediapipe's taskvision
// here video corresponds to 'HTMLVideoElement' with live webcam stream
const landmarks = handDetector.detectForVideo(video, performance.now());
vg.detect(landmarks.landmarks[0], performance.now());
// Virtual cursor can be seen once model loading and detection started successfully

For more information about ‘handDetector’, refer to the mediapipe handLandmarker documentation.

3. Currently offered cursor control events

// comprehensive list of all potential event types can be found within the 'EVgMouseEvents'
import { EVgMouseEvents } from "@learn-hunger/visual-gestures/dist/app/utilities/vg-constants";

// currently offered cursor control events
vgPointerMove();  // corresponds to 'onmousemove'
vgPointerEnter(); // corresponds to 'onmouseenter'
vgPointerLeave(); // corresponds to 'onmouseleave'
vgPointerDown(); // corresponds to 'onmousedown'
vgPointerUp(); // corresponds to 'onmouseup'
vgPointerClick(); // corresponds to 'onclick'
vgPointerDrag(); // corresponds to 'onmousedrag' ('onclick'+'onmousemove')
vgPointerDrop(); // corresponds to 'onmousedrop' ('onclick'+'onmousemove'+'onmouseup')

For each event, a callback on the ‘vgInstance[3.1]’ or via traditional event listeners [3.2], similar to cursor-based controls,, can be used

3.1. Instance Based Listening
Function corresponds to ‘onmousemove’ event in traditional cursor-based controls.

vg.mouseEvents.onPointerMove = () => {
  // console.log("callback pointer moved");
};

3.2. Traditional Event Based Listening

Function corresponds to ‘onmousemove’ event in traditional cursor-based controls.

import { EVgMouseEvents } from "@learn-hunger/visual-gestures/dist/app/utilities/vg-constants";
document.addEventListener(EVgMouseEvents.MOUSE_MOVE, () => {
  // console.log("callback pointer moved");
});

Similarly, ‘MOUSE_ENTER’, ‘MOUSE_LEAVE’, ‘MOUSE_DOWN’, ‘MOUSE_UP’, ‘MOUSE_CLICK’, ‘MOUSE_DRAG’, ‘MOUSE_DROP’ events can be listened to via instance based or traditional based listening.

4. Quick User Guide
Refer to the quick guide below for effective gesture usage.

Running the Code Locally
1. Clone the repository

git clone https://github.com/learn-hunger/visual-gestures.git
cd visual-gestures

2. Install the dependencies of source and examples

npm install
cd example2
npm install

3. Link the source code to example

cd ../
npm run clone

4. Run the example

cd example2
npm run dev

By default, the webpage runs using the hostname http://localhost:5173/

Comprehensive Ecosystem
Our custom-built project seamlessly integrates tools like Central Logger, Vercel auto build, GitHub release management, debugging tools, CI/CD pipelines, and automated code reviews, providing developers with the flexibility and performance needed to innovate and contribute effortlessly.

1. Google Analytics: Enables tracking and analyzing user behavior on applications, offering insights to improve user engagement and optimize performance.
2. NPM Package: A pre-built library of code that can be easily integrated into projects to extend functionality and simplify development processes.
3. CI/CD: Continuous Integration and Continuous Deployment pipelines automate the testing and deployment of code, ensuring seamless updates and rapid delivery of new features.
4. GitHub Releases: A streamlined way to distribute versions of your software, enabling easy version control and access to key updates or patches.
5. Auto Code Reviews: Automates the code review process to ensure quality, consistency, and adherence to coding standards, reducing manual oversight.
6. Vercel Application: A platform for deploying and hosting web applications with real-time performance optimization and ease of deployment, perfect for scalable projects.

Debugging panel

There are various debugging tools placed under single interface for smooth development, quick troubleshooting, and inference. It includes:

1. Performance Monitors
2. Debug UI
3. Graphical Visualization(s)
4. Custom landmark plotting (click here)

You can view in local by adding ‘#debug;’ path to host, i.e., using the URL: http://localhost:5173/#debug

Watch our promo to explore all the innovative features in action.

Compatibility
Optimized for seamless compatibility across a wide range of devices and platforms.

Use-cases across industries

Classroom Presentation Control: Enhances digital learning environments by managing presentations in real-time with precise hand gestures, making instruction more dynamic and interactive.
Hands-Free Home Entertainment: Transforms home entertainment systems by controlling your TV and media devices through gesture-based commands, eliminating the need for a traditional remote.
Seamless Corporate Presentations: Enables professionals to deliver smooth, hands-free presentations in corporate settings, allowing for efficient content navigation during important meetings.
Touchless Interaction in Healthcare: Improves hospital hygiene and efficiency by integrating gesture controls for medical equipment, enabling touchless operations in critical care environments.
Industrial Automation & Control: Streamlines industrial processes by allowing operators to control machinery and interfaces using gesture recognition, enhancing precision and safety on factory floors.
Social Media Gesture Navigation: Offers users a cutting-edge experience by enabling them to browse and interact with social media platforms hands-free, creating a futuristic and engaging user journey.
Gesture-Controlled Ecommerce Experiences: Elevates the online shopping experience by allowing customers to browse, select, and purchase products using intuitive gesture controls, providing a more interactive shopping platform.

Future Scope

As we continue to refine our system, future improvements will focus on enhancing algorithmic precision for more accurate gesture recognition in diverse environments, including low-light conditions and faster gestures, ensuring a more robust and responsive interaction experience.

Conclusion
VisualGestures.js is poised to revolutionize the way users interact with digital systems, providing a touchless, intuitive experience that elevates productivity and engagement. Its cutting-edge gesture control technology offers limitless possibilities for businesses and individuals alike. As the demand for innovative and efficient interfaces grows, VisualGestures.js is at the forefront of this transformation, ready to reshape the future of user interaction. Embrace the power of touchless control and unlock new potential today.

Maintainers
Nagendra Dharmireddi& Boddu Sri Pavan
Join our Discord community to engage with us directly, and don’t forget to follow us on LinkedIn and GitHub to stay updated on our latest projects, collaborations, and innovations!

Citation

@software{
package = {@learn-hunger/visual-gestures},
authors = {Nagendra Dharmireddi& Boddu Sri Pavan},
title = {{visual-gestures}},
year = {2024},
version = {0.0.1},
url = {https://github.com/learn-hunger/visual-gestures},
howpublished = {\url{https://www.npmjs.com/package/@learn-hunger/visual-gestures}}
}

Key Takeaways

• visualGestures.js delivers seamless, touchless control, enhancing user interaction through intuitive hand and finger gestures.
• It is designed for flexibility and scalability, making it adaptable across a wide range of industries and use cases.
• The system offers full offline functionality, ensuring uninterrupted performance in remote or restricted environments.
• Advanced debugging tools provide real-time insights, enabling quick optimization and smoother implementation.
• With easy integration and robust compatibility, VisualGestures.js is ideal for projects aiming to boost productivity and modernize interaction models.