How to Turn Smartphone into a Peripheral Keyboard and Barcode QR Scanner in Flutter

Barcode scanning technology is crucial for many enterprises, such as those in the supermarket, logistics, and warehousing industries. The data obtained from scanning barcodes is used to track inventory, manage orders, and process payments. Handheld barcode scanners are widely used as peripheral devices in these industries. However, traditional handheld laser barcode scanners can only detect one-dimensional codes, while more expensive scanners with an embedded camera are needed to detect two-dimensional codes like QR codes. Smartphones with built-in cameras can detect both one-dimensional and two-dimensional codes. Nowadays, smartphones have become ubiquitous. Everyone has at least one smartphone in their pocket. If smartphone can do the same job as a handheld barcode scanner, no additional hardware is needed. This article presents a solution that combines technologies like Flutter, Python, Bonjour, web sockets, PyAutoGUI, and Dynamsoft Barcode Reader SDK to help enterprises save costs by using smartphones instead of barcode scanners.

Supported Platforms

Android
iOS

Demo Video: Smartphone as a Peripheral Keyboard and Barcode Scanner

The demo shows how to input text from an Android phone to Windows notepad and macOS notes simultaneously.

Overview of the Utilized Technologies and Why They Are Used

Before diving into the details of the solution, let’s first take a look at the technologies used in this solution.

Flutter: Flutter is a cross-platform UI framework. It enables developers build native applications for Windows, Linux, macOS, Android, iOS and web from a single codebase. Using Flutter can save a lot of time and effort for building the client user interface.
Python: With Python, it only takes a few lines of code to implement network services and simulate keyboard input on operating systems.
Bonjour: Bonjour is a zero-configuration networking (zeroconf) protocol developed by Apple. It allows devices on the same network to discover each other. In this solution, Bonjour is used to discover the web socket server on the same network.
WebSocket: Web sockets are a computer communications protocol that allows for bidirectional communication between a client and a server. In this solution, web sockets are used to communicate between mobile devices and PCs.
PyAutoGUI: PyAutoGUI is a Python module that can be used to automate mouse movements and keyboard presses. In this solution, PyAutoGUI is used to simulate keyboard input.
Dynamsoft Barcode Reader SDK: Dynamsoft Barcode Reader SDK is a cross-platform barcode recognition SDK that can be used to recognize 1D and 2D barcodes in images. In this solution, Dynamsoft Barcode Reader SDK is used to recognize barcodes in images captured by the smartphone camera.

Building a Discoverable Web Socket Server with Bonjour in Python

Let's get started with the server-side implementation. The server-side implementation is written in Python.

Install the Required Python Modules

pip install pyautogui websockets zeroconf

Get the Local IP Address in Python

There may be multiple network interfaces on your computer. A simple way to list all of them is to use the ipconfig command on Windows or the ifconfig command on Linux and macOS. The Python subprocess.run() method can be used to execute the command and get the output. The desired IP address can be obtained from the output by using a regular expression, such as r'(\d+\.\d+\.\d+\.\d+)'.

import subprocess
import re
import sys

def get_ip_address():
    ip_address = ''
    if sys.platform == 'win32':
        result = subprocess.run(['ipconfig', '/all'], capture_output=True, text=True)   
    else:
        result = subprocess.run(['ifconfig'], capture_output=True, text=True)

    foundEthernet = False
    for line in result.stdout.split('\n'):
        if sys.platform == 'win32':
            match = re.search(r'Ethernet adapter Ethernet:', line)
            if match:
                foundEthernet = True

            if foundEthernet:
                match = re.search(r'IPv4 Address.*? : (\d+\.\d+\.\d+\.\d+)', line)
                if match:
                    ip_address = match.group(1)
                    break
        else:
            if sys.platform == "linux" or sys.platform == "linux2":
                match = re.search(r'eth0:', line)
            else:
                match = re.search(r'en0:', line)
            if match:
                foundEthernet = True

            if foundEthernet:
                match = re.search(r'(\d+\.\d+\.\d+\.\d+)', line)
                if match:
                    ip_address = match.group(1)
                    break

    return ip_address

Note: You may need to modify the regular expression used in the above code to match the network interface name on your computer.

Implement the Web Socket Server

After obtaining the local IP address, the web socket server can be implemented with asyncio and websockets.

import asyncio
import websockets
import socket
import pyautogui
import signal

connected = set()
isShutdown = False

async def handle_message(websocket, path):
    async for message in websocket:
        if isinstance(message, str):
            pass

        elif isinstance(message, bytes):
            pass            


async def server(websocket, path):
    connected.add(websocket)
    try:
        await handle_message(websocket, path)
    except:
        connected.remove(websocket)


def ctrl_c(signum, frame):
    global isShutdown
    isShutdown = True

async def main(ip_address):
    global isShutdown
    # Start the server
    s = await websockets.serve(server, ip_address, 4000)

    while not isShutdown:
        await asyncio.sleep(1)

    s.close()
    await s.wait_closed()

signal.signal(signal.SIGINT, ctrl_c)
asyncio.run(main(ip_address_str))

We use signal.signal() to handle keyboard input for interrupting the infinite loop of the server.

Simulate Keyboard Input with PyAutoGUI

Once a message is received from the mobile client, the server can simulate keyboard input with PyAutoGUI.

if message != '':
    if message == 'backspace':
        pyautogui.press('backspace')
    elif message == 'enter':
        pyautogui.press('enter')
    else:
        pyautogui.typewrite(message)

The pyautogui.press method can be used to simulate pressing a key on the keyboard. The pyautogui.typewrite method can be used to simulate typing text on the keyboard.

Integrate Bonjour into the Web Socket Server

The last step is to make the web socket server discoverable with Bonjour. The zeroconf module can be used to integrate Bonjour into the web socket server.


from zeroconf import ServiceBrowser, ServiceInfo, ServiceListener, Zeroconf

class MyListener(ServiceListener):

    def update_service(self, zc: Zeroconf, type_: str, name: str) -> None:
        print(f"Service {name} updated")

    def remove_service(self, zc: Zeroconf, type_: str, name: str) -> None:
        print(f"Service {name} removed")

    def add_service(self, zc: Zeroconf, type_: str, name: str) -> None:
        info = zc.get_service_info(type_, name)
        print(f"Service {name} added, service info: {info}")

ip_address_str = get_ip_address()
ip_address = socket.inet_aton(ip_address_str)

info = ServiceInfo("_bonsoirdemo._tcp.local.", socket.gethostname().split('.')[0] +
                " Web Socket Server._bonsoirdemo._tcp.local.",
                port=7000, addresses=[ip_address])

zeroconf = Zeroconf()
# Register the Bonjour service
zeroconf.register_service(info)
listener = MyListener()
browser = ServiceBrowser(zeroconf, "_bonsoirdemo._tcp.local.", listener)

# Start the web socket server
signal.signal(signal.SIGINT, ctrl_c)
asyncio.run(main(ip_address_str))

# Unregister the Bonjour service
print("Unregistering Bonjour service")
zeroconf.unregister_service(info)
zeroconf.close()

The socket.gethostname() may return a host name that contains a . inside. The . must be filtered out, otherwise the Bonjour service name will be invalid.

Building a Mobile App with Flutter, Bonjour, WebSocket and Dynamsoft Barcode Reader SDK

In this section, we will build a mobile app with Flutter, Bonjour, WebSocket and Dynamsoft Barcode Reader SDK. The mobile app can automatically discover the web socket server and send barcode data via the web socket connection.

Install the Required Flutter Packages

flutter pub add bonsoir flutter_riverpod device_info web_socket_channel dynamsoft_capture_vision_flutter

For Android, change the minimum SDK version to 21 in android/app/build.gradle:

  defaultConfig {
      minSdkVersion 21
  }

For iOS, add the following description to ios/Runner/Info.plist:

  <key>NSCameraUsageDescription</key>
  <string>Can I use the camera please?</string>
  <key>NSLocalNetworkUsageDescription</key>
  <string>Looking for local tcp Bonjour service</string>
  <key>NSBonjourServices</key>
  <array>
    <string>_bonsoirdemo._tcp</string>

The value of NSBonjourServices must be the same as the service type of your Bonjour service.

Implement Web Socket Connection and Bonjour Service Discovery in Flutter

We use Bonjour service discovery because we want to avoid manually entering the IP address of the web socket server.

The example code of bonsoir is helpful for implementing the Bonjour service discovery. We can copy app_service.dart, discovery.dart, and service_list.dart files from the example code to our project, and then modify them to meet our needs.

app_service.dart: Change the service type to _bonsoirdemo._tcp.


  static const String type = '_bonsoirdemo._tcp';

discovery.dart: No change is needed.
service_list.dart: Add an elevated button to trigger the web socket connection.

  class ItemWidgetState extends State<ItemWidget> {
    IOWebSocketChannel? _channel;
    String _connectAction = 'Connect';
    bool _connected = false;

    void _connect(String msg) {
      if (_connected) {
        print('disconnect to $msg');
        _channel!.sink.close(status.goingAway);
        channels.remove(_channel!);
        _connected = false;
        _connectAction = 'Connect';
        setState(() {});
        return;
      }

      _channel = IOWebSocketChannel.connect('ws://$msg');

      _channel!.ready.then((_) {
        channels.add(_channel!);
        print('connected to $msg');
        _connected = true;
        _connectAction = 'Disconnect';

        setState(() {});
        _channel!.stream.listen((message) {
          print('received: $message');
        }, onError: (error) {
          print('error: $error');
        }, onDone: () {
          if (_channel!.closeCode != null) {
            print('WebSocket is closed with code: ${_channel!.closeCode}');
          } else {
            print('WebSocket is closed');
          }
          _connected = false;
          _connectAction = 'Connect';
          setState(() {});
        });
      });
    }

    @override
    void dispose() {
      if (_channel != null) {
        _channel!.sink.close(status.goingAway);
        channels.remove(_channel!);
      }
      super.dispose();
    }

    @override
    Widget build(BuildContext context) {
      return Padding(
        padding: const EdgeInsets.all(8.0),
        child: Row(
          children: [
            Expanded(
              child: ListTile(
                title: Text(widget.service.name),
                subtitle: Text('IP : ${widget.service.ip}, Port: 4000'),
              ),
            ),
            ElevatedButton(
              onPressed: () {
                _connect('${widget.service.ip}:4000');
              },
              child: Text(_connectAction),
            ),
          ],
        ),
      );
    }
  }

Implement Barcode QR Code Scanning in Flutter

The dynamsoft_capture_vision_flutter package also contains excellent example code, that guides you how to use the API of the Dynamsoft Barcode Reader SDK in Flutter.

Here are the basic steps to implement barcode QR code scanning in Flutter:

Initialize the SDK with a license key.

late final DCVCameraEnhancer _cameraEnhancer;
late final DCVBarcodeReader _barcodeReader;

final DCVCameraView _cameraView = DCVCameraView();

@override
void initState() {
  super.initState();
  WidgetsBinding.instance.addObserver(this);
  _sdkInit();
}

Future<void> _sdkInit() async {
  try {
    await DCVBarcodeReader.initLicense(
        'DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ==');
  } catch (e) {
    print(e);
  }
  _barcodeReader = await DCVBarcodeReader.createInstance();
  _cameraEnhancer = await DCVCameraEnhancer.createInstance();

  DBRRuntimeSettings currentSettings =
      await _barcodeReader.getRuntimeSettings();
  currentSettings.barcodeFormatIds =
      EnumBarcodeFormat.BF_ONED | EnumBarcodeFormat.BF_QR_CODE;

  currentSettings.expectedBarcodeCount = 0;
  await _barcodeReader
      .updateRuntimeSettingsFromTemplate(EnumDBRPresetTemplate.DEFAULT);
  await _barcodeReader.updateRuntimeSettings(currentSettings);

  _cameraEnhancer.setScanRegion(Region(
      regionTop: 30,
      regionLeft: 15,
      regionBottom: 70,
      regionRight: 85,
      regionMeasuredByPercentage: 1));

  _cameraView.overlayVisible = true;

  _cameraView.torchButton = TorchButton(
    visible: true,
  );

  await _barcodeReader.enableResultVerification(true);

  _barcodeReader.receiveResultStream().listen((List<BarcodeResult>? res) {
    if (mounted) {
      decodeRes = res ?? [];

      setState(() {});
    }
  });

  await _cameraEnhancer.open();

  _barcodeReader.startScanning();
}

Add the camera view to the widget tree.

Widget listItem(BuildContext context, int index) {
  BarcodeResult res = decodeRes[index];

  return ListTileTheme(
      textColor: Colors.white,
      // tileColor: Colors.green,
      child: ListTile(
        title: Text(res.barcodeFormatString),
        subtitle: Text(res.barcodeText),
      ));
}

@override
Widget build(BuildContext context) {
  return Scaffold(
      appBar: AppBar(
        title: const Text('Barcode Scanner'),
      ),
      body: Stack(
        children: [
          Container(
            child: _cameraView,
          ),
          SizedBox(
            height: 100,
            child: ListView.builder(
              itemBuilder: listItem,
              itemCount: decodeRes.length,
            ),
          ),
        ],
      ));
}

Send the barcode result to the server.

Set<IOWebSocketChannel> channels = {};

void sendMessage(String msg) {
  if (channels.isEmpty) {
    return;
  }

  for (final channel in channels) {
    channel.sink.add(msg);
  }
}

_barcodeReader.receiveResultStream().listen((List<BarcodeResult>? res) {
  if (mounted) {
    decodeRes = res ?? [];
    String msg = '';
    for (var i = 0; i < decodeRes.length; i++) {
      msg += '${decodeRes[i].barcodeText}\n';
    }
    if (msg.isNotEmpty && _ready) {
      _ready = false;
      sendMessage(msg);

      Future.delayed(const Duration(seconds: 2), () async {
        _ready = true;
      });
    }

    setState(() {});
  }
});