How to Build Mobile Document Scanning App with Xamarin.Forms for Android and iOS

Mobile document scanning apps bring convenience to users by allowing them to scan travel documents, ID cards, passports and other types of documents on the go. This article aims to help C# developers, who do not have the background in computer vision, to build a mobile document scanning app for both Android and iOS using Xamarin.Forms and Dynamsoft Document Normalizer SDK.

Dynamsoft.DocumentNormalizer.Xamarin.Forms

Xamarin.Forms enables developers to build cross-platform mobile apps with C# and XAML from a single codebase. Dynamsoft.DocumentNormalizer.Xamarin.Forms is a Xamarin.Forms wrapper for Dynamsoft Document Normalizer SDK. The primary features of the SDK include:

Camera view for real-time scanning
Document edge detection
Perspective correction
Parameter settings for image normalization
File saving

Online Documentation

https://www.dynamsoft.com/document-normalizer/docs/programming/xamarin/user-guide.html

Steps to Build a Mobile Document Scanning App with Xamarin.Forms

In the following paragraphs, we will use Visual Studio 2022 for Windows to build the document scanning app.

Create and Configure Xamarin.Forms Project for Android and iOS

Create an empty Xamarin.Forms project in Visual Studio 2022 and pair the project to a remote macOS.

Try to build the project for Android and iOS respectively. If you suffered from the iOS build issue Error MT4109: Failed to compile the generated registrar code caused by Xcode 14, you can visit https://github.com/xamarin/xamarin-macios/issues/15954 to find the solution. The issue can be fixed by installing xamarin.ios-16.0.0.72.pkg. To run the app on iOS 16, turn on Settings > Privacy & Security > Developer Mode.

The Xamarin.Forms version used by Dynamsoft.DocumentNormalizer.Xamarin.Forms is 5.0.0.2515, whereas the default version of Xamarin.Forms in Visual Studio 2022 is 5.0.0.2196. To install the SDK, you need to update the version of Xamarin.Forms by right-clicking the project and selecting Manage NuGet Packages.

Then install the SDK by searching Dynamsoft.DocumentNormalizer.Xamarin.Forms in the NuGet package manager.

The final step is to add the camera permission for Android and iOS.

AndroidManifest.xml

<uses-permission android:name="android.permission.CAMERA" />

Info.plist

<key>NSCameraUsageDescription</key>
<string>This app is using the camera</string>

Initialize the Document Normalizer SDK

Before getting started to write the shared code, you need to initialize the SDK in the platform-specific code. It is a little bit different between Android and iOS.

// Android MainActivity.cs
using DDNXamarin.Droid;

protected override void OnCreate(Bundle savedInstanceState)
{
    LoadApplication(new App(new DCVCameraEnhancer(this), new DCVDocumentNormalizer(), new DCVLicenseManager(this)));
}

// iOS AppDelegate.cs
using DDNXamarin.iOS;
public override bool FinishedLaunching(UIApplication app, NSDictionary options)
{
    LoadApplication(new App(new DCVCameraEnhancer(), new DCVDocumentNormalizer(), new DCVLicenseManager()));
}

In the shared code App.xaml.cs, we register the license key and create static variables to store the DCVCameraEnhancer and DCVDocumentNormalizer instances for later use. The license key can be obtained from https://www.dynamsoft.com/customer/license/trialLicense.

using DDNXamarin;

public partial class App : Application, ILicenseVerificationListener
{
    public static ICameraEnhancer dce;
    public static IDocumentNormalizer ddn;
    public static ILicenseManager licenseManager;

    public App(ICameraEnhancer enhancer, IDocumentNormalizer normalizer, ILicenseManager manager)
    {
        ...

        dce = enhancer;
        ddn = normalizer;

        licenseManager = manager;
        licenseManager.InitLicense("DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ==", this);
        ...
    }

    public void LicenseVerificationCallback(bool isSuccess, string msg)
    {

    }
}

Content Page for Camera Preview and Real-time Scanning

Create a new content page CustomRendererPage.xaml and add a DCVCameraView control to it.

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
             xmlns:dynamsoft = "clr-namespace:DDNXamarin;assembly=DDN-Xamarin"
             x:Class="DocumentScanner.CustomRendererPage">
    <ContentPage.Content>
        <AbsoluteLayout>
            <dynamsoft:DCVCameraView AbsoluteLayout.LayoutBounds="0,0,1,1" AbsoluteLayout.LayoutFlags="All" x:Name="preview">
            </dynamsoft:DCVCameraView>
            <Button x:Name="capture" Text="Capture"
                    AbsoluteLayout.LayoutBounds="0.5,0.5,120,50" AbsoluteLayout.LayoutFlags="PositionProportional"
                    Clicked="OnButtonClicked">
            </Button>
        </AbsoluteLayout>
    </ContentPage.Content>
</ContentPage>

The DCVCameraView control is a custom renderer for Xamarin.Forms. It does not only wrap the native camera view, but also provides the real-time document edge detection feature.

public partial class CustomRendererPage : ContentPage, IDetectResultListener
{
    public static ICameraEnhancer dce;
    public static IDocumentNormalizer ddn;


    public CustomRendererPage()
    {
        InitializeComponent();
        App.ddn.SetCameraEnhancer(App.dce);
        App.ddn.AddResultListener(this);
    }

    public void DetectResultCallback(int id, ImageData imageData, DetectedQuadResult[] quadResults)
    {

        if (imageData != null && quadResults != null)

        {

            Device.BeginInvokeOnMainThread(async () => {
                await Navigation.PushAsync(new QuadEditorPage(imageData, quadResults));
            });
        }
    }

    protected override void OnAppearing()
    {

        base.OnAppearing();
        App.dce.Open();
        App.ddn.StartDetecting();
    }

    protected override void OnDisappearing()
    {
        base.OnDisappearing();
        App.dce.Close();
        App.ddn.StopDetecting();

    }

    void OnButtonClicked(object sender, EventArgs e)
    {
        App.ddn.EnableReturnImageOnNextCallback();
    }
}

Android

iOS

Once we get the detected document edges via the callback function, we can launch a new page to edit the document edges.

Content Page for Editing Document Edges

Create a new content page QuadEditorPage.xaml and add a DCVImageEditorView control to it.

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
             xmlns:dynamsoft = "clr-namespace:DDNXamarin;assembly=DDN-Xamarin"
             x:Class="DocumentScanner.QuadEditorPage">
    <ContentPage.Content>
        <AbsoluteLayout>
            <dynamsoft:DCVImageEditorView AbsoluteLayout.LayoutBounds="0,0,1,1" AbsoluteLayout.LayoutFlags="All"
                x:Name="imageEditor">
            </dynamsoft:DCVImageEditorView>

            <Button
                x:Name="normalize"
                Clicked="OnNormalizeClicked"
                Text="Save"
                AbsoluteLayout.LayoutBounds="0.5,0.5,120,50"
                AbsoluteLayout.LayoutFlags="PositionProportional">
            </Button>

        </AbsoluteLayout>

    </ContentPage.Content>
</ContentPage>

The DCVImageEditorView control allows you to edit the document edges and save the points to a Quadrilateral object.

 public partial class QuadEditorPage : ContentPage
{
    ImageData data;
    DetectedQuadResult[] results;
    public QuadEditorPage(ImageData imageData, DetectedQuadResult[] results)
    {
        InitializeComponent();
        data = imageData;
        this.results = results;
    }

    protected override void OnAppearing()
    {
        base.OnAppearing();
        if (data != null)
        {
            imageEditor.OriginalImage = data;
        }
        if (results != null)
        {
            imageEditor.DetectedQuadResults = results;
        }
    }

    void OnNormalizeClicked(object sender, EventArgs e)
    {
        try
        {
            var quad = imageEditor.getSelectedQuadResult();
            if (quad != null)
            {
                Navigation.PushAsync(new NormalizedPage(data, quad));
            }
        }
        catch (Exception exception)
        {
            Device.BeginInvokeOnMainThread(async () => {
                await DisplayAlert("Error", exception.ToString(), "OK");
            });
        }

    }
}

Android

iOS

After checking and editing the document edges, we can launch a new page to normalize the document based on the selected quadrilateral.

Content Page for Document Cropping and Perspective Correction

Create a new content page NormalizedPage.xaml to show the normalized document.

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
             xmlns:local="clr-namespace:DocumentScanner"
             x:Class="DocumentScanner.NormalizedPage">
    <ContentPage.Content>
        <AbsoluteLayout>
            <local:GestureView x:Name="gestureView" AbsoluteLayout.LayoutBounds="0.5,0.5,1024,1024"
                AbsoluteLayout.LayoutFlags="PositionProportional">
            </local:GestureView>

            <Button
                x:Name="shareButton"
                Clicked="OnShareClicked"
                Text="Share"
                AbsoluteLayout.LayoutBounds="0.5,1,120,50"
                AbsoluteLayout.LayoutFlags="PositionProportional">
            </Button>

            <StackLayout RadioButtonGroup.GroupName="colors" Orientation="Horizontal">
                <RadioButton Content="Binary" IsChecked="True" CheckedChanged="RadioButton_CheckedChanged"/>
                <RadioButton Content="Color" CheckedChanged="RadioButton_CheckedChanged"/>
                <RadioButton Content="Grayscale" CheckedChanged="RadioButton_CheckedChanged"/>
            </StackLayout>

        </AbsoluteLayout>
    </ContentPage.Content>
</ContentPage>

We put the image in a GestureView.xaml control to allow the user to zoom and pan the image.

<?xml version="1.0" encoding="UTF-8"?>
<ContentView xmlns="http://xamarin.com/schemas/2014/forms" 
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
             x:Class="DocumentScanner.GestureView">
  <ContentView.Content>
        <StackLayout  HorizontalOptions="CenterAndExpand"
                 VerticalOptions="CenterAndExpand"
                     >
            <Image 
                    x:Name="image">
            </Image>

        </StackLayout>
    </ContentView.Content>
</ContentView>

The gesture detection code is from Microsoft's Xamarin.Forms Cookbook.

public partial class GestureView : ContentView
{
    double currentScale = 1;
    double startScale = 1;
    double x = 0;
    double y = 0;

    public Image getImage()
    {
        return image;
    }
    void OnPinchUpdated(object sender, PinchGestureUpdatedEventArgs e)
    {
        ...
    }

    void OnPanUpdated(object sender, PanUpdatedEventArgs e)
    {
        switch (e.StatusType)
        {
            case GestureStatus.Running:
                Content.TranslationX = Math.Max(Math.Min(0, x + e.TotalX), -Math.Abs(Content.Width * Content.Scale - App.ScreenWidth));
                ...
                break;

            ...
        }
    }

    public GestureView()
    {
        InitializeComponent();

        var pinchGesture = new PinchGestureRecognizer();
        pinchGesture.PinchUpdated += OnPinchUpdated;
        GestureRecognizers.Add(pinchGesture);

        var panGesture = new PanGestureRecognizer();
        panGesture.PanUpdated += OnPanUpdated;
        GestureRecognizers.Add(panGesture);
    }
}

Code change is required for pan translation calculation.

- Content.TranslationX = Math.Max (Math.Min (0, x + e.TotalX), -Math.Abs (Content.Width - App.ScreenWidth));
- Content.TranslationY = Math.Max (Math.Min (0, y + e.TotalY), -Math.Abs (Content.Height - App.ScreenHeight));
+ Content.TranslationX = Math.Max(Math.Min(0, x + e.TotalX), -Math.Abs(Content.Width * Content.Scale - App.ScreenWidth));
+ Content.TranslationY = Math.Max(Math.Min(0, y + e.TotalY), -Math.Abs(Content.Height * Content.Scale - App.ScreenHeight));

Back to the NormalizedPage, we specify the color template by selecting a radio button.

private void UpdateNormalizedImage(string template)
{
    App.ddn.InitRuntimeSettings(template);
    normalizedImage = App.ddn.Normalize(imageData, quadrilateral);

    Image image = gestureView.getImage();

    image.Source = normalizedImage.image.ToImageSource();
    if (Device.RuntimePlatform == Device.iOS)
    {
        image.RotateTo(normalizedImage.image.orientation);
    }
}

private void RadioButton_CheckedChanged(object sender, CheckedChangedEventArgs e)
{
    RadioButton button = sender as RadioButton;
    if (button != null)
    {
        if (button.Content.Equals("Binary") && button.IsChecked)
        {
            UpdateNormalizedImage(Templates.binary);
        }
        if (button.Content.Equals("Color") && button.IsChecked)
        {
            UpdateNormalizedImage(Templates.color);
        }
        if (button.Content.Equals("Grayscale") && button.IsChecked)
        {
            UpdateNormalizedImage(Templates.grayscale);
        }
    }
}

The templates are defined in the Templates class.

public class Templates
{
    public static string binary = @"{
    ""GlobalParameter"":{
        ""Name"":""GP""
    },
    ""ImageParameterArray"":[
        {
            ""Name"":""IP-1"",
            ""NormalizerParameterName"":""NP-1""
        }
    ],
    ""NormalizerParameterArray"":[
        {
            ""Name"":""NP-1"",
            ""ColourMode"": ""ICM_BINARY"" 
        }
    ]
}";

    public static string color = @"{
    ""GlobalParameter"":{
        ""Name"":""GP""
    },
    ""ImageParameterArray"":[
        {
            ""Name"":""IP-1"",
            ""NormalizerParameterName"":""NP-1""
        }
    ],
    ""NormalizerParameterArray"":[
        {
            ""Name"":""NP-1"",
            ""ColourMode"": ""ICM_COLOUR"" 
        }
    ]
}";

    public static string grayscale = @"{
    ""GlobalParameter"":{
        ""Name"":""GP""
    },
    ""ImageParameterArray"":[
        {
            ""Name"":""IP-1"",
            ""NormalizerParameterName"":""NP-1""
        }
    ],
    ""NormalizerParameterArray"":[
        {
            ""Name"":""NP-1"",
            ""ColourMode"": ""ICM_GRAYSCALE"" 
        }
    ]
}";
}

Finally, we can save and share the normalized document image as JPEG, PNG or PDF.

string file = Path.Combine(FileSystem.CacheDirectory, "normalized.png");
App.ddn.SaveToFile(normalizedImage, file);
Device.BeginInvokeOnMainThread(async () => {

    await Share.RequestAsync(new ShareFileRequest
    {
        Title = Title,
        File = new ShareFile(file),
        PresentationSourceBounds = DeviceInfo.Platform == DevicePlatform.iOS && DeviceInfo.Idiom == DeviceIdiom.Tablet
        ? new System.Drawing.Rectangle(0, 20, 0, 0)
        : System.Drawing.Rectangle.Empty
    });
});