The post was published in Medium.

For the majority of healthcare systems, COVID has been one of the most challenging periods and the one with the most impactful digital transformation on their services. Additionally, the pandemic amplified and revealed the structural challenges and the predicament of the healthcare system as a whole in terms of scaling, adapting, and adopting new strategies to respond to the challenges posed by the pandemic. With an already perennial shortage of healthcare professionals, the pandemic could saturate the healthcare infrastructure through a compound effect of increasing hospitalizations and limited resources due a higher risk of infection among healthcare professionals. Hence, aside from a lack of personal protective equipment, ventilators and medical supplies, staff shortage has been the most critical risk to address.

To mitigate this risk, the Center for Disease Control and Prevention (CDC) issued recommendations and a contingency capacity strategy to maximize the availability of healthcare personnel and reduce the quarantine period of health personnel who have been exposed to COVID-19. The strategy requires all health personnel to report temperature and absence of symptoms each day before starting work and made strict protocols for healthcare systems to monitor their status. In case of reporting symptoms consistent with COVID-19, individuals had to follow a notification, isolation, and “fast-track” testing protocol.

For healthcare systems with thousands of employees, the implementation of this contingency capacity strategy was vital; though it required a new process to screen, notify, and guide thousands of personnel 24 hours a day. A manual process was unlikely to succeed and would have created a huge bottleneck. Moreover, the speed of implementation and the ever-changing landscape of the pandemic discarded the traditional path of implementing a change management and training regimen for staff to adhere to this new normal of “checking in” on your way to work at a healthcare facility.

So, with a small team at Inception Labs — our clinical informatics group at the Medical College of Wisconsin — and in partnership with Inception Health innovation and operations team, we deployed a fully digital solutions for screening employees using React Native combined with a native cloud solution developed on Amazon Web Services (AWS).

In a nutshell, the platform was composed of two components:

• A frontend mobile application embedded in Froedtert & MCW mobile application.

• A Serverless backend using Amazon Web Services.

Frontend Mobile App

Design and User Experience Workflow

Rather than gathering requirements, we explored how to design an experience — digital and physical — that can help the end-user start their day and be confident to go to work in the midst of the pandemic. In a gist, we extracted the tasks that each individual would have to perform daily and grouped them in the following actions:

• Screening for COVID-19 symptoms has to be easily accessible. Ideally, the user can do it right from their phone with as as few clicks as possible. This task of submitting symptoms has to be reliable, easy, and intuitive. An initial prototype using RedCap deployed in our academic medical center showed that people will stop using the tool if it is too cumbersome to access. Even remembering the web link is a barrier.

• Each individual will have to demonstrate at the entrance of the healthcare facility that she/he has completed the screening right before coming to work . The clearance has to be timestamped and only valid for a specific duration (i.e. 12 hours for instance).

• In case of suspected symptoms, the individual should be notified immediately. The notification must be clear and concise with enough information about next steps. The experience has to be supportive of the end-user. It must help navigate the various options rather than acting as an alert system. Because of the rapid rollout and the constant change of guidelines, most users will need as much information as possible at this step without creating confusion. Otherwise, they might miss important steps and/or perform wrong actions.

• “Jumping” to the digital loop of COVID testing should be seamless. At the time of the design phase, our digital team has released a fully digital and asynchronous e-Visit for COVID testing solution. We decided to integrate with the solution and enable a seamless “hand-over” between the screening platform and the e-Visit service.

• Last but not least, we had to provide a knowledge-base to help users learn about the service, the current guidelines, and the updates related to the program and guidelines.

Integration in Froedtert & MCW Mobile App

This initial exploration phase helped us discard several alternatives and understand the boundaries of the few options we had. We chose to embed the experience within our publicly available Froedtert & MCW mobile app. The app was first released in March 2020 and it has become the backbone of our digital care services. The app delivers our On-Demand Video Visits, our asynchronous e-Visits and embeds several digital services such as scheduled appointment check-ins and patient messaging. The choice of integrating the screening service in the mobile application was obvious for a few reasons:

• The screening for employees has to be intuitive and easy. Hence, a mobile app is one of the best medium to enable a seamless experience. With the mobile application, we can create a custom on-boarding, integrate an identity solution, and use the mobile notification services.

• We own the entire mobile platform. Hence, we can ensure that the experience is tailored to our design heuristics. More importantly, since the mobile app is developed in-house, we had the ability to experiment, deploy, and redesign the digital experience at a much higher velocity than relying on third-party vendors. At the peak of our design of the screening service, we were updating the mobile application several times a day.

• Full control of the data flow and a clear separation of the data was fundamental. We knew that embedding the service in our app will ensure a full end-to-end privacy by-design and ensure that we have full control of the chain of custody of the data submitted by our personnel. We chose from the beginning to have separate APIs and authentication flow for the screening service inside the app. There was no lateral movement of data and the code was logically segregated and compiled in its own bundle.

Backend Architecture

Design Principles

We had to build a backend architecture that will enable 4 core tenants:

• Privacy by-design was one of the core foundations of our platform choice. We chose not to rely on any external vendor or solution. Even when considering HIPAA compliant alternatives, our decision was to make privacy and security as restrictive as possible and hence restricted the data flow to the program itself. As you will read below, the solution was designed as a self-contained component within the mobile application and the navigation to other services was performed without any data transfer or data sharing between the screening service and our digital services. From a compliance perspective, we worked with our legal team to develop custom user terms and data protection in addition to the one we use currently for our mobile app.

• Scalability was also a big concern. With thousands of employees potentially using the solution within the same time window around coming to work, we had to make sure that the service was scalable. We previously experimented using Fargate and Kubernetes. But, we worried about the resource overhead to deploy such architecture. Alternatively, we chose to use a purely Serverless architecture using AWS Lambda function and AWS API Gateway.

• Adaptive Algorithm for Screening was also important. We needed to make sure that we controlled as much as possible of the flow and the screening logic of the frontend. Hence, we chose to create backend APIs to manage the screening logic and several of the frontend interfaces. This makes most of the changes to the logic and the interfaces of the app centrally controlled and deployable anytime without any update to the app itself.

• Reporting was a key function. We created two types of report. One is a generic traditional reporting. The second was a more reactive, event-based reporting and alerting. This latter approach was for monitoring the infrastructure (using AWS CloudWatch alerts) and for notification of exposure (using SNS Topics and SES for notifications).

Infrastructure as a Code — CDK

The cloud can be thought as a remote data center. However, that view does not take into account that the cloud also creates new challenges for infrastructure engineers and distributed computing scientists which in turn create new solutions and new modalities to deliver infrastructure. Those solutions — as in all things related to scalability — embed by-design Automation and Repeatability. This is where AWS Cloud Development Kit (AWS CDK) really shines!

AWS CDK is an open-source programming framework that helps design platforms using the programming language of your choice. In our case, since we spend the majority of our time working in the field of data science and clinical informatics, we chose Python 3.8 (in the most recent iterations, we shifted to Typescript for better support).

Here is an example below of how the infrastructure was written. We create a Stack that creates an API Gateway (line 6), configure a lambda function (lines 11–14), and then exposes the lambda function through the API Gateway (line 16).

The code is debuggable, testable, and deployable using a CI/CD pipeline. This helped us ensure that the platform development is robust and allowed us to increase the speed of deployment and the release cycle of our solution exponentially.

Serverless Architecture

Using CDK, we designed a fully serverless architecture. In a nutshell, the platform is composed of the following components:

• AWS Cognito User Pool for identity management. We used Cognito User Pool to manage the signup flow, and the authorization of all requests to the API Gateway. AWS Cognito is natively supported in AWS API Gateway. We leveraged AWS Cognito User Pool custom lambda triggers to build a passwordless email authentication without performing a deep integration with our On-Prem identity provider (a great AWS blog post discusses this topic in more details).

• API Gateway with WAF and Cognito Authorizer: API Gateway is a fully managed API platform offered by AWS. It integrates seamlessly with AWS Cognito, Lambda functions, and has several built-in functionalities such as throttling, api key management, tracing, data model schema validation, and WAF integration. It is reliable and scalable to support thousands of transactions per second.

• Lambda Functions: We used lambda functions throughout the architecture. From custom User Pool Workflows with Lambda Triggers, to running our screening algorithm, to triggering alerts and notifications using DynamoDB Streams and AWS Lambda Triggers, Lambda function is perhaps the most versatile service we had the chance to use. This is a huge shift from our traditional development which often relied on an end-to-end framework such as Django, or Hapi.js. We found for our use case that distinct and small self-contained lambda functions were the best approach to operate a scalable and loosely-coupled architecture; although, the limited number of functionalities of the screening service helped with the limitation of using Lambda functions without tight integration. We are currently exploring using NestJS serverless integration and AWS Serverless Application Framework (SAM) for larger projects that require more structure. For the screening project though, using bare Lambda functions was sufficient.

• CloudWatch and X-Ray: We enabled by default X-Ray tracking in AWS Lambda and API Gateway. We used an amazing Python package AWS-Lambda-powertools to enable tracing, monitoring, and added custom metrics for all our lambda functions.

• DynamoDB: DynamoDB is blazing fast, cheap, scalable, and the default option for any cloud development in AWS. Although we explored the option of using RDS and other relational databases, the choice of using a document-based DB was trivial.

• Simple Notification Service (SNS) Topics for notifications and alerts: SNS is a very simple, yet very powerful Pub/Sub service. We implemented SNS to enable notifications and alerts that can be sent by email, stored in S3 bucket if needed, or published to any HTTPS endpoint.

• **AWS Amplify **is an open source Mobile SDK that basically offers most of the boilerplate to integrate our mobile application with AWS Cognito, AWS API Gateway, and other AWS Services. The SDK is battle tested and hence helped us speed the development of the mobile integration extremely well. It is not yet well documented and the documentation takes a very opinionated approach to configure the SDK.

Results

Rollout

We started the project in July 2020 and we had the first prototype ready within 10 days. The screening service was officially launched by mid-August. We updated the app more than a dozen times and upgraded the API multiple times.

The initial rollout had several last mile hiccups. for instance, an SMS activation service that we first rolled out was quickly replaced by an in-app activation flow. The main challenge with the SMS activation was that it was quickly overwhelmed with thousands of requests per hour during the first rollout and Apple deep-linking was not reliable. Although we had an integration and scaling tests in place for the app, those last mile steps were difficult to test at scale before going live.

Another major issue was connectivity and the availability of internet access in the “hotspot” locations where users were likely going to use the app such as parking lots or main entrances at our healthcare facilities. We had a team including myself checking the connectivity level at those locations.

Operational Metrics

Since we launched the service, we had more than 18,000 employees signing up for the service; with more than 12,000 screenings performed daily. In total, more than half a million screening were performed using the application since its launch. On average, our API response time was 180 ms with a peak of few hundreds requests per second. We had less than 0.002% 5xx errors through the API Gateway. The only major glitch was when AWS us-east-1 went down.

Customer Support

In addition to the technical team, we established a small support center to help users. With thousands of users trying to use the app, we had a few hundreds (or perhaps less) frustrated users. In most cases, it was out of the scope of the application itself. It was either a network issue, an email activation code in the spam, or an incompatible device. However, all those cases required attention.

Customer support is more than product support. It means helping users — in our case it was helping colleagues — check their phone settings, verifying that they used the correct email address, and in some cases fixing bugs.

Lessons learned

We learned a few lessons along the way:

• Build a contingency plan/solution: if we intend to deliver the service for everyone, we must build the platform to account for a myriad of use cases (or at least 95% of them). Within few weeks releasing the screening program, we launched a website as an alternative option. While this is not as intuitive and straightforward as the mobile development, some users chose to use the website rather than the mobile application.

• Responsiveness and Observability of the entire experience, not only of the components we have control over. This means that we had to test and check the Wifi connectivity in the parking lots (which was very limited), make sure that the mobile application was performing well on all mobile platforms (which was not true, specifically for Android), and that we had good monitoring of the user experience (work-in-progress…). Observability is hard.

• Innovation and Product Management are two faces of the same coin. We had to develop the experience, roll the product, build the documentation, update the help center, deliver support to end users, and maintain the algorithm in accordance with CDC guidelines. This is a snapshot of the miscellaneous tasks and operations that had to be in place and maintained in order to support digital transformation. This was required to maintain a stable and seamless experience. Without such full-responsibility model, the results would have been unfavorable.

• Socio-technical model is always at play. Building a digital solution requires careful consideration of the balance between potential benefits and its side-effects on users, and vice-versa. While technology could be an enabler, it could also create a wider digital divide. This is more pronounced when we build digital care services. The socio-technical interaction has to be considered throughout the design, development, and release.

In Summary

The pandemic has been a shockwave of the entire healthcare ecosystem. Most providers are today aware of the importance of resiliency and ability to be Agile in delivering healthcare services and being digital rather than merely doing digital. As I mentioned in a previous post, we will learn rapidly whether what we are witnessing is a patchwork under unusual circumstances or the beginning of a new era of healthcare transformation.
The Challenges and Opportunities of Digital Health following COVID-19
*In a stroke of a brush, digital was no longer a gadget. It became the foundation of the care delivery and a critical…*medium.com

Our screening service has been one among the many forays of our informatics team at Inception Labs and Inception Health in designing, building, and deploying large scale digital tools. What made this project unique is that it represented the first digital service that we developed for our staff and employees. As the say goes, there is nothing better than “eating your own dog food” to learn and grow. And we surely did…

Acknowledgment

I would like to thank all the people who contributed to this work including — non-exhaustively — Brad Crotty, Michael Anderes, David Hotchkiss, Lauren Marvell, Inception Health team, Froedtert & Medical College legal, communication and HR teams, AWS NPO Team, and our entire healthcare staff and personnel.