archive blog about

2020/1/26

Building a Web QR Scanner with Snowpack and WebAssembly

With the capabilities of the modern web, there are tons of applications that can be built that were traditionally only possible within native apps. Today, we're gonna build a basic QR Scanning app using novel web standards and practices, including ES Modules and WebAssembly.

Snowpack and App Structure

First thing we'll want to do is create our app directory structure. In order to utilize the fantastic tooling of modern Javascript, we normally have to use a bundler like Webpack or Rollup in order to get our CommonJS dependencies to compile on the browser.

However, nowadays there's a new project called pika.dev that's helping out Javascript developers utilize ES Modules for external dependencies. We're gonna use a program called Snowpack to help us out. Snowpack will help us convert our dependencies in our node_modules directory into a web_modules directory. These new web_modules dependencies can be directly fetched by our browser and run without any bundling required. Super cool.

We'll assume that you have npm and node installed. If not, I'd recommend downloading the required software for your operating system here. If this is your first time using npm, I'd also recommend reading this article here about how to use it and what it's good for.

Let's started by installing Snowpack:

$ npm install --save-dev snowpack

Or with yarn:

$ yarn add --dev snowpack

Let's create our project directory. We'll start with:

$ mkdir wasm-qrscanner
$ cd wasm-qrscanner
$ npm init --yes

Next we install our dependencies. We'll be using Preact as it's a light and fast React alternative, ideal for small apps that still want to use modern techniques.

$ npm install --save preact

$ npx snowpack
npx: installed 212 in 56.593s
✔ snowpack installed: preact. [0.22s]

We'll want to be using Typescript, so we'll add that as part of our dev dependencies

$ npm install --save-dev typescript serve concurrently

You'll see in our web_modules directory we now have preact.js file that will be served out to the browser with no additional tooling required. Let's get some basics created to verify this is all working. First, we'll create our HTML index file.

<!-- src/index.html -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>WASM QRScanner</title>
    <meta charset="utf-8">
    <meta name="author" content="Stephen Peterkins">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <meta name="mobile-web-app-capable" content="yes">
    <meta name="apple-mobile-web-app-capable" content="yes">
    <link rel="stylesheet" href="/style.css">
  </head>
  <body>
    <div id="app"></div>
    <script type="module" src="/index.js"></script>
  </body>
</html>

Next, we'll create our src directory and populate it with app.js.

// src/index.tsx

import { h, render, FunctionalComponent} from '/web_modules/preact.js'

const App: FunctionalComponent = () => (
  <div>
    <h1>Hello World</h1>
  </div>
);

const appMount = document.querySelector('#app')
if (appMount) render(<App />, appMount)

export default App

// package.json

{
  "name": "wasm-qrscanner",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "npm run build:ts && npm run build:esm",
    "build:esm": "npx snowpack --dest dist/web_modules --optimize",
    "build:ts": "rm -rf dist && tsc",
    "build:ts:watch": "tsc -w",
    "dev": "npm run build && concurrently 'npm run build:ts:watch' 'serve -s dist'",
    "prestart": "npm run build",
    "start": "serve -s dist"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "snowpack": {
    "webDependencies": [
      "preact"
    ]
  },
  "dependencies": {
    "preact": "^10.2.1"
  },
  "devDependencies": {
    "concurrently": "^5.0.2",
    "serve": "^11.3.0",
    "typescript": "^3.7.5"
  }
}

// tsconfig.json

{
  "compilerOptions": {
    "baseUrl": ".",
    "esModuleInterop": true,
    "jsx": "react",
    "jsxFactory": "h",
    "lib": ["dom", "esnext"],
    "module": "esnext",
    "moduleResolution": "node",
    "outDir": "dist",
    "sourceMap": true,
    "strict": true,
    "target": "esnext",
    "paths": {
      "/web_modules/*.js": [
        "node_modules/@types/*",
        "node_modules/*",
        "public/web_modules/*"
      ]
    }
  },
  "exclude": ["node_modules", "web_modules"]
}

Run

$ npm install
$ npm start

to take a gander at what it looks like!

That completes our initial project structure setup! Next, we'll want to add the our WebAssembly files to do the heavy lifting of our app.

Web Assembly and QUIRC files

WebAssembly is a new file format that can run uber performant code in the browser. We'll want to be using .wasm files for image recognition and QR decoding process. The first thing we want to do is get our wasm files from a fork of Daniel Beer's QR decoder library. To save some time, we'll be pulling the pre-compiled files from this repo.

The files we're most interested in are these located in the wasm directory. Here are links to download the files we want:

And this is where we'll want to place them in our directory structure:

src/lib/quirc/quirc.js
src/lib/quirc/quirc.wasm

We'll create a quirc directory in a new lib directory located in src. It's good practice to separate external libraries and our app logic.

Our quirc.js file will be calling our quirc.wasm to do most of the heavy lifting of qr decoding. However, we'll ned a way to instantiate and call our quirc.js module and have it function with the rest of our existing app logic. The easiest way to achieve this will be running our quirc logic as a Javascript Worker.

Let's create a file called quirc_worker.js in our quirc directory.

// src/lib/quirc/quirc_worker.js

var image=null;
var width, height;
var counted;

var Module = {};

// We start by importing our quirc.js script
importScripts('/quirc/quirc.js');


self.onmessage = function(msg) {
  quirc_process_image_data(msg.data);
  postMessage('done');
}

// Our worker recieves raw image data from the decoder,
// then it posts the message back to our listeners.
self.decoded = function(i, version, ecc_level, mask, data_type, payload, payload_len) {
  var payload_string = String.fromCharCode.apply(null,
    new Uint8Array(Module.HEAPU8.buffer, payload, payload_len));
  postMessage({
    i,
    version,
    ecc_level,
    mask,
    data_type,
    payload,
    payload_len,
    payload_string
  });
}

// Receives a simple string with an error
self.decode_error = function(errstr) {
  console.log("decode error: " + errstr);
}

function quirc_process_image_data(img_data) {
  if (!image) {
    width = img_data.width;
    height = img_data.height;
    image = Module._xsetup(width, height);
  }

  var data = img_data.data;

  for (var i=0, j=0; i < data.length; i+=4, j++) {
    // We convert our image data into grayscale here
    // This is to help with edge detection when quirc is attempting pattern recognition
    Module.HEAPU8[image + j] = (data[i] * 66 + data[i + 1] * 129 + data[i + 2] * 25 + 4096) >> 8;
  }

  // Note that "decoded" and/or "decode_error" will be called from within
  var a = Module._xprocess();
}

We'll also need to add to our copy script in package.json so that we're copying over our quirc library when we're building out our dist dir.

// package.json

// ...
"scripts": {
  // ...
  "copy": "copyfiles 'src/*.html' 'src/assets/*' 'src/lib/quirc/*.js' ''src/lib/quirc/*.wasm'' 'src/**/*.gif' 'src/*.css' dist -u 1",
  // ...
  }
},

Great! We're ready to add our view logic to our QR Scanner. Let's create start to flesh out our app and put our WebAssembly module to good use.

Preact and WebRTC Video Streams

Let's first start by creating a components folder and populating it with the components we'll be using. From the root directory:

$ cd src
$ mkdir components
$ mkdir components/App
$ touch components/App/index.jsx
$ mkdir components/QRScanner
$ touch components/QRScanner/index.jsx

Next, we're gonna want to restructure our entrance into the app to fit our new app architecture. In our src/index.tsx file, we'll change it to the following:

// src/index.tsx

import { h, render } from '/web_modules/preact.js'
import App from './components/App/index.js'

const appMount = document.querySelector('#app')
if (appMount) render(<App />, appMount)

export default App

// src/components/App/index.tsx

import { h, FunctionalComponent } from '/web_modules/preact.js'
import QRScanner from '../QRScanner/index.js'

const App: FunctionalComponent = () => (
  <div>
    <QRScanner />
  </div>
)

export default App

// src/components/QRScanner/index.tsx

import { h, Component } from '/web_modules/preact.js'

type QRScannerProps = {}
type QRScannerState = {
  decoded_string: ''
}

class QRScanner extends Component<QRScannerProps, QRScannerState> {
  render() {
    <div>
      <h1>QRSCanner App</h1>
    </div>
  }
}

export default QRScanner

Cool, we have all of the pieces in place. All of our logic will exist in our QRScanner component. We'll first add the most important part of a QRSCanner,access to the camera! We want to be retrieving whatever camera accesses our device has, and use it to get image data to our quirc module will process and decode.

Modern web standards allow us to do this quite easily with APIs provided by WebRTC. We'll start first by adding our HTML video element to our QRScanner component.

// src/components/QRScanner/index.tsx

import { h, Component } from '/web_modules/preact.js'

type QRScannerProps = {}
type QRScannerState = {
  decoded_string: ''
}

class QRScanner extends Component<QRScannerProps, QRScannerState> {
  render() {
    return (
      <div>
        <div class="video-container">
          <video playsInline autoPlay></video>
        </div>
        <canvas id="qr-canvas"></canvas>
      </div>
    )
  }
}

export default QRScanner

We also want to be adding some styling to our video element. We'll create a style.css file in our root directory and add our styling:

/* src/style.css */
html, body {
  position: relative;
	width: 100%;
	height: 100%;
}

body {
  margin: 0;
	padding: 0;
	box-sizing: border-box;
}

.video-container {
  position: relative;
  top: 0;
  bottom: 0;
  width: 100vw;
  height: 100vh;
  overflow: hidden;
}

.video-container video {
  /* Make video to at least 100% wide and tall */
  min-width: 100%;
  min-height: 100%;

  /* Setting width & height to auto prevents the browser from stretching or squishing the video */
  width: auto;
  height: auto;

  /* Center the video */
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%,-50%);
}

#qr-canvas {
  display: none;
}

OK, so we have a video element which is full screen in our browser. You may have noticed we also have a canvas element that's set to be invisible with display: none. This canvas will be important when we're processing our video stream images.

We won't be able to see any output yet until we enable WebRTC to bind a video stream to our video element. Let's do that next.

In QRScanner.tsx, we want to initialize WebRTC video when we mount our component.

import { h, createRef, Component } from '/web_modules/preact.js'

type QRScannerProps = {}
type QRScannerState = {
  decoded_string: ''
}

class QRScanner extends Component<QRScannerProps, QRScannerState> {

  video = createRef();
  canvas = createRef();

  isStreamInit = false;
  constraints = {
    audio: false,
    video: {
      facingMode: 'environment'
    }
  };

  async componentDidMount() {
    try {
      let stream = await navigator.mediaDevices.getUserMedia(this.constraints);
      this.handleSuccess(stream);
    } catch (err) {
      this.handleError(err);
    }
  }

  handleSuccess(stream: any) {
    this.video.current.srcObject = stream;
    this.isStreamInit = true;
  }

  handleError(error: Error) {
    console.log('navigator.MediaDevices.getUserMedia error: ', error.message, error.name);
  }

  render() {
    return (
      <div>
        <div class="video-container">
          <video playsInline autoPlay ref={this.video}></video>
        </div>
        <canvas id="qr-canvas" ref={this.canvas}></canvas>
      </div>
    )
  }
}

export default QRScanner

There's a lot going on here, so let's break it down a bit further.

First, we're creating references for our video and canvas DOM elements so we can explicitly access these elements in our program.

We're obtaining our video stream by making a request to the browser's navigator, which manages permissions surrounding the devices that our app can communicate with. This will trigger a permission request to the user to allow the use of the camera.

If we're given permission, we'll set this video stream provided by the camera to our video element. This will then display the stream in our browser window.

Try running it with

$ npm start

and if you've been following along, you should be able to see your beautiful face right there on your screen 😁

Now, it's time for us to add our quirc logic. First, we're gonna want to declare a few more attributes in our QRScanner component.

  // src/components/QRScanner/index.tsx

  //...

  isStreamInit = false;
  constraints = {
    audio: false,
    video: {
      facingMode: 'environment'
    }
  };

  // quirc wasm
  decoder = new Worker('/lib/quirc/quirc_worker.js');
  last_scanned_raw = new Date().getTime();
  last_scanned_at = new Date().getTime();

  // In milliseconds
  debounce_timeout = 750;

Next, we'll be using this attributes in our componentDidMount function to kickoff the decoder logic using our Worker.

async componentDidMount() {
  this.decoder.onmessage = (msg) => { this.onDecoderMessage(msg) };

  try {
    let stream = await navigator.mediaDevices.getUserMedia(this.constraints);
    this.handleSuccess(stream);
  } catch (err) {
    this.handleError(err);
  }

  setTimeout(() => { this.attemptQRDecode() }, this.debounce_timeout);
}

Let's break down what we're doing here. First, we're getting our canvas_context, which will help us get a "snapshot" of our video stream that will be fed into the quirc module to start the process of scanning for QR codes.

We're overriding our decoder Worker's onmessage function to a function called onDecoderMessage. onDecoderMessage will accept a decoded message passed from the quirc module.

We'll also be delaying initializing the scanning function attemptQRDecode. This function is pretty self explanatory, we'll be attempting to decode a snapshot of our video stream to detect if there's any QR codes in frame. If there is, it will attempt to successfully decode it and pass the decoded message to our onDecoderMessage function.

Let's create both our attemptQRDecode and onDecoderMessage functions next.

// src/components/QRScanner/index.tsx

// ...

attemptQRDecode() {
  if (this.isStreamInit)  {
    try {
      let canvas_context = this.canvas.current.getContext("2d");
      this.canvas.current.width = this.video.current.videoWidth;
      this.canvas.current.height = this.video.current.videoHeight;
      canvas_context.drawImage(this.video.current, 0, 0, this.canvas.current.width, this.canvas.current.height);

      var imgData = canvas_context.getImageData(0, 0, this.canvas.current.width, this.canvas.current.height);

      if (imgData.data) {
        this.decoder.postMessage(imgData);
      }
    } catch (err) {
      if (err.name == 'NS_ERROR_NOT_AVAILABLE') setTimeout(() => { this.attemptQRDecode() }, 0);
        console.log("Error");
        console.log(err);
    }
  }
}

In attemptQRDecode, we're getting the frame data of the video stream and converting it into an image using our canvas_context attribute. Once we have this image saved in our imgData variable, we fire it off to our Worker's decoder function to start the processing.

If the decoder is able to detect and decode a QR code successfully, it will run the onmessage function that we've set to run our onDecoderMessage. Let's create that logic right now.

// src/components/QRScanner/index.tsx

// ...

onDecoderMessage(msg: any) {
  if (msg.data != 'done') {

    const qrid = msg.data['payload_string'];
    const right_now = Date.now();

    if (qrid != this.last_scanned_raw || this.last_scanned_at < right_now - this.debounce_timeout) {
      this.last_scanned_raw = qrid;
      this.last_scanned_at = right_now;

      alert(qrid);
    } else if (qrid == this.last_scanned_raw) {
      this.last_scanned_at = right_now;
    }
  }
  setTimeout(() => { this.attemptQRDecode() }, 0);
}

In our function, we want to receive our payload_string from the msg, as this is our decoded message. We also cache the last successful scan data in last_scanned_raw so that the user doesn't get spammed over and over with the their decoded information. We also manually check that we're passed our set debounce_timeout. Once these factors have been cleared, we'll show the qrid with the built in alert function to display the information in the browser.

Let's run our code with npm start once again to test it out. For testing purposes, I'd recommend using this online QR code generator to create a QR code on your phone. Running the app, you'll see yourself on your screen using your camera. Pointing our phone screens (with our newly generated QR code on it) at our laptop screens will allow our program to decode it.

Success! We officially have a working QR Code Scanner! 🎉🎊 Give yourself a pat on the back for making it this far.

While we have a working prototype, it's still lacking a lot of the style that makes it "feel" like it's really scanning anything. For first time users, they wouldn't know what the app is supposed to do. Let's add some visual styling to make it clear what our app means to accomplish.

QR Scanner Styling

Most QR scanners are going to put a dark filter over the screen except for a square in the middle. This is the target that the user is supposed to line up the QR code with. This sort of styling has a purpose, it's there to provide intuitive knowledge of how to use the scanner for first time users.

We'll want to edit our QRScanner component's render function to add the necessary elements:

// src/components/QRScanner.index.tsx

// ...

render() {
  return (
    <div>
      <div id="qr-hud">
        <div class="opaque-black" id="qr-hud-header">
          <h2>QR Code Scanner</h2>
        </div>
        <div id="qr-hud-body">
          <div class="opaque-black" id="qr-hud-top"></div>
          <div id="qr-hud-mid">
            <div class="opaque-black"></div>
            <div id="qr-hud-target"></div>
            <div class="opaque-black"></div>
          </div>
          <div class="opaque-black" id="qr-hud-bot"></div>
        </div>
      </div>
      <div class="video-container">
        <video playsInline autoPlay ref={this.video}></video>
      </div>
      <canvas id="qr-canvas" ref={this.canvas}></canvas>
    </div>
  )
}

Now that we have our elements in place, we'll add append our new styling here:

.opaque-black {
  background: rgb(0,0,0,0.4);
}

#qr-hud {
  position: absolute;
  z-index: 100;
  width: 100vw;
  height: 100vh;
  display: grid;
  grid-template-rows: 4em auto;
}

#qr-hud-header {
  color: white;
  padding: 0 1em;
  display: grid;
  align-items: center;
}

#qr-hud-header h2 {
  text-align: center;
  font-family: arial;
}

#qr-hud-body {
  display: grid;
  grid-template-rows: 2fr auto 3fr;
}

#qr-hud-mid {
  display: grid;
  grid-template-columns: 1fr auto 1fr;
}

.qr-hud-mid-opacity {
  opacity: 0.4;
  background: black;
}

#qr-hud-target {
  width: 18em;
  height: 18em;
  border: 2px solid red;
  background-color: transparent;
}

Let's take a quick preview to see how this looks.

Not too shabby (and I'm not just talking about myself here).

Conclusion

In this tutorial, we have created a QR Code scanner with WebAssembly and the latest in modern web technology. As a component, you'll easily be able to export this functionality to any app you'll be creating in the future.

We've showcased how to one can use WebAssembly for the most performance critical portions of our application, and how bundlers are not a requirement for modern web development.

GitHub repo: https://github.com/BearGuy/wasm-qrscanner

Demo: https://gifted-bhaskara-a52e4f.netlify.com

© | Stephen Peterkins | Powered by Zola