We aim to misbehave

Evergreen: Progressive Web App

2017 Evergreen International Conference

2017-04-07

Everyone wants an app

Offline access! Part of my device!
But apps take time to develop:
- iOS vs. Android vs. Android vs. Android
Half of US users download 0 apps per month
- Finding the right app can be hard
- App install banners can be annoying
24% of people abandon an app after one use (Localytics, 2017-03-30)

Let's improve the web app we have

Y'all got on this boat for different reasons, but y'all come to the same place. So now I'm asking more of you than I have before. Maybe all. Sure as I know anything, I know this - they will try again. […] They'll swing back to the belief that they can make [native apps]… better. And I do not hold to that. So no more runnin'. I aim to misbehave. Serenity, performance by Nathan Fillion, 2005-09-30.

Progressive Web Apps are... what?

From graceful degradation to progressive enhancement

Progressive Web Apps (PWAs)

Baseline - make web apps more like native apps:
- Responsive: UI adapts to different devices
- Fast to load and render
- Secure: encrypted by HTTPS
- Reliable under any network condition
- Installable to your device home screen

Progressive Web Apps (PWAs)

Usually built on frameworks like or
Common model: app shell built with templated HTML, minified CSS and JavaScript, with REST calls to JSON data
Recent example: Mobile Twitter (2017-04-06)
Almost like the JSPAC of yore…
Let's see what we can do with the TPAC

Auditing PWA progress

Lighthouse audits web apps for quality
Focused on PWA baseline criteria
Offers advice for exemplary PWAs as well
Chrome extension & command line tool

Baseline: 37/100

Fresh Evergreen 2.12 install, without HTTPS
Praised for:
- Page load performance is fast
- Site is progressively enhanced
- Design is mobile-friendly

Baseline: 37/100

Fresh Evergreen 2.12 install, without HTTPS
Punished for:
- Network connection is not encrypted
- User will not be prompted to Add to Homescreen
- Installed web app will not launch with a splash screen
- Address bar will just be default colour
- App can't handle offline/flaky connections

Let's encrypt that connection

Use certbot to get a free TLS certificate from LetsEncrypt
Point the Apache config at the certificates
Enable TLS for all connections

certbot certonly --webroot -w /openils/var/web/ \
        -d shiny.example.org

vim /etc/apache2/sites-enabled/eg.conf

SSLCertificateFile /etc/letsencrypt/live/shiny.example.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/shiny.example.org/privkey.pem

vim /etc/apache2/eg_vhost.conf

# Uncomment the following to force SSL for everything.
# Note that this defeats caching and you will suffer
# a performance hit.
RewriteCond %{HTTPS} off
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [NE,R,L]

Baseline with HTTPS: 50/100

Zoë: We mentioned that we were out of rations, and 10 minutes later, a bunch of apples rained into the trench.
Wash: And they grew into a big tree, and they all climbed up the tree into a magical land with unicorns and a harp. "War Stories." Firefly, performance by Gina Torres and Alan Tudyk, season 1, episode 9, 2002-12-06.

Manifest destiny

The purpose of the manifest is to install web applications to the homescreen of a device, providing users with quicker access and a richer experience. "Web App Manifest." Mozilla Developer Network

Manifest basics

A JSON file with metadata about the web app
A link to the manifest from the <head>

{
  "short_name": "Shiny Cap'n",
  "name": "Shiny Cap'n",
  "description": "A real shiny public catalogue!",
  "start_url": "/eg/opac/home"
}

<link rel="manifest" href="/manifest.json" />

Manifest icons

Multiple icons for multiple screen densities

{
  "short_name": "Shiny Cap'n",
  "icons": [{
      "src": "/images/icon48.png", "type": "image/png",
      "sizes": "48x48"
    },
    {
      "src": "/images/icon96.png", "type": "image/png",
      "sizes": "96x96"
    },
    {
      "src": "/images/icon192.png", "type": "image/png",
      "sizes": "192x192"
    }],
}

Manifest theming

While the app launches, give it nice colours

{
  "short_name": "Shiny Cap'n",
  "background_color": "#007a54",
  "theme_color": "#007a54",
}

Manifest `display`

When the app launches, set display to either:
- Hide all user agent chrome: fullscreen
- Launch as a separate application: standalone
Set orientation to either portrait or landscape

{
  "short_name": "Shiny Cap'n",
  "display": "standalone",
  "orientation": "portrait"
}

`link` and `meta` matching

<head> should contain <link rel="icon" /> and <meta rel="theme-color" /> elements that match the manifest values:


                        <meta rel="theme-color" value="#007a54" />
                        <link rel="icon" sizes="192x192" href="images/icon192.png" />

HTTPS + manifest: 85/100

'Yes. Yes, this is a fertile land, and we will thrive. We will rule over all this land, and we will call it… "This Land."' "Serenity." Firefly, performance by Alan Tudyk, season 1, episode 1, 2002-09-20. Netflix, https://www.netflix.com/watch/70133870?trackId=200257859.

Optimizations

/eg/opac/home first paint: 1648.1ms

…opac/home (shiny)
  …opac/semiauto.css (shiny) - 719.2ms, 14.61KB
  …opac/simple.js (shiny) - 796ms, 16.4KB
  …tundra/tundra.css (shiny) - 1,060.6ms, 54.93KB
  …css/style.css (shiny) - 1,175.7ms, 60.35KB
  …dojo/openils_dojo.js (shiny) - 1,456.5ms, 12.82KB
  …opensrf/JSON_v1.js (shiny) - 1,541.4ms, 15.97KB
  …opensrf/opensrf_xhr.js (shiny) - 1,571.1ms, 17.78KB
  …themes/dijit.css (shiny) - 1,640.9ms, 36.78KB
  …images/small_logo.png (shiny) - 1,747.4ms, 15.37KB
  …opensrf/opensrf.js (shiny) - 1,840.6ms, 39.31KB
  …images/eg_tiny_logo.png (shiny) - 1,926.7ms, 14.96KB
  …images/main_logo.png (shiny) - 1,941.4ms, 21.25KB
  …dojo/dojo.js (shiny) - 2,106ms, 92.59KB

Dojo?!? Didn't we try to avoid Dojo?

Kaylee: Everything's shiny, Cap'n. Not to fret.
Mal: You told me those entry couplings would hold for another week!
Kaylee: That was six months ago, Cap'n. Serenity, performances by Jewel Staite and Nathan Fillion, 2005-09-30.

`opac/parts/header.tt2`

# Dojo is required to use the copy locations advanced search
# filter, therefore, it should always be enabled.
want_dojo = 1;

IF use_autosuggest.enabled == "t";
    want_dojo = 1;
END;

IF ctx.google_books_preview;
    want_dojo = 1;
END;

IF ENV.OILS_NOVELIST_URL;
    want_dojo = 1;
END;

The fix is in bug 1411699

…opac/home (shiny)
  …opac/semiauto.css (shiny) - 592ms, 11.48KB
  …opac/simple.js (shiny) - 642.3ms, 13.26KB
  …css/style.css (shiny) - 862.8ms, 58.53KB
  …images/eg_tiny_logo.png (shiny) - 1,041.5ms, 11.82KB
  …images/main_logo.png (shiny) - 1,071.5ms, 18.12KB

Still 85/100 , but first paint is down to 1011.9ms!
Most of the points are docked because we fail the App can load on offline/flaky connections test

Are we the Serenity?

Kept in the air by love (and Kaylee)

Or are we a Reaver Ship?

Belching smoke and uncontained radiation...

and Dojo 1.3

Handling offline/flaky connections:

Introducing service workers

Service workers

A separate JavaScript per origin scope
Runs on a separate thread
No access to the DOM
Can intercept and modify network requests for fine-grained caching strategies
Enables push notifications, too!

Service worker support

Supported by Chrome, Firefox, Opera today
Microsoft Edge is working on it
Safari… 🤷
But progressive enhancement means it's okay!

Service worker registration

<script>
if ('serviceWorker' in navigator) {
  window.addEventListener('load', function() {
    // Your service worker *must* be located at
    // the top-level directory relative to your site.
    // It won't be able to control pages unless it's
    // located at the same level or higher than them.
    navigator.serviceWorker.register('/sw.js')
      .then(function(reg) { /* … stuff! */ })
      .catch(function(e) { console.error('Error:', e); });
  }
}
</script>

Adapted from Google SW registration boilerplate

Service worker: Fetch API

More flexible than XMLHttpRequest()
- Generic Request, Response, and Headers interfaces
FetchEvent handler + fetch() + Cache API + cache dedicated to a given web app:
- Fetch network requests from the cache
- Store network responses in the cache
- Overcome network flakiness!

Let there be patterns!

serviceworke.rs offers examples of caching strategies

Network-or-cache

self.addEventListener('fetch', function(evt) {
  if evt.respondWith(fromNetwork(evt.request, 400)
  .catch(function () {
    return fromCache(evt.request);
  }));
});
function fromNetwork(request, timeout) {
  return new Promise(function (fulfill, reject) {
  var timeoutId = setTimeout(reject, timeout);
  fetch(request).then(function (response) {
      clearTimeout(timeoutId);
      fulfill(response);
    }, reject);
  });
}

Cache and update

self.addEventListener('fetch', function(evt) {
  evt.respondWith(fromCache(evt.request));
  evt.waitUntil(update(evt.request));
});
function fromCache(request) {
  return caches.open(CACHE).then(function (cache) {
    return cache.match(request).then(function (matching) {
      return matching || Promise.reject('no-match');
    });
  });
}
function update(request) {
  return caches.open(CACHE).then(function (cache) {
    return fetch(request).then(function (response) {
      return cache.put(request, response);
    });
  });
}

Let there be tools!

We could roll our own service worker based on the serviceworke.rs patterns
But libraries provide a nice layer of abstraction:
- sw-toolbox implements network handlers with five request/caching patterns
- sw-precache builds on sw-toolbox to generate a service worker based on a configuration file

package.json

{
  "name": "evergreen-pwa-stuff",
  "version": "1.0.0",
  "description": "Evergreen PWA stuff",
  "author": "Dan Scott",
  "license": "GPL-2.0+",
  "dependencies": {
    "sw-precache": ">4.3.0"
  }
}

$ npm install

sw-precache basic dumb config

module.exports = {
  staticFileGlobs: [
    '/openils/var/web/css/skin/default/opac/semiauto.css',
    '/openils/var/web/js/ui/default/opac/simple.js',
    '/openils/var/web/opac/images/small_logo.png',
    '/openils/var/web/opac/images/progressbar_green.png',
    '/openils/var/web/opac/images/main_logo.png',
    '/openils/var/web/opac/images/eg_tiny_logo.png'
  ],
  stripPrefix: '/openils/var/web/',
  runtimeCaching: [{
    urlPattern: /^https:\/\/shiny.example.org\//,
    handler: 'networkFirst'
  }]
};

Enable our service worker

Generate and install the service worker
Install the boilerplate registration code
Add registration code to templates/opac/parts/base.tt2


$ node_modules/sw-precache/cli.js --config sw-precache.js
Total precache size is about 18.2 kB for 6 resources.
service-worker.js has been generated.
$ cp service-worker.js /openils/var/web/sw.js


$ cp sw-register.js /openils/var/web/.

HTTPS + manifest + service worker: 100/100

Shut. Up.

100/100 PWA score!^*

Specifically, no My Account pages are available offline
That was kind of the point of the exercise

^*Score may not reflect real-world experiences

"My Account" headers


cache-control: no-store, no-cache, must-revalidate
expires: -1

no-store means "never cache this"
expires: -1 means "this expires immediately"
No network connection? You shall not see your checked out items!

Radical surgery


diff --git a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm
index 6aa1f58..6548edc 100644
--- a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm
+++ b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm
@@ -184,8 +184,7 @@ sub load {
 return $self->redirect_auth unless $self->editor->requestor;

 # Don't cache anything requiring auth for security reasons
-$self->apache->headers_out->add("cache-control" => "no-store, no-cache, must-revalidate");
-$self->apache->headers_out->add("expires" => "-1");
+$self->apache->headers_out->add("cache-control" => "no-cache, must-revalidate");

 return $self->load_email_record if $path =~ m|opac/record/email|;

Rapid decay

If we remove those headers, Expires becomes + 5 seconds due to /etc/apache2/eg_vhost.conf:


<Location /eg/opac>
  PerlSetVar OILSWebContextLoader "OpenILS::WWW::EGCatLoader"
  # Expire the HTML quickly since we're loading dynamic data
  ExpiresActive On
  ExpiresByType text/html "access plus 5 seconds"
</Location>

It's dynamic, but there are alternatives to Expires

Enter ETag


diff --git a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm
index da18d7e..21d4715 100644
--- a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm
+++ b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGWeb.pm
@@ -129,11 +129,16 @@ sub handler_guts {
 $vhost_processor_cache{$processor_key} = $tt;
 $ctx->{encode_utf8} = sub {return encode_utf8(shift())};

-unless($tt->process($template, {ctx => $ctx, ENV => \%ENV, l => $text_handler}, $r)) {
+my $_out = '';
+unless($tt->process($template, {ctx => $ctx, ENV => \%ENV, l => $text_handler}, \$_out)) {
     $r->log->warn('egweb: template error: ' . $tt->error);
     return Apache2::Const::HTTP_INTERNAL_SERVER_ERROR;
 }

+my $etag = md5_hex(Encode::encode_utf8($_out);)
+$r->headers_out->add('Etag' => $etag);
+$r->print($_out);
+
 return Apache2::Const::OK;

ETag and Last-Modified enable the browser to check for updates

Whither `Expires`?

Given ETag and no-cache, the browser will always make a request to the server
If the server can't respond (is down, or has no network connection), the service worker can
Bonus: for other HTML pages, Expires can be turned off


<Location /eg/opac>
  PerlSetVar OILSWebContextLoader "OpenILS::WWW::EGCatLoader"
  ExpiresActive Off
</Location>

Add to Home Screen: Shiny!

Android notification that the PWA has been added to the home screen

Android PWA launcher splash screen transition

HTTP/2

Lighthouse recommends using HTTP/2
One TCP connection multiplexes requests and compresses headers
More efficient than four to eight separate connections per origin (http2 FAQ)
Nginx--which we're already using to proxy websockets--supports HTTP/2
Let's get this thing going

HTTP/2 on Debian Jessie

Jessie doesn't deliver a new enough version, so:


apt-get -t jessie-backports install nginx

HTTP/2: nginx config


diff --git a/examples/nginx/osrf-ws-http-proxy b/examples/nginx/osrf-ws-http-proxy
index d079230..2fbf832 100644
--- a/examples/nginx/osrf-ws-http-proxy
+++ b/examples/nginx/osrf-ws-http-proxy
@@ -19,7 +19,7 @@ server {
 }

 server {
-    listen 443;
+    listen 443 ssl http2;
     ssl on;

     # Use the same SSL certificate as Apache.

HTTP/2: Are we good?


$ curl --http2 -H "Accept-Encoding: gzip" \
       -I https://shiny.example.org/eg/opac/home
HTTP/2 200
server: nginx/1.10.3
date: Sat, 01 Apr 2017 15:44:15 GMT
content-type: text/html; encoding=utf8
etag: 1735fc80e7ea268e656d53ac9126e438

nginx falls back to HTTP 1.1 for browsers that don't support HTTP/2

(Yes, I tested with w3m)

Brain dead approach PWA results

Not bad. Almost ready for prime time!

We've inadvertently enabled the web staff client--yay!?

But we're lacking a few things…

An offline fallback page

A search-driven app will lead to cache misses
Ideally, tell the user what they can access (perhaps via History API?)
This will require custom service worker code

Better control over GET params

Default behaviour is to treat the following pages as completely different:
- https://shiny.example.org/eg/opac/myopac/holds?query=potter
- https://shiny.example.org/eg/opac/myopac/holds?query=harry
For the /myopac/ pages, at least, GET params shouldn't matter
This will require custom service worker code

Separate the apps

/eg/opac/, /eg/staff/, and /eg/myopac/ are conceptually separate apps
But common JS, image, CSS assets under / require them all to use the same service worker
Instead, use /eg/opac/assets/, /eg/staff/assets/, and /eg/myopac/assets to serve up static assets from /openils/var/web/?

A modern `/eg/myopac`?

Precache a static app shell
Load fines, holds, lists, messages, etc as JSON over REST
Deliver native device notifications with the Push API

I got people with me, people who trust each other, who do for each other and ain't always looking for the advantage. There's good people in the 'verse. Not many, lord knows, but you only need a few. "Our Mrs. Reynolds." Firefly, performance by Nathan Fillion, season 1, episode 3, 2002-10-02.

Details

Slides: stuff.coffeecode.net/2017/evergreen-progressive-web-app
Email: dscott@laurentian.ca
Social: @denials / +DanScottCAN
Blog: coffeecode.net

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

We aim to misbehave

Evergreen: Progressive Web App

Everyone wants an app

Let's improve the web app we have

Progressive Web Apps are... what?

From graceful degradation to progressive enhancement

Progressive Web Apps (PWAs)

Progressive Web Apps (PWAs)

Auditing PWA progress

Baseline: 37/100

Baseline: 37/100

Let's encrypt that connection

Baseline with HTTPS: 50/100

Manifest destiny

Manifest basics

Manifest icons

Manifest theming

Manifest display

link and meta matching

HTTPS + manifest: 85/100

Optimizations

Dojo?!? Didn't we try to avoid Dojo?

opac/parts/header.tt2

The fix is in bug 1411699

Are we the Serenity?

Or are we a Reaver Ship?

Handling offline/flaky connections:

Introducing service workers

Service workers

Service worker support

Service worker registration

Service worker: Fetch API

Let there be patterns!

Network-or-cache

Cache and update

Let there be tools!

package.json

sw-precache basic dumb config

Enable our service worker

HTTPS + manifest + service worker: 100/100

100/100 PWA score!*

"My Account" headers

Radical surgery

Rapid decay

Enter ETag

Whither Expires?

Add to Home Screen: Shiny!

HTTP/2

HTTP/2 on Debian Jessie

HTTP/2: nginx config

HTTP/2: Are we good?

Brain dead approach PWA results

An offline fallback page

Better control over GET params

Separate the apps

A modern /eg/myopac?

Details

Manifest `display`

`link` and `meta` matching

`opac/parts/header.tt2`

100/100 PWA score!^*

Whither `Expires`?

A modern `/eg/myopac`?