DIALLED

Distributed Index of Allsome Library Location and Event Data

Dan Scott, January 29, 2016

The 1-minute version

There is linked data!
Current data dump in Turtle format is at https://dialled.ca
There is not a huge amount of data
There is still a lot of work to do
Budget = my time + $0
Please help (both lightweight and heavyweight participation is encouraged!)

Inspired by CWRC

Thinking about the challenge of turning this around, and of convincing thousands of libraries and archives to adopt schema.org on just the most basic of all of their web pages, I *almost* gave up. That's a lot of inertia to overcome. But in January 2015, members of the Royal Society of Canada expert panel report on the future of libraries in Canada presented their work at the OLA SuperConference--and part of that work included a map of almost 5,000 Canadian libraries and archives. That map was an effort contributed by the Canadian Writing Research Collaboratory (CWRC), and the work on the initial version wrapped up in June 2014.

I was impressed--every one had lat/long data, along with some related institutional data, including URL. For sure, there were problems: all of the lat/longs for Toronto were the same, perhaps for clustering purposes. But still!

A challenge was that it was almost entirely a manual effort, dated June 2014. And the clock continues to tick, and things continue to change, and the data continues to get more out of sync.

It drives me crazy that, despite more than 90% of the entries having URLs, we can't simply regularly crawl the sites and update and augment the data.

Which is why this summer I was delighted to meet Susan Brown, who spearheaded the CWRC effort. She recognizes the problem, understands the value of linked open data, and wants to collaborate on solutions to keeping the data fresh and relevant.

Visualization of RDFa for a book in Evergreen

Visualization of RDFa for a book in Evergreen, focusing on the schema:Offer section

Visualization of RDFa for a book in Evergreen, focusing on the schema:seller section of the offer

Library linked data principles

Every bibliographically described resource, its instances
Every instance, its offering library
Every library, its URL!
... with hours, location, contact data so people can get to your stuff

I started at the bottom

Our MARC records now link to OCLC Work URIs and LoC IDs
- Evergreen expresses these using schema.org in RDFa
OCLS is similarly enriching their MARC records
Zepheira offers a MARC->BIBFRAME->HTML + schema.org service

WorldCat Registry

authoritative single source for institutional metadata

Problematic non-commercial license
Mostly missing or out-of-date data

Library and Archives Canada

Screenshot of 4 results for type = Public and city = Toronto in LAC Union Catalogue

Germans rock linked open data

beta.lobid.org offers a web API for a comprehensive dataset of library institutions in Germany
Major hat tip to Adrian Pohl!

GET http://beta.lobid.org/organisations/DE-6
{
  "isil" : "DE-6",
  "wikipedia" : "http://de.wikipedia.org/wiki/Universitäts-_und_Landesbibliothek_Münster",
  "@context" : "http://beta.lobid.org/organisations/context.jsonld",
  "linkedTo" : "http://lobid.org/organisation/DE-605",
  "@type" : "http://schema.org/Library",
  "rs" : "055150000000",
  "name" : "Universitäts- und Landesbibliothek Münster",
  "url" : "http://www.uni-muenster.de/ULB/",
  "provides" : "http://www.uni-muenster.de/ULB/katalog/",
  "@id" : "http://beta.lobid.org/organisations/DE-6#!",
  "telephone" : "0049 251 83-24040",
  "email" : "mailto:info.ulb@uni-muenster.de",
  "address" : {
    "postOfficeBoxNumber" : "80 29",
    "addressLocality" : "Münster",
    "@type" : "http://schema.org/PostalAddress",
    "addressCountry" : "Germany",
    "postalCode" : "48043"
  },
  "location" : [ {
    "type" : "http://schema.org/Place",
    "address" : {
      "postalCode" : "48143",
      "streetAddress" : "Krummer Timpen 3",
      "addressLocality" : "Münster",
      "addressCountry" : "Germany",
      "@type" : "http://schema.org/PostalAddress"
    },
    "openingHoursSpecification" : {
      "@type" : "http://schema.org/OpeningHoursSpecification",
      "description" : "Mo-Fr 8.00-22.00, Sa-So 10.00-20.00"
    },
    "geo" : {
      "lat" : "51.96286",
      "lon" : "7.62076"
    }
  } ]
}

So, about that CWRC work...

Screenshot of the CWRC map of libraries and archives in Canada

CWRC's dataset

{
    "community": "Sudbury",
    "group": "Libraries",
    "institutionType": "Library",
    "label": "Laurentian University",
    "latLng": "46.5011111,-80.9775",
    "latitude": "46.5011111",
    "libraryType": "Academic",
    "longitude": "-80.9775",
    "population": null,
    "provCode": "ON",
    "province": "Ontario",
    "sort": "707",
    "startDate": "1960",
    "subGroup": "Academic libraries",
    "url": "http://library.laurentian.ca/content/library"
}

CWRC very generously made the data they had collected available to download on a province-by-province basis, in a simple JSON format.

So here's a typical entry of what we're starting with. The granularity of the address info is low: we're working with the city level, so the latitude and longitude doesn't really help us if we want to see the distribution of branches across a given city, but it's a starting point.

The crucial piece is the URL. Given a URL, you can grab the HTML for the library's home page and (perhaps!) pull some current data from it.

At the very least, you can find out if the URL is still current.

Step 1. Crawl the URLs

4,798 entries in the CWRC dataset
4,250 with URLs
3,223 unique URLs
561 URLs redirected
3,141 URLs retrieved

Step 2. Update the CWRC dataset

{
    "...": "...",
    "url": "http://laurentian.ca/library",
    "urlPrevious": "http://library.laurentian.ca/content/library"
}

Here's where we start annotating the CWRC dataset so that we can improve for subsequent runs. In this case, adding a urlPrevious entry when we receive a redirect.

Step 3. Extract structured (or linked) data

Microdata (itemscope, itemtype, itemprop): 191 pages
RDFa (vocab, typeof, property): 197 pages
JSON-LD (<script type="application/ld+json">) : 112 pages

Encouraging!

Step 4. Evaluate the extracted data

Some of it is weaksauce...

Step 4. Evaluate the extracted data

Some of it rocks!

Step 5. Publish the linked open data

One big Turtle file

<http://biblio.uottawa.ca/en/morisset-library> md:item ( [ a schema:Library ;
        schema:branchOf [ a schema:Library ;
                schema:address [ a schema:PostalAddress ;
                        schema:addressCountry "Canada"@en ;
                        schema:addressLocality "Ottawa"@en ;
                        schema:addressRegion "ON"@en ;
                        schema:postalCode "K1N 6N5"@en ;
                        schema:streetAddress "65 University Private"@en ] ;
                schema:alternateName "Bibliothèque Morisset"@en ;
                schema:geo [ a schema:GeoCoordinates ;
                        schema:latitude "45.423241"@en ;
                        schema:longitude "-75.684126"@en ] ;
                schema:name "Morisset Library"@en ;
                schema:url <https://biblio.uottawa.ca/en/morisset-library> ] ;
        schema:name "uOttawa Library"@en ] ) ;
    rdfa:usesVocabulary schema: .

The picture in Canada

schema.org/Library Usage: 3 libraries--one of which is an archives
schema.org/LocalBusiness Usage: 47 statements
schema.org/Organization : Usage: 15 statements
schema.org/Event : Usage: 44 statements
schema.org/PostalAddress : Usage: 103 statements

This should be in our wheelhouse!

Why should you care--really?

Visibility and accuracy in common search engines
Ease of building lightweight applications
General linked open data use cases

Why does this matter, beyond the unfortunately abstract "linked open data enables others to build exciting new services!"? People increasingly search on mobile devices, meaning that they're getting funneled through Google, Bing, and Yahoo, and when they search for their local library they should get accurate information back.

Unfortunately, resources like Google Maps, Bing Maps, Yahoo Local, etc are a hodge-podge of data culled from social media properties like Google+ and FourSquare, drive-by edits, and attempted parses of the human-oriented markup on pages.

Like it or not, this data is effectively considered authoritative; if Google Maps has the right name and location, but says your library is closed because it couldn't figure out the difference between your summer hours and your fall hours, then many people aren't going to make the extra effort to click through to double-check on your web page.

But we also want this data to be available to support applications that we haven't envisioned yet.

And we want to be present in the web of data.

What can you do?

Add linked data to your library home page
Help manage the list of libraries
Help build dialled.ca

Adding linked data

Scott, D. (2015). White Hat Search Engine Optimization (SEO): Structured Web Data for Libraries. Partnership: The Canadian Journal of Library and Information Practice and Research, 10(1). Retrieved from https://journal.lib.uoguelph.ca/index.php/perj/article/view/3328

Web page before linked data

Screenshot of the Library One-Pager template for library websites

Web page after linked data

Education by demonstration!

An augmented Library One-Pager
Live at dialled.ca/one_pager
git branch at github.com/dbs/One-Pager/tree/schema.org

Standard RDFa tactics

@@ -14,14 +14,14 @@
        <link rel="stylesheet" href="deps/ie.css">
        <script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script>
        <![endif]-->
-</head><body><div class="page">
+</head><body vocab="http://schema.org/" typeof="Library"><div class="page">
 
 <header>
        <nav>
                <a href="#">Log in/Sign up</a>
                <a href="#footer">Location/Hours</a>
        </nav>
-       <h1><a href="#">Niceville Library</a></h1>
+       <h1 property="name"><a href="#">Niceville Library</a></h1>
        <form id="search"><fieldset><legend>Search Catalog</legend>
                <label for="searchText"><a href="#">Catalog</a></label><input id="searchText" placeholder="Search books and more" type="search">
                <input type="submit" value="Search">

Reality check

How many of you use a static HTML library web site?
How many use a CMS like Drupal?
How many of you want a deep knowledge of RDFa, microdata, or JSON-LD and HTML to be the minimum bar for someone to change the opening hours on your website?

Drupal One-Pager

A theme with user-friendly widgets that hide any implementation details.
So easy, a University Librarian could use it!™

I've also taken that One-Pager and converted it into a Drupal theme, with a handful of associated modules (as is the Drupal way), that builds on standard modules such as Opening Hours and Addressfield to create a schema.org-enhanced website. My goal was to prove that it should be possible to build a CMS template based on something reasonably rational that normal humans can use to set opening hours, contact information, etc, without having to even think about what's happening with schema.org under the covers; it should just work.

And it does. It's not perfect; a Drupal expert could probably clean up a lot of the rough edges pretty quickly, and all of the "tailor to your specific users and environment" warnings apply, but as a proof of concept I think it's a damn fine start.

Location and contact info

Screenshot of standard Drupal form for entering telephone number, email, and address via the Addressfield widget

This really isn't hard!

NO WEBSITE REDESIGN NECESSARY
Nobody needs to know except you and the world of linked open data

How to help dialled.ca directly

Ensure your own library is listed, with an accurate URL: see https://goo.gl/ZpsB8M
Create and maintain machine-readable lists
Write HTML scrapers for existing human-oriented lists

The future of dialled.ca

Generate periodic data dumps and statistics of the crawled data
Normalize the data across different vocabularies
Add more data from more sources (ILL symbols, etc)
Integrate checks and updates via external lists
Offer a dynamic web interface for querying the existing data
Enhance the dialled.ca user experience
Contributions welcome at https://github.com/dbs/dialled-crawler