Introducing OpenSRF

In the beginning there was OpenSRF - a scalable framework for making services written in multiple languages available across multiple servers, knit together via the fabric of XMPP, communicating in the common tongue of JSON.

srfsh is a convenient command-line tool for making OpenSRF requests.

A basic OpenSRF request
# request <service> <method> <JSON>

bash$ srfsh
srfsh# request

Received Data: [

Request Completed Successfully
Request Time in seconds: 0.012795

srfsh# request 3

Received Data: [
    "050":"3489 V.96"

Request Completed Successfully
Request Time in seconds: 0.033568

JSON is great because there are libraries for any relevant programming language that speak JSON. Also, you generally don’t have to do the work of converting between JSON and the native types in the programming language; the Perl and C (and Python!) OpenSRF modules/libraries translate for you.

OpenSRF requests in Perl (adapted from
# Create a session to communicate to a service
my $_search = OpenSRF::AppSession->create( '' );
my $_storage = OpenSRF::AppSession->create( 'open-ils.cstore' );

# Set the search params for a particular library
my $o_search = { shortname => uc($ou) };

# Issue a request for actor.org_unit data for that library
my $one_org = $_storage->request(

# Request a count of copies at that library for a given bib record
my $count_req = $_search->request(
    '', $one_org->id, $bib

# Issue the request and gather all results into an array
my $copy_counts = $count_req->gather(1);

# Iterate over the array of results and get the properties of each object
foreach my $count (@$copy_counts) {
    my $num_transparent = $count->transparent;
    my $num_available = $count->available;
    my $depth = $count->depth;

We can also speak OpenSRF via JavaScript over HTTP to the OpenSRF gateway or the OpenSRF HTTP Translator, but only to the services that are publicly exposed per opensrf_core.xml.

For more on OpenSRF, please refer to the Code4lib Journal article:

Introduction to Evergreen’s interface definition language (IDL)

Evergreen maintains a dictionary of the objects that it works with at the database layer in the fieldmapper IDL — see /openils/conf/fm_IDL.xml. The IDL provides:

  • the class hint for each object, such as aou for objects corresponding to the actor.org_unit table

  • whether the data in the object is accessible via pcrud (no authentication required for public access), cstore (authentication required, full access to data service), and/or reporter-store (authentication required, accessible via the reporter service)

  • the primary, identifying field for a given object

  • all of the properties of an object, including the name and type and whether the value of the property is translatable

  • relationships (links) between this object and other objects

  • permissions required for access via pcrud for create, read, update, and delete operations

IDL definition for the SIP Statistical Category Field Identifier
<class id="actscsf" controller="open-ils.cstore open-ils.pcrud"
        reporter:label="SIP Statistical Category Field Identifier">
    <fields oils_persist:primary="field">
        <field reporter:label="Field Identifier" name="field"
            reporter:datatype="text" reporter:selector="name"/>
        <field reporter:label="Field Name" name="name"
        <field reporter:label="Exclusive?" name="one_only"
    <permacrud xmlns="">
            <create permission="CREATE_PATRON_STAT_CAT" global_required="true"/>
            <retrieve />
            <update permission="UPDATE_PATRON_STAT_CAT" global_required="true"/>
            <delete permission="DELETE_PATRON_STAT_CAT" global_required="true"/>
Database table underlying the SIP Statistical Category Field Identifier
evergreen=# \d actor.stat_cat_sip_fields
        Table "actor.stat_cat_sip_fields"
  Column  |     Type     |       Modifiers
 field    | character(2) | not null
 name     | text         | not null
 one_only | boolean      | not null default false
    "stat_cat_sip_fields_pkey" PRIMARY KEY, btree (field)
Referenced by:
    TABLE "actor.stat_cat" CONSTRAINT "stat_cat_sip_field_fkey"
        FOREIGN KEY (sip_field) REFERENCES actor.stat_cat_sip_fields(field)

As you can see in this simple example, each column in the actor.stat_cat_sip_fields table maps directly to a <field> element in the IDL, with attributes providing hints useful to the reporter interface. The open-ils.pcrud service uses the permissions in the <permacrud> element to limit access to the table data to those users with matching permissions.

Retrieving data in Perl via CStoreEditor: base table queries

The majority of Evergreen’s logic is written in Perl, and naturally there are convenience classes and methods available for common operations. The CStoreEditor class provides a set of methods for retrieving data by object ID and by simple attribute queries. When you request an object by ID, you get the instance of the object directly (assuming that an object with that ID exists).

Basic CStoreEditor retrieval requests: by ID
# Get an instance of the CStoreEditor
my $e = $self->editor;

# Retrieve an acq.claim object by ID
my $claim = $e->retrieve_acq_claim($claim_id);

CStoreEditor also makes it easy to search by your choice of attributes of the targeted objects. The result of a search is a reference to an array of results.

Basic CStoreEditor retrieval requests: search by attribute
# Get an instance of the CStoreEditor
my $e = $self->editor;

# Search the actor.org_unit table for the top of the hierarchy (no parent)
# Equates to "SELECT * FROM actor.org_unit WHERE parent_ou IS NULL"
my $tree = $e->search_actor_org_unit([
    { parent_ou => undef },

Fleshing results returns linked objects at the indicated depth (flesh) that have been fleshed out with the designated fields (flesh_fields). You can also use the order_by property to define how the results are to be sorted.

Basic CStoreEditor retrieval requests: search and flesh related objects
# Get an instance of the CStoreEditor
my $e = $self->editor;

# In this case, the related objects are other actor.org_unit objects;
# effectively a self-join
my $tree = $e->search_actor_org_unit([
    { parent_ou => undef },
    { flesh            => -1,
      flesh_fields    => {aou =>  ['children']},
      order_by        => {aou => 'name'}

Retrieving data via JSON queries

CStoreEditor and the open-ils.cstore service also support complex free-form queries via JSON through the json_query method. For complete details, see the excellent JSON query tutorial; just bear in mind that you have to translate the JSON syntax to Perl or the language you are working in. For example, in Perl you use undef to represent a JSON null value, and you use the => operator to map a key to a value.

Note that if you find yourself writing ridiculously complex JSON queries, you might want to consider simply defining a view in the IDL instead and referring to that as a virtual object for read-only purposes. That way, you get to control the underlying SQL precisely and can avoid having to interpret how JSON will get mapped to the actual SELECT statement.

A candidate ridiculously complex JSON query
my $query = {
    select => {
        acp => ['id', 'barcode', 'circ_lib', 'create_date', 'age_protect', 'holdable'],
        acpl => [
            {column => 'name', alias => 'copy_location'},
            {column => 'holdable', alias => 'location_holdable'}
        ccs => [
            {column => 'name', alias => 'copy_status'},
            {column => 'holdable', alias => 'status_holdable'}
        acn => [
            {column => 'label', alias => 'call_number_label'},
            {column => 'id', alias => 'call_number'}
        circ => ['due_date'],

    from => {
        acp => {
            acn => {
                join => {bre => {filter => {id => $rec_id }}},
                filter => {deleted => 'f'}
            circ => { # If the copy is circulating, retrieve the open circ
                type => 'left',
                filter => {checkin_time => undef}
            acpl => {},
            ccs => {},
            aou => {}

    where => {'+acp' => {deleted => 'f' }},

    order_by => [
        {class => 'aou', field => 'name'},
        {class => 'acn', field => 'label'}

    limit => $copy_limit,
    offset => $copy_offset

my $cstore = OpenSRF::AppSession->create('open-ils.cstore');
my $copy_rec = $cstore->request(

You can see that, just like SQL, the JSON query language includes select, from, where, order_by, limit, and offset clauses. It is a common pattern in Evergreen to build up JSON queries gradually or to adjust clauses programatically. The structure itself is nothing more than a simple Perl hash.

Database functions

A trend in Evergreen development is to reduce network communication overhead by pushing more of the logic closer to the data. The page rendering delays associated with the traditional catalogue’s heavy use of AJAX and corresponding round trips from browser to server are being replaced by the Template Toolkit catalogue’s (TPAC) delivery of the bulk of the content to the browser in a single request. Network overhead with the TPAC model is reduced drastically; the Apache server still makes a multiple API requests to assemble the data for a single page, but most of those requests are to servers that are located on the same rack.

A logical progression of this trend has been to push business logic directly into the database via the use of database functions. A example relevant to TPAC development is the set of in-database unAPI functions. The unapi schema defines a number of functions corresponding to IDL objects meant to facilitate the retrieval of those objects in an XML format, including:

  • unapi.bre - bibliographic records

  • unapi.acn - call numbers

  • unapi.acp - copies

  • unapi.aou - organizational units / libraries

Invoking the unapi.aou() database function
evergreen=# SELECT * FROM unapi.aou (4, NULL, 'circlib', NULL, NULL, NULL, NULL);
 <circlib xmlns=""
    ident="4">Example Branch 1</circlib>
(1 row)

On their own, these functions do little more than wrap existing data in an XML format that mirrors the unAPI methods previously implemented in Perl by the open-ils.supercat service. However, the formats can be combined to return a complete XML document containing all of the relevant bibliographic metadata required to render a single record in a catalogue—or indeed, a complete set of search results—with all calls happening directly on the database server.

Following is a realistic invocation of the unapi.bre function which demonstrates the ability to return the bibliographic record and related holdings information including call numbers with prefixes and suffixes:

Invoking the unapi.bre() database function

evergreen=# SELECT * FROM unapi.bre(
    '3',                                 -- <1>
    'marcxml',                           -- <2>
    'record',                            -- <3>
    '{holdings_xml,mra,acp,acnp,acns}',  -- <4>
    'CONS',                              -- <5>
    '0', '5'                             -- <6>
    ) AS "unapi.bre" ;
  1. Object identifier

  2. Requested format of the output

  3. Root element name

  4. Other XML objects to embed within the returned object:

    • holdings_xml = call numbers, copies, located URIs

    • mra = metabib record attributes

    • acp = copies

    • acnp = call number prefixes

    • acns = call number suffixes

  5. Search scope for this request

  6. Paging variables: begin at page 0, showing 5 results per page (focusing on copies in this case)

The following results of the call have been formatted and trimmed for the sake of formatting and brevity—but the amount of detail that is available to a caller should be evident.

Example results from the unapi.bre call
    <leader>01221njm a2200289 a 4500</leader>
    <controlfield tag="001">3</controlfield>
    <controlfield tag="003">CONS</controlfield>
    <datafield tag="245" ind1="0" ind2="0">
        <subfield code="a">Latin quarter /</subfield>
        <subfield code="c">with Dave Samuels</subfield>
    <attributes xmlns=""
      id="" record="">
        <field name="ctry" filter="true" sorter="false">inu</field>
        <field name="mrec" filter="true" sorter="false"></field>
        <field name="date1" filter="true" sorter="false">2001</field>
        <field name="date2" filter="true" sorter="false"></field>
        <field name="pubdate" filter="false" sorter="true">2001</field>
        <field name="cat_form" filter="true" sorter="false">a</field>
        <field name="language" filter="true" sorter="false">eng</field>
    <holdings xmlns="">
            <count type="public" depth="0" org_unit="1"
              transcendant="0" available="2" count="2" unshadow="2"/>
            <count type="staff" depth="0" org_unit="1" transcendant="0"
              available="2" count="2" unshadow="2"/>
            <volume id="" lib="BR2"
              opac_visible="true" deleted="false" label="FAKE 3"
              label_sortkey="FAKE 3" label_class="1" record="3">
                <owning_lib xmlns=""
                  id="" shortname="BR2"
                  name="Example Branch 2" opac_visible="true"/>
                    <copy id=""
                      circulate="true" deposit="false" ref="false"
                      holdable="true" deleted="false"
                      deposit_amount="0.00" barcode="FAKE 100"
                        <status ident="0" holdable="true"
                        <location ident="1" holdable="true"
                        <circ_lib xmlns=""
                          id="" shortname="BR2"
                          name="Example Branch 2" opac_visible="true"/>
                        <circlib xmlns=""
                          ident="5">Example Branch 2</circlib>
                <call_number_prefix ident="-1" label="" label_sortkey=""/>
                <call_number_suffix ident="1" label="REFERENCE" label_sortkey="reference"/>