Evergreen Development 101
=========================
Dan Scott <dscott@laurentian.ca>
December 2011

This work is licensed under a
http://creativecommons.org/licenses/by-sa/3.0/[Creative Commons
Attribution-ShareAlike 3.0 Unported License.]

Introducing OpenSRF
-------------------

In the beginning there was OpenSRF - a scalable framework for making services
written in multiple languages available across multiple servers, knit together
via the fabric of XMPP, communicating in the common tongue of JSON.

`srfsh` is a convenient command-line tool for making OpenSRF requests.

.A basic OpenSRF request
[source, bash]
------------------------------------------------------------------------------
# request <service> <method> <JSON>

bash$ srfsh
srfsh# request open-ils.cat open-ils.cat.marc_template.types.retrieve

Received Data: [
  "K_audio",
  "K_book",
  "K_video"
]

 ------------------------------------
Request Completed Successfully
Request Time in seconds: 0.012795
 ------------------------------------

srfsh# request open-ils.cat open-ils.cat.biblio.record.marc_cn.retrieve 3

Received Data: [
  {
    "050":"3489 V.96"
  }
]

 ------------------------------------
Request Completed Successfully
Request Time in seconds: 0.033568
 ------------------------------------
------------------------------------------------------------------------------

JSON is great because there are libraries for any relevant programming language
that speak JSON. Also, you generally don't have to do the work of converting
between JSON and the native types in the programming language; the Perl and
C (and Python!) OpenSRF modules/libraries translate for you.

.OpenSRF requests in Perl (adapted from SuperCat.pm)
[source,perl]
------------------------------------------------------------------------------
# Create a session to communicate to a service
my $_search = OpenSRF::AppSession->create( 'open-ils.search' );
my $_storage = OpenSRF::AppSession->create( 'open-ils.cstore' );

# Set the search params for a particular library
my $o_search = { shortname => uc($ou) };

# Issue a request for actor.org_unit data for that library
my $one_org = $_storage->request(
    "open-ils.cstore.direct.actor.org_unit.search",
    $o_search
)->gather(1);

# Request a count of copies at that library for a given bib record
my $count_req = $_search->request(
    'open-ils.search.biblio.record.copy_count', $one_org->id, $bib
);

# Issue the request and gather all results into an array
my $copy_counts = $count_req->gather(1);

# Iterate over the array of results and get the properties of each object
foreach my $count (@$copy_counts) {
    my $num_transparent = $count->transparent;
    my $num_available = $count->available;
    my $depth = $count->depth;
}
------------------------------------------------------------------------------

We can also speak OpenSRF via JavaScript over HTTP to the OpenSRF gateway or
the OpenSRF HTTP Translator, but only to the services that are publicly exposed
per `opensrf_core.xml`.

For more on OpenSRF, please refer to the http://journal.code4lib.org[Code4lib
Journal] article:

  * http://journal.code4lib.org/articles/3284[Easing gently into OpenSRF, part 1]
  * http://journal.code4lib.org/articles/3365[Easing gently into OpenSRF, part 2]

Introduction to Evergreen's interface definition language (IDL)
---------------------------------------------------------------
Evergreen maintains a dictionary of the objects that it works with at the
database layer in the fieldmapper IDL -- see `/openils/conf/fm_IDL.xml`. The
IDL provides:

  * the class hint for each object, such as `aou` for objects corresponding to the
    `actor.org_unit` table
  * whether the data in the object is accessible via `pcrud` (no authentication
    required for public access), `cstore` (authentication required, full
    access to data service), and/or `reporter-store` (authentication required,
    accessible via the reporter service)
  * the primary, identifying field for a given object
  * all of the properties of an object, including the name and type and whether
    the value of the property is translatable
  * relationships (links) between this object and other objects
  * permissions required for access via `pcrud` for create, read, update, and
    delete operations

.IDL definition for the SIP Statistical Category Field Identifier
[source,xml]
------------------------------------------------------------------------------
<class id="actscsf" controller="open-ils.cstore open-ils.pcrud" 
        oils_obj:fieldmapper="actor::stat_cat_sip_fields"
        oils_persist:tablename="actor.stat_cat_sip_fields"
        reporter:label="SIP Statistical Category Field Identifier">
    <fields oils_persist:primary="field">
        <field reporter:label="Field Identifier" name="field"
            reporter:datatype="text" reporter:selector="name"/>
        <field reporter:label="Field Name" name="name"
            reporter:datatype="text"/>
        <field reporter:label="Exclusive?" name="one_only"
            reporter:datatype="bool"/>
    </fields>
    <links/>
    <permacrud xmlns="http://open-ils.org/spec/opensrf/IDL/permacrud/v1">
        <actions>
            <create permission="CREATE_PATRON_STAT_CAT" global_required="true"/>
            <retrieve />
            <update permission="UPDATE_PATRON_STAT_CAT" global_required="true"/>
            <delete permission="DELETE_PATRON_STAT_CAT" global_required="true"/>
        </actions>
    </permacrud>
</class>
------------------------------------------------------------------------------

.Database table underlying the SIP Statistical Category Field Identifier
------------------------------------------------------------------------------
evergreen=# \d actor.stat_cat_sip_fields
        Table "actor.stat_cat_sip_fields"
  Column  |     Type     |       Modifiers        
----------+--------------+------------------------
 field    | character(2) | not null
 name     | text         | not null
 one_only | boolean      | not null default false
Indexes:
    "stat_cat_sip_fields_pkey" PRIMARY KEY, btree (field)
Referenced by:
    TABLE "actor.stat_cat" CONSTRAINT "stat_cat_sip_field_fkey"
        FOREIGN KEY (sip_field) REFERENCES actor.stat_cat_sip_fields(field)
        ON UPDATE CASCADE ON DELETE SET NULL DEFERRABLE INITIALLY DEFERRED
------------------------------------------------------------------------------

As you can see in this simple example, each column in the
`actor.stat_cat_sip_fields` table maps directly to a `<field>` element
in the IDL, with attributes providing hints useful to the reporter interface.
The `open-ils.pcrud` service uses the permissions in the `<permacrud>` element
to limit access to the table data to those users with matching permissions.

Retrieving data in Perl via CStoreEditor: base table queries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The majority of Evergreen's logic is written in Perl, and naturally there
are convenience classes and methods available for common operations. The
`CStoreEditor` class provides a set of methods for retrieving data by object ID
and by simple attribute queries. When you request an object by ID, you get the
instance of the object directly (assuming that an object with that ID exists).

.Basic CStoreEditor retrieval requests: by ID
[source,perl]
------------------------------------------------------------------------------
# Get an instance of the CStoreEditor
my $e = $self->editor;

# Retrieve an acq.claim object by ID
my $claim = $e->retrieve_acq_claim($claim_id);
------------------------------------------------------------------------------

`CStoreEditor` also makes it easy to search by your choice of attributes of the
targeted objects. The result of a search is a reference to an array of results.

.Basic CStoreEditor retrieval requests: search by attribute
[source,perl]
------------------------------------------------------------------------------
# Get an instance of the CStoreEditor
my $e = $self->editor;

# Search the actor.org_unit table for the top of the hierarchy (no parent)
# Equates to "SELECT * FROM actor.org_unit WHERE parent_ou IS NULL"
my $tree = $e->search_actor_org_unit([
    { parent_ou => undef },
})->[0];
------------------------------------------------------------------------------

'Fleshing' results returns linked objects at the indicated depth (`flesh`) that
have been fleshed out with the designated fields (`flesh_fields`). You can also
use the `order_by` property to define how the results are to be sorted.

.Basic CStoreEditor retrieval requests: search and flesh related objects
[source,perl]
------------------------------------------------------------------------------
# Get an instance of the CStoreEditor
my $e = $self->editor;

# In this case, the related objects are other actor.org_unit objects;
# effectively a self-join
my $tree = $e->search_actor_org_unit([
    { parent_ou => undef },
    { flesh            => -1,
      flesh_fields    => {aou =>  ['children']},
      order_by        => {aou => 'name'}
    }
])->[0];
------------------------------------------------------------------------------

Retrieving data via JSON queries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
`CStoreEditor` and the `open-ils.cstore` service also support complex free-form
queries via JSON through the `json_query` method. For complete details, see the
http://docs.evergreen-ils.org/2.0/draft/html/JSON_Queries.html[excellent JSON
query tutorial]; just bear in mind that you have to translate the JSON syntax
to Perl or the language you are working in. For example, in Perl you use
`undef` to represent a JSON `null` value, and you use the `=>` operator to
map a key to a value.

Note that if you find yourself writing ridiculously complex JSON queries, you
might want to consider simply defining a view in the IDL instead and referring
to that as a virtual object for read-only purposes. That way, you get to
control the underlying SQL precisely and can avoid having to interpret how
JSON will get mapped to the actual SELECT statement.

.A candidate ridiculously complex JSON query
[source,perl]
------------------------------------------------------------------------------
my $query = {
    select => {
        acp => ['id', 'barcode', 'circ_lib', 'create_date', 'age_protect', 'holdable'],
        acpl => [
            {column => 'name', alias => 'copy_location'},
            {column => 'holdable', alias => 'location_holdable'}
        ],
        ccs => [
            {column => 'name', alias => 'copy_status'},
            {column => 'holdable', alias => 'status_holdable'}
        ],
        acn => [
            {column => 'label', alias => 'call_number_label'},
            {column => 'id', alias => 'call_number'}
        ],
        circ => ['due_date'],
    },

    from => {
        acp => {
            acn => {
                join => {bre => {filter => {id => $rec_id }}},
                filter => {deleted => 'f'}
            },
            circ => { # If the copy is circulating, retrieve the open circ
                type => 'left',
                filter => {checkin_time => undef}
            },
            acpl => {},
            ccs => {},
            aou => {}
        }
    },

    where => {'+acp' => {deleted => 'f' }},

    order_by => [
        {class => 'aou', field => 'name'}, 
        {class => 'acn', field => 'label'}
    ],

    limit => $copy_limit,
    offset => $copy_offset
};

my $cstore = OpenSRF::AppSession->create('open-ils.cstore');
my $copy_rec = $cstore->request(
    'open-ils.cstore.json_query.atomic', 
    $query
);
------------------------------------------------------------------------------

You can see that, just like SQL, the JSON query language includes `select`,
`from`, `where`, `order_by`, `limit`, and `offset` clauses. It is a common
pattern in Evergreen to build up JSON queries gradually or to adjust clauses
programatically. The structure itself is nothing more than a simple Perl hash.

Database functions
------------------
A trend in Evergreen development is to reduce network communication overhead
by pushing more of the logic closer to the data. The page rendering delays
associated with the traditional catalogue's heavy use of AJAX and corresponding
round trips from browser to server are being replaced by the Template Toolkit
catalogue's ('TPAC') delivery of the bulk of the content to the browser in a
single request. Network overhead with the TPAC model is reduced drastically; 
the Apache server still makes a multiple API requests to assemble the data for
a single page, but most of those requests are to servers that are located on
the same rack.

A logical progression of this trend has been to push business logic directly
into the database via the use of database functions. A example relevant to
TPAC development is the set of in-database unAPI functions. The `unapi`
schema defines a number of functions corresponding to IDL objects meant to
facilitate the retrieval of those objects in an XML format, including:

  * `unapi.bre` - bibliographic records
  * `unapi.acn` - call numbers
  * `unapi.acp` - copies
  * `unapi.aou` - organizational units / libraries

.Invoking the unapi.aou() database function
[source,sql]
------------------------------------------------------------------------------
evergreen=# SELECT * FROM unapi.aou (4, NULL, 'circlib', NULL, NULL, NULL, NULL);
 ---------------------------------------------------
 <circlib xmlns="http://open-ils.org/spec/actors/v1" 
    ident="4">Example Branch 1</circlib>
(1 row)
------------------------------------------------------------------------------

On their own, these functions do little more than wrap existing data in an
XML format that mirrors the unAPI methods previously implemented in Perl by
the `open-ils.supercat` service. However, the formats can be combined to return
a complete XML document containing all of the relevant bibliographic metadata
required to render a single record in a catalogue--or indeed, a complete set of
search results--with all calls happening directly on the database server.

Following is a realistic invocation of the `unapi.bre` function which
demonstrates the ability to return the bibliographic record and related
holdings information including call numbers with prefixes and suffixes:

.Invoking the unapi.bre() database function
[source,sql]
------------------------------------------------------------------------------

evergreen=# SELECT * FROM unapi.bre( 
    '3',                                 -- <1>
    'marcxml',                           -- <2>
    'record',                            -- <3>
    '{holdings_xml,mra,acp,acnp,acns}',  -- <4>
    'CONS',                              -- <5>
    '0', '5'                             -- <6>
    ) AS "unapi.bre" ;
------------------------------------------------------------------------------

<1> Object identifier
<2> Requested format of the output
<3> Root element name
<4> Other XML objects to embed within the returned object:
  ** holdings_xml = call numbers, copies, located URIs
  ** mra = metabib record attributes
  ** acp = copies
  ** acnp = call number prefixes
  ** acns = call number suffixes
<5> Search scope for this request
<6> Paging variables: begin at page 0, showing 5 results per page (focusing
    on copies in this case)

The following results of the call have been formatted and trimmed for the
sake of formatting and brevity--but the amount of detail that is available
to a caller should be evident.

.Example results from the unapi.bre call
[source,xml]
------------------------------------------------------------------------------
                 unapi.bre
 ----------------------------------------------------------------------------
 <record>
    <leader>01221njm a2200289 a 4500</leader>
    <controlfield tag="001">3</controlfield>
    <controlfield tag="003">CONS</controlfield>
    <datafield tag="245" ind1="0" ind2="0">
        <subfield code="a">Latin quarter /</subfield>
        <subfield code="c">with Dave Samuels</subfield>
    </datafield>
    <attributes xmlns="http://open-ils.org/spec/indexing/v1"
      id="tag:open-ils.org:U2@mra/3" record="tag:open-ils.org:U2@bre/3">
        <field name="ctry" filter="true" sorter="false">inu</field>
        <field name="mrec" filter="true" sorter="false"></field>
        <field name="date1" filter="true" sorter="false">2001</field>
        <field name="date2" filter="true" sorter="false"></field>
        <field name="pubdate" filter="false" sorter="true">2001</field>
        <field name="cat_form" filter="true" sorter="false">a</field>
        <field name="language" filter="true" sorter="false">eng</field>
    </attributes>
    <holdings xmlns="http://open-ils.org/spec/holdings/v1">
        <counts>
            <count type="public" depth="0" org_unit="1"
              transcendant="0" available="2" count="2" unshadow="2"/>
            <count type="staff" depth="0" org_unit="1" transcendant="0"
              available="2" count="2" unshadow="2"/>
        </counts>
        <volumes>
            <volume id="tag:open-ils.org:U2@acn/100" lib="BR2"
              opac_visible="true" deleted="false" label="FAKE 3"
              label_sortkey="FAKE 3" label_class="1" record="3">
                <owning_lib xmlns="http://open-ils.org/spec/actors/v1"
                  id="tag:open-ils.org:U2@aou/5" shortname="BR2"
                  name="Example Branch 2" opac_visible="true"/>
                <copies>
                    <copy id="tag:open-ils.org:U2@acp/100"
                      create_date="2011-12-05T12:47:11.451332-05:00"
                      edit_date="2011-12-05T12:47:11.451332-05:00"
                      circulate="true" deposit="false" ref="false"
                      holdable="true" deleted="false"
                      deposit_amount="0.00" barcode="FAKE 100"
                      opac_visible="true">
                        <status ident="0" holdable="true"
                          opac_visible="true">Available</status>
                        <location ident="1" holdable="true"
                          opac_visible="true">Stacks</location>
                        <circ_lib xmlns="http://open-ils.org/spec/actors/v1"
                          id="tag:open-ils.org:U2@aou/5" shortname="BR2"
                          name="Example Branch 2" opac_visible="true"/>
                        <circlib xmlns="http://open-ils.org/spec/actors/v1"
                          ident="5">Example Branch 2</circlib>
                        <copy_notes/>
                        <statcats/>
                        <foreign_records/>
                    </copy>
                </copies>
                <uris/>
                <call_number_prefix ident="-1" label="" label_sortkey=""/>
                <call_number_suffix ident="1" label="REFERENCE" label_sortkey="reference"/>
            </volume>
        </volumes>
        <foreign_copies/>
    </holdings>
</record>
------------------------------------------------------------------------------