This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Introducing OpenSRF
In the beginning there was OpenSRF - a scalable framework for making services written in multiple languages available across multiple servers, knit together via the fabric of XMPP, communicating in the common tongue of JSON.
srfsh is a convenient command-line tool for making OpenSRF requests.
# request <service> <method> <JSON> bash$ srfsh srfsh# request open-ils.cat open-ils.cat.marc_template.types.retrieve Received Data: [ "K_audio", "K_book", "K_video" ] ------------------------------------ Request Completed Successfully Request Time in seconds: 0.012795 ------------------------------------ srfsh# request open-ils.cat open-ils.cat.biblio.record.marc_cn.retrieve 3 Received Data: [ { "050":"3489 V.96" } ] ------------------------------------ Request Completed Successfully Request Time in seconds: 0.033568 ------------------------------------
JSON is great because there are libraries for any relevant programming language that speak JSON. Also, you generally don’t have to do the work of converting between JSON and the native types in the programming language; the Perl and C (and Python!) OpenSRF modules/libraries translate for you.
# Create a session to communicate to a service my $_search = OpenSRF::AppSession->create( 'open-ils.search' ); my $_storage = OpenSRF::AppSession->create( 'open-ils.cstore' ); # Set the search params for a particular library my $o_search = { shortname => uc($ou) }; # Issue a request for actor.org_unit data for that library my $one_org = $_storage->request( "open-ils.cstore.direct.actor.org_unit.search", $o_search )->gather(1); # Request a count of copies at that library for a given bib record my $count_req = $_search->request( 'open-ils.search.biblio.record.copy_count', $one_org->id, $bib ); # Issue the request and gather all results into an array my $copy_counts = $count_req->gather(1); # Iterate over the array of results and get the properties of each object foreach my $count (@$copy_counts) { my $num_transparent = $count->transparent; my $num_available = $count->available; my $depth = $count->depth; }
We can also speak OpenSRF via JavaScript over HTTP to the OpenSRF gateway or the OpenSRF HTTP Translator, but only to the services that are publicly exposed per opensrf_core.xml.
For more on OpenSRF, please refer to the Code4lib Journal article:
Introduction to Evergreen’s interface definition language (IDL)
Evergreen maintains a dictionary of the objects that it works with at the database layer in the fieldmapper IDL — see /openils/conf/fm_IDL.xml. The IDL provides:
-
the class hint for each object, such as aou for objects corresponding to the actor.org_unit table
-
whether the data in the object is accessible via pcrud (no authentication required for public access), cstore (authentication required, full access to data service), and/or reporter-store (authentication required, accessible via the reporter service)
-
the primary, identifying field for a given object
-
all of the properties of an object, including the name and type and whether the value of the property is translatable
-
relationships (links) between this object and other objects
-
permissions required for access via pcrud for create, read, update, and delete operations
<class id="actscsf" controller="open-ils.cstore open-ils.pcrud" oils_obj:fieldmapper="actor::stat_cat_sip_fields" oils_persist:tablename="actor.stat_cat_sip_fields" reporter:label="SIP Statistical Category Field Identifier"> <fields oils_persist:primary="field"> <field reporter:label="Field Identifier" name="field" reporter:datatype="text" reporter:selector="name"/> <field reporter:label="Field Name" name="name" reporter:datatype="text"/> <field reporter:label="Exclusive?" name="one_only" reporter:datatype="bool"/> </fields> <links/> <permacrud xmlns="http://open-ils.org/spec/opensrf/IDL/permacrud/v1"> <actions> <create permission="CREATE_PATRON_STAT_CAT" global_required="true"/> <retrieve /> <update permission="UPDATE_PATRON_STAT_CAT" global_required="true"/> <delete permission="DELETE_PATRON_STAT_CAT" global_required="true"/> </actions> </permacrud> </class>
evergreen=# \d actor.stat_cat_sip_fields
Table "actor.stat_cat_sip_fields"
Column | Type | Modifiers
----------+--------------+------------------------
field | character(2) | not null
name | text | not null
one_only | boolean | not null default false
Indexes:
"stat_cat_sip_fields_pkey" PRIMARY KEY, btree (field)
Referenced by:
TABLE "actor.stat_cat" CONSTRAINT "stat_cat_sip_field_fkey"
FOREIGN KEY (sip_field) REFERENCES actor.stat_cat_sip_fields(field)
ON UPDATE CASCADE ON DELETE SET NULL DEFERRABLE INITIALLY DEFERRED
As you can see in this simple example, each column in the actor.stat_cat_sip_fields table maps directly to a <field> element in the IDL, with attributes providing hints useful to the reporter interface. The open-ils.pcrud service uses the permissions in the <permacrud> element to limit access to the table data to those users with matching permissions.
Retrieving data in Perl via CStoreEditor: base table queries
The majority of Evergreen’s logic is written in Perl, and naturally there are convenience classes and methods available for common operations. The CStoreEditor class provides a set of methods for retrieving data by object ID and by simple attribute queries. When you request an object by ID, you get the instance of the object directly (assuming that an object with that ID exists).
# Get an instance of the CStoreEditor my $e = $self->editor; # Retrieve an acq.claim object by ID my $claim = $e->retrieve_acq_claim($claim_id);
CStoreEditor also makes it easy to search by your choice of attributes of the targeted objects. The result of a search is a reference to an array of results.
# Get an instance of the CStoreEditor my $e = $self->editor; # Search the actor.org_unit table for the top of the hierarchy (no parent) # Equates to "SELECT * FROM actor.org_unit WHERE parent_ou IS NULL" my $tree = $e->search_actor_org_unit([ { parent_ou => undef }, })->[0];
Fleshing results returns linked objects at the indicated depth (flesh) that have been fleshed out with the designated fields (flesh_fields). You can also use the order_by property to define how the results are to be sorted.
# Get an instance of the CStoreEditor my $e = $self->editor; # In this case, the related objects are other actor.org_unit objects; # effectively a self-join my $tree = $e->search_actor_org_unit([ { parent_ou => undef }, { flesh => -1, flesh_fields => {aou => ['children']}, order_by => {aou => 'name'} } ])->[0];
Retrieving data via JSON queries
CStoreEditor and the open-ils.cstore service also support complex free-form queries via JSON through the json_query method. For complete details, see the excellent JSON query tutorial; just bear in mind that you have to translate the JSON syntax to Perl or the language you are working in. For example, in Perl you use undef to represent a JSON null value, and you use the => operator to map a key to a value.
Note that if you find yourself writing ridiculously complex JSON queries, you might want to consider simply defining a view in the IDL instead and referring to that as a virtual object for read-only purposes. That way, you get to control the underlying SQL precisely and can avoid having to interpret how JSON will get mapped to the actual SELECT statement.
my $query = { select => { acp => ['id', 'barcode', 'circ_lib', 'create_date', 'age_protect', 'holdable'], acpl => [ {column => 'name', alias => 'copy_location'}, {column => 'holdable', alias => 'location_holdable'} ], ccs => [ {column => 'name', alias => 'copy_status'}, {column => 'holdable', alias => 'status_holdable'} ], acn => [ {column => 'label', alias => 'call_number_label'}, {column => 'id', alias => 'call_number'} ], circ => ['due_date'], }, from => { acp => { acn => { join => {bre => {filter => {id => $rec_id }}}, filter => {deleted => 'f'} }, circ => { # If the copy is circulating, retrieve the open circ type => 'left', filter => {checkin_time => undef} }, acpl => {}, ccs => {}, aou => {} } }, where => {'+acp' => {deleted => 'f' }}, order_by => [ {class => 'aou', field => 'name'}, {class => 'acn', field => 'label'} ], limit => $copy_limit, offset => $copy_offset }; my $cstore = OpenSRF::AppSession->create('open-ils.cstore'); my $copy_rec = $cstore->request( 'open-ils.cstore.json_query.atomic', $query );
You can see that, just like SQL, the JSON query language includes select, from, where, order_by, limit, and offset clauses. It is a common pattern in Evergreen to build up JSON queries gradually or to adjust clauses programatically. The structure itself is nothing more than a simple Perl hash.
Database functions
A trend in Evergreen development is to reduce network communication overhead by pushing more of the logic closer to the data. The page rendering delays associated with the traditional catalogue’s heavy use of AJAX and corresponding round trips from browser to server are being replaced by the Template Toolkit catalogue’s (TPAC) delivery of the bulk of the content to the browser in a single request. Network overhead with the TPAC model is reduced drastically; the Apache server still makes a multiple API requests to assemble the data for a single page, but most of those requests are to servers that are located on the same rack.
A logical progression of this trend has been to push business logic directly into the database via the use of database functions. A example relevant to TPAC development is the set of in-database unAPI functions. The unapi schema defines a number of functions corresponding to IDL objects meant to facilitate the retrieval of those objects in an XML format, including:
-
unapi.bre - bibliographic records
-
unapi.acn - call numbers
-
unapi.acp - copies
-
unapi.aou - organizational units / libraries
evergreen=# SELECT * FROM unapi.aou (4, NULL, 'circlib', NULL, NULL, NULL, NULL); --------------------------------------------------- <circlib xmlns="http://open-ils.org/spec/actors/v1" ident="4">Example Branch 1</circlib> (1 row)
On their own, these functions do little more than wrap existing data in an XML format that mirrors the unAPI methods previously implemented in Perl by the open-ils.supercat service. However, the formats can be combined to return a complete XML document containing all of the relevant bibliographic metadata required to render a single record in a catalogue—or indeed, a complete set of search results—with all calls happening directly on the database server.
Following is a realistic invocation of the unapi.bre function which demonstrates the ability to return the bibliographic record and related holdings information including call numbers with prefixes and suffixes:
evergreen=# SELECT * FROM unapi.bre( '3', -- <1> 'marcxml', -- <2> 'record', -- <3> '{holdings_xml,mra,acp,acnp,acns}', -- <4> 'CONS', -- <5> '0', '5' -- <6> ) AS "unapi.bre" ;
-
Object identifier
-
Requested format of the output
-
Root element name
-
Other XML objects to embed within the returned object:
-
holdings_xml = call numbers, copies, located URIs
-
mra = metabib record attributes
-
acp = copies
-
acnp = call number prefixes
-
acns = call number suffixes
-
-
Search scope for this request
-
Paging variables: begin at page 0, showing 5 results per page (focusing on copies in this case)
The following results of the call have been formatted and trimmed for the sake of formatting and brevity—but the amount of detail that is available to a caller should be evident.
unapi.bre ---------------------------------------------------------------------------- <record> <leader>01221njm a2200289 a 4500</leader> <controlfield tag="001">3</controlfield> <controlfield tag="003">CONS</controlfield> <datafield tag="245" ind1="0" ind2="0"> <subfield code="a">Latin quarter /</subfield> <subfield code="c">with Dave Samuels</subfield> </datafield> <attributes xmlns="http://open-ils.org/spec/indexing/v1" id="tag:open-ils.org:U2@mra/3" record="tag:open-ils.org:U2@bre/3"> <field name="ctry" filter="true" sorter="false">inu</field> <field name="mrec" filter="true" sorter="false"></field> <field name="date1" filter="true" sorter="false">2001</field> <field name="date2" filter="true" sorter="false"></field> <field name="pubdate" filter="false" sorter="true">2001</field> <field name="cat_form" filter="true" sorter="false">a</field> <field name="language" filter="true" sorter="false">eng</field> </attributes> <holdings xmlns="http://open-ils.org/spec/holdings/v1"> <counts> <count type="public" depth="0" org_unit="1" transcendant="0" available="2" count="2" unshadow="2"/> <count type="staff" depth="0" org_unit="1" transcendant="0" available="2" count="2" unshadow="2"/> </counts> <volumes> <volume id="tag:open-ils.org:U2@acn/100" lib="BR2" opac_visible="true" deleted="false" label="FAKE 3" label_sortkey="FAKE 3" label_class="1" record="3"> <owning_lib xmlns="http://open-ils.org/spec/actors/v1" id="tag:open-ils.org:U2@aou/5" shortname="BR2" name="Example Branch 2" opac_visible="true"/> <copies> <copy id="tag:open-ils.org:U2@acp/100" create_date="2011-12-05T12:47:11.451332-05:00" edit_date="2011-12-05T12:47:11.451332-05:00" circulate="true" deposit="false" ref="false" holdable="true" deleted="false" deposit_amount="0.00" barcode="FAKE 100" opac_visible="true"> <status ident="0" holdable="true" opac_visible="true">Available</status> <location ident="1" holdable="true" opac_visible="true">Stacks</location> <circ_lib xmlns="http://open-ils.org/spec/actors/v1" id="tag:open-ils.org:U2@aou/5" shortname="BR2" name="Example Branch 2" opac_visible="true"/> <circlib xmlns="http://open-ils.org/spec/actors/v1" ident="5">Example Branch 2</circlib> <copy_notes/> <statcats/> <foreign_records/> </copy> </copies> <uris/> <call_number_prefix ident="-1" label="" label_sortkey=""/> <call_number_suffix ident="1" label="REFERENCE" label_sortkey="reference"/> </volume> </volumes> <foreign_copies/> </holdings> </record>