Code4Lib - User contributions [en]

ServiceResponse data structures and generation

2011-01-24T00:33:13Z

68.50.223.109: /* Statelessness */

Part of [[Umlaut Technical Overview]]

==ServiceResponse and related data structures==

Before talking about how the services generate data, we should talk about the data format of a [http://umlaut.rubyforge.org/api/classes/ServiceResponse.html ServiceResponse]. A ServiceResponse is basically a unit of information generated by a Service, generally for display somewhere on the link resolver menu page. For example, there might be a ServiceResponse representing a fulltext link, a help link, or an abstract. ServiceResponses almost always link out somewhere, along with providing other data for display.

The ServiceResponse entity has a few 'standard' properties (display_text, url, notes), but also a property, service_data, consisting of a serialized hash for holding arbitrary key/value information. Different service types might require different key/values here. The [] operator on ServiceResponse conveniently allows you to store arbitrary key/value information in this property (and also access/set the 'built-in' properties). For appropriately loose coupling between data stored, service generating it, and view, we define some conventions for what key/value pairs are used for what purposes in each response type, in comments on [http://umlaut.rubyforge.org/api/classes/ServiceResponse.html ServiceResponse class definition].

A ServiceResponse also records which Service generated it, using that Service's service_id/name as defined in config/umlaut_config/services.yml, and generally retrievable from the ServiceList.

So what do we mean by a 'service type'? The list of all valid service types is defined in the ServiceTypeValue table. Each ServiceTypeValue has a one-word internal identifier token (name), a display_name for user presentation, and optionally a display_name_pluralized (to over-ride standard Rails pluralization). The values in this table are initialized from db/orig_fixed_data/service_type_values.yml when you run rake umlautdb:load_initial_data. We intend the local implementer to be able to create locally defined ServiceTypeValues too, if necessary.

ServiceTypeValue uses the acts_as_enumerated plug-in to conveniently allow the developer to refer to an individual ServiceTypeValue by name: ServiceTypeValue[:fulltext] ==> the ServiceTypeValue object with name == 'fulltext'. acts_as_plugin does efficient caching.

So obviously which ServiceTypeValue a given ServiceResponse is intended for needs to be registered somewhere. But you won't find it in ServiceResponse, which might be confusing at first. In fact, there's a somewhat confusingly named three-way join object called ServiceType, which ties together:
* a ServiceResponse
* a ServiceTypeValue
* a Request

This architecture theoretically allows:
* One ServiceResponse to belong to multiple Requests (ServiceResponse cacheing accross requests/sessions).
* One ServiceResponse to be assigned ''multiple'' ServiceTypeValues and thus listed multiple times with a given Request.

In fact, Umlaut does not currently use ServiceResponse caching across requests; it turned out to be tricky to get right without clear gain. And very few (if any?) current services register the same ServiceResponse to a request with multiple ServiceTypeValues. But, the architecture is there to support it if needed in the future.

This data structure architecture ends up somewhat confusing (and ServiceType is probably not a clear name for that three-way join) but there are usually convenience methods defined to avoid the complexity; they should be used. See for example (tbd).

[http://bibwild.wordpress.com/files/2008/02/umlaut-serviceresponse.jpg Data structure diagram] Trying to figure out how to make this display inline, sorry.

==Obligations of Service logic==

What you need to know to write a new Service. How to generate data, and callback methods service logic can or must provide.

Recall that an umlaut "service" is defined in config/umlaut_config/services.yml to be a particular class holding the service logic, and some configuration parameters.

That class holding the service logic is called a "service adaptor", or somewhat ambiguously, sometimes times just a "service". Service adaptors live in lib/service_adaptors, and extend [http://umlaut.rubyforge.org/api/classes/Service.html Service].

===Statelessness===

(Not sure this is actually true any longer, working to make it not so. jrochkind jan 2010). Service logic should generally be written to be state-less. The same Service object, defined in services.yml, is initialized once and generally re-used for the life of an application instance (cached by ServiceList). So any state you store can end up persisting from request to request and session to session, which you probably don't intend. Umlaut architecture for background services also involves threads and forks, and while there's normally no reason a given service object would be in two threads simultaneously, better safe than sorry. It's safest to store no non-universal state in the service object.

===Disclosure methods===

A service adaptor must define [http://umlaut.rubyforge.org/api/classes/Service.html#service_types_generated service_types_generated()] to return an Array of ServiceTypeValues constituting the types of ServiceResponses the service

A service adaptor may optionally list some required configuration params. If they are not supplied, an exception will be thrown when the service is initialized from services.yml. eg:
: required_config_params :api_key, :base_url

===The handle method===

The heart of a typical service is in implementing the [http://umlaut.rubyforge.org/api/classes/Service.html#handle handle] method. When Umlaut wants a service to do it's thing, Umlaut will pass the request in, and it's up to the Service to do it's work. Note that while the individual umlaut request is passed into handle() as an argument for legacy purposes, it's also available from Service#request at any time, the argument isn't really necessary.

The service can examine all metadata from the request, and even examine ServiceResponses generated by other services, and the status of other services in progress or finished. (See [http://umlaut.rubyforge.org/api/classes/Request.html Request#dispatched_services], Request#dispatched, Request#services_in_progress, etc.)

The service can then enhance any metadata if desired (likely data in [http://umlaut.rubyforge.org/api/classes/Referent.html Referent], from Request#referent).

The service can create one or more ServiceResponses. A ServiceResponse normally represents a discrete package of data that will be displayed on some part of the resolve menu. ServiceResponses should generally be created with the convenience method [http://umlaut.rubyforge.org/api/classes/Request.html Request]#add_service_response.

The add_service_response call has you specify the particular service class, the ServiceTypeValue, and arbitrary key value pairs as appropriate/conventional for the given time. For conventions on these key/value pairs, see [http://umlaut.rubyforge.org/api/classes/ServiceResponse.html ServiceResponse]. For example:

<pre>
request.add_service_response(:service=>self,
:service_type_value => :cover_image,
:display_text => 'Cover Image',
:key=>size,
:url => img.inner_html,
:asin => asin,
:size => size )
</pre>

The Service code is also responsible for registering a DispatchedService object with the completion state of the service. This should be done with the convenience method [http://umlaut.rubyforge.org/api/classes/Request.html Request]#dispatched . If the service throws an uncaught exception, Umlaut itself will register a DispatchedService with status FailedFatal. But otherwise, the service is responsible for registering a completion status, or Umlaut may not realize the service is complete and continue running it over and over again, or reporting it as timed out.

===callback methods===

The Service can play an interactive role with the view elements of Umlaut in determining how to display the ServiceResponse and how to generate an external url for it if the user clicks on it. A Service doesn't need to do this--it can simply include properties in generated ServiceResponses for the conventional keys mentioned in [http://umlaut.rubyforge.org/api/classes/ServiceResponse.html ServiceResponse], including a pre-generated url in the :url property.

However, for more complicated processing (including not generating urls until the point-of-need when a user actually clicks on one), callback methods can instead be implemented.

These callback methods include [http://umlaut.rubyforge.org/api/classes/Service.html]#view_data_from_service_type(service_type_obj) , #to_[name of service goes here] (eg #to_fulltext, or #to_help ), Service#response_to_view_data, and Service#response_url.

For more information, see the Technical Overview section on view logic. (tbd).

==Alternate Service Tasks==

Services were originally designed to do one thing, as described above. However, it has been useful to use the service architecture to perform other 'tasks' too, basically other sorts of plug-ins. What plug-in 'task' a service will be called upon to perform depends on the task config property in services.yml, which defaults to 'standard' when empty.

The other service task we have defined currently as 'link_out_filter'. A task:link_out_filter service will never have it's handle method called. Instead, it will have a [http://umlaut.rubyforge.org/api/classes/Service.html Service#]link_out_filter method defined, and called at the appropriate control point. Examples of link_out_filter services are ezproxy, and sfx_backchannel_record.

[[Category: Umlaut]]

Umlaut wishlist

2010-10-05T01:45:47Z

68.50.223.109:

[[Category:Umlaut]]

Some actual current future plans:

* Fix HathiTrust adapter to use new HT plugin, including not showing fulltext for just portions of a serial. (Or showing it in 'see also' section only)

* Rails3

* Internet Archive -- use new OL/IA api, discover search-inside-the-book.

* WorldCat, use new api, link directly to nearest public library in 'see also' or elsewhere.

* CiteSeerX -- source of 'cited by' info, AND, most excitingly, open access pre-prints. But their Atom/RSS feeds (the only API I could find) don't seem to advertise enough info to actually use these features. Would need to talk to developer team -- possibly offer to help code? Also not entirely clear how big their corpus actually is, if it's worth it.

'''old''' Desired or planned features.

* Check for similar articles from: http://biosemantics.org/jane/faq.php#api

* Full-text availability check from http://chroniclingamerica.loc.gov/ -- check by title/city, check by lccn (?), able to check particular dates/link to particular dates and/or pages of paper?

* Allow a service_response to have a tree relationship to children, so for instance alternate versions of a text can be attached as children of the main link, expandable by the user.

* http://export.arxiv.org/api_help/ !!!!

* PubMed Central full text lookup http://www.ncbi.nlm.nih.gov/entrez/query/static/esearch_help.html (SFX may already do this?)

* Journal ToC from CiteULike

* Parsing of formatted references from an entry screen. Use http://wing.comp.nus.edu.sg/parsCit/ package. Very interesting! Or a similar UCOP package: http://purl.net/net/egh/hmm-citation-extractor/ See list of such packages here under "Other Parsing Tools" http://freecite.library.brown.edu/

* LibraryThing open knowledge API for more data. http://www.librarything.com/blog/2008/08/free-web-services-api-to-common.php

* Connect to internet linked movie database on movies: http://www.linkedmdb.org/

* Add information about the conversation happening around an article with Scintilla if we have a URL, PMID or DOI (Alf at Scintilla would prefer us NOT to use the API for high-traffic. But we can copy his techniques internally to Umlaut. CrossRef and PubMed for "cited by" on DOI and PMID identifiers are a good idea. He has also reverse engineered the Scopus javascript api to allow server-side json access. http://hublog.hubmed.org/archives/001512.html):
http://hublog.hubmed.org/archives/001609.html
Unofficially it will return json:
http://scintilla.nature.com/conversations?uri=info%3Adoi%2F10.1371%2Fjournal.pmed.0020124&format=json

* Rochester “Getting Users Fulltext” style code to skip right to the full text, skipping content-provider metadata pages.

* UMich Mirlyn for metadata enrichment?
http://webservices.itcs.umich.edu/mediawiki/MLibraryAPI/index.php/Mirlynapi:Home

* xISBN/thingISBN use. (Some thought is required in how to integrate this while avoiding false positives). Bowker ISSN service for metadata enhancement. OCLC xISSN? Integrate preceding/succeeding title information from OPAC or xISSN?

* LibraryLookup: http://xisbn.worldcat.org/liblook/index.htm At least until xISBN is baked in we could provide a link to this service. Increases the chances of finding a desired book in the catalog through work set grouping. Used by LibX.
http://xisbn.worldcat.org/liblook/resolve.htm?res_id=http://www.iucat.iu.edu&rft.isbn=0451530942&url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:book

* Journal covers from Ulrich's via screen-scraping (or Ulrich's/sersol built in api?)

* Connotea integration

* Fetch ToC from LC. Screen scrape, I guess? Or z3950? Any other content from LC?

* Link to Books In Print ala Notre Dame.
http://www.library.nd.edu/eresources/findit/findit.cgi?doc_num=001939269&aleph_session=U5AVHRXD5QB1CGDFDSVJ9DSY2UA6QNCGVEU8EYRX9NNMIQ429Q-54668%22
example

* bip search url? :
http://www.booksinprint.com/merge_shared/Search/advsearch.asp%3FdateState%3DY%26txtAction%3D%26BooleanSearch%3D%26SType%3Dadv%26collection%3DBIP%26QueryMode%3DSimple%26ResultCount%3D25%26ResultTemplate%3Dmbbookresult_fl.hts%26navPage%3D1%26SrchFrm%3DAdv%26ScoreThreshold%3D0%26Criteria1%3DISBN%26CriteriaText1%3D0838935370

* SFX plugin: Notice when first title given is non-roman, and look for roman title to enhance metadata with when so.

* HIP and other OPAC searchers should pull ToC from MARC 505 when present. And 856's judged to be ToC in ToC, not full text.

* Fix Umlaut Referent to more easily allow multiple authors. Architectural change neccessary to get a lot of this stuff working right.

* "Cited by" service. Scopus via screen scraping? (scopus javascript api? http://www.scopus.com/scsearchapi/ See also http://hublog.hubmed.org/archives/001512.html ) ISI Web of Science is too hard to even screen scrape the interface is such a mess, but Scopus looks do-able. Google scholar?

* Enhance metadata to have full metadata for a refworks etc export. Using: CrossRef? Metalib? Anything else?

* A general purpose responsecache. Schema: Date, service/source, key. Use for caching image urls, ToC urls from LC, etc.

* Fix Worldcat registry auto-discovery.

* Add a Worldcat search that uses API, instead of screen scrape.

* Switch OCA search to use OCA native APIs, instead of indexdata mirror index.

* fix unapi in umlaut. unapi to rsi? For zotero.

* Change background to use Spawn plugin instead of manual threading. Investigating using spawn with fork instead of thread (terry reese on limited pool of forks).

* Crazy idea for an abstract interface/architecture to support querying web service apis that require client side javascript, like Google Books and Scopus.

* Integrate my various local document delivery services into menu of options when full text isn’t available. More generally, a clear architecture for providing localized doc delivery services in addition to a single ILL link.

* SFX adaptor: Add a "rollup" feature that pays attention to dates to avoid eliminating coverage.

Umlaut wishlist

2010-10-05T01:40:00Z

68.50.223.109:

[[Category:Umlaut]]

Some actual current future plans:

* Fix HathiTrust adapter to use new HT plugin, including not showing fulltext for just portions of a serial. (Or showing it in 'see also' section only)

* Rails3

* Internet Archive -- use new OL/IA api, discover search-inside-the-book.

* WorldCat, use new api, link directly to nearest public library in 'see also' or elsewhere.

* CiteSeerX -- source of 'cited by' info, AND, most excitingly, open access pre-prints. But their Atom/RSS feeds (the only API I could find) don't seem to advertise enough info to actually use these features. Would need to talk to developer team -- possibly offer to help code?

'''old''' Desired or planned features.

* Check for similar articles from: http://biosemantics.org/jane/faq.php#api

* Full-text availability check from http://chroniclingamerica.loc.gov/ -- check by title/city, check by lccn (?), able to check particular dates/link to particular dates and/or pages of paper?

* Allow a service_response to have a tree relationship to children, so for instance alternate versions of a text can be attached as children of the main link, expandable by the user.

* http://export.arxiv.org/api_help/ !!!!

* PubMed Central full text lookup http://www.ncbi.nlm.nih.gov/entrez/query/static/esearch_help.html (SFX may already do this?)

* Journal ToC from CiteULike

* Parsing of formatted references from an entry screen. Use http://wing.comp.nus.edu.sg/parsCit/ package. Very interesting! Or a similar UCOP package: http://purl.net/net/egh/hmm-citation-extractor/ See list of such packages here under "Other Parsing Tools" http://freecite.library.brown.edu/

* LibraryThing open knowledge API for more data. http://www.librarything.com/blog/2008/08/free-web-services-api-to-common.php

* Connect to internet linked movie database on movies: http://www.linkedmdb.org/

* Add information about the conversation happening around an article with Scintilla if we have a URL, PMID or DOI (Alf at Scintilla would prefer us NOT to use the API for high-traffic. But we can copy his techniques internally to Umlaut. CrossRef and PubMed for "cited by" on DOI and PMID identifiers are a good idea. He has also reverse engineered the Scopus javascript api to allow server-side json access. http://hublog.hubmed.org/archives/001512.html):
http://hublog.hubmed.org/archives/001609.html
Unofficially it will return json:
http://scintilla.nature.com/conversations?uri=info%3Adoi%2F10.1371%2Fjournal.pmed.0020124&format=json

* Rochester “Getting Users Fulltext” style code to skip right to the full text, skipping content-provider metadata pages.

* UMich Mirlyn for metadata enrichment?
http://webservices.itcs.umich.edu/mediawiki/MLibraryAPI/index.php/Mirlynapi:Home

* xISBN/thingISBN use. (Some thought is required in how to integrate this while avoiding false positives). Bowker ISSN service for metadata enhancement. OCLC xISSN? Integrate preceding/succeeding title information from OPAC or xISSN?

* LibraryLookup: http://xisbn.worldcat.org/liblook/index.htm At least until xISBN is baked in we could provide a link to this service. Increases the chances of finding a desired book in the catalog through work set grouping. Used by LibX.
http://xisbn.worldcat.org/liblook/resolve.htm?res_id=http://www.iucat.iu.edu&rft.isbn=0451530942&url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:book

* Journal covers from Ulrich's via screen-scraping (or Ulrich's/sersol built in api?)

* Connotea integration

* Fetch ToC from LC. Screen scrape, I guess? Or z3950? Any other content from LC?

* Link to Books In Print ala Notre Dame.
http://www.library.nd.edu/eresources/findit/findit.cgi?doc_num=001939269&aleph_session=U5AVHRXD5QB1CGDFDSVJ9DSY2UA6QNCGVEU8EYRX9NNMIQ429Q-54668%22
example

* bip search url? :
http://www.booksinprint.com/merge_shared/Search/advsearch.asp%3FdateState%3DY%26txtAction%3D%26BooleanSearch%3D%26SType%3Dadv%26collection%3DBIP%26QueryMode%3DSimple%26ResultCount%3D25%26ResultTemplate%3Dmbbookresult_fl.hts%26navPage%3D1%26SrchFrm%3DAdv%26ScoreThreshold%3D0%26Criteria1%3DISBN%26CriteriaText1%3D0838935370

* SFX plugin: Notice when first title given is non-roman, and look for roman title to enhance metadata with when so.

* HIP and other OPAC searchers should pull ToC from MARC 505 when present. And 856's judged to be ToC in ToC, not full text.

* Fix Umlaut Referent to more easily allow multiple authors. Architectural change neccessary to get a lot of this stuff working right.

* "Cited by" service. Scopus via screen scraping? (scopus javascript api? http://www.scopus.com/scsearchapi/ See also http://hublog.hubmed.org/archives/001512.html ) ISI Web of Science is too hard to even screen scrape the interface is such a mess, but Scopus looks do-able. Google scholar?

* Enhance metadata to have full metadata for a refworks etc export. Using: CrossRef? Metalib? Anything else?

* A general purpose responsecache. Schema: Date, service/source, key. Use for caching image urls, ToC urls from LC, etc.

* Fix Worldcat registry auto-discovery.

* Add a Worldcat search that uses API, instead of screen scrape.

* Switch OCA search to use OCA native APIs, instead of indexdata mirror index.

* fix unapi in umlaut. unapi to rsi? For zotero.

* Change background to use Spawn plugin instead of manual threading. Investigating using spawn with fork instead of thread (terry reese on limited pool of forks).

* Crazy idea for an abstract interface/architecture to support querying web service apis that require client side javascript, like Google Books and Scopus.

* Integrate my various local document delivery services into menu of options when full text isn’t available. More generally, a clear architecture for providing localized doc delivery services in addition to a single ILL link.

* SFX adaptor: Add a "rollup" feature that pays attention to dates to avoid eliminating coverage.

2011 nominations list

2010-07-30T18:29:25Z

68.50.223.109:

'''This page is under development'''

Below is a list of nominations for invited speakers for Code4Lib 2011.

* [http://jfwilliams.com/ Joan Frye Williams] - a sharp thinker of an extremely practical bent, I would expect her to tell us the cold, hard facts as she sees them in libraries today, which I think is what we should be hearing. - Roy

* [http://managemetadata.org/blog/ Diane Hillman] -- has years of experience working with library metaata, starting with traditional cataloging but over the past many years moving into modern metadata for library applications. Involved in RDA schema modelling. Knows what modern metadata looks like, knows what our legacy data is like, has a lot of insight into where we need to go and how to get there -- and on how systems people can work with catalogers to do it. --jrochkind

[[Category: Code4Lib2011]]

2011 nominations list

2010-07-30T18:28:04Z

68.50.223.109: