21
edits
Changes
no edit summary
[[HAMR: Human/Authority Metadata Reconciliation]]
Initial design/prototype by: Sean Chen, Tim Donohue, Joshua Gomez, Ranti Junus, Ryan Scherle
A tool for a curator to determine whether the various fields of a metadata record are correct. Takes a metadata record, locates any identifiers (e.g., DOI, PMID). Retrieves a copy of the metadata record from an authoritative source (e.g., CrossRef, PubMed). Displays a human-readable page that compares fields in the initial record with fields in the authoritative record. Each field is color-coded based on how well it matches, so the curator can quickly identify discrepancies.
== UI Prototype (uses static data) ==
http://dl.dropbox.com/u/9074989/code4lib/unverified.html
* Dublin core (maybe qualified)
* framework that allows multiple authority sources
* NOT focusing on author names ([http://www.orcid.org/ ORCID] is already working on this), except the fact that they are strings, and we'll do basic string matching
* 1 to 1 matching. Even if you want to eventually match with multiple authorities, you'd only do one at a time
Possible authority sources:
* pubmed - PubMed** Sample pubmed query (in Java): [https://wiki.duraspace.org/display/DSPACE/PubMedPrefill-PubmedPrefillStep.java DSpace PubMedPrefillPubMedPrefillStep.java] (From [https://wiki.duraspace.org/display/DSPACE/PopulateMetadataFromPubMed Populate Metadata from PubMed])* crossref** See 'retrievePubmedXML()' in above java code for actual call to PubMed*** Mapping happens here: See [https://wiki.duraspace.org/display/DSPACE/PubMedPrefill-pmid+dim.xsl pmid-to-dim.xsl] for a sample XSLT crosswalk to translate PubMed format to a qualified dublin core (internal DSpace metadata format)** More examples of querying PubMed: http://www.my-whiteboard.com/how-to-automate-pubmed-search-using-perl-php-or-java/** Useful tool for finding PubMed IDs: http://www.ncbi.nlm.nih.gov/entrez/getids.cgi* CrossRef** simply send the DOI to crossref, and get JSON /XML back** example: * http://api.labs.crossref.org/10.1111/j.1558-5646.2009.00626.x.json*** http://api.labs.crossref.org/10.2307/1935157.xml*** [http://code.google.com/p/dryad/source/browse/trunk/dryad/dspace/modules/doi/dspace-doi-webapp/src/main/java/org/dspace/doi/DOIServlet.java java code that includes a lookup]** [http://labs.crossref.org/site/crossref_metadata_search.html Metadata Search] -- send a text query, receive a list of matching records** [http://labs.crossref.org/site/quick_and_dirty_api_guide.html OpenURL search]
* google scholar - does it have an API?
* [http://www.mendeley.com mendeley] - [http://dev.mendeley.com/ Mendeley API]
* [http://vivoweb.org/ vivo]
* [http://bibapp.org/ bibapp]
Thoughts / Questions:
* Is there a way to do most/all of this via Javascript/AJAX/JQuery? Could it be a simple Javascript framework you could "drop" into any metadata editing interface?
** Unfortunately, it seems this wouldn't work out. In order to perform querying of external authorities, they'd all need to support [http://en.wikipedia.org/wiki/JSON#JSONP JSONP] or similar (and they don't)
== Code ==
* [http://gitref.org/ quick reference for Git]
* [https://github.com/ryscher/hamr Ryan's really stupid scratch implementation]
=== Draft Matching Algorithm ===
<pre>
function compareRecords(localDubCore, authDubCore)
recordMatches = []
for each element-type:
loc = array of local values
auth = array of authority values
// arrays are actually lists of dictionaries
// a1
// 0 value="Benson, Arnold", match="", strength=""
// 1 value="Terrence, D.", match="a2[3]", strength="100%"
elementMatches = compareElements(loc, auth)
recordMatches.add(elementMatches)
function compareElements(loc, auth)
output = []
//nested loops run through values and assigns strongest matches to each element
for each element in loc
for each element in auth
strength = string distance between the two elements
if strength = 100
//if match is perfect go ahead pop each element and add their values to output array
//output array is also list of dictionaries
//0 loc="Hector", auth="Hector", strength="100"
//1 loc="Albert", auth="Alberto", strength="90"
if strength > auth element's current strength value
overwrite auth element's strength and match values
if strength > loc element's current strength value
overwrite loc element's strength and match values
//this second set of non-nested loops pull out the strongest matches
for each item in auth
//x = some arbitrary barrier for a decent enough match
if element strength > x AND if matching element is still in the loc list
pop each element and add their values to output array
for each item in loc
if element strength > x AND if matching element is still in the auth list
pop each element and add their values to output array
//now do cleanup and look for values that have no decent matches
for each element in loc
pop element and add to output array without match //x loc="Heyward", auth="", strength=""
for each element in auth
pop element and add to output array without match //x loc="", auth="Perry", strength=""
return output
</pre>
== Output Spec ==
* We will use a simple XML output consisting of paired (and possibly unpaired) values.
* The root element will contain an attribute signifying the source of the authority metadata.
* The <match> element will be used to pair values, with a strength attribute to signify the string distance.
* Within each match element will be exactly 2 metadata elements with attributes signifying the source of each value: either the local input or the remote authority data.
* An <nonmatch> element will be used for unpaired values.
=== Sample Output ===
<pre>
<hamr authority="PubMed">
<match strength="100%">
<creator src="input">Trojan, Tommy</creator>
<creator src="authority">Trojan, Tommy</creator>
</match>
<match strength="90%">
<title src="input">Great American Article</title>
<title src="authority">Great American Article, The</title>
</match>
<nonmatch>
<subject src="input">Medical Stuff</subject>
</nonmatch>
<nonmatch>
<type src="authority">text</type>
</nonmatch>
</hamr>
</pre>
== Need to do ==
# Implement metadata retrieval from authority ''(done for crossref in ryan's code)''
# Design structure of plugins
## crosswalk from authority format to simple dc
# Design matching algorithm
