9
edits
Changes
no edit summary
The resulting functionality is in production at http://franklin.library.upenn.edu/. It is still under heavy development, and questions/comments/criticism would be welcome. The source code has not been released open source, but hopefully that will change in the near future.
==Queue Programming -- how using job queues can make the Library coding world a better place==
*Birkin James Diana, Brown University (birkin_diana@brown.edu)
**I've given one or two C4L 20-minute talks and a few lightning ones over the years
In 2007 we built a system that dumped certain user web-requests for books into a database for offline-processing triggered via cron. We wanted to make the magic happen live, but knew it would take too long. Thus we created, sort of accidentally, a kind of old-fashioned static procedural job queue.
Over the years we we've been repeatedly impressed with how useful and robust this unintended architecture has been, and it fostered thinking about using real job queues in Library workflows.
Fast-forward to the present. We now are using _real_ job queueing, in production, for parts of the functioning of Brown Digital Repository. We've also used it for ingestion scripts, and plan to move more lots more code to this architecture.
I'd like to share & show:
* our lightweight rq/redis job queueing setup
* how using job queues can speed up workflows via using multiple workers
* how job queueing can make workflows more robust, especially by simplifying failure handling
* a way we've smoothly avoided race-conditions that can occur in concurrent-programming
* a technique for using task-processing job queues to simplify complex workflows
rq: http://python-rq.org
redis (python): https://pypi.python.org/pypi/redis/
[[:Category:Code4Lib2014]]
