<h1 id='ri-daemon'>Update daemon</h1>

<toc/>


## Introduction

Update daemon keeps ACIS database tables up-to-date with the
metadata collections that an ACIS service uses.  I also
sometimes call it "RI daemon".  RI stands for RePEc-Index,
which is a historical name for the modules behind the
daemon.

The daemon reads data about documents, software components,
series, institutions from the collections' data files and
inserts it into corresponding tables.  If a record
disappears from the underlying metadata collection, RI
daemon deletes corresponding data from the tables.

It is a daemon because it does not do anything by its own
will.  It sits there and waits for somebody to send him a
request.  Update requests are basically, orders for work.
The only way the daemon communicates with the world is by
writing a log.


Sometimes ACIS itself sends an update request to the daemon.
Sometimes you might want to send an update request to the
daemon.



<h2 id='collection'>Metadata collections</h2>


Collections is how ACIS thinks about your metadata.  You may
have zero, one or more separate collections.  (If you have
zero, ACIS won't be of much use.)  Each collection has an
<b>identifier</b>, a <b>type</b> and is stored in data files
somewhere in the local filesystem.  These things are
specified in the collections' configuration (see below).

Collection identifiers must be unique for any particular
ACIS service installation.  "<co>ACIS</co>" is a reserved
collection identifier.

Collection type must be "<co>AMF</co>" if your collection
will consist of AMF files.  (For <a
href='http://repec.org/'>RePEc</a>'s ReDIF it is
"<co>RePEcRec</co>".)  Other collection types may be defined
in the future.

Collection consists of any number of data files with stable
names, grouped into any directory structure that fits.  The
directory structure can not contain circular symbolic links.
Update daemon must have file-system access to the files and
directories of the collection.  

Each file of a collection may contain zero, one or many
<b>data records</b>.  Each data record must have a globally
unique identifier.  Identifiers are treated in
case-insensitive manner.  Having been lowercased they still
must be unique.

If two or more records in a collection have the same
identifier, update daemon excludes them all from the ACIS
database.

All other records are processed and its data is saved to the
ACIS database.


For AMF collections only files with <co>.amf.xml</co> name
extension (case-insensitive) are treated as data files;
other files are ignored.


If data of a collection has changed, you want ACIS to check
and process it to reflect these changes in the database.
Use <c>bin/updareq</c> utility is for this.  If your
collection changes often, you will want to run this utility
at regular intervals.



<h3 id='ri-collections'>Configuring collections</h3>


You configure the collections with
<c>metadata-collections</c> parameter of the
<f>main.conf</f> file.  For each collection you put the
collection identifier into this parameter and you specify
its type and path to it with <c>metadata-<i>X</i>-type</c>
and <c>metadata-<i>X</i>-home</c> parameters, where
<co><i>X</i></co> is the collection identifier.

If you have several collections, separate their identifiers
in <c>metadata-collections</c> with a space.

"<co>ACIS</co>" is a reserved collection identifier, system
uses it for its own generated data.

You use collection identifiers when you request a data
update of a collection with <c>bin/updareq</c> utility.

Right now ACIS understands data in two metadata formats:
[RePEc](http://repec.org)'s ReDIF and
[AMF](http://amf.openlib.org/).  Support for other metadata
formats and collection structures can be added.  It is not
very difficult to develop, given that they are file-based or
have a pretty simple way to map records to files.

Parsed collections configuration is written into
<co>{HOME}/RI/collections</co> file.







## Installation

<p><strong>
Normally, the update daemon is installed when you install 
ACIS and no special installation is needed.
</strong></p>

If you need a separate installation of the update daemon or
if you want to manually upgrade the update daemon, you may
follow these instructions.  Grab the latest RePEc-Index
package from <a
href='http://acis.openlib.org/code/'>http://acis.openlib.org/code/</a>.
You unpack the package and then run:
 
    $ RePEc-Index-0.XX/install.sh HOME

where HOME is the path to your ACIS home dir (or the
directory you want to install the daemon to).

The actual daemon script will be installed to
<f>bin/control_daemon.pl</f>.  But you do not normally need
it.  ACIS includes everything you need to work with it, like
scripts to <a ref='binrid'>start and stop</a> the daemon and
to <a ref='binupdareq'>send update requests</a> to it.


I suggest that for a serious ACIS installation you make a
private copy of Berkeley DB.  This will protect you from
system-wide software updates.  Read [why and how to do
that](bdb-private.html).




## Administration 


### Starting and stopping the daemon

Use <c>bin/rid</c><co> start</co> and <c>bin/rid</c><co>
stop</co> to start and stop update daemon, respectively.

You better run the daemon all the time while ACIS is
working.  If for some reason it were not running for some
time, it is not a catastrophe.  Some update requests may be
lost, but generally you can recover by running
"<co>bin/updareq ACIS /</co>".



<h3 id='sending-update-requests'>Sending update requests</h3>

<dl>

<dt><co>bin/updareq COLLECTION PATH [TOO_OLD]</co></dt>

<dd>Sends an update request to the daemon.  It asks to
update file or dir <co>PATH</co> in collection
<co>COLLECTION</co>.  <co>TOO_OLD</co> is time in seconds.
If a file was last time processed more than <co>TOO_OLD</co>
seconds ago, the daemon will process it again (even if it
didn't change since).  By default, <co>TOO_OLD</co> is
<code>86400*12</code> seconds, which means 12 days.  </dd>

</dl>



### Logs

The main log it keeps is in <f>RI/daemon.log</f>.  It is
the general log of requests comming in and what *processing
channel* took it for processing.  

All details of processing particular requests go into logs
<f>RI/update\_ch0.log</f>, <f>RI/update\_ch1.log</f>, ...
<f>RI/update\_ch5.log</f>.  Each of these correspond to a
processing channel.  

These logs protocol what is being done, what files are read,
what records are found in them and so on.  If there were any
problems with processing data, it will be logged in there.



<h3 id='db-house'>Database housekeeping</h3>


Recent versions of update daemon use Berkeley DB <a
href='http://www.sleepycat.com/products/transactional.shtml'>Transactional
Data Store</a> for its database.  This causes it to work a
little slower (when compared to plain Data Store), but it
gives us parallel processing feature and great
fault-protection.

The database files are stored in <f>{HOME}/RI/data</f>.
General files (so called "database environment") are stored
in this dir, and files for specific collections are stored
in subdirs of it.  For example, data a for collection called
"bliss" would live in <f>{HOME}/RI/data/bliss</f>.

When data is added or modified in the database, the Berkeley
DB library creates sequentially numbered log files, e.g.
<f>RI/data/log.0000000001</f>.  If you have lots of data
going in and getting modified in the database, number of
these files can grow fast and they'll occupy a real huge
amount of disk space.

To solve this you need to do two things:

1. initiate chekpoints regularly;

2. delete unnecessary log files. 

You can read about it in the Berkeley DB <a
href='http://www.sleepycat.com/docs/'>documentation</a>, but
basically, you can do it this way:

    db_checkpoint -1 -h {HOME}/data
    db_archive -d -h {HOME}/RI/data

<co>db\_checkpoint</co> and <co>db\_archive</co> utilities are
from the Berkeley DB library package.  Some care you need
here. Becaust you may have these utilities installed
system-wide.  And those can be of wrong version of BDB than
you actually use.  If you have installed a <a
ref='bdb-private'>private copy</a> of Berkeley DB, then
you'll have to refer to these utilities by full path.


For instance, in my case, I had to use
<f>/home/ivan/lib/bdb/bin/db\_checkpoint</f> and
<f>/home/ivan/lib/bdb/bin/db\_archive</f>.  So, I have a
cron entry which runs a little script like this:

     #!/bin/sh
     /home/ivan/lib/bdb/bin/db_checkpoint -1 -h /opt/ACIS/RI/data
     /home/ivan/lib/bdb/bin/db_archive -d -h /opt/ACIS/RI/data


Here are some documentation links, just in case you need to
know more:

- <a
  href='http://www.sleepycat.com/docs/ref/transapp/checkpoint.html'>about 
  checkpoints</a>

- <a
  href='http://www.sleepycat.com/docs/ref/transapp/logfile.html'>about
  log files removal</a>

- <a
  href='http://www.sleepycat.com/docs/utility/db_checkpoint.html'>db_checkpoint
  utility doc</a>


- <a
  href='http://www.sleepycat.com/docs/utility/db_archive.html'>db_archive
  utility doc</a>







* * *

If you want to know more about the daemon, you may look into
<a href='internal.html'>internals</a> doc.


