EPrints-related stuff in ACIS
=============================

<toc/>


## Intro

<a href='http://software.eprints.org'>GNU EPrints</a> is an
open-source online repository software for preprints and
other scholarly materials.  It's primary usage scenario is
self-archiving.  It is being maintained and developed
actively.

In ACIS we have specified and developed some <a
href='cooperate.html' >features for cooperation</a> with
document submission services.  We chose EPrints as the basis
for a reference implementation of these features.  This
document explains in detail what features do we introduce
into EPrints and how to use them.



<h2 id='terminology'>Terminology</h2>


EPrints uses some terms specificly.  We will try with all
care to avoid confusion.  For this, we will use the EPrints'
terms in their EPrints-specific sense.

For instance, let's take term *document*.  In EPrints it
means *a rendering of a document in a file of a particular
type*.  To mean a document as a piece of content, like a
working paper or an article, which gets submitted to
EPrints, they use word *eprint* (note lower case and
singular number) and we will use that.

When EPrints talks about an installation of EPrints it would
usually say "an archive".  *An archive*, therefore is *a service,
powered by EPrints*.  

One EPrints installation can power several archives.  Each
EPrints archive has an id-name and an accordingly-named
subdirectory in <f>{EPrints-dir}/archives/</f>.  That is
called *an archive's directory*.

Each archive is configured via several configuration files.
Most basic and vital things are set in an XML file, named
<f>{EPrints-dir}/archives/{archive-id}.xml</f>.  Several
other configuration files lie in a directory called
<f>{EPrints-dir}/archives/{archive-id}/cfg</f>.

This configuration directory contains the file
<f>ArchiveConfig.pm</f>, that holds the main configuration
parameters for an archive.  When further below we refer to
the *archive configuration*, we mean the <co>get_conf()</co>
function in this file.  This Perl function returns a hash
reference with the configuration parameters as the keys of
the hash.

A typical parameter setting looks like this:

    # If 1, users can request the removal of their submissions from the archive
    $c->{allow_user_removal_request} = 1;

<p>If you need to add a configuration parameter, you'd want to
add a line like this, with your parameter name instead of
the <co>allow_user_removal_request</co>.  You can do it
right before the return statement of the <co>get_conf()</co>
function, or, maybe, right after the initial lines:
</p>

    sub get_conf
    {
        my( $archiveinfo ) = @_;
        my $c = {};

Choose your topping.

A link on the topic:

<ul><li><a
href='http://software.eprints.org/docs/php/configarchive.php'>The
Archive Configuration Files</a> from the EPrints online
documentation</li>
</ul>



<h2 id='eprints-installation-brief'>Installation</h2>


<h3 id='eprints-install-script'>Automatic installation</h3>

The primary way to install these extensions onto an EPrints
service is to use the <f>install.pl</f> script in the
<f>EPrints</f> directory of ACIS.  Give the path to your
EPrints archive directory as a parameter to that script and
follow the instructions.  For example:

    $ cd ACIS-1.2.3-20051010d/EPrints
    $ perl install.pl /opt/eprints2/archive/Archive


If this does not work for you, there are more details on
installation in the <a
href='eprints-install.html'>Installation of ACIS'
extentsions to EPrints</a> document.

The further sections of this document contain installation
and configuration instructions which are specific to that
particular level of ACIS-EPrints cooperation.  But all that
is covered by the automatic installation and is present here
for historical reasons.  You may also use it to change an
existing configuration.



<h2 id='level1'>Level 1.  Export eprints metadata in
AMF</h2>


EPrints can export its metadata in a simple XML format.  It
includes a utility <co>export_xml</co> for that.  EPrints
has an <a href='http://openarchives.org' >OAI-PMH</a>
interface in it, and so, includes a Dublin Core conversion.
But we need:

  1. each document's data in a separate datafile;

  2. datafiles created, updated and deleted immediately as
     EPrints editors accept or remove individual eprints.

And that's what our AMF metadata export does.


<h3 id='metadata-export-amf'>AMF metadata export module</h3>

The module's name is ACIS::EPrints::MetadataExport::AMF and
it is located in the
<f>EPrints/perl_lib/ACIS/EPrints/MetadataExport/AMF.pm</f>
file.


#### Installation and configuration

The <a ref='eprints-install-script'>installation script</a>
will patch the <f>perl_lib/EPrints/EPrint.pm</f> module.
This will make EPrints call our AMF export module exactly
when it is needed: on occasions when a new eprints has been
created (accepted into the repository), when an eprint data
has been modified and when an eprint is being deleted.  It
will add several lines into the generate\_static() method of
the EPrints::EPrints class:

    require ACIS::EPrints::MetadataExport::AMF;
    if ( $ds_id eq 'archive' ) {
       ACIS::EPrints::MetadataExport::AMF::export_metadata( $self );
    }

and these two lines into the \_move\_from\_archive() method:

    require ACIS::EPrints::MetadataExport::AMF;
    ACIS::EPrints::MetadataExport::AMF::clear_metadata( $self );


To configure the AMF export module itself, add two
configuration parameters to the get_conf() subroutine of
your archive's configuration <f>cfg/ArchiveConfig.pm</f>:

<dl>

<dt><C>eprint_metadata_export_AMF_dir</C></dt>

<dd>Pathname of the directory to create AMF files in.</dd>

<dt><C>eprint_metadata_export_AMF_idprefix</C></dt>

<dd>A string; will be used as a prefix for AMF text
identifiers.  Default value: empty string.</dd>

</dl>

Example: 

    $c->{eprint_metadata_export_AMF_dir} = "/opt/eprints2/archives/Musasi/amf";
    $c->{eprint_metadata_export_AMF_idprefix} = "Musasi:";


####  What it does.  Techinical details

Upon a call to <co>export_metadata()</co> function in this
module, it will:

1. get and check the configuration parameters specified
   above.  

   If the directory parameter points to an inexistant
   directory, try to create it.  Fail silently if this
   fails.

2. extract internal numeric eprint identifier from the
   passed EPRINT object;

3. construct datafile pathname from the
   configuration-specified directory and the numeric id as
   follows:

        my $filename = $dir . "/" . $id . ".amf.xml";

4. create empty AMF *text* object with ID
   <code>"$prefix$id"</code>.  Populate it with other data
   of the eprint (see next section).

5. get XML rendering of it;

6. write it to the file, adding standard AMF prolog and
   epilog (the &lt;amf&gt; element) along the way.



The <co>clear_metadata()</co> function in this
module will:

1. get and check the directory configuration parameter
   specified above;

2. build the datafile pathname exactly as in previous
   function;

3. delete the file if it exists.



### The AMF generation

<a href='http://acis.openlib.org/documents/saskatoon/' >Saskatoon</a> 
doc <a
href='http://acis.openlib.org/documents/saskatoon/#sec3-3-1'>says
that</a> AMF from EPrints must reflect:

   1. the work's title

   2. the authors' full names

   3. EPrints' internal id of the eprint

   4. the URL of the work's page in EPrints


<!-- XXX Thomas dislikes "we" in the following text. -->

Given an eprint object, we first build an empty AMF *text*
noun with ID <code>"$prefix$id"</code>, where
<co>$prefix</co> comes from configuration
(<c>eprint\_metadata\_export\_AMF\_idprefix</c>) and
<co>$id</co> is the eprint's internal id (the
<code>eprintid</code> field).  That covers point no. 3.

Then we add <co>title</co> and <co>displaypage</co>
adjectives, these are points 1 and 4.  


Then we go through all the work's authors (the
<code>creators</code> field) and editors (the
<code>editors</code> field).  For each of them we create a
*person* noun, see next section.  And then we save each
author data via the <co>hasauthor</co> verb and each editor
data via the <co>haseditor</co> verb.  That gives us point
number 2.



#### From eprint's personal data to AMF *person* noun


That is made in a separate function <c>decode_person()</c>.
First, we create an empty AMF *person* noun.

We populate the person noun with the <co>givenname</co> and
<co>familyname</co> adjectives, if corresponding fields are
defined.  We use the EPrints utility function
<co>EPrints::Utils::make\_name\_string()</co> to get full
name of the person.  Save result as adjective <co>name</co>.

We check the personal data for presence of the <co>id</co>
field.  If defined, we check it for what kind of value it
contains.  It may contain a short-id or an email.  If it
contains an email address, we save it into an <co>email</co>
adjective.  If it is a short-id, we save it into
<co>identifier</co> adjective.  It is part of <a
href='#level2'>level 2</a>, actually.




#### Additional: from eprint type to AMF text type


Each eprint object has a <co>type</co> field.  EPrints is
flexible in the types of document that one can store in it.
Administrators can change the available types, can define
new types.  It means the range of values that this field can
take is not fully defined.  We will rely on default types
configuration.  Given that this is not a required feature,
we will not try too hard to find a matching AMF type.

Here are the basic eprints types, defined in
<f>defaultcfg/metadata-types.xml</f> (from EPrints dir) and
their AMF counterpart, if defined.


<table class='files'>
<tr><th align='left'
>eprint <i>type</i> field &#160; </th><th> AMF text type adjective</th></tr>
<tr><td><co>article</co></td>         <td><co>article</co></td></tr>
<tr><td><co>book</co></td>            <td><co>book</co></td></tr>
<tr><td><co>book_section</co> </td>   <td><co>bookitem</co></td></tr>
<tr><td><co>conference_item</co> &#160; </td> <td><co>conferencepaper</co></td></tr>
<tr><td><co>monograph</co>       </td><td>see below</td></tr>
<tr><td><co>thesis</co>          </td><td>-</td></tr>
<tr><td><co>patent</co>          </td><td>-</td></tr>
<tr><td><co>other</co>           </td><td>-</td></tr>
</table>

Eprints of type *monograph* also have a
<co>monograph\_type</co> field.  This field takes 
following values (again with their AMF text type equivalents):

<table class='files'>
<tr><th align='left'
>eprint <i>monograph_type</i> field &#160; </th>
                                 <th> AMF text type adjective</th></tr>
<tr><td><co>technical_report</co> &#160; </td>
                                 <td><co>preprint</co></td></tr>
<tr><td><co>project_report</co></td>
                                 <td>-</td></tr>
<tr><td><co>documentation</co> </td>
                                 <td>-</td></tr>
<tr><td><co>manual</co> </td>    <td>-</td></tr>
<tr><td><co>working_paper</co></td>
                                 <td><co>preprint</co></td></tr>
<tr><td><co>discussion_paper</co> &#160; </td>
                                 <td><co>preprint</co></td></tr>
<tr><td><co>other</co>           </td><td>-</td></tr>
</table>




<h3 id='amf-plugin'>AMF output plugin</h3>

EPrints, since version 2.3.14, supports plugins for metadata
conversion and output.  Plugins are special modules, made as
drop-in files.  You simply put the plugins you want to use
into the <f>plugins/output/</f> directory of your EPrints
installation (or to that of your EPrints archive), restart
EPrints (i.e. Apache) and they work.  Default EPrints
installation already has a number of plugins pre-installed:
for BibTeX and for a simple XML format metadata output.

One of the ways to see them working is to look at such URL:

    http://your.eprints.url/perl/export/#id#

where <code>http://your.eprints.url</code> is the base URL
of your EPrints service, and <code>#id#</code> is an actual
id of an eprint (a submission) in your service.

You should then see a page offering you metadata of this
item in different formats, each format representing an
installed plugin.

Now the AMF plugin is in ACIS file
<f>EPrints/plugins/output/amf</f>.  But if you followed the
<a ref='eprints-installation-brief'>general installation</a>
instruction above, it is already installed and should be
working.

Note that the AMF plugin depends on the
ACIS::EPrints::MetadataExport::AMF module, <a
ref='metadata-export-amf'>described above</a>.


<h2 id='level2'>Level 2.  Personal id metadata
field</h2>


For each person, author or editor of a work, EPrints asks
and stores a number of fields.  Main are the name fields,
like given name and family name.  In default installations
there is also a field called "Creators email (if known)".
Internally and by design it is named "id".  We will use it
for the personal identifiers in author/editor metadata.

Typical traditional usage of this field is for email
addresses.  We will assume it is used for both email
addresses and personal id.

EPrints allows administrator to configure the field labels.
We suggest to change this field's label to "Id or
email".

To change the field's label edit the <f>phrases-en.xml</f>
file in your archive's configuration directory.  Find and
modify the &lt;ep:phrase&gt; items identified
<code>eprint\_fieldname\_creators\_id</code> and
<code>eprint\_fieldname\_editors\_id</code>.  The first one
looks like this:

    <ep:phrase ref="eprint_fieldname_creators_id">Creators email (if known)</ep:phrase>

If your installation of EPrints does not have this field,
you need to enable it in your archive's
<f>cfg/ArchiveMetadataFieldsConfig.pm</f> file.

For that: 

  1. find line which says

        $fields->{eprint} = [

  2. near below it find the line, which starts with

            { name => "creators", type => "name", 
	  
  and make sure <code>hasid</code> parameter in this
  block is set to 1.  The block will look like this:
  
            { name => "creators", type => "name", multiple => 1, input_boxes => 4,
                    hasid => 1, input_id_cols=>20,
                    family_first=>1, hide_honourific=>1, hide_lineage=>1 },

  3. Do the same for the "editors" field.  Here the block
  starts with

            { name => "editors", type => "name" 

  Again, set parameter <code>hasid</code> to 1.




This is what you get on the "Core Bibliographic Information"
page of EPrints if you follow the above instructions:

<p>
<img class='snap'
src='img/eprints/core-bibliographic-info-author-area-with-shortid-or-email-field-with-empty-fields.png'/>
</p>



If you had any problems with above instructions, you may
want to look at <a
href='http://software.eprints.org/docs/php/metadata.php'>Configuring
the Archive Metadata</a> section of the EPrints
documentation.



<h2 id='level3'>Level 3.  Author identification aid</h2>




An eprint submission in EPrints takes a number of screens to
go through.  We will modify just one of the screens, the one
called "Core Bibliographic Information" in the default
English-language installation.

On this screen a user enters title of the work being
submited and data about its authors.  As she enters author
information, we will search the records database in ACIS to
see if we have any matching records.  And we offer found
personal records for the user to choose.  If the user
accepts an offered record, we automatically fill in the name
and id fields for this author.  Having correct short-id in
the document metadata is what we want after all.

It is important that we minimize user errors at this stage.
If many EPrints users start assigning documents to wrong
persons, it would cause bad metadata.

We do not change the normal user workflow.  We avoid
intervening in it.  Instead we try to gently offer our help,
where we can.  

The next section starts a functional specification of the
interface changes that we introduce.  Later on there is a
description of the technical side of things: how to install
and configure it, what components does it consist of and
they work.




<h3 id='level3-ui'>User interface</h3>

This is a functional specification of the user interface
enhancements to EPrints at level 3.  The basic general logic
of the enhancements is right below.  In further sections
you'll see it detailed.

For data of the authors the "Core Bibliographic Information"
screen has a table with input fields in it.  After <a
href='#level2'>level 2</a>, the columns are titled: "Family
name", "Given name / Initials", "Id or email".  (As an
aside, the order of the columns on the screen may be
different.  You can configure EPrints to display "Given name
/ Initials" before the "Family name".)

The user types her information about the authors into these
fields.  We look at what she types, do search queries of our
database.  If we have something to say, we insert a row into
the author data input table and put our message into this
row.  This shifts further rows of the table further below.
So our message appears just below the relevant input fields.

There should be no problem if the user does not notice our
message, does not understands it or prefers to ignore it.
She can ignore it and go on.  When the user starts working
with another row in the personal data table, we will make
the message that was relevant to the previous author
disappear.  If the user returns back to the row (the field),
which caused the message, we display it again.

Our message is usually a menu of personal records that our
search yielded.  We do not display the menu if we have too
many search results.  (That would probably mean the search
conditions are too vague to be useful.)  <i>Too many</i> is
more than a magical number <co>max_results</co>, with 15 as
the default value.  It is configurable via the <a
ref='acis-pidaid'>ACIS::PIDAID module</a>.


<p>
<img class='snap'
src='img/eprints/core-bibliographic-info-author-area-with-shortid-or-email-field-with-walsh-and-menu.png'/>
</p>


The menu is shown as a little table.  Each row of the table
represents one person.  Next to the person's name is a
mini-icon link to further information about the person -- to
his or her profile page.  Then comes a link to his or her
homepage.  Next is an inviting mini-icon to choose the
person.

A click on the name or on the choose icon would mean user
has made a choice.  It would mean she was entering data
about that particular person.  At this point we remove the
menu and fill in the personal data fields with the data that
we have: the family name, the given name and the id.  

If search yielded any results, we always build a menu of
these results and display it.  If search yielded no results
or too many results and there is a previously-generated menu,
then we remove the menu.





#### Search by name


##### Sample scenario

A user has typed in the title of the work, and then switched
to the next input field, i.e. the family name input box for
the first author.  She types the first few letters of the
author's surname.  The page makes a search and brings up a
little list of personal names, right below the input box.
User finds a matching person in this list, clicks on that
person's name.  Immediately the family, the given name and
the id/email input boxes are filled in with data from
ACIS.  Then the menu disappears.  The user is happy because
she doesn't have to type that info manually.  We get an
exactly identified author in the eprint's metadata.

The user clicks on the second author's family name input box
and continues the process.


##### Principles

Name search is case-insensitive.

We attempt to search for the matching personal records each
time user acts in a name input box, be it family name box or
given name box.  Pressing a key, entering or leaving an
input box are the acts to react upon.


If search yielded any results, we build a menu of these
results.  If the search yielded no results or too many
results and there is a previously-generated menu, then we
remove the menu.

When user is in an input box, we assume she is editing it
and probably has not finished it yet.  When focus belongs to
the family input box, we search for lastnames **starting
with** what user has typed so far.  So if user typed "Mendel",
it would match both "Mendel" and "Mendelssohn".

As soon as user leaves a family box, we assume the surname
typed in it is complete and cut off the items, which do not
produce a whole-string match.  So if user left the family
name box with "Mendel" in it, only this family name will
match.


Search requests are asynchronous, which means the page does
not stop reacting to user actions while a search is taking
place.  Search results are processed only when a response
from server has arrived.





#### Search by email address and by short-id


##### Sample email-search scenario 

A user types in the title of a work and then enters the
family name and the given name of an author.  ACIS doesn't
recognize the name for some reason.  It may be a consequence
of a different (not known to ACIS) variation of the name
spelling.  It may be that ACIS did find some matching
records, but user didn't notice or prefers not to use the
menu of the offered records.

Then the user switches to the next field, the "Id or email"
box.  She enters the author's email address and presses
"TAB" key.  TAB key switches focus to the next row of the
table.  The page sends a request to check if that email
address is known to ACIS.  ACIS finds a personal record with
a matching email address and offers that personal record to
the user.  User clicks on the offered personal name.  The
family, the given name input boxes are filled in with data
from ACIS.  The value of the email address that she entered
is replaced with the person's short-id.


##### Short-id scenario #1, basic

A user enters personal name of an author.  For some reason
ACIS doesn't recognize the person or user doesn't use the
menu offered.  Then she TABs to the id / email field and
types in the identifier for the person that she knows.
After she types in the first 4 characters
(e.g. "<code>pme4</code>") a message appears below the input
box: "Id 'pme4' belongs to:" with a single-item menu below
it.  The item is the record, which corresponds to this
short-id.

Soon after she types another char, the page rechecks and
offers her a new record -- the one that corresponds to the
current value in the short-id input box.

Once she typed the identifier completely (it may be 4, 5, 6
characters long), she sees the name of the person she meant.
She is now sure that she did no mistake.


##### Short-id scenario #2: Advanced user

An advanced experienced user comes to the page with the
short-id of the author at hand.  Instead of entering the
author's name, she enters the short-id and waits to see the
page has recognized it.

A second later she sees the expected name (i.e. name of the
author), clicks on it and name fields are automatically
filled in.  Everybody is happy.



##### Short-id scenario #3: User makes a mistake

A user enters the name details for a person and then enters
an identifier for him.  She doesn't look at the messages
appearing or they were not appearing because of slow
connection or because the server was pretty loaded at the
moment.

She goes on to complete details of the next author/creator.
A few moments away the identifier that she entered becomes
red-colored.  She notices that, hoovers mouse cursor over
the field, and sees a little floating hint: "No such record:
id unknown".

She clicks on the field.  Immediately she sees this same
message below the field.  She rechecks her notes and finds
out she made a mistake.  She corrects the value of the
field, waits a second or two to see if the page complains
about it.  Soon she sees that all is fine this time: the
page offers the record she meant.  She continues to enter
next person's details.  The menu is hidden as she starts to
type another person's family name.



##### Principles

Email search and short-id search happen in the same input
field.  The page looks at the format of the value string to
understand, whether it is an email or an id.

The email search and the short-id search are
case-insensitive.

The id and email searches do exact full-string matching.

We attempt to search for the matching personal records each
time user acts in the input box.  Pressing a key, entering
or leaving an input box are the actions to react upon.

If search yielded any results, we build a menu of these
results.  If search yielded no results or too many results
and there is a previously-generated menu, then we remove the
menu.

Search requests are asynchronous, which means the page does
not stop reacting to user actions while a search is taking
place.  Search results are processed only when a response
from server has arrived.


If user has left the field with a non-empty value, which
does not look like an email address and is not a valid
short-id (there was a successful search by short-id attempt,
which returned an empty result list), the value is shown in
red color and a hint "No such record: id unknown" is
attached to the field (via <code>title</code> attribute).

If user enters the field while it is value is known to be
invalid, we display message "No such record: id '...'
unknown." below the input field; red color and the floating
hint is removed.

When such message is shown and user starts to edit the
value, the message is removed.





#### Problems and weaknesses

The user interface enhancements, that are described above,
have a number of potential problems and weak sides.

First, most users (until they learn) will not expect the
search results menu and other messages to appear.  We hope
it will be a nice surprise.  But novice users may feel
irritated by unexpected, unrequested things appearing on a
page.

Second, latency and server load problem.  If the user has a
slow network connection or the server, handling search
queries, is overloaded, the features may distort the
experience.  At the same time, with growing usage of an
EPrints installation, the number of db search queries may
grow dramatically, thus increasing the load on the EPrints
service server and the ACIS database server.

<div class='note'> 
<p>Situation is as follows: the page
sends a CGI query back to EPrints, to the <f>cgi/pidaid</f>
script.  The <f>cgi/pidaid</f> script sends a database query
to the ACIS' MySQL server.  MySQL returns some result.  The
<f>cgi/pidaid</f> XML-encodes it and returns to the
page.</p>

<p>Each component of the structure may become a bottleneck.
But I guess that MySQL is under biggest strain here, because
<f>cgi/pidaid</f> will run under mod_perl, so it won't eat too much
resources -- no new processes to start for each request, at
least.  MySQL also doesn't have to start a process for each
request, but it must search the database.  Of course, it
becomes even more strained if several EPrints installations
would query the same ACIS database at the same time.
</p>

<p>
On the other hand, how many document submissions are we
going to handle?  I doubt any of the existing EPrints
installations has 10 submissions per day on average.
But of course, this will grow.  Or at least we have to
account for that.
</p>
</div>


Third, such implementation may create accessibility problems
for people with disabilities, for users with
speach-browsers, with memory and orientation difficulties.

Fourth, there is no way for the user to enter both an email
address and a short-id for a person.

Fifth, with the current setup, having chosen an item from a
menu, user has no way to undo this choice and to return to
the previous state.

Sixth, the for technical reasons, it won't work for users
with older versions of the browsers and some modern, but
incompatible browsers (e.g.  Opera 7 and earlier).  For
those users it will work as a usual EPrints service, only
with a differently-named email field.





#### Undo feature (additional)

Whenever user gets names from a personal records menu
inserted into the name and id fields, right under the id
box appears a small "undo choice" button.  If user clicks on
the button, the previous values are restored and the button
disappears.  The previous values are the ones that were in
those fields at the moment when the user clicked on a menu
item.

If after making a choice the user starts editing any of the
name field values or the id field value, the button
disappears also.  (Otherwise it may confusingly suggest that
it is capable of undoing his editing.)







<h3 id='level3-tech'>Implementation technical specification</h3>


#### Components and technology

Our enhancement of EPrints for author/person identification
aid consists of:

- a CGI script added to EPrints, file <f>cgi/pidaid</f>;

- JavaScript code, file <f>pidaid.js</f>;

- little additional <abbr title='Cascading Style
  Sheets'>CSS</abbr> styling, file <f>pidaid.css</f>.

The JavaScript part defines certain reactions to certain
user actions (events).  These reactions include background
queries back to server and modification of the page without
full page reload.  JavaScript communicates with the CGI
script on the EPrints server via XMLHttpRequest technology
(see <a href='#xmlhttprequest'>below</a>).

The <a ref='acis-pidaid'>ACIS::PIDAID module</a> executes
the database queries for the CGI script.  


<h4 id='pidaid-install'>Installation</h4>

1. Copy <f>sql\_helper/sql\_helper.pm</f> and
   <f>sql\_helper/sql\_result.pm</f> files from ACIS
   distribution into the <f>perl_lib</f> directory of the
   EPrints installation directory.

2. Change to the <f>EPrints/</f> directory of ACIS
   distribution.

3. Copy <f>cgi/pidaid</f> into <f>cgi/</f> dir of the
   EPrints installation.

4. Copy recursively <f>perl_lib</f> directory to the EPrints
   installation directory.

5. Copy <f>pidaid.js</f> and <f>pidaid.css</f> and
<f>images/</f> directory (recursively) to the <f>html/en/</f>
dir of the EPrints archive.

6. Edit <f>cfg/template-en.xml</f> in the archive directory to
add <f>pidaid.js</f> and <f>pidaid.css</f> to every HTML
page built.  It may now look like this:

        ...
        <html xmlns="http://www.w3.org/1999/xhtml"
              xmlns:ep="http://eprints.org/ep2/template">
          <head>
            <title>&archivename; - <ep:pin ref="title" textonly="yes" /></title>
            <style style="text/css" media="screen">
        @import  url(&base_url;/eprints.css);
        @import  url(&base_url;/pidaid.css);     /* PIDAID */
            </style>
            <style style="text/css" media="print">
        @import  url(&base_url;/eprints.css);
        @import  url(&base_url;/pidaid.css);     /* PIDAID */
        @import  url(&base_url;/print.css);
            </style>
            <!-- and PIDAID JavaScript: -->
            <script type='text/javascript' src='&base_url;/pidaid.js' />  
            <link rel="Top" href="&frontpage;" />
        ...


7. Obtain mysql access parameters from the ACIS service
   administrator.  Then configure the <a
   ref='acis-pidaid'>ACIS::PIDAID</a> module by defining the
   <co>$ACIS::PIDAID::CONF</co> variable in the archive
   configuration file (<f>cfg/ArchiveConfig.pm</f>).  It may
   look like this:

        $ACIS::PIDAID::CONF = {
           host => 'acis.super.edu',
           port => '9099',
           db   => 'ACIS',
           user => 'peter',
           pass => 'jolly',
           max_results => '25', # optional, default: 15
        };

8. Stop and then start again EPrints, i.e. the web server it
   runs on.  The usual <code>apachectl restart</code> may
   not be enough -- you need to separately stop and then
   start it.


#### <f>cgi/pidaid</f> script

This CGI script searches for personal records.  The script
is installed into EPrints' <f>cgi/</f> directory.  It
accepts HTTP GET and POST requests and the usual CGI-style
parameters.  Returns an XML document.

Usage: 

    http://eprints.super.edu/perl/pidaid?l=mendel*
    http://eprints.super.edu/perl/pidaid?s=pme4

Response's <co>Content-type</co> is "<co>text/xml; charset=utf-8</co>".

The script uses the <a ref='acis-pidaid'>ACIS::PIDAID</a>
module to make the actual searches and the <a
ref='acis-pidaid-xml'>ACIS::PIDAID::XML</a> module to
produce an XML string of search results.

The CGI parameters this script accepts:

 - <co>f</co>: given name to search by;

 - <co>l</co>: family name to search by;

 - <co>s</co>: short-id to search by;

 - <co>e</co>: email address to search by.

There are three search modes:

 - by name -- <co>l</co> and <co>f</co> parameters are used;

 - by short-id -- <co>s</co> parameter is used;

 - by email address -- <co>e</co> parameter is used.

Specifying together <co>l</co> and <co>s</co> or <co>l</co>
and <co>e</co> or <co>s</co> and <co>e</co> makes no sense
and return from such requests is not defined.

The script need not to be configured in any way; it is
assumed that admin has configured the <a
ref='acis-pidaid'>ACIS::PIDAID</a> module.

If a search request returns too many matching records, the
script will return XML string
"<code>&lt;toomany/&gt;</code>".


In case of an internal problem the script may return XML
string "<code>&lt;problem/&gt;</code>".



##### Technical details

The <f>cgi/pidaid</f> script employs the normal EPrints'
infrastructure for accepting and processing requests.  It
creates an EPrints::Session object and uses it to get the
input parameters parsed.




#### <f>pidaid.js</f> add-on JavaScript


##### XMLHttpRequest technology 

XMLHttpRequest is a piece of browser technology that bacame
widely known just recently, although existed for a long
while now.  It allows to create "responsive" web pages.
With it a web page can communicate to a server and then
modify itself based on what server returned.  To modify
itself, the page can use the HTML <abbr title='Document
Object Model'>DOM</abbr> interface.  XMLHttpRequest is a
client-side technology and requires in-browser scripting;
normally JavaScript is used.

XMLHttpRequest has asynchronous mode, when communication
with the server happens in the background.  While a request
is sent and a response is recieved, the page is still
usable, it can react to user input in a usual way.  This
contrasts with what happens at the page reload phase of a
usual user-to-a-website interaction.

The XMLHttpRequest requests can contain parameters, as if
sent by a submitted web form.  The only important limitation
is that most browsers restrict to which servers it can send
requests; the server must be on the same domain name as the
server from which the current page was loaded.  This is a
security measure.

Response to a XMLHttpRequest request can be of any
type, e.g. HTML or plain text; just as usual HTTP responses
can be.  But if a response contains some data in XML,
XMLHttpRequest provides an interface to parse and traverse
it.  Again it is <abbr title='Document Object
Model'>DOM</abbr> interface.



Not all browsers support XMLHttpRequest.  It works in
Internet Explorer 5.0 and higher, in Mozilla 1.0+ and
FireFox; Safari 1.2+.  It does not work in Opera 7 and
earlier, but Opera 8 beta already has an early
implementation of it.  So, we can hope that soon there will
be very little number of web users who don't have it.  Of
course, some users have JavaScript disabled.  Nothing we can
do about it.



Here are some examples of using the technology:

1. [Google Suggest](http://www.google.com/webhp?complete=1)

2. Meetup.com [registration form](http://www.meetup.com/register/).
   Choose a country and see how the page reacts -- it loads the
   list of cities from the server.



Some external resources:

1. [Using the XML HTTP Request object](http://jibbering.com/2002/4/httprequest.html)

2. [Dynamic HTML and XML: The XMLHttpRequest Object](http://developer.apple.com/internet/webcontent/xmlhttpreq.html)

3. [Mozilla.com: XML Extras](http://www.mozilla.org/xmlextras/)

4. [AJAX -- Asynchronous JavaScript And XML](http://en.wikipedia.org/wiki/AJAX)

5. [SPA -- Single Page Application](http://en.wikipedia.org/wiki/Single_Page_Application)


##### Overview

This section offers a brief, simplified overview of how the
<f>pidaid.js</f> script works.  

The script is executed in a user's browser in context of
an EPrints webpage.  

It starts by calling the <co>install\_check\_handlers()</co>
function (via the page's body "onload" event).  The function
goes through all the &lt;input&gt; elements of the page.
Along the way, it checks each of them to see if it looks
like a personal data entry element.  If there are no such
elements, this is not a page that is interesting to us.

For each group of personal data entry input boxes on the
page the function creates an object and puts it into the
global <var>name\_groups</var> array.  We use it to store
certain parameters of each group of controls.  (Each group
of controls corresponds to one person, whose name data can
possibly be entered on the page.)

Also, as <co>install\_check\_handlers()</co> cycles through
the &lt;input&gt; elements and finds groups of controls, it
assigns handlers for "onfocus", "onblur" and "onkeyup" events
to some of them.  To be precise, it assigns handlers for
these events to the family name, given name and id/email
input boxes.  Depending on the EPrints configuration, name
controls may also include honourific name and lineage name
boxes.  They are disabled by default and we ignore them even
if they are present.

The handler functions that are assigned to the
above-mentioned events are different; they depend on the
event type and the input box type to correctly implement all
details of the <a href='#level3-ui'>user interface</a>.

What all handler functions share in common is that they
attempt to launch a search request.  For instance, all
family name and given name box handlers call
<co>check\_name()</co> function, which starts a search by
name.


A search is done by creating an XMLHttpRequest object, and
using it to send a request to the <f>cgi/pidaid</f> script.
The <co>send\_request()</co> function does that.  Each
request includes search parameters, e.g. a family name user
has typed so far.  Before a request is sent, a call-back
function is set for processing the response.  If the request
was sent successfully and a response arrives, the
<co>process\_search\_results()</co> function analyses the
response and takes further action.

It uses the data from the search response to build a
personal records menu (<co>create\_menu()</co> function) or
produce some other message or hide a previously shown
message.

Here it becomes important to know that personal data entry
input boxes in EPrints are arranged in a HTML table.  One
table row -- one person.  To show a message we create a new
row in the table and a cell in that row and put our message
into that cell.  (The cell's <co>TD</co> element has
<co>colspan</co> attribute set to <co>4</co> value--it spans
4 table columns.)

When we build a menu, we create a new table (inside the
message cell), and put each personal record item in a
separate row.  Each menu table row presents a clickable
personal name and has links to homepage & profile page and
an "inviting" choice mini-icon.  The personal name and the
choice mini-icon have "onclick" event handled by a choice
function.  The choice function sets the name and id input
boxes to the values of the corresponding personal record and
hides the menu.

That's how it works, basically.  You are welcome to look
into the script source if you need more details; it has some
comments for help.



##### Optimizations

These are the rules the script follows for minimal
efficiency and to avoid server overload problems.

At any given time there is only one search request going on.
If necessary, the new one is sent immediately after the
first one has finished.


**Explanation.** It may happen that user continued entering
data while a search request was being sent and a response
was recieved.  This may mean circumstances have changed and
a new request has to be sent.  But if there is a previous
request going on, we do not send a new one.  We set a flag
instead, which instructs the response-processing function to
restart the search when it is executed.  And the
response-processing function will run and check this flag
only when the previous request resulted in a response.


We maintain a simple global cache for search requests to the
server (the <var>cache</var> variable).  It ensures we never
have to repeat the same query twice.

We never actually search when the search expression doesn't
look useful, for example, when the search field's value is
empty.

We always whitespace-normalize key expression before
searching: we strip leading and trailing whitespace and we
compact inner whitespace sequences.  In perl we would do
that by:

    $var =~ s/^\s+//g;
    $var =~ s/\s+$//g;
    $var =~ s/\s+/ /g;



###### Short-id / email search optimizations

There's no point to search for a short-id or email
address until it looks like an ACIS short-id or an email
address.

We use a JavaScript equivalent of the following Perl-style
regular expression for matching email addresses:

    /^\s*([\&\+a-z\d\-\.\=\_]+\@(?:[a-z\d\-\_]+\.)+[a-z]{2,})\s*$/i

For matching ACIS personal short-ids we use this regular
expression:

    /p[a-z]+\d+/i




<ignore>

<!-- this is not needed anymore, I think --> 

#### How to make a testing installation

<div class='note'>
<p>This is primarily for ourselves, the
developers.
</p>
</div>

The screen in EPrints, on which the work author's data is
entered and on which our astonishing menu of personal
records will appear, is only available to registered users
and only while they submit or edit an eprint record.  (The
screen is called "Core Bibliographic Information", but that
doesn't matter.)  It means a stranger Mrs Floggy, that came
to our site by a mysterious coincidence and poked it further
out of extreme bore, or a steering committee member won't
see what we are doing.  Furthermore, it will be complicated
to test these features for us ourselves.  Therefore, we need
to fix it.

Fixing it means creating a page, like the "Core
Bibliographic Information" screen of EPrints, with author
info entry fields, were <f>pidaid.js</f> will work, and
people can come and see it searching the database, offering
a menu, checking a personal id, reading their thoughts.

For that we need to have an ACIS service with large enough
number of personal records in the database.  After
considering <span title='a joke'>a dozen of services</span>,
it is obvious that <a href='http://authors.repec.org/'>RAS</a> 
is the best candidate.

Given an ACIS service, we do the following:

1. install EPrints 2, make an EPrints archive; (BTW, it
   doesn't have to be the same machine as ACIS.)

2. make the <var>records</var> and <var>names</var> tables
   of the ACIS service <a
   ref='pidaid-db-expose'>accessible</a> (possibly over
   network);
   
3. configure the <a ref='acis-pidaid'>ACIS::PIDAID</a>
   module via the EPrints archive's configuration file
   <f>cfg/ArchiveConfig.pm</f>;

4. <a ref='pidaid-install'>install ACIS personal
   identification extensions</a> to it.   

The last thing is to add a static HTML file to EPrints,
which has the basic content of the "Core Bibliographic
Information" screen, and has <f>pidaid.js</f> and
<f>pidaid.css</f> files correctly linked from it.

This can be done by running EPrints, registering a user,
starting a document submission, proceeding to the "Core Bib
Info" screen and then saving the page source into a HTML
file.  Or, we can use the file I have made already (by using
the same complicated methodology as described):
<f>EPrints/pidaid.html</f>.  It has an additional benefit of
containing a "debugging display": a little area in the
page where debugging messages from <f>pidaid.js</f> appear.

If the EPrints archive name is <co>Alexandria</co>, then we
put this file into the
<f>/opt/eprints2/archives/Alexandria/html/en/</f> directory.
After that it becomes available at the top level URL of the
archive.  For instance, if the EPrints <co>Alexandria</co>
archive's frontpage URL was
<co>http://eprints.alexandria.edu/</co>, then the
<f>pidaid.html</f> now becomes available at the
<co>http://eprints.alexandria.edu/pidaid.html</co> URL.

Wasn't it a bit <i>too</i> easy?


</ignore>




<h2 id='level4'>Level 4.  The metadata update request</h2>


This level's goal is to notify an ACIS service about an
updated eprint record.  An updated eprint record here means
a metadata file, to which a new or edited eprint record has
been just exported.

So, it is as an extension to the <a
ref='metadata-export-amf'>ACIS::EPrints::MetadataExport::AMF
module</a>, introduced at level 1.  The MetadataExport::AMF
module writes a metadata file and then immediately notifies
an ACIS service about it.  We use the <a
ref='metaupdate-request'>ACIS::MetaUpdate::Request</a>
module to actually send the request.


### Configuration

Via the <co>$c</co> variable in get_conf() function in the
<f>cfg/ArchiveConfig.pm</f> file of the EPrints archive:

     $c->{eprint_metadata_export_AMF_metaupdate} = {
                'request-target-url' => 'http://acis.super.org/meta/update',
                'archive-id'         => 'michigan',
                'log-filename'       => '/opt/eprints/archives/History/super-org-metaupdate.log',
                'object-dir-levels'  => 1,
     };


Here three first items: <co>request-target-url</co>,
<co>archive-id</co> and <co>log-filename</co> are
configuration for the ACIS::MetaUpdate::Request module.  

You can also configure the ACIS::MetaUpdate::Request module
via the <co>$ACIS::MetaUpdate::Request::CONF</co> variable
(<a ref='metaupdate-request-conf'>details</a>).



<h4 id='object-dir-levels'><co>object-dir-levels</co>
parameter</h4>

The <co>object-dir-levels</co> parameter helps us to
calculate the update request OBJECT value from the absolute
path of a just written AMF file.  It specifies how many
levels of directory structure will be included into the
OBJECT.  If <co>object-dir-levels</co> is 0, the OBJECT will
just include the name of the AMF file (something like
<code>"100.amf.xml"</code>).

But if, for example,
<co>$c->{<c>eprint\_metadata\_export\_AMF\_dir</c>}</co> is
<code>"/opt/eprints/metadata/amf/History/"</code> and
<co>object-dir-levels</co> is 1, and the AMF file 
written was
<code>"/opt/eprints/metadata/amf/History/100.amf.xml"</code>,
then OBJECT will be <code>"History/100.amf.xml"</code>.

It means the ACIS service, to which the update request is
sent, will try to retrieve
<code>"History/100.amf.xml"</code> from us, the document
service archive.

Default value for <co>object-dir-levels</co> is 0.



### Other considerations


It is up to the EPrints service administrator and the ACIS
service administrator to agree on a transport technology for
the data files between the services.  We assume the
simpliest case here: an exported file becomes immediately
available for retrieval by ACIS, probably via a public web
server.  All what is left is to notify ACIS about it.



<id>
$Id: eprints.text,v 2.1 2005/12/30 20:55:37 ivan Exp $
</id>






