Stage D: full-text links, document-to-document links and fuzzy name searches / ACIS documentation

# Stage D: full-text links, document-to-document links and fuzzy name searches

### Table of contents

## Fuzzy name searching

### TODO

- documentation

- fuzzy search enabled for default background search (not only under APU)?

- XXX marks?

### notions

distance level = Levenshtein distance / string length of the name variation

The default critical distance level of distance is 1/m, i.e. 1/7 e.g. if the Levenshtein distance is 2 edits and the name variation’s length is 20, the distance level is 2/20 = 0.1 ; O.1 is less then 1/7, so, this would be a successful and positive match.

### configuration

- fuzzy-name-search — on or off

- fuzzy-name-search-min-common-prefix — The number of characters n of at the start of a name variation that has to match in the name expressions exactly. Default: 3.

- fuzzy-name-search-min-variation-length — The minimum number of characters m that a name variation would have to have in order to qualify for being fuzzy matched. The default is 7.

- fuzzy-name-search-max-name-occurr-in-doc-names — The maximum number of occurrences of a name expression in the document author names table before it is considered for fuzzy matching. The default is 1. If this parameter is set to 0 or is not set, no maximum is checked.

- fuzzy-name-search-max-name-occurr-in-name-variations — The maximum number of occurrences of a name expression in the name variations table before it is considered for fuzzy matching. By default, maximum is 0, ie. a name expression should not be present among name variations. Set is to -1 to disable this limit.

- fuzzy-name-search-via-web — should fuzzy name searches be run when research search is initiated by the online user? (When a search is APU-initiated, this is a question of fuzzy-name-search.)

### administration

Run bin/fuzzy_search_table script regularly to update the rare table.

### implementation components

1. given a set of name expressions (i.e. document author names) compare it to a set of name variations and find close matches.

2. given a name variation and an ACIS configuration, find all relevant name expressions, which should be fuzzy-compared to that variation.

mysql> select name,count(name) from testrdb.res_creators_separate group by name;

mysql> select name,count(name) from names group by name;

### tools

String::Approx

## Full-text file recognition

### TODO

1. input data — support LoTEc data format ftp://all.repec.org/LotEc

   - configure test.acis to use that

2. improve / fix the screen

   - add listed works count? - not critical, maybe

   - scroll the page vertically so that the menu is in the screen middle? - maybe; not critical

   - add explanations or links to explanations — Thomas hasn’t yet sent me the wording corrections and/or explanations

### input

authoritative:

    <text id="..">
     <file>
      <url>url</url>
     </file>
    </text>

automatically found:

    <text id="..">
     <hasversion>
      <text>
       <file>
        <url>url</url>
       </file>
      </text>
     </hasversion>
    </text>

### user choices

In the recognition menu, there are four mutually exclusive options.

1. This is not a file related to this paper.
2. This is a web page describing the paper.
3. This is a full-text file for another version of the paper.
4. This is a full-text file for this version of the paper.

The default is option 4.

The options will be expressed in shorter words, but there will be a help button.

The premissions menu is only shown if one of the last two options has been chosen. In the permission menu, there are two options

1. This full-text file may be archived as is.
2. This full-text file may be archived, but check for updates.
3. This full-text file may not be archived.

The default is option 1.

Co-authors may make contradictory choices. An ACIS installation records such choices, but does not resolve them.

Short versions of the menu items, first menu (“recognition”):

1. wrong (n)
2. abstract page (d)
3. full-text file of another version (r)
4. correct full-text file (y)

second menu (“permission”):

1. may archive (y)
2. archive, but check for updates (c)
3. do not archive (n)

### configuration

conf parameter:

- full-text-urls-recognition — on or off

### data storage

The URLs are stored in the ft_urls table. The user decisions (choices) are stored in the ft_urls_choices table. (See below.) They are not stored in userdata! (Contrary to where every other piece of data given by users is stored.)

### data tables

table ft_urls:

- dsid char(15) not null
- url blob not null
- checksum char(16) binary not null
- nature ENUM('authoritative','automatic') not null
- source varchar(255) not null

`PRIMARY KEY( dsid, checksum ), index url_i(url(30)), index source_i(source(50))`

table ft_urls_choices:

- dsid char(15) not null
- checksum char(16) binary not null
- psid char(15) not null
- choice char(2) not null — first char: d|y|r|n (corresponding menu items: 2|4|3|1); second char: y|c|n (1|2|3)
- time datetime not null

`primary key prim(dsid, checksum, psid), index t_i(time), index psid_i(psid)`

### implementation components

1. input processing, ft_urls table making

2. recording user choices, inserting into ft_urls_choices

3. reading previous user choices

4. user screen

One idea is that we can create a special vacuum process for the ft_urls_choices table. It would scan the table and if for the same person/document/url there are several decisions, it would delete the old ones, and only leave the most recent one.

(it could also move the old ones to a separate archive table, optionally.)

## Document to document links

### TODO

- screen: improve destination document listing (menu); for instance, follow Sune’s recommendation. Or Thomas’ idea of alphabetical order.

- use Ajax to send the new link data and update the page?

- use Ajax to delete a link?

- document

- fix XXX marks?

### notions

link type — has a name, description and, possibly, a reverse link type name; some link types have no reverse.

### configuration

- document-document-links-profile — yes / no

$HOME/doclinks.conf.xml.eg is an example configuration file. Rename to $HOME/doclinks.conf.xml and edit to your liking.

### components

1. get configuration

   - parse it (ACIS::DumpXML::Parse)

   - check it (validate)

   - query configuration:

     $lab = $conf->type(‘follow-up’)->label; $rev = $conf->type(‘follow-up’)->rev; $rnm = $conf->type(‘follow-up’)->rev->name;

     or

     $lab = $conf->label(‘follow-up’); $rna = $conf->rev(‘follow-up’); $rla = $conf->revlabel(‘follow-up’);

2. get all current links for a record - an optional, but potentially useful abstraction level

   my $l = getdoclinks( $record ); savedoclinks( $record, $l );

3. get current links for a document

   $l->fordocument( $dsid ); $l->todocument( $dsid ); #? $l->fromdocument( $dsid ); #? $l->label($type) $l->allexpanded; $l->all_compact; $l->count;

4. create a link, drop a link

   $l->add($dsid1,$type,$dsid2); $l->drop($dsid1,$type,$dsid2);

5. screen

   - test on IE/Windows and in other browsers

### screen design

- use Yahoo! UI javascript library?

- use http://script.aculo.us/?