wiki:QemuGiftServer

Installing GNU Gift on QemuGiftServer

Assumptions

We're going to assume:

  • you're using a virtual machine manufactured to the specifications of the FAI configuration maintained by user juri_ at gitorious.org

Preparing Source

Following the directions on Gitorious, use git to download the newest version:

mkdir -p /home/gift/gift/src/orig/
cd /home/gift/gift/src/orig/
git clone git://gitorious.org/gnu-gift/gnu-gift.git gnuift-0.1.14

Create an original tarball, for use while building debian packages:

tar -cvzf gnuift_0.1.14.orig.tar.gz gnuift-0.1.14/

Building from Source

Please note that this is here for completeness. GNU GIFT is buildable as a debian package, and that process is described below.

Generate files required to build:

cd gift
./bootstrap-cvs.sh

Set up the build system:

./configure

Build Gift:

make

Building a Debian Package

First, get the debian/ directory to place into our source tree, for debian package building.

mkdir -p /home/gift/gift/src/debian_package/
cd /home/gift/gift/src/debian_package/
git clone git://gitorious.org/gnu-gift-debian-package/gnu-gift-debian-package.git debian
tar -cvzf gnuift_0.1.14.debian.tar.gz debian/

Grab the gift tarball we created earlier

cp /home/gift/gift/src/orig/*.tar.gz /home/gift/gift/src/debian_package/

Extract gift, adding a debian control directory.

tar -xzf gnuift_0.1.14.orig.tar.gz
cd gnuift-0.1.14
tar -xzf ../gnuift_0.1.14.debian.tar.gz

Finally, build the package.

debuild

Installing Your New Debian Packages

Since image indexing takes at least twice(!!) as long with whats in debian, versus the packages we just built, make sure to install the packages you just created.

sudo dpkg -i /home/gift/gift/src/debian_package/*.deb

Setting up the YACMS GIFT Frontend

First, check out the YACMS PHP5 Framework.

cd /var/www/
git clone git@gitorious.org:yacms/yacms.git yacms

Next, check out the yacms module containing the YACMS GIFT Frontend.

cd modules
git init .
git remote add origin git://gitorious.org/rainbowsock/elishys-rainbowsock.git
git remote update
git pull

Copy the main configuration file, and edit it.

cp configs/main.conf.example.php configs/main.conf.php

make sure to add the 'session', 'ystyle', 'input', and 'js' classes to the autostart list.

Copy the example database configuration file, and edit it.

cp configs/database.conf.example.php configs/database.conf.php

Set up a mysqli interface to localhost. set it so that user yacmsgift with password fai accesses a database named 'yacms_gift'.

install mysql.

sudo apt-get install mysql-server php5-mysql

Now, create a database and a username/password to back the database settings we just created.

cd /var/www/yacms/modules/sql/
mysql -u root -p < users_and_databases.sql
mysql -u yacmsgift -p yacms_gift <  yacms_gift.sql

Setting up a Frontend

Remove the default "it works!" page.

sudo rm /var/www/index.html

Add an alias for /home/gift/gift, to serve the images and their thumbnails to the public through apache.

Add the following in /etc/apache2/sites-available/default, between the VirtualHost? tags.

    Alias /gift/ "/home/gift/gift/"
    <Directory /home/gift/gift/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

Improving the Frontend

See GiftNewFrontend?

Using GIFT for sorting images

This uses a tag-and-move approach, to draw down from a large pool of images into smaller pools of sorted images.

Import Batch of Images

DONE: We need a web interface for importing a collection of images, that stores image metadata (relationships of one image to another, whether an image is part of the image database or a 'stock' image, tags, etc) in an sql database.

Create a default set that is all images in the imported collection, named 'collectionname-origin'. This set should be read-only.

Sort Images into Sets

Using the 'search' page, decide form type by making a search that only returns forms of the type you are looking for. To do this, run a search, and create a set by using that search's results, plus a floor value for how alike of results to tag.

cutting forms up

Create a new collection out of the result set, instead of using convert to resize the entire image in the import script, crop it to known region.. (look at css for defining regions?)

This will cut the field segments we want out. you may want to import them into a new collection, along side some 'known' field images (known signatures, printed client IDs, etc)

Getting Results

Compare the imported/generated images with the cut fields. matches should be in the high percentage with matching fields. Use this corelation to file forms by field match.

Obtaining Sample Image Data

I grabbed my sample image data from The Open Clip Art Project.

OpenClipArt? 0.19

Specifically, I downloaded their 0.19 release, then ran the following command, to move all the .png files into one flat directory, for gift to index:

cd /home/gift/
wget http://www.openclipart.org/downloads/0.19/openclipart-0.19.tar.bz2
tar -xjf openclipart-0.19.tar.bz2
mkdir -p gift/openclipart-0.19/
cp -a `find openclipart-0.19 -name *.png` gift/openclipart-0.19/

OpenClipArt? 2.0

Download the 2.0 release, then move all the .png files into one flat directory, for gift to index:

cd /home/gift/
wget http://www.openclipart.org/downloads/2.0/openclipart-2.0-full.tar.bz2
tar -xjf openclipart-2.0-full.tar.bz2
mkdir -p gift/openclipart-2.0/
cp -a `find openclipart-2.0-full -name *.png` gift/openclipart-2.0/

Indexing your Collection

To index your collection, first copy in a default configuration file, then run gift-add-collection with the full path of the images you want to import.

The gift packaged in lenny and the current CVS version fail to create a config file. copying the file from the default, before running gift-add-collection.pl works around the problem.

cp /usr/share/libmrml1/gift-config.mrml /home/gift/
gift-add-collection.pl /home/gift/gift/openclipart-*/

now wait a long, long time.

Fixing Problems

Failure to add Collection ID

PROGRESS: 99%


Copying /home/gift/gift-config.mrml to /home/gift/gift-config.mrml-old

XML::DOM::Attr=ARRAY(0x9e06b78)
Can't locate object method "getAttribute" via package "XML::DOM::Attr" at /usr/b
in/gift-add-collection.pl line 855, <LOCALELIST> line 274.
----> collection-id c-59-50-8-18-9-111-2-290-0  <----

This happened after a 'successful' run, where the mergesort had gone sideways. i had removed the gift data directory, and forgotten to restore the gift-config to default.

to fix it, since the collection had already written out this file successfully with the wrong collection ID, it is meerly required to find the collection ID in the file, and replace it with the one presented in the error message (not the one from this site!).

MergeSort? Blowup

...finished
before mergesort
Starting quicksort: 1048576 elements per page.
Sorting files /home/gift/gift-indexing-data/images//gift-auxiliary-1
to            /home/gift/gift-indexing-data/images//gift-auxiliary-2
NOW ALLOCATING A PAGE1048576
HIERFIRSTLEVELQUICK226868124;0
................gift-generate-inverted-file: ../../libGIFTAcInvertedFile/include
/merge_sort_streams.h:282: void first_level_quicksort(int, const char*, const ch
ar*) [with T = CIFBuilderTriplet]: Assertion `lTemporary' failed.

PROGRESS: 99%

This was the result of running out of disk during the mergesort. removed everything, freed up some disk space, and started over.

Out of Memory

PROGRESS: 99%

Copying /home/gift/gift-config.mrml to /home/gift/gift-config.mrml-old

Ran out of memory for input buffer at /usr/lib/perl5/XML/Parser/Expat.pm line 469, <LOCALELIST> line 270.                                                             

Run out of memory? create a 256 meg swap file, and use it:

dd if=/dev/zero of=/home/gift/swapfile bs=4k count=65536
sudo mkswap /home/gift/swapfile
sudo swapon /home/gift/swapfile

now skip down to the section on re-starting your run.

Infinity when re-starting a run

STARTING mit MERGESIZE1
MERGESORT MergeSize 12
endmerge
after mergesort. The last file I used was /home/gift/gift-indexing-data/clipart/
/gift-auxiliary-2
Opening sorted stream for reading. State (should be '1'): 0xbff84bec
[inFeatureID:4/0;inPosition:16/0==0]20
Writing Chunk for Feature ID 0. The Offset is 0x0=0
The collection frequency is: inf
gift-generate-inverted-file: CInvertedFileChunk.cc:117: bool CInvertedFileChunk:
:writeBinary(std::ostream&, TID, size_t) const: Assertion `!"collection frequenc
y out of range"' failed.


PROGRESS: 99%


Copying /home/gift/gift-config.mrml to /home/gift/gift-config.mrml-old

Ran out of memory for input buffer at /usr/lib/perl5/XML/Parser/Expat.pm line 46
9, <LOCALELIST> line 274.

This happened when I re-ran gift-add-colllection after a failed run due to no config file.

because it did not regenerate any .fts files, the script fails the next step.

Fix the root cause of the failure, then follow the directions on re-starting a broken run:

cp /usr/share/libmrml1/gift-config.mrml /home/gift/

Re-starting A Broken Run

Assuming you have fixed the root cause of whatever failure you encountered...

Backup the completed mapping between URIs and feature files.

cp /home/gift/gift-indexing-data/openclipart-*/url2fts.xml /home/gift

Next, clean out generated files.

rm /home/gift/gift-indexing-data/openclipart-*/InvertedFile*
rm /home/gift/gift-indexing-data/openclipart-*/gift-aux*
rm /home/gift/gift-indexing-data/openclipart-*/00*

re-run gift-add-collection, and place the backup url2fts.xml file in place.

gift-add-collection.pl /home/gift/gift/openclipart-*/
cp /home/gift/url2fts.xml /home/gift/gift-indexing-data/openclipart-*/

now, get a count of images in your image repository, and insert the count into /home/gift/gift-config.mrml.

cat gift-config.mrml | sed 's/images="[0-9]*/images="'`find gift/openclipart-*/ -type f | wc -l`'/' > gift-config.new
cp gift-config.new gift-config.mrml

Finally, have gift-add-collection update the collection ID in the configuration file.

gift-add-collection.pl -fix-config /home/gift/gift/openclipart-*/

Importing into OpenEMR

sort forms based on type, AND field match. perform action with form based on form type.

Last modified 8 years ago Last modified on Dec 4, 2013, 6:12:31 AM