Workpackage 4 - Image Analysis
User Requirements Analysis
Kirk Martinez, Paul Lewis and Steve Chan
Analysis of Requirements
The results of Work Package 2 on gathering user requirements have led to a table of goals, from which a set of requirements for image analysis have been constructed. Below is the original Goal list with inserted notes on the possible image analysis approaches. Some requirements are actually composed of sub-requirements, especially G4.
G1 Matching of similar images (includes "have you got this picture")
aspect ratio, histograms, colour regions, edge maps, multiresolution shape finder?
Also finding "Lost" images
G2 Automatic search using synonyms
G3 Search based on the concept of style
depends what sort of "style", texture classifier, colour + texture classifier, neural networks to analyse image features
shape styles etc – need shape detectors.
Artist technique (?)
G4 Search based on features oriented to the restoration framework
histogram based threshold plus counter, binary shrink to count large regions only?
x-ray and reverse pic views of frames
line detectors, hough?, lp filter first?
edge/line detectors tuned to cracks, chain + classify? or texture of binary result? Snakes?
Search based on framework features
"butterfly", wood knots, planks, retouching detectors. These use rear-view images and require shape detectors.
Parquetage, stretchers, bars.
G5 Access information quickly and easily
Avoiding brute-force searching and image processing during queries. Fast indexing strategies required. May use Java Viseum image viewer for browsing large images.
G6 Search based on colour
Image with these colours? (pantone or Lab or sRGB)
convert to sRGB or Lab, look for colour regions at low res
3D colour histogram, similarity matrix
CIE/picked colour (plus damage): as above plus damage detector appropriate
G7 Query by low quality images (especially faxes)
scalespace versions of edge maps
G8 Query by sketch
scalespace edge map search, shapes searches?
G9 Query refinement
G10 Joint retrieval by content and by text
G11 Use of publishing products built on the Artiste system
G12 Detail finding
(eg finding which image a close-up is from): CCV , template matching?: edge map?(scalespace) searching
G13 Search using multilingual vocabulary
G14 Respect installation site privacy and security policy
G15 Produce a sustainable system after the end of the project
Document algorithms and code. Help system.
G16 Be consistent with partner’s pre-defined technical constraints.
G17 (new!) shape of the painting - eg oval, round top etc. (also useful for restricting image matching to real areas)
G18 (new!) - if time allows! - Canvas Weave count. (possibly using X-ray image)
It is noticeable that there are vague requirements about speed, success rate and accuracy as well as human interface issues. It is expected that systems will initially be slower than expected but benefit eventually from faster hardware and software optimisation. It is also hoped that HCI issues will surface as users start to use the system, which is probably their first time with such a system.
We will identify the most commonly needed algorithms and implement these first, as many solutions are based on a combination of functions. Some are available in VIPS or MAVIS already, while others will need coding from scratch. They will be written in C/C++ under Unix for performance reasons, in such a way as to be portable to NT. A project web page will contain the latest software for download, together with help pages and an API.
Goals will be tackled in roughly this order: 1, 12, 6, 4, 7, 8, 3. This will depend on the success rate of algorithms as they are implemented as it is possible that a solution for one goal is found while tackling another.
Clearly multiresolution/multiscalar techniques will be involved due to the nature of some of the problems (eg find where this detail image is from) as well as the range of feature scales (eg small cracks on a 10kx10k image). From the list above certain types of algorithm are needed:
Many of these are found in any image processing package but the rest are more specialised or even problem specific variants.
Image processing in artiste
As images are uploaded to the database they are passed to image processing functions which generate image feature vectors (IFVs). These will be stored in the database and comparison functions provided for searching using them. It seems that blobs will be used for the data or one of the built-in types (eg int arrays).
An API will be agreed on for the image processing functions which produce the IFVs. There should be a list of functions, descriptions of their strengths etc, data definitions (for the db fields). Then there is the list of associated "matchers" which produce scores given sets of test images.
To reduce compute-intensive operations the database will store image primitives which are higher-level and faster to compare if possible. For example histograms, averages, perhaps low-res edge maps, FFT, 3D colour histogram etc.
Starting with the simplest image processing and moving through to the latest, we expect a growing list of IFVs like:
aspect ratio, h/w, float
mean, mean of all pels, int
realscale eg mm/pel?
FFT power spec, intint
texture... various, int
colour texture measures
spatial colour histograms
In ECS/IAM we will be cable to develop the image processing funtions and test them standalone or with MAVIS2. Then wrap them for the NCR database, or our next version of MAVIS using our database infrastructure.