Shared Development

MagicMimeTypeIdentifier update

2007-02-11T07:02:00.000-05:00

MagicMimeTypeIdentifier, which scored so highly on my comparison of Java file type detectors, has been updated. Here is the email I received from Christian Fluit:

I have just updated Aperture's MagicMimeTypeIdentifier based on the results of your benchmark, in order to achieve the best score possible. This led to the addition of the following MIME types:

audio/x-aiff
audio/x-ms-wma (previously mistakenly labeled as audio/x-ms-wmv)
application/x-ms-wm (artificial supertype of wma and wmv, they share the same magic number and can only be distinguished when they have the proper file name extension or when you interpret the container's contents in more depth)
image/svg
image/x-icon
image/x-raw
image/x-tga
application/x-freemind

Also, some MIME types had their description updated, e.g. .rmi files are now also labeled as audio/midi.

Unfortunately, a 100% score is not achievable at the moment, as the magic number of TGA files also matches with that of certain versions of Quattro Pro spreadsheets. At the moment this cannot be expressed in the identifier's config file, you would almost need a rule based language to express this.

Running the test with the new version of MagicMimeTypeIdentifier (available from the CVS repository) lead to a great improvement: 95% accuracy! Most of this improvement came from recognition of the SVG and ICO files. Great job Chris!

Chris also mentions in his email:

A remark about your benchmark: I noticed that all your files had the proper file extensions. Our MIME type identifier primarily uses magic numbers and only switches to checking file extensions when magic number matching fails or when it is unable to discriminate between a family of related file formats (e.g. the MS Office formats). I wonder what the outcome of your test would be if you would remove all file name extensions :)

Renaming the files without an extension reduced the accuracy of MagicMimeTypeIdentifier to only 78%--a small decrease from 84% accuracy with file extensions. Still, this beats the heck out of other detectors.

Current project deadlines prevent me from doing more comprehensive testing, but I would like to thank Chris for his response.

Microsoft Photo Info

2007-01-24T16:19:00.001-05:00

Microsoft Photo Info is a new software add-in for Microsoft Windows that allows photographers to add, change and delete common "metadata" properties for digital photographs from inside Windows Explorer.

Visualization Links

2007-01-24T16:02:00.000-05:00

Swivel is a Web site for curious people to explore data. They "use farms of powerful computers and algorithms ... to transform a lonely grid of numbers and letters into hundreds - sometimes thousands - of graphs that can be explored and compared with any other public data ... have ratings and comments and publishing shortcuts for bloggers, so folks can share ideas, talk about insights and understand data together ... we transform the sometimes tedious task of reading someone else's spreadsheet into a fun experience of clicking through a Web site full of images, graphs and color."

Gapminder "is a non-profit venture for development and provision of free software that visualize human development. This is done in collaboration with universities, UN organizations, public agencies and non-governmental organizations. The main project during the coming three years is a collaboration with UN Statistic Division with the aim to visualize UN common database..."

The IBM Visual Communication Lab "develop[s] visualization algorithms that help people see and exchange information in novel ways. Our designs aim to transform visualization from a solitary activity into a collaborative one. Some application areas are online discussions, email archives, social networks, software development, and executive decision support tools. By allowing people to observe and orient themselves in complex information landscapes, our inventions enable faster, more insightful decisions."

Bauhaus-Universität Weimar

2007-01-22T07:33:00.000-05:00

I recently stumbled upon Bauhaus-Universität Weimar (english), a univerisity for creative studies in Weimar, Germany. The university conducts research in Web Technology and Information Systems.

The site has information about:

The AItools suite which addresses text-based information retrieval tasks. It is comprised of basic and advanced algorithms, data structures, and design patterns to model complex real-world retrieval processes.

The AIsearch mining tool for the intelligent analysis of document collections. It offers a convenient interface for Web-based search and combines algorithms for the formation, labeling, and visualization of categories along with a smart spelling analysis.

The International Workshop on Text-based Information Retrieval which addresses researchers, users, and practitioners from different fields: data mining and machine learning, document and knowledge management, semantic technologies, computer linguistics, and information retrieval in general.

The TIR workshop is occurs in conjunction with DEXA which also hosts other interesting workshops on data processing, data management, data mining and retrieval, semantics, knowledge, self adaption, and autonomic computing.

File Type Metadata Discovery, Part 2: Images

2007-01-19T07:23:00.000-05:00

File Type Metadata Discovery, Part 2: Images

In a previous article, I evaluated various libraries to determine which most accurately identified a file's type. This article represents part two in a series of articles that explore how to discover metadata about a file after its type has been detected.

Java ImageIO

The primary Java library for image handling is javax.imageio which provides a pluggable architecture for working with images stored in files and accessed across the network and a framework for the addition of format-specific plugins. Plug-ins for several common formats are included with Java Image I/O, but third parties can use this API to create their own plugins to handle special formats.

There is also a jai-imageio project on java.net which is a set of ImageReader and ImageWriter plugins for the ImageIO API, primarily built by the JAI team (see this thread at java.net).

The javax.imageio has comprehensive metadata capability included in the ImageReader.getStreamMetadata() and ImageReader.getImageMetadata() methods. These methods generate an IIOMetadataFormat object whose values are accessible through a DOM tree. The amount of image specific metadata available is staggering, and probably wouldn't be useful to someone unless they were creating a very image-centric application. However, in a digital asset management system, simple metadata such as height and width are readily available from ImageReader.getImageMetadata().

ImageMetadataDiscoverer

I have created a simple library that gathers all available image metadata into a Map set. Download ImageMetadataDiscoverer at Sourceforge or read the javadocs.

Need More?

More specific needs can be met by the wide variety of image tools available for Java. Marco Schmidt has a nice list of raster and vector libraries. DMOZ also maintains a directory of libraries.

The Future of Semantic Search

2007-01-17T15:38:00.000-05:00

Steven Arnold recently stated some facts that are very closely related to my project and research interests:

...what will carry us into 2007 is a collection of technologies we think of as text mining, where software algorithms look at documents and find the names of people, places and things and attempt to relate them to one another.

...companies like Attensity Corp. and nStein Technologies ... are focused on figuring out the nuances, relationships and the important concepts in a document. Their systems generate index terms that an enterprise search system can suck in.

...new companies ... are approaching the problem both mathematically and by doing vocabulary and knowledge-based analysis. Their software decomposes sentences into subjects, verbs and adjectives and analyzes the results with the predictive algorithms.

Wikiseek

2007-01-17T15:02:00.000-05:00

A newly launched service called Wikiseek focuses on complimenting Wikipedia by restricting search results to articles and references in the encyclopedia. Wikiseek's about page claims that this method makes it "an authoritative source of information less subject to spam and SEO schemes."

Wikiseek suggests search refinements "based on user tagging and categorization within Wikipedia." When I do a search for the word "apple," Wikiseek returns a category cloud displaying categories which which my search phrase appears most often. The "Apple hardware" category shows most prominently. Actually, I was more interested in the fruit because I had one with lunch today. Clicking on the fruit category gives me a list of those references in Wikipedia that refer to "apple" as a fruit. Pretty nifty! If I go back and choose "Apple hardware" I see the expected list of articles about current and legacy Apple products.

There are some instances of strange results, such as the search for "wiki," and comments at the TechCrunch article are quick to point out that since Wikipedia is editable, article spammers now have more incentive if their actions will effect a Wikipedia-specific search engine.

Personally, I find the category refinement feature and reference results useful although not enough to build an entire site around. It would be better utilized as a module in a large package... perhaps Wikipedia itself.

The Future of Enterprise Search

2007-01-15T13:27:00.000-05:00

Dana Gardner of Interarbor Solutions recently interviewed members of FAST Search & Transfer. Their discussion brought up several interesting topics about semantic, search-centered applications.

Dr. Bjørn Olstad (CTO) makes the following insightful remarks:

The fundamental difference between search and such things as database, content management, and document management systems is that search starts with the user and then does reverse engineering -- what is necessary to realize this user's experience. It's starting with the content and then trying to deduce what should we do with this content. When you do that, you end up with a user-driven experience.

...increasingly the user experience will be driven by algorithms and will be dynamic, so that you can actually optimize the user experience...

...search can auto-generate metadata and find ways to use that structure and to improve the discovery. Then, the allocation elements that the traditional Semantic Web talks about can be aimed at how to improve algorithms, as opposed to starting from scratch. In doing that, I think search has the opportunity to deliver on the premise of the Semantic Web, by applying algorithms as opposed to altering tools.

Technically, if you use an open-ended query like, "List the innovative people in my company," or something, you could get back kind of a menu of people that have been referred to in the documents or discussions where there is talk of innovation, and then get the facts related to these people. So it’s not coming back with an answer, but it is coming back with an analysis -- and giving you the opportunity to refine the query.

I really liked Zia Zaman's ideas as well (Vice President, Strategic Market Development):

I often like to talk about Greek philosophers, and Socrates is one of my favorites. The reason is because of what he did with a seeker. He didn’t answer a question with an answer. Rather he asked another question, and that allowed the seeker to refine his or her question, until finally they got to what they were looking for. In many ways, technology -- whether it’s information discovery, search, and business intelligence, whatever it might be -- shouldn’t insult the intelligence of the seeker. Rather it should allow individuals to find the answer that they’re looking for, either through tacit information that’s stored in the enterprise, or semantic information, or structured information, whatever it might be. What we’re trying to do is mimic the type of dialogue that Socrates had.

Listen to the podcast or read the transcription...

Free Music Identification and Metadata Service

2007-01-12T14:45:00.000-05:00

Yesterday I briefly mentioned that there were services that could help you identify non-technical audio metadata. I just become aware of a press release from MusicIP that announces their free music identification service that provides track metadata.

Check out the press release or the MusicIP website for more info.

Flexible Taxonomies

2007-01-11T15:23:00.000-05:00

In Web Analytics And Content Group Management, Gary Angel writes about the use of taxonomies in web analytics, stating that "no single taxonomy is likely to support a very wide range of analytic problems. [...] there are only taxonomies appropriate for more or fewer analytic problems."

He mentions the common yet limiting use of navigational taxonomy and points out that the analysis process/application should "be able to construct multiple 'point' taxonomies that can be used for specific analytic purposes [...] The combination of a graphical drag-and-drop interface, ability to apply regex rules and the ability to create analysis specific taxonomies on the fly [...] would make it relatively easy to 'manufacture' a taxonomy for analysis..."

Angel is proposing a method of flexible taxonomy creation, which would be useful not only in web analytics, but also in the analysis of any type of document collection, including business, scientific or humanities.

So how is a flexible taxonomy implemented? Angel mentions that taxonomy often needs to be based on the content of page instead of its function. The semantic content of a page can be determined by computational linguistics, which includes natural language processing and semantic relatedness/differential.

Theoretically, a document (or even a paragraph) could be analyzed to determine its semantic meaning. The semantic data could then be used to build a taxonomy, either automatically without user-intervention, or as Angel proposes, dynamically, allowing the analyst to determine what characteristics define their custom taxonomic unit.

For example, in building a taxonomy that describes communications technology, an analyst would undoubtedly include Apple's new iPhone. Depending on its commercial success, the present and future technology included in the iPhone could heavily influence the rest of the communications industry. Semantic analysis of documents describing the iPhone would reveal that its multi-touch interface is, in part, the result of research conducted by Jefferson Han at NYU. New taxonomic units could then be created that classify the various influences on a particular technology, such as market, military or academic.

I greatly appreciate Angel's thoughtful remarks about flexible taxonomies. The fields of web analytics and document management obviously have much in common.

File Type Metadata Discovery, Part 1: Audio

2007-01-11T07:20:00.000-05:00

In a previous article, I evaluated various libraries to determine which most accurately identified a file's type. This article represents part one in a series of articles that explore how to discover metadata about a file after its type has been detected.

Audio Metadata

Based on the nature of digital audio Wikipedia cites "sample rate, resolution and number of channels" as important audio file format parameters. This type of technical metadata is important in classifying and organizing digital assets. Determining non-technical metadata, such as title, author and date are beyond the scope of this article, although there are many resources that address this type of metadata discovery.

Much of Java media development revolves around playing, streaming, recording and editing. However, only metadata discovery is consider in digital asset management (and in this article).

Discovering Audio Metadata with the Java Sound API

According to Sun, the Java Sound API "provides low-level support for audio operations." The javax.sound.sampled.AudioFormat and javax.sound.sampled.AudioFileFormat both allow access to an audio file's metadata. Using the static methods of the javax.sound.sampled.AudioSystem class, we can get an AudioFileFormat based on a File, InputStream or URL. An AudioFormat is obtained by invoking the AudioFileFormat's getFormat() method. Java Sound Resources provides an excellent example of how this is done. In summary, the following list shows what metadata is accessible through the Java Sound API.

AudioFileFormat
getType()	audio file type, such as WAVE or AU
properties()	unmodifiable map of properties that specify additional informational meta data (like a author, copyright, or file duration). Properties are optional information, and file reader and file writer implementations are not required to provide or recognize properties
getByteLength()	size in bytes of the entire audio file (not just its audio data)
getFrameLength()	length of the audio data contained in the file, expressed in sample frames
AudioFormat
getChannels()	number of channels
getEncoding()	type of encoding for sounds in this format
getFrameRate()	frame rate in frames per second
getFrameSize()	frame size in bytes
getSampleRate()	sample rate
getSampleSizeInBits()	size of a sample
properties()	an unmodifiable map of properties
NOTE: The duration of an audio file (in seconds) can be computed by multiplying the frame length by the frame rate.

The Java Sound API appears to meet our needs well, however, it only supports AIFF, AU, WAV and some MIDI based formats! To remedy this, the API provides a service provider interface (SPI) to support more file formats. MP3, OGG, APE (Monkey's Audio), FLAC and even Speex SPIs are available. An alternative implementation of the Java Sound API, Tritonus, also provides some SPI plug-ins (the OGG SPI requires Tritonus).

It should also be noted here that an SPI will implement the properties() method of the AudioFileFormat and AudioFormat classes. The properties returned may include non-technical metadata, but this is solely dependent upon the audio file format.

What about other Java media libraries?

According to Sun, the Java Media Framework (JMF) "can capture, playback, stream, and transcode multiple media formats." In comparing the JMF to Java sound, we find that the JMF does have more codecs, however the ability to capture audio specific metadata is limited. The focus of the JMF is not on metadata. At least from what I can tell. If anyone has found the case to be different, please let me know. The same goes with the FMJ and Enterprise Media Beans.

Conclusion

The Java Sound API currently provides the best means of discovering an audio file's technical metadata.

Centiare on the Heels of Wikipedia

2007-01-05T18:54:00.000-05:00

"semantic web" technology is installed on Centiare. This means registered users can perform amazing searches on Centiare that just wouldn’t be possible on Google, MySpace, or Wikipedia. Imagine multi-level searches for very specific things, like:

* Are there any white males, over 6 feet tall, with a Masters degree, in Pennsylvania?
* Locate all home heating oil companies, at least 50 years in business, in New Jersey.
* Find all Centiare Directory entities situated between the 39th and 40th parallels.

These searches would be practically impossible in a wiki database without semantic web enabled. It's no wonder that the inventor of the World Wide Web, Tim Berners-Lee, has literally written a roadmap for semantic web -– he knows it's that important. And Internet prognosticators are finally agreeing that in 2007, semantic web will usher in the "Web 3.0" era.

File type detection follow up

2007-01-03T11:25:00.000-05:00

earl.strain.at ran the test files used in my file type detection article against file, "the open source implementation of the file(1) command that ships with every free operating system."

Read about his results...

Intellisophic Achieves Patent Milestone on Document Indexing System & Methods

2007-01-02T13:52:00.000-05:00

Intellisophic Inc., a leading provider of information products to the search and text mining industry, announced that it has been granted allowance by the U.S. Patent and Trademark Office on the systems and methods used for indexing documents based on relevance to reference publications such as a book, web page, newspaper or encyclopedia. The patent also covers core technology relating to generating keyword indices and topic hierarchy from the reference materials for the purposes of scoring or matching other documents based on relevance.

Read the full article at TechWeb...

Hakia: A New Google?

2007-01-02T13:37:00.000-05:00

"...the triumph of a semantic search technology over the irrelevance of syntactic search illustrates why companies like Hakia will garner more attention in 2007."

An article on Line56.com discusses the power of semantic search.

Information Scarcity to Information Overload

2007-01-02T11:10:00.000-05:00

"The focus in enterprise content management (ECM) is shifting from ending information scarcity to dealing with information overload. This dynamic explains why the disparate technologies of search, records management and analytics are now hot."

Read full article at DMReview...

Character encoding detection

2007-01-01T13:06:00.000-05:00

Problem

Documents that are stored as plain text, such as XML or XHTML, often have a particular character encoding, also known as a character set or codepage. This character encoding allows applications to identify how characters should be displayed. This is especially the case with CJK languages.

Detailed analysis of the problem is available in A composite approach to language/encoding detection by Shanjian Li and Katsuhiko Momoi (2001). Li and Momoi's approach has become Mozilla's Universal Charset Detector.

Implementations

jchardet is a Java port of Mozilla's character set detection algorithm. (MPL)

International Components for Unicode (ICU) is a set of C/C++ and Java libraries for Unicode support, software internationalization and globalization (i18n/g11n). It grew out of the JDK 1.1 internationalization APIs, which the ICU team contributed, and the project continues to be developed for the most advanced Unicode/i18n support. ICU is widely portable and gives applications the same results on all platforms and between C/C++ and Java software. (X license)
cpdetector is a Java framework for configurable code page-detection of documents. (MPL)
monq.stuff.EncodingDetector is part of the Java Finite Automata class library from the European Bioinformatics Institute. (GPL)
com.sun.syndication.io.XmlReader handles the character encoding of XML documents in Files, raw streams and HTTP streams by offering a wide set of constructors. Part of the ROME project for reading RSS and Atom feeds. A nice explanation of how ROME detects character encoding is available here. (Apache License)
com.glaforge.i18n.io.CharsetToolkit is a utility class that guesses the charset used in a byte buffer. (Unknown license)
MLang is a MSDN library that lists, “detection of which possible code pages and languages text data is written in,” as one of its features. (Microsoft Windows DLL)
Charset Detector is a stand alone executable module for automatic charset / encoding detection based on Mozilla's i18n component. It can be compiled for MS Windows using Delphi or Free Pascal or Linux using Delphi/Kylix. (LGPL)

Test Results

Click for full-size image

Raw data (PDF) | Source and test files

The HTML Character Set Tables from Columbia University all have META tags that specify the encoding. cpdetector did an excellent job in recognizing this while the other detectors failed to do so.

The XHTML files created from the Character Encoding Test Page all have XML prologs and META tags that specify the encoding. cpdetector and monq.stuff.EncodingDetector both shine here, while com.sun.syndication.io.XmlReader generated several java.io.UnsupportedEncodingExceptions.

The Japanese XML files from the W3C (“pr-” and “weekly-” prefix) were best handled by ICU and com.sun.syndication.io.XmlReader, despite the fact that XML prologs were not always available.

Finally, the Chinese TXT and XML files (“zh-” prefix), which all have XML prologs, were handled by cpdetector, monq.stuff.EncodingDetector and com.sun.syndication.io.XmlReader.

Conclusion

The strength of a character encoding detector lies in whether or not its focus is on statistical analysis or HTML META and XML prolog discovery. If you are processing HTML files that have META, use cpdetector. Otherwise, your best bet is either monq.stuff.EncodingDetector or com.sun.syndication.io.XmlReader.

What about files that have no META or prolog? That is a big question. If an XML file has neither, but contains multiple languages, such a Chinese content and English markup, a statistical analysis skewed by the more prevalent markup language may not be enough to display a document properly. This is where Unicode comes in. Unicode with multiple languages is handled by the application used to render the document. Check out Wikipedia's entry on Multilingual support.

Test files

Other Useful Resources

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) By Joel Spolsky, Wednesday, October 08, 2003
Letter Database lists the characters and corresponding code pages for specific languages.
W3C I18N tutorial: Character sets & encodings in XHTML, HTML and CSS
netscape.public.mozilla.i18n has been abandoned but replaced by mozilla.dev.i18n
XHTML Test Cases
MIME Test - Character Encoding Test

While doing research, I used the following phrase in Google:
("character set" OR "charset" OR "codepage" OR "character encoding")

File type detection

2006-12-26T14:52:00.000-05:00

NOTE: Since this article was written, updates have been made to the MagicMimeTypeIdentifier in the Aperture Framework. Read more...

Problem Description

My current project, vyasa, is a digital library management system. One of the features of a digital library is the ability to recognize the types of files (digital assets) that are loaded into the repository.

The process of detecting a file's type (also known as a file's MIME) is non-trivial, yet there are important benefits. For example, type-related metadata, such the length and bitrate of an audio file, or the size and DPI of an image file, lead to comprehensive asset management.

Possible Solutions

The problem and various solutions to file type detection are briefly explained in this Wikipedia article. Detailed coverage is available in the following academic papers (PDF):

Content Based File Type Detection Algorithms, Mason McDaniel and M. Hossain Heydari, Computer Science Department, James Madison University, Harrisonburg.

File Type Detection Technology, Douglas J. Hickok, Daine Richard Lesniak, Michael C. Rowe, Ph.D., Computer Science and Software Engineering Department, University of Wisconsin-Platteville.

Java Implementations

javax.activation.FileDataSource is part of the JavaBeans(TM) Activation Framework used by the JavaMail(TM) API to manage MIME data.
Java Mime Magic, "retrieves file and stream mime types by checking magic headers," according to Réal Gagnon. (LPGL)
ffident is a Java metadata extraction, file format identification library created by Marco Scmidt. (LPGL)
JHOVE (JSTOR/Harvard Object Validation Environment) will identify, validate and characterize file types (LPGL).
MagicMimeTypeIdentifier is from the Aperture Framework. It determines the MIME type of a binary resource based on magic number-based heuristics. (AFL, OSL)
MimetypeRegistryService is part of the Nuexo project. (LGPL)

Non-Java Resources

complex.filter.detection.typeDetection is OpenOffice.org's code for detecting file types (LGPL).
Magic DB is a file containing magic numbers for identifying file type as well as several other file metadata. The format of the file is specified by Optima SC. (see license on magicdb.org)
Marco Schmidt lists several non-Java resources on his file formats page.
FileType is an internal filetype detection engine for other coders who wish to have a simple to use C module.
org.mmbase.util.magicfile determines file types based on a parsing of the UNIX magic command.

Character Encoding

Another important aspect of file type detection is character encoding (aka codepage) detection for plain text files such as HTML or XML. I will cover this topic in a future article.

Testing

A series of text, image, audio, video and "other" files were used to test the Java libraries listed above. The details of the files I used are available here. The results of the test are summarized in the chart below:

The detection accuracy of most libraries was less than 50%. However, the Aperture Framework's MagicMimeTypeIdentifier was extremely accurate. It was able to correctly identify many proprietary formats. The code used to perform the actual testing, along with the test files themselves are available here. More detail about the results of the tests are available in PDF form here.

Conclusion

MagicMimeTypeIdentifier from the Aperture Framework appears to be the most reliable and accurate file type detector.