The Future of Enterprise Search

Dana Gardner of Interarbor Solutions recently interviewed members of FAST Search & Transfer. Their discussion brought up several interesting topics about semantic, search-centered applications.

Dr. Bjørn Olstad (CTO) makes the following insightful remarks:

The fundamental difference between search and such things as database, content management, and document management systems is that search starts with the user and then does reverse engineering -- what is necessary to realize this user's experience. It's starting with the content and then trying to deduce what should we do with this content. When you do that, you end up with a user-driven experience.

...increasingly the user experience will be driven by algorithms and will be dynamic, so that you can actually optimize the user experience...

...search can auto-generate metadata and find ways to use that structure and to improve the discovery. Then, the allocation elements that the traditional Semantic Web talks about can be aimed at how to improve algorithms, as opposed to starting from scratch. In doing that, I think search has the opportunity to deliver on the premise of the Semantic Web, by applying algorithms as opposed to altering tools.

Technically, if you use an open-ended query like, "List the innovative people in my company," or something, you could get back kind of a menu of people that have been referred to in the documents or discussions where there is talk of innovation, and then get the facts related to these people. So it’s not coming back with an answer, but it is coming back with an analysis -- and giving you the opportunity to refine the query.

I really liked Zia Zaman's ideas as well (Vice President, Strategic Market Development):

I often like to talk about Greek philosophers, and Socrates is one of my favorites. The reason is because of what he did with a seeker. He didn’t answer a question with an answer. Rather he asked another question, and that allowed the seeker to refine his or her question, until finally they got to what they were looking for. In many ways, technology -- whether it’s information discovery, search, and business intelligence, whatever it might be -- shouldn’t insult the intelligence of the seeker. Rather it should allow individuals to find the answer that they’re looking for, either through tacit information that’s stored in the enterprise, or semantic information, or structured information, whatever it might be. What we’re trying to do is mimic the type of dialogue that Socrates had.

Listen to the podcast or read the transcription...

Free Music Identification and Metadata Service

Yesterday I briefly mentioned that there were services that could help you identify non-technical audio metadata. I just become aware of a press release from MusicIP that announces their free music identification service that provides track metadata.

Check out the press release or the MusicIP website for more info.

Flexible Taxonomies

In Web Analytics And Content Group Management, Gary Angel writes about the use of taxonomies in web analytics, stating that "no single taxonomy is likely to support a very wide range of analytic problems. [...] there are only taxonomies appropriate for more or fewer analytic problems."

He mentions the common yet limiting use of navigational taxonomy and points out that the analysis process/application should "be able to construct multiple 'point' taxonomies that can be used for specific analytic purposes [...] The combination of a graphical drag-and-drop interface, ability to apply regex rules and the ability to create analysis specific taxonomies on the fly [...] would make it relatively easy to 'manufacture' a taxonomy for analysis..."

Angel is proposing a method of flexible taxonomy creation, which would be useful not only in web analytics, but also in the analysis of any type of document collection, including business, scientific or humanities.

So how is a flexible taxonomy implemented? Angel mentions that taxonomy often needs to be based on the content of page instead of its function. The semantic content of a page can be determined by computational linguistics, which includes natural language processing and semantic relatedness/differential.

Theoretically, a document (or even a paragraph) could be analyzed to determine its semantic meaning. The semantic data could then be used to build a taxonomy, either automatically without user-intervention, or as Angel proposes, dynamically, allowing the analyst to determine what characteristics define their custom taxonomic unit.

For example, in building a taxonomy that describes communications technology, an analyst would undoubtedly include Apple's new iPhone. Depending on its commercial success, the present and future technology included in the iPhone could heavily influence the rest of the communications industry. Semantic analysis of documents describing the iPhone would reveal that its multi-touch interface is, in part, the result of research conducted by Jefferson Han at NYU. New taxonomic units could then be created that classify the various influences on a particular technology, such as market, military or academic.

I greatly appreciate Angel's thoughtful remarks about flexible taxonomies. The fields of web analytics and document management obviously have much in common.

File Type Metadata Discovery, Part 1: Audio

In a previous article, I evaluated various libraries to determine which most accurately identified a file's type. This article represents part one in a series of articles that explore how to discover metadata about a file after its type has been detected.

Audio Metadata

Based on the nature of digital audio Wikipedia cites "sample rate, resolution and number of channels" as important audio file format parameters. This type of technical metadata is important in classifying and organizing digital assets. Determining non-technical metadata, such as title, author and date are beyond the scope of this article, although there are many resources that address this type of metadata discovery.

Much of Java media development revolves around playing, streaming, recording and editing. However, only metadata discovery is consider in digital asset management (and in this article).

Discovering Audio Metadata with the Java Sound API

According to Sun, the Java Sound API "provides low-level support for audio operations." The javax.sound.sampled.AudioFormat and javax.sound.sampled.AudioFileFormat both allow access to an audio file's metadata. Using the static methods of the javax.sound.sampled.AudioSystem class, we can get an AudioFileFormat based on a File, InputStream or URL. An AudioFormat is obtained by invoking the AudioFileFormat's getFormat() method. Java Sound Resources provides an excellent example of how this is done. In summary, the following list shows what metadata is accessible through the Java Sound API.

AudioFileFormat
getType()	audio file type, such as WAVE or AU
properties()	unmodifiable map of properties that specify additional informational meta data (like a author, copyright, or file duration). Properties are optional information, and file reader and file writer implementations are not required to provide or recognize properties
getByteLength()	size in bytes of the entire audio file (not just its audio data)
getFrameLength()	length of the audio data contained in the file, expressed in sample frames
AudioFormat
getChannels()	number of channels
getEncoding()	type of encoding for sounds in this format
getFrameRate()	frame rate in frames per second
getFrameSize()	frame size in bytes
getSampleRate()	sample rate
getSampleSizeInBits()	size of a sample
properties()	an unmodifiable map of properties
NOTE: The duration of an audio file (in seconds) can be computed by multiplying the frame length by the frame rate.

The Java Sound API appears to meet our needs well, however, it only supports AIFF, AU, WAV and some MIDI based formats! To remedy this, the API provides a service provider interface (SPI) to support more file formats. MP3, OGG, APE (Monkey's Audio), FLAC and even Speex SPIs are available. An alternative implementation of the Java Sound API, Tritonus, also provides some SPI plug-ins (the OGG SPI requires Tritonus).

It should also be noted here that an SPI will implement the properties() method of the AudioFileFormat and AudioFormat classes. The properties returned may include non-technical metadata, but this is solely dependent upon the audio file format.

What about other Java media libraries?

According to Sun, the Java Media Framework (JMF) "can capture, playback, stream, and transcode multiple media formats." In comparing the JMF to Java sound, we find that the JMF does have more codecs, however the ability to capture audio specific metadata is limited. The focus of the JMF is not on metadata. At least from what I can tell. If anyone has found the case to be different, please let me know. The same goes with the FMJ and Enterprise Media Beans.

Conclusion

The Java Sound API currently provides the best means of discovering an audio file's technical metadata.

Centiare on the Heels of Wikipedia

"semantic web" technology is installed on Centiare. This means registered users can perform amazing searches on Centiare that just wouldn’t be possible on Google, MySpace, or Wikipedia. Imagine multi-level searches for very specific things, like:

* Are there any white males, over 6 feet tall, with a Masters degree, in Pennsylvania?
* Locate all home heating oil companies, at least 50 years in business, in New Jersey.
* Find all Centiare Directory entities situated between the 39th and 40th parallels.

These searches would be practically impossible in a wiki database without semantic web enabled. It's no wonder that the inventor of the World Wide Web, Tim Berners-Lee, has literally written a roadmap for semantic web -– he knows it's that important. And Internet prognosticators are finally agreeing that in 2007, semantic web will usher in the "Web 3.0" era.

Read more...

File type detection follow up

earl.strain.at ran the test files used in my file type detection article against file, "the open source implementation of the file(1) command that ships with every free operating system."

Read about his results...

Intellisophic Achieves Patent Milestone on Document Indexing System & Methods

Intellisophic Inc., a leading provider of information products to the search and text mining industry, announced that it has been granted allowance by the U.S. Patent and Trademark Office on the systems and methods used for indexing documents based on relevance to reference publications such as a book, web page, newspaper or encyclopedia. The patent also covers core technology relating to generating keyword indices and topic hierarchy from the reference materials for the purposes of scoring or matching other documents based on relevance.

Read the full article at TechWeb...