SearchTools Blog

ESS West. next week in San Jose

Wed, 17 Sep 2008 22:17:31 GMT

Enterprise Search Summit, 22-24 September, 2008

Focused on real-world issues of implementing and enhancing search for intranets, portals and large web sites. It's been a wonderful conference every time, because it is just about search and related issues. It's a great mix of case studies, specialist presentations, and even the vendor talks are good.

Avi Rappoport will be presenting a pre-conference workshop, Enterprise Search 101, and a talk, Inside the Black Box of the Search Index which will start with the basic inverted index, and some of the more interesting aspects, including tokenizing and document caching.

Online registration available. There's also a vendor exhibit hall, shared with the KMworld and Intranets conferences. To just see the exhibits, fill in the registration, and scroll down to the "Exhibits only" button, that's free. But you'll be missing a lot of fascinating talks.

Search Solutions meeting in London, 23 September, 2008

Wed, 17 Sep 2008 22:14:41 GMT

Search Solutions meeting in London, 23 September, 2008

Sponsored by the British Computer Society, Information Retrieval Specialist Group, this is an interactive and collegial meeting, focusing on innovations in information search and retrieval.

Great overview article on Inverted Indexing for Search

Tue, 26 Aug 2008 19:37:13 GMT

I am doing a talk about going inside the black box of the search index for the Enterprise Search Summit in September in San Jose (more on that later).

While I have a lot to say about indexes, I used the opportunity to check around and look for current research on the topic, and pretty much struck gold. Although this paper is from 2006, it is exhaustive and detailed, with both practical and theoretical information, including finding that inverted indexes are both significantly faster to search and easier to maintain than relational database management systems, signature files and suffix arrays. It also has a thorough annotated bibliography. Best of all, Zobel and Moffat agree with me on lowercasing all words in the index and including stopwords, which they say "have an important role in phrase queries".

Inverted files for text search engines
by Justin Zobel and Alistair Moffat
ACM Computing Surveys. 2006;38(2) (56 pages).
Available from: http://doi.acm.org/10.1145/1132956.1132959

Unfortunately, this article is firmly behind the ACM firewall, so if you or your institution don't have a subscription, you have to go through a few hoops to get it. Click the PDF link, you will be denied access and have to go through their free registration form. After that, there's a little form and you can buy the article for $10 by credit card. I think it's worth it.

ETA: improved my title

Sphinx (open source free search engine): New SearchTools Report

Fri, 11 Jul 2008 18:36:25 GMT

Sphinx is an open source search engine, written in C, using both SQL and custom index files to provide a very fast text search. The architecture scales to over a billion records by distributing the index and querying among multiple virtual and real processors.

While it does a full text search, Sphinx is designed to work with structured content (music lyrics, products), and semi-structured content (RSS feeds, blog posts, magazine articles). Sphinx is much faster and more flexible than the internal SQL functions such as where, order by, and group by. This structure allows it to display results in a faceted metadata, for example in the widepress.com results, showing graphical facets including country, source, theme and date.

Sphinx does not have a robot crawler, although it can accept input in XML which can be generated by a crawler. It connects directly to mySQL and PostgreSQL, and has web scripts for external sources. APIs are available in PHP, Python, Java, Perl and Ruby.

Read more and tell me about your Sphinx experience here.

x-robots-tag

Thu, 10 Jul 2008 20:00:51 GMT

In the Robots Exclusion Protocol June 08 Agreement, the leading webwide search engines announced that they would recognize a new element in the HTTP header, the X-Robots-Tag. Google started using it at first, then Yahoo and now Microsoft Live Search is supporting it.

When a browser or robot sends a request to the web server for a URL, part of the response is the invisible HTTP header, including information about the file type, encoding, and date modified. This information is generated by the web server.

The new X-Robots-Tag, within the HTTP response header, can contain same values as the Robots META tags: NOINDEX, NOFOLLOW, NOARCHIVE, NOODP, NOSNIPPET.

There are several cases where the X-Robots-Tag values will be very valuable:

For non-HTML documents, including plain text, XML, PDF, office documents, audio, and video files. While many of these documents are able to carry Properties information or metadata such as XMP, they rarely do, and even then, it's often incorrect or duplicated.

For situations where the web site publisher cannot change the content of the HTML files, but wishes to control some of the site interaction with search indexing robots.

Sites with large amounts of changing content, where updating individual files is too hard or expensive.

This is not something anyone can type in by hand, but it's easily added by programmatically by server-side tools such as Perl, Ruby, or PHP. For simple cases, the Apache .htaccess file is easy enough to configure, as in this example where the crawler is told not to index content in robots.txt:
<FilesMatch "robots\.txt">
	Header set X-Robots-Tag "NOINDEX, FOLLOW"
</FilesMatch>
or to avoid following links in".doc" files
<FilesMatch "\.doc$">
	Header set X-Robots-Tag "NOFOLLOW"
</Files>
I think this is a very clever way to add the known functionality of Robots META tags to non-HTML file formats, collated from an external metadata repository. It's likely to be particularly useful to intranet search engines, and portals which may not have access to the documents themselves.

I have added an X-Robots-Tag test suite to the SearchTools testing section and will report if I find anything interesting.

H/T to: Playing with the X-Robots-Tag; Controlling Your Robots; Handling Google's neat X-Robots-Tag

Webwide search robots now indexing Flash (and filling in forms)

Thu, 03 Jul 2008 00:04:15 GMT

The SWF (Flash) file format has been open for a while, and a lot of search engines have used the format to get at some of the static text in in the Flash files. However, Flash is now an interactive web site application builder, and there is a lot of text that just does not exist until someone comes along and clicks. This has meant that people who wanted their sites properly indexed by webwide search engines could not use Flash, or would have to go to extra lengths to provide static text for search engine robots to find.

What Adobe and Google have just announced is that Adobe is making a special version of the Flash code that can approximate a human interacting with the Flash application in the SWF file, triggering as many application states as it can. As far as I can tell, the Flash client within the indexing robot will be clicking every possible button and entering text in text fields. While indexing the labels on buttons seems odd at first, it makes sense to think of that as as anchor text pointing at other pages (or at least URLs).

This is similar to what the googlebot is doing on some site forms: automatically clicking every combination of buttons, menus, and checkboxes, and submitting words from the site in text boxes. This has ended up creating phantom shopping carts and search queries. They only do this on GET actions, not on POST, and presumably will not do so if the page has meta NOINDEX and NOFOLLOW tags.

The chief concerns I've seen from web site publishers include: the lack of clarity about exactly which JavaScript Flash loading links will be acceptable (especially SWFObject); how external XML files loaded by Flash will be indexed, and how the deep linking into Flash files will work. Adobe has some explanations on their FAQ At the moment, it's SWF only, all versions from the oldest to the current, whether generated by Flash or Flex, which they call "RIAs" (rich Internet applications). However, they are not providing access to FLV files, which are used on YouTube etc. to contain video for playback, and rarely have textual metadata.

Adobe says Yahoo is working on this as well, and Adobe says that they are "exploring ways to make the technology more broadly available" to other search vendors.

No word on whether that includes enterprise and site search developers. There's an excellent writeup from the SEO point of view at Searchengineland, and searchmarketinggurus has a skeptical response.

Search usability research findings

Fri, 27 Jun 2008 17:55:35 GMT

Whitney Quesenbery and her colleagues convey the findings of a long study about how search is used at the UK's Open University, She gave a talk at the Enterprise Search Summit, and presented more formally at the Usability Professionals’ Association conference, in June 2008

The study included search log analysis, heuristic reviews, remote and local usability testing on the search user experience, over the course of several years, and they are linked from Whitney's valuable Search Usability page.

Designing for Search: Making Information Easy(PDF) covers both search and content. It recommends focusing improvements first on the most frequent terms, the short head of search popularity. The results of tests with eyetracking "heat map" visualizations show that both students and those outside the system will scan the whole search results page, and confirm the user tests stressing the strong value of meaningful titles.

Search is now normal behavior. what do we do about that?(PDF) has a different perspective. In addition to the classic long tail frequency of search terms, it showed that the most popular search terms (the short head) remained much the same over the course of three years, though there is also some seasonal variation. Gratifyingly, because I've been saying this for a while, the need for search as a supplement to navigation went down, when the site navigation changed. The study finds that topical metadata and improving titles makes search results significantly more useful. A comparison of four search engines found significant differences in results, in particular, in the variety of top results for common terms. Only one of the four search engines hid duplicate entries, consistently had all links on the first page be appropriate, and displayed links to several different locations, rather than a single subsite or directory.

It's great to see more research done over time and with a large amount of data. I'm keeping a listing of what I've found at CiteULike with the tag search-interface, and planning to update my Search Usability page

The Short Head and Long Tail of Search

Thu, 26 Jun 2008 23:07:58 GMT

I've just posted an article on the Long Tail, Short Head and Search. Every site, intranet and enterprise search log I've analyzed fits the model of the Long Tail, with a very few very popular search terms, then tailing off very quickly to unique queries (the Long Tail), creating a Zipf curve.

The Short Head -- the few most frequently used search terms -- is the best place to start in analyzing search engine usage. My article also gives some suggestions for taking the information and using it to improve a search engine.

HCI/IR workshop

Tue, 24 Jun 2008 00:03:12 GMT

HCIR 2008: Workshop on Human-Computer Interaction and Information Retrieval

Making the connection between interface and search, this workshop is focused this year on complex search tasks. The 2007 Workshop presentations ranged from visual text analysis to online consumer choice. This year's workshop will be 23 October, 2008, in Redmond, Washington, USA.

article on the new Robots Exclusion Protocol

Sun, 15 Jun 2008 22:22:55 GMT

My article is up on InfoToday: New Robots Exclusion Protocol Agreement Among Yahoo!, Google, and Microsoft Live Search. Nothing earthshaking, just a summary from a library point of view, and a quote from Danny Sullivan saying that it's an important first step.

More Information on the new Robots Exclusion Protocol

Fri, 13 Jun 2008 21:18:53 GMT

More Information on the new Robots Exclusion Protocol

Search indexing robot writers and web publishers should definitely look at the new extensions to the REP, as there are useful additions to both robots.txt directives and Robots META tags. Most of these features have been supported by the big three search engines (Google, Yahoo, MSN Live), but it's nice to have that formalized, and other search robots can take advantage of the new functionality.

The new X-Robots-Tag (added to the HTTP header for non-HTML files) is a good way to send the meta information, but requires automated extensions to the servers. For example, if content is available in both HTML and PDF formats, it's easy to send NOINDEX values for all PDF, directing search engines away from the printable format and towards the browser-readable format.

It turns out that NOODP comes in handy when a page is linked from the ODP (Open Directory Project), and the title or text in that entry is not accurate, which happens sometimes. Using the NOODP robots meta tag value tells the search engines not to use the ODP entry, but rather the title and text from the page. NOYDIR does the same for the Yahoo Directory, but is only officially supported by Yahoo and its Slurp robot.

For pages with frequent changes, NOARCHIVE makes some sense: the old content may be in the searchable index, but at least the search engines will not display the old version of the page itself.

However, I have yet to figure out when someone would use NOSNIPPET (which also disables archive display). Limiting a listing in the search results page to the title and URL seems like such a bad idea. Why would anyone do this?

New Robot Exclusion Protocol!

Wed, 04 Jun 2008 00:47:29 GMT

Supported by webwide search engines Yahoo, Google and Microsoft, this adds directives to robots.txt:

"Allow" directives
wildcards in URLs
Sitemap Location

There are also HTML meta tags and document properties directives for

NOSNIPPET
NOARCHIVE
NOODP (don't use ODP information for this page).

Yahoo has a nice long blog entry on this, as does Google and MS Live Search. Great news for web developers, who've been waiting for this for a very long time.

But there's nothing from the robots mailing list or the RobotsTxt.org which is a shame.

This is also a test for all site and intranet search crawlers -- any abandoned software will not recognize these new directives.

I'll dig further into this in the next week and provide more analysis and details.

Comments?

Yahoo vs. Google interactive geosearch

Tue, 03 Jun 2008 22:00:51 GMT

I wanted to find the trendy-but-good shop I stopped by yesterday: it's not really a cafe or a coffehouse or a restaurant. They sell sandwiches and savory chicken pie and strawberry shortcake, with cartons of strawberries stacked high in the front. I knew where it was, but not the name, so I compared Yahoo Maps and Google Maps, to learn a little about geosearch.

Yahoo maps knew where I live, so it started there, and then just scrolled and zoomed until I found the corner of 51st and Telegraph. (It would have been easier if I'd used the hybrid map & satellite photo, much better for me than plain maps). Then I used the "Find a Business" and typed "pie". Instantly, Bakesale Betty's came up, with the word "pie" harvested from a Yahoo user review (5 stars!) in which a customer praised their "pot-pies". That was about 5 scrolls and 4 clicks and one short search, and success

Google doesn't know where I live, so typed in "51st and telegraph, oakland, ca" -- and there I was. A couple of clicks to get to the closest street level. Then I clicked on the "Find businesses" link and typed in "pie" and got a display of... 20 results out of 5,645 hits and a much higher view, so they could show me all the other pie matches. Oops. I tried "pies" and that didn't work, and I couldn't think of anything else to search on as there area a million sandwich and cookie places. 2 searches (one long), 1 scroll and 4 clicks, but that got me stuck.

Then I typed "betty" and of course it was right there, with links to reviews.

Just to check, I clicked the Explore this area link, no luck, all about garbage pickup. I tried using the "Street View" and looking at that corner, but in one view the storefront is too dark to see, and in the other there's a big truck in front of the shop.

Why did Google not find me the store when I searched for pie? You may ask, and I have an answer. Because their index and/or their query parsing did not match "pie" with "pot-pies" in one of the customer reviews that Google automatically connected to the business. If they'd used even lightweight pluralization, it would have worked. And zipping me up to the whole Bay Area view was really annoying.

Yahoo did a stemmed match and showed it right there, without changing my map level. It wins this comparison.

off to ESS

Thu, 15 May 2008 22:16:36 GMT

I will be leaving for the Enterprise Search Summit tomorrow (taking my family to New York for a little adventure). I'll be teaching a workshop, Enterprise Search 101, on Monday the 19th, and starting the Search Analytics track on Wednesday the 21st. If you see me, please introduce yourself, I'd love to meet people who read this blog.

A First Taxonomy for "Search Log Junk"

Tue, 06 May 2008 17:57:06 GMT

Search logs contain a lot of weird things, and some of them can have a significant effect on search log analysis. Having looked at tens of thousand lines of search log entries, I offer this first attempt at defining some of the weirdest and least useful kinds of log entry, which I call "Search Log Junk". Here are the types of junk that I've seen most frequently:

Empty Queries: Queries without any query text or usable parameters. These can appear when people think the "Search" button is important in and of itself. Or perhaps search is in the first page form, and the cursor gets into that field and users press Return. These are often sent from the home page, according to the referer fields I've seen.

The first thing is to make sure that the search engine is doing something reasonable in this case. This could be just bringing up a helpful search page, adding a script to bring up an error dialog, or a script to ignore the empty query. I'm leaning towards the last option.

I've found only a couple of ways to use this information. They are still useful for traffic and response time metrics, and I think it's useful to check the top referring pages occasionally. A lot of empty queries for a page deep within a site may indicate some navigation problems.
Repeat Queries: Multiple identical queries to the search engine from the same IP or user ID. My best guess is that the client is calling for a refresh automatically -- my favorite was thousands of queries over months for two dots: "..".

Again, this is useful for traffic metrics and possibly for identifying really weird incoming links. For most situations, it won't affect the statistics in any important way. But if there are hundreds of repeat queries by the same client, removing them from the database allows you to concentrate on the real data. You may also want to ban that IP address.
Robot crawlers: Having search and intelligent agents crawl search results may be a good thing. Incoming links are always good and it may be that the search results on your site for emerald green widgets is number one in webwide search results and drives good traffic. However, there may be other robots wasting your search engine cycles: for those, a combination of robots.txt and banning their IP address will help.
Server hacks: Search engines are attacked by the standard web server hacking parameters, such as "phpmyadmin". They may also be subject to buffer overflow and other attacks, so should be included in standard website security audits and checklists.
Guestbook spam: There are automated advertising services that insert fake comments with URLs into form fields, guestbooks, blogs and wikis (and there's a wikipedia page about them). Many of them do the same with search fields, which explains why logs contain bizarre queries with spaces, HTML formatting and URLs in them.; For sites with light search traffic, these meaningless entries can cause problems with both traffic metrics and top query listings. Even for sites with thousands of queries per day, they can distort statistics about the average length of query, so removing them from your analysis database is a good idea.

It's fairly easy to identify these queries with simple regular expressions looking for href, http and .com. I haven't heard of any search engines which filter this, though some may be doing it without bothering their customers about it.

partly offline due to injury

Tue, 04 Mar 2008 23:22:11 GMT

I slipped on a stepladder and broke my left leg (tibial plateau fracture) and then chipped my right heel while on crutches. My office is not really wheelchair accessible, nor can I go down my house's steps without great effort, so I'm working remotely, part-time.

I am trying to read email every day and respond in a timely way, so if you've left a voice message or sent email that I have not answered, please try again (by email if possible). Apologies for your inconvenience.

Avi

where has Entopia gone?

Wed, 23 Jan 2008 21:55:04 GMT

One of my clients is interested in Entopia, so I was taking a look.

I tried to go to the web site and it was replaced by one of those placeholder spam sites which pops up several spammy windows. It seems like the kind of thing that might have viruses, worms or trojans, so I'd suggest against opening the site in IE, or really, at all on a Windows machine.

No one answered at one phone number, the other two I found were disconnected.

Casualty of the recession? Acquired by someone? It's a mystery, and I'm curious.

ETA: The Wayback Machine (archive.org) has an actual home page as of June 13, 2006 and an empty page as of July 1 of that year. I always thought they were promising more than they could deliver, so this is perhaps confirmation.

Small updates to Search Tools reports

Wed, 19 Dec 2007 01:28:15 GMT

We've updated the following reports on search engines large and small in the last few weeks:

i411 has changed its name to Intelligenx and added autocatagorization and multiple language support.
Engenium now has OEM library and automatic clustering module.
FreeFind now has wildcards for excluding URL paths from indexing, indexes common office document file formats, relevance weight adjustments for URL paths (with wildcards), and some really nice indexing reports -- URLs extracted, server response, status, and which URLs are actually in the searchable index.
HomePageSearchEngine now indexes more file types.
Doclinx now has a web monitoring agent, with support for speech recognition, for research and competitive intelligence, and a language analyzer.
Boolean Search now runs natively on both PPC and Intel Mac OS X systems, includes web-based admin, spellchecking and match term highlighting in search results, template and AppleScript integration for search results formatting, standalone search server, and regular expressions in queries.
Crawl-it remote service is still being supported.
Datagold is no longer a separate search, it's part of an online archiving suite.
Educasoft has no indication of continuing development

Search Conferences Listing updated

Wed, 19 Sep 2007 23:58:28 GMT

This list covers all the search and related related conferences I know about.

At the Enterprise Search Summit West I will be doing a pre-conference workshop on Critical Success Factors (how search engines work and how to make them better), a presentation on Tuning Search using Analytics and a moderating a panel on Good Practices for Search User Interfaces. At the Web Builder 2.0 conference, I'll be presenting on Web Site Search and the User Experience. If you are a reader of this web site, please come and say hi, and if you'd like an online presentation to your organization or company, I do those as well.

To suggest a conference or the listing, please leave a comment and I'll add it.

Critique of the Google Custom Search Traffic Report

Wed, 29 Aug 2007 22:35:21 GMT

Edward Tufte would be disappointed in Google. The traffic reports in the Google Custom Search Business Edition are not only insufficient, but somewhat misleading.

Below is a picture from a CSBE search for a B2B site that I helped install in August 2007. The fact that it's a line chart, with no data points given, filled underneath,makes it look active. It seems as though something's happening, the traffic is making progress, or worse, losing ground. The deep dips look scary, as though the site has done something wrong.

The problems come about because it's the wrong graph format for the content. This is very simple data: one point per day. Look at it as a simple bar graph and it suddenly seems more reasonable. The traffic resolves itself into a rhythm: the dips are on weekends -- all the customers are home. I don't know why they got it so wrong, but it's worth getting right.

Edward Tufte wrote some enlightening books on these topics, including The Visual Display of Quantitative Information, which taught those of us paying attention that how data is presented deeply affects how it is received. I highly recommend getting some of Tufte's books, from Amazon, from Powell's or from your library (using WorldCat).

Please comment whether you agree or disagree. I'm haven't seen quite this problem in other search engine traffic reports, but I'm wondering what other interfaces might look like, and what you think is best. Tell me your opinions, please!

Google Search Appliance and Mini - SearchTools Report Updated

Mon, 20 Aug 2007 20:55:48 GMT

I have updated my report on the GSA and Mini search appliances, with detail based in part on my recent experiences customizing a Google Mini. The report includes information on the pricing as far as I could find it, the terms of licensing, new features, links to informative documents, and features that are not included with the Mini appliance.

Once I update my full product review, I will have a chance to pay attention to other search engines, and that will be lovely.

Google CSE - different results when searching more than three sites

Thu, 16 Aug 2007 22:03:07 GMT

A support document for the Google CSE (Custom Search Engine)and CSBE (Custom Search Business Edition) notes that some results may be different than those found in the same search on Google.com. It attributes this to including more than three sites in the CSE, and says that the CSE is using a subset of the Google.com index.

They recommend limiting the CSE to three sites, changing the behavior to 'Search the entire web but emphasize included sites', or adding refinements that have the same effect.

As of August 16, 2007, the support note says "We're working to bring more complete results to all Custom Search Engines.".

Google Launches Site Search Service for Business

Fri, 03 Aug 2007 17:43:11 GMT

Google's Custom Search Business Edition uses the Google web search index limited by site or sites. It provides most of the Google web search features and is very cheap, only $100 per year for up to 50,000 pages, $500 for up to 500,000 pages. More here at my InfoToday article / more at the SearchTools Google Service report page.

What do you think of it?

New Google hosted search with no advertising

Thu, 19 Jul 2007 23:48:35 GMT

Called the Google Custom Search Business Edition, this is a hosted site search, designed for small businesses with web site content, who don't want the advertising displayed on the older Custom Search Engine.

This version uses Google's existing index of the Internet, searching all the pages they know about it on the specified sites including non-HTML file types, using their query language, retrieval and relevance algorithsm, and searching in multiple languages and character sets. Like the web search engine, there is no way to index pages protected by access control such as passwords or ACLs.

The default interface customization is limited to a logo and colors of the results page border, title, background, text and links, but the XML results format is fairly configurable using the Google AJAX Search API. While there is no structure in place to display site advertising on search results, presumably one could do that very easily with XML results. Reports are limited to top queries and queries per day/week/month/all, but can be connected to the Google Activity Monitor site traffic analysis tool.

Note that Google will not guarantee that they'll crawl all of the pages of a particular site, update on-demand, or even update frequently. Using this service will not improve a site's position in the Google.com search results.

Pricing is $100 per year for up to 5,000 pages; $500 per year for up to 50,000 pages (both payable by credit card via Google Checkout). According to ecommerce-guide.com, it seems to go to a $15,000 per year fee for up to 1 million pages, but potential customers should contact the company. (Non-profits, university and government agencies can use the standard Custom Search and opt-out of advertising).

Swish-e - SearchTools Report Updated

Fri, 04 May 2007 00:00:24 GMT

Swish-e, a free open-source Unix search engine, Swish-e is fast at indexing and searching, and quite flexible. It can handle simple authentication, indexes HTML, text, XML, and (via converters), PDF, MS Word, Excel and MP3 ID3 tags, with an emphasis on storing feilds/tags for specifying during search. Resuts can be sorted by relevance, date, size, and other fields. It runs as a CGI to a web server (Apache recommended), and has a fairly active user and developer base. New features include adjustments to the relevance algorithm, "near" operator and "?" single character wildcard operator (in addition to "*").