If your business is global, then you likely have many documents in foreign languages. In order to help employees find documents in languages other than their first, Google has enabled cross-language searching for enterprise customers using the Google Search Appliance.

Queries will be passed through a translation engine which will enable search in several languages. Users can choose the document to be returned to them in any of the available languages.

Google Mini and Google Search Appliance owners can head to this link at Enterprise Labs to check out the new feature.

It’s easy to pick on Microsoft. It’s practically a national pastime at this point. So when they announced their Cashback program to help grow Live Search, the snears came fast and furious.

“You have to pay people to search?” they said.

Turn your laughter into claps, people. It turns out that Cashback is working. Microsoft is seeing positive results on three goals it said it would report. They are:

• Number of offers in the system
• Advertiser return on investment (ROI)
• Percentage of commercial search queries as measured by comScore

There’s been a 30% increase in the number of products offered via Cashback. 4.5 million unique users per month are generating 68 million commercial queries. eBay has seen an increase of 50% on their ROI.

“We believe this early traction speaks to the differentiated and unique value proposition of Microsoft Live Search cashback for both consumers and advertisers, especially in these tough economic times,” said Brad Goldberg, general manager of Microsoft Live Search.

Is it really any surprise that incentives work? No. Have you ever been listening to the radio and they’re having a $1,000 giveaway? You have to be listening at the right times to call in.

Incentives are nothing new. Microsoft was smart to implement them into Live Search. People don’t necessarily use Google because it’s any better but because it’s familiar. Live Search needs a way to get people searching, and Cashback is working to help accomplish those goals.

That may be why Microsoft is expanding Cashback by partnering with shopping cart providers Miva Merchant, Early Impact Inc. (ProductCart) and 3DCart. Through the agreement, merchants who use the shopping carts are eligible for Cashback.

Google had been doing a series of posts about search quality. Today, the latest post in the series discusses how evaluation enters into the the process.

Scott Huffman, Engineering Director, gave four insights into the nuances of difficulty experienced in search evaluation:

• First, understanding what a user really wants when they type a query — the query’s “intent” — can be very difficult. For highly navigational queries like [ebay] or [orbitz], we can guess that most users want to navigate to the respective sites. But how about [olympics]? Does the user want news, medal counts from the recent Beijing games, the IOC’s homepage, historical information about the games, … ? This same exact question, of course, is faced by our ranking and search UI teams. Evaluation is the other side of that coin.
• Second, comparing the quality of search engines (whether Google versus our competitors, Google versus Google a month ago) is never black and white. It’s essentially impossible to make a change that is 100% positive in all situations; with any algorithmic change you make to search, many searches will get better and some will get worse.
• Third, there are several dimensions to “good” results. Traditional search evaluation has focused on the relevance of the results, and of course that is our highest priority as well. But today’s search-engine users expect more than just relevance. Are the results fresh and timely? Are they from authoritative sources? Are they comprehensive? Are they free of spam? Are their titles and snippets descriptive enough? Do they include additional UI elements a user might find helpful for the query (maps, images, query suggestions, etc.)? Our evaluations attempt to cover each of these dimensions where appropriate.
• Fourth, evaluating Google search quality requires covering an enormous breadth. We cover over a hundred locales (country/language pairs) with in-depth evaluation. Beyond locales, we support search quality teams working on many different kinds of queries and features. For example, we explicitly measure the quality of Google’s spelling suggestions, universal search results, image and video searches, related query suggestions, stock oneboxes, and many, many more.

Not sure if I’m buying that Olympics example. Google didn’t do a great job with the Beijing Olympics, and surely their algorithm could handle serving up more relevant search results during the time surrounding the event.

I’m not saying that search query intent evaluation is easy, just that the Olympics query is not quite as problematic as Google is making it out to be.

The rest of the points are things we’ve been hearing from Google for a long time. We know they’re progressing on universal and personalization search efforts, all in their famous intent to create the best user experience.

So, what methods does Google employ to address these evaluations? Huffman offered up the following:

• Human evaluators. Google makes use of evaluators in many countries and languages. These evaluators are carefully trained and are asked to evaluate the quality of search results in several different ways. We sometimes show evaluators whole result sets by themselves or “side by side” with alternatives; in other cases, we show evaluators a single result at a time for a query and ask them to rate its quality along various dimensions.
• Live traffic experiments. We also make use of experiments, in which small fractions of queries are shown results from alternative search approaches. Ben Gomes talked about how we make use of these experiments for testing search UI elements in his previous post. With these experiments, we are able to see real users’ reactions (clicks, etc.) to alternative results.

What do you think of Google’s search evaluation? What evaluations would you like to see them conduct? Discuss in the comments.

Video search has been a key topic at the TechCrunch50 show in San Francisco. Lots of newly launched companies presented (kind of like DEMO minus the payola). In a “rich media” grouping of companies, the proverbial challenge of making images and video searchable kept coming up.

Presenting companies were
• VideoSurf: video search
• GazoPa: image search
• Fotonauts: image search
• Bojam: online music collaboration (like garage band in the cloud)

On the panel of judges was Bradley Horowitz who was a key part of Yahoo’s acquisition of Flickr back in 2005. Now he in charge of Google Gadgets and recently oversaw the integration of facial recognition software in Picassa.

“For a number of years I worked at MIT on image recognition technology and the first time I saw Flickr, I wanted to rip up my diploma,” he said from the stage. “It turns out humans are very good at this. You don’t need a hyper-technical solution. It can sometimes be the activity around the video, not the content in the video that defines relevance.

This point underscores many of the companies walking different lines between algorithmic solutions and user interaction/tagging in order to solve this longstanding indexing issue.

For the sake brevity, I’ll just talk about the best one I saw — San Mateo, Calif based video search engine VideoSurf. CEO Lior Delgo agrees that users should be brought into the mix but asserts that relying on this alone is flawed.

The company still got Horowitz’ vote for its underlying technology, which includes a great deal of intellectual property around visual recognition. This is paired with some social features that together make it a viable choice for video search over and above existing engines like Truveo and Blinkx.

The technology essentially goes beyond the voice recognition, meta data, and surrounding content that many existing sites rely on, and instead tags each frame with more granular information. This includes character names in popular shows and other things that are more contextually relevant and likely as search terms. Relevancy is weighed on frequency of these keywords, click throughs and a few other factors.

This could also be the right time for this technology, given the expanding corpus of video being brought online. This isn’t just the long tail YouTube ranks but head content that will get more search queries such as network shows on sites like Hulu. Lots more to the nuts and bolts, and it’s clear video search will continue to be a tough nut to crack. But this will be one to watch.

I hope to dive deeper in a column later this month on this and the many other search related companies here — including a social network for bird watchers (seriously).

The art and science of understanding how search engines identify pages that are relevant to a query made by a visitor and designing marketing strategies based on this is called search engine optimization. Search engines offer the most cost effective mechanism to acquire “real” and “live” business leads. It is found that in most cases, search engine optimization delivers a better ROI than other forms such as online advertisements, e-mail marketing and newsletters, affiliate and pay per click advertising, and digital campaigns and promotions.

Search Quality is the name of the team responsible for the ranking of Google search results. Our job is clear: A few hundreds of millions of times a day people will ask Google questions, and within a fraction of a second Google needs to decide which among the billions of pages on the web to show them — and in what order. Lately, we have been doing other things as well. But more on that later.

For something that is used so often by so many people, surprisingly little is known about ranking at Google. This is entirely our fault, and it is by design. We are, to be honest, quite secretive about what we do. There are two reasons for it: competition and abuse. Competition is pretty straightforward. No company wants to share its secret recipes with its competitors. As for abuse, if we make our ranking formulas too accessible, we make it easier for people to game the system. Security by obscurity is never the strongest measure, and we do not rely on it exclusively, but it does prevent a lot of abuse.

The details of the ranking algorithms are in many ways Google’s crown jewels. We are very proud of them and very protective of them. By some estimate, more than one thousand programmer/scientist years have gone directly into their development, and the rate of innovation has not slowed down.

But being completely secretive isn’t ideal, and this blog post is part of a renewed effort to open up a bit more than we have in the past. We will try to periodically tell you about new things, explain old things, give advice, spread news, and engage in conversations. Let me start with some general pieces of information about our group. More blog posts will follow.

I should take a moment to introduce myself. My name is Udi Manber, and I am a VP of engineering at Google in charge of Search Quality. I have been at Google for over two years, and I have been working on search technologies for almost 20 years.

The heart of the group is the team that works on core ranking. Ranking is hard, much harder than most people realize. One reason for this is that languages are inherently ambiguous, and documents do not follow any set of rules. There are really no standards for how to convey information, so we need to be able to understand all web pages, written by anyone, for any reason. And that’s just half of the problem. We also need to understand the queries people pose, which are on average fewer than three words, and map them to our understanding of all documents. Not to mention that different people have different needs. And we have to do all of that in a few milliseconds.

The most famous part of our ranking algorithm is PageRank, an algorithm developed by Larry Page and Sergey Brin, who founded Google. PageRank is still in use today, but it is now a part of a much larger system. Other parts include language models (the ability to handle phrases, synonyms, diacritics, spelling mistakes, and so on), query models (it’s not just the language, it’s how people use it today), time models (some queries are best answered with a 30-minutes old page, and some are better answered with a page that stood the test of time), and personalized models (not all people want the same thing).

Another team in our group is responsible for evaluating how well we’re doing. This is done in many different ways, but the goal is always the same: improve the user experience. This is not the main goal, it is the only goal. There are automated evaluations every minute (to make sure nothing goes wrong), periodic evaluations of our overall quality, and, most importantly, evaluations of specific algorithmic improvements. When an engineer gets a new idea and develops a new algorithm, we test their ideas thoroughly. We have a team of statisticians who look at all the data and determine the value of the new idea. We meet weekly (sometimes twice a week) to go over those new ideas and approve new launches. In 2007, we launched more than 450 new improvements, about 9 per week on the average. Some of these improvements are simple and obvious — for example, we fixed the way Hebrew acronym queries are handled (in Hebrew an acronym is denoted by a (”) next to the last character, so IBM will be IB”M), and some are very complicated — for example, we made significant changes to the PageRank algorithm in January. Most of the time we look for improvements in relevancy, but we also work on projects where the sole purpose is to simplify the algorithms. Simple is good.

International search has been one of our key focus areas in the past two years. This means all spoken languages, not just the major ones. Last year, for example, we made major improvements in Azerbaijani, a language spoken by about 8 million people. In the past few months, we launched spell checking in Estonian, Catalan, Serbian, Serbo-Croatian, Ukranian, Bosnian, Latvian, Filipino Tagalog, Slovenian and Farsi. We organized a network of people all over the world who provide us with feedback, and we have a large set of volunteers from all parts of Google who speak different languages and help us improve search.

Another team is dedicated to new features and new user interfaces. Having a great engine is necessary for a great car, but it is not sufficient. The car has to be comfortable and easy to drive. The Google search user interface is quite simple. Very few of our users ever read our help pages, and they can do very well without them (but they’re good reading nevertheless, and we’re working to improve them). When we add new features we try to ensure that they will be intuitive and easy to use for everyone. One of the most visible changes we made in the past year was Universal Search. Others include the Google Notebook, Custom Search Engines, and of course, many improvements to iGoogle. The UI team is helped by a team of usability experts who conduct user studies and evaluate new features. They travel all over the world, and they even go to people’s homes to see users in their natural habitat. (Don’t worry, they do not come unannounced or uninvited!)

There is a whole team that concentrates on fighting webspam and other types of abuse. That team works on variety of issues from hidden text to off-topic pages stuffed with gibberish keywords, plus many other schemes that people use in an attempt to rank higher in our search results. The team spots new spam trends and works to counter those trends in scalable ways; like all other teams, they do it internationally. The webspam group works closely with the Google Webmaster Central team, so they can share insights with everyone and also listen to site owners.

There are other teams devoted to particular projects. In general, our organizational structure is quite informal. People move around, and new projects start all the time.

One of the key things about search is that users’ expectations grow rapidly. Tomorrow’s queries will be much harder than today’s queries. Just as Moore’s law governs the doubling of computing speed every 18 months, there is a hidden unwritten law that doubles the complexity of our most difficult queries in a short time. This is impossible to measure precisely, but we all feel it. We know we cannot rest on our laurels, we have to work hard to meet the challenge. As I mentioned earlier, we will continue providing you with updates on search quality in the coming months, so stay tuned.

Link to Original Story

ü They search the Internet - or select pieces of the Internet - based on important words,

ü They keep an index of the words they find, and where they find them, and

ü They allow users to look for words or combinations of words found in that index.

Early Search Engines held an index of a few hundred thousand pages and documents, and received maybe one or two thousand inquiries each day. Today, a top Search Engine will index hundreds of millions of pages, and respond to tens of millions of queries per day.

Before a Search Engine can tell you where a file or document is, it must be found. To find information on the hundreds of millions of Web pages that exist, a Search Engine employs special software robots, called spiders, to build lists of the words found on Web sites.

When a spider is building its lists, the process is called web crawling.

In order to build and maintain a useful list of words, a Search Engine’s spiders have to look at a lot of pages. How does any spider start its travels over the Web? The usual starting points are lists of heavily used servers and very popular pages. The spider will begin with a popular site, indexing the words on its pages and following every link found within the site. In this way, the spidering system quickly begins to travel, spreading out across the most widely used portions of the Web.

Once the spiders have completed the task of finding information on Web pages, the Search Engine must store the information in a way that makes it useful. There are two key components involved in making the gathered data accessible to users:

ü The information stored with the data, and

ü The method by which the information is indexed.

In the simplest case, a Search Engine could just store the word and the URL where it was found. In reality, this would make for an engine of limited use, since there would be no way of telling whether the word was used in an important or a trivial way on the page, whether the word was used once or many times or whether the page contained links to other pages containing the word. In other words, there would be no way of building the ranking list that tries to present the most useful pages at the top of the list of search results.

To make for more useful results, most Search Engines store more than just the word and URL. A Search Engine might store the number of times that the word appears on a page. The engine might assign a weight to each entry, with increasing values assigned to words as they appear near the top of the document, in sub-headings, in links, in the META tags or in the title of the page. Each commercial Search Engine has a different formula for assigning weight to the words in its index. This is one of the reasons that a search for the same word on different Search Engines will produce different lists, with the pages presented in different orders.

An index has a single purpose: it allows information to be found as quickly as possible. There are quite a few ways for an index to be built, but one of the most effective ways is to build a hash table. In hashing, a formula is applied to attach a numerical value to each word.

The formula is designed to evenly distribute the entries across a predetermined number of divisions. This numerical distribution is different from the distribution of words across the alphabet, and that is the key to a hash table’s effectiveness.

When a person requests a search on a keyword or phrase, the Search Engine software searches the index for relevant information. The software then provides a report back to the searcher with the most relevant web pages listed first.