Posts Tagged ‘autonomy’
I spent yesterday morning at Ovum’s briefing on Enterprise Search, and they kindly invited me to sit on a discussion panel. One of the more controversial topics raised by analyst Mike Davis was ‘Is Enterprise Search dead?’ which provoked some lively discussion. We also heard from Tyler Tate of Twigkit on Search UX, Exalead on Search Based Applications and Search Technologies on data conditioning and why metadata is so important.
One can’t deny that the search market is going through some huge changes at the moment. Larger vendors are being acquired which can lead to some major (and not always welcome) changes in the product, pricing and service. Smaller vendors are finding it increasingly hard to compete with the plethora of powerful open source solutions (we’ve heard rumours of prices of closed source solutions being dropped radically to attempt to secure new business). There are also some interesting moves towards more comprehensive Business Intelligence and Unified Access solutions, such as Attivio.
I don’t think enterprise search is dying as a market or an offering, simply changing – and hopefully for the better, into an era of more realistic pricing, solutions that actually work (rather than promising ‘magic’) and more openness in terms of the technology and capability.
The blogotweetosphere has been positively buzzing since last night’s announcement that Hewlett Packard will be buying Autonomy for £7.1bn, while divesting itself of its PC business. Many commentators have put a positive spin on this, pointing to Autonomy’s meteoric rise from a small office in Cambridge to the behemoth it is today. It’s undoubtedly good news for Autonomy’s shareholders. Dave Kellogg correctly identifies Autonomy as a “finance company dressed in (meaning-based) technology company clothing” with a “happy ending”.
However the reaction isn’t all positive – the FT implies this deal is at the “lunatic end of the valuation spectrum”. Law Technology News says “Autonomy’s e-discovery revenue stream is high-end but unsustainable” and quotes users of the system with problems: “We had a lot of issues with the applications crashing, the documents tending not to get checked in”….”"[Autonomy sales staff] were pricey, arrogant, and they couldn’t care less about us. … It cannot get any worse.”.
HP will have to work hard to integrate Autonomy into both its corporate culture and software frameworks – a problem currently faced by Microsoft since its acquisition of FAST a short while ago. Stephen Arnold thinks this process will be “risky”. What it means for the rest of the search sector is harder to guess, although Martin White of Intranet Focus says this deal indicates HP can see a “future in search applications” and, interestingly, “A number of privately-held search vendors are probably working out what their valuation would be”.
My view is that this is just the latest of huge shifts in the enterprise search market, partly spurred on by the rise of open source options and the gradual realisation that the huge license fees charged by some vendors may be unsustainable. I don’t think Autonomy will be the last company looking for a safe haven in the years to come.
This week I was passed a link to a European Commission report on the Enterprise Search market, which I’ve just finished ploughing through (it’s 123 pages and not exactly light reading). It provides an overview of the history of the market and some current trends, but sadly misses out almost completely the rapidly growing open source sector. The authors say “…open source solutions have been disregarded because they do not seem yet to be a real alternative for company use…” – a point of view both I and our satisfied clients would disagree with. The report does at least acknowledge that “open source components are frequently used and integrated in some commercial solutions”.
However there are some very interesting numbers in the latter part of the report. For example, we hear that an Exalead customer, the automotive logistics specialist Gefco, paid 700,000 Euros for the solution built for them to track around 100,000 events a day regarding 1 million vehicles. Appendix 2 has a list of various search vendors and associated costs: for example “The average selling price for the [Autonomy] IDOL tool is $375,000″ and “The price for the Oracle Secure Enterprise Search is $34,500 per processor and $70 per referenced user (with a minimum of 100 users).”
I would question whether these prices are sustainable given that alternative solutions based on proven, scalable open source software are now available at a fraction of the cost. Perhaps the authors of the report should have considered more deeply how this might impact the enterprise search market.
Cambridge, U.K. has a long history of hosting search experts and businesses. Back in the 1980s two firms arose – Cambridge CD Publishing, founded by Martin Porter and John Snyder grew into Muscat, and Cambridge Neurodynamics became Autonomy. We believe Smartlogic still have a small office here. Stephen Robertson, co-author of the probabilistic theory of information retrieval (which Xapian uses for ranking) is based here at Microsoft Research.
Today, the city is still home to innovative search companies, including True Knowledge, Grapeshot and of course ourselves. We know of many more ‘under the radar’ developing search technologies both to complement existing systems and as completely new approaches to information retrieval, including visual search.
To encourage networking and to help keep the city at the forefront of search developments, we’ve created the Enterprise Search Cambridge Meetup group and our first meeting is on February 16th – all are welcome, whether currently working with search and related technologies or simply interested in the possibilities. Hope to meet you there!
It’s been an interesting and busy twelve months here at Flax – we’ve worked on some fantastic customer projects, spoken at conferences at home and abroad and made some great alliances and partnerships. We are talking to more people than ever before about the advantages of open source search and we’ve even started a local Meetup group.
This has been the year when open source search moved out of the shadows and became a force to reckon with – whether handling billions of queries or millions of customers, powering innovative new APIs for open content from forward-looking media companies or simply making it easier for search applications to be developed. Commercial support is now available to rival anything offered by the closed source world and there are now fully packaged solutions built on open source. In some sectors open source may even become the default choice (see what IDC said about the embedded/OEM market).
There’s still significant change to come in the search sector – I expect a few vendors will be in trouble by this time next year as they realise their business models (often built on per-document charges) are out-of-date, and we might also see further acquisitions by the usual behemoths. All this leads to reduced choice and increased costs for customers, and this is where open source can help – you can build your search solution in-house, or engage companies like ours to help, but you’re no longer locked in to a vendor’s roadmap and shackled to their business plan (or the consequences of its failure!).
I’ll leave the final word to Matt Asay of Canonical, who says: “Open source is how we do business 10 years into this new millennium.”
We’ve been aware that some FAST customers will be considering migration for a while now – but Autonomy have finally caught up.
However, if you migrate from one closed source solution to another, how can you guarantee that the same sort of events that have led to the current situation won’t happen again? With open source, there’s no vendor lock-in, a wide choice of companies to assist you with development an integration, a wealth of different support options and of course no license fees to pay. Migrating from FAST is a common topic at conferences at the moment – read Jan Høydahl’s presentation, or see Michael McIntosh’s video. There are even open source document processing pipeline frameworks to replace the popular FAST one, and we’ve been evaluating some alternative language processing frameworks. Scaling isn’t an issue and some cases you could significantly reduce your hardware budget.
We dropped in to the Online 2010 event at Olympia this week, and were immediately struck by how quiet the event was: yes, there’s been some terrible weather recently in the UK but there were fewer stalls than last year, a smaller exhibition space and very few exhibitors in the enterprise search space – no Autonomy, Google, Vivisimo or Endeca for example. Unlike previous years there was no dedicated ’search’ area on the exhibition floor, and we did see a few unmanned stands from mid afternoon. Is this is a sign of difficult times or of an event that needs a rethink about its focus?
We didn’t attend the conference that runs next to the exhibition hall this year. This report on the closing panel shows that one question to the panel was about the rise of open source search – not surprisingly, the panel members (all being from closed source companies) weren’t very enthusiastic about this. According to Autonomy open source is only for the commodity end of the market, which is the smallest part. I’m not sure Twitter (1 billion queries a day), LinkedIn (30 million users), The Guardian (innovative open platform) or the Financial Times would agree…
A new year, and a chance to think about what might happen in the world of enterprise search over the next twelve months. I’ll make a stab at some predictions:
- Price cuts – possibly driven by even harsher competition between Google and Microsoft FAST, I can see prices coming down for packaged enterprise search. Autonomy will probably raise theirs
- Real time search matures – not just Twitter or Facebook, but real time data from many sources being part of enterprise search results
- More geolocation-aware search – in the U.K. at least, we’re seeing signs that the source data is finally being freed up, which should make it a lot simpler and cheaper to build location-aware solutions
- A few less second-tier players in the market – it’s still difficult out there, I’m afraid not every company will survive the next year.
You’re welcome to take any of these with a generous pinch of salt!
Stephen Arnold recently posted some rather impressive performance figures for Autonomy’s IDOL search engine. This kind of data is all very well, but without independent testing and more detail it’s hard to know how these figures apply to the real world.
So here’s an idea. Why not create an openly available collection of test data, a set of searches and a set of conditions, then compare the performance of the various available engines for indexing and searching? Recording the software and hardware used as well, of course. Making the data and conditions public would allow for independent verification.
I’m not sure commercial search vendors would ever agree to this, but it’s a nice idea.
