Where the FAST Enterprise Search Platform (ESP) is going now…

I have spent the last week in Las Vegas attending the FAST Forward 09 conference. About a year ago the Norvegian company FAST Search & Transfer was acquired by Microsoft and like me customers all over the world wonder what would happen. Some thought it was great to have a huge company with its R&D resources to take the platform forward while others like me feared a technology transition which would include cancelling support for other operating systems and integration with nothing but Microsoft technology.

It was very clear that the Microsoft Marketing department had a lot to say about the conference and what messages that were to be conveyed. Somewhere behind all that you could still see some of the old FAST mentality but it was really toned down. To me the conference was about convincing existing customers that MS is committed to Enterprise Search and to give Sharepoint customers some idea of what Enterprise Search is all about.

It is clear that the product line is diversifying in a common Microsoft strategy:

Solutions for Internet Business

  • FAST Search for Internet Business
  • FAST Search for Sharepoint Internet sites
  • FAST AdMomentum
  • Solutions for Business Productivity

  • FAST Search for Sharepoint
  • FAST Search for Internal Application
  • FAST Search for Sharepoint won’t be available until Office Wave 14 (incl Sharepoint) will be released so in the meantime there will be a product called FAST ESP for Sharepoint that can be used today and will have a license migration path towards FAST Search for Sharepoint. That product will have product license of aroudn 25 000 USD and then additional Client Access License (CAL) will follow in a standrad MS manner.

    So what does all of this means for us who like to see FAST ESP continue as an enterprise component in a heterogenous environment? Well, MS has commited to 10 years of support for current customers, I guess in a gesture towards those who are worried. Over and over again I heard representatives talking about how important those high-end installations on other operating systems are. The same message appeared when it came to connectors and integration with Enterprise Content Management systems like EMC Documentum. Still, most if not all demos was connected to Sharepoint and/or other MS-specific technologies.

    The technical roadmap means that the past year has been devoted in rewriting their next generation search platform from Java to .Net. The first product that will be released is the Content Integration Studio (CIS) which consist of Visual Studio (I guess earlier in Eclipse) component and a server-side execution engine. This will only be available on Windows since it is deeply connected to the .Net-environment. It looks like a promising product with support for flows instead of linear pipeline to handle the processing of information before it is handed of to the index engine. CIS therefore sits in-front of FAST ESP and a combination of actions in flow and in old pipelines can be executed. Information from CIS is written to the ESP which then creates the index and also processes queries to it.

    What I think we can expect is that new innovation is focused on creating a modular architecture where CIS is the first one. Features in ESP will the be gradually reengineered in a .Net-environment and thus creating a common search platform some years into the future. It will likely mean that we will still see one or two upgrades to the core ESP as we know it today to enable it to function together with the new components. Content Fusion will most likely be the next module that will extend ESP but on a .Net-architecture.

    When it comes to the presentation logic where we today have the FAST Search Front-End (SFE) we will see them either as Web parts for Sharepoint or as AJAX Aerogel from MS. These are currently developed using Javascript but will include Silverlight later on.

    These will initially be offered in both a IIS and a Tomcat flavour and possibly others if there is demand. They will intitially integrated with ESP and Unity and thus opening up for a new approach of developing a search experience on top of them.

    I general I don’t like the Microsoft approach of insisting of owning the whole technology stack by themselves and refusing to invest in other standards-based projects. Instead of developing their own AJAX libraries they could have used ExtJS or even Google Web Toolkit. While it is not open source MS argues that it is a very Permissive licence from MS that has many of the same qualities. A good thing is that MS was comitted to make sure that this framework works on all major browsers including FireFox, Safari and Chrome. It is interoperable with JQuery.

    In summary I think it is kind of a mixed experience. The new features being developed are truly needed to make FAST keep being one of the most advanced search engines available. I think many of the features look really promising and I can’t wait to get my hands on then. On the other hand it is clear that things are going proprietary (FAST ESP had a lot of open source in it), it is being aligned in a Microsoft stack and thus gradually minimizing options. That includes how new technologies are being implemented (MS-ones instead of open source), what operating systems it will run on and how the support for developing presenation logics look like. It means I have to have people how know both Java and .Net, both Flash and Silverlight (possibly JavaFx) and both ExtJS/GWT and MS AJAX/Aerogel.

    We are deeply invested in the EMC Documentum Platform and would of course like to continue use ESP as a way to add advanced capabilities and performance to our architecture. However, I think I will over time get sick and tired on Microsoft sales people trying to convince me to use Sharepoint instead of Documentum. For anybody who know how both platform work it is almost a joke but I will most likely have to keep explaining and explaining. I just hope that we can have decent connector developed for Documentum.

    Too read more you can go to the FAST Forward Blog which has many interviews, look at videos at the Microsoft Press Room and check out the chatter on ffc09 tagged tweets on Twitter. An finally here is what CMS Watch has to say about it.

    Share

    My speech at the Software Development Track at Momentum 08

    On Tuesday it was time for my speech at Momentum 08. It was a first for me at a conference of this kind. I found my room at the top of the conference and it was huge and had seats for around 200 people I would guess. The projector screen was of the back-projection kind and built into a wall with some elaborate theme in blue/gren and yellow details. There was a speakers booth but I almost immediately decided not to use it. I know my voice isn’t the strongest one so I was happy to see a wireless Sennheiser transmitter lying there on the booth. The computer was preloaded with my presentation so it was just to get going. Did not actually feel nervous this time and it feels great to be able to enjoy these situations now. I guess it helps that I am completely passionate about the subject and feel we are doing really cool stuff with Documentum.

    My speech started with outlining how Sweden have changed its focus from a neutral country preparing for a Soviet invasion to an expeditionary force taking part in multi-national peace support operations all over the world. Next I wanted to stress that our project is not driven by the IT-department but rather from the ”business” side although the concept of that is a bit strange in the Armed Forces where actually not make money but anyway.

    Because of that we base our ideas and the architecture based on challenges at an operational level military HQ that we have identified and what we think needs improvement. The main ideas is to be able to put context around all the information that needs to be handled in the HQ and use that context to provide a flexible structure of all pieces of information. That structure allows us to handle a vast amount of information in ”all” formats and be able to ensure traceability and reuse of it. To do that an advanced ECM-system like Documentum is crucial.

    So what have we done to do that then?

    Digital Asset Manager
    DAM 6.5 is our main client and we have made a couple of customizations to that. Some small ones include filtering away object types that users don’t need to see, putting a number in brackets on the Inbox in the left tree structure and to simplify setting object level security. The bigger ones include a new relationship object that extends dm_relation but with a lot more attributes which we expose in a new relationship dialogue. If there are relationships on objects that is also visible in the folder view and clicking on that link called ”View Rel” allows the user to explore objects which are related, edit relationship properties and continue ”surfing” the relationship paths. One big nuisance is that the current release of DAM has a bug (possibly related to the IP-rights integration) which has a significant impact on performance. At least ten times more queries than WebTop to view contents in a folder. Will be fixed with DAM 6.5 SP1 in January 2009.

    Documentum Reporting Services
    So far the information or content analytics in Documentum has not exposed in any obvious ways. However, thanks to the advanced repository all the information is available – it is just a matter of visualizing. Reports which uses Crystal Reports technology provides a really powerful way to create great-looking outputs based on queries to the repository. We use it to report on user behaviour which can tell us how users or groups interact with the content. Another set of reports focuses on the characteristics of the content such as what formats are used but also based on our custom attributes to see trends on how different kind of content is used and created. Finally, reports provide a way to create nice looking template-based outputs on the actual values of all the attributes. An object whose attributes represent organisations can through reports provide documents of all organisations in alphabetical order for instance. In discussions with EMC representatives I found out that information analytics is one of the areas that EMC is now investing a lot of resources into. Based on what we were talking about I really look forward to se the first outcomes from that effort. It will truly make the power of advanced ECM much more evident for everyday users.

    Share

    Speaking at Momentum 08 in Prague on Tuesday

    I have arrived to Prague today to attend EMC’s Momentum 08 conference which focuses around their Documentum product line. I love travelling and especially flying and have no problem at all spending time at big international airports. However, being by yourself in big towns when heading for a dinner is not always that nice. However, now I am having a great time after a warm bath and lying in my bathrobe watching BBC World and using my computer.

    On Tuesday at 0945 I will be speaking at the conference. My session in the Software Development Track is called ”Active Knowledge Management using Documentum D6.5”. The full description looks like this:

    The Swedish Armed Forces uses EMC Documentum D6.5 at the Joint Concepts Development and Experimentation Centre to develop staff procedures for a next-generation operational HQ. Based on the knowledge support concept, a customized EMC Documentum solution, using multiple user interfaces to access all kinds of digital content, has been developed.

    My presentation will cover our workflow implementation using TaskSpace and BAM, our customized relationship object, graphical visualization of relationship and our integration with external search engines and GIS-systems. It will be interesting to get some reactions to our concept and how we make use of the Documentum platform. I hope for reactions not only from EMC employees but also from other big customers.

    Download the full conference programme

    Share

    Reflections from a busy week

    For the third year in a row I have the busiest week of the year when we are conducting what we call an ”Experiment” at work. What it means is that we bring real people in to work according to the outlines in the concept we are developing. It is as interesting as it sometimes is tiresome. Since I am more or less the ”mum” of the design of our information technology platform I struggle hard to make it work as much as I have previously ”advertised” to my colleagues and participants. This year I think we have made a really interesting (and cool) integration of Documentum 6.5, a 4D-database tool called Beata (space+time), FAST ESP 5.2, IBM Lotus SameTime 8.0.1, Carmenta Server and the portal based on InfoGlue.

    This week meant luckily that I could stay at a hotel in Enköping so I did not have to drive two hours back and forth each day. Since the traditional Viking-exercise was going on at the same time in Enköping there where a lot of military officers from other countries staying at the hotel. I wrote about my experiences at at a previous exercise before and this time it was nothing like that at all. I could wear my nice blue uniform with a skirt which makes me feel both pretty and comfortable. I was together with a colleague more or less in control of all that happened and the experiment participants have been really nice. However, at the hotel I had somewhat of a deja vu some mornings. Maybe it was my uniform or the sounds of my heels, I don’t know but with all these mostly male officers from all over the world I felt so stared at that it became uncomfortable at times. During the evenings when we usually had some food in the bar it felt much better, maybe because I was sitting with somebody all the time. I told a friend of mine about it and he just told me that they were only looking because I was so good-looking. Hmm…I like to believe that but there is that something in people’s eyes that says something. Maybe it is because I am a female officer – I know other women in the Armed Forces have told me similar stories.

    Share

    First impressions of Documentum Digital Asset Manager 6.5

    Before going on my planned sick-leave I played around with DAM 6.5 for a while. I will try to summarize a few reflections I have on this brand new release.

    Good things
    The interface have got yet another refresh but rather small modifications that I guess I won’t even notice in a couple of weeks. The biggest change is that some functions have got modal windows meaning that when you click on properties you no longer see the big full screen page but instead a new browser window that allows you to see where you where when you clicked. A great improvement I think. The import/export/check-in process also has small modal windows with a nice looking update progress bar.

    A thing that I just love is the new clusters/facets features which appears when performing a search. Your results can then be drilldowned based on user, topic, date and so forth. Will improve findability hugely. We had these installed in D6 SP0 but they did not work then and seem to be connected more closely to ECI Services back then.

    In general the interface is prettier and looks more distinct and modern. The icons have been slightly improved as well.
    Another small improvement is that attributes which have both value assistance (dropdowns) but also allows entering of an own value now have the correct width.

    I guess it is not really connected to this upgrade but I finally manage to find how one creates Presets (rules) for specific folders and users which was great. Look at the three structure in DA – not in the menu.

    Bad things
    The left tree structure has been cleaned up with clearer icons and the update is based on AJAX (or should see bad below). This works fine in Documentum Administrator 6.5 but for some reason they seemed to have missed something in compiling DAM because there is small refresh anyway when you click on a folder. Our partner suggest that they simply have inherited from the wrong WDK-class.

    Another interesting thing is that some features that are highly marketed at EMC World are turned off by default in the configuration files. Those include Deep Export and OLE-linking support (resolving links in Office documents and imports associated files if desired). That is rather strange I think since those are really handy features. The OLE-linking can also be toggled on/off in Preferences. The effect of that is that there was no folder export available at all which is fairly strange. We also had some issues with getting import of more than two folders working.

    We also have a an irritating issue around thumbnails. It seems that those can not be created for PDF-files at all which also means no storyboarding. When reading through the release notes this is noted as a known bug and it seems that despite our bug report from earlier this year nothing has been done to fix it. From a usability standpoint that is not so good.

    Share

    EMC Documentum CenterStage

    If you haven’t done it already I recommend a look at the site for the beta of EMC Documentum’s new web client called CenterStage. This modern Web 2.0 client has earlier been called both Magellan and IntelliSpace but EMC now seem to have settled on the name CenterStage. It is kind of funny because I associate CenterStage with a TV application for Mac OS X which is found at the CenterStage Project site. Anyway it is interesting to see how the interfaces and feature looks for the free CenterStage Essentials (included in any Content Server license) and the paid version called CenterStage Pro. I have long waited to for a good application that both can do ”Facebook for the Enterprise” while still having all the features of an advanced and full-fledged Enterprise Content Management platform. This seem to be a big step towards that. The key thing is to be able to collaborate both around content (documents etc) but also around people, groups and projects. Although there are good collaboration platforms out there such as Clearspace which has some basic integration with Documentum it is still creating a lot of duplicate information in separate ”stove-pipes”. I want the content objects found in the Clearspace platform stored in Documentum but this is not the case today. What we are looking at is being able to search Documentum content from Clearspace for the immediate future.

    Again, back to Centerstage I believe it will provide a lot of organisations with a client that will be a lot more intuitive and useful out of the box than we have ever seen from Documentum before. This is thanks to an ambitions usability project run by Gideon Ansell in the Documentum User experience group. However, after having had a looked at the project release matrix found in the beta community it looks like the beta of CenterStage essentials will not have enough features to be the flexible collaboration client I need. Those features will be added later on this year. We just have to wait for the full CenterStage Pro version I think. I also hope that the few missing pieces like a full fledged personal profile, expert location and integration with external presence/Instant Messaging systems will be on the schedule for the next update of it.

    This week I will be able to play with the Documentum 6.5 release for the first time since it is being installed by our Documentum partners at work. I especially look forward to see the new Digital Asset Manager (DAM) 6.5 client and TaskSpace 6.5. I also hope that I can get some further information about what release of the embedded FAST InStream Search engine is used in this release.

    Share

    I will be a speaker at Momentum 08 in Prague

    This weekend I got a nice surprise in an email from EMC which confirmed that my speaking proposal had been accepted. So now I guess I will be going to Momentum 08 Prague as a speaker! It feels great and I look forward to it a lot. Unfortunately it is in the middle of a very important week at work but I got confirmation from my commanding officer today that I can at least fly down for the day to give the speech and possibly also check out the other sessions that day. My sessions is entitled ”Active Knowledge Management using Documentum D6” and will be focusing on how we use Documentum as a foundation to work proactively to support end users in finding, organizing and utilizing their information in a military HQ. Fortunately it looks like we are going to do another major step in taking our platform forward and that will not only include a very early upgrade to D6.5 but I will also get my data-deduplication feature 🙂 It sure will be an interestung autumn.

    Share

    Interesting details about Documentum licensing

    Today I learned some interesting things about the names and licensing of Documentum products. MediaWorkSpace which was previously announced as included in the DAM license will now require its own license. A little bit strange since it is merely handling image files so far. The Magellan essentials license however will be included in each Content Server license which is good. The full Magellan client (which for a while was called IntelliSpace) will be a separate license when it will become available. I also learned that there are no longer any licenses for Reporting Services developer which limits the number of Crystal Reports designer tools in your system. Instead there are is a CPU license for Reporting Services allowing as many designers as you like in your system. Furthermore I learned that what is labeled Business Process Management includes both Documentum Process Builder and Forms Builder but only used one at the time on one machine. Forms Builder can be bought as a separate license. We had intended to get Forms User licenses but we chose to for some TaskSpace clients instead where Forms User is included. At EMC World we also learned that the Process Engine DocApp is required to run BAM in Documentum D6.5

    Grateful for any clarifications or corrections if someone have found out something else.

    Share

    EMC World 2008 Reflections

    This was a great conference I think which does not mean that things can’t be improved but let me come to that later on. I like the fact that the conference is very technical because it gives me a way to find out what can be done with this large toolbox that Documentum is. I found most more business-oriented presentations to superficial and the key point of all of them is usually reduced to: save money, do things faster, do things more efficiently and sell more stuff. All of which is pretty self-evident. I think it is great to have the ability to meet the product managers and ask those tricky questions that you have collected during the year. EMC also seem to be a fairly open company so roadmaps are frequently presented and discussion.

    So how do I feel about Documentum now then. To sum it up: Finally, they understand the importance of sleek, nice and usable user interfaces. I have heard many saying that Documentum is a great platform but with rather crappy user interfaces. Now that is definately going away. It was so great to see the new Magellan interface which actually addresses many of the real capabilities that modern so called Knowledge Workers need. The MediaWorkSpace application which soon will have DAM capabilities is even cooler. A lot of the design is also inspired from a great company in Cupertino, California called Apple. MediaWorkSpace almost feel like a server-based version of iPhoto sometimes. Everything is getting service oriented which is just great for the integration efforts we all have. WebTop/DAM is also getting a visual overhaul which makes them on par with what you can see in Alfresco for instance. TaskSpace is getting even cooler in 6.5 and the configuration of the interface can almost be called easy now. So Documentum is definately looking better than ever now.

    What can be improved then (and here I agree with Word of Pie a lot):
    – Strengthen the Documentum (CMA) community by having all of us wear the yellow tag so we know who we can talk to about these things because the 6000 storage nerds does not seem to be interested at all in software.
    – Move all the sessions into one area with a soft hang-out area in between.
    – Previous year they apparently had something called ”Meet the speakers” and that seem like a great event.
    – I would like to have more focus in the sessions on particular products so you can plan for getting updates on all the ones you are using. I would for instance have liked a designated one about the DAM client. Combined them if necessary but state explicitly which products that are covered.
    – More plugs for my laptop powercord. After all this is an IT-conference I really like to take notes on my computer. Even with double batteries for my PowerBook it is hard to last from 8.30 am to 6.15 pm.
    – Bigger screens. Some rooms are huge and it is real hard to read the presentations from the back.
    – Why not record at least the CMA sessions on video. After all EMC is all about digital content and it could be a nice show case of the rich media capabilities.
    – Case studies are better than overview sessions. Always to hear more concrete experiences from customers.

    Next year it is in Orlando. Maybe I am giving a speech then 🙂

    Share