Bob Balaban's Blog

     
    alt

    Bob Balaban

     

    Huzzah! New LSX Toolkit is posted (Part 1)!!

    Bob Balaban  March 31 2009 02:13:39 AM
    Greetings, Geeks!

    I am pleased to report that a very long, and sorry saga, lasting more than 6 years, has finally come to a close! I am, of course referring to the LSX Toolkit. I got my new version of it today, the previous release having come out in 2001. That's a long time to wait, but Friends! The waiting is Over!

    Few living now remember (or care) what the brouhaha (brew? ha ha!) as all about, but I will give you a radically shortened history.

    What is the LSX Toolkit anyway?


    LSX (originally "LotusScript Extention") is an architectrue that was invented in the course of developing the LotusScript "back-end classes" for Notes version 4.0, back in 1994 or so. We (primarily me) were developing the original set of back-end classes (meaning, they would run on either the client or the server) using what we'd now call "extension points" in the LotusScript interpreter code.

    English translation:
    A developer could create a bunch of C++ classes, put them in a DLL, and have the "host product" (Notes in the initial case) load the DLL. The host would, as part of the load-and-register protocol, would provide to the DLL code a specially crafted object instance of the LotusScript interpreter. This object has methods that allow "foreign" (or "extension") code to "register" the classes and methods and properties and errors that it implements with LotusScript. From the LotusScript developer's point of view, suddenly a whole bunch of new classes were available to be used in the LS editor and runtime environments.

    The Notes back-end classes were the first ever LSX. The LSX-i-ness of that DLL (nlsxbe.dll on windows, other name prefixes and suffixes on other OS platforms) resided in the fact that without that DLL, Notes had no "native" LotusScript classes at all (beyond the ones built to implement the editor and debugger). The DLL is hard-coded to load automatically when Notes (and now Domino) starts up. Other LSXs can be dynamically loaded when needed with a "USELSX " type statement in a LS program.

    This had two very interesting and immediate impacts on the thinking of how LotusScript could be used:
      1. Since, at that time (1995) Lotus had several products which embedded LotusScript (remember SmartSuite? 1-2-3? Freelance? Wordpro? Approach?), these "desktop products" could (easily, it turned out) be made to ALSO use the Notes LSX! You could run a script in the spreadsheet that could (behind the scenes, as these were the "back-end" classes) have 1-2-3 talk directly to the Notes objects. Cool!

      2. Anyone could write ANY DLL that did, well, ANYTHING, and make it a "plug-in" to either the client or the server, exposing entirely new (and not necessarily strctly Notes-related) functionality.

    And both of these things were done. SmartSuite products learned to be able to load the Notes LSX. People actually built apps using it. More people created new LSXs to be used with Notes/Domino to do different things: connectors to outside data sources (eventually rationalized and collected into the Lotus Connector LSX, a single LotusScript API to talk to a variety of outside (non-Notes) relationsl databases. Still in use today!!

    One major problem remained: it was HARD to configure an LSX just so such that your classes and methods and constants and errors would all get registered properly. And it was HARD to set up the code so that when the LS Interpreter needed to invoke a function whose code resided in an LSX, it would happen correctly, passing through the right data structures and so on. The first few LSXs that were built were all hand-crafted, with a lot of help from those few who'd done it before.

    So the cry went out across the Lotus Rogers St. building in Cambridge, Massachusetts: "We need a TOOLKIT for this bugger! It needs to be easier! And it CAN be made easier!"

    Fortunately, management at the tiem were very responsive and supportive. A team was created at Lotus, and they architected and built the first edition of the LSX Toolkit. It was AWESOME. A thing of beauty. You operated it by creating documents in a Notes database, describinb what you wanted (classes, methods, etc). Then you pushed a button, and it generated a crapload of C++ code that created the "skeleton" of your implementation (of course, the "Wizard" as it was called knew nothing about your business logic, but it COULD create all the code you needed to build, register with LotusScript and accept callbacks for evaluation at runtime. You would then go and "flesh out the skeleton" to add the business logic.

    I think that first version came out in 1998 or 1999, only 4 years after Notes v4.0 had shipped.

    I wrote an LSX to sell as a product at my then new company (I had departed Iris Associates in 1997 to create Looseleaf Software). I had decided that it might make me some money (and be fun) to write an LSX that expanded the reach of the Lotus-owned back-end classes across more of the CAPI functionality than was currently available to LotusScript. Is there a Notes CAPI function you want that you can't (easily) access from LotusScript? Tell me, and I'll "wrap" it in a class in my "ScriptExpander" LSX, and you can buy it from me.

    I found lots, and people suggested others. I sold a bunch of copies over a few years. In fact: Here's what I'm gonna do. EVERYONE reading this blog post can HAVE (for free!) my Notes database that contains the entire product, except for the DLL itself. You can read the documentation, take a look at all the sample agents that show you how each and every one of the 100 or so LSX functions worked. Go on, take it! I said it was free (about 1.5mb zip file containing 1 NSF):
    lsxdoc.zip

    Much cool stuff in there. You want to set the replicaid of a database from LS? It's in there. You want to create a relica stub on another server (not a full replica, the way Notes -- at the time -- made you do it? It's in there. Need text fields up to 2gb in size? It's there.

    Other people wrote LSXs too. The most famous is probably Ben Langhinrich's MIDAS rich text conversion classes (http://www.geniisoft.com). Bill Buchan (http://www.hadsl.com/) wrote one for his FIRM product.

    Life was good (or at least, fun). The toolkit supported all of the client and server platforms for Notes and Domino. An updated kit came out in 2001. Some of the code that glues your LSX stuff to the LotusScript stuff and brings them together is actually delivered in the kit in source code form. I found a couple of bugs there (Reported them, of course!), but I was able to just fix them locally, 'cause I had the code. Life went on.

    As a few of us used LSXs more and more frequently, we noticed some, er, glitches that made things, um, inconvenient. Some could be worked around by tweaking the skeleton code after it got generated, other stuff (crashes in the toolkit wizard, for example), we had no source code for. A list of complaints began to accumulate, but Lotus was, um, slllloooooowwww to acknowledge that fixing these were a priority. The group that had been maintaining the kit was disbanded (ALWAYS a bad sign).

    Yet, I (and oathers) had real products out in the market depending on this technology. What could we do? It was frustrating. So. I think it was 2003, or maybe it was 2004 (I admit to being hazy about this, heck it might have been 2002) that I showed up at the "Meet the Developers" Session at Lotusphere (I think now it's called "Ask the Developers", we just always called it "Beat the Developers"). I happened to be invited to ask the first question. I mentioned that I was having a bunch of problems with the lsx toolkit, and said that, if Lotus wasn't really staffed to fix the problems, I would be happy to donate some time for free to sit in the Westford lab (where I used to work) and do it for them.

    At first I thought something might actually happen. One Vice President asked for my bizcard, and said he'd call me to set it up. He never did. The following year, I was again able to position myself to be the first one called on at "Beat the Developers" at Lotusphere. I repeated my offer. And again the following year! This continued right up to Lotusphere 2006, the first oneat which I was again an IBM employee. This time, a few other people asked if, now that Bob was back at IBM, would the kit get fixed. Damn thing was turning into a joke, and all I wanted was for it to be fixed.

    I was given permission at Lotusphere 2007 to pull a project together to begin the work. Of course things had deteriorated. Whereas the 2001 version of the kit built fine (on Windows) with Visual Studio 2003, it did not work with the 2005 version. There was an extensive list of outright bugs and a fairly long list of important enhancements. To be honest, I didn't spend a lot of time on this project. But I did convince a couple of other developers to do most of the work. We engaged with the release management team to get trhough the paperwork and legal review. At Lotusphere 2008 "Beat the Developers", I "announced" that we were finally going to ship it within a few months.

    Then, in February of 2008, I left IBM to join a Lotus Business Partner company. LSX pretty much fell off my radar. But I did notice, around early January of 2009, that the updated kit had still not shipped. EIGHT YEARS, and they were still sitting on it. Once again, for the 5th or 6th (maybe) time, I complained at "Beat the Developers, Lotusphere 2009 Edition". Brent Peters made a joke at my expense, implying that the delay had been caused by the poor quality of my code. I guess he didn't realize that virtually none of the code in the LSX Toolkit was mine.

    But! Brent also promised that the kit would be released by Feb. 1. Whoo hoo! Light-at-end-of-tunnel! AND, light-is-not-train-coming-my-way! I forgave him for the joke.

    After Lotusphere, Feb. 1 came and went. Brent blogged that the date had slipped to Feb. 15, which also came and went. Buchan blogged about the delays in a way designed (successfully) to embarrass IBM. Brent stepped up in a posted reply and said he agreed, enough was enough, he was going to make sure it got out. Soon.

    Sure enough! I read on Brent's blog last week (http://www.LotusStaffNotes.com/LotusStaffNotes/LSNblog.nsf/) that it was REALLY READY!!! (And! he dedicated it to me. How sweet!).

    BUT! I couldn't find it on the old toolkit site (http://www.notes.net). Only the old one was there. But TODAY! I got the link to the page where it REALLY lives:
    http://www14.software.ibm.com/webapp/download/preconfig.jsp?id=2006-09-14+09%3A05%3A37.162620R&S_TACT=104CBW71&S_CMP=

    Note: This is the windows-only version, unix version to come....soon. You have to register to get the download, but it's free.

    Phew. I'm exhausted. Eight years! And now it's here. I'm thinking I may just begin a project to revive and extend ScriptExpander with this baby, see how that goes. You can assume I will be blogging about that experience.

    But for now, MANY thanks to:
       Brent Peters, George Langlais, Scott Morris, Willie Arbuckle, Roberto Olivares, John Beck
    and everyone else who helped get the darn thing FINALLY out the door.

    The saga continues! This has been but the end of the beginning!

    (Need expert application development architecture/coding help? Contact me at: bbalaban, gmail.com)
    Follow me on Twitter @LooseleafLLC
    This article ┬ęCopyright 2009 by Looseleaf Software LLC, all rights reserved. You may link to this page, but may not copy without prior approval.