In Theory...

"Clouds", the sequel

Bob Balaban — Tue, 24 Jun 2008 20:15:27 -0400

Greetings, Geeks!

I was going to write a follow-up post to my "Clouds" article of about 2 weeks ago, and the focus was going to be some interesting aspects of "cloud" architecture: how it's different from "regular" client/server, how you use it differently, and what benefits you derive thereby. I'm still going to write that article, but not today. Today is still about cloud computing, but after looking at several of the replies to my earlier post, it's clear that many people see real problems with buying into the cloud model of computing, particularly with the fact that you're handing all your data over to someone else, and you may very well not even know where that data is being stored.

Commentator Kerr points out (rightly) that companies give their data to 3rd parties all the time, so what's the big deal? (I'm paraphrasing)

Commentator (and former Burton Group analyst) Karen Hobert usefully distinguishes between "web centric/saas" clouds and "data center centric" clouds. Google is, I think, mostly in the former category, whereas Microsoft Online, or IBM Strategic Outsourcing, say, are in the 2nd category. Yes, Google runs from data centers, but you don't know where they are (the locations are secret), there are many of them (the number is secret), and you have no idea in most cases which one (or ones) have your data.

Kerr's point that others already have their hands on your data is true enough, but I would claim that in order to be successful as a business, companies like Google who want to "host" not only your apps, but your data and your email/application infrastructure as well need to convince their customers that they have certain things under control:
- Reliability. If I give you my stuff, I need to be confident that I can get it back in good shape, whenever I want it.
- Security. I need to be confident that nobody else can hack into my stuff
- Privacy. I need to know that the holder of my stuff (Google, Amazon, whoever) is not reading it for their own purposes. It's ok to create a search index for authorized users to access, but you better not be mining MY stuff for aggregations, usage patterns, or anything else.
- Locality. Many users of "cloud" services don't care where their data are stored. Others do care, some are required by law to care. Financially regulated companies, for example corporate auditors, are often required by law to keep their data stored within certain political boundaries (country, state, whatever). This is true in some states within the United States, and also for many countries within the European Union. Google offers to guarantee (for an extra cost, one imagines) that if a customer has this requirement, and if Google has a data center that meets the locality requirement imposed on the customer, then that customer's data will not be stored outside that data center.

As an aside to that last point on Locality, I heard a Google executive remark during a presentation that Google is receiving a lot of requrests from non-U.S. customers for guarantees that their data will be stored only OUTSIDE the United States. The reason for this, of course, is due to the rather fascistic (my opinion) provisions of the USA Patriot Act, which allows certain government agencies (FBI, CIA, NSA, probably others) to snoop your data by simply declaring to the holder of that data (your ISP, your phone company, a public library you visit to use their books or computers) that they need to. AND, the provider of access to your stuff is PROHIBITED from notifying you that this intrusion has happened. [sarcasm]Real "land of the free, home of the brave" stuff, huh?[/sarcasm]. As usual, those who can will vote with their feet, the rest of us have to hope that a new regime will overturn this nasty law.

One other legalistic implication of cloud computing I wanted to point out: today it is perfectly legal for companies to snoop employees email (traffic and/or repositories) and other files, so long as the company is the owner of the computer on which the data resides. This is true in most states within the U.S., and (I am told) throughout the E.U., and probably other places as well. Of course, it's much more of a speculation to try to guess how many companies actually take advantage of this legally permissable behavior. I'm sure some do. Others may only snoop when they have a real reason to: they're worried about a specific case of industrial espionage, perhaps, or maybe a crime is being investigated.

Naturally, the normal reaction by the "sensitive" among us to this situation is to start (or continue) encrypting everything. But there are limits on this. In Notes, you can encrypt databases stored locally on your Client machine (laptop, say), but you can't encrypt the server replica of your mail database, because if you did, the server would not be able to deliver mail to it. You can encrypt all of your outgoing mail traffic in Notes pretty easily, but you're still reliant on your Notes-generated key to do so. Are you completely certain that your employer hasn't sequestered a pre-determined set of bits which are used in all keys? Really? You're sure? Well, it was well known in the 90's that Lotus did a deal with the U.S. Government to provide them with some of the bits used in all non-NorthAmerican keys, reducing the effectiveness of any keys used outside NAmerica, and thereby making it easy for U.S. government agencies to crack encrypted messages. In return, Lotus won the right to not have to have 2 separate security systems in Notes (one for the US one for everywhere else), and also got Notes removed from the list of "weapons" maintained by the Defense Department. Being on that list was awful, it (technically/legally, though it was rarely enforced) prevented people from taking computers containing the US version of Notes out of the country.

Anyway, the real point I'm trying to work towards is this: do employers lose the legal right to snoop employees' data when they've outsourced data storage and applications like email to Google? The employer no longer owns the computers that the stuff sits on, Google does. So your employer may no longer have the right (or the ability) to read your email. Maybe that's a good thing (depends on whose side you're viewing the issue from, I guess). But consider the flip side: if all of your stuff is on computers owned by Google, does that give Google the right to snoop it, if they choose to? I'm not a lawyer, so I don't know. I suspect part of the answer lies in the terms of whatever agreement the employer and Google contract to. Perhaps Google will readily agree not to peek at your stuff in their data centers. But if they do anyway, are they obligated to tell you about it? What is the enforcement mechanism?

Yet another wrinkle: within some regulated industries, the law (such as Sarbanes/Oxley in the U.S., as one example) imposes so-called "Chinese walls" between certain groups of employees. This means that for some organizations, they must institute email tracking and archiving, AND they must prevent email from flowing between certain groups. Nobody in the trading department (for example) is allowed to send email to or receive email from anyone in the analysis department. I've worked on such security features for a couple of customers, generally with Notes/Domino you would write a special purpose Extension Manager plug-in to monitor the traffic and kick back "invalid" messages (there are other ways of doing it, too). In this situation the "security" code is not reading the message (though a copy is probably being stored away somewhere, in case it's needed later), it's only scanning the From field and the recipient lists, and looking for invalid pairings.

Still, I don't think (as of now) that Google offers a service like this (they do offer archiving and other security/spam filtering features through their Postini acquisition), I could be wrong. If someone uses a company computer to commit a crime (download kiddie porn, perpetrate embezzlement or fraud, or whatever), the company may have to accept some liability. If you set up a scam site on Google Apps and cheat people by stealing credit card numbers, or something, does that make Google liable too, because you used their computers?

The point is that when you shove everything into "the cloud" you give up a measure of control over your content and over your operations. The implications are sometimes subtle, but nonetheless important. Should be great for the lawyers!

(ok, NEXT post on cloud computing will, I promise, get into the architecture/geeky stuff....)

"I’ve looked at clouds from both sides, now"

Bob Balaban — Mon, 16 Jun 2008 14:50:08 -0400

I've been working with Google Apps APIs and trying out various features of the mail, calendar and contacts services (products?) for a few months now. I think I'm getting a handle on the whole "cloud computing" thing. So I thought I'd collect a few observations in a blog post and see what y'all think as well. This is neither a sales job nor a trash job on Google, nor is it meant to be a comprehensive feature comparison of Google Apps vs. Notes. I'm really using Google Apps as a prominent example of "cloud computing", and starting to think about the implications for the collaborative software world.

What is 'cloud computing'?

Here's a Business Week article that attempts to define it. I think it does a reasonable job, and it points to IBM as a technology leader in that space. But while I"m sure that IBM has pockets of advanced technology and even offerings in the "cloud", it seems fairly obvious to me that Google is far and away both the technology and the market leader in cloud computing (Microsoft has stuff too, naturally). One could argue with that conclusion, I suppose, and point out that while Google has more offerings in the consumer/mass-market space (gadgets, youtube, etc), they are not necessarily the leader in "enterprise" markets.

Maybe. I don't know, and that's not the point I'm trying to make, anyway: this isn't a vendor comparison.

To me, the basic point (or maybe there's more than one point here) of cloud computing appears to be this: cloud providers (if we can call them that) will not only host apps for you, they will supply you with virtually unlimited storage space, and virtually unlimited scalability of access, as well.

Sign up for a free GMail account with Google, and you get 6 GIG of storage. Sign up for a $50 per seat per year (+ or -) enterprise account, and each GMail user gets 25 GIG of storage, plus more features. That's a lot of email. On the scalability front, Google will happily host your enterprise email regardless of whether you have 3 seats, or 300,000 seats. They offer good searching (duh), security, spam filtering, archiving and so on. And you hardly ever have to delete anything, with that much space available.

Major Benefits of 'cloud computing'

The biggest argument I see cloud vendors making for people (and companies) to switch over is the economic one: when you go to Google Apps you're not just going to a "Software As A Service" (SAAS) model, you're outsourcing your entire email infrastructure. No more in-house maintenance of hardware and networking, no more paying pesky admins to wake up in the middle of the night and fix stuff that broke, no more having to re-evaluate and do software/hardware upgrades every year. Google handles ALL of that for you, and it's pretty cheap. You just hired another 100 employees? Go online and create accounts for them on your hosted site (or write a program to do it for you via the Google provisioning APIs), and you're done.

Google never goes down, they handle all the backups, authentication, access control, routing, everything. And it's all accessible with a browser.

What are you giving up for 'cloud computing'?

So what's the downside? Well, you do give up a few things:

- Functionality. Let's face it, GMail is not the best email system out there, feature-wise or UI-wise. It's still in "beta" 4 years after its launch, it does WAY less than, say, Notes. Same for the Google Apps calendar (one annoying example: they don't allow attachments on meeting invitations...)

- "Ownership". By that I mean that when all your email is in GMail, you don't ever know where it really is. Same with all the other "cloud" apps -- your data could be anywhere. For a lot of people this isn't a big problem (once they get over that initial slightly queasy feeling, especially when they remember how much money they're saving). But, for some people, it's a show-stopper. Accounting firms, for example. In some countries (and in some states within the U.S.), regulated firms are prohibited from storing sensitive data outside the boundaries of their political unit. Other firms worry that data stored on Google servers that happen to reside in the United States might be vulnerable to arbitrary U.S. Patriot Act subpoena.

- Application deployment. Email is one thing, enterprise applications (especially custom-built ones) are quite another. You can't go to Google and ask them to give you a Domino server to run your mission critical apps. You can deploy Web apps on Google now, but only if they're written in Python (or something like that). Maybe that will change, but in the meantime, I think most companies who may want to use Google for email will hang on to their app servers for quite a while. But that leads to what I call the "coexistence" question: Since quite a lot of Notes/Domino based applications use email messaging as a transport (many workflow apps, for example, send email notifications to users), how does that work when my email moves to the cloud? Do I have to rewrite all those apps? Yikes. (Quick plug here - no, you don't. Binary Tree (and some other vendors) have coexistence solutions you can, uh, buy).

Bottom line?

Are the negatives show-stoppers? Clearly not, Google (and Microsoft, and maybe even IBM someday) seem to be very happy with the rate of growth of their cloud offerings.

On the first point (Functionality) -- yeah, ok, maybe it's not so great now, but it'll get better over time. And there are evidently a lot of CIOs who look at the cost savings and (perhaps, I'm just speculating here) say to themselves, "Wow! My bonus is going to be HUGE this year. I guess my users can live without a few bells and whistles on their email. No more whining about increasing the mail quotas! Yee-HAH!"

On the "Ownership" issue, obviously this is a nonstarter for some types of organizations (I don't see the CIA or the military going for it anytime soon). For others, it's just another cost issue: Google will (for a fee) offer certain guarantees about where your organization's data will reside. If you're in (I'm just making up this example) say, the Cook Islands (yes, it's a real place, go look it up), and you really, really, need all your bytes to stay in the Cook Islands, and you can show Google that there's enough like-minded organizations in the Cook Islands to make some money, then maybe Google would simply build a data center there. Or, maybe not, and you find another solution.

As far as coexistence goes, there are technology solutions, and if you have really important apps that you want to keep on Notes, then the cost is still probably worth it. Careful planning is required, and YMMV.

I could go on (and on), about APIs, RESTful interfaces, message interoperability architectures, you name it. BUT, I won't, this post is already long enough. Maybe I'll do a Part 2, if there's interest.

What I REALLY want is for YOU to tell me what you think -- impressions, experiences, razzing, you know, the usual.

And, in the end, "It's cloud's illusions I recall. I really don't know clouds at all!"
:-)

I will be showing up....at ILUG (and presenting a session)

Bob Balaban — Fri, 16 May 2008 13:20:20 -0400

Greetings, Geeks!

Just wanted to let you know that I will be presenting a session (and attending many others) at the Irish Lotus User Group (ILUG) get-together in Dublin next month (June 4-6). My session title and abstract are:

Achieving "Peaceful" Coexistence Between Microsoft Exchange and Lotus Notes
While many organizations strive to standardize their messaging infrastructures on a single platform, many others do not. Whether through merger, acquisition or simple user preference, many organizations face a need to achieve peaceful coexistence in the messaging wars, perhaps for only a short time (while they transition from one to the other), but perhaps (if they can pull it off) for a very long time. The technical challenges involved in maintaining a well-performing dual messaging infrastructure are not trivial, but they can be met successfully with proper planning and the right technology. The big issues are all around directory synchronization, message transport, rich text, scheduling logic differences and data formats. Come to this session to learn how to overcome these technical challenges how to keep both Notes and Exchange up and running, and how to avoid mutually assured destruction!

Hope to see you there/then!

QA and Dev, the "Itchy and Scratchy" of Software Production?

Bob Balaban — Tue, 13 May 2008 13:13:39 +0100

Greetings, Geeks!

I'm hoping that many of you out there are fans (or recovering fans) of The Simpsons, in which case you'll have no trouble recognizing the names in this week's post as the characters in the always-fighting cat and mouse TV show beloved by all the children of fictional Springfield. If you're closer to my age (Boomer) than to Gen-X, and don't have any kids that you watch TV with, the equivalent characters of our time were probably Ben and Jerry (the cat and mouse cartoons, NOT the ice cream makers!)

So, the thing I want to get at thoday is, why can't we all just get along? Nah, that's not it. Let me try again: Why is it so darn hard to release quality software? Nope, that's not really it either (but it's a good question). Ok, I think I have it now: If you were working in a relatively small software-making company (let's say, between 20 and 200 employees, and by "software maker" I mean, creates and sells software as a primary business), and you had to figure out how to organize the people doing the actual software creation and (we hope) testing, how would you set that up?

Of course we need to constrain the question a little bit more, so as to be answerable in our lifetimes. Here are the (arbitrarily ranked) goals I would want to maximize in such a situation:

Minimize the nuber of defects (bugs, glitches, doc errors, user errors whatever) reported by customers (defined as, the people you pay you for your software). Notice that I lump "user errors" in with the more common kinds of problems. I do that because, IMHO, a "user error" (the user did something "wrong", resulting in a bad outcome, but the software is "working as designed") is rather more likely to result from a problem with the User Interface (UI) or perhaps with the documentation, leading the user to expect a result other the one s/he got when they did whatever it was. We could argue whether these are "real" bugs, or "equal" in some way to "real" software problems, but I don't care. By my definition, this kind of error is a quality problem in the product.

Maximize the amount of automated testing a product can undergo before release. I put this item in here because experience (and arithmetic) has shown me that release cycles are dramatically reduced when you apply automated testing to a product. I won't belabor the point, and yes, it's true that you can't generally achieve 100% automation (at reasonable cost), but test automation is a good thing.

Keep as many employees happy in their jobs as possible. Yeah, yeah, I know. There are a LOT of factors that affect employee satisfaction, but here's one thing I have noticed myself, and at more than one company where I have worked: implementing caste systems leads to unhappiness among employees. Creating sub-divisions within the broader "development organization" where one group of people (let's call them the "Lords") is responsible for the fun, creative, and more highly paid work of writing (inventing, architecting, designing, coding...) software, and another group (the "Serfs") are responsible for taking the work product of the Lords and testing it (looking for defects, broadly defined, which might also include things like unacceptable performance, poor UI, etc.). Most companies at which I have been an employee implement (whether blindly, or on purpose) this kind of 2-tiered system. The Lords (developers) get more money, and more status than the Serfs (QA/QE*, pick your terminology). Some people feel that this system is "natural", after all, the developer's job is harder and more creative. The "QE" role is to receive and test, a rather more "mundane", yet unfortunately necessary step in the release cycle. Anyhow, the point here is, your job (as the hypothetical keeper/maker of the org chart in this example) is to try to not have a caste system.

Minimize the cost of producing quality software. Since we're talking about the software business, there has to be a business constraint on all of this. Headcount is expensive. Back in the days when Lotus was mostly a spreadsheet company, the normal ratio of QA to Developer headcount on any major project was 3:1 or so. For every developer creating code, there were 3 people testing it. Today that would be inconceivable.

So? Now what? We can probably all agree that there's a difficult problem here. What's the answer?

Speaking for myself, I don't have an "answer". There probably is no one single "answer". What I have, though, is a hypothesis. Which is:

[hypothesis]
If you view the problem of "software product quality" from a broad perspective (as I described it above), then your approach to building the software cannot treat "QA" (or "QE", or whatever you call it) as a distinct operation from "development". ^**
[/hypothesis]

One of the things this implies for software companies is that "Development" should probably not be a separate organization, or department from "Testing" (QA, etc.). I would almost advocate that within a "Software Development" organization (let's say, a team dedicated to creating and evolving a single product, or suite of products), there is indeed reason to declare and nurture a "specialization" of skill that is distinct (i.e., not all architects/designers/coders need to be expert in testing, and not all testers need to be expert in software design/implementation), but really, isn't there an awful lot of overlap? Don't testers benefit from understanding the product's construction? Don't developers benefit from an understanding of a testing process, so that they can (as one possible example) instrument their code appropriately? For sure, once you get to thinking about test automation, someonehas to build the automation harnesses, script the test runs, etc. Right? Is that not "development"?

So, is there a benefit to organizing around a "QD" (Quality Development?) job function, and organize a primarily "development" team where perhaps there is some internal specialization (but not necessarily a clear distinction) between creating software and making sure it works properly?

What do YOU think? How would YOU organize for quality, if you could?
-------------------------------------------------------------------------------------------------
(Footnotes) ^* In fact, when I first started working at Lotus Development in the late '80s, the "testing" department was called "QA" (Quality Assurance). Around the time that we shipped Notes V4 (mid-'90s, before IBM bought Lotus), the Lotus QA organization working on Notes was merged with the Iris organization (developers of Notes), which had been composed almost entirely of developers. Somewhere in there, the term "QA" was explicitly switched to "QE", or "Quality Engineering". When I asked why, I was told that people felt it was a better name, because it more accurately reflected the engineering basis of the task, and that it might improve the level of respect given to the job title and to the people. To this day, the terminology (at the Lotus division of IBM, anyway) remains "QE", though I, personally, haven't detected any significant reduction in the caste differential (though of course these things are always in the eye of the beholder, YMMV). ^**Again, the underlying assumption here is that you're trying to optimize for quality. Naturally, many, many companies (businesses) do not inherently optimize for quality, they try to optimize for profits, or maybe for market share, or maybe for personal career growth, or maybe for something else. I'm not primarily a business person (maybe if I were, I would still be running my own company!), so I can't say whether quality is the thing for which a business should globally optimize, and certainly, from an employee perspective, working on a "quality" product doesn't make me happy if I can't get paid for it. But it does seem to me that quality should be an important objective of product development, a "differentiator", if you like. :-) But that's another discussion.

Calling Notes CAPI from C#/Visual Studio

Bob Balaban — Mon, 5 May 2008 13:11:47 -0400

Greetings, Geeks!

How many of you have ever written LotusScript code that needed to call C entry points in some DLL (such as, maybe, the Notes C API, in nnotes.dll)? I've done it, many of you probably have too.

Well, I spent a few days recently trying to figure out how to do the same thing, but from a C# ("c-sharp") program running as "managed code" in .NET. I'm using Visual Studio 2005 as my IDE. In the end I cracked it, and it wasn't too difficult. The real secret was to find the magic incantation equivalent to "DECLARE" in LotusScript.

Background:
I inherited some C# code that uses the Notes COM interfaces to do some stuff with user mail files. The COM interfaces are pretty much the same as the LotusScript back-end classes, they work great in a .NET environment for accessing NSF data. As an enhancement to the existing code (which loops over all documents in a user mail NSF and does things to them), I wanted to find out which documents were unread for the mail file's owner. This involved two extra steps that I needed to research:

Figure out the name of the mailbox owner. NotesPeek came to my rescue (as it has so many times before), and I found what I needed. It's a Profile document called "calendarprofile" (just use NotesDatabase.GetProfileDOcument("calendarprofile", "") ), and the name is on an item named "Owner". So far, so good. It would be a good idea to make sure that the name you get is in "distinguished name" format (i.e., cn=xxx/o=yyy...)
Figure out if the curernt document is unread for the name we found in step 1. Unfortunately, there is no back-end class support for this functionality, you have to drop down to the C API. I knew how to do this in LotusScript, but not from .NET. Fortunately, some Google research found me a couple of articles that were very helpful (links at the end of this post). Knowing what C API calls to use, I was able to write a C# class to tell me whether a NOTEID is in the unread list or not.

It turns out that the magic incantation in C# is easy to do, once you know it. You have to tell CLR ("common language runtime") where the DLL is that you want to access, and what the signature (call name and parameter types) of the entry point you need is. Just like Declare statements in LotusScript (with different syntax, of course).

Here's an extract from my code showing the Notes entry points I needed:

/*
* STATUS LNPUBLIC NSFDbOpen(
const char far *PathName,
DBHANDLE far *rethDB);
*/
[DllImport("nnotes.dll")]
public static extern STATUS NSFDbOpen(String path, ref HANDLE phDB);

/*
* STATUS LNPUBLIC NSFDbClose(DBHANDLE hDB);
*/
[DllImport("nnotes.dll")]
public static extern STATUS NSFDbClose(HANDLE hDb);

/*
* STATUS LNPUBLIC NSFDbGetUnreadNoteTable2(
DBHANDLE hDB,
char far *UserName,
WORD UserNameLength,
BOOL fCreateIfNotAvailable,
BOOL fUpdateUnread,
HANDLE far *rethUnreadList);
*/
[DllImport("nnotes.dll")]
public static extern STATUS NSFDbGetUnreadNoteTable2(
HANDLE hDb, String user, ushort namelen, bool create, bool update, ref HANDLE hList);

/*
* STATUS LNPUBLIC NSFDbUpdateUnread(
DBHANDLE hDataDB,
HANDLE hUnreadList);
*/
[DllImport("nnotes.dll")]
public static extern STATUS NSFDbUpdateUnread(HANDLE hDb, HANDLE hUnreadList);

/*
* BOOL LNPUBLIC IDIsPresent(HANDLE hTable, DWORD id);
*/
[DllImport("nnotes.dll")]
public static extern bool IDIsPresent(HANDLE hTable, DWORD id);

/*
* STATUS LNPUBLIC OSMemFree(HANDLE Handle);
*/
[DllImport("nnotes.dll")]
public static extern STATUS OSMemFree(HANDLE h);

Note that every call is preceded by " [DllImport("nnotes.dll")]", that's a requirement (and the DLL must be on the system PATH). There's no "#include" directive in C# to bring in header files, so you have to map all the special Notes datatypes (like DBHANDLE, NOTEID, etc.) to native C# types. I did this with "using" statements, like this:

using HANDLE = System.UInt32;
using DWORD = System.UInt32;
using STATUS = System.UInt16;

After doing this, I just called the routines as if they were native C# code. CLR took care of mapping string buffers to "unmanaged" memory and so on. It worked great! Since this post is already too long, I won't post the class I wrote to use these CAPI calls, but it's very simple. There's an Init() routine that takes a server, db and username, and opens the database. With the username, it gets the IDTable representing the user's unread list (and updates it). Then, every time I get a new Document, I convert it's NOTEID string to a number, and see if that ID is in the unread list. On shutdown I free the IDList and close the Database (actually, I could almost certainly close the database after retrieving the IDList....)

Not bad! The articles I read said there was a way to invoke all kinds of entry points: using structs, passing callback functions, everything (someday I'm sure I'll need to do those things too, just didn't have to this time).

Here are the 2 links I promised:
http://msdn.microsoft.com/en-us/magazine/cc301501.aspx
http://msdn.microsoft.com/en-us/library/ms123402.aspx (I swear this one was there on Saturday, but today when I went to verify it, I got "content not found". Oh well, keep trying...)

Enjoy!

Do we need new kinds of replication?

Bob Balaban — Tue, 22 Apr 2008 07:55:00 -0400

Greetings, Geeks!
Today I wish to talk about a favorite topic of mine, Notes-style Replication. We all know about it and love it, naturally. It's one of the great foundational pillars of Notes and Domino, it's value to the product and to the applications built on the platform cannot be under-estimated.

BUT (I claim) it is time to start thinking about the need for an enhanced replication facility for Notes and especially for Domino.

The need (I claim) for a couple of new kinds of Notes/NSF replication has been around for a while, as I'll explain. However, I think the need will accelerate in the next 1 or 2 releases of Notes/Domino, if people (Developers, mainly) start really embracing some of the new features that have appeared in N/D v8.0, and which are likely to appear in v8.5.

What "new kinds" of replication am I talking about? Two, mainly:
1) Multi-NSF "packages"
2) Non-NSF files in the N/D data tree

Multi-NSF applications have been around for years. The very first consulting project I worked on after leaving Iris Associates in 1997 involved distributing data across as many as 7 Notes databases per dataset. The reason, of course, was that (at the time, in V4.6) the maximum allowed size of a single NSF was something like 2GB (it might have been 4GB, I can't remember specifically). And databases larger than 1GB led to pitifully poor performance. Aside from the difficulty of coding and app set up like that, the fact that a given installation required several instances of an NSF to replicate all at once meant severe headaches, both at application install time (setting up multiple replication instances across a few servers, and ensuring that all the app NSFs replicated at the same time), but it was a headache for the admins, too. They had to know either not to touch ANY of the NSFs in a given group, or to apply certain kinds of changes to ALL of them, at once.

The other type of "replication" that I mention above is also not terribly new. Ever since Domino became an HTTP platform (1996 or so, with V4.5), developers have been storing pieces of their app as HTML files in the "data tree" (domino\data\domino\html, and associated folders) on disk, instead of "inside" NSF files. My extensive research shows (Ok, I called a couple of people...) that there were (are) two reasons for this:
1) Simplicity of referencing with HTML links. Domino-generated URLs were then, and are now, rather complex to deal with. But if you put an HTML file in a certain place on the server disk, you always know how to reference it.
2) Performance. I am told by a reliable source (they guy at Lotus who actually maintains the HTTP server code) that it's 2 or 3 times faster to serve a file off of the server's disk than it is to serve the same file embedded somewhere in an NSF (on a Page, perhaps, or as a file resource). This is especially true because in more recent versions of Domino, the amount of caching that the HTTP server tries to do on disk has been scaled way back.

Ok, so you have a Domino Web application that relies on specific files on disk. Replicating the NSF (the major piece of the app) to another server is no big deal, but how do you install and/or keep non-NSF files synchronized? There are ways (run a scheduled agent that can access a mapped drive to another server and copy files, buy a third party utility that does it, write an agent to detect modified disk files, attach them to an NSF, replicate, then have another agent detach newer files to disk, etc. None of these solutions is particularly satisfying, especially because none of them (that I am aware) handles merges and conflicts very well.

So I think we'd all pretty much agree that this issue is not a new one. So why bring it up now?

I'm mentioning it now because I see the need for a solution becoming more important, not less. There are two sets of enhancements (one released in v8.0, one coming in v8.5) that will potentially cause this problem to become even worse, in the sense that more people will require it. The two new features are:
1) Composite Applications (released for Domino in v8.0)
2) Dojo JavaScript libraries shipped with the Domino Server (scheduled for v8.5, this year).

The pressures on multi-file replication are clear: CompApps involving Notes-based components pretty much require you to use multiple NSFs. The "Application" database itself has almost nothing in it, except some attached XML files that define the CompApp. There will be references in there to 1 (or possibly several) additional NSFs containing data and/or logic components that make up parts of the overall application. At application launch time, a Notes user double-clicks on the icon representing "the composite application" (the mostly empty one), and the right thing happens in the Notes Client - the multi-pane window opens up and the right components appear in the right places. BUT, it only works if ALL the referenced NSFs are where they should be.

I've had many experiences of trying to open a CompApp that was replicated to a server that was not the one used to originally install the app. If one database comprising part of the overall app is not replicated, or if it is replicated to the new server, but to a location different from its originally referenced one, stuff is broken.

Furthermore, go tell a Domino server administrator (even an experienced one) that you have a Composite Application on ServerA that needs to be moved (or copied) to ServerB, with replication enabled between the two instances. Go ahead, I dare ya! This exposes another, related problem with multi-NSF applications -- you can't easily figure out from looking at the "main" NSF in a Composite Application what OTHER NSFs are required to make the app work, unless you're willing to go read some XML files or poke around in Domino Designer, looking for database references. Yet, the whole group of them MUST be replicated as a unit, and must follow the original folder layout, neither of which is generally a requirement for "traditional" Notes replication.

The second new featureset, scheduled to appear in v8.5, is a terrific enhancement for developers working on Domino-based Web application development. Shipping a set of Dojo JavaScript libraries with the product (and exposing easy ways of hooking those libraries into your app via Domino Designer) is a huge step forward for Web apps on Domino. Dojo is a popular, browser based set of JavaScript code, distributed as open source. It provides some very nice features to the Web application developer, among them: browser isolation; sophisticated AJAX support; accessibility features; a full complement of UI widgets, from which a JavaScript developer can inherit and customize new widgets, and so on.

Now, for performance and other reasons (as mentioned above), the plan (which could, of course, change at any time) is to ship the Dojo libraries as a set of .JS files on disk. This will work great if your Dojo-enabled Web apps all run on newly-installed servers, all of the same release of Domino. But if not, then you may have a problem. Let's say (hypothetically) that Dojo version 1.2 ships with all Domino 8.5 servers, on all OS platforms. Then let's say that Domino 8.5.1 is released with Dojo v1.3, which has some nice bug fixes and/or new features in it. Now let's assume that I've deployed an 8.5 Web App using Dojo 1.2 on one of my new 8.5 servers. If I have a second server that gets 8.5.1 and I want to now deploy my app there, I could have a problem.

I now need to pick not only which version of the server I should QE my app for, I have two different versions of Dojo to deal with, too. Do I pick one or the other? If so, how do I get the 8.5.1 version of Dojo (hypothetically v1.3) over to my 8.5 server (which normally only has Dojo v1.2 on it)? PLUS, if I've deployed pieces of my JavaScript based client as .JS files to disk, how do I synchronize those between the two servers (or between any 2 servers, even if they are running identical versions of Domino and Dojo)?

My point is that, although I think Dojo is a terrific enhancement for Domino, using it will increase the level of deployment complexity and synchronization complexity for both developers and admins.

An enhancement to the way Domino does replication to include non-NSF files as well as to be able to treat a collection of related NSFs as a unit would be very, very helpful (Note: Of course I am not naive enough to expect that non-NSF file replication could have all the features of NSF-to-NSF replication, but that's a topic for another post...)

Guess what? It’s been a whole year already

Bob Balaban — Tue, 8 Apr 2008 19:50:02 -0400

Greetings, Geeks!

Naturally, I missed my own anniversary -- the first posting I made on this blog was March 30, 2007, just over a year ago. As my dad might say: "Hoo hah!" (if you're reading this and you're my father, Hi Dad!).

Pretty cool, if I say so myself, I really wasn't sure when I started whether I'd be able to keep it going this long. As it turned out, it wasn't as hard to come up with topics as I thought it would be. Here are some stats:

41 -- Approximate number of main postings in the blog (yes, I was too lazy to count...)
1100 -- Approximate number of comments posted
2 -- Number of guest blog postings (1 by Mark Vincenzes, one by Steve Nikopoulos)
between 3000 and 9500 -- the range of the number of page hits on the blog per month
166 - largest number of comments on any one post (this one, I wonder how many of these suggestions will make it into 8.5?)
a LOT - the amount of fun I've had doing this little project

Thanks to you, fellow Geeks, for making it worthwhile to keep this thing going. Keep posting!

Remember: "In theory" means "not really"!

Recursion or Iteration? YOU decide...

Bob Balaban — Fri, 4 Apr 2008 08:23:05 -0400

Greetings, Geeks!

Today our topic is....recursion. No, fellow geeks, this does NOT mean that we are going to practice cursing, and cursing again....

The thing is, recursion (loosely defined as the technique whereby a subroutine processes a piece of a data structure, and then invokes itself ("recursively") to process another piece. The classic example that we all learn in Computer Science 101 class is... "walking a tree". Note that this has very little, if anything to do with the technique we learn in real life, called "walking the dog" (and YES, I KNOW that that is also a yo-yo maneuver....sheesh).

There's even a pretty good example of a recursive tree-walker routine, written in LotusScript, in the Designer Help file (search or index to "Domparser class", go to that article, and follow the "Example" link). In this case, the code is reading in an XML file, parsing it into a tree using the standard XML DOM parser, and then travelling the tree to visit every node. Because the algorithm is recursive, it does a "depth-first" tree walk, meaning that it visits the first child of the top-level node, then the first child of that node, then the first child of that node, etc., until it reaches a node that has no children. Along the way, the "welktree" subroutine calls itself to "process" the next child node. In this way, it stores the "context" at each layer of the tree on the LS invocation stack. When it hits a node with no children, it "processes" that node, then backs up a layer, and continues to the next child of tht layer's parent node.

Yes, it sounds confusing, but if you step through the code in the debugger, you'll get it pretty quickly.

One of our developoers actually copied this code and used it in developing an enhancement to one of our products. It was at that point that he discovered a serious limitation of recursion as an algorithm, and thereby learned an important lesson. To wit: Just because they teach you a "standard" algorithm for data structure traversal in CS101, that doesn't automatically mean that you should actually USE it in code that you sell to people for money, and therefore have to support for real.

Why do I say this? Because, dear Geeks, it turned out that the whole thing blew up when the XML DOM tree got big; especially when it got "deep" (meaning, a lot of nested layers). Why? Because, unlike in Windows (and many other operating systems), the size of the calling stack in LotusScript is fixed. Yes, that is correct. The calling stack for a LotusScript program is pre-allocated at program start time, and it never changes. In the Windows C (and .NET) runtime, by contrast, you can specify a calling stack's initial size (there is one stack per thred in most OS processes), but if you run out, the OS (or more properly, the language runtime servics) will grow it automtically. But in LS, when you (for example) recurse too deeply at runtime, you get an "out of memory" exception. I'm not sure how big the LS pre-allocated stack is, but if I had to guess, I would guess 64KB.

So, copying that particular example didn't work out so well for us. My suggestion for fixing this problem was to process the nodes iteratively instead of recursively -- do them in a loop. If you traverse the tree "breadth first" instead of "depth first", you can just build a queue of nodes as you "walk" the tree, and at the same time, you can pull nodes off the front of the queue for "processing". We did indeed verify that for the same data inputs, this new technique did not run out of memory.

Here's the original example, copied from Designer Help:

Original version:
This agent parses the origXML file, walks the DOM node tree and creates a report in outputFile about the nodes encountered.

(Declarations) Dim domParser As NotesDOMParser Dim LF As String

Sub Initialize Dim session As NotesSession Dim db As NotesDatabase Dim inputStream As NotesStream, outputStream As NotesStream Dim docNode As NotesDOMDocumentNode Dim origXML As String, outputFile As String origXML = "c:\dxl\xmldom.xml" outputFile = "c:\dxl\DOMtree.txt" Dim header As String header = "Walk Tree agent" LF = Chr(13)+Chr(10) On Error Goto errh Set session = New NotesSession Set db = session.CurrentDatabase 'create the output file Set outputStream =session.CreateStream outputStream.Open (outputFile) outputStream.Truncate 'write report title outputStream.WriteText ("DOM Parser Report - " ) outputStream.WriteText (header+LF) 'open the XML file Set inputStream = session.CreateStream inputStream.Open (origXML) If inputStream.Bytes = 0 Then outputStream.WriteText (origXML+" is empty"+LF) Goto results End If 'create DOM parser and process Set domParser=session.CreateDOMParser(inputStream, outputStream) domParser.Process 'get the document node Set docNode = domParser.Document Call walkTree(docNode) results: Call outputStream.Close Exit Sub errh: outputStream.WriteText ("errh: "+Cstr(Err)+": "+Error+LF) Resume results End Sub

Sub walkTree ( node As notesdomnode) Dim child As notesdomnode Dim elt As notesdomnode Dim attrs As notesdomnamednodemap Dim a As notesdomattributenode Dim piNode As Notesdomprocessinginstructionnode LF = Chr(13)+Chr(10) If Not node.IsNull Then Select Case node.NodeType Case DOMNODETYPE_DOCUMENT_NODE: ' If it is a Document node domParser.Output( "Document node: "+node.Nodename ) Set child = node.FirstChild ' Get the first node Dim numChildNodes As Integer numChildNodes = node.NumberOfChildNodes domParser.Output(" has "+Cstr(numChildNodes)+" Child Nodes"+LF) While numChildNodes > 0 Set child = child.NextSibling ' Get next node numChildNodes = numChildNodes - 1 Call walkTree(child) Wend Case DOMNODETYPE_DOCUMENTTYPE_NODE: ' It is a tag domParser.Output("Document Type node: "+ node.NodeName+LF) Case DOMNODETYPE_TEXT_NODE: ' Plain text node domParser.Output("Text node: "+node.NodeValue+LF) Case DOMNODETYPE_ELEMENT_NODE: ' Most nodes are Elements domParser.Output("Element node: "+node.NodeName ) Set elt = node Dim numAttributes As Integer, numChildren As Integer numAttributes = elt.attributes.numberofentries domParser.Output(" has "+Cstr(numAttributes)+" Attributes"+LF) Set attrs = elt.Attributes ' Get attributes Dim i As Integer For i = 1 To numAttributes ' Loop through them Set a = attrs.GetItem(i) ' Print attr. name & value domParser.Output("Attribute "+a.NodeName+": "+a.NodeValue+LF) Next numChildren = elt.NumberOfChildNodes Set child = elt.FirstChild ' Get child While numChildren > 0 Call walkTree(child) Set child = child.NextSibling ' Get next child numChildren = numChildren - 1 Wend domParser.Output( elt.nodeName+LF) Case DOMNODETYPE_COMMENT_NODE: ' Comments domParser.Output("Comment node: "+node.NodeValue+LF) Case DOMNODETYPE_PROCESSINGINSTRUCTION_NODE: ' Handle PI nodes Set piNode = node domParser.Output("Processing Instruction node: " ) domParser.Output(" with Target "+piNode.Target+_ " and Data "+piNode.Data+LF) Case DOMNODETYPE_CDATASECTION_NODE: ' CDATA sections domParser.Output("CDATA Section node: "+node.NodeName) domParser.Output(" has value of "+node.NodeValue+LF) Case DOMNODETYPE_ENTITYREFERENCE_NODE: ' Handle entities domParser.Output("Entity Reference node: "+node.NodeName+LF) Case Else: domParser.Output("Ignoring node: "+Cstr(node.NodeType)+LF) End Select 'node.NodeType End If 'Not node.IsNull End Sub

------------------End of Example--------------------------

It's a perfectly fine example, it's just that there are a lot of real-world cases where it will crash (at least, in LotusScript it will).

Here's a new version of the example that I re-wrote to use a node queue. The startup code parses the input XML file, and gets the top-level ("document") node. It puts that node on a queue (I wrote a LS "Queue" class to manage the queue, it's in the Declarations section). Then, as long as there's something in the queue, we grab the first Node, and see if it has any children. If so, we do a loop and put each child at the end of the queue. You could modify the code to "process" (in this example "process" just means to write out some info about the current node) the node before you put it on the queue, or to do it when you take it off the front of the queue, it doesn't matter that much.

So, in sum, we're stuffing nodes onto the tail of the queue as we encounter them, and pulling them off the front for processing. We're done when the queue is empty. Not too hard, eh?

Enjoy!

New version:
'TreeWalker:

Option Public
Option Declare

Dim domParser As NotesDOMParser
Dim LF As String
Const INCREMENT = 5

Public Class Queue
Private q() As NotesDOMNode
Private first As Integer
Private last As Integer

Sub new()
first = 0
last = -1
Redim q(INCREMENT - 1) ' INCREMENT represents the count, not the ubound
End Sub

Public Sub Add(node As NotesDOMNode)
Dim origFirst As Integer
Dim i As Integer

origFirst = Me.first

' does the queue have space to just add?
If Me.last < Ubound(Me.q) Then
Me.last = Me.last + 1
Set Me.q(last) = node
Goto XEnd
End If

' if there's space at the head of the queue, shift everything over
' (this is cheaper than redim-ing the array)
If Me.first > 0 Then
i = 0
While Me.first <= Me.last
Set Me.q(i) = Me.q(Me.first)
Set Me.q(Me.first) = Nothing
Me.first = Me.first + 1
i = i + 1
Wend
Me.last = Me.last - origfirst
Me.first = 0

' now add the new node
Me.last = Me.last + 1
Set Me.q(last) = node
Goto XEnd
End If

' if we reach here, the array is full but we still need to add a node,
' so no choice but to expand the array
Redim Preserve Me.q(Ubound(Me.q) + INCREMENT )
Me.last = Me.last + 1
Set Me.q(last) = node
Goto XEnd

XEnd:
End Sub

Public Function FirstNode() As NotesDOMNode
If Me.first <= Me.last Then
Set FirstNode = Me.q(Me.first)
Set Me.q(Me.first) = Nothing
Else
Set FirstNode = Nothing
Exit Function
End If
& nbsp; Me.first = Me.first + 1
End Function
End Class
Sub Initialize
Dim session As NotesSession
Dim db As NotesDatabase
Dim inputStream As NotesStream, outputStream As NotesStream
Dim docNode As NotesDOMDocumentNode

Dim origXML As String, outputFile As String
origXML = "c:\temp\xmldom.xml"
outputFile = "c:\temp\DOMtree.txt"

Dim header As String
header = "Walk Tree agent"
LF = Chr(13)+Chr(10)

On Error Goto errh

Set session = New NotesSession
Set db = session.CurrentDatabase

'create the output file
Set outputStream =session.CreateStream
outputStream.Open (outputFile)
outputStream.Truncate

'write report title
outputStream.WriteText ("DOM Parser Report - " )
outputStream.WriteText (header+LF)

'open the XML file
Set inputStream = session.CreateStream
inputStream.Open (origXML)
If inputStream.Bytes = 0 Then
outputStream.WriteText (origXML+" is empty"+LF)
Goto results
End If

'create DOM parser and process
Set domParser=session.CreateDOMParser(inputStream, outputStream)
domParser.Process

'get the document node
Set docNode = domParser.Document

' Call walkTree(docNode)
Dim q As New Queue()
q.Add docNode ' get the first node in there
Process q

results:
Call outputStream.Close
Exit Sub
errh:
outputStream.WriteText ("errh: "+Cstr(Err)+": "+Error+LF)
Resume results
End Sub

Sub walkTree ( node As notesdomnode)
Dim child As notesdomnode
Dim elt As notesdomnode
Dim attrs As notesdomnamednodemap
Dim a As notesdomattributenode
Dim piNode As Notesdomprocessinginstructionnode
LF = Chr(13)+Chr(10)

If Not node.IsNull Then
Select Case node.NodeType
Case DOMNODETYPE_DOCUMENT_NODE: ' If it is a Document node
domParser.Output( "Document node: "+node.Nodename )
Set child = node.FirstChild ' Get the first node
Dim numChildNodes As Integer
numChildNodes = node.NumberOfChildNodes
domParser.Output(" has "+Cstr(numChildNodes)+" Child Nodes"+LF)

While numChildNodes > 0
Set child = child.NextSibling ' Get next node
numChildNodes = numChildNodes - 1
Call walkTree(child)
Wend

Case DOMNODETYPE_DOCUMENTTYPE_NODE: ' It is a tag
domParser.Output("Document Type node: "+ node.NodeName+LF)

Case DOMNODETYPE_TEXT_NODE: ' Plain text node
domParser.Output("Text node: "+node.NodeValue+LF)

Case DOMNODETYPE_ELEMENT_NODE: ' Most nodes are Elements
domParser.Output("Element node: "+node.NodeName )
Set elt = node

Dim numAttributes As Integer, numChildren As Integer
numAttributes = elt.attributes.numberofentries
domParser.Output(" has "+Cstr(numAttributes)+" Attributes"+LF)

Set attrs = elt.Attributes ' Get attributes

Dim i As Integer
For i = 1 To numAttributes ' Loop through them
Set a = attrs.GetItem(i)
domParser.Output("Attribute "+a.NodeName+": "+a.NodeValue+LF)
Next

numChildren = elt.NumberOfChildNodes
Set child = elt.FirstChild ' Get child
While numChildren > 0
Call walkTree(child)
Set child = child.NextSibling ' Get next child
numChildren = numChildren - 1
Wend
domParser.Output( elt.nodeName+LF)

Case DOMNODETYPE_COMMENT_NODE: ' Comments
domParser.Output("Comment node: "+node.NodeValue+LF)

Case DOMNODETYPE_PROCESSINGINSTRUCTION_NODE: ' Handle PI nodes
Set piNode = node
domParser.Output("Processing Instruction node: " )
domParser.Output(" with Target "+piNode.Target+_
" and Data "+piNode.Data+LF)

Case DOMNODETYPE_CDATASECTION_NODE: ' CDATA sections
domParser.Output("CDATA Section node: "+node.NodeName)
domParser.Output(" has value of "+node.NodeValue+LF)

Case DOMNODETYPE_ENTITYREFERENCE_NODE: ' Handle entities
domParser.Output("Entity Reference node: "+node.NodeName+LF)

Case Else:
domParser.Output("Ignoring node: "+Cstr(node.NodeType)+LF)

End Select 'node.NodeType
End If 'Not node.IsNull
End Sub
Sub Process(q As Queue)
Dim node As NotesDOMNode
Dim count As Integer
Dim child As NotesDOMNode
Dim i As Integer

Set node = q.FirstNode
While Not (node Is Nothing)
count = node.NumberOfChildNodes
If count > 0 Then
For i = 0 To count-1
If i = 0 Then
Set child = node.FirstChild
Else
Set child = child.NextSibling
End If

Speaking at The View Admin/Developer 2008 conference, Boston

Bob Balaban — Mon, 31 Mar 2008 07:10:01 -0400

Greetings, Geeks!

Hope to see you at this annual Boston event (April 30 - May 2). Here are the 2 sessions I will be presenting:

Achieving "Peaceful" Coexistence Between Microsoft Exchange and Lotus Notes
While many organizations strive to standardize their messaging infrastructures on a single platform, many others do not. Whether through merger, acquisition or simple user preference, many organizations face a need to achieve peaceful coexistence in the messaging wars, perhaps for only a short time (while they transition from one to the other), but perhaps (if they can pull it off) for a very long time. The technical challenges involved in maintaining a well-performing dual messaging infrastructure are not trivial, but they can be met successfully with proper planning and the right technology. The big issues are all around directory synchronization, message transport, rich text, scheduling logic differences and data formats. Come to this session to learn how to overcome these technical challenges how to keep both Notes and Exchange up and running, and how to avoid mutually assured destruction!

Can Domino Applications Be "Web 2.0"-Compliant?
Lotus Domino-based Web applications have been around for well over 10 years. The upcoming release of version 8.5 promises lots of new tools and functionality to make it vastly easier to create great-looking and well-performing Web apps. But will these new style apps and the tools you use to build them be architecturally compliant with "Web 2.0"? Come to this session to hear what "Web 2.0" really means for Domino Web application development, to hear how you can keep your Domino apps "standards compliant", and perhaps to learn some new Best Practices for designing and creating Web applications.

Show ’n’ Tell Thursday - what folders is a document in?

Bob Balaban — Mon, 24 Mar 2008 15:43:04 -0400

Greetings, Geeks!

I'm very late to the SNTT club, but I'm hoping to be able to throw in some decent techie tips at least semi-regularly, in addition to the more high-falutin' thought-pieces and questionnaires.

So! Have you ever needed to figure out what folders a document is in? Here's a brief synopsis of the documented way to do it, and then I'll show you a neat way I discovered that works better and faster.

You can go to the Designer Help database, select the Index tab, and find the "FolderReferences property" entry. You'll see in the doc that this is a property on NotesDocument (works in Java, too), but there are some constraints: the db has to have the "FolerReferencesEnabled" property turned on. But, there's no UI to turn it on (no InfoBox checkbox or anything), you have to do it from a program. And the db has to include 2 hidden views, which you can copy from a recent mail template: $FolderInfo and $FolderRefInfo. Eh, not great, but (so far) not so bad. Then you should go look at Technote number 1092899, which gives a bit more information on how folder references work.

It turns out there are a couple of limitations. For one thing, references to folders containing a document are only accumulated once you turn the database-level feature on, it doesn't go back and figure out anything about documents that were put into folders before the feature was enabled. And, having documents in privatefolders causes problems (like, doesn't work....). And they warn you that turning on this feature can slow down "database operations".

So, for the project I'm working on, as I did this research it sure started to sound like "not the best way to peel this onion" (meaning, I started to get the idea that if I went down this road, it was going to mean a lot of tears for me).

Luckily I came up with another way to do it that seems to work pretty well, and wasn't too hard to code up. I'll describe the algorithm, then show you some code snippets.

Algorithm:
1. Get a list of all the folders in the database (scan the array of View objects returned by db.Views, save the actual folders (view.IsFolder) in an array
2. Figure out what documents I want to test for folder-ization (this will vary depending on the application, but that's not important)
3. For each document in whatever list (or DocumentCollection, or whatever):
4. Iterate over each folder object (which is really an instance of the View class), and for each one:
5. Create a ViewNavigator object, using the View.CreateViewNavFrom()
6. Call ViewNav.Count. If the result is 0, the document is NOT in the folder, otherwise, it is

Cool, huh? I must say, I was very pleased to have figured this out. Why does it work? The trick is in using the ViewNavigator class. When you call any of the CreateViewNavXXX methods on the View class, you will ALWAYS get a non-null result, it always creates the object. BUT, if you use CreateViewNavFrom(document), the navigator class has nothing to do if the document is not in the view/folder. So Count will be 0. It's really nice that the View and ViewNavigator classes were coded to be tolerant of what is essentially an error condition! You're asking for a navigator object where the first position in the navigator's "list" is the document you specify. There is a call in the CAPI that this logic uses (NIFLocateNote(), you can look it up) which, given a document id, finds that document in a view or folder collection.

Code:
This code is written in C#, because the project I happen to be working on requires a .NET implementation. However, this would work just as well if it were in LotusScript or java.

// "allFolders" is a previously-populated array of NotesView objects representing all the folders in a db

string f = null; // list of folder names, delimited by ";"
for (int i = 0; i < allFolders.Count; i++)
{
NotesView v = (NotesView)allFolders[i];
try
{
NotesViewNavigator nav = v.CreateViewNavFrom(doc, 0);
if (nav != null && nav.Count > 0)
f += (f == null) ? v.Name : "; " + v.Name;
}
catch (Exception e) {} // guess it didn't work, keep going
}

Simple, huh? Note that this is a doubly-nested loop: outer loop over documents, inner loop over array of folders. If you can keep the numbers reasonable, this isn't too slow.

Enjoy!
lol, g2g, ttyl

Integration vs. Migration (an oldie, but a goodie)

Bob Balaban — Sun, 23 Mar 2008 09:16:19 -0400

Greetings, Geeks!

Given the time of year, I was feeling a little bit lazy, so I decided to resurrect a couple of postings I wrote a few years ago on a late, lamented "group blog" called "3C-Interop". It used to be at http://www.3cinterop.com, and the contributors were industry experts such as Karen, Rocky, Peter, and Amy. The idea was to focus on topics related to technology inter-operability, and some good content resulted. The blog kind of dried up after Amy got a job at Microsoft, Rocky and I took jobs at IBM, and Karen went to work at the Burton Group. Of course, you can still find cached stuff on Google....

Anyhow, since Binary Tree (my new employer) is big in this area, I decided to go back and review my earlier thinking on the topic of migration vs. integration. What appears below is the original text of what was a 2-part posting in 3C-Interop, unmodified. Of course, if I were to write this now, 3 1/2 years later, it would be a little different. But perhaps not much different. In fact, I think I'll let y'all absorb this for a few days, then post my own critique. The world (this part of it, anyway) has, after all, evolved in the intervening time.

Here goes:

Integration vs. Migration (December, 2004)

"Interoperability"
I view "interoperability" as the capacity of pieces of an application or infrastructure's architecture to talk to other pieces, regardless of whether those pieces are located on the same machine or not, implemented in the same language or not, running on the same operating system or not. This implies quite a wide range of possibility in terms of implementation and communication technologies, but that's what you want when you architect an application -- the ability to move big boxes of functionality around until the arrangement makes sense. "I'll put the UI over here, the database over there, and the business logic will be a combination of this kind of component and a few of those kinds over there." Knowing that the big boxes can talk to each other when and how you need them to makes it possible to architect the solution with full (or at least solid) awareness of the performance, implementation, security and maintenance implications.

As a slight digression, this is exactly why Web Services shows a lot of promise in the appdev world -- it's a technology that uses basic transport and data encoding models to let application "boxes" talk to each other. Of course, it's not completely baked yet.

Another characteristic of interoperability is that (if you do it right) you're planning it from the beginning. It's one of those things that seems obvious when you start to architect or design an application, but which, if you don't plan for it up front, can be very difficult to add on later. Taking such buzzwords as "looseley coupled" and "services oriented" into account early on tend to increase the ease of interoperability. You buy yourself flexibility later if, for example, you want to replace the UI, or the RDBMS, or the middle-tier business logic platform. Which, of course, sounds a lot like "migration".

"Migration"
I see migration as a different kind of beast. Migration happens when you take a working piece of your architecture and re-host it elsewhere (a different OS, a different "platform" or "framework"). This is something you do AFTER the app is up and running, typically. It generally involves some kind of "porting" or re-coding, either because the OS is different, or because the services on which the functionality rests are different, or differently implemented (e.g., porting a J2EE servlet coded in Java to a C# ASP in .NET), even if the functionality is supposed to be the same.

So, is migration fundamentally different from building for interoperability? In a sense no, because architecting for interoperability up front makes any migrations you do later a lot cheaper, at a component level. From a big-picture, full-application point of view, however, I think it's true that architecting for interoperability ultimately means that you will have LESS need to migrate, or at least, IF you have to migrate, it won't be the entire app. And that's a good thing.

Part 3 of this little series [editor's note: see below] will address some additional points on this topic: Why do people migrate apps? When should you migrate? When should you not migrate? If you're going to migrate, how should you do it? And last, but not least, is "migrate" etymologically related to "migraine"?

After some briefings at IBM a few weeks ago and at Microsoft this past week, [editor's note: remember, this was written in 2004] it seems that it's no longer good enough to sell developers and their management on your platform for doing new development. Now you have to have a strategy and a marketing plan to "convert" organizations from the competition's stuff to your own, including any already-existing apps. I don't know, maybe it was always that way, and maybe it has to be that way in a relatively mature market.

So both behemoths are focusing a lot of energy on "migration", and both have their sights set on "harvesting" the large Domino installed base. Godzilla vs. Mothra? Don't get me started. IBM wants Domino apps to migrate to Workplace [editor's note: heh heh] , and Microsoft wants them to migrate to .NET and whatever Windows server platform is current (win2003 today).

First off, let me say that these efforts by both companies, while understandable, are misguided. Neither Lotus Workplace nor .NET is nearly at par functionally with Domino (and Notes) in the collaborative space -- yet. Both are working hard to get there, of course, but until they can offer true functional replacements, why should Domino guys and gals even consider throwing out perfectly good, operational Notes/Domino apps and starting over? They shouldn't (IMHO).

So, back to the point in my 2 prior postings on this topic in this blog: Integration is what you do to solve particular problems you may be having with existing apps; migration is what you do when you want to port your app to another framework or platform. Often it is true that what begins as integration (moving a piece of an existing, working app somewhere else) ends as migration (all the pieces end up getting moved). But not always.

And, just like JIT, migration happens anyway. Why do people migrate things? I can think of reasons (some valid, some not) which apply at different levels in an organization:

Strategic reasons: "My company is moving to .NET"
Political reasons: "The new CIO hates Domino"
Tactical reasons: "We have more Java developers than C#"
Personal reasons: "I want C# on my resume"

The issue question for solution architects and developers is: given that you're going to do some of this, what's the best way to get it done so as to a) minimize cost and b) minimize downtime of the app while making changes.

How to turn "migration" into "integration"
If you look at the problem as "Which piece to Migrate first?", you're back to where you should be -- thinking about how to "integrate" the app. The choices tend to be:

UI
Business logic
Other content management/workflow logic
Data store
Web Service(s)

Each has its own problems, solutions and trade-offs, and there is no one-size-fits-all solution, technique or product. Different platforms/frameworks have different strengths and weaknesses. So you pick the low-hanging fruit first, go for maximum impact with minimum time/effort, and demonstrate an early success. Then you decide whether you need to keep going with the next hunk of app, and the next, and so on.

What % of the time are YOU online?

Bob Balaban — Mon, 10 Mar 2008 20:30:48 -0400

Greetings, Geeks!

To quote one of my all-time favorite TV characters, Roseanne Roseannadanna (ok, so I'm a baby-boomer, so sue me), "What's all this I hear about discombobulated obligations?" Or, in plain English, "disconnected applications". Today I took the Amtrak Acela (the "fast" train, or, as one of my European friends calls it, the "faster" train) from Newark, New Jersey (home of the corporate hq of my new company, Binary Tree) back home to Boston. It's a 4 hour ride, give or take, and I got bored reading "Office 2007 For Dummies", so I decided ot go online to check my email.

Well! It turns out that the fancy, shmancy "Fast" train from Newark (New Jersiey) to Boston (Massachusetts), does NOT have wireless Internet connectivity!! I thought I was going to DIE!! I said to the conductor guy, "HEY! Mister CONDUCTOR guy on the fancy shmancy "fast" train from Newark New Jersity to Boston Massachusetts!! How come there's no wireless Internet connectivity here?? Huh? Is this the fancy shmancy FAST Amtrak train, or what??"

He said, "HEY! Mr. Passenger guy. You sound just like Roseanne Roseannadanna from that old Baby Boomer TV show! NO, we do NOT have wireless Internet connectivity on this train! Unless, of course, you purchased yourself a fancy shmancy telephone company broadband wireless connection card! Did you do that, Mr. I-want-to-be-connected-everywhere, Mr. "Baby Boomer" TV-watching guy? Did you? Huh?"

Ok, enough, before I get sued by somebody. Here's the question -- does anyone still care in a major way about disconnected apps? It's been a huge differentiating/selling point of Notes for decades, and I use it all the time. I love having a local replica of my email file, so that I can still read and write email when I can't get an Internet connection for one reason or another. I also make use of the local-replica feature when I can get a connection, but it's slow. I let my dbs replicate in the background, and get decent performance using a local copy in the meantime.Admittedly, I use it a lot less than I used to. Just 2 or 3 years ago, wi-fi was way less ubiquitous, and in airports and hotels I frequently had to rely on slow dial-up connections. But even now, I find that on airplanes (and yes, even on fancy-shmancy trains), I'm forced to be offline. In these situations Notes' replication/disconnected support is a real win.

But now I'm wondering a) how big a percentage of most people's time is this case? and b) is it stabilized, or shrinking?

I've noticed over the last few years that the Microsoft suite of Web and collaborative applications pretty much assumes all-connected-all-the-time. And they have some valid rationales for making that trade-off -- it's certainly easier to create the software baking in that assumption, and it's certainly true that more people are actually connected a greater percentage of the time.

So, weigh in for me, fellow Geeks! Is "offline" support still important? Is it of growing, steady, or shinking importance, In Your Humble Opinions? Will anyone still care in 3 years? 5?

Goodnight Geeks! (and goodnight, little Roseanne Roseannadanna!)

And just when you thought you had division down....

Bob Balaban — Mon, 3 Mar 2008 14:45:58 -0400

Greetings, Geeks!

Everyone who knows me knows that I'm a big fan of LotusScript (it's not dead yet!). I've been writing code using the language for about 15 years now, and I was pretty sure that I'd explored most of the sometimes strange nooks and crannies of its functionality (with the possible exception of "Like", which I have yet to find a real use for....but that's another article).

Until this morning, that is, when I tripped over a subtle issue with, of all things, arithmetic division.

I won't go into all the details of my app, they're not relevant. Let's just say I wanted to know the maximum number of 2s in some number, call it N. If N did not have an even number of 2s in it (i.e., if N is not a multiple of 2), then I wanted the nearest possible integer result -- that is, I wanted to do a division and have the result truncated if there was a fractional value.

So, for example, if N is 6, the result should be 3. If N is 7, the result should be 3, if N is 8, the result should be 4. Simple, right?? You just divide N by 2 (you know both are integers when you start), and assign the result to another integer variable, which should force it to truncate any remainder. Here's the basic code:

Dim two as Integer, N as Integer, result as Integer
two = 2
'....obtain N from somewhere
result = N / two

Easy, right? Simple, right? WRONGWRONGWRONG!!

Rather surprisingly, I noticed I was getting the wrong answer sometimes! When N was 4, "result" was 2 (well, it better be!). But when N was 5, "result" was 3!

So I muttered to myself, "Self! someone or something evil is rounding on me! But the variables are all integers! What gives?" I went to the Designer Help, which contains surprisingly good and thorough documentation on the LotusScript language. I noticed that "division" is defined as always returning a floating point number!! And, unlike in C and other languages, when LotusScript converts a float to an integer, it automatically ROUNDS the number (C and C++ truncate).

As an interesting (to me, anyway) aside, this has always been confusing for people, there's nothing in many (or even most) programming languages that makes it obvious whether float-to-int assignment will either truncate or round. Perhaps that's why in Pascal if you code a direct assignment, the compiler will complain. You have to specify either the Trunc() or Round() function explicitly in that case.

All very interesting, but I needed a way to force the assignment in LS to give me a truncated integer result. So I thought to myself, "Self! There must be another operator for integer division!" I know I'd never heard of trunc/round functions in LS before, and I sure didn't want to write them. I hadn't ever heard of there being 2 division operators before either, but given the observed behavior of "/" (the floating point division operator), I assumed there would be another one for integer division. I knew about the Cint() function, which converts any data type to integer, but I already knew that that would also round a floating point value.

Back to Designer Help. Found it! It's the backslash: "\". Thank goodness "\" isn't used as an escape character in LotusScript, as it is in C/C++!

I replaced "/" with "\" in my code, and all was well. Go figure!

What Goes Around, Comes Around (or, should that be "WHO goes around..."?)

Bob Balaban — Sun, 17 Feb 2008 09:55:16 -0400

Greetings, Geeks!

Yes, I have been away from blog-land for a few weeks, I was disappeared into the Lotusphere/QuitMyJob/GotANewJob parallel universe there for a while. And then, of course, along with the new job came a new laptop, which (as usual) took 3 or 4 days to get fully set up and functioning properly (and we're still not completely there yet, for some reason the new SonicWall VPN thingie just won't do its job...)

So, here I am once again, on the outside of the IBM firewall, back in Business-Partner-Land. Coming up to speed on my new company's products and services. It turns out there are a lot of those, so it may take a little time. I was given a terrific suggestion on how to expedite that -- join in as a lurker on any and all sales presentations/demos being given to our existing or prospective customers on the Net.

In the meantime, it looks as though I will be immersing myself in a couple of new (to me, but not new to many of you, I suspect) technical areas. One that I'm particularly interested in so far is the particular problems faced by organizations trying to run BOTH Notes/Domino and Exchange messaging infrastructures. I'll call it "Coexistence" (peaceful or otherwise). There are a bunch of them out there, though I have no idea how common it really is. Even though I'm no Admin expert (yet, anyway), I can easily see how maintaining/synchronizing/integrating two directory systems, two (or more) mail clients, and two calendaring systems could be a real nightmare.

[full disclosure]
Naturally, Binary Tree (my new employer) have some products in this space, and my "interest" is therefore not entirely academic, nor will my opinions on this topic be entirely objective (for one thing, I can't imagine why anyone would keep Exchange around when they already have Notes/Domino....). But since I am not a sales/marketing guy, my interest and focus in this blog will remain almost entirely technical. And there are PLENTY of interesting technical issues needing resolution in this space.
[/full disclosure]

So, in the sprit of me-learning-from-you with which I started this blog in the first place, I would love to know:

- How common is a dual IBM/Microsoft messaging infrastructure?
- How often do you run into other enterprise messaging systems in terms of integration/coexistence needs? (I hear Groupwise still have a few million seats...)
- Is it really the nightmare to administer that I imagine it is? Or have you figured it out?
-What specially difficult technical issues do those of you who deal with this situation for a living encounter?
(One example that I've heard mentioned: Sending Notes email to Exchange, when the email contains an attached form and/or an LS button or doclink in the email body...)

Another, related-but-less-technical question I have is:

When you have encountered this 2-headed situation, has it been "stable"? By that I mean, does the organization intend things to stay that way over time? I can imagine that in many cases, there are 2 messaging systems in place, but the situation is temporary, because the organization is actually migrating from one to the other.

I'd love to know how often this is the case (i.e., there are 2 but 1 is on the way out), versus how often it's the case because of historical accident (acquisition/merger, or whatever), and the organization intends to leave it that way.

Let's hear that feedback!

Obscure (Rob says that works as a title)! In Which A Certain Food Fight is Explained

Bob Balaban — Tue, 29 Jan 2008 21:35:53 -0400

Greetings, Geeks!!

Well, like many of you, I'm recently back from Lotusphere08, relatively rested, and with the usual new collection of t-shirts and schwag. Hey, did any of you like the buttons I handed out? I got lots of compliments from Lotus management (one VP asked why I hadn't done it as an official project so I could expense it. The answer -- because then it wouldn't have been as good or as much fun. It was worth having to pay for the whole thing to be able to do it just the way I wanted).

Unlike many of you (I assume, anyway) I came back from Lotusphere with a new job. That's right, as of January 29, I have resigned my position at IBM. My new job is Vice President of R & D at Binary Tree. I'm real excited about jumping in to a great company that I'm already familiar with (they were one of my first customers back in 1997/98 when I left Iris the first time, and started Looseleaf Software). My last day as an IBM employee will be Friday, Feb. 8

I feel totally comfortable about the change, I'm happy that I'll still be part of the Lotus community, and I'm really looking forward to throwing my weight around as an executive, and getting to tell people what to do (ok, kidding about that part).

I intend to keep this blog running, though of course the focus will change a little.

So, many of you will be wondering what's happening at IBM, who's running the AppDev stuff now, whether the LSX Toolkit will still ship, and who's going to ask the first question at Ask the Developers at Lotusphere09. Not being omniscient, I cannot provide reliable answers to all of these questions, but here's my best shot: at a little FAQ:

Q: Who is taking over your AppDev Architect position at Lotus?
A: I don't know yet, but if/when I find out I'll post an announcement (if I am given permission to do so, of course)

Q: Does this mean that the LSX Toolkit is not going to ship after all?
A: I sure hope it doesn't mean that. The thing is really very, very close to being ready to go. All it needs is a final legal clearance to go to interested Design Partners, and some QE. As I said at Lotusphere, I really do expect it to be generally available by the time Notes/Domino 8.5 ships, or possibly even sooner.

Q: So who will be asking the first question at Meet the Developers/Lotusphere09?
A: With any luck at all, it will be me! Binary Tree almost always participates in the Lotusphere showcase (and in fact BT was a Lotusphere sponsor this year), and I intend to submit some abstracts, and hope to be selected as a speaker.

Q: If you get to ask the first question, what will it be?
A: That would be telling! I have some ideas, but I think it's way too early to say.

Q: What the heck is the story with that "food fight" over on Rocky's blog?
A: Ah. Yes, well. Ok, time to come clean: The whole thing was a gag that Rocky and I cooked up (some other people were in on it too, but it was essentially my idea). We thought it would be funny to replay the post on Volker's blog that started a whole foofurraw, except this time about me (though nobody was to know it was me). It was all fake, including some of the stronger comments posted by me and some others. I have to admit that, for me, the result wasn't as funny as I'd hoped it would be, though it was amusing. But, it went on long enough (all day yesterday and into the night), and now it's time that it stopped. I suspect there are lessons to be learned by all from this. If anyone was hurt or offended, I deeply apologize.

Q: So, Rocky really isn't a jerk?
A: No! Of course not! Rocky is a totally forthright, honest, stand-up sort of guy, as anyone who has ever met him can immediately tell. I am grateful to have had his friendship lo these many years, and look forward to continuing it for many years to come (even though in our new jobs we are technically, sort-of competitors). If he has any major flaws at all, it is that he shares (with me) a rather strange sense of humor that expresses itself (at times) in odd ways. AND, for the record, he is VERY good at keeping secrets.

Q: What's your new contact info?
A: I will post my official Binary Tree email and such here when it's all set up. Otherwise, if necessary, I can be reached after 2/8 via email, bbalaban AT gee em a eye ell DOT com.

For those attending Lotusphere

Bob Balaban — Fri, 18 Jan 2008 11:04:07 -0400

Greetings, Geeks!

I'm taking off for Lotusphere08 in a couple of hours, and I wanted to alert those of you who will also be there to a few things:

1) I have schwag! If you are a Lotusphere first-timer, or have attended 10 or more Lotuspheres (including '08 of course), I have something for YOU! (while supplies last)

2) I will be presenting one session (AD302) showing off some new stuff coming for Web App Development in Domino/Next (Tuesday, 11:15am, Swan 5-6)

3) I will be roaming the showcase floor as much as possible, but.....

4) I will also be spending a bunch of time in the "Meet the Developers" Lab (usually right off the rotunda lobby in the Dolphin). Not sure what my duty schedule there is yet, but if I'm not there when you show up, ask for my schedule at the door.

So, look for me here, there and around, or just wave and say hi as we pass in the hallways (no tuxedo this year, just the regular IBM staff shirt or the traditional Hawaiian shirt).

ciao!