"32K limit exceeded", huh?? Why SUMMARY is not always your friend

Bob Balaban  October 20 2010 04:41:37 PM

Greetings, Geeks!

So there I was (don't you love stories that start out that way?). I was writing an agent to take journaled email messages, expand group names in the various recipient lists (SendTo, CopyTo, etc.), and then save a copy of the message with expanded lists in another database.

The group expansion issue was a little tricky, because Notes lets you nest group names (an email or security group can include the name of another group). That was fun to solve, maybe I'll post that code sometime. BUT, in testing, it turned out that if an expanded recipient item got too big, Notes would throw an exception (I was using Java) on the Document.save() call, "32K field limit exceeded". What? B-b-b-but I SPECIFICALLY coded the agent to watch how long the field is getting....

Ok, background: The limits I always knew about were pretty straightforward: No single data item, of any type, can be bigger than 64KB (and that includes overhead for type codes and stuff). The CAPI documentation says you should really limit the size to 40KB, plus or minus. "SUMMARY" items (those which can appear in a View) have to be 15KB or less in size. Rich text items can be any size, but that's because when a single item containing rich text data gets too big (more than 40K or so), Notes just creates another item of the same name and tacks it onto the chain of data items that make up the document.

My group expansion processing yields a Vector containing a String object for each name in the recipient field. Vectors can be any size, no problem). In order to obey the Notes limits, though, I chose to create a "recipient list" Vector, and add name to it from the original Vector, counting the length of each String as I add it, until I hit my 40K (or so) limit.^** Then I can just take the new Vector and hand it to Document.replaceItemValue(). If I fill up the original field name (this is in the copy, of course, not in the original journal message), I start a cascade: SendTo_1, SendTo_2, etc., until I've used up all the names in the expanded list. No single item has more than my pre-defined maximum of 40K characters (String.length returns number of characters, remember, NOT number of bytes! Pay attention, this might be important later).

So, why, after doing all this work to keep the translated items short enough for Notes, do I get the "32K limit exceeded"?? At first I thought it was because i was counting characters in Java, not bytes. Each character in a Java String object (which is normally encoded as UTF-16) actually consumes 16 bits, or TWO bytes, not one. So a 40Kchar string is actually using up 80KBytes! But.....no. Because when I pass the String objects to Notes, Notes automatically translates each one to LMBCS (the internal Notes character set, see my character set tutorial here for a full explanation), which is mostly 1 byte per character (not for Kanji and other multi-byte character sets, but in this case that's not what I was dealing with).

Ok, so....whatever. I dialled the max item size down to 30K and ran it again. Same error. Crap.

So then I did what I often do when I'm really, really stuck. I called my friend Daniel. We brainstormed. To make a longish story short-ish, he made a suggestion which, although it sounded a little weird, actually WORKED (he's very good at that).

The key was: Turn OFF the SUMMARY flag on the new items. Here's an approximation of how I did that in my Java code?

Vector expanded;     // the expanded list of names. We get that in other code I'm not showing you today. List of String objects
Vector newField;       // the list for a single document Item we're going to build, and track the length of
int newLength;
String nextOutputItem;   // name of next item to write to the document

// skipping details of outer loop over recipient fields....
// do a loop over names for the next output item
// (skipping tedious logic about cascading output items when they get too big...boring)
newLength = 0;
nextOutputItem = "";
for (int i = 0; i < expanded.size(); i++)
    {
    String aName = (String) expanded.elementAt(i);
    if (newLength >= MAX_LEN_WE_MADE_UP_SOMEWHERE)
         {
         // current newField is full, write it out
         // MUST turn off the SUMMARY flag before doc save
         Item newItem = doc.replaceItemValue(nextOutputItem, newField);
         newItem.setSummary(false);
         nextOutputItem = "";
         }
    else {
         newLength += aName.length();
         newField.append(aName);
         }
    } // end for i

Ok, I hope you get the basic idea. replaceItemValue() returns the Item object (convenient), so just use it to turn off the SUMMARY flag. Then when you save the document, you don't get that nasty 32K error.

BUT, the question remains: WHY does this work? We haven't reduced the size of the items we're writing. And if the limit for SUMMARY items is 32K, why did we still get the error when we starting limiting the items to 30KB??

I'm afraid, my friends, that the answer has to be: The error message is wrong. Bogus. Full of it. Misleading, at best. There IS NO 32K LIMIT!! 14K (or so) limit, maybe. THAT's why turning off the SUMMARY flag works, we're telling Notes that the item content is allowed to go all the way to 64KB (or so).

So, there you have it. Go forth and code, with no pesky, wrong error messages to befuddle and comfusticate your day.

Geek ya later!

** "Got the Vector, Victor?" (Lord, I just love the movie "Airplane"...)

(Need expert application development architecture/coding help?  Want me to help you invent directory services based on RDBMS?? Contact me at: bbalaban, gmail.com)
Follow me on Twitter @LooseleafLLC
This article ©Copyright 2010 by Looseleaf Software LLC, all rights reserved. You may link to this page, but may not copy without prior approval.

• Comments [0]