Bob Balaban's Blog

     
    alt

    Bob Balaban

     

    "32K limit exceeded", huh?? Why SUMMARY is not always your friend

    Bob Balaban  October 20 2010 04:41:37 PM
    Greetings, Geeks!

    So there I was (don't you love stories that start out that way?). I was writing an agent to take journaled email messages, expand group names in the various recipient lists (SendTo, CopyTo, etc.), and then save a copy of the message with expanded lists in another database.

    The group expansion issue was a little tricky, because Notes lets you nest group names (an email or security group can include the name of another group). That was fun to solve, maybe I'll post that code sometime. BUT, in testing, it turned out that if an expanded recipient item got too big, Notes would throw an exception (I was using Java) on the Document.save() call, "32K field limit exceeded". What? B-b-b-but I SPECIFICALLY coded the agent to watch how long the field is getting....

    Ok, background: The limits I always knew about were pretty straightforward: No single data item, of any type, can be bigger than 64KB (and that includes overhead for type codes and stuff). The CAPI documentation says you should really limit the size to 40KB, plus or minus. "SUMMARY" items (those which can appear in a View) have to be 15KB or less in size. Rich text items can be any size, but that's because when a single item containing rich text data gets too big (more than 40K or so), Notes just creates another item of the same name and tacks it onto the chain of data items that make up the document.

    My group expansion processing yields a Vector containing a String object for each name in the recipient field. Vectors can be any size, no problem). In order to obey the Notes limits, though, I chose to create a "recipient list" Vector, and add name to it from the original Vector, counting the length of each String as I add it, until I hit my 40K (or so) limit.** Then I can just take the new Vector and hand it to Document.replaceItemValue(). If I fill up the original field name (this is in the copy, of course, not in the original journal message), I start a cascade: SendTo_1, SendTo_2, etc., until I've used up all the names in the expanded list. No single item has more than my pre-defined maximum of 40K characters (String.length returns number of characters, remember, NOT number of bytes! Pay attention, this might be important later).

    So, why, after doing all this work to keep the translated items short enough for Notes, do I get the "32K limit exceeded"?? At first I thought it was because i was counting characters in Java, not bytes. Each character in a Java String object (which is normally encoded as UTF-16) actually consumes 16 bits, or TWO bytes, not one. So a 40Kchar string is actually using up 80KBytes! But.....no. Because when I pass the String objects to Notes, Notes automatically translates each one to LMBCS (the internal Notes character set, see my character set tutorial here for a full explanation), which is mostly 1 byte per character (not for Kanji and other multi-byte character sets, but in this case that's not what I was dealing with).

    Ok, so....whatever. I dialled the max item size down to 30K and ran it again. Same error. Crap.

    So then I did what I often do when I'm really, really stuck. I called my friend Daniel. We brainstormed. To make a longish story short-ish, he made a suggestion which, although it sounded a little weird, actually WORKED (he's very good at that).

    The key was: Turn OFF the SUMMARY flag on the new items. Here's an approximation of how I did that in my Java code?

    Vector expanded;     // the expanded list of names. We get that in other code I'm not showing you today. List of String objects
    Vector newField;       // the list for a single document Item we're going to build, and track the length of
    int newLength;
    String nextOutputItem;   // name of next item to write to the document

    // skipping details of outer loop over recipient fields....
    // do a loop over names for the next output item
    // (skipping tedious logic about cascading output items when they get too big...boring)
    newLength = 0;
    nextOutputItem = "";
    for (int i = 0; i < expanded.size(); i++)
        {
        String aName = (String) expanded.elementAt(i);
        if (newLength >= MAX_LEN_WE_MADE_UP_SOMEWHERE)
             {
             // current newField is full, write it out
             // MUST turn off the SUMMARY flag before doc save
             Item newItem = doc.replaceItemValue(nextOutputItem, newField);
             newItem.setSummary(false);
             nextOutputItem = "";
             }
        else {
             newLength += aName.length();
             newField.append(aName);
             }
        } // end for i

    Ok, I hope you get the basic idea. replaceItemValue() returns the Item object (convenient), so just use it to turn off the SUMMARY flag. Then when you save the document, you don't get that nasty 32K error.

    BUT, the question remains: WHY does this work? We haven't reduced the size of the items we're writing. And if the limit for SUMMARY items is 32K, why did we still get the error when we starting limiting the items to 30KB??

    I'm afraid, my friends, that the answer has to be: The error message is wrong. Bogus. Full of it. Misleading, at best. There IS NO 32K LIMIT!! 14K (or so) limit, maybe. THAT's why turning off the SUMMARY flag works, we're telling Notes that the item content is allowed to go all the way to 64KB (or so).

    So, there you have it. Go forth and code, with no pesky, wrong error messages to befuddle and comfusticate your day.

    Geek ya later!

    ** "Got the Vector, Victor?" (Lord, I just love the movie "Airplane"...)

    (Need expert application development architecture/coding help?  Want me to help you invent directory services based on RDBMS?? Contact me at: bbalaban, gmail.com)
    Follow me on Twitter @LooseleafLLC
    This article ©Copyright 2010 by Looseleaf Software LLC, all rights reserved. You may link to this page, but may not copy without prior approval.

    Comments

    1Erik Brooks  10/20/2010 5:36:11 PM   32K limit exceeded , huh?? Why SUMMARY is not always your friend

    Fun stuff... also, you can't have more than 64K of summary data in an entire doc, and I can't remember (it's been forever since I've seen this) but you may get that bogus error once you hit *that* limit, too. Or maybe not... I might be remembering wrong. :-)

    2Nathan T. Freeman  10/20/2010 6:02:20 PM   32K limit exceeded , huh?? Why SUMMARY is not always your friend

    I'm confused as to the source of your confusion. You say yourself...

    "SUMMARY" items (those which can appear in a View) have to be 15KB or less in size.

    ...and yet you were trying to put 40KB in a new item, which by default is Summary. I would expect that to fail.

    And so i would also expect that turning off the summary flag would fix the problem, as it apparently has. So what is surprising here? That the summary flag is on by default?

    The error message is misleading I guess, since it's informing you that there's a 32KB limit when it's documented at 15KB. Maybe that limit was increased at some point?

    Anyway, because the API behavior is so coarse when dealing with summary information, my own rule of thumb is that if you don't need to show it in a view, mark it nonsummary. It helps just about everywhere. Even on viewentry.getColumnValues(). This definitely seems like a good exampe of that.

    3Bob Balaban  10/20/2010 6:26:05 PM   32K limit exceeded , huh?? Why SUMMARY is not always your friend

    Hey Nathan - No, I wasn't confused about SUMMARY being on by default (forgot to mention that, not everyone knows it). I WAS confused that when I created items smaller than 30K I still got the error. Your rule of thumb seems reasonable, though.

    4Richard Schwartz  10/20/2010 7:55:43 PM   32K limit exceeded , huh?? Why SUMMARY is not always your friend

    Ditto what Nathan said. In your 3rd paragraph you state outright that the C API doc says that: "'SUMMARY' items (those which can appear in a View) have to be 15KB or less in size." But then in the 2nd paragraph after the code sample, you contradict yourself, saying: "And if the limit for SUMMARY items is 32K..."

    It clearly can't be both 15K limit and 32K limit, and it isn't! You've confused the 15K limit on an individual text list item with the 32K limit on the total size of the summary buffer.

    From the C API Reference doc for TYPE_xxx [TEXT_LIST]: "The total length of a Text List field must not exceed 15 kilobytes. By default, the SUMMARY flag is set for Text List fields."

    And from the C API Reference doc for NSFItemSetText (and repeated verbatim in several other places): "The maximum size of the summary buffer is 32K per note. If you append more than 32K bytes of data in items that have the ITEM_SUMMARY flag set, NSFItemSetText will succeed, but NSFNoteUpdate will return ERR_SUMMARY_TOO_BIG."

    5YCI  12/21/2010 7:03:12 PM   32K limit exceeded , huh?? Why SUMMARY is not always your friend

    the issue rely on the followings though I guess

    1) Whenever you create an item it's a summary one by default. You have to explicitly change this settings.

    2)An Item can be used in a view only if it's a summary item.

    3) a summary item is limited to 16KB

    4) LMBCS handle char data I guess just like unicode in a double byte way

    So we may face this situation :

    Trying to store a standard item ( 1) => summary) with more than 16K chars will be translated to a 32KB LMBCS storage which yield an error with the 32KB LMBCS size limit...

    6YCI  12/21/2010 7:25:56 PM   32K limit exceeded , huh?? Why SUMMARY is not always your friend

    Sorry submitted previous post talking about LMBCS but i think it's Unicode that is used by the interpreter. the point is that the size limit should be defined with the right charset. The error doesn't mention this charset. if evaluated as lenb(string) compared to len(string) it would result as a double size / standard char which i guess are used for group naming. What is confusing then is the missing charset in limitation & error msg.

    7Mike Woolsey  04/13/2011 4:20:23 PM  32k, 15k, 64k and so forth

    Hey Bob B. Long time no see.

    Here're the limits I remember -- things may've changed, but the more they change &etc. ...

    32k : the maximum size of the SUMMARY area for each document. Otherwise the view collection has a cow and spits up over your index.

    15k: the maximum size one SUMMARY field is allowed to take up by the Notes Editor. The editor is whiny about this, but that's also saved us often enough to where I don't mind. The editor will also throw an error if the document's SUMMARY area is too big: 32k.

    64k: the maximum size of any non-rich-text field.

    The recommendation to knock out the SUMMARY flag is perfect. We also asked Lotus to give us a checkbox to disable SUMMARY default on forms. After a decade, everybody wants what I want! =chuckle=

    8Smithc763  05/18/2014 7:11:11 AM  John

    Thank you for every other informative blog. The place else may just I get that type of info written in such a perfect means? I've a project that I'm just now working on, and I have been at the glance out for such information. dgefkkbefadebbfe

    9Smithd441  06/03/2014 3:29:10 PM  John

    Your weblog is 1 of a kind, i really like the way you organize the topics. ddcegcagdedcekeg

    10Smithk622  06/06/2014 12:34:38 AM  John

    Great website! I am loving it!! Will be back later to read some more. I am taking your feeds also kkdfedkbdeffbkfa

    11Bob Balaban  03/12/2015 3:40:41 AM  Untitled

    Thanks! Did anyone ask about the LSX Toolkit? :-)

    12Smithb778  09/24/2016 6:23:48 PM  John

    Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point. You clearly know what youre talking about, why throw away your intelligence on just posting videos to your weblog when you could be giving us something enlightening to read? ddbafedddcdbaaek

    13Bob Balaban  09/25/2016 5:32:46 PM  John

    John, Thanks for your comment, but are you sure you posted it in the correct blog? I don't think I have ever posted an entry with a video. Certainly not recently.

    14Smithd942  03/21/2017 7:08:35 PM  John

    Oh my goodness! an amazing article dude. Thanks Nonetheless I am experiencing issue with ur rss . Dont know why Unable to subscribe to it. Is there anybody getting similar rss drawback? Anybody who is aware of kindly respond. Thnkx edkcccaddgcfadda