inadvertent load test against Sybase ASE --- what did a Hobbit find

Last few weeks, I have been busy building up NMS for our site using Hobbit Monitor, an open-source NMS suite mimicking Bigbrother. For the most part, Hobbit Monitor (4.2-RC-20060712 w/o all-in-one patch) works out of the box for non-SELINUX server. I did my SELinux tweaks so it works on CentOS 4 w/o disabling SELinux. My job this week is mainly adding/customizing various health-checks, with some trials & errors here and there.
From time to time, for production database server running Sybase ASE 12.5.x on CentOS 4/i386, a few checks complained no data reported by Hobbit client. After a few days, I discerned the pattern. That is, [ports] and NMS servers's [hobbitd] are always bad when [files] [msgs] [procs] went CLEAR (no data, but considered OK by design) or PURPLE (non-reporting). Hobbitd's history page has 'client data' shows various truncation message. A post to the mailing list got answer from the author Henrik himself. I bumped up the size limit on client data/message from the default 256K to 1024K. Lo-and-behold, there are over 9000 TCP TIME_WAIT connections (~= new db connection) to the database engines.
Since TIME_WAIT is 2 x MSL (Maximum Segment lifetime), it means the application servers made over 9000 connections to the database server within a four-minute sliding window. Talking with our application architect turns fruitful. He said he has been contemplating to use pooled connection for some auxiliary processes. The main web applcation counts for only 20~40 connections (TCP ESTABLISHED) to the database since they are pooled. Recently in our performance lab we are testing how scalable Sybase ASE is for our web application. Now we sorta know. It can handle >= 2125 new database connections per minute, or 3 millon a day :)

Hobbit monitor :: a nice BigBrother replacement

it took me a bit to hunt down the good community of Hobbits, and it is well worthy. Hobbit Monitor's sourceforge.net site is not kept up-2-date. Wonder whether the wrong impression people get off the SF site misled people to believe it is not maintained up-to-date. The Hobbit Monitor seems to be nice & functional Big Brother replacement. The active development and the author responds to mailing-list question and code/attach patch while discussing a bug with an end-user, is reminiscent of the early open-source days or early Internet days. I definitely like the ACK feature and the zoom-in on history and trend graph (guess it'd be part of the rrdtool function as well).