Wednesday, August 09, 2006

inadvertent load test against Sybase ASE --- what did a Hobbit find

Last few weeks, I have been busy building up NMS for our site using Hobbit Monitor, an open-source NMS suite mimicking Bigbrother. For the most part, Hobbit Monitor (4.2-RC-20060712 w/o all-in-one patch) works out of the box for non-SELINUX server. I did my SELinux tweaks so it works on CentOS 4 w/o disabling SELinux. My job this week is mainly adding/customizing various health-checks, with some trials & errors here and there.
From time to time, for production database server running Sybase ASE 12.5.x on CentOS 4/i386, a few checks complained no data reported by Hobbit client. After a few days, I discerned the pattern. That is, [ports] and NMS servers's [hobbitd] are always bad when [files] [msgs] [procs] went CLEAR (no data, but considered OK by design) or PURPLE (non-reporting). Hobbitd's history page has 'client data' shows various truncation message. A post to the mailing list got answer from the author Henrik himself. I bumped up the size limit on client data/message from the default 256K to 1024K. Lo-and-behold, there are over 9000 TCP TIME_WAIT connections (~= new db connection) to the database engines.
Since TIME_WAIT is 2 x MSL (Maximum Segment lifetime), it means the application servers made over 9000 connections to the database server within a four-minute sliding window. Talking with our application architect turns fruitful. He said he has been contemplating to use pooled connection for some auxiliary processes. The main web applcation counts for only 20~40 connections (TCP ESTABLISHED) to the database since they are pooled. Recently in our performance lab we are testing how scalable Sybase ASE is for our web application. Now we sorta know. It can handle >= 2125 new database connections per minute, or 3 millon a day :)

No comments: