inadvertent load test against Sybase ASE --- what did a Hobbit find

Last few weeks, I have been busy building up NMS for our site using Hobbit Monitor, an open-source NMS suite mimicking Bigbrother. For the most part, Hobbit Monitor (4.2-RC-20060712 w/o all-in-one patch) works out of the box for non-SELINUX server. I did my SELinux tweaks so it works on CentOS 4 w/o disabling SELinux. My job this week is mainly adding/customizing various health-checks, with some trials & errors here and there.
From time to time, for production database server running Sybase ASE 12.5.x on CentOS 4/i386, a few checks complained no data reported by Hobbit client. After a few days, I discerned the pattern. That is, [ports] and NMS servers's [hobbitd] are always bad when [files] [msgs] [procs] went CLEAR (no data, but considered OK by design) or PURPLE (non-reporting). Hobbitd's history page has 'client data' shows various truncation message. A post to the mailing list got answer from the author Henrik himself. I bumped up the size limit on client data/message from the default 256K to 1024K. Lo-and-behold, there are over 9000 TCP TIME_WAIT connections (~= new db connection) to the database engines.
Since TIME_WAIT is 2 x MSL (Maximum Segment lifetime), it means the application servers made over 9000 connections to the database server within a four-minute sliding window. Talking with our application architect turns fruitful. He said he has been contemplating to use pooled connection for some auxiliary processes. The main web applcation counts for only 20~40 connections (TCP ESTABLISHED) to the database since they are pooled. Recently in our performance lab we are testing how scalable Sybase ASE is for our web application. Now we sorta know. It can handle >= 2125 new database connections per minute, or 3 millon a day :)

sybase ase hourly transaction log dump keep growing in size (continued)

To me, the replication transactions should not be considered part of normal db transactions, in the sense that they don't alter the data state of the database, but that of the replicated twins. At least, no reason for 'truncation point' for the real db to get held hostage if a gone-wild REP thread declined/forgot/neglected to commit its replication transaction.

sybase ase hourly transaction log dump keep growing in size

our dba asked me for more space on database servers since it ran out. I opened my trending graph by MRTG. sure enough, the file system holding the backups has creeping up for three weeks now. however, it grows too fast, something like 80G a day instead of 8G a day I'd expect. with the help of 'du' and 'sort', I found the hourly transaction dump is the culprit. It has grown from 10M each to 1G each in the course of three weeks. reported this back to the sybase DBA. after some digging, he said it is a gone-wild replication thread holding a transaction open, somehow. even nightly full dump failed to mark the db such that hourly dump would cover diff instead of accumulative...