perc 4e/Di on Dell PE6850 saga continues...part A

We ended up applying BIOS upgrade (A00->A01) and PERC 4e/Di firmware upgrade (521A to 522A A13) for the system lockup problems we had on the production database server running on a Dell PE6850. Home-made load tests didn't cause panic for 18 hours. The server was then rushed back into production since the fail-over spare server couldn't stand the load.

The server (the Sybase database engines) has been up for 14 days today. At 09:50am, just when the server started to ramp up to its daily load peak (CPU load ~=4) , some processes failed to write to the disk and 'date > junk' from cmdline just hang there. I canceled that 'date>junk'. All is good after less than 4 minutes. Nothing interesting (warn/error/abort) in the system log, exportlog from PERC controller, or database log. PR was running at the time.

The symptoms definitely differ, so the BIOS and firmware upgrade did make some difference towards the better. For the previous two lockups and the only two for 15 months, we lost access to the disks totally, getting "reject i/o to offlined disk" without kernel panic or corruption. This time, this is merely a hiccup or pause or suspension of sorts.

Older postings on similar topic on dell-linux-poweredge forum suggested PR could be the culprit if BIOS/firmware is up-to-date. On the system, I get the following output from '"megapr -dispPR -a0" today. Is #Iterations current count of the total PR has run or a threshold or some sort? If the former, how to clear it? If the latter, how to increase? Basically I am looking into why it locked up exactly 30 days (could be coincidence too. and we are now using newer BIOS and firmware). Dell diag from OMSA 4.4 on 10/17/2006 suggests nothing wrong the controller, memory, or underlying disks. (omreport on the controller is appended below too).

********PR INFO********
Mode :AUTO
#Iterations:2200
Status :PR In Progress

# omreport storage controller
Controller PERC 4e/Di (Embedded)

Controllers
ID

: 0
Status : Ok
Name : PERC 4e/Di
Slot ID : Embedded
State : Ready
Firmware Version : 522A
Driver Version : Not Applicable
Minimum Required Firmware Version : Not Applicable
Minimum Required Driver Version : Not Applicable
Number of Channels : 2
Rebuild Rate : 30%
Alarm State : Not Applicable
Cluster Mode : Not Applicable
SCSI Initiator ID : 7

Also, we upgraded the BIOS from A00 to A01, instead of to the latest A04, since the release notes of A02 through A04 didn't read pertinent at the time. At second read of A03's release notes, I noticed the following two fixes that could be relevant to the system. Where can I find more detailed notes other than PE6850-BIOSA03.TXT ? I don't quite understand why the developers or release managers so minced on words.

• Added support for Virtualization Technology in the processor.

Should I assume this is not referring to HT, but of special server virtualization assistance from Intel's VT (?) technology or alike ?

• Added support for 800MHz system configurations.

Does this mean BIOS prior to A03 doesn't support 800MHZ system configurations?

Although the megaraid* driver is dated early 2005. The CHANGLOG.megraid in /kernel/Documentation doesn't have much interesting changes either.

step-by-step: how to set up a large ramdisk on CentOS 4.4/i386

On our new database server running CentOS 4.4 on a Dell PE 6850, a 2G ramdisk is needed. The option to use 'tmpfs' was considered and forfeited due to its potential of using disk swap.

• To make the ramdisk 2G apiece, a kernel parameter (ramdisk_size) needs to be passed ( grub.conf or lilo.conf) and a quick reboot is required to make it effective. The ramdisk by default is 16M in size.
  

Here is my grub.conf entry:

kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ console=ttyS0,57600n8 console=tty1 ramdisk_size=2097152

• To make the ramdisk accessible to the database after reboot, I cooked a chkconfig-compliant init script

---cut----x------/etc/init.d/ramdisk4syb-----x-----cut---------
#!/bin/bash
#
# $Id: rc.ramdisk4syb,v 1.00 2006/10/23 13:57:14 jer Exp $
#
# rc file for set up ramdisk fs for sybase to use for tempdb
#
#

# For Redhat-ish systems
#
# chkconfig: 2345 30 70
# processname: ramdisk4syb
# config: none
# description: set up ramdisk for sybase to use for tempdb
case $1 in
start)
test -d /mnt/ram1 || mkdir /mnt/ram1
mke2fs -m 1 -b 1024 /dev/ram1 >/tmp/ramdisk.mke2fs.log &&
mount -t ext2 /dev/ram1 /mnt/ram1 &&
chmod 770 /mnt/ram1 && chown sybase:dba /mnt/ram1
;;
stop)
umount /mnt/ram1
;;
*)
echo "Usage: $0 start|stop"
;;
esac

• to pin the init script to run levels

chkconfig --add ramdisk4syb

• to test the init script before reboot

/etc/rc3.d/S30ramdisk4syb start && df -k /mnt/ram1 && ls -ld /man/ram1

• to make all this effective by reboot.
  

init 6

• to verify the resultant ramdisk file system
  

df -k /mnt/ram1 && ls -ld /man/ram1

Much to my surprise, the ramdisk is not mounted after the reboot! The mke2fs log showed the mke2fs was successful. When I attempted to mount it manually, I got
# mount /dev/ram1 /mnt/ram1 mount: wrong fs type, bad option, bad superblock on /dev/ram1, or too many mounted file systems

In the log, I did notice the block size was 4K was instead of the 1K block size I observed with a 512M ramdisk.
Block size=4096 (log=2) Fragment size=4096 (log=2) 262144 inodes, 524288 blocks

Since the Sybase ASE uses 2K page size, I tried mke2fs using a 2K block. No joy! Mounting attempts were met with the same errors.

1K block size is known to have worked for 512M ramdisk, so I gave it a try. Bingo!
#mke2fs -b 1024 /dev/ram1 Block size=1024 (log=0) Fragment size=1024 (log=0) 262144 inodes, 2097152 blocks
Now we have a working ramdisk as below:
/dev/ram1 2063763 3086 1955820 1% /mnt/ram1
I don't see much of a point to waste space on 5% reserved for superuser. So, I tuned it down to 1%
# tune2fs -m 1 /dev/ram1
/dev/ram1 2063763 3086 2039706 1% /mnt/ram1
From the 'free' output, I am a bit surprised to see the 2G is not counted as 'used'.
total used free shared buffers cached Mem: 16627288 231236 16396052 0 48300 89240 -/+ buffers/cache: 93696 16533592 Swap: 16779884 0 16779884

bootparam man[7] page actually says " These days ram disks use the buffer cache, and grow dynamically." If so, ramdisk truly behaves similar to tmpfs at this point, except

• ramdisk's maximum size (the reserved?) needs a reboot to adjust, while tmpfs can be adjusted on the fly.
  
• ramdisk won't have swap in the picture at all. [good thing]
  

However, by growing dynamically, FWIW, it may not able to grow to its full reserved size when it needs it [bad bad thing], if physical RAM is used up by some real processes already. Questions yet to be answered:

• Does the 2.6 kernel or other series know or set ramdisk has higher priority on its claim to thus reserved memory?!
• why ext2 file system made with block sizes other than 1K failed to be mounted ? ramdisk is the only case here?

inadvertent load test against Sybase ASE --- what did a Hobbit find

Last few weeks, I have been busy building up NMS for our site using Hobbit Monitor, an open-source NMS suite mimicking Bigbrother. For the most part, Hobbit Monitor (4.2-RC-20060712 w/o all-in-one patch) works out of the box for non-SELINUX server. I did my SELinux tweaks so it works on CentOS 4 w/o disabling SELinux. My job this week is mainly adding/customizing various health-checks, with some trials & errors here and there.
From time to time, for production database server running Sybase ASE 12.5.x on CentOS 4/i386, a few checks complained no data reported by Hobbit client. After a few days, I discerned the pattern. That is, [ports] and NMS servers's [hobbitd] are always bad when [files] [msgs] [procs] went CLEAR (no data, but considered OK by design) or PURPLE (non-reporting). Hobbitd's history page has 'client data' shows various truncation message. A post to the mailing list got answer from the author Henrik himself. I bumped up the size limit on client data/message from the default 256K to 1024K. Lo-and-behold, there are over 9000 TCP TIME_WAIT connections (~= new db connection) to the database engines.
Since TIME_WAIT is 2 x MSL (Maximum Segment lifetime), it means the application servers made over 9000 connections to the database server within a four-minute sliding window. Talking with our application architect turns fruitful. He said he has been contemplating to use pooled connection for some auxiliary processes. The main web applcation counts for only 20~40 connections (TCP ESTABLISHED) to the database since they are pooled. Recently in our performance lab we are testing how scalable Sybase ASE is for our web application. Now we sorta know. It can handle >= 2125 new database connections per minute, or 3 millon a day :)

sybase ase hourly transaction log dump keep growing in size (continued)

To me, the replication transactions should not be considered part of normal db transactions, in the sense that they don't alter the data state of the database, but that of the replicated twins. At least, no reason for 'truncation point' for the real db to get held hostage if a gone-wild REP thread declined/forgot/neglected to commit its replication transaction.

sybase ase hourly transaction log dump keep growing in size

our dba asked me for more space on database servers since it ran out. I opened my trending graph by MRTG. sure enough, the file system holding the backups has creeping up for three weeks now. however, it grows too fast, something like 80G a day instead of 8G a day I'd expect. with the help of 'du' and 'sort', I found the hourly transaction dump is the culprit. It has grown from 10M each to 1G each in the course of three weeks. reported this back to the sybase DBA. after some digging, he said it is a gone-wild replication thread holding a transaction open, somehow. even nightly full dump failed to mark the db such that hourly dump would cover diff instead of accumulative...