miles wide miles deep

how to manage SSL certificates for Tomcat using IIS tools

2011-12-13T15:27:00.001-05:00

I worked in a shop where Windows & IIS servers dominates the operations. A few third-party applications were hosted by Tomcat, which terminates SSL connection as well. Engineers loathes the use of keytool, a java utility to manage certificate and keys.

I first found that it is relatively easy to convert keystores from the SUNW format used by keytool to PKCS#12 (.pfx) format used by IIS, and back. This means one can easily follow normal procedures to obtain a new certificate and export it to .pfx format.

After tinkering with it a bit, I read the tomcat doc. It seems that Tomcat (at least for v6) can support PKCS12 format directly.

Fixed: Cisco VPN client minimized on Windows 7 taskbar & disconnected

2011-09-27T09:59:00.001-04:00

Cisco Systems VPN Client 5.0.0.7.0410 has worked for a month since I first installed it on Windows 7 Enterprise x64. One day, the VPN Client 'connect' window started minimized and stayed in the tool bar when launching. There's no useful choices in pop-up menus for the client window minimized on the taskbar and the VPN service (cvpnd.exe) icon in the Windows tray. Not choices were listed to 'minimize' , 'maximize', 'restore', or 'move'. Since the Cisco VPN client was launched as 'disconnected', it was basically useless!

Googling found it is rather common a problem and people usually blame it on older versions of the Cisco VPN clients, latest Windows updates, etc. Originally I intended to write a javascript to launch the binary (vpngui.exe) with its window maximized. The javascript snippet I wrote in the past is not handy, so I continued my search a little bit more, only to stumble upon this thread in a Microsoft forum. One of the answers pointed me to the direction. That is, the direct cause is rather simple. That is, window sizing somehow got messed up in vpnclient.ini under "C:\Program Files\Cisco Systems\VPN Client".

vpnclient.ini looked like below. Instead of LogWindow* to have super big numbers, my copy has WindowX & WindowY assigned super big numbers. Once I changed WindowX and Window Y to match WindowWidth & WindowHeight, respectively, it started to launch the Cisco VPN client Connect window normally again. I experimented a little bit more. It seems that the absolute number does not matter such, as long as they are not ridiculously big.

I have dual heads (dual display) setup. HP Elitebook laptop drives two HP L2245wg monitors. A similar thread on Virtual PC 2007 console disappearing problem suspected that dual display may have contributed to erroneous setting with this huge number. On the other hand, this work laptop is a managed node, so, I won't be surprised if something was pushed by our dear IT department to my laptop image to have caused this by accident.

Important thing is, the fix is simple and I'm happy for now. I will keep an eye on this to see if it gets messed up again.

Before (messed-up version)

---------cut-------------8<---------C:\Program Files\Cisco Systems\VPN Client\vpnclient.ini ----------------8<----------

[main]

ClientLanguage=

[GUI]

DefaultConnectionEntry=OneOfFortune500ciscoVPN

WindowWidth=485

WindowHeight=99

WindowX=4294935296

WindowY=4294935296

VisibleTab=0

ConnectionAttribute=0

AdvancedView=1

LogWindowWidth=0

LogWindowHeight=0

LogWindowX=0

LogWindowY=0

---------cut-------------8<----------------------------------8<----------

After (fixed and verified to be working).

---------cut-------------8<---------C:\Program Files\Cisco Systems\VPN Client\vpnclient.ini ----------------8<----------

[main]

ClientLanguage=

[GUI]

DefaultConnectionEntry=OneOfFortune500ciscoVPN

WindowWidth=485

WindowHeight=99

WindowX=485

WindowY=99

VisibleTab=0

ConnectionAttribute=0

AdvancedView=1

LogWindowWidth=0

LogWindowHeight=0

LogWindowX=0

LogWindowY=0

migrate to VirtualBox from VMWare Server : sadly & happy to

2011-08-17T17:17:00.001-04:00

I have been a loyal VMWare server 1.0 user since it became free in 2005. It is used to build various sandbox on my Windows XP laptop at work. The guest OS include RHEL, CentOS, Fedora, Solaris 10, Windows server 2008, Windows server 2003, Windows XP & Windows Enterprise 7. One feature I like a lot is its capability, with an easy-to-use GUI, to modify network settings to choose subnet, DHCP lease, and port forwarding, etc.

After upgrading my home desktop to long-awaited CentOS 6 early this month, I could no longer compile & run VMWare server. Since my home PC runs a mini lab of 3~4 nodes to play with cobbler & puppet, I need to find a solution that can play the same vmware machine (VMDK storage). Virtualbox from Oracle (Sun/Virtualbox.org) seems to be a good choice, given its popularity in LinuxJounal's 2010 poll.

The one RPM installation is great.
It uses DKMS to compile and it compiles successfully in one shot.
Guess OS support is up to date, with WIndows 7 and Server 2008. It does not differentiate versions of RHEL, RHL, Fedora though.
PXE capability comes from the extension pack. you'll need to download & load. The extension pack license is free only for evaluation or personal use.
Nowhere to find UI to modify network settings, other than choosing types of networking (NAT, bridged, etc.).

add NAT adapter, only when you need this VM go talk with the world linked to the host
add 'intnet' adapter and provides a meaningful name. This name apparently is used by Virtualbox to wire the other guest os into the same network, if they share the same name for their intnet adapter.

Virtualbox won't import vmware guest images as is, since it imports only Open Virtualization format. On the other hand, it does support VMDK format, such that you can create a new virtualbox and choose to use an existing virtual disk (the vmware guest image you intend to import or run from virtualbox).
Too bad the proxy preference do not extends to guests (behind the NAT). Instead, I had to configure http proxy at various places (profile.d, yum.conf, gnome) inside my CentOS 5 or CentOS 6 guests.

The other thing noteworthy is what type of controller Virtualbox associates your existing vmware disk (vmdk) to. It needs to match. Otherwise the working vmware image may fail to boot properly. It happened to me, when Virtualbox associated an IDE-based disk to SATA controller. Once I powered off the guest, then redid the association, I were able to boot up the image w/o a glitch from there on.

The installation & look & feel are pretty smooth & consitent & faster(?),on both a Windows 7 Enterprise x64 host (HP Elitebook laptop. 4G RAM) and a CentOS 6.0 i386 host (a Compaq PC, 2G RAM). For now, I think I'll continue the path of migrating to VirtualBox. VMWare server will stay on for a bit, just in case I decide to roll-back. I am definitely looking forward for taking advantage of the capability to take multiple snapshots, with notes! It bothered me when I could keep only one snapshot and couldn't annotate what is this snapshot, with VMWare Server 1.x & 2.x

willing to pay extra to keep the service you like in business

2011-08-16T11:48:00.001-04:00

Recently a premium local computer VAR went out of business. Many are sad.
The question is, are you, willing to pay extra to keep the services you like in business?
Take a moment & note your position in the comment below.

RHEL 5 is released today

2007-03-14T22:46:00.001-04:00

Tonight, I read a review by Linux Format , the best-selling Linux magazine in the UK. The review is done for the latest RHEL 5 (Redhat Enterprise 5) which is released today. Technology enhancements are welcomed, such as those in SELinux administration and full integration of Xen virtualization as expected from Fedora Core 6. The review also included an Q/A interview with Nick Carr, general manager, RHEL.

More noteworthy is that, Redhat Inc., as a company, has revamped its support, marketing, customer relation plan, and its relationship with the open source community. As stated in the review, the company has realigned the whole company to work better with the open source community as well as their enterprise customers.

The one page SLA
server/client flavor versus the traditional WS/ES/AS flavors for RHEL3 and RHEL4.

first impression of Bestpractical's RT, an open source ticketing system

2007-02-14T00:02:00.001-05:00

I heard quite a bit of good things about RT, an open source ticketing system. Here is the first impression I had after retrieving the source tarball off bestpractical.com's web site.

The current stable version is 3.6.1.

Per README in the source tarball,

"RT is an enterprise-grade issue tracking system. It allows organizations to keep track of what needs to get done, who is working on which tasks, what's already been done, and when tasks were (or weren't) completed."
GPLv2
commercial support is available including training and such: sales@bestpractical.com .

The installation seems a little too involving, even for a professional UNIX/linux system engineer.

it requires Perl >5.8.3 with a lot of CPAN modules. To the developer's credit, a RT util tool is built to help get CPAN modules and compile them in place.
It support a few RDBMS as its backend: MySQL/Postgres/Oracle/SQLite
Apache with mod_perl (or FastCGI)

linttylog hang my Linux system on a Dell PowerEdge 6850 at first run

2007-02-13T23:55:00.001-05:00

When troubleshooting a Dell PowerEdge 6850 server, I read on Dell's linux-poweredge mailing list that a small utility named linttylog may be useful to dump log from the PERC raid controller. The function sounded a lot like what action=exportlog does using OMSA's srvadmin services. However, I was desperate enough to try anything reportedly working. So, without much ado, I downloaded the utility. It came in as an RPM and was a bit aged (probably out-of-date too).

syb04:/# rpm -ivh linttylog-1.00-0.i386.rpm
Preparing...                ########################################### [100%]
   1:linttylog              ########################################### [100%]

syb04:/# linttylog

        Serial Port Output History Handling Application 1.00 (Date 10/01/2003)
                Copyright (c) 2003 LSI Logic Corp.

Depending on the buffer size, it may take several minutes...Please wait...

        TTY History Updated in the file tty.log.

Exiting.
syb04:/# ls -ltr
====================================
The ssh shell session got disconnected from here. I couldn't log back in either. No response on serial console either. Nothing on the console screen. Nothing logged onto the remote syslogd server either, even though all kernel.* and all *.warn are redirected via /etc/syslogd.conf and verified to be working.

I had to power cycle the server to get it back online again. Once the server was back, I re-ran the utility successfully without any problem. What the ... The moral of this post is not to run linttylog on your mission critical system in production.

lay off ~= business strategy ~= choked by your own success

2007-02-07T08:30:00.001-05:00

Four weeks ago, I called in to a meeting from home. In that meeting, John, my manager was let go, because the business strategy has renewed. The rest of us were assured in a later company-wide town-hall meeting, that "this is the team we are going forward with." John wasn't present. I was told later that he was asked not come in at all and his employment terminated that day. Why the rush? We were just about to roll out a major release on the ASP platform our team work on for the first third of our customer base. John was a hands-on manager, so the big void caused by his departure was apparent.

Three weeks ago, Stevie, Matt, and Prakash, three more team members, were let go. Their positions were eliminated, because the management is confident that the latest release on the ASP platform is "feature-rich and robust enough to go forward without new development, for years to come". Again, my team or the left half of it were assured that "this is the team we are going forward with." Why the rush? We were yet to roll out the major release to the rest two thirds of our customer base.

Does this second layoff justification suggest that our team choked by our own success? If less feature were rushed in by John and squeezed in by Sales/marketing and we didn't work days and nights and weekends to smooth the edges, we'd all still have jobs? I couldn't help wondering whether this why ISVs release bug-ridden code or QA/test inadequately.

At any rate, the red PANIC button has been pounded on. Seasoned professionals as we are, we instantly hit the floor. Calling recruiters, line up old friends and contacts, update resumes and post them. It pretty much resembles a scene where rats rush off a sinking boat.

Various news starts to tickle in through the cubicle walls. Hallway meetings anytime. Then, Matt's sense of humor struck me. He said, "Yes they were a bit clumsy firing people. It could only be a good thing. At least they are not used to this kind of thing or hadn't do it often enough to become proficient and smooth."

I don't think I am thinking straight right now. I probably will add a few more comments when I have leisure to reflect, after I find my new job somewhere else.

how to verify ownership to Yahoo on wordpress.org blog server

2007-01-18T16:29:00.001-05:00

The mischief caused by 404.php template is not a problem for authentication, when I stumbled into Yahoo!'s Site Explorer. To claim your site, you were instructed to place a special file with special content under / of the site. This way, one GET will do and have no problem with customized 404 pages. The latter is pretty common in use for sites managed by a CMS (Content Management System) or blogging servers. I wonder how come the smart engineers at Google decide to do two GET instead...

Once I added the required file on my wordpress.org blog server's / and clicked to continue, the next page asked me to keep the file there for 24 hours, till Yahoo's bots take their sweet time to crawl, literally! To a sharp contrast, Google's webmaster tool authenticates site ownership real-time, and sucks in sitemaps real-time too!

Placing a special file under / is the only way to authenticate your ownership on Yahoo Site Explorer. For millions of hosted sites (blogs or otherwise) whereas content owners don't have access to the /, they'd be out of luck. For now, at least. Hopefully when Yahoo! Site Explorer comes out of beta, they'd come up with a way to authenticate sites whose content owner have content-level access (META tokens, maybe?) instead of file-level access.

In comparison with Google's web master tools, Yahoo's site explorer is so spartan right now. Its own blog hasn't been updated for a few months now. I guess it is real beta then.

trial & errors :: SEO Dave's adSense-spiked themes

2007-01-14T12:13:00.001-05:00

It's a God-send when I found SEO Dave's themes for wordpress.org blogs. He spiked some default themes with Google's adSense and optimized the placements. I gladly took his words for it, since he is a SEO consultant by trade. This way, I can get new sites up quickly w/o laboring on SEO first :)

On a new wordpress.org server I set up recently, I attempted to unzip the five theme zip files obtained from his blog. I was surprised to be prompted to overwrite this file and that file. I said 'None'. It turned out that only "connections", one of five zip files holds everything under a directory, as expected for a compliant theme package, which is to be extracted under /wordpress/wp-content/themes. No biggie, a little command line bash magic, I got it.

/wordpress/themes.php, the admin page to preview and activate a theme, now showed all of them. Many, however, didn't show preview screenshots. Some of them even reported errors. Long listing of these files on the server revealed that the permission was too restrictive: 0700 for directory and 0600 for files. The root user's umask is 0022, so I am sure the restrictive permissions came from the theme zip, instead of from my sometimes overly-secure setup on the server. A few find and chmod later, all is well again.

There may be a bug in the original or spiked Blix theme, as it treats any new page as top-level, even if it is specified as child page for the ubiquitous About page. Same parent-child page relationship was handled properly by other themes such as Kubrick.

I posted a comment on SEO Dave's site in hope he may check it out, along with a wish of using Google's search box instead of the default search box.

how to verify ownership to Google on wordpress.org blog server

2007-01-14T11:57:00.001-05:00

I was checking out Google's web master tools site the other night. In order to verify the ownership, I opted to create a static HTML file on my site. Google failed to verify the ownership, stating it received a 404 error inside a 220-status page. I saw the file on the server and could browse to it properly using a browser too.

Puzzled, I looked at the server's access log. It turned out the Google attempted to retrieve two files. One was the long-winding name it stipulated. The other was the former file with its name prefixed with 'noexist_'. The logic is clear: Google wants be sure the 220 code returned for the "magic" file is real, by verifying a different code (404 in this case) would be returned if the "magic" file doesn't exist.

66.249.74.2 - - [11/Jan/2007:20:12:20 -0500] "GET /google0467d40068c96de7.html HTTP/1.1" 200 59
66.249.74.2 - - [11/Jan/2007:20:12:20 -0500] "GET /noexist_0467d40068c96de7.html HTTP/1.1" 200 5306

The help page claims Goggle does HEAD only. This obviously isn't true, or isn't true any more, per Apache's access log entries above.

<META name="verify-v1" content="DGxlTrIdDwI9xwBYeYOMddr34POYb934o45vCpf3t+nvcI=" />
I ended up use the META tag instead. I copy+pasted it into /wp-content/themes/myTheme/header.php right before </HEAD><BODY>. This time it worked just fine.

The mischief was caused by a beautified 404 page generated by the /wp-content/themes/myTheme/404.php. Vaguely recalling Apache's manual pages do state that ErrorDocument directive and some other types of redirect tend to lose the original response status code, be it 404, 501, or 403.

AJP proxy enabled by default for Apache 2.2.3 on Fedora Core 6

2007-01-10T05:25:00.001-05:00

As part of hardening an Apache instance on a new Fedora Core 6 Linux server, I commented out all _proxy_ modules in the main httpd.conf. When checking for syntax, however, I got
# /etc/init.d/httpd configtest
httpd: Syntax error on line 209 of /etc/httpd/conf/httpd.conf: Syntax error on line 2 of /etc/httpd/conf.d/proxy_ajp.conf: Cannot load /etc/httpd/modules/mod_proxy_ajp.so into server: /etc/httpd/modules/mod_proxy_ajp.so: undefined symbol: proxy_module

Surprised by the something actually required proxy_module, I took a look at the proxy_ajp.conf.
#cat proxy_ajp.conf

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

#
# When loaded, the mod_proxy_ajp module adds support for
# proxying to an AJP/1.3 backend server (such as Tomcat).
# To proxy to an AJP backend, use the "ajp://" URI scheme;
# Tomcat is configured to listen on port 8009 for AJP requests

This surely is nice. The building of mod_jk for Apache 2.0 on CentOS 4 has got old pretty fast. From a security standpoint, I'd think this comes off some httpd-tomcat package. Not exactly!
# rpm -qf proxy_ajp.conf
httpd-2.2.3-5

For my purpose, I just commented out the LoadModule directive inside proxy_ajp.conf. However, it'd make more sense, if

it comes off an optional module package, something like httpd-tomcat or httpd-ajp. Or,
disabled by default.

weird wormholes :: set reply-to header to the list, or not to

2007-01-09T17:42:00.000-05:00

Recently I asked on a mailing list how come I had to remember to copy the list address to the CC whenever I reply to a post. To my surprise, it turned to a big flame war of sorts. Unwilling to make any changes, the list administrator essentially asked both parties to go away, politely.

In its current form, you need to 'reply-all' to mailing list and the list administrator will make sure the poster's email address is in a special list such that the poster not receive two copies. Occasionally, someone may request his/her email address be added to the special list, yet again. What a hassle! Why the administrator wants all this hassle for nothing?!

It puzzled me in the past. It still puzzles me now. To my simple mind, the benefits of setting reply-to header to the list is so obvious:

less keystrokes for the list members. Many modern MUA (Mail User Agent) has default to 'reply' instead of 'reply-to-all', esp. web-mail UI.
less unnecessary mental note to reply-to-all instead of reply, or to copy list's address to CC if you already hit 'reply'
no missing discussion
consistently threaded discussion in archive and in live discussion. A post sent to the poster alone was often seen forwarded to the list, as an after-thought and after-fact good intention efforts. The thread is then broken, making it extreme difficult to follow a discussion you found an interesting excerpts by googling.

Instead,

Some said it is hard to be done.
Some insisted that it is philosophically wrong to reply-to the list. <= Hello, the purpose of subscribing to a mailing list is to publish to and read from the list, not to find sensible partners to conduct private conversations!
Some insists reply-to-all is great enough and is the only sensible way, so they went ahead to hack their mail clients (MUA) to detect whether a message is post or private message and automate to save the unnecessary keystrokes.
Well, maybe some of these people just accept the dysfunctional setting as the inevitable fact of life, and just find a get-around and moved on. If so, it is pretty sad.

Reflecting a bit more today on the observed need for some people to invent convoluted ways to satisfy themselves, I thought of what the agent in Men in Black says in a StarBucks coffee shop. There are people who can't control their own destiny or fate or fortune, and they are well aware of it. Instead, they opt to pay premium to select from a bloated feel-rich selection of lattes, as if they were the master of the universe.

downloads.wordpress.org needs a face lift

2007-01-08T09:21:00.001-05:00

Over this past weekend, I built a new wordpress.org blog server on a new CentOS 4.4 server. It seems the 'official' plugin/theme repository page at http://downloads.wordpress.org needs quite a face lift. In its current form, the plug-in and theme repository page is

spartan: It gives an ordered list of plugins and themes by name (and/or versions).
poor in function:

No descriptions to tell whether you need or like a theme or plugin. I had to just download it and install it, then read the descriptions from /wp-admin/plugins.php. Quite some wasted bandwidth on the server and time & efforts on the users.
no option to download it all to try it out. Instead, you have to click on each one. The lack of description certainly exacerbates the problem.

not up-to-date. For example, the link to tiger-admin plugin doesn't work. Upstream now has tiger-admin-v3.0.zip.
malformed HTML. plugin page has an empty link text for tiger-admin plugin, causing the page to display funny in FireFox 1.5/Linux and FireFox 2.0/windows.
Many of the themes don't show preview screens under /wp-admin/themes.php. Yet to check whether the themes themselves are at fault, or the tiger-admin v3.0 theme is the culprit.
no security or integrity assurance: no checksum or digital signature is provided to verify the authenticity and/or integrity of the file.

I guess I'll generate a more functional version and contribute to the site. Or alternatively, host a beautified version here myself.

how to coerce OMSA 5.1 to install & run properly on CentOS 4 (part III)

2006-12-22T12:23:00.001-05:00

Here is the summary of what I did for OMSA 5.1 on CentOS 4.1 ( the white-box twin of RHAS 4.1)

For installation, on a CentOS 4.1 without OpenIPMI or net-snmp, you just need to

append /etc/redhat-release with "Nahant", RHEL 4's code name.

1c1
< CentOS release 4.1 (Final) Nahant --- > CentOS release 4.1 (Final)
2. start installation by running ./linux/suppportscripts/srvadmin-install.sh
After installation, answer NO to start start all services. Instead, conduct the following steps first:
1. insert a line 'test -e /dev/ipmi0 || mknod -m 0600 /dev/ipmi0 c 253 0' at the beginning of /etc/init.d/dsm_sa_ipmi. w/o it, /etc/init.d/dsm_sa_ipmi will fail miserablly. It seems that udev doesn't put /dev/ipmi0 back as expected.
22,24d21 < # my hack < test -e /dev/ipmi0 || mknod -m 0600 /dev/ipmi0 c 253 0 < #
2. insert '/etc/init.d/dsm_sa_ipmi start' at the beginning of 'start' section inside /usr/bin/srvadmin- services.sh
w/o it, srvadmin-services.sh start will fail with IPMI drivers fail to load. However, if you start IPMI manually by '/etc/init.d/dsm_sa_ipmi start', you'd be just fine.
241,243d240 < # my hack::ipmi failed to start when called from later this script < /etc/init.d/dsm_sa_ipmi start <
3. now start it all with /usr/bin/srvadmin-services.sh start
4. verify it with ' omreport chassis temps' and 'omreport storage controller '. These two rely on different things to work. I'll elaborate on this later.

The rc scripts are placed in the S50 instead of their own start sequence number mandated in the scripts. I corrected them manually with my understanding of chkconfig directives in the scripts. However, I didn't get to change run level to test whether that would guarantee OMSA startup succesfully, since all my boxens are in production. Besides, Hobbit Monitor will let me know soon enough upon such failure.

It is such a great relief to have constant & comprehensive monitoring against critical systems. As a system engineer, you don't have to make all these mental or paper notes to check this or that. A decent NMS will do its job to nag you (whether you like or not) when necessary.

how to change IP address on a Linux server :: a project?

2006-12-20T13:34:00.001-05:00

I was asked this question once, "how do you change IP address on a Linux server?" At its face value, it's a simple technical question on systems administration or network administration using a Linux server. So, my instant response is

edit /etc/sysconfig/ifcfg-eth0 (or eth?). or system-config-network TUI or GUI.
adjust routes and personal fw on the host as needed
ifconfig eth1 down && ifconfig eth1 up (or /etc/init.d/network restart)
verify: ifconfig eth1 (to see if the new IP has taken effect)
check & verify from remote host it is actually accessible (TCP/IP level, plus other services the server may provide)
update DNS if applicable

To dwell on the question a little more, and depends on the context of the conversation, the lines of questions, and who asks, I would expand a bit from the perspective of process management, configuration management, and knowledge management. In other words, I'd prefer to drive such a "simple" change as a small project. Thus, here comes the addition to my initial answer:

So, the actual change is simple. Depends on the function of the server and its inter-relationship with other systems, you probably need to manage such a change as a project. As they say, a little bit more of thinking and planning goes a long way.

gather requirement from functional groups and management
set a date to cut-over and obtain sign-off from stake-holders
plan for it (procedures/fall-back plan/check-list/notification/etc.)
communicate the planned change and its potential impact to stake-holders and end users
dry run if necessary
prior to cut-over: for days leading up to the cut-over, shorten TTL for A record for that IP, if it has DNS record and is accessed by DNS name instead of IP alone.
prior to cut-over: check to be sure ACL or conduits or fw rules get updated on router/firewalls and systems the server gain access to or from
during cut-over: keep team and stake-holders abreast of up-to-date status
post cut-over: verify by going through your check-list to determine success
post cut-over: enter change log for this event & notify users the cut-over is done
post cut-over: alias or real NIC to keep servicing the IP for limited period. (could also just alias the new IP onto the NIC serving the current IP, the cut-over becomes switching who's alias and who's real)
post cut-over: added DNAT or reverse proxy rules to catch traffic to the old IP and redirect/log/alert as necessary
post cut-over: document this in the knowledge-management system: a scrapbook, a WIKI, a technical writer.

Hobbit Monitor :: how to report multiple temperture probe results

2006-12-18T09:34:00.001-05:00

For Dell PowerEdge servers, OMSA can report temperature for multiple probes, notably, "BMC Ambient", "BMC Planaar", "BMC Riser", and one for each physical processor. I initially wrote my own extension script named 'temps' , which reports all probes under one 'status server1.temps green' call. It works for a test server after I did the following on the Hobbit server:

defined a RRD graph section in hobbitgraph.cfg for all probes available on that server
NCV_temps='*:GAUGE" in hobbitserver.cfg
appended temps=ncv to test2rrd line in hobbitserver.cfg
restart hobbitd server

When I deployed the same extension code to a different server, the limitation of such an extension became painfully obvious: the graph definition failed when a different server reports more or less probes. Since hobbitgraph.cfg's graph section is keyed to the test name in hobbitgraph.cfg, I can't add custom graph section per server configuration, not to mention it is not scalable!. As for alerting, I tried setting up thresholds on the server. However, it didn't seem to generate any alert even if the threshold is obviously surpassed. wonder if the 'status blah GREEN' reported by the extension script kinda blocked such centralized checking.

After posting the above questions to Hobbit's mailing list and got no answers, I took upon myself to find the answers. Good thing that Hobbit Monitor is licensed using GPLv1. I found the answers in the source code, rrd/do_temperature.c. [[ All hail goes to Open Source & Henrik! ]]

the 'temperature' test is built-in. duh...
if 'temperature' is used as 'test name' on the client, nothing needs to be changed on the server end.
a lump-all status command will do. Format of the data portion is somewhat restrictive. Each probe needs to be in the format of '&green BMCambient 17 62'.
the test can be done locally on the client and the overall $status is reported by 'status server1.temperature $status'.

Wishes:

It'd be nice to have such data report format documented elsewhere other than the source code.
The current code (4.2RC1-20060712) didn't take input other than integer too well. It confused the hell out of it, in fact, to the extent it lump the whole long after $status as the probe name. Most of problems I experienced was actually the '.' in my data report confused Hobbit.
It'd be nice to be able to specify threshold on the server centrally.

How to coerce OMSA 5.1 to install & run properly on CentOS 4 (part II)

2006-12-13T16:50:00.000-05:00

Not so fast! The same fix doesn't work on CentOS 4.1 running on a Dell PE6850. srvadmin-install.sh somehow determined it needed a newer version of OpenIPMI from its own package, regardless of whether the OS has latest OpenIPMI installed or not. Recall that the same hack worked w/o OpenIPMI installed at all on CentOS 4.4.

# Here is the excerpt from srvadmin-install.sh
Server Administrator requires a newer version of the OpenIPMI driver modules
for the running kernel than are currently installed on the system. The OpenIPMI
driver modules for the running kernel will be upgraded by installing an
OpenIPMI driver RPM.
warning: RPMS/supportRPMS/openipmi-33.13.RHEL4-1dkms.noarch.rpm: V3 DSA signature: NOKEY, key ID 23b66a9d
Preparing... ########################################### [100%]
1:openipmi ########################################### [100%]

Loading kernel module source and prebuilt module binaries (if any)
Installing prebuilt kernel module binaries (if any)

Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.

Your DKMS tree now includes:
openipmi, 33.13.RHEL4, 2.6.9-5.EL, i686: built
openipmi, 33.13.RHEL4, 2.6.9-5.EL, x86_64: built
openipmi, 33.13.RHEL4, 2.6.9-5.ELsmp, i686: built
openipmi, 33.13.RHEL4, 2.6.9-5.ELsmp, x86_64: built
openipmi, 33.13.RHEL4, 2.6.9-5.ELhugemem, i686: built

The OpenIPMI packages is from the CentOS 4.4 release DVD. So, it couldn't be newer !
# rpm -aq|grep -i OpenIPMI
OpenIPMI-libs-1.4.14-1.4E.13
OpenIPMI-1.4.14-1.4E.13

All our production servers don't have kernel source or development kits available. So, dkms approach isn't the one we'd take easily if we have other viable alternatives. So, I modified srvadmin-openipmi.sh to force it not to upgrade IPMI.
# diff srvadmin-openipmi.sh.orig srvadmin-openipmi.sh
1055c1055,1056
< loc_recommended_action="$?
> #LOC_RECOMMENDED_ACTION=$?
> LOC_RECOMMENDED_ACTION=4

This worked for the installation. However, upon start-up the services, the IPMI services failed to start. I ran into problem before with OMSA's own ipmi init script (dsm_sa_ipmi) deleting /dev/ipmi0 while udev never puts it back. So, as a simple hack, I just added a line to add the device at the beginning of the script.
test -e /dev/ipmi0 || mknod -m 0600 /dev/ipmi0 c 253 0

After this, /etc/init.d/dsm_sa_ipmi would start ok, w/o OpenIPMI package installed. However, when it is called from srvadmin-services.sh (I suspect instsvcdrv is the actually caller). It failed each every time, causing omreport could't get any data. For now, I can manually start dsm_sa_ipmi and then start/stop srvadmin-services.sh just fine.

What makes worse (maybe even the cause of this IPMI start-up problem) is that the init sequence numbers assigned to all these init scripts are S50, instead of S00(mptctl), S96(instsvcdrv), and S97 for all the rest. I don't think this will work upon reboot or run level changes. I fixed it manually per chkconfig directives inside the scripts. However, I am curious why chkconfig got confused.

P.S. A more complete tweak would be at beginning of the function in srvadmin-openipmi.sh. The previous tweak on srvadmin-openipmi.sh works only when you have OpenIPMI installed.

# diff srvadmin-openipmi.sh.orig srvadmin-openipmi.sh
973a974,976
> LOC_RECOMMENDED_ACTION=4
> return ${LOC_RECOMMENDED_ACTION}

How to coerce OMSA 5.1 to install & run properly on CentOS 4

2006-12-13T16:45:00.000-05:00

To monitor Dell PowerEdge servers running Linux, I downloaded OMSA (Open Manage Server Assistant) 5.1 package for Redhat Enterprise Linux 4 (RHEL4) on a bunch of Dell PowerEdge 6850 servers. The package was listed under 'system management' of the 'drivers & download' section. The actual file name is 'OM_5.1_ManNode_LIN_A00.tar.gz'.

On the PE6850 server, I unpacked the tarball, and attempted to install using the srvadmin-install.sh. It failed silently, with an error code of 2.
# supportscripts/srvadmin-install.sh
# echo $?
2
Examing srvadmin-install.sh closely, I realized that it didn't support CentOS 4. Instead, it supports & detects RHEL4 by its code node 'Nahant'. Instead of mucking the code to support CentOS, I decided to fake RHEL4 by modifying /etc/redhat-release.
# cat /etc/redhat-release
fake RHEL 4 Nahant
It worked.The installation went a-ok and all services started properly. 'omreport storage controllers' now reports various information on various storage controllers.

Thinking the faking was needed only by the installation scripts, I reverted /etc/redhat-release to its original version. To be sure, I restarted OMSA by ' srvadmin-services.sh stop|start'. Well, it didn't pan out that well.

# /usr/bin/srvadmin-services.sh start
Starting mptctl:
Waiting for mptctl driver registration to complete:
[ OK ] Starting Systems Management Device Drivers:
Starting dell_rbu: [ OK ]
Starting ipmi driver: [FAILED]
Starting Systems Management Device Drivers:
Starting dell_rbu: Already started [ OK ]
Starting ipmi driver: [FAILED]
Starting DSM SA Shared Services: [ OK ]

Starting DSM SA Connection Service: [ OK ]

Now it seemed the srvadmin-ipmi package's init script still relies on /etc/redhat-release to tell it which OS it is running on. The server doesn't have OpenIPMI installed. Without any IPMI driver loaded, no report can be run against storage or temps using OMSA.

To have it work for both world (OMSA and CentOS), I cooked up this /etc/redhat-release by appending the RHEL4 code name to the original /etc/redhat-release
# cat /etc/redhat-release.fake
CentOS release 4.4 (Final) Nahant
# /usr/bin/srvadmin-services.sh start
Starting mptctl:
Waiting for mptctl driver registration to complete: [ OK ] Starting Systems Management Device Drivers:
Starting dell_rbu: [ OK ]
Starting ipmi driver: [ OK ]
Starting Systems Management Data Engine:
Starting dsm_sa_datamgr32d: [ OK ]
Starting dsm_sa_eventmgr32d: [ OK ]
Starting DSM SA Shared Services: [ OK ]
Starting DSM SA Connection Service: [ OK ]

This cooked /etc/redhat-release had been verified to work well with 'yum -y update' and OMSA's 'srvadmin-services.sh start'

trick to enable non-bridged wireless access on a Netgear Wireless router (WGR614v6)

2006-12-06T17:58:00.000-05:00

I bought a Netgear wireless router (WGR614v6) from CompUSA yesterday. It is going to a customer's site for WI-FI blackberries (BB7270). I need a wireless router that can simply

obtain its own TCP/IP via DHCP from a network port
self-contain its WLAN: NAT/SPI protect all wi-fi devices

I would be happier with a Netgear MR814v3, which we have in the lab and is known to work well with BB7270. All the bugs and glitches we ran into in the past year turned out to be on RIM's end, instead of the "poor" quality perceived by RIM's engineer of consumer-grade wireless access points.

Followed the auto-run wizard from the installation CD to the letter, I connected my own laptop to one of the LAN ports on the unit, with the unit's own Internet port plugged into the wall jacket in my office. Once I did a 'ipconfig /renew' on my laptop, the laptop got TCP/IP settings via DHCP as promised. It is a bit odd to see the DNS server is 192.168.1.1. This means the Netgear unit proxies DNS queries for devices utilizing its DHCP/NAT services. However, two things do not work right:

nslookup anything resolves to 192.168.1.1 ?
the LED on the front of the unit is not on for the WI-FI? 'Router status' says 'wireless :: off', while the checkbox to turn on 'wireless router radio' was shadowed out!

Searching on Netgear's site, I was relieved/disappointed to find the firmware was up to date. There's a link from WGR614's support page on how to enable the unit as a wireless access point. However, such enabling is to bridge the WLAN to the local LAN. I attempted that to satisfy my curiosity. It requires to use a regular LAN port as the autosense uplink instead of the separate Internet port on the unit. Once I did it, the laptop and a wifi blackberry were bridged to the real Ethernet LAN and got their TCP/IP settings from the real corporate DHCP server instead of from the unit's own DHCP server.

Lo & behold, everything went peachy all of a sudden, after this diversion. The checkbox to turn on wireless router radio option is now available, even if after I moved the cable from regular LAN port to Internet port on the unit. The wireless icon on the unit is blinking green. Power cycle the unit didn't eliminate such capability.

I am a bit at loss why wireless capability was disabled before and how it could be enabled by the steps I took. One thing I should have done at the beginning, is to reset it to factory default, just in case someone has messed it up somehow. The unit has all its seals and should be a new one. Anyhow, I am glad it worked out. However, I can't imagine any non-techie can get this far w/o picking up the phone or returning it to the retailer.

One thing to note is that WPA-PSK worked as advertised, both on the access point and on the BB7270. A lengthy 'mydoaminDS-24random' was used for SSID and the maximum allowed '63RANDOM' as the PSK (Pre-shared key). I couldn't imagine that I can type them in w/o a typo, so I emailed such lengthy secret to the email account currently assigned to my test BB7270. On the BB7270, I copy+paste these lengthy random bits from email to a new WLAN profile creation form under Options/WLAN. It worked great. With older version of BB7270 firmware, copy+paste WEP key fails each every time. I'd assume the bug was fixed by now (from v4.0.1.84 to v.4.0.1.104). On the BES (Blackberry Enterprise Solution) server, a site-specific "IT Policy" got created on the using these WLAN secrets worked great too. However, the SSID somehow took several kicks to get really sucked into the "IT policy". The BBM (Blackberry Manager) didn't complain when SSID didn't take, resulting that a BB7270 provisioned from scratch after nuking didn't carry any WLAN profile at all. Only upon reviewing the IT policy, I was shocked to find SSID field was blank while everything else under the WLAN policy section looked good.

Garmin StreetPilot C320 :: review by Experience

2006-11-27T20:21:00.000-05:00

I bought a new Garmin StreetPilot C320 gps right before my trip to Florida over the thanksgiving weekend.
Pros: It worked for most of the part.
Cons:

The online registration to unlock the map was extra hassle I didn't expected.
Too much other junk in the map causing a 128M SD card hold no more than three states at a time. I was told some other brand has less than 4M a state.
Reception is poor. It failed to make its first round of contacts with satellites even if I placed it on the picnic table on the back deck or the front patio stairs. One time on interstate, it failed too, when there's a 16-wheeler nearby. Consider the radio still works, you'd think they should augment/supplement the signal somehow to make it work too. Otherwise, you simply can't drive in a crowded traffic and expect it to guide you.

Here is the other parts when it failed to work or could use some improvements:

In St Augustine Beach, FL, I was directed to turn left to Mandrid St for my destination from SR A1A South. It turned out Mandrid street was blocked by a private gate.
In Atlanta, GA, I was instructed to turn left in 4 miles off 285 to my house. Nothing was mentioned about a right turn (T-junction) from Ashford-Dunwoody Rd to Mt. Vernon Rd.
In Kissimmee, FL, I was directed to a Super Wal-Mart 3.2miles away from the resort we stayed in instead of the one 1.2 miles away. Apparently both had been there for quite few years already.
In St. Augustine, FL, we wanted to go to a Thai restaurant. However, the unit complained it couldn't find '4900 SR US 1". By searching 'shopping' on the unit, however, we located a Publix at '1127 SR US 1'. It has been more than one occasions when it claimed no such address can be found even though the whole FL state is loaded.
Driving back from Orlando, FL to Atlanta, GA, it said turn in 310 miles when we paid $2.50 at the 2nd toll plaza. I figured $2.50 is cheap for a 300-mile toll road. It turned out it failed to mention the merge from Florida turnpike to I-75 N in 27 miles.

The extra junk turned out to be very handy when we were in Orlando

To find a super Wal-Mart to buy a special brand of organic milk for my son, I just selected 'shopping' and spotted the first super Wal-Mart, as the list is sorted by distance from your current location. We left our laptops home on purpose even though places we stayed have free WI-FI. Without the GPS, I guess we'd have to drive around to spot one on the run. We found a CVS and a Publix the same way.
We had tickets for Downtown Disney. However, both the tickets and brochures don't list street address. I guess the smart Disney writers assumed everybody know where it is ?! The unit listed it under 'theme parks' as 'Disney Quest'. That solved our problem pretty easily.

All in all, it eliminated quite some headache for my wife and me.

perc 4e/Di on Dell PE6850 saga continues...part C

2006-11-10T19:55:00.000-05:00

With the load from the full database dump plus the application load burst we set up last Friday , the problematic server 'syb04' generated a few alerts over the weekend. The alerts complained the stamp didn't show up right after the 'logger' call. We were very excited, thinking we were able to reproduce the problem this quick. The next thing would be just to pick out what to upgrade from a decent list of potential upgrades.

Examining closely the local log as well as the log on the remote syslogd server, however, showed that such 'missing' stamps appeared up right after the complaints of their absence. Most were within the same second, and only one was one second late. Therefore such alerts were identified as false positives.

Needless to say, we were very disappointed. Even worse, this remained the case for the full week. We started to toss around the idea maybe the hiccup was merely a delay and we overacted a little by flipping the switch.

To "add insult to the injury", our constant attention was demanded by a lot of database problems related to application peak load which was coerced to repeat. The problems were:

Sybase ASE log device filled up, causing the application peak load come to a sudden halt, until Sybase is restarted with log cleared.
Hourly transaction has grown from 20M each to over 1G each. It seemed like some transaction failed to be committed.
In turn the transaction dumps filled up the disk.

So far, the server endured seven 24-hour days of load run, which totals 24x7x3= $504 load peaks. A regular day have only 2 peaks, therefore, this equals to 250 days worth of load.

I am betting a small sum of money on the PR (patrol read), whose background scheduling may be surprised by the sudden spike in disk IO caused by the nightly full database backup as well as the daily application peak. To force PR to collide with the load, I wrote a script to check PR status and start one if none is 'In Progress' already, as reported by 'megapr -dispPR -a0'.

BTW, the 'megapr -dispPR -a0' command alone causes the following errors in PERC controller's exportlog. My inquiry on this error got no response from Dell's linux-PowerEdge forum, which is monitored by a few Dell engineers.
11/07 10:25:51: MPT_Rec: INQ Error - Negotiating LD[6] pRfm a07517c0
11/07 10:25:51: MPT_Rec: INQ Error - Negotiating LD[16] pRfm a0743360
11/07 10:25:51: GET: SCSI_chn=ff, rtn status=0

debugfs :: handy utility to debug an ext2 or ext3 file system

2006-11-07T12:16:00.000-05:00

Today I am learning a useful utility program named 'debugfs'. It is part of e2fsprogs package, an essential package containing axillary programs for ext2 and ext3 file system under Linux.

For a regular file, 'debugfs' can help you find an inode by any data block the file or dir entry is using. Then you can turn around and ask for the name of the inode. This could be handy when some mysterious files causing df and du to disagree whether the filie system is full, or the file system is corrupted or can't mounted to be accessed as usual. More advanced file system features are available too.

# to find what inode is claiming a given data block
# debugfs -R "icheck 12345" /dev/hda1
debugfs 1.35 (28-Feb-2004) Block Inode number 12345 340

# to find the file name given the inode number
# debugfs -R "ncheck 49153" /dev/hda1 debugfs 1.35 (28-Feb-2004) Inode Pathname
49153 /usr/share/locale/ar/LC_MESSAGES/libbonobo-2.0.mo

# Print the location of the inode data structure
# debugfs -R "imap /boot/vmlinuz-2.6.9-42.0.2.EL" /dev/hda1 debugfs 1.35 (28-Feb-2004)
Inode 557516 is part of block group 34
located at block 1114128, offset 0x0580

# to dump the direntry (filespec, per man page)
debugfs -R "dump -p /boot/vmlinuz- 2.6.9-42.0.2.EL /tmp/vmlinuz_dumped" /dev/hda1
# md5sum /boot/vmlinuz-2.6.9-42.0.2.EL /tmp/vmlinuz_dumped e5c536b539b5ffcaa03b22bd7fcc164a /boot/vmlinuz-2.6.9-42.0.2.EL e5c536b539b5ffcaa03b22bd7fcc164a /tmp/vmlinuz_dumped

# to get the contents of a file, assume the fs can't be mounted and accessed the usually way.
# debugfs -R "cat /etc/redhat-release" /dev/hda1
debugfs 1.35 (28-Feb-2004) CentOS release 4.4 (Final)

Noteworthy is, for files under /selinux ( a pseudo fs), it can find inode number associated with a data block. However, it couldn't find the file name for the very inode number.
# debugfs -R "ncheck 8" /dev/hda1 debugfs 1.35 (28-Feb-2004)
Inode Pathname 8
# find / -inum 8
/selinux/relabel
# ls -id /selinux/relabel
8 /selinux/relabel
# debugfs -R "icheck 4567" /dev/hda1 debugfs 1.35 (28-Feb-2004) Block Inode number
4567 8
# / is on /dev/hda1
/dev/hda1 8127400 6738524 1306308 84% /

There are a lot of powerful (and dangerous) features such as

feature you can set or clear various file system features in the superblock
freeb to mark data blocks as unallocated vs. setb
freei to free the inode specified
clri to clear the contents of the inode
chroot to chroot to the directory
find_free_block
find_free_inode
init_filesys to create an ext2 file system
kill_file deallocate the file and its blocks. It doesn't remove any direntry to this inode. not ' rm' or 'unlink'.
logdump to dump the ext3 journal
modify_inode modify the contents of the inode structure
ls/mkdir/mknod/rm/rmdir

'debugfs' starts interactively by default, unless you have '-R' to request one-time use only. A session would be like below:
# debugfs
debugfs 1.35 (28-Feb-2004)
debugfs: open /dev/hda1
debugfs: icheck 12345
Block Inode number
12345 340
debugfs: ncheck 340
Inode Pathname
340 /usr/X11R6/lib/xscreensaver/mountain
debugfs: close
debugfs: quit

AUUG November meeting :: Legal Concerns on Blogger & GPLv3

2006-11-07T09:33:00.001-05:00

7:26pm last night, I finally made my way into the HP building across I-285 from Perimeter Mall. I couldn't find the front entry with some contruction blocked some part of the road, so I pulled in wherever I can into the only standing-tall building. A few employees passed me from the next lane and were kind enough to roll down their windows to inform me that a badge is required for after-hour access. I buzzed the security guy. Once I told him that I was here for the AUUG meeting, he attempted to buzz me in a few times, to no avail. So, he dispatched someone to come down to get it for me. While I was parking my car, he was standing there holding the door for me. I am really touched.
The presentation is on legal considerations for Bloggers and legal implications of new provisions in GPLv3. It is well done by two lawyers from Manning, Morris, and Somebody, a local law firm and one of the corporate sponsors of AUUG. I knew Linus was not a fan of GPLv3 because of some of the restrictive clauses. It is much clear made the presenter, the motivation behind it is some kinda vendetta against commercialism of some sort by an academic and a pure idealist.

Let me quote below the three most problematic clauses in GPLv3, as discussed in the presentation

patent retaliation
bundle rights
no locked key [ The lawyer's own research ]

perc 4e/Di on Dell PE6850 saga continues...part B

2006-11-04T11:22:00.001-05:00

After searching up & down, I compiled a decent list of potential upgrades and toggles to try out on syb04. None of them is apparently pertinent enough to have you say 'ahh-ha'. I purchased and put into production a new server named syb06, the one killed by oom-killer and cured by kernel-hugemem. With the improved production configuration mix, time is more affordable than last time when syb04 locked up. So, our team of 'experts' decided to reproduce the recent hiccup or the older lockup problem reliably before we attempt a fix this time around.

Hobbit Monitor is running all the servers, so it is rather easy to catch the old lockup problem wherein all checks went to 'purple', as in 'stale', or 'no report received' status. It is a bit tricky to detect when a hiccup happens. If it happens squarely inside the 5-minute interval Hobbit Monitor uses, we'd miss the signal! It seems it is not all that easy to change monitor frequency down to 1 minute for one single client, as nobody has answered my question on the Hobbit mailing list for three days now. After much discussion of alternatives, I come up with a way and verified it works.

With the monitor fine-tuned and focused on syb04, load is added to it first. Count full nightly database backup and daily peak as two load situations, we need have at least 28 peaks to equate to the 14 days leading up to the lockup. The nightly database backup takes only 25 minutes, and is very easy to run it continuously by simply changing cron schedule to every 30 minutes instead of every day. So, we did that. After 20 hours (~= 40 load peaks), nothing happened. Since we don't plan to work over the weekend, it is decided to simulate the daily load peak and let it run continuously. It took some Java code change and it is done. So, we'd have both the application load and the backup load against the server over the weekend.

* fingers-crossed *

Maxtor Personal Storage 3200 320 GB external USB 2.0 drive to stage backups under Linux

2006-11-04T11:20:00.001-05:00

Today I bought a Maxtor Personal Storage 3200 320 GB (Model U01H320) from Circuit City. 320G with 8M cache for merely $159. They matched their online price by taking $20 off the $179 in-store price. Amazon.com is selling the same item on behalf of CompUSA for $169.
The drive will be used to stage backups at our co-lo. At work, we've waited too long for a budget to come through for a real NAS or a tape library/magzine/autoloader. It is one of those day-to-day challenges of small businesses which face the same challenges with much more limited resources. The daily full backup set now amounts to 12G, all compressed by gzip, my favorite tool.

From a RHEL 4.4 AS (CentOS 4.4, to be more exact) guest inside VMWare, the drive was detected as USB 1.1 (full speed) instead of USB 2.0. I assume it is a limitation of the VMWare emulation.
usb 1-1: new full speed USB device using address 2
Initializing USB Mass Storage driver...
scsi1 : SCSI emulation for USB Mass Storage devices
Vendor: Maxtor Model: 3200 Rev: 0341
Type: Direct-Access ANSI SCSI revision: 02 USB Mass Storage device found at 2
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB) sda: assuming drive cache: write through
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB) sda: assuming drive cache: write through
sda: sda1 Attached scsi disk sda at scsi1, channel 0, id 0, lun 0

Pretty interesting to see it actually has NTFS as partition type. wonder how it would fly with MacOS out of the box?
Disk /dev/sda: 320.0 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 38913 312568641 7 HPFS/NTFS
Since I am gonna use it under RHEL 4, I go ahead relabel the partition as 'Linux' and formatted it as EXT3.
/root# mke2fs -L E320G -O sparse_super,dir_index,filetype -T largefile4 -j /dev/sda1
mke2fs 1.35 (28-Feb-2004)
Filesystem label=E320G
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
76320 inodes, 78142160 blocks
3907108 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=79691776 2385 block groups
32768 blocks per group, 32768 fragments per group
32 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616 Writing inode tables: done
Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Ahh, I always forgot to reduce the percentage reserved for super user from the default 5% to 1%.
/dev/sda1 312538740 98368 296811940 1% /mnt
/root# tune2fs -m 1 /dev/sda1
tune2fs 1.35 (28-Feb-2004)
Setting reserved blocks percentage to 1 (781421 blocks)
/root# df -kv /mnt
Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 312538740 98368 309314688 1% /mnt

Comcast connection problem (Motorolla SF4100) for recent two weeks

2006-11-03T17:39:00.001-05:00

I have been having connection problem with Comcast connection problem (Motorola SF4100) for the past two weeks. The speed was not stable at all and may even lost connection. Why the broadband ISP can't provide a robust service just running water or electricity? Is it because
we as customers are not that demanding

we fail to scream at them for each every outage or service degradation
we fail to find time to talk to our legislators
we fail to fail to vote with our feet
we fail to penalize them even within the current framework. If everyone can just report the problem, ask for a ticket be opened, calling back to Billing with the ticket number to get a refund! This would help in two fronts

the sheer volume of calls/tickets would open some eyes when BI or data warehouse report is in
the sheer volume of the refund check would cut into the profit margin
the faster and more we complain, the sooner they stop using us as a measuring gauge whether the service is up or not

For the least, we should demand a registration be passed so that refund for system-wide outages be automated and to all customer base, instead of the current more-hoopla-for-you scheme: refund is only available for those who went through the trouble reporting the problem, asking for a ticket, calling back to Billing with the ticket number to get a teeny-weeny refund.

Here is the symptoms of the week:

power-cycle the cable modem and/or wireless router may or may not help.
download speed ranges from 200K to .4KBps to a stall. several times within a hour.
Web interface of the cable modem itself shows the following error/warnings. I looked at it when it works properly, it should be all Debug/Informational.
When I asked 'router' to 'renew' address, it failed to renew for over a dozens of retries and timed out.
Speed test via http://www.speakeasy.net/speedtest/ shows the connection is not stable at all: 600KB/14KB at one point, and 200KB/104KB at another. It really should stay around 700KB/80KB.

061103155029	8-Debug	F504.1	Bridge Ethernet Hook. Failed to learn CPE MAC Address.
061103155029	4-Error	F507.5	MAC Filters. Add MAC Address can't add entry. Table is full.
061103155019	8-Debug	M570.2	Motorola CM certificate present
061103155019	8-Debug	M571.7	CM Cert Upgrade Enabled. Initiate after Registration
061103155019	8-Debug	I503.0	Cable Modem is OPERATIONAL
061103155019	7-Information	B401.0	Authorized
061103155018	8-Debug	F502.1	Bridge Forwarding Enabled.
061103155018	8-Debug	F502.3	Bridge Learning Enabled.
061103155018	7-Information	B0.0	Baseline Privacy
061103155016	7-Information	X518.9	Configuration - GGFMMD - Unit Update Enabled by CVC
061103155016	8-Debug	I500.1	DOCSIS 1.0 Registration Completed
061103155016	7-Information	I500.4	Attempting DOCSIS 1.0 Registration
061103155016	7-Information	D509.0	Retrieved TFTP Config File SUCCESS
061103155013	7-Information	D507.0	Retrieved Time....... SUCCESS
061103155013	7-Information	D511.0	Retrieved DHCP .......... SUCCESS
061103155013	5-Warning	D3.0	DHCP WARNING - Non-critical field invalid in response
061103155013	4-Error	D530.8	DHCP - Invalid Log Server IP Address.
061103155013	5-Warning	D520.2	DHCP Attempt# 6 BkOff: 5s Tot DSC:6 OFF:3 REQ:3 ACK:1
061103155013	3-Critical	D1.0	DHCP FAILED - Discover sent, no offer received
061103154959	5-Warning	D520.2	DHCP Attempt# 4 BkOff:27s Tot DSC:4 OFF:2 REQ:2 ACK:0
061103154959	3-Critical	D1.0	DHCP FAILED - Discover sent, no offer received
061103154932	5-Warning	D520.2	DHCP Attempt# 3 BkOff:13s Tot DSC:3 OFF:2 REQ:2 ACK:0
061103154932	3-Critical	D2.0	DHCP FAILED - Request sent, No response
061103154927	5-Warning	D520.2	DHCP Attempt# 2 BkOff: 4s Tot DSC:2 OFF:1 REQ:1 ACK:0
061103154927	3-Critical	D1.0	DHCP FAILED - Discover sent, no offer received
061103154923	5-Warning	D520.2	DHCP Attempt# 1 BkOff: 4s Tot DSC:1 OFF:1 REQ:1 ACK:0
061103154923	3-Critical	D2.0	DHCP FAILED - Request sent, No response
061103154918	7-Information	D0.0	DHCP CM Net Configuration download and Time of Day
061103154918	7-Information	T500.0	Acquired Upstream .......... SUCCESS
061103154918	8-Debug	T503.1	Acquire US with status OK, powerLevel 19, tempSid 1378
061103154918	8-Debug	T505.0	Acquired Upstream with status OK
061103154916	7-Information	T501.0	Acquired Downstream (687000000 Hz)........ SUCCESS
061103154916	8-Debug	T509.0	Acquired DS with status OK, DS Freq 687000000, US Id 5
061103154906	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154905	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 105000000, US Id 0
061103154905	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154905	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 99000000, US Id 0
061103154905	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154905	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 93000000, US Id 0
061103154905	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154904	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 855000000, US Id 0
061103154904	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154904	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 849000000, US Id 0
061103154904	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154903	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 843000000, US Id 0
061103154903	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154903	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 837000000, US Id 0
061103154903	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154902	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 831000000, US Id 0
061103154902	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154902	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 825000000, US Id 0
061103154902	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154901	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 819000000, US Id 0
061103154901	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154901	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 813000000, US Id 0
061103154901	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154900	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 807000000, US Id 0
061103154900	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154900	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 801000000, US Id 0
061103154900	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154859	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 795000000, US Id 0
061103154859	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154859	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 789000000, US Id 0
061103154859	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154859	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 783000000, US Id 0
061103154859	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154858	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 777000000, US Id 0
061103154858	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154858	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 771000000, US Id 0
061103154858	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154857	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 765000000, US Id 0
061103154857	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154857	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 759000000, US Id 0
061103154857	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154856	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 753000000, US Id 0
061103154856	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154856	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 747000000, US Id 0
061103154856	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154855	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 741000000, US Id 0
061103154855	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154855	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 735000000, US Id 0
061103154855	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154855	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 729000000, US Id 0
061103154855	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154854	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 723000000, US Id 0
061103154854	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154854	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 717000000, US Id 0
061103154854	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154853	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 711000000, US Id 0
061103154853	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154853	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 705000000, US Id 0
061103154853	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154852	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 699000000, US Id 0
061103154852	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154852	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 693000000, US Id 0
061103154852	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154851	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 687000000, US Id 0
061103154851	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154851	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 681000000, US Id 0
061103154851	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154850	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 675000000, US Id 0
061103154850	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154850	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 669000000, US Id 0
061103154850	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154850	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 663000000, US Id 0
061103154850	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154849	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 657000000, US Id 0
061103154849	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154849	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 651000000, US Id 0
061103154849	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154848	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 645000000, US Id 0
061103154848	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154848	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 639000000, US Id 0
061103154848	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154847	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 633000000, US Id 0
061103154847	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154847	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 627000000, US Id 0
061103154847	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154847	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 621000000, US Id 0
061103154847	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154846	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 615000000, US Id 0
061103154846	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154846	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 609000000, US Id 0
061103154846	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154845	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 603000000, US Id 0
061103154845	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154845	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 597000000, US Id 0
061103154845	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154845	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 591000000, US Id 0
061103154845	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154844	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 585000000, US Id 0
061103154844	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154844	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 579000000, US Id 0
061103154844	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154843	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 573000000, US Id 0
061103154843	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154843	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 567000000, US Id 0
061103154843	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154842	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 561000000, US Id 0
061103154842	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154842	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 555000000, US Id 0
061103154842	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154842	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 549000000, US Id 0
061103154842	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154841	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 543000000, US Id 0
061103154841	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154841	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 537000000, US Id 0
061103154841	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154840	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 531000000, US Id 0
061103154840	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154840	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 525000000, US Id 0
061103154840	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154840	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 519000000, US Id 0
061103154840	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154839	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 513000000, US Id 0
061103154839	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154839	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 507000000, US Id 0
061103154839	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154838	8-Debug	T509.0	Acquired DS with status NO FEC lock, DS Freq 501000000, US Id 0
061103154838	3-Critical	T2.0	SYNC Timing Synchronization failure - Failed to acquire FEC framing
061103154837	7-Information	D519.0	DHCP Client shutting down.
061103154837	7-Information	H501.2	HFC: Shutting Downstream Down
061103154837	3-Critical	I2.0	REG RSP not received
061103154837	1-Emergency	I506.0
061103154828	7-Information	I500.4	Attempting DOCSIS 1.0 Registration
061103154828	7-Information	D509.0	Retrieved TFTP Config File SUCCESS
061103154828	7-Information	D507.0	Retrieved Time....... SUCCESS
************	7-Information	D511.0	Retrieved DHCP .......... SUCCESS
************	5-Warning	D3.0	DHCP WARNING - Non-critical field invalid in response
************	4-Error	D530.8	DHCP - Invalid Log Server IP Address.
************	5-Warning	D520.2	DHCP Attempt# 1 BkOff: 5s Tot DSC:1 OFF:1 REQ:1 ACK:1
************	7-Information	D0.0	DHCP CM Net Configuration download and Time of Day
************	7-Information	T500.0	Acquired Upstream .......... SUCCESS

perc 4e/Di on Dell PE6850 saga continues...part A

2006-11-01T20:19:00.000-05:00

We ended up applying BIOS upgrade (A00->A01) and PERC 4e/Di firmware upgrade (521A to 522A A13) for the system lockup problems we had on the production database server running on a Dell PE6850. Home-made load tests didn't cause panic for 18 hours. The server was then rushed back into production since the fail-over spare server couldn't stand the load.

The server (the Sybase database engines) has been up for 14 days today. At 09:50am, just when the server started to ramp up to its daily load peak (CPU load ~=4) , some processes failed to write to the disk and 'date > junk' from cmdline just hang there. I canceled that 'date>junk'. All is good after less than 4 minutes. Nothing interesting (warn/error/abort) in the system log, exportlog from PERC controller, or database log. PR was running at the time.

The symptoms definitely differ, so the BIOS and firmware upgrade did make some difference towards the better. For the previous two lockups and the only two for 15 months, we lost access to the disks totally, getting "reject i/o to offlined disk" without kernel panic or corruption. This time, this is merely a hiccup or pause or suspension of sorts.

Older postings on similar topic on dell-linux-poweredge forum suggested PR could be the culprit if BIOS/firmware is up-to-date. On the system, I get the following output from '"megapr -dispPR -a0" today. Is #Iterations current count of the total PR has run or a threshold or some sort? If the former, how to clear it? If the latter, how to increase? Basically I am looking into why it locked up exactly 30 days (could be coincidence too. and we are now using newer BIOS and firmware). Dell diag from OMSA 4.4 on 10/17/2006 suggests nothing wrong the controller, memory, or underlying disks. (omreport on the controller is appended below too).

********PR INFO********
Mode :AUTO
#Iterations:2200
Status :PR In Progress

# omreport storage controller
Controller PERC 4e/Di (Embedded)

Controllers
ID

    : 0
Status                            : Ok
Name                              : PERC 4e/Di
Slot ID                           : Embedded
State                             : Ready
Firmware Version                  : 522A
Driver Version                    : Not Applicable
Minimum Required Firmware Version : Not Applicable
Minimum Required Driver Version   : Not Applicable
Number of Channels                : 2
Rebuild Rate                      : 30%
Alarm State                       : Not Applicable
Cluster Mode                      : Not Applicable
SCSI Initiator ID                 : 7

Also, we upgraded the BIOS from A00 to A01, instead of to the latest A04, since the release notes of A02 through A04 didn't read pertinent at the time. At second read of A03's release notes, I noticed the following two fixes that could be relevant to the system. Where can I find more detailed notes other than PE6850-BIOSA03.TXT ? I don't quite understand why the developers or release managers so minced on words.

Added support for Virtualization Technology in the processor.

Should I assume this is not referring to HT, but of special server virtualization assistance from Intel's VT (?) technology or alike ?

Added support for 800MHz system configurations.

Does this mean BIOS prior to A03 doesn't support 800MHZ system configurations?

Although the megaraid* driver is dated early 2005. The CHANGLOG.megraid in /kernel/Documentation doesn't have much interesting changes either.

Fedora Core 6 security improvement :: refined SELinux policy managment and more

2006-10-31T21:02:00.001-05:00

After a stock installation of the Fedora Core Linux 6, I am pleasantly surprised at the first boot, with the SELinux policy management GUI at part of the 'firstboot' program. You can collapse to set granular policies for services of your interest. For each service, all you need to do is to check or to uncheck policy items .Such a nice GUI management would definitely help greatly to unleash the raw power of SELinux to system administrators or even security administrators who would otherwise have to spend much more time to study, research, and to manage SELinux policy effectively.

The same SELinux policy management tool is readily accessible under System-> Administration->Security Level and Firewall. I guess the name of the applet has not really caught up with this new integration.

Of course, a system won't be secure unless it can be kept up-to-date with all good security patches and updates. In that regard, FC6 is improved as well. The update notification applet is improved to make the update process point-and-click compliant. You will get Package Updater running all GUI to assist regular desktop users to keep their system up to date.

kernel panic when using a 2G ramdisk on a CentOS 4.4 serverCD installation

2006-10-31T09:31:00.001-05:00

I was burnt recently with a minimal installation using CentOS 4.4 ServerCD. CentOS project has this one-CD flavor for server use in addition to the regular current 4-CD full distro, mirroring Redhat Enterprise Advanced Server (RHAS). I found it awfully convenient to download only one CD of 530M, instead of the full set of 2.4G. The installations I do at work are all server installations, if my own Linux desktop is not counted. It has been a god-given ever since I first noticed its existence in CentOS 4.2 /iso folder.

Kernel paniced for a new production Sybase ASE database server when Sybase ASE 12.5 database engines started to zero out ~2G tmpdb devices (data & log devices). The hardware specs: Dell PowerEdge 6850 with 16G of DDR2 RAM and quad CPUs. The box survived my own stress test on CPU and disk. However, as an hindsight, no attempt was made to exhaust the memory.

The Sybase seemed to have finished its task, when the DBA called me saying he lost connection to the machine. I attached to its serial console and opened a connection using Kermit. Input was not rejected, but no response (no echo) from the system. After a quick reboot, I found nothing interesting in the log tail around the panic ( syslog.conf is set to write kernel.* to /var/log/messages). So, I decided to log the console session and asked the DBA to start Sybase. Sure enough, it happened again, with tons of 'out of memory' messages, followed by desperate yet fatal attempts by oom-killer to kill all processes to free up memory.
oom-killer: gfp_mask=0xd0
Mem-info: DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
cpu 4 hot: low 2, high 6, batch 1
cpu 4 cold: low 0, high 2, batch 1
cpu 5 hot: low 2, high 6, batch 1
cpu 5 cold: low 0, high 2, batch 1
cpu 6 hot: low 2, high 6, batch 1
cpu 6 cold: low 0, high 2, batch 1
cpu 7 hot: low 2, high 6, batch 1
cpu 7 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
cpu 4 hot: low 32, high 96, batch 16
cpu 4 cold: low 0, high 32, batch 16
cpu 5 hot: low 32, high 96, batch 16
cpu 5 cold: low 0, high 32, batch 16
cpu 6 hot: low 32, high 96, batch 16
cpu 6 cold: low 0, high 32, batch 16
cpu 7 hot: low 32, high 96, batch 16
cpu 7 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 cpu 2 hot: low 32, high 96, batch 16 cpu 2 cold: low 0, high 32, batch 16 cpu 3 hot: low 32, high 96, batch 16 cpu 3 cold: low 0, high 32, batch 16 cpu 4 hot: low 32, high 96, batch 16 cpu 4 cold: low 0, high 32, batch 16 cpu 5 hot: low 32, high 96, batch 16 cpu 5 cold: low 0, high 32, batch 16 cpu 6 hot: low 32, high 96, batch 16 cpu 6 cold: low 0, high 32, batch 16 cpu 7 hot: low 32, high 96, batch 16 cpu 7 cold: low 0, high 32, batch 16 Free pages: 12506892kB (12493888kB HighMem) Active:208204 inactive:791964 dirty:101636 writeback:26 unstable:0 free:3126723 slab:20412 mapped:563179 pagetables:DMA free:12564kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4 all_unreclaimable? yes protections[]: 0 0 0 Normal free:440kB min:928kB low:1856kB high:2784kB active:644696kB inactive:16036kB present:901120kB pages_scanned:1069066 all_unreclaimable? yes protections[]: 0 0 0 HighMem free:12493888kB min:512kB low:1024kB high:1536kB active:180824kB inactive:3159244kB present:16908288kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 3*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12564kB Normal: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 440kB HighMem: 1418*4kB 535*8kB 128*16kB 85*32kB 119*64kB 38*128kB 10*256kB 4*512kB 2*1024kB 2*2048kB 3041*4096kB = 12493888kB Swap cache: add 0, delete 0, find 0/0, race 0+0 0 bounce buffer pages Free swap: 16779884kB 4456448 pages of RAM 3963834 pages of HIGHMEM 299626 reserved pages 849369 pages shared 0 pages swap cached Out of Memory: Killed process 6586 (dataserver). oom-killer: gfp_mask=0xd0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 cpu 2 hot: low 2, high 6, batch 1 cpu 2 cold: low 0, high 2, batch 1 cpu 3 hot: low 2, high 6, batch 1 cpu 3 cold: low 0, high 2, batch 1 cpu 4 hot: low 2, high 6, batch 1 cpu 4 cold: low 0, high 2, batch 1 cpu 5 hot: low 2, high 6, batch 1 cpu 5 cold: low 0, high 2, batch 1 cpu 6 hot: low 2, high 6, batch 1 cpu 6 cold: low 0, high 2, batch 1 cpu 7 hot: low 32, high 96, batch 16 cpu 7 cold: low 0, high 32, batch 16 Free pages: 12507020kB (12493888kB HighMem) Active:46134 inactive:953784 dirty:101636 writeback:26 unstable:0 free:3126755 slab:20379 mapped:563179 pagetables:1589 DMA free:12564kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4 all_unreclaimable? yes protections[]: 0 0 0 Normal free:568kB min:928kB low:1856kB high:2784kB active:0kB inactive:659820kB present:901120kB pages_scanned:1845199 all_unreclaimable? no protections[]: 0 0 0 HighMem free:12493888kB min:512kB low:1024kB high:1536kB active:180824kB inactive:3159244kB present:16908288kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 3*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12564kB Normal: 20*4kB 3*8kB 3*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 568kB HighMem: 1418*4kB 535*8kB 128*16kB 85*32kB 119*64kB 38*128kB 10*256kB 4*512kB 2*1024kB 2*2048kB 3041*4096kB = 12493888kB Swap cache: add 0, delete 0, find 0/0, race 0+0 0 bounce buffer pages Free swap: 16779884kB 4456448 pages of RAM 3963834 pages of HIGHMEM 299626 reserved pages 849602 pages shared 0 pages swap cached Out of Memory: Killed process 6587 (dataserver). oom-killer: gfp_mask=0xd0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 cpu 2 hot: low 2, high 6, batch 1

The server was rebooted fine with sybase up the day before. Today, the DBA got some message claiming the tmpdb devices are not initialized. Since these tmpdb devices are created from scratch when sybase starts, it really didn't make sense! DBA decides to drop the current set of tmpdb devices and create a new set instead. Just when the new set is finishing initialization per Sybase logs, the system stopped responding. Before I lost my console, I were able to issue 'ps -e -o rss,pid,cmd,args|sort -n', 'free', and 'top'. Only about 6G is used out of the total 16G. So, the memory is not really exhausted. So, this got to do with how Kernel is carving and using the memory. Googling ' oom-killer: gfp_mask=0xd0' yields a single post which referred to a bug of sorts by a overly-zealous OOM-killer and how to disable it, due to problem with memory addressing scheume switch between high mem and low mem area.

For that, I wonder if ramdisk was claiming HIMEM. If HIMEM were not addressable or addressed improperly, that may have the appearance of OOM (out of memory). Once I started to examine '/var/log/messages' line by line, there, it was painfully obvious that hugemem kernel was not selected by CentOS ServerCD 4.4. The regular kernel-smp was selected.
Oct 23 12:35:30 syb06 kernel: Linux version 2.6.9-42.ELsmp (buildcentos@build-i386) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-3 )) #1 SMP Sat Aug 12 09:39:11 CDT 2006
Oct 23 12:35:30 syb06 kernel: BIOS-provided physical RAM map:
Oct 23 12:35:30 syb06 kernel:********************************************************
Oct 23 12:35:30 syb06 kernel: * This system has more than 16 Gigabyte of memory. *
Oct 23 12:35:30 syb06 kernel: * It is recommended that you read the release notes *
Oct 23 12:35:30 syb06 kernel: * that accompany your copy of Red Hat Enterprise Linux *
Oct 23 12:35:30 syb06 kernel: * about the recommended kernel for such configurations *
Oct 23 12:35:30 syb06 kernel: **********************************************

Once I downloaded the corresponding kernel-hugemem and rebooted the server with it, (grub.conf needed to be edited manually to to boot the new kernel by default). All is peachy again.

P.S. To answer Barry's question below, here is the 12-day memory RRD graph. The panic happened on the last Thursday or Friday, where lines were broken due to missing data. It would be nicer if Blogger actually allows image insertion inside comments. -20061030
P.P.S. Obviously the correct selection of the right kernel package and the maintenance thereof had been a problem/pain for Redhat's maintainers and customers. Per its release notes, the newly minted Fedore Core 6 (FC6) has one single kernel package per architecture. Kernel parameters optimized for different hardware specs (SMP or UNP, 4G/8G/16G/32G of RAM, etc.) would be set dynamicly upon boot, instead of being hard coded in different kernel packages: kernel, kernel-smp, and kernel-hugemem. Of course, with RHEL5 beta in the horizon, I guess we won't see such a change (in RHEL6) for a few years :(

step-by-step: how to set up a large ramdisk on CentOS 4.4/i386

2006-10-23T14:05:00.001-04:00

On our new database server running CentOS 4.4 on a Dell PE 6850, a 2G ramdisk is needed. The option to use 'tmpfs' was considered and forfeited due to its potential of using disk swap.

To make the ramdisk 2G apiece, a kernel parameter (ramdisk_size) needs to be passed ( grub.conf or lilo.conf) and a quick reboot is required to make it effective. The ramdisk by default is 16M in size.

Here is my grub.conf entry:

kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ console=ttyS0,57600n8 console=tty1 ramdisk_size=2097152

To make the ramdisk accessible to the database after reboot, I cooked a chkconfig-compliant init script

 ---cut----x------/etc/init.d/ramdisk4syb-----x-----cut---------
#!/bin/bash
#
# $Id: rc.ramdisk4syb,v 1.00 2006/10/23 13:57:14 jer Exp $
#
# rc file for set up ramdisk fs for sybase to use for tempdb
#
#

 # For Redhat-ish systems
#
# chkconfig: 2345 30 70
# processname: ramdisk4syb
# config: none
# description: set up ramdisk for sybase to use for tempdb
case $1 in
start)
          test -d /mnt/ram1 || mkdir /mnt/ram1
   mke2fs -m 1 -b 1024 /dev/ram1  >/tmp/ramdisk.mke2fs.log &&
   mount -t ext2 /dev/ram1 /mnt/ram1 &&
   chmod 770 /mnt/ram1 && chown sybase:dba /mnt/ram1
          ;;
stop)
   umount /mnt/ram1
         ;;
*)
  echo "Usage: $0 start|stop"
  ;;
esac

to pin the init script to run levels

chkconfig --add ramdisk4syb

to test the init script before reboot

/etc/rc3.d/S30ramdisk4syb start && df -k /mnt/ram1 && ls -ld /man/ram1

to make all this effective by reboot.

init 6

to verify the resultant ramdisk file system

df -k /mnt/ram1 && ls -ld /man/ram1

Much to my surprise, the ramdisk is not mounted after the reboot! The mke2fs log showed the mke2fs was successful. When I attempted to mount it manually, I got
# mount /dev/ram1 /mnt/ram1 mount: wrong fs type, bad option, bad superblock on /dev/ram1, or too many mounted file systems

In the log, I did notice the block size was 4K was instead of the 1K block size I observed with a 512M ramdisk.
Block size=4096 (log=2) Fragment size=4096 (log=2) 262144 inodes, 524288 blocks

Since the Sybase ASE uses 2K page size, I tried mke2fs using a 2K block. No joy! Mounting attempts were met with the same errors.

1K block size is known to have worked for 512M ramdisk, so I gave it a try. Bingo!
#mke2fs -b 1024 /dev/ram1 Block size=1024 (log=0) Fragment size=1024 (log=0) 262144 inodes, 2097152 blocks
Now we have a working ramdisk as below:
/dev/ram1 2063763 3086 1955820 1% /mnt/ram1
I don't see much of a point to waste space on 5% reserved for superuser. So, I tuned it down to 1%
# tune2fs -m 1 /dev/ram1
/dev/ram1 2063763 3086 2039706 1% /mnt/ram1
From the 'free' output, I am a bit surprised to see the 2G is not counted as 'used'.
total used free shared buffers cached Mem: 16627288 231236 16396052 0 48300 89240 -/+ buffers/cache: 93696 16533592 Swap: 16779884 0 16779884

bootparam man[7] page actually says " These days ram disks use the buffer cache, and grow dynamically." If so, ramdisk truly behaves similar to tmpfs at this point, except

ramdisk's maximum size (the reserved?) needs a reboot to adjust, while tmpfs can be adjusted on the fly.
ramdisk won't have swap in the picture at all. [good thing]

However, by growing dynamically, FWIW, it may not able to grow to its full reserved size when it needs it [bad bad thing], if physical RAM is used up by some real processes already. Questions yet to be answered:

Does the 2.6 kernel or other series know or set ramdisk has higher priority on its claim to thus reserved memory?!
why ext2 file system made with block sizes other than 1K failed to be mounted ? ramdisk is the only case here?

which is eth0 on a PowerEdge 6850 running CentOS 4.4/i386

2006-10-22T10:35:00.001-04:00

When setting up the refurbished Dell PowerEdge 6850 I purchased recently, I noticed it is newer and has a few oddities

no video connectors at front panel nor the back. SVGA video only on the DRAC 5 card.
a dual-port Intel Pro copper Ethernet card
PERC adapter is of 4e/DC instead of the 4e/Di with the other two PE6850 bought new from Dell last summer.

A minimal installation of Linux is done smoothly using a CentOS 4.4/i386 server-cd. Well, I did run into one hitch that caused problems getting the network connectivity going. Initially, I didn't have confidence in ports on both ends of the cat5e cable:

The "new" port is in an aged Cisco Catalyst 2900XL switch bought off eBay in 2000. My boss told me several times one port is known to be bad on this switch, since I joined the company a year ago. I used twelve free ports so far without problem. This port is one of the last two "free" ports on this 24-port switch.
The port was in VLAN #1 to mirror two neighboring ports and had not been in active use. I programmed the switch such that the port now participates VLAN#3 instead of VLAN#1, and no longer mirrors any port.
I was hoping to see a steady green LED and a blinking yellow LED on the NIC, by switching the cable from port to port on the server. The results were not telling. The first embedded Broadcom port had that, but no connectivity. The two Intel-pro ports had only green LED, no blinking yellow LED nor connectivity. I later came to the realization that the fast switching from port to port won't work well, probably due to ARP cache and such on the switch.

After a bit of frustration, I decided to leave the cable in the first embedded port and to do some serious work on the server side to make it work. Need a quieter place to think and type, I borrowed a serial cable to access its serial console. All the boxen I set up have serial console enabled in /etc/inittab and GRUB or LILO. For Dell PowerEdge servers, with BIOS-redirect-to-serial enabled too. Too busy to bargain hunt a serial condenser. I used Digi board before and it is nice. However, Digi board is too expensive a proposition for the current employer who's satisfied with the capability to call the ISP technician to hook up monitor and keyboard to their servers.

Once sitting down in the much quieter guest working area, I immediately put the pieces together and realized that the embedded Broadcom Ethernet ports were enumerated as eth2 and eth3, while the two ports on the Intel-Pro PCI-e card were as eth0 and eth1. Here is my "detective work":

grep eth /etc/modprobe .conf
alias eth0 e1000 alias eth1 e1000 alias eth2 tg3 alias eth3 tg3
dmesg
e1000: 0000:17:06.0: e1000_probe: (PCI:66MHz:64-bit) 00:04:23:cd:6e:b8 divert: allocating divert_blk for eth0 e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection ACPI: PCI interrupt 0000:17:06.1[B] -> GSI 133 (level, low) -> IRQ 50 e1000: 0000:17:06.1: e1000_probe: (PCI:66MHz:64-bit) 00:04:23:cd:6e:b9 divert: allocating divert_blk for eth1 e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection tg3.c:v3.52-rh (Mar 06, 2006) ACPI: PCI interrupt 0000:14:02.0[A] -> GSI 64 (level, low) -> IRQ 201 divert: allocating divert_blk for eth2 eth2: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:13:72:40:e1:bd eth2: RXcsums[1] LinkChgREG[1] MIirq[1] ASF[1] Split[0] WireSpeed[1] TSOcap[0] eth2: dma_rwctrl[769f4000] ACPI: PCI interrupt 0000:14: 02.1[B] -> GSI 65 (level, low) -> IRQ 209 divert: allocating divert_blk for eth3 eth3: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:13:72:40:e1:be eth3: RXcsums[1] LinkChgREG[1] MIirq[1] ASF[0] Split[0] WireSpeed[1] TSOcap[1] eth3: dma_rwctrl[769f4000]
grep HWA /etc/sysconfig/network-scripts/ifconfig-eth?
ifcfg-eth1:HWADDR=00:04:23:CD:6E:B9 ifcfg-eth2:HWADDR=00:13:72:40:E1:BD ifcfg-eth3:HWADDR=00:13:72:40:E1:BE ifcfg-eth0:HWADDR=00:04:23:CD:6E:B8

With the card-to-driver mapping in dmesg and in the modprobe.conf, it is easy to see the embedded Broadcom ethernet ports are mapped to eth2 and eth3, while the add-on Intel-pro ports are mapped to eth0 and eth1. The mac addresses in the 'dmesg' and the HWADDR recorded in ifcfg-eth1 through ifcfg-eth3 files hammered down the last nail. Without the last nail, it'd rather hard to tell if all ports is of the same make and model.

mii-tool sorta told the tale as well. lspci can help too.
eth0: no link eth1: no link eth2: 10baseTx, HD, link ok eth3: no link
Once I duplicated the IP/netmask on eth0 to eth2, then ifdown eth0 and ifup eth2, all is well now per ' mii-tool'
eth2: negotiated 100baseTx-FD, link ok

It'd be an interesting exercises to force eth0 to be the first embedded Broadcom port. It makes more sense for Linux to enumerate the embeded ports first, ethernet or not. What do you think?

step-by-step: how to host WordPress under Linux (FC5/RHEL4)

2006-10-21T04:35:00.001-04:00

Why:

on Blogger: missing certain nice features ( static page support, import/export). Or,
on WordPress.com: don't like your own html get stripped off. Your little piggy-bank gets left empty since Google Adsense code get stripped or disabled. Or,
want your own professional domain name and such. Or,
simply love the experience of DIY and the freedom of customization

Assumptions & Prerequisites:

Fedora Core 5 stock installation (All should be applicable to CentOS 4 or RHEL 4 as well, unless the FC5 SELinux changes get in your way.)

apache (httpd package) installed.
php package installed.
mysql-server package installed.

The stable release (latest.tar.gz) has been downloaded from wordpress.org and extracted under /var/www/html
comfortable with MySQL and SQL commands.
starter's knowledge of how a blog works.

Steps to follow:
# set up MySQL access

This is to set up a MySQL database dedicated for WordPress and a user to access this database from the Apache/PHP combo. The schema and data for the database itself will be populated by install.php later. The names used below assume you want user 'wp_master' to have all access to a local database named 'wordpressDB'.

/etc/init.d/mysqld start
/usr/bin/mysqladmin -u root password 'mysqlAdminpassword'
/usr/bin/mysqladmin -u root -p create wordpressDB
mysql -Uroot -p mysql

insert into user (host, user, password) values ('localhost', 'wp_master', password('mastersecret'));
insert into db (host, db, user) values ('localhost', 'wordpressDB', 'wp_master');

the 'db' table entry should also grant the 'wp_master' user with most of the privileges to the 'wordpressDB' . So, I simply replaced the 'test' database entry therein, since I am lazy, plus I consider a wild-open test db a security risk.
update db set host='localhost',db='wordpressDB',user='wp_master' where host='%' and db='test';

commit;
restart mysqld by '/etc/init.d/mysqld restart'

# set up MySQL PHP extension support (Otherwise you may get this error, "Your PHP installation appears to be missing the MySQL which is required for WordPress. ")

yum -y install php-mysql
/etc/init.d/httpd restart

# to set up wp-config.php with your own set of credentials for WP to use to access MySQL
cd /var/www/html/wordpress && cp wp-config-sample.php wp-config.php && vi wp-config.php
# RTFM using your favorite browser

point to your browser to http://localhost/wordpress/readme.html

# initialize/populate your new WP database. Do record the 'admin' account information generated.
point to your browser to follow http://localhost/wordpress/wp-admin/install.php

# Now, time to verify. If it worked, you'd see the default 'hello world' post using the 'default' theme.
point to your browser to http://localhost/wordpress/index.php

# to customize the blog right away, click on 'edit' link inside the blog and log on with the 'admin' account

# about them themes:
Only two themes are bundled (classic/default). To add a theme, find ones you like and download from a trustworthy source.

wget http://downloads.wordpress.org/theme/pool.zip
cd /var/www/html/wordpress/wp-content/themes && unzip pool.zip
Now the new theme(s) would just show up if you go back to edit/presentation/themes

# Get them roamers home

import existing blogs and comments from Blogger and other top blog sites.
if new, register this new blog with technorati.com, weblogs.com
new or updated, brief your readers of the changes [advance notice would be appreciated too]

Blogger :: a work in progress, beta or not

2006-10-18T21:22:00.001-04:00

I joined Technorati and I learned it is supposedly a great thing to have proper label (Blogger's or Google's terminology) or tags (Technorati's term). A few nights back. I started to retrofit my then fifty some posts with labels. I selected a post to edit, added a label or two, then published it again. Back to the main 'edit posts' screen, all the posts looked the same, with or without labels. I said to myself, wouldn't it be great the labels be showed next to the associated post?!

Today I resumed my label-retrofitting task. Lo & behold, the 'edit posts' page has all current labels displayed, just like Gmail does for each messages. Wow, progress!

Editing posts, I noticed the oldest few have hard breaks (
) which broke the lines. The new posts don't have this problem, even if all of them were posted from Gmail.

PERC 4e/di went offline on a Dell PowerEdge 6850, with nothing apparently bad

2006-10-18T12:14:00.001-04:00

I got paged 2:15 in the morning, only to find the main RAID controller on a Dell PowerEdge 6850 yanked itself away from under the running OS (CentOS 4.1/i386), again. The OS seemed to be fine, except any task involving read/write to the disk failed.

The PE6850 server was bought new from Dell last June, and has been in production since. Same day same time last month, it gave us its first outage showing the same symptoms. Questions we asked ourselves are: why starts now? And why exactly one month apart?

The serial console showed the following message repeating on the screen:

  scsi0 (0:0): rejecting I/O to offline device  EXT3-fs error (device sda2): ext3_find_entry: reading directory #178817 offset 0

The nightly full database backup kicked off at 2:00 and should finish by 2:45am.
Log exported from the PERC 4e/di controller showed that PR (patrol read) is scheduled to run every four hours and one started at 1:00 and didn't get to finish either.
Same RAID controller log also has two interesting entries that coincided with the two outages. These are the only two "ProcessHostDmaInterrupt: No requests active" for a log stretching back to last June. The DMA thingy made the BIOS update A01 interesting in that it corrects memory address assignment or claims by the raid controller ( The server has 16G DDR2 ECC RAM, 2G chips)

09/18  2:05:47: ProcessHostDmaInterrupt: No requests active! (ch=a0102be8)
09/18  2:05:47: ProcessHostDmaInterrupt: No requests active! (ch=a0102be8)
10/18  2:13:34: ProcessHostDmaInterrupt: No requests active! (ch=a0102be8)
10/18  2:13:34: ProcessHostDmaInterrupt: No requests active! (ch=a0102be8)

Some postings suggests turning off PR may help in case it conflicts with heavy disk I/O from application tasks. However, at time of both outages, the disk I/O and the CPU load is not the highest, as we've seen much higher ones during the day every work day, per the baselines neatly graphed using RRDTOOL by our new Hobbit Monitor.
Dell Diag is run and nothing is reported wrong, for the memory, disks, RAID controller and such.

Checking Dell's Driver download site with the server's service tag turned up a few interesting potential fixes:

LSI Logic Perc 4e/Di, v.522A, A13 [[We are at 521S, A00]] Release Date: 9/11/2006

"Fixed an issue that could cause a blue screen, file system error or system hang when using EVPD inquiry commands."
Anybody has any idea what is exactly "an issue" ?!

BIOS upgrade A01 [[ we are at A00 ]]

Added workaround for lockup resulting from the systems with 8GB RAM or more and RAID storage controller potentially claiming inappropriate addresses.

SCSI controller firmware update JT00 for various Maxtor drives

"Under certain circumstances a hard disk drive may go offline, hard disk drives (HDD), may report offline due to a timeout condition. If the HDD is unable to complete commands, this may result in the controller reporting the HDD off line due to a timeout condition."
"Higher than expected failures rates have been reported on the Maxtor Atlas 10K V ULD (unleaded or lead free) SCSI hard disk drives. If the hard disk drive (HDD) is unable to complete commands, this may result in the controller reporting the HDD offline due to the timeout condition. The primary failure modes have been the HDD failing to successfully rebuild and also failing after a rebuild has completed."
Sounds promising, but it turned out that drives in our PE6850 are all Seagate.

Google's add-comment Firefox extension fails to log me in to Blogger beta

2006-10-15T17:30:00.001-04:00

I downloaded Google's add-comment FireFox extension. I thought it is neat to comment on a interesting web page and have the link preserved at the same time in a relevant blog post (This works more like a scrapbook, only for the web). However, first try at www.citrix.com the extension failed to log me in. I tried both my Google account and my blogger account.

As described by previous posts, I converted to Blogger beta last week. Maybe this is one of the glitches for the SSO (single sign-on) the new Blogger beta is using. Strange thing is course, none of those valid accounts work.

Citrix Netscaler NS7000 : how to create a content switched load balanced farm/service (Part II)

2006-10-15T09:47:00.000-04:00

In the previous part of this article, how to create a load balance virtual server (or 'lb vserver' using CLI) was described. Now we'll describe how to create content switching. Currently, as of Netscaler 7.0 beta, Citrix Netscaler supports only HTTP protocol for content switching.

Again, the logical, conceptual flow is clear and intuitive. However, the work flow is less so, the same as any object-oriented model applied to a more procedural mind-set or work flow.

In Citrix Netscaler's terminology, you need to create

a Content Switching (CS) policy. The policy basics states a matching rule set that the HTTP request object must comply. Three basic

domain, aka, URI
URL
any other request attributes/headers using regular expression or absolute matching.

create a CS virtual server to direct traffic regulated by this policy to go to either individual service or a load balancing virtual server.

a default mapping is allowed. The policy field would be 'blank'. It'd make more sense to actually call it 'Default' and displays it as such. This basically directs all traffic that fails to satisfy all the policy-vserser mapping to a default LB virtual server.

Regular Expression can be created on the fly under the 'policy' node, or pre-created with a name from under 'System' node.

to claim my blog from technorati

2006-10-14T20:12:00.000-04:00

This post is merely to claim this blog from technorati.
Technorati Profile

bargain shopping :: a recent Dell PowerEdge server purchase

2006-10-13T12:39:00.001-04:00

My company needed to buy a Dell PowerEdge 6850. As the only systems engineer and the resident tough-nose bargainer, I was tasked to find a good deal and fast. I ended up buying a "certified refurbished" from Dell Outlet for about $5700. The specs are like:

quad Xeon 3.66GHZ/1M
16G DDR2 ECC RAM (2G chips)
2x 146G drive (in RAID1) + 3x 300G drive (in RAID5). Ultra320 scsi, 10K RPM
rapid rails
CDRW/DVD ROM combo + floppy drive
dual power supply
RAC card
3-year NBD warranty (bronze)
ship the next day.

The viable options I had were:

$13,000, for a brand-new from Dell Business. Drives are smaller: 2x73G + 3x146G drive. ship in two weeks or so. no RAC card.
$8,900, used, from an eBay store. 90-days warranty and no rails. no RAC card.
$8,500, new, from yet another eBay store. 3-year NBD (till March 2009), with quad 3.16GHZ/1M Xeon CPU. ship in a week. no RAC card.
NA, from two local VARs. After a few days of hunting for a Dell (with enough profit margin, I guess), they started asking me whether a Compaq can be quoted instead.

For the Dell Outlet purchase itself,

Initially I thought it's hard to cut the red tape (PO, approval, and stuff) fast enough to close the deal online. Fortunately, I talked to a Dell representative. He told me I could simply email/phone him of my pick so he can take it offline and hold for me for a few days. How nice!
On top of that, I asked for 'best offer' from Dell Outlet, saying that I had quotations roughly $2k under Dell's price for a similar server. Similar alright, the quotations had slower CPU, or smaller HDD, or less denser RAM chips, or lesser days in warranty, or all the above. It is a small bluff :) The fellow said he'd check with his people. Next thing you know, he came back the next day and slashed the online price by $1977. With that, I had no hesitation whatsoever to ink the deal right away, before it disappears!

The lesson I took from this round of bargain shopping is "Do your homework". Know how much they differ before you even ask for a quotation. Don't overlook or dismiss too quickly the value of bigger GB/RPMs/Cache of the drives, bigger density of the memory chips, bigger GHZ/cache of CPUs, since they can cost you dearly enough to shift a nice-looking quotation from a bargain to a bargain look-like! Like they say, details kill.

face lift :: switched to Blogger beta (continued)

2006-10-11T20:22:00.000-04:00

Other nice features (some are more like bug fixes) are

perma link if you click on the title of any post
secondary bullets work now. Before Blogger beta, secondary bullets were automatically promoted, even though the HTML code ( UL embedded inside UL) and WYSIWYG editors on Blogger's 'Post Edit' or Gmail's composer shows proper layout of double indented secondary bullets.
the mysterious slice of the text ad link on top of my blog disappeared.
The editor used to 'edit posts' has two tabs. One is 'Edit Html'. The other is 'Compose'. Those two should really be 'simple WYSIWYG editing' versus 'advanced HTML coding'. I'll leave it to the legend of creative workers at Google/Blogger to cook up proper pair of names.
The use cases for the multi-blog feature are not as thought-through as other features. The profiles(about-me) share the same profile. As a test, I created a private blog. Email post to that blog doesn't work yet. Two emails were returned with permanent error by alpha1.blogger.com or something to that effect. Since the blogger id is the same, I wondered which blog of the two I own under the Google account should the email post be directed to. Since I had only one blog before Blogger beta, so I wouldn't know whether this is an old bug/feature or not.

face lift :: switched to Blogger beta

2006-10-11T09:00:00.000-04:00

Experimenting a blog on Wordpress.com, I liked many features. Yet to find time to try the open-source blog hosting software itself from wordpress.org. However, I was very disappointed to find Wordpress.com strips down the Wordpress blog software. Even html code to insert a Google adsense ad for Firefox was stripped out. Blogger has great support for adsense and such. However, it doesn't do static page. It is pretty desirable to have static page to hold, say, some static contents.

Sigh, do I really have to host my own? As a systems engineer by trade, I really don't need the hassle of hosting. Only if I can get the features I need for free. Only if Blogger can do 'page' to hold semi-static content.

Why not?! I googled for it. Among the first links, I found a link to 'Blogger beta' talking about new features with widgets and such. Click through and end up switched this blog to the new Blogger BETA.

Changes that met my eyes are:

blogger account is now assimilated to your google account

In other words, now Blogger is yet another Google service, side by side with Gmail, calendar, maps, search, spreadsheets, and such.

Template is now widgetized:

Pros: You move blocks (widgets) around on a page layout. I guess they copy the behavior of WordPress. I got to say it is such a nice feature.
Pros: Arbitrary html/javascript can be used to create a widget of your own.
Cons: A custom widget should have an extra label of sort so you can distinguish which is which on the page layout when you come back to edit. Currently, it only has a title, which is displayed on the actual blog. Imagine if you have more than one custom widgets.
Cons: Ad-sense widget couldn't be added, generating some error. I wonder if it is because my adsense account has different password. Search-box html code copied to a custom widget doesn't work quite right either. In fact, you can see a text ad link on top of my blog and a image ad link on the right column. I didn't asked for the top link! == as always, Google has wisely branded it as 'BETA'.

access control is in place, replacing the current membership. Yet another feature copied from WordPress, I assume.

authors can be added to co-own the blog
access can be changed from Public (the default) to private

the power of outsourcing, as felt by a vending machine operator

2006-10-10T14:58:00.000-04:00

Standing in the break room warming up my lunch, I noticed a mid-aged man was replenishing the vending machine. To break the ice, I said something like, "I heard operating vending machines is a pretty profitable business." He smiled, nodded, then shook his head. Here is what he said,

He has operated vending machines for over twenty years here in Atlanta, GA
His business hires four employees and runs over 100 vending machines all over town.
Blue-collar workers are 'good customers', in that they eat anything everything you put in the machine, w/o complaints. nice & courteous. On the opposite end of the spectrum is the ladies in the offices, who complains about anything everything. To put the consumption rate quantitatively, thirty blue-collar workers consume the same a week as one hundred office workers for a month.
Due to all the outsourcing of blue-collar works and white-collar work, the need for vending machine had stayed the same if not shrunk.
Competition to operate them in office buildings are intense. Too many operators. It takes a good sales person with good pricing/service/incentives/suave talking to win new accounts, after a lot of cold calls. These accounts most likely are converts instead of brand-new accounts.
It doesn't take much to lose an existing account, once the facility manager or other self-elected vending machine manager changes. Your good service/good will instilled to the leaving management is all gone. It is a victimless victory for the latter to make a mark, to say services has been improved for the better.

free hosting service :: good business plan or scam

2006-10-10T14:04:00.000-04:00

I was searching for free hosting services earlier. Criteria is simple: over 100MB and allows php/mysql and perl CGI, >1G/month bandwidth. frihost.net came as top choice. Its requirement is simple too, 'register to post to its forums five times (and accumulate 10 points)." So I did. registered on its forums, posted 6 messages, and accumulated over 15 points (apparently judged by length of the post).
went ahead to apply for the hosting account. a day later got a PM (private message) informing of rejection due to poor quality of the posts. I don't really agree, since I believe my posts' quality is in line with other posts therein. Nevertheless, I complied by posting a few more then applying again. This time I got through. The control panel seems to be powerful (HTTP not HTTPS, though). Yet, I couldn't find a way to publish my first Hello-world page. Based on my understanding of Apache and such, I'd think 'echo Hello World > ~/public_html/index.html' would be it. However, my virtual domain was not resolvable after two days or so. Fake DNS entries in my own BIND setup or hosts entries, getting to the generic 'Apache is up' stock page.
A bit annoyed, so I started to read the FAQ section of the hosting service. Much to my shock, it states that one had to stay active on their forum with quality posts, otherwise your hosting service will be canceled. Now I appreciate the old saying, "when it sounds too good to be true, it probably is." What a good business plan, to hold hostage of people with quality posts in the so-called frihost.net forums community.

nice security enhancements on Vanguard.com against phishing scams

2006-10-10T08:53:00.000-04:00

I received an email from Vanguard. Therein two secure measures are added. one on authenticating server and the other on authenticating client/customer. Both are pretty smart moves (multi-factor authentication) and will prevent bulk of the current phishing attacks.

After entering your user name, you'll select and name a security image. Whenever you log on, you'll see your security image—and you'll know you're on the authentic Vanguard.com site."

You'll also answer three security questions. If we don't recognize the computer you're using to log on, we'll ask you to answer one of your security questions"

vulnerability assessment of Citrix Netscaler NS7000 using Nessus

2006-10-04T16:59:00.000-04:00

I used Nessus on the latest Pentoo Linux live cd to assess the Citrix Netscaler NS7000. For the ports

The BGP port should really not be there, since it is not enabled. The connection was closed immediately. however, it shouldn't allow connection at all.
HTTPS server allows SSL V2 ciphers which can give administrators false sense of security if they happen to use aged browser.
UDP in general: Traceroute is enabled. bad. bad.
TCP in general:

The remote host does not discard TCP SYN packets which
have the FIN flag set.

Depending on the kind of firewall you are using, an
attacker may use this flaw to bypass its rules.

See also : http://archives.neohapsis.com/archives/bugtraq/2002-10/0266.html
http://www.kb.cert.org/vuls/id/464113

Solution : Contact your vendor for a patch
Risk factor : Medium
BID : 7487

====================================================================

The remote host accepts loose source routed IP packets.
The feature was designed for testing purpose.
An attacker may use it to circumvent poorly designed IP filtering
and exploit another flaw. However, it is not dangerous by itself.

Solution : drop source routed packets on this host or on other ingress
routers or firewalls.

Risk factor : Low

Citrix Netscaler NS7000 : how to create a content switched load balanced farm/service (Part I)

2006-10-03T11:23:00.000-04:00

The Netscaler (now Citrix) load balancer has pretty clear conceptual, logical, and work flow. That is, well, to a system/network engineer like me anyway. To touch it off visually by a GUI, all this is neatly grouped under the 'load balancing' leaf node and the 'content switching' leaf node on the left pane of the Applet or Web Start GUI.

define servers. A server is created by a pair of server name and its physical IP address: node8/192.168.88.8 node9/192.168.88.9 ... node100/192.168.88.100.
define services. A service is created by binding the server IP address with a service port (port number and protocol, e.g. TCP/80, UDP/53, UDP/123).

Take web service for example, a service named "prod_web_node1" can be created by binding TCP/80 with node1's IP address.

Custom and default health monitored can be applied to the service.
Multiple monitors can be applied a service with different weight and such.
Web service is the most popular for an application switch or a load balancer. In fact, it is the only protocol rendered capable of content-switching by the Citrix Netscaler series.

create a load-balancing "virtual server" (From CLI, it is 'add lb vserver'.) to front a group of services to be load-balanced.
- weight can be applied to each LB member service.
- load balancing algorithm can be chosen (many variations of round-robin, least connection, LTRM, etc.)
- advanced setting can be adjusted here, such as timeout values for an idle server or an idle client.

Any 'lb vserver' can be utilized in two different ways:

As a public service by itself. It can be exposed to clients as a service by assigning a virtual IP:Port. In other words, the load balanced service group becomes a service usable by the clients (directly). If load balancing is all you need, your work is done here.
As a private service to a Content Switching (CS) virtual server (vserver). No IP:port is assigned to it. In other words, the load balanced service group won't be a service usable by the clients (directly). It can only be accessed/referenced by its name, as an internal object, from within the Netscaler running context (aka, by the the content switching engine we'll discuss in Part II of this post) in the latter case. That's right, it is more like a call-by-reference.

a void left by Dell discontinuing PowerEdge 1850

2006-09-29T10:14:00.000-04:00

I was tasked to purchase yet another Dell PowerEdge 1850, to be dedicated to certain tasks currently done on some other-purposed server. The Dell account rep sent me an quote for PE1950, in response to my RFQ for an PE1850. I thought it is merely a typo, so I called him up. He told me, it is not a typo but that Dell decided to discontinue 1850, as the industry moves towards dual-core processors, aka, the new x9xx series for Dell PE. I told him that we can't go with dual-core processors, due to software licensing restrictions, since DBMS (Sybase & Oracle) will count each core as 0.8 CPU or so . We either go with single dual-core CPU (loss of redundancy and an underutilized 2-CPU license), or go with dual dual-core CPU ( now we need to purchase 1.2 CPU license to make up 3.2 CPU license instead of 2 CPU license). He informed me that the next best alternative for us would be 850, which only features Celeron processors.
This move contradicts Dell's intention to cater to SMB users, who use smaller boxen to house database applications whose licenses often involve CPU count. Not sure this void will be filled by Dell herself later, or by some third party such as HP. Either way, wish they had a published life expectancy for each model. So, you can make an informed decision when you starts. Case in point, I wouldn't buy six PE1850 last summer or Dell for that matter, if I knew then it will be discontinued only a year later or less.

f5 support site top-notch : web UI useful & friendly & easy.

2006-09-27T05:02:00.000-04:00

After signing up for an askf5.com, I was redirected to tech.f5.com site. Therein I found documentations, release management information, release/patch/hotfixes downloads in a well-organized fashion. Most other tech support sites I visited before gave you the feeling of a dump-all-here repository with a thin cover of HTML. The F5 sites allows you to select a version, which has a short description of what this version is on a neatly tabulated fashion. MD5 is provided for downloads. The provided md5 checksums is way above par for commercial entities, even though PGP or GnuPG signature is more desirable and more secure. Three different downloading mechanism is provided: FTP/HTTP/HTTPS. Again, all neatly tabulated and explained in plain English. It seems that this site itself had gone through a real OO modeling design & implementation, to ensure the UI be useful & friendly & easy. One cue to support this suspicion is a reference of 'choose a container' instead of saying "choose a version number'.
The F5 LTM/GTM software release scheme has recently been revamped and are well documented on their site. The scheme is pretty straightforward and makes an easy read for a professional systems engineer like me instead of software engineer. basically a x.y.z scheme with y being odd/even to distinguish a feature release versus a maintenance release. As I am writing this, I came to wonder whether this F5 scheme had been influenced by the odd/even version scheme of Linux kernel development. For instance, kernel 2.5 was unstable/cutting-edge while 2.4 was stable releases with maintenance fixes only). I will consider to recommend this scheme to our internal release team, such that our next release number will not be an 'ass-pull' - a quirky code invented by an obnoxious-wanna-be Yankee veteran and has been circulated around our office.
No wonder askf5.com is the first and only site the F5 Big-IP SE (systems engineer) referred me to. It gave me warm & fuzzy feeling about F5 that all this releases and patches are manged so well, so consistent, and for free. This came in stark contrast with Citrix Netscaler SE. He made such a big deal of gaining access to newer builds and releases and didn't give me access until he agreed the feature I need to evaluate is not there in the bundle. Even then, he had to grab them somewhere and slap them into a FTP server in his own iBook (OS X), and gave me an one-time ad hoc account to download. One year or three years post-sales, I have no doubt that I can find bug fixes and new features with ease from F5. Needless to say, I don't have that confidence with Citrix Netscaler team. Hope they are merely temporary quirks produced by the Citrix acquisition.

old 9G not in supply any more?

2006-09-20T15:35:00.000-04:00

A 9G disk failed on a production server. It is a Dell PowerEdge 2450 and I know the service tag, so I searched on Dell's site. Amazingly it recognized the service tag of the old boxen, but produced only 18G or above and started at $250 a pop.

technical interview for an enterprise job (continued)

2006-09-20T12:47:00.000-04:00

Three days after the interview (the labor day weekend), I was offered the job. So, the hints I picked up are real, when they showed me around the office and introduced me to the potential peers after the technical interview. The offer was initially ~10K short to meet my current full compensation & ~5K below my current base, so I told my headhunter who relayed the good news.
The headhunter's account manager kept assuring me that they won't be able to offer more and insisted that he just signed up a professional with a 10K pay-cut. He said, and I quote, "people do that all the time." So I said, and I truly meant it, "I'll to have to decline the offer as it stands now." He didn't say anything and we bid adios . Next day, he called me. Obviously he picked up only part of the message (5k lower than my current base) and went on to bat for me. Appreciatively speaking, it worked, the client offered to match my current base. That did make me feel good and bad. "Good" in the sense that my skills and experience are truly appreciated at this new place, so my pay rate is rectified in more than one places. Sometimes I wondered whether my current "high" pay was only because my current employer was desperate in need of a systems engineer when they hired me on. "Bad" in the sense that their offer was such a low ball, so I'd expect an up-hill fight to get big raises in the future if I get in. Hope it was only because I am making too high a salary now (hmm, should I feel good about that?!)
Since the new place doesn't have ESPP, stock option, or bonus for that team, the improved offer was still a 5~10K pay-cut for me. Once I made that clear, the account manager again assured me that their client wouldn't be able to offer any more. Sure enough, the client rescinded the offer the next day. Later I learned from my source, that the hiring manager was offended, since my pay situation was not explained or presented clearly to him, thus he took that as a needy/greedy give-me-more-and-more request, taking advantage of his good faith effort to sweeten the offer last time.

to flush a jsp

2006-09-20T03:17:00.000-04:00

in order to test how a load balancer (F5 BIG-IP or Citrix Netscaler or others) time-outs an idle server or client, I cooked a simple JSP. what I desire is to flush any byte as soon it is written to the response stream. Using PERL, one'd set $| = 1; or autoflush(1) and be over with it. In an Apache/Tomcat environment, I ended up using <@ page buffer="none" %> and response.flushBuffer() combined. The API for 'buffer' page parameter led me believe it should accomplish what I desire by itself. It doesn't seem to be the case. Maybe it is buffered by Apache?

F5 BIG-IP version 9 sends a RST packet to the client after timeout

2006-09-20T03:08:00.000-04:00

The load balancer in production on our site currently leaves the client hanging. After timeout on the server or client, it terminates the connection to the server but doesn't terminates the connection to the client. As a result, the FireFox or IE browser spins its hourglass or whirls forever, as confused the hell out of the unsuspecting non-techie end users.
With that in mind, it is quite a relief to me to see a RST packet received by the client, after 320s of idle time ( thread.sleep(320000)) in a jsp served by Apache/tomcat. Since FireFox actually sent 'keep-alive 300' in the http request header, so I am not so sure the 300s is per FireFox's request, or per F5 Big-IP v9's own hard-coded client (or server) idle timeout. If I recall correctly, Citrix Netscaler EE9000 (version 6.1 and 7.0 beta) defaults to timeout on idle client at 180 seconds and on idle server at 300 seconds.

F5 version 9 is based on Redhat Linux 9 microkernel

2006-09-15T00:23:00.000-04:00

I received a F5 LTM 3400 on Tuesday. Today I finally squeezed in some time to boot it up and checked it out. It seems to be based on Redhat Linux 9 microkernel.
It has two processors, so the SMP kernel is used. However, I was told by the F5 SE that only one is used for the application switching task (load balancing + content switching).

wonder of modern technologies: MSN video conference allows my son dance for his grand-parents half-globe away

2006-09-12T23:14:00.000-04:00

Tonight, I put my son on the desk in my study. He danced, clapped hands, while his grandma and grandpa sang a song and cheered him on, via MSN Messeger & a pair of Logitec Pro 4000 webcams. All in real time, across half a globe. Hearing ans sharing the laughter and joy on both sides of the MSN screen, I am once again amazed by the wonders of video conferencing and the technology advancement that made all this possible, even though I am an IT professional by trade.

It is certainly inconceivable to send electronic mail just forty years ago, much less a video conference. My uncle lives several states away from home since his teens. His siblings including my mom visited only once when he turned 60. The only connection they have is a greeting card for the new year.

custom transport for postfix on RHEL 4/AS (CentOS 4.3)

2006-09-06T10:00:00.000-04:00

I was told to use a special server to relay outbound emails for a new NMS server I built. The NMS server has POSTFIX installed and is configured to send production alerts to a new duty pager, a Blackberry 6230 with BIS account served by T-mobile. I did POSTFIX relay thingy once elsewhere, so, I instantly did the following

vi /etc/postfix/transport & added the following two lines at the bottom of the file

tmo.blackberry.net smtp:smartrelay2.intranet.com
intranet.com smtp:10.9.9.99

run 'postmap' to generate the hashed transport:

postmap /etc/postfix/transport

vi /etc/postfix/main.cf and commented out the global relay

#relayhost = 10.9.9.99

reload configuration for the running postfix. Reloading was met with some nonsense error/warnings, so I ended up with a full restart.

/etc/init.d/postfix reload OR service postfix reload

tested it and it didn't work. it seemed that the transport map doesn't exist, since it looked up MX record to find SMTP servers to send mail to.

man postmap, man postfix, man 5 transport, grep -i transport main.cf.

It became obvious that main.cf doesn't have reference to use which transport hash at all. so I added the following then restarted POSTFIX. Bingo!

transport_maps = hash:/etc/postfix/transport

It still made me curious why this is not enabled by default since one needs to consciously insert rules into the transport configuration file as well as to consciously run postmap to generate the hash. This would save me the hassle to figure out it is not there, and save a service outage in production environment should custom transport becomes necessary. The only argument I can find up to now is the security, only if it takes different credentials to modify transport.db and to start use transport.db at first place.

Of course, "crazy" default behavior can be identified on other modern operating systems also. Don't get me started with that Solaris halts if it receives the POWER Fail signal sent by a resetting serial console.

technical interview for an enterprise job

2006-09-01T17:32:00.000-04:00

Two days ago, I went for an on-site interview at a major media company. The lobby is similar to Google's in that you have their namesake news channel on a big LCD TV instead of a big screen showing all the current buzz words people are searching using Google Search. The schedule seemed a bit rushed than Google's, since it is 1.5 hours instead of 3. Of course, I flew from ATL to SFO just for the Google interview, thus they'd want to ask any/all questions they have at one shot.

At this point of my life and career, I am really interested in a technical lead or hands-on manager type position, as a transition to move up the ladder, so to speak. I have advanced degree (MS/MIS) and broad exposure to major IS/IT components such as OS, RDBMS, network, security, etc. Over the course of the interview, I started to get the feeling that this position was purely technical thus no upward opportunity would be there. That said, the group is sizable and currently has no manager or lead, plus the job description in the ad requires tech lead experience. So, we'll see. Before he took off for another meeting, the director asked me a PM-type question and told me that my answer was a good one though a little bit too detailed. Is this a hint ?!

Two potential peers showed me around the working places and introduced me to the other three guys/gals. A classic textbook hint that they view you as a potential peer (a good thing). Or, I could have read too much into it, since they could have been just showing off their manners.

On my drive back to work, I thought of a couple of cons:

The cube is tiny though private
The drive would be bad and I don't think telecommuting would be an option.
On-call duties would definitely be rotated.
Money won't be so great either, since the high end of the range is lower than what I make now.
could be pigeon-holed into an OS-only type person
could be victim of outsourcing or politics in a large organization

And, of a few Pros:

The name recognition of the group
The enterprise experience you definitely won't have from working with SMBs.
opportunity to specialize & get deep onto topics of interest

Working with SMBs are more stressful, since you are the ONE, plus very limited resource to address same type of concerns that large enterprises pour money/man-hour into.

windows 2003's RAID5 doesn't support hot-spare --- observed with help from the great VMWare Player

2006-08-26T02:09:00.000-04:00

I finally found some scsi virtual disk image to be used for my Windows 2003 server, a virtual machine I set up inside the 'VMWare Player" in prep for 70-290, my first MCSE test. In the past weeks, I had a hard time to have more than 2 free (IDE) disks in order to simluate an environment where I can set up RAID5 volumes.

The successful count of the RAID-5 capability inside windows 2003 server:

lsiLogic is the controller type supported by the Windows 2003 server. busLogic is not supported w/o external driver.
run Device manager to remove the old/broken busLogic controller then re-scan for hardware change
run Disk Manager (diskmgmt.msc)

initialize all four brand-new identical 500MB scsi disks
convert them all into dynamic disks
create a RAID-5 volume using 64MB on each disk: right click on one of them, select 'create a volume' from the pop-up menu, check 'RAID-5', select disks & sizes to create.

Nowhere, hot-spare is found in the create-a-volume wizard, or the resultant 3-disk RAID5 volume with one free dynamic disk available in the system

No wonder 'VMWare Player' is voted the best technology of the year by some IT pro magzine. Using 'VMWare Player', I just inserted a few entries to the .vmx configuration file claiming I have a few new disks and a SCSI controller now, on a suspended server. One the server is resumed, the controller & the disks are detected and installed properly. Can you imagine the hassle I would have had in a non-virtualized world. Arghhhhhhhhhhhhhh....

buy a busLogic SCSI controller, only to find it doesn't work.
waste time searching for proper driver, or return it and get a lsiLogic one to try my luck.
buy 4 scsi disk to set up 3-disk RAID5, only to find the 4th disk can't be used as hot-spare anyway
shutdown os, power off the box, open the case, screw driver, struggling with free dribe bays & cables, etc.

To have four SCSI disks on a SCSI controller, I just add the following to my blah.vmx:
---x------cut----------cut-----x------cut------------x----
scsi0.virtualDev = "lsilogic"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "sda.vmdk"
scsi0:0.deviceType = "disk"
scsi0:1.present = "TRUE"
scsi0:1.fileName = "sdb.vmdk"
scsi0:1.deviceType = "disk"
scsi0:2.present = "TRUE"
scsi0:2.fileName = "sdc.vmdk"
scsi0:2.deviceType = "disk"
scsi0:3.present = "TRUE"
scsi0:3.fileName = "sdd.vmdk"
scsi0:3.deviceType = "disk"
---x------cut----------cut-----x------cut------------x----

VMWare player automatically added a few lines to the blah.vmx file after my lines too:
---x------cut----------cut-----x------cut------------x----
scsi0:0.redo = ""
scsi0:1.redo = ""
scsi0:2.redo = ""
scsi0:3.redo = ""
scsi0.present = "TRUE"
---x------cut----------cut-----x------cut------------x----

sp5 for BES v4.0 :: many more fixes than other service packs

2006-08-24T17:49:00.000-04:00

I finally got some time to myself today: my son is asleep when I am
back from work. So, I went ahead to download the release-notes for
service pack 5 for BES v4.0. For work, I administrate a BES 4 server
currently on v4.0.4. To my big surprise, there's tons of fixes in
this service pack, many are for bugs for old sp1/sp2 and many are for
serious BES performance/reliability issues. I don't recall this many
fixes in release notes for other service packs. (To be fair, I started
with 4.0.2) I guess this could be that RIM finally can concentrate on
its product and customers, after settling the patent lawsuits.

on-call duties :: to pay or not to pay

2006-08-24T17:23:00.000-04:00

at work, we are talking about setting up a duty pager and a group of
us will join the rotation to be on-call 24x7x365. Some of us don't
think, or think the boss won't like, that the rest of us actually want
to be compensated for the on-call duties. To me, it makes sense to be
compensated for the inconvenience of carrying it, and also be
compensated for the excessive time working off-hour by comp-time or
cash.
A strong counter argument is that labor laws categorizes most of IT
professionals as exempt, in that we can't bill for hours outside a 8-5
work day. Many employers forget to see/emphasize/respect the other
side of the token. That is, the exempt status would be void if an
employer ducks our pay if we take a few hours off here and there.
Most don't deduct pay, but they frown upon and ask you to make up the
time. As a result, most of us opt not to take random hours off. Plus,
at smaller places, your chores pile up and your projects get delayed.
In the end, your performance (real or in the eyes of the boss-man) is
hurt, so are the salary and upward movement on the corporate ladder,
in turn.
It gets even murkier to us laypersons if on-call duties is actually
part of your job description. Does that entitle an employer to get you
work whole weekend plus a full week? what do you think?

fedora core 5 -- better SELinux policy

2006-08-20T23:44:00.000-04:00

Tonight I created a virtual machine inside my vmware player, with 256M ram and 10G hard disk, to install Fedora Core 5 from the DVD Andy@redhat handed out at the Intel/Redhat breakfast seminar series. The installation wizard has some good selection menus for SELinux set-up.

This reminds me of what I read in FC5's release notes that the targeted policy in RHEL4 (or CentOS 4 or FC4) is now deprecated. Plus the integration of Zen and JBoss, FC5 now began to look like a larger bite. Ken @ worldspan is certified for RHL 8 and he said he will take RHCE once RHEL 5 is out. wonder if I should pair up with him to get my heart-thorn over with.

Sybase 15.0 certification is ready

2006-08-20T23:37:00.000-04:00

I received an email from Sybase mid week. Seeing it says 'free
certification voucher', I forwarded to my former co-worker, who is a
Sybase DBA hunting for a job. I thought it'd be a nice addition to
have a renewed certification on the resume, plus a good distraction
against anxiety while staying home.
After sending the email, I click on 'display image' inside gmail. now
I knew it was not "free' free, but free after taking a few prep
courses offered by Sybase for a few grands. Don't I feel like a
jack-ass?!

Redhat/Intel breakfast : Linux para-virtualization on Intel dual core processor

2006-08-19T15:54:00.000-04:00

This past Tuesday, I attended the breakfast seminar series of Linux Virtualization and Dual Core processors at Westin near I-285 and peachtree-dunwoody interaction. Once the Westin waitress followed me into the breakfast area and lifted several party tray covers, I instantly understood why the invitation email said 'hot breakfast'. The breakfast consists of hot scrambled eggs, steamed sausages, fried bacon, oven-warm bread, pancakes, and ice-chilled fresh orange juice. "Nice", many said.

Ken from WorldSpan sat next to me. He's a programmer turned systems administrator. He told me that the WorldSpan building at I-75 and I-285 I drive by everyday is the headquarters. Its data center is down at the airport and it used to be part of Delta. They have mostly mainframes with code written in the '60s, a few Solaris servers, and a few Linux servers.

The Intel speaker is OK. A few points I take home are:

* AMD has a very very small footprint in terms of market share. Based on the noise it generates in the media, I thought it is making strides.

* Intel's dual Core processors will come hard and heavy to the market in the coming
years.

* Intel led AMD in 20 of the 22 industry benchmark results.

Andy from Redhat presents the Xen in Fedora Core 5 (and the coming
RHEL 5, beta1 at the end of August). A charming Brit. Xen basically
allows Para-Virtualization, with which the guest OS is aware of the
fact that it is a guest and its kernel is also modified to expedite
its access to raw hardware, esp. CPU/memory. One thing it doesn't do
is the bandwidth allocation for guests.
Two cool features I like about Zen (or enhanced feature available via
RHN, Redhat Network, subscriptions) are:
* snapshot of a guest. such a snapshot can then be used to restore or replicate
* a guest can be migrated, alive, to another physical host, with
minimal downtime appreciable to the end users.

I got a FC5 DVD (good thing I didn't start my download a few days ago)
from a pile Andy put out. Ken and I recalled the fact that we used the
3.5" floppy diskettes to install Debian 0.92 in '96 onto a 486dx.

Farewell group lunch at Pappadeaux

2006-08-18T22:54:00.000-04:00

I attended a farewell group lunch today at Pappadeaux . The farewell is to a manager who was replaced. He moved down from MD to GA a year ago, after some sweet&heavy courting.

This is the first time I ever entered a Pappadeaux. It is trendy, expensive, crowded at 12:00pm-2:00pm. Feeling a bit cool, I noticed the sweat on several waiters' foreheads. Entrees started at $15 each. No free bread or butter while waiting. Half of us finished, while the other half waited over 45 minutes to get our plates.

I planned for an hour and ended up spending two hours there for a rather simple lunch. It is not like anybody ordered a full course of meals. Most ordered just a salad, a gumbo bowl, or a sea-food plate. Fortunately my boss sat next to me and we discussed some project priorities. So, I count that as 'working' to cut my lunch time back to an hour, the originally allocated time slot. It is Friday after all, who wants to start his/her weekend one hour late?

For a person who religiously brings brown-bag every day and often frets over a $12 dinner entree at one's favorite authentic ethnic restaurant, I ordered a fried shrimp & scallop lunch plate for $14.99. There's hardly anything you can order to get below that price anyway. I know some at the table earn much less than me and have much more obligations than me. Not sure how they feel about the price tag. Everybody looked and acted happy and cheerful.

Near the end of it, the waiter was about to get checks for everybody. The supervisor of the parting manager suddenly said, "let me pay for all." Joy and shriek across the tables. Many regret that they didn't order a lobster to-go :)

Blog is not part of the Web -- overengineering at Google ?

2006-08-18T22:18:00.000-04:00

More often than not, when I search for a non-mainstream topic, I couldn't find it in Web or Group using Google search. However, search by 'blog' may get me a boat-load of results. I wonder why Google doesn't integrate blogs into its web search.

Blogs are mainly addition/appending, just like regular web page. Therefore blogs should be friendly enough to lend themselves to the same storage and indexing algorithms Google uses for web contents. Why bother to separate them into two tab? Even worse, 'blog' is now a few clicks away, instead of just another tab. Is there some over-engineering sprouted at Google ?

Citrix Netscaler :: wildcard not so wild for Content Switch (continued)

2006-08-18T17:09:00.000-04:00

I dialed into a conference call with the Citrix lady this afternoon. She told me that answers she got internally are that the RegExp pattern "jack*.blogspot.com" wont' work at all for any builds for current Netscaler 6.1 release. What a disappointment! Her inquiry on whether the coming 7.0 release, currently in RC-CR phase (RC stands for release candidate while CR stands for controlled release), will have it was redirected to Citrix developers in India. Answer is pending,
since people are sound asleep at that part of the world right now.

Not sure why this would be a question for developers instead of for some project manager here in the states. You'd think there got to be some form of project plan or use cases or feature lists or fixed bugs for a major release now in RC mode? Should I hold that against Citrix's development or PM process?

Could I just read too much into it, as usual? No. This is by stark contrast with F5's much more mature and systematic approach towards release management.

certification or a M.S./MBA

2006-08-18T05:37:00.000-04:00

At work, a biz discussion turned into a chit-chat this morning. A bit of surprise to me, my fellow IS engineer has decided to give up his SCJP4 certification prep and started to prep for GMAT instead. Being a CCNP & MCSE, he came to a painful conclusion that the certification path is too tiresome since they expire (in real term, as well as fading in your mind) before you can use it for work or for job hunting.

Over years of turmoils about to pursue certifications or not, I total agree. Unfortunately, I have a MS in MIS from a prominent school. A few years later, I still don't see a door opened for me to the management track. So, I'd say MS doesn't work, since it labels you as a techie more than anything else. A techie needs something else to package/wrap/color himself or herself in order to impress your future employer that he/she is biz/management material. Recently, a few of my techie friends spent a lot (tuition/time/energy) to get their MBA. That seemed to have paid off. All found positions that a techie would never have dreamed of getting.

ethereal renamed to wireshark -- in June 2006

2006-08-18T05:03:00.000-04:00

Yesterday, while reading a RHSA (redhat security advisory, I signed up for notification emails) regarding ethereal, I was surprised to find that ethereal was renamed recently to 'wireshark'. not sure why. since it is such a powerful & popular network troubleshooting tool, I imagine there'd be a lot of people updating their resumes to insert '(wireshark)'.

It was back in June that ethreal was renamed, a tool I use on a daily basis (if not hourly). Am I too aloof to what's going on outside my little "circle of life"? Maybe it is about time for me to get onto some RSS feed or radio feed, so I can automatically pick up such vibes w/o conscious efforts.

citrix netscaler :: wildcard not so wild for Content Switch

2006-08-18T04:33:00.000-04:00

With my son going to church group meetings, I had a bit free time to resume playing with the citrix Netscaler EE-9000. I set up a policy using in-line expression to say I want "http.header.host == jack*.blogspot.com", then I created a content switch virtual server to associate the policy with a LB virtual server I created earlier. Simple enough, huh? Not really, all attempts to hit the CS-LB service (firefox, IE, curl) were met with "500 service not available error". Ethereal dumps on the client and tcpdump dumps on the Citrix box couldn't give me anything wrong with communication either.

Combing through the ICG (Installation & Configuration guide), I found such an error usually is due to the fact that an underlying feature is not enabled. I went back to the system node on the configuration GUI. And sure enough, 'content switching' is not checked by default under 'Features'. (while writing this, I am thinking, would it be better to shadow it out if such a feature is not enabled!) Checked & saved. Still got the same error. reboot the box, delete/recreate the policy & cs virtual server, all to no avail.

I cried for help by calling up my friend. He in turn got his lady friend, a Citrix SE, to chat with me. Off the bat, she told me that she's young with the company and may not know all the answers. I said to myself, "oh lord...sigh...at least she's honest." After a few go-to-meeting sessions and hauling other SEs into it, she finally broke it to me, "no, no wildcard would work for content switching." Great, isn't it?!

So, she and I went down the path to make-do with the limit set of operands on the GUI, trying to come up with a compound regular expression to mimic this behavior. None could really works so far. She kept telling me that newer builds won't help since they only fix bugs and won't change how RegExp works. She also told me that this IS how a PERL Reg object works for Citrix Content Switching. I was smiling along, begging to differ as a certified PERL programmer.

At this point, I am somewhat perplexed since it is hard for me to believe it didn't have true regular expression support. My employer opts not to use sub-domains or URL to divide traffic to load-balanced clusters. Instead, the sales/marketing geniuses want the customer to have any URL they want, only to find us engineers and architects scrambling to direct requests to these URL to a proper LB cluster. Sina and Google are using Netscaler in thousands. Does it mean that they don't need wildcards?

Employer questionnaire

2006-08-16T21:37:00.000-04:00

While stir-frying long beans for dinner, I received a call from a recruiter named Jack. He told me that an employer wants an interview and wants me to fill out some simple Q/A. I could tell that he was disappointed that I was not as excited as he was. Plainly, I asked him to send it my way via email then back to cooking. Barely in time to save my stir-fry long beans from burning into dark sticks.

Not sure why he just sent an email saying the same thing w/o any interaction from me. Human nature, or occupational hazard? After dinner, I retreated to my study and checked my inbox. It turned out that the Q/A is not for HR, but simple technical questions regarding Linux/Solaris operating systems. I answered them. To be sure, I googled to double check myself, before I shoot it back to the recruiter.

Three thoughts:

If I know where/how to find the info and can use the info properly, would it be enough ?
I am certified on both Linux and Solaris and have years of professional experience with them. However, I haven't touched either for a year. Who am I to remember three -v is needed to get the most verbose output from openSSH? Is this a sign that I should run away from this employer if this is his/her expectation?
If I do a 'man ssh' on my Linux Desktop to get that the exact answer is '-vvv' instead of 'a few -v' I initially wrote down, am I cheating ?

took smartcert.net practice exam for MCSE (70-290)

2006-08-16T21:28:00.000-04:00

I ran into a link to smartcert.net today. My son is taking his nap thus I have half an hour to forty-five minutes to kill. So, I think, why not. I took the example exam for 70-290. the quiz session won't start within Firefox, just spinning forever. It took me a while to find IE, since I've deleted the shortcuts on the desktop and taskbar, since I rarely use it if not forced.
The test itself is pretty good, I'd say. Not sure whether they'd be this many ntbackup/vss question though. Questions are pretty long and the 2-minute clock is literally ticking in my face! got to read quick.

scsi disk image by QEmu for VMWare Player for a Windows 2003 guest

2006-08-16T21:12:00.000-04:00

I need to have at least three spare disks to check out the RAID5 in Windows 2003 server, a guest OS running inside VMWare player with Windows XP as the host OS. I usually use QEMU's qemu-img command to create IDE disk images. However, VMWare player declined to take IDE2, and declined to allow me to use IDE1:0 (the cd-rom) as a disk device.

So, I thought I'd just use SCSI disks then. Not so fast, they say. QEMU doesn't support SCSI disk image. A Romaian hacker reported an altered qemu-img to create SCSI disk image. Unfortunately, his link on 'hack a day' returned a 404 error. not sure how far I want to go to get more than three spare disks for the RAID5 configuration practice needed for my MCP 70-290.

Ignoring the warning about the disks are IDE disks labeled as SCSI disks, I booted up the server. found-new-hardware wizard started. a driver was found but an 'aged' one, thus Windows 2003 server fails to install the new hardware named 'Buslogic multimaster scsi adapater' or something like that. bummer. I guess I'd just give it up for now, and take their words for it?

useless junk with important information

2006-08-13T13:35:00.000-04:00

Finally got some free time today to clean up the junk pile on my desk. For a few weeks, I could barely find enough space to put my hands on the keyboard to type.

Majority in the pile are actually important mails, such as a few statements from brokerages, bank, premium increase of home insurance, termite warranty payment stub, 2006 county tax notification, and on and on. I never can draw a clear line whether I should thread them right away or keep them for a year just in case. If I keep, then I need to sort them now and remember to come back to sort out them for threading or archiving yet again. what a bore.

All these smart people on earth, why nothing has been invented for this mess of junk?! Or the blame falls right on me. That is, I have too many possessions and decide not to give up an inch or thread, depending what it is. If I were able to follow the spirit of Zen, I'd be able to live with myself with much less from this earthy world, I'd be just as happy through the warranty period of this bio-form of life in this cycle.