How to enable tftpd or TFTP service on OS X or Mac

I needed to update firmware on a TP-LINK router on a recent trip. One option is to download the firmware bundle and place it on a TFTP server accessible by the TP-link router. At the time, I had only a MacBook Pro running OS X 10.11.5 (El Capitan). 

Knowing OS X is BSD-based, I assume tftpd or tftp service is baked in. It is! Googling yielded a few articles on how to enable tftpd or tftp service on Mac or OS X. None is immediately reusable, by which, I mean to copy+paste :-)

To benefit those who wants to run TFTP service on Mac or OS X, below is a simple MOP (manual of operation) that worked for me. The instructions below assume you have a terminal open already.

• to enable and run tftpd or tftp service. '-F' to ignore the Disabled key for the service.
  

sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist

• to verify it is running properly

• to add a test file into the /private/tftpboot/, which is the tftp service root directory.
  

date > /tmp/test1

sudo cp /tmp/test1 /private/tftpboot/

• to run tftp client program to retrieve the test file

tftp localhost

get test1

quit

• to serve any file via TFTP service now running on your Macbook, simply copy it to /private/tftpboot/

sudo cp file2 /private/tftpboot/

• to clean-up
• to purge files you no longer wants to serve.

sudo rm /private/tftpboot/test1

• to unload the service configuration and stop the tftpd or tftp service

sudo launchctl unload /System/Library/LaunchDaemons/tftp.plist

The above assume you need tftpd or tftp service just to run one time or other ad-hoc purposes. If you like to keep tftpd running, you may need to load the tftp service configuration using a different option. Namely, '-w' instead of '-F', to override the Disabled key for the service.

sudo launchctl load -w /System/Library/LaunchDaemons/tftp.plist

how to profile a python program to pinpoint performance bottlenecks

Recently, I wrote some python programs to help integrating Cisco's UCS (Unified Computing System) with our homegrown infrastructure backends.  The programs run ok, though slower than we expected.

The python programs run from a CentOS-6.5 virtual machine on my MacBook (Mavrerick) against the famous Cisco's UCS Platform Emulator (UCSPE v2.2 1b) also running as a virtual machine on the same MacBook.

To pinpoint performance bottlenecks in these python programs, I started with the standard Python profiler.

# to get a text output of cpu time stats

python -m cProfile myutil.py  > /tmp/myutil.py-cProfile.out

# to ignore calls spent less than 1s in tottime.

grep -v -P "^\s*\d+/*\d*\s+0\."  /tmp/myutil.py-cProfile.out

With the above, we immediately noted that a sigificant percentage of cpu time (tottime) was spent on FindClassIdInMoMetaIgnoreCase and String.lower(). 

         1849158 function calls (1848366 primitive calls) in 109.533 CPU seconds
   Ordered by: standard name
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   863     45.159    0.052   88.026    0.102 UcsBase.py:553(FindClassIdInMoMetaIgnoreCase)        1     2.361    2.361    2.371    2.371 urllib.py:23(<module>)
      764    8.904    0.012    8.904    0.012 {built-in method read}   11825    6.999    0.001    6.999    0.001 {method 'append' of 'list' objects}
1735232  42.783    0.000   42.783    0.000 {method 'lower' of 'str' objects}

Looking at UcsBase.py closely, I was able to locate a silly performance bug and fixed it. As you can see in the new results below, the quick fix reduced the number of calls to String.lower() to half!

    978415 function calls (977623 primitive calls) in 48.234 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

      863   12.825    0.015   33.887    0.039 UcsBase.py:553(FindClassIdInMoMetaIgnoreCase)

      764   11.436    0.015   11.436    0.015 {built-in method read}

 873738   21.037    0.000   21.037    0.000 {method 'lower' of 'str' objects}

It still seems too many though. Trying to reduce calls to lower() everywhere in UcsUtils.py and UcsBase.py didn't reduce much. Now we need to identify which/what calls contributed to this many invocation of String.lower().

pycallgraph seems to a magnificent tool to examine a running python program to see which calls which and for how many times. Unfortunately, I couldn't get it to work on centos-6.5 (Python-2.6.6). I ended up resorting to gprof2dot. 

# to capture profiling results in pstat format

python -m cProfile -o /tmp/python-cProfile-SDK_orig.pstats myutil.py

# use gprof2dot & dot to graph into a svg

python ~/gprof2dot/gprof2dot.py -f pstats /tmp/python-cProfile-SDK_orig.pstats | dot -Tsvg -o  python-cProfile-SDK_orig.svg

Safari & Chrome open up the .svg pictures just fine. Isn't it a beauty! It is very nice to scale up to zoom in on these pictures. After all, .svg stands for Scalable Vector Graphics image :-)

From the pretty pictures, I can see most String.lower() calls were indeed by FindClassIdInMoMetaIgnoreCase().  Oh, well, guess I'll have to tackle calls to FindClassIdInMoMetaIgnoreCase() another day.

If there's interest, I'll follow up with a post on how to set up these python performance profiling utilities. Please let me know what you think, in the comments below.

how to bring VirtualBox out of 'Guru meditation'

I have been using Oracle Virtualbox for work for quite a few years now. This time around I have a few relatively big virtual machines running. A Netapp simulator and a UCS platform Emulator. I keep switching between the two, by pausing one to free up memory to work on the other.

Yesterday, the UCS one suddenly can not be started again from the Virtualbox Manager GUI. All possible actions for the vm are grayed out, except that I could review logs. 

Under the vm icon, it says 'Guru meditation'. The mouse-over hint said 'session locked'.  

The actual directory looks pretty clean. The Logs/VBox.log indeed has an entry confirming this unfortunate 'guru meditation' state.

41:15:26.089214 UIMediumEnumerator: Machine (or snapshot) event processed, ID = d2b17907-9bef-4e1d-b755-d6be33e5570a

41:15:26.089683 UIMediumEnumerator: Machine (or snapshot) event received, ID = d2b17907-9bef-4e1d-b755-d6be33e5570a

41:15:26.089856 UIMediumEnumerator:  Old usage: 0210f54c-ea12-4a15-bc4d-f8fa690759a8, 658b6fcb-ee59-4065-abd4-c3aeb1eab3a8, 6d3ed09f-42cd-4b36-8652-b4a75e8181bd, 7b83e1e3-ffc2-445d-a9fc-9d6586833b4e, 8589d5fd-4ff3-4c80-b915-7cee87a2e974, e6964c46-790b-4aff-82fb-b22a83792882, f623a9aa-4b0c-44a0-b34c-34d9a3f09dd6, fea35c17-bc46-4d31-84b0-08f4620e7cc4

41:15:26.101502 UIMediumEnumerator:  New usage: 658b6fcb-ee59-4065-abd4-c3aeb1eab3a8, e6964c46-790b-4aff-82fb-b22a83792882, f623a9aa-4b0c-44a0-b34c-34d9a3f09dd6, 8589d5fd-4ff3-4c80-b915-7cee87a2e974

41:15:26.101557 UIMediumEnumerator:  Items excluded from usage: 7b83e1e3-ffc2-445d-a9fc-9d6586833b4e, fea35c17-bc46-4d31-84b0-08f4620e7cc4, 0210f54c-ea12-4a15-bc4d-f8fa690759a8, 6d3ed09f-42cd-4b36-8652-b4a75e8181bd

41:15:26.113783 UIMediumEnumerator: Machine (or snapshot) event processed, ID = d2b17907-9bef-4e1d-b755-d6be33e5570a

41:15:26.135516 UIMediumEnumerator: Medium-enumeration finished!

41:21:23.219316 ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={8ab7c520-2442-4b66-8d74-4ff1e195d2b6} aComponent={Console} aText={Invalid machine state: GuruMeditation}, preserve=false

I made a copy of the virtual machine directory, embracing for the worst.
Then it dawned on me that I may be able to bring it out of 'session locked' state via CLI, namely, VBoxManage. 

Viola!

Mac:/Users/jack/VirtualBoxVMs$ VBoxManage controlvm ucspe resume

VBoxManage: error: Cannot resume the machine as it is not paused (machine state: GuruMeditation)

VBoxManage: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component Console, interface IConsole, callee nsISupports

VBoxManage: error: Context: "Resume()" at line 118 of file VBoxManageControlVM.cpp

Mac:/Users/jack/VirtualBoxVMs$ VBoxManage controlvm ucspe poweroff

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

As soon as VBoxManage finished executing, all the grayed-out actions became available again on the GUI for this vm. I started it. 

The virtual machine came up just fine. What a relief :-)

In case more specifics help somebody, I am running Virtualbox v3.4.8 on Mac OS X 10.9.2 (Maverick). The two VMs in question are Linux 2.6 and FreeBSD.

how to manage SSL certificates for Tomcat using IIS tools

I worked in a shop where Windows & IIS servers dominates the operations. A few third-party applications were hosted by Tomcat, which terminates SSL connection as well. 

Engineers loathe the use of keytool, a java utility to manage certificate and keys. 

I first found that it is relatively easy to convert keystores from the SUNW format used by keytool to PKCS#12 (.pfx) format used by IIS, and back. 

This means one can easily follow normal procedures to obtain a new certificate and export it to .pfx format.

After tinkering with it a bit, I read the tomcat doc. It seems that Tomcat (at least for v6) can support PKCS12 format directly. 

Fixed: Cisco VPN client minimized on Windows 7 taskbar & disconnected

Cisco Systems VPN Client 5.0.0.7.0410 has worked for a month since I first installed it on Windows 7 Enterprise x64. One day, the VPN Client 'connect' window started minimized and stayed in the tool bar when launching. There's no useful choices in pop-up menus for the client window minimized on the taskbar and the VPN service (cvpnd.exe) icon in the Windows tray. Not choices were listed to 'minimize' , 'maximize', 'restore', or 'move'.  Since the Cisco VPN client was launched as 'disconnected', it was basically useless!

Googling found it is rather common a problem and people usually blame it on older versions of the Cisco VPN clients, latest Windows updates, etc. Originally I intended to write a javascript to launch the binary (vpngui.exe) with its window maximized. The javascript snippet I wrote in the past is not handy, so I continued my search a little bit more, only to stumble upon  this thread in a Microsoft forum. One of the answers pointed me to the direction. That is, the direct cause is rather simple. That is, window sizing somehow got messed up in vpnclient.ini under "C:\Program Files\Cisco Systems\VPN Client". 

vpnclient.ini looked like below. Instead of LogWindow* to have super big numbers, my copy has WindowX & WindowY assigned super big numbers. Once I changed WindowX and Window Y to match WindowWidth & WindowHeight, respectively, it started to launch the Cisco VPN client Connect window normally again.  I experimented a little bit more. It seems that the absolute number does not matter such, as long as they are not ridiculously big.

I have dual heads (dual display) setup. HP Elitebook laptop drives two HP L2245wg monitors. A similar thread on Virtual PC 2007 console disappearing problem suspected that dual display may have contributed to erroneous setting with this huge number. On the other hand, this work laptop is a managed node, so, I won't be surprised if something was pushed by our dear IT department to my laptop to cause this by accident or regression. 

 

Important thing is, the fix is simple and I'm happy for now. I will keep an eye on this to see if it gets messed up again.

Before (messed-up version)

---------cut-------------8<---------C:\Program Files\Cisco Systems\VPN Client\vpnclient.ini ----------------8<----------

[main]

ClientLanguage=

[GUI]

DefaultConnectionEntry=OneOfFortune500ciscoVPN

WindowWidth=485

WindowHeight=99

WindowX=4294935296

WindowY=4294935296

VisibleTab=0

ConnectionAttribute=0

AdvancedView=1

LogWindowWidth=0

LogWindowHeight=0

LogWindowX=0

LogWindowY=0

---------cut-------------8<----------------------------------8<----------

After (fixed and verified to be working).

---------cut-------------8<---------C:\Program Files\Cisco Systems\VPN Client\vpnclient.ini ----------------8<----------

[main]

ClientLanguage=

[GUI]

DefaultConnectionEntry=OneOfFortune500ciscoVPN

WindowWidth=485

WindowHeight=99

WindowX=485

WindowY=99

VisibleTab=0

ConnectionAttribute=0

AdvancedView=1

LogWindowWidth=0

LogWindowHeight=0

LogWindowX=0

LogWindowY=0

migrate to VirtualBox from VMWare Server : sadly & happy to

I have been a loyal VMWare server 1.0 user since it became free in 2005. It is used to build various sandbox virtual machines on my Windows XP laptop at work. The guest OS include RHEL, CentOS, Fedora, Solaris 10, Windows server 2008,  Windows server 2003,  Windows XP & Windows Enterprise 7. 

One feature I like a lot is its capability, with an easy-to-use GUI, to modify network settings to choose subnets, DHCP lease, and port forwarding, etc.

After upgrading my home desktop to long-awaited CentOS 6 early this month, I could no longer compile & run VMWare server. Since my home PC runs a mini lab of 3~4 nodes to play with cobbler & puppet, I need to find a solution that can play the same vmware machine (VMDK storage). Virtualbox from Oracle (Sun/Virtualbox.org) seems to be a good choice, given its popularity in LinuxJounal's 2010 poll. 

• The one RPM installation is great.
• It uses DKMS to compile and it compiles successfully in one shot.
• Guess OS support is up to date, with WIndows 7 and Server 2008. It does not differentiate versions of RHEL, RHL, Fedora though.
• PXE capability comes from the extension pack. you'll need to download & load. The extension pack license is free only for evaluation or personal use.
  
• Nowhere to find UI to modify network settings, other than choosing types of networking (NAT, bridged, etc.).
• add NAT adapter, only when you need this VM go talk with the world linked to the host
• add 'intnet' adapter and provides a meaningful name. This name apparently is used by Virtualbox to wire the other guest os into the same network, if they share the same name for their intnet adapter.
• Virtualbox won't import vmware guest images as is, since it imports only Open Virtualization format. On the other hand, it does support VMDK format, such that you can create a new virtualbox and choose to use an existing virtual disk (the vmware guest image you intend to import or run from virtualbox).
• Too bad the proxy preference do not extends to guests (behind the NAT). Instead, I had to configure http proxy at various places (profile.d, yum.conf, gnome) inside my CentOS 5 or CentOS 6 guests.

The other thing noteworthy is what type of controller Virtualbox associates your existing vmware disk (vmdk) to. It needs to match. Otherwise the working vmware image may fail to boot properly. It happened to me, when Virtualbox associated an IDE-based disk to SATA controller. Once I powered off the guest, then redid the association, I were able to boot up the image w/o a glitch from there on.

The installation & look & feel are pretty smooth & consitent & faster(?),on both a Windows 7 Enterprise x64 host (HP Elitebook laptop. 4G RAM) and a CentOS 6.0 i386 host (a Compaq PC, 2G RAM).  For now, I think I'll continue the path of migrating to VirtualBox. VMWare server will stay on for a bit, just in case I decide to roll-back.  I am definitely looking forward for taking advantage of the capability to take multiple snapshots, with notes!  It bothered me when I could keep only one snapshot and couldn't annotate what is this snapshot, with VMWare Server 1.x & 2.x

willing to pay extra to keep the service you like in business

Recently a premium local computer VAR went out of business. Many are sad.
The question is, are you, willing to pay extra to keep the services you like in business?
Take a moment & note your position in the comment below.

RHEL 5 is released today

Tonight, I read a review by Linux Format , the best-selling Linux magazine in the UK. The review is done for the latest RHEL 5 (Redhat Enterprise 5) which is released today. Technology enhancements are welcomed, such as those in SELinux administration and full integration of Xen virtualization as expected from Fedora Core 6.  The review also included an Q/A interview with Nick Carr, general manager, RHEL.

More noteworthy is that, Redhat Inc., as a company, has revamped its support, marketing, customer relation plan, and its relationship with the open source community. As stated in the review, the company has realigned the whole company to work better with the open source community as well as their enterprise customers. 

• The one page SLA
• server/client flavor versus the traditional WS/ES/AS flavors for RHEL3 and RHEL4.

first impression of Bestpractical's RT, an open source ticketing system

I heard quite a bit of good things about RT, an open source ticketing system.  Here is the first impression I had after retrieving the source tarball off bestpractical.com's web site.

The current stable version is 3.6.1.

Per README in the source tarball,

• "RT is an enterprise-grade issue tracking system. It allows organizations to keep track of what needs to get done, who is working on which tasks, what's already been done, and when tasks were (or weren't) completed."
• GPLv2
• commercial support is available including training and such: sales@bestpractical.com .
  

The installation seems a little too involving, even for a professional UNIX/linux system engineer.

• it requires Perl >5.8.3 with a lot of CPAN modules. To the developer's credit, a RT util tool is built to help get CPAN modules and compile them in place.
  
• It support a few RDBMS as its backend: MySQL/Postgres/Oracle/SQLite
• Apache with mod_perl (or FastCGI)

linttylog hang my Linux system on a Dell PowerEdge 6850 at first run

When troubleshooting a Dell PowerEdge 6850 server, I read on Dell's linux-poweredge mailing list that a small utility named linttylog may be useful to dump log from the PERC raid controller.  The function sounded a lot like what action=exportlog does using OMSA's srvadmin services. However, I was desperate enough to try anything reportedly working. So, without much ado, I downloaded the utility. It came in as an RPM and was a bit aged (probably out-of-date too).

syb04:/# rpm -ivh linttylog-1.00-0.i386.rpm
Preparing...                ########################################### [100%]
   1:linttylog              ########################################### [100%]

syb04:/# linttylog


        Serial Port Output History Handling Application 1.00 (Date 10/01/2003)
                Copyright (c) 2003 LSI Logic Corp.

Depending on the buffer size, it may take several minutes...Please wait...

        TTY History Updated in the file tty.log.

Exiting.
syb04:/# ls -ltr
====================================
The ssh shell session got disconnected from here. I couldn't log back in either. No response on serial console either. Nothing on the console screen. Nothing logged onto the remote syslogd server either, even though all kernel.* and all *.warn are redirected via /etc/syslogd.conf and verified to be working.

I had to power cycle the server to get it back online again. Once the server was back, I re-ran the utility successfully without any problem. What the ... The moral of this post is not to run linttylog on your mission critical system in production.

lay off ~= business strategy ~= choked by your own success

Four weeks ago, I called in to a meeting from home. In that meeting, John, my manager was let go, because the business strategy has renewed. The rest of us were assured in a later company-wide town-hall meeting, that "this is the team we are going forward with." John wasn't present. I was told later that he was asked not come in at all and his employment terminated that day. Why the rush? We were just about to roll out a major release on the ASP platform our team work on for the first third of our customer base. John was a hands-on manager, so the big void caused by his departure was apparent. 

Three weeks ago, Stevie, Matt, and Prakash, three more team members, were let go. Their positions were eliminated, because the management is confident that the latest release on the ASP platform is "feature-rich and robust enough to go forward without new development, for years to come". Again, my team or the left half of it were assured that "this is the team we are going forward with." Why the rush? We were yet to roll out the major release to the rest two thirds of our customer base. 

Does this second layoff justification suggest that our team choked by our own success? If less feature were rushed in by John and squeezed in by Sales/marketing and we didn't work days and nights and weekends to smooth the edges, we'd all still have jobs? I couldn't help wondering whether this why ISVs release bug-ridden code or QA/test inadequately. 

At any rate, the red PANIC button has been pounded on. Seasoned professionals as we are, we instantly hit the floor. Calling recruiters, line up old friends and contacts, update resumes and post them. It pretty much resembles a scene where rats rush off a sinking boat. 

Various news starts to tickle in through the cubicle walls. Hallway meetings anytime. Then, Matt's sense of humor struck me. He said, "Yes they were a bit clumsy firing people. It could only be a good thing. At least they are not used to this kind of thing or hadn't do it often enough to become proficient and smooth." I don't think I am thinking straight right now. I probably will add a few more comments when I have leisure to reflect, after I find my new job somewhere else.

how to verify ownership to Yahoo on wordpress.org blog server

The mischief caused by 404.php template is not a problem for authentication, when I stumbled into Yahoo!'s Site Explorer. To claim your site, you were instructed to place a special file with special content under / of the site. This way, one GET will do and have no problem with customized 404 pages. The latter is pretty common in use for sites managed by a CMS (Content Management System) or blogging servers. I wonder how come the smart engineers at Google decide to do two GET instead...

Once I added the required file on my wordpress.org blog server's / and clicked to continue, the next page asked me to keep the file there for 24 hours, till Yahoo's bots take their sweet time to crawl, literally! To a sharp contrast, Google's webmaster tool authenticates site ownership real-time, and sucks in sitemaps real-time too!

Placing a special file under / is the only way to authenticate your ownership on Yahoo Site Explorer. For millions of hosted sites (blogs or otherwise) whereas content owners don't have access to the /, they'd be out of luck. For now, at least. Hopefully when Yahoo! Site Explorer comes out of beta, they'd come up with a way to authenticate sites whose content owner have content-level access (META tokens, maybe?) instead of file-level access.

In comparison with Google's web master tools, Yahoo's site explorer is so spartan right now. Its own blog hasn't been updated for a few months now. I guess it is real beta then.

trial & errors :: SEO Dave's adSense-spiked themes

It's a God-send when I found SEO Dave's themes for wordpress.org blogs. He spiked some default themes with Google's adSense and optimized the placements. I gladly took his words for it, since he is a SEO consultant by trade.  This way, I can get new sites up quickly w/o laboring on SEO first :)

On a new wordpress.org server I set up recently, I attempted to unzip the five theme zip files obtained from his blog. I was surprised to be prompted to overwrite this file and that file. I said 'None'.  It turned out that only "connections", one of five zip files holds everything under a directory, as expected for a compliant theme package, which is to be extracted under /wordpress/wp-content/themes. No biggie, a little command line bash magic, I got it.

/wordpress/themes.php, the admin page to preview and activate a theme,  now showed all of them. Many, however, didn't show preview screenshots. Some of them even reported errors. Long listing of these files on the server revealed that the permission was too restrictive: 0700 for directory and 0600 for files. The root user's umask is 0022, so I am sure the restrictive permissions came from the theme zip, instead of from my sometimes overly-secure setup on the server. A few find and chmod later, all is well again.

There may be a bug in the original or spiked Blix theme, as it treats any new page as top-level, even if it is specified as child page for the ubiquitous About page. Same parent-child page relationship was handled properly by other themes such as Kubrick.

I posted a comment on SEO Dave's site in hope he may check it out, along with a wish of using Google's search box instead of the default search box.

how to verify ownership to Google on wordpress.org blog server

I was checking out Google's web master tools site the other night.  In order to verify the ownership, I opted to create a static HTML file on my site. Google failed to verify the ownership, stating it received a 404 error inside a 220-status page. I saw the file on the server and could browse to it properly using a browser too.

Puzzled, I looked at the server's access log. It turned out the Google attempted to retrieve two files. One was the long-winding name it stipulated. The other was the former file with its name prefixed with 'noexist_'.  The logic is clear: Google wants be sure the 220 code returned for the "magic" file is real, by verifying a different code (404 in this case) would be returned if the "magic" file doesn't exist.

66.249.74.2 - - [11/Jan/2007:20:12:20 -0500] "GET /google0467d40068c96de7.html HTTP/1.1" 200 59
66.249.74.2 - - [11/Jan/2007:20:12:20 -0500] "GET /noexist_0467d40068c96de7.html HTTP/1.1" 200 5306

The help page claims Goggle does HEAD only. This obviously isn't true, or isn't true any more, per Apache's access log entries above.

<META name="verify-v1" content="DGxlTrIdDwI9xwBYeYOMddr34POYb934o45vCpf3t+nvcI=" />
I ended up use the META tag instead. I copy+pasted it into /wp-content/themes/myTheme/header.php right before </HEAD><BODY>. This time it worked just fine.

The mischief was caused by a beautified 404 page generated by the /wp-content/themes/myTheme/404.php. Vaguely recalling Apache's manual pages do state that ErrorDocument directive and some other types of redirect tend to lose the original response status code, be it 404, 501, or 403.

AJP proxy enabled by default for Apache 2.2.3 on Fedora Core 6

As part of hardening an Apache instance on a new Fedora Core 6 Linux server, I commented out all _proxy_ modules in the main httpd.conf. When checking for syntax, however, I got
# /etc/init.d/httpd configtest
httpd: Syntax error on line 209 of /etc/httpd/conf/httpd.conf: Syntax error on line 2 of /etc/httpd/conf.d/proxy_ajp.conf: Cannot load /etc/httpd/modules/mod_proxy_ajp.so into server: /etc/httpd/modules/mod_proxy_ajp.so: undefined symbol: proxy_module

Surprised by the something actually required proxy_module, I took a look at the proxy_ajp.conf.
#cat proxy_ajp.conf

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

#
# When loaded, the mod_proxy_ajp module adds support for
# proxying to an AJP/1.3 backend server (such as Tomcat).
# To proxy to an AJP backend, use the "ajp://" URI scheme;
# Tomcat is configured to listen on port 8009 for AJP requests

This surely is nice. The building of mod_jk for Apache 2.0 on CentOS 4 has got old pretty fast. From a security standpoint, I'd  think this comes off some httpd-tomcat package. Not exactly!
# rpm -qf proxy_ajp.conf
httpd-2.2.3-5

For my purpose, I just commented out the LoadModule directive inside proxy_ajp.conf. However, it'd make more sense, if

• it comes off an optional module package, something like httpd-tomcat or httpd-ajp. Or,
• disabled by default.

weird wormholes :: set reply-to header to the list, or not to

Recently I asked on a mailing list how come I had to remember to copy the list address to the CC whenever I reply to a post. To my surprise, it turned to a big flame war of sorts. Unwilling to make any changes, the list administrator essentially asked both parties to go away, politely.

In its current form, you need to 'reply-all' to mailing list and the list administrator will make sure the poster's email address is in a special list such that the poster not receive two copies. Occasionally, someone may request his/her email address be added to the special list, yet again. What a hassle! Why the administrator wants all this hassle for nothing?!

It puzzled me in the past. It still puzzles me now. To my simple mind, the benefits of setting reply-to header to the list is so obvious:

• less keystrokes for the list members. Many modern MUA (Mail User Agent) has default to 'reply' instead of 'reply-to-all', esp. web-mail UI.
  
• less unnecessary mental note to reply-to-all instead of reply, or to copy list's address to CC if you already hit 'reply'
• no missing discussion
• consistently threaded discussion in archive and in live discussion. A post sent to the poster alone was often seen forwarded to the list, as an after-thought and after-fact good intention efforts. The thread is then broken, making it extreme difficult to follow a discussion you found an interesting excerpts by googling.
  

Instead,

• Some said it is hard to be done.
  
• Some insisted that it is philosophically wrong to reply-to the list. <= Hello, the purpose of subscribing to a mailing list is to publish to and read from the list, not to find sensible partners to conduct private conversations!
• Some insists reply-to-all is great enough and is the only sensible way, so they went ahead to hack their mail clients (MUA) to detect whether a message is post or private message and automate to save the unnecessary keystrokes.
• Well, maybe some of these people just accept the dysfunctional setting as the inevitable fact of life, and just find a get-around and moved on. If so, it is pretty sad.

Reflecting a bit more today on the observed need for some people to invent convoluted ways to satisfy themselves, I thought of what the agent in Men in Black says in a StarBucks coffee shop. There are people who can't control their own destiny or fate or fortune, and they are well aware of it. Instead, they opt to pay premium to select from a bloated feel-rich selection of lattes, as if they were the master of the universe.

downloads.wordpress.org needs a face lift

Over this past weekend, I built a new wordpress.org blog server on a new CentOS 4.4 server.  It seems the 'official'  plugin/theme repository page at http://downloads.wordpress.org needs quite a face lift. In its current form, the plug-in and theme repository page is

• spartan: It gives an ordered list of plugins and themes by name (and/or versions).
  
• poor in function:
  
• No descriptions to tell whether you need or like a theme or plugin. I had to just download it and install it, then read the descriptions from /wp-admin/plugins.php. Quite some wasted bandwidth on the server and time & efforts on the users.
• no option to download it all to try it out. Instead, you have to click on each one. The lack of description certainly exacerbates the problem.
  
• not up-to-date. For example, the link to tiger-admin plugin doesn't work. Upstream now has tiger-admin-v3.0.zip.
• malformed HTML. plugin page has an empty link text for tiger-admin plugin, causing the page to display funny in FireFox 1.5/Linux and FireFox 2.0/windows.
• Many of the themes don't show preview screens under /wp-admin/themes.php.  Yet to check whether the themes themselves are at fault, or the tiger-admin v3.0 theme is the culprit.
  
• no security or integrity assurance: no checksum or digital signature is provided to verify the authenticity and/or integrity of the file.
  

I guess I'll generate a more functional version and contribute to the site. Or alternatively, host a beautified version here myself.

how to coerce OMSA 5.1 to install & run properly on CentOS 4 (part III)

Here is the summary of what I did for OMSA 5.1 on CentOS 4.1 ( the white-box twin of RHAS 4.1)

For installation, on a CentOS 4.1 without OpenIPMI or net-snmp, you just need to

1. append /etc/redhat-release with "Nahant", RHEL 4's code name.

1c1
< CentOS release 4.1 (Final) Nahant --- > CentOS release 4.1 (Final)
2. start installation by running ./linux/suppportscripts/srvadmin-install.sh
After installation, answer NO to start start all services. Instead, conduct the following steps first:
1. insert a line 'test -e /dev/ipmi0 || mknod -m 0600 /dev/ipmi0 c 253 0' at the beginning of /etc/init.d/dsm_sa_ipmi. w/o it, /etc/init.d/dsm_sa_ipmi will fail miserablly. It seems that udev doesn't put /dev/ipmi0 back as expected.
22,24d21 < # my hack < test -e /dev/ipmi0 || mknod -m 0600 /dev/ipmi0 c 253 0 < #
2. insert '/etc/init.d/dsm_sa_ipmi start' at the beginning of 'start' section inside /usr/bin/srvadmin- services.sh
w/o it, srvadmin-services.sh start will fail with IPMI drivers fail to load. However, if you start IPMI manually by '/etc/init.d/dsm_sa_ipmi start', you'd be just fine.
241,243d240 < # my hack::ipmi failed to start when called from later this script < /etc/init.d/dsm_sa_ipmi start <
3. now start it all with /usr/bin/srvadmin-services.sh start
4. verify it with ' omreport chassis temps' and 'omreport storage controller '. These two rely on different things to work. I'll elaborate on this later.

The rc scripts are placed in the S50 instead of their own start sequence number mandated in the scripts. I corrected them manually with my understanding of chkconfig directives in the scripts. However, I didn't get to change run level to test whether that would guarantee OMSA startup succesfully, since all my boxens are in production. Besides, Hobbit Monitor will let me know soon enough upon such failure.

It is such a great relief to have constant & comprehensive monitoring against critical systems. As a system engineer, you don't have to make all these mental or paper notes to check this or that. A decent NMS will do its job to nag you (whether you like or not) when necessary.

how to change IP address on a Linux server :: a project?

I was asked this question once, "how do you change IP address on a Linux server?"  At its face value, it's a simple technical question on systems administration or network administration using a Linux server. So, my instant response is

• edit /etc/sysconfig/ifcfg-eth0 (or eth?).   or system-config-network TUI or GUI.
• adjust routes and personal fw on the host as needed
• ifconfig eth1 down && ifconfig eth1 up (or /etc/init.d/network restart)
• verify: ifconfig eth1 (to see if the new IP has taken effect)
• check & verify from remote host it is actually accessible (TCP/IP level, plus other services the server may provide)
• update DNS if applicable
  

To dwell on the question a little more,  and depends on the context of the conversation, the lines of questions, and who asks, I would expand a bit from the perspective of process management, configuration management, and knowledge management. In other words, I'd prefer to drive such a "simple" change as a small project. Thus, here comes the addition to my initial answer:

So, the actual change is simple. Depends on the function of the server and its inter-relationship with other systems, you probably need to manage such a change as a project. As they say, a little bit more of thinking and planning goes a long way.

• gather requirement from functional groups and management
  
• set a date to cut-over and obtain sign-off from stake-holders
  
• plan for it (procedures/fall-back plan/check-list/notification/etc.)
• communicate the planned change and its potential impact to stake-holders and end users
  
• dry run if necessary
  
• prior to cut-over: for days leading up to the cut-over, shorten TTL for A record for that IP, if it has DNS record and is accessed by DNS name instead of IP alone.
  
• prior to cut-over: check to be sure ACL or conduits or fw rules get updated on router/firewalls and systems the server gain access to or from
• during cut-over: keep team and stake-holders abreast of up-to-date status
  
• post cut-over: verify by going through your check-list to determine success
  
• post cut-over: enter change log for this event & notify users the cut-over is done
  
• post cut-over: alias or real NIC to keep servicing the IP for limited period. (could also just alias the new IP onto the NIC serving the current IP, the cut-over becomes switching who's alias and who's real)
  
• post cut-over: added DNAT or reverse proxy rules to catch traffic to the old IP and redirect/log/alert as necessary
• post cut-over: document this in the knowledge-management system: a scrapbook, a WIKI, a technical writer.
  

Hobbit Monitor :: how to report multiple temperture probe results

For Dell PowerEdge servers, OMSA can report temperature for multiple probes, notably, "BMC Ambient", "BMC Planaar", "BMC Riser", and one for each physical processor. I initially wrote my own extension script named 'temps' , which reports all probes under one 'status server1.temps green' call. It works for a test server after I did the following on the Hobbit server:

• defined a RRD graph section in hobbitgraph.cfg for all probes available on that server
• NCV_temps='*:GAUGE" in hobbitserver.cfg
• appended temps=ncv to test2rrd line in hobbitserver.cfg
• restart hobbitd server

When I deployed the same extension code to a different server, the limitation of such an extension became painfully obvious: the graph definition failed when a different server reports more or less probes. Since hobbitgraph.cfg's graph section is keyed to the test name in hobbitgraph.cfg, I can't add custom graph section per server configuration, not to mention it is not scalable!. As for alerting, I tried setting up thresholds on the server. However, it didn't seem to generate any alert even if the threshold is obviously surpassed. wonder if the 'status blah GREEN' reported by the extension script kinda blocked such centralized checking.

After posting the above questions to Hobbit's mailing list and got no answers, I took upon myself to find the answers. Good thing that Hobbit Monitor is licensed using GPLv1. I found the answers in the source code, rrd/do_temperature.c. [[ All hail goes to Open Source & Henrik! ]]

• the 'temperature' test is built-in. duh...
  
• if  'temperature' is used as 'test name' on the client, nothing needs to be changed on the server end.
  
• a lump-all status command will do. Format of the data portion is somewhat restrictive. Each probe needs to be in the format of '&green BMCambient 17 62'. 
  
• the test can be done locally on the client and the overall $status is reported by 'status server1.temperature $status'.

Wishes:

• It'd be nice to have such data report format documented elsewhere other than the source code.
• The current code (4.2RC1-20060712) didn't take input other than integer too well. It confused the hell out of it, in fact, to the extent it lump the whole long after $status as the probe name. Most of problems I experienced was actually the '.' in my data report confused Hobbit.
• It'd be nice to be able to specify threshold on the server centrally.