Fedora Core 6 security improvement :: refined SELinux policy managment and more

After a stock installation of the Fedora Core Linux 6, I am pleasantly surprised at the first boot, with the SELinux policy management GUI at part of the 'firstboot' program. You can collapse to set granular policies for services of your interest. For each service, all you need to do is to check or to uncheck policy items .Such a nice GUI management would definitely help greatly to unleash the raw power of SELinux to system administrators or even security administrators who would otherwise have to spend much more time to study, research, and to manage SELinux policy effectively.

The same SELinux policy management tool is readily accessible under System-> Administration->Security Level and Firewall. I guess the name of the applet has not really caught up with this new integration.

Of course, a system won't be secure unless it can be kept up-to-date with all good security patches and updates. In that regard, FC6 is improved as well. The update notification applet is improved to make the update process point-and-click compliant. You will get Package Updater running all GUI to assist regular desktop users to keep their system up to date.

kernel panic when using a 2G ramdisk on a CentOS 4.4 serverCD installation

I was burnt recently with a minimal installation using CentOS 4.4 ServerCD. CentOS project has this one-CD flavor for server use in addition to the regular current 4-CD full distro, mirroring Redhat Enterprise Advanced Server (RHAS). I found it awfully convenient to download only one CD of 530M, instead of the full set of 2.4G. The installations I do at work are all server installations, if my own Linux desktop is not counted. It has been a god-given ever since I first noticed its existence in CentOS 4.2 /iso folder.

Kernel paniced for a new production Sybase ASE database server when Sybase ASE 12.5 database engines started to zero out ~2G tmpdb devices (data & log devices). The hardware specs: Dell PowerEdge 6850 with 16G of DDR2 RAM and quad CPUs. The box survived my own stress test on CPU and disk. However, as an hindsight, no attempt was made to exhaust the memory.

The Sybase seemed to have finished its task, when the DBA called me saying he lost connection to the machine. I attached to its serial console and opened a connection using Kermit. Input was not rejected, but no response (no echo) from the system. After a quick reboot, I found nothing interesting in the log tail around the panic ( syslog.conf is set to write kernel.* to /var/log/messages). So, I decided to log the console session and asked the DBA to start Sybase. Sure enough, it happened again, with tons of 'out of memory' messages, followed by desperate yet fatal attempts by oom-killer to kill all processes to free up memory.
oom-killer: gfp_mask=0xd0
Mem-info: DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
cpu 4 hot: low 2, high 6, batch 1
cpu 4 cold: low 0, high 2, batch 1
cpu 5 hot: low 2, high 6, batch 1
cpu 5 cold: low 0, high 2, batch 1
cpu 6 hot: low 2, high 6, batch 1
cpu 6 cold: low 0, high 2, batch 1
cpu 7 hot: low 2, high 6, batch 1
cpu 7 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
cpu 4 hot: low 32, high 96, batch 16
cpu 4 cold: low 0, high 32, batch 16
cpu 5 hot: low 32, high 96, batch 16
cpu 5 cold: low 0, high 32, batch 16
cpu 6 hot: low 32, high 96, batch 16
cpu 6 cold: low 0, high 32, batch 16
cpu 7 hot: low 32, high 96, batch 16
cpu 7 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 cpu 2 hot: low 32, high 96, batch 16 cpu 2 cold: low 0, high 32, batch 16 cpu 3 hot: low 32, high 96, batch 16 cpu 3 cold: low 0, high 32, batch 16 cpu 4 hot: low 32, high 96, batch 16 cpu 4 cold: low 0, high 32, batch 16 cpu 5 hot: low 32, high 96, batch 16 cpu 5 cold: low 0, high 32, batch 16 cpu 6 hot: low 32, high 96, batch 16 cpu 6 cold: low 0, high 32, batch 16 cpu 7 hot: low 32, high 96, batch 16 cpu 7 cold: low 0, high 32, batch 16 Free pages: 12506892kB (12493888kB HighMem) Active:208204 inactive:791964 dirty:101636 writeback:26 unstable:0 free:3126723 slab:20412 mapped:563179 pagetables:DMA free:12564kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4 all_unreclaimable? yes protections[]: 0 0 0 Normal free:440kB min:928kB low:1856kB high:2784kB active:644696kB inactive:16036kB present:901120kB pages_scanned:1069066 all_unreclaimable? yes protections[]: 0 0 0 HighMem free:12493888kB min:512kB low:1024kB high:1536kB active:180824kB inactive:3159244kB present:16908288kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 3*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12564kB Normal: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 440kB HighMem: 1418*4kB 535*8kB 128*16kB 85*32kB 119*64kB 38*128kB 10*256kB 4*512kB 2*1024kB 2*2048kB 3041*4096kB = 12493888kB Swap cache: add 0, delete 0, find 0/0, race 0+0 0 bounce buffer pages Free swap: 16779884kB 4456448 pages of RAM 3963834 pages of HIGHMEM 299626 reserved pages 849369 pages shared 0 pages swap cached Out of Memory: Killed process 6586 (dataserver). oom-killer: gfp_mask=0xd0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 cpu 2 hot: low 2, high 6, batch 1 cpu 2 cold: low 0, high 2, batch 1 cpu 3 hot: low 2, high 6, batch 1 cpu 3 cold: low 0, high 2, batch 1 cpu 4 hot: low 2, high 6, batch 1 cpu 4 cold: low 0, high 2, batch 1 cpu 5 hot: low 2, high 6, batch 1 cpu 5 cold: low 0, high 2, batch 1 cpu 6 hot: low 2, high 6, batch 1 cpu 6 cold: low 0, high 2, batch 1 cpu 7 hot: low 32, high 96, batch 16 cpu 7 cold: low 0, high 32, batch 16 Free pages: 12507020kB (12493888kB HighMem) Active:46134 inactive:953784 dirty:101636 writeback:26 unstable:0 free:3126755 slab:20379 mapped:563179 pagetables:1589 DMA free:12564kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4 all_unreclaimable? yes protections[]: 0 0 0 Normal free:568kB min:928kB low:1856kB high:2784kB active:0kB inactive:659820kB present:901120kB pages_scanned:1845199 all_unreclaimable? no protections[]: 0 0 0 HighMem free:12493888kB min:512kB low:1024kB high:1536kB active:180824kB inactive:3159244kB present:16908288kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 3*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12564kB Normal: 20*4kB 3*8kB 3*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 568kB HighMem: 1418*4kB 535*8kB 128*16kB 85*32kB 119*64kB 38*128kB 10*256kB 4*512kB 2*1024kB 2*2048kB 3041*4096kB = 12493888kB Swap cache: add 0, delete 0, find 0/0, race 0+0 0 bounce buffer pages Free swap: 16779884kB 4456448 pages of RAM 3963834 pages of HIGHMEM 299626 reserved pages 849602 pages shared 0 pages swap cached Out of Memory: Killed process 6587 (dataserver). oom-killer: gfp_mask=0xd0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 cpu 2 hot: low 2, high 6, batch 1

The server was rebooted fine with sybase up the day before. Today, the DBA got some message claiming the tmpdb devices are not initialized. Since these tmpdb devices are created from scratch when sybase starts, it really didn't make sense! DBA decides to drop the current set of tmpdb devices and create a new set instead. Just when the new set is finishing initialization per Sybase logs, the system stopped responding. Before I lost my console, I were able to issue 'ps -e -o rss,pid,cmd,args|sort -n', 'free', and 'top'. Only about 6G is used out of the total 16G. So, the memory is not really exhausted. So, this got to do with how Kernel is carving and using the memory. Googling ' oom-killer: gfp_mask=0xd0' yields a single post which referred to a bug of sorts by a overly-zealous OOM-killer and how to disable it, due to problem with memory addressing scheume switch between high mem and low mem area.

For that, I wonder if ramdisk was claiming HIMEM. If HIMEM were not addressable or addressed improperly, that may have the appearance of OOM (out of memory). Once I started to examine '/var/log/messages' line by line, there, it was painfully obvious that hugemem kernel was not selected by CentOS ServerCD 4.4. The regular kernel-smp was selected.
Oct 23 12:35:30 syb06 kernel: Linux version 2.6.9-42.ELsmp (buildcentos@build-i386) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-3 )) #1 SMP Sat Aug 12 09:39:11 CDT 2006
Oct 23 12:35:30 syb06 kernel: BIOS-provided physical RAM map:
Oct 23 12:35:30 syb06 kernel:********************************************************
Oct 23 12:35:30 syb06 kernel: * This system has more than 16 Gigabyte of memory. *
Oct 23 12:35:30 syb06 kernel: * It is recommended that you read the release notes *
Oct 23 12:35:30 syb06 kernel: * that accompany your copy of Red Hat Enterprise Linux *
Oct 23 12:35:30 syb06 kernel: * about the recommended kernel for such configurations *
Oct 23 12:35:30 syb06 kernel: **********************************************

Once I downloaded the corresponding kernel-hugemem and rebooted the server with it, (grub.conf needed to be edited manually to to boot the new kernel by default). All is peachy again.

P.S. To answer Barry's question below, here is the 12-day memory RRD graph. The panic happened on the last Thursday or Friday, where lines were broken due to missing data. It would be nicer if Blogger actually allows image insertion inside comments. -20061030
P.P.S. Obviously the correct selection of the right kernel package and the maintenance thereof had been a problem/pain for Redhat's maintainers and customers. Per its release notes, the newly minted Fedore Core 6 (FC6) has one single kernel package per architecture. Kernel parameters optimized for different hardware specs (SMP or UNP, 4G/8G/16G/32G of RAM, etc.) would be set dynamicly upon boot, instead of being hard coded in different kernel packages: kernel, kernel-smp, and kernel-hugemem. Of course, with RHEL5 beta in the horizon, I guess we won't see such a change (in RHEL6) for a few years :(

step-by-step: how to host WordPress under Linux (FC5/RHEL4)

Why:

• on Blogger: missing certain nice features ( static page support, import/export). Or,
  
• on WordPress.com: don't like your own html get stripped off. Your little piggy-bank gets left empty since Google Adsense code get stripped or disabled. Or,
  
• want your own professional domain name and such. Or,
  
• simply love the experience of DIY and the freedom of customization
  

Assumptions & Prerequisites:

• Fedora Core 5 stock installation (All should be applicable to CentOS 4 or RHEL 4 as well, unless the FC5 SELinux changes get in your way.)
  
• apache (httpd package) installed.
  
• php package installed.
• mysql-server package installed.
  
• The stable release (latest.tar.gz) has been downloaded from wordpress.org and extracted under /var/www/html
• comfortable with MySQL and SQL commands.
• starter's knowledge of how a blog works.
  

Steps to follow:
# set up MySQL access

This is to set up a MySQL database dedicated for WordPress and a user to access this database from the Apache/PHP combo. The schema and data for the database itself will be populated by install.php later. The names used below assume you want user 'wp_master' to have all access to a local database named 'wordpressDB'.

• /etc/init.d/mysqld start
• /usr/bin/mysqladmin -u root password 'mysqlAdminpassword'
• /usr/bin/mysqladmin -u root -p create wordpressDB
• mysql -Uroot -p mysql
• insert into user (host, user, password) values ('localhost', 'wp_master', password('mastersecret'));
  
• insert into db (host, db, user) values ('localhost', 'wordpressDB', 'wp_master');
  
• the 'db' table entry should also grant the 'wp_master' user with most of the privileges to the 'wordpressDB' . So, I simply replaced the 'test' database entry therein, since I am lazy, plus I consider a wild-open test db a security risk.
  
• update db set host='localhost',db='wordpressDB',user='wp_master' where host='%' and db='test';
  
• commit;
• restart mysqld by '/etc/init.d/mysqld restart'
  

# set up MySQL PHP extension support (Otherwise you may get this error, "Your PHP installation appears to be missing the MySQL which is required for WordPress. ")

• yum -y install php-mysql
• /etc/init.d/httpd restart

# to set up wp-config.php with your own set of credentials for WP to use to access MySQL
cd /var/www/html/wordpress && cp wp-config-sample.php wp-config.php && vi wp-config.php
# RTFM using your favorite browser

point to your browser to http://localhost/wordpress/readme.html

# initialize/populate your new WP database. Do record the 'admin' account information generated.
point to your browser to follow http://localhost/wordpress/wp-admin/install.php

# Now, time to verify. If it worked, you'd see the default 'hello world' post using the 'default' theme.
point to your browser to http://localhost/wordpress/index.php

# to customize the blog right away, click on 'edit' link inside the blog and log on with the 'admin' account

# about them themes:
Only two themes are bundled (classic/default). To add a theme, find ones you like and download from a trustworthy source.

• wget http://downloads.wordpress.org/theme/pool.zip
• cd /var/www/html/wordpress/wp-content/themes && unzip pool.zip
• Now the new theme(s) would just show up if you go back to edit/presentation/themes
  

# Get them roamers home

• import existing blogs and comments from Blogger and other top blog sites.
• if new, register this new blog with technorati.com, weblogs.com
• new or updated, brief your readers of the changes [advance notice would be appreciated too]
  

fedora core 5 -- better SELinux policy

Tonight I created a virtual machine inside my vmware player, with 256M ram and 10G hard disk, to install Fedora Core 5 from the DVD Andy@redhat handed out at the Intel/Redhat breakfast seminar series. The installation wizard has some good selection menus for SELinux set-up.

This reminds me of what I read in FC5's release notes that the targeted policy in RHEL4 (or CentOS 4 or FC4) is now deprecated. Plus the integration of Zen and JBoss, FC5 now began to look like a larger bite. Ken @ worldspan is certified for RHL 8 and he said he will take RHCE once RHEL 5 is out. wonder if I should pair up with him to get my heart-thorn over with.

Redhat/Intel breakfast : Linux para-virtualization on Intel dual core processor

This past Tuesday, I attended the breakfast seminar series of Linux Virtualization and Dual Core processors at Westin near I-285 and peachtree-dunwoody interaction. Once the Westin waitress followed me into the breakfast area and lifted several party tray covers, I instantly understood why the invitation email said 'hot breakfast'. The breakfast consists of hot scrambled eggs, steamed sausages, fried bacon, oven-warm bread, pancakes, and ice-chilled fresh orange juice. "Nice", many said.

Ken from WorldSpan sat next to me. He's a programmer turned systems administrator. He told me that the WorldSpan building at I-75 and I-285 I drive by everyday is the headquarters. Its data center is down at the airport and it used to be part of Delta. They have mostly mainframes with code written in the '60s, a few Solaris servers, and a few Linux servers.

The Intel speaker is OK. A few points I take home are:

* AMD has a very very small footprint in terms of market share. Based on the noise it generates in the media, I thought it is making strides.

* Intel's dual Core processors will come hard and heavy to the market in the coming
years.

* Intel led AMD in 20 of the 22 industry benchmark results.

Andy from Redhat presents the Xen in Fedora Core 5 (and the coming
RHEL 5, beta1 at the end of August). A charming Brit. Xen basically
allows Para-Virtualization, with which the guest OS is aware of the
fact that it is a guest and its kernel is also modified to expedite
its access to raw hardware, esp. CPU/memory. One thing it doesn't do
is the bandwidth allocation for guests.
Two cool features I like about Zen (or enhanced feature available via
RHN, Redhat Network, subscriptions) are:
* snapshot of a guest. such a snapshot can then be used to restore or replicate
* a guest can be migrated, alive, to another physical host, with
minimal downtime appreciable to the end users.

I got a FC5 DVD (good thing I didn't start my download a few days ago)
from a pile Andy put out. Ken and I recalled the fact that we used the
3.5" floppy diskettes to install Debian 0.92 in '96 onto a 486dx.