which is eth0 on a PowerEdge 6850 running CentOS 4.4/i386

When setting up the refurbished Dell PowerEdge 6850 I purchased recently, I noticed it is newer and has a few oddities

• no video connectors at front panel nor the back. SVGA video only on the DRAC 5 card.
• a dual-port Intel Pro copper Ethernet card
• PERC adapter is of 4e/DC instead of the 4e/Di with the other two PE6850 bought new from Dell last summer.

A minimal installation of Linux is done smoothly using a CentOS 4.4/i386 server-cd. Well, I did run into one hitch that caused problems getting the network connectivity going. Initially, I didn't have confidence in ports on both ends of the cat5e cable:

• The "new" port is in an aged Cisco Catalyst 2900XL switch bought off eBay in 2000. My boss told me several times one port is known to be bad on this switch, since I joined the company a year ago. I used twelve free ports so far without problem. This port is one of the last two "free" ports on this 24-port switch.
  
• The port was in VLAN #1 to mirror two neighboring ports and had not been in active use. I programmed the switch such that the port now participates VLAN#3 instead of VLAN#1, and no longer mirrors any port.
  
• I was hoping to see a steady green LED and a blinking yellow LED on the NIC, by switching the cable from port to port on the server. The results were not telling. The first embedded Broadcom port had that, but no connectivity. The two Intel-pro ports had only green LED, no blinking yellow LED nor connectivity. I later came to the realization that the fast switching from port to port won't work well, probably due to ARP cache and such on the switch.

After a bit of frustration, I decided to leave the cable in the first embedded port and to do some serious work on the server side to make it work. Need a quieter place to think and type, I borrowed a serial cable to access its serial console. All the boxen I set up have serial console enabled in /etc/inittab and GRUB or LILO. For Dell PowerEdge servers, with BIOS-redirect-to-serial enabled too. Too busy to bargain hunt a serial condenser. I used Digi board before and it is nice. However, Digi board is too expensive a proposition for the current employer who's satisfied with the capability to call the ISP technician to hook up monitor and keyboard to their servers.

Once sitting down in the much quieter guest working area, I immediately put the pieces together and realized that the embedded Broadcom Ethernet ports were enumerated as eth2 and eth3, while the two ports on the Intel-Pro PCI-e card were as eth0 and eth1. Here is my "detective work":

grep eth /etc/modprobe .conf
alias eth0 e1000 alias eth1 e1000 alias eth2 tg3 alias eth3 tg3
dmesg
e1000: 0000:17:06.0: e1000_probe: (PCI:66MHz:64-bit) 00:04:23:cd:6e:b8 divert: allocating divert_blk for eth0 e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection ACPI: PCI interrupt 0000:17:06.1[B] -> GSI 133 (level, low) -> IRQ 50 e1000: 0000:17:06.1: e1000_probe: (PCI:66MHz:64-bit) 00:04:23:cd:6e:b9 divert: allocating divert_blk for eth1 e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection tg3.c:v3.52-rh (Mar 06, 2006) ACPI: PCI interrupt 0000:14:02.0[A] -> GSI 64 (level, low) -> IRQ 201 divert: allocating divert_blk for eth2 eth2: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:13:72:40:e1:bd eth2: RXcsums[1] LinkChgREG[1] MIirq[1] ASF[1] Split[0] WireSpeed[1] TSOcap[0] eth2: dma_rwctrl[769f4000] ACPI: PCI interrupt 0000:14: 02.1[B] -> GSI 65 (level, low) -> IRQ 209 divert: allocating divert_blk for eth3 eth3: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:13:72:40:e1:be eth3: RXcsums[1] LinkChgREG[1] MIirq[1] ASF[0] Split[0] WireSpeed[1] TSOcap[1] eth3: dma_rwctrl[769f4000]
grep HWA /etc/sysconfig/network-scripts/ifconfig-eth?
ifcfg-eth1:HWADDR=00:04:23:CD:6E:B9 ifcfg-eth2:HWADDR=00:13:72:40:E1:BD ifcfg-eth3:HWADDR=00:13:72:40:E1:BE ifcfg-eth0:HWADDR=00:04:23:CD:6E:B8

With the card-to-driver mapping in dmesg and in the modprobe.conf, it is easy to see the embedded Broadcom ethernet ports are mapped to eth2 and eth3, while the add-on Intel-pro ports are mapped to eth0 and eth1. The mac addresses in the 'dmesg' and the HWADDR recorded in ifcfg-eth1 through ifcfg-eth3 files hammered down the last nail. Without the last nail, it'd rather hard to tell if all ports is of the same make and model.

mii-tool sorta told the tale as well. lspci can help too.
eth0: no link eth1: no link eth2: 10baseTx, HD, link ok eth3: no link
Once I duplicated the IP/netmask on eth0 to eth2, then ifdown eth0 and ifup eth2, all is well now per ' mii-tool'
eth2: negotiated 100baseTx-FD, link ok

It'd be an interesting exercises to force eth0 to be the first embedded Broadcom port. It makes more sense for Linux to enumerate the embeded ports first, ethernet or not. What do you think?