Citrix Netscaler NS7000 : how to create a content switched load balanced farm/service (Part II)

In the previous part of this article, how to create a load balance virtual server (or 'lb vserver' using CLI) was described. Now we'll describe how to create content switching. Currently, as of Netscaler 7.0 beta, Citrix Netscaler supports only HTTP protocol for content switching.

Again, the logical, conceptual flow is clear and intuitive. However, the work flow is less so, the same as any object-oriented model applied to a more procedural mind-set or work flow.

In Citrix Netscaler's terminology, you need to create

• a Content Switching (CS) policy. The policy basics states a matching rule set that the HTTP request object must comply. Three basic

• domain, aka, URI
• URL
• any other request attributes/headers using regular expression or absolute matching.
  

• create a CS virtual server to direct traffic regulated by this policy to go to either individual service or a load balancing virtual server.
• a default mapping is allowed. The policy field would be 'blank'. It'd make more sense to actually call it 'Default' and displays it as such. This basically directs all traffic that fails to satisfy all the policy-vserser mapping to a default LB virtual server.
  

• Regular Expression can be created on the fly under the 'policy' node, or pre-created with a name from under 'System' node.
  

Citrix Netscaler NS7000 : how to create a content switched load balanced farm/service (Part I)

The Netscaler (now Citrix) load balancer has pretty clear conceptual, logical, and work flow. That is, well, to a system/network engineer like me anyway. To touch it off visually by a GUI, all this is neatly grouped under the 'load balancing' leaf node and the 'content switching' leaf node on the left pane of the Applet or Web Start GUI.

• define servers. A server is created by a pair of server name and its physical IP address: node8/192.168.88.8 node9/192.168.88.9 ... node100/192.168.88.100.
  
• define services. A service is created by binding the server IP address with a service port (port number and protocol, e.g. TCP/80, UDP/53, UDP/123).
  

• Take web service for example, a service named "prod_web_node1" can be created by binding TCP/80 with node1's IP address.

• Custom and default health monitored can be applied to the service.
• Multiple monitors can be applied a service with different weight and such.
• Web service is the most popular for an application switch or a load balancer. In fact, it is the only protocol rendered capable of content-switching by the Citrix Netscaler series.
  
• create a load-balancing "virtual server" (From CLI, it is 'add lb vserver'.) to front a group of services to be load-balanced.
  
  • weight can be applied to each LB member service.
    
  • load balancing algorithm can be chosen (many variations of round-robin, least connection, LTRM, etc.)
    
  • advanced setting can be adjusted here, such as timeout values for an idle server or an idle client.
    

Any 'lb vserver' can be utilized in two different ways:

• As a public service by itself. It can be exposed to clients as a service by assigning a virtual IP:Port. In other words, the load balanced service group becomes a service usable by the clients (directly). If load balancing is all you need, your work is done here.
• As a private service to a Content Switching (CS) virtual server (vserver). No IP:port is assigned to it. In other words, the load balanced service group won't be a service usable by the clients (directly). It can only be accessed/referenced by its name, as an internal object, from within the Netscaler running context (aka, by the the content switching engine we'll discuss in Part II of this post) in the latter case. That's right, it is more like a call-by-reference.
  

F5 BIG-IP version 9 sends a RST packet to the client after timeout

The load balancer in production on our site currently leaves the client hanging. After timeout on the server or client, it terminates the connection to the server but doesn't terminates the connection to the client. As a result, the FireFox or IE browser spins its hourglass or whirls forever, as confused the hell out of the unsuspecting non-techie end users.
With that in mind, it is quite a relief to me to see a RST packet received by the client, after 320s of idle time ( thread.sleep(320000)) in a jsp served by Apache/tomcat. Since FireFox actually sent 'keep-alive 300' in the http request header, so I am not so sure the 300s is per FireFox's request, or per F5 Big-IP v9's own hard-coded client (or server) idle timeout. If I recall correctly, Citrix Netscaler EE9000 (version 6.1 and 7.0 beta) defaults to timeout on idle client at 180 seconds and on idle server at 300 seconds.

Citrix Netscaler :: wildcard not so wild for Content Switch (continued)

I dialed into a conference call with the Citrix lady this afternoon. She told me that answers she got internally are that the RegExp pattern "jack*.blogspot.com" wont' work at all for any builds for current Netscaler 6.1 release. What a disappointment! Her inquiry on whether the coming 7.0 release, currently in RC-CR phase (RC stands for release candidate while CR stands for controlled release), will have it was redirected to Citrix developers in India. Answer is pending,
since people are sound asleep at that part of the world right now.

Not sure why this would be a question for developers instead of for some project manager here in the states. You'd think there got to be some form of project plan or use cases or feature lists or fixed bugs for a major release now in RC mode? Should I hold that against Citrix's development or PM process?

Could I just read too much into it, as usual? No. This is by stark contrast with F5's much more mature and systematic approach towards release management.

citrix netscaler :: wildcard not so wild for Content Switch

With my son going to church group meetings, I had a bit free time to resume playing with the citrix Netscaler EE-9000. I set up a policy using in-line expression to say I want "http.header.host == jack*.blogspot.com", then I created a content switch virtual server to associate the policy with a LB virtual server I created earlier. Simple enough, huh? Not really, all attempts to hit the CS-LB service (firefox, IE, curl) were met with "500 service not available error". Ethereal dumps on the client and tcpdump dumps on the Citrix box couldn't give me anything wrong with communication either.

Combing through the ICG (Installation & Configuration guide), I found such an error usually is due to the fact that an underlying feature is not enabled. I went back to the system node on the configuration GUI. And sure enough, 'content switching' is not checked by default under 'Features'. (while writing this, I am thinking, would it be better to shadow it out if such a feature is not enabled!) Checked & saved. Still got the same error. reboot the box, delete/recreate the policy & cs virtual server, all to no avail.

I cried for help by calling up my friend. He in turn got his lady friend, a Citrix SE, to chat with me. Off the bat, she told me that she's young with the company and may not know all the answers. I said to myself, "oh lord...sigh...at least she's honest." After a few go-to-meeting sessions and hauling other SEs into it, she finally broke it to me, "no, no wildcard would work for content switching." Great, isn't it?!

So, she and I went down the path to make-do with the limit set of operands on the GUI, trying to come up with a compound regular expression to mimic this behavior. None could really works so far. She kept telling me that newer builds won't help since they only fix bugs and won't change how RegExp works. She also told me that this IS how a PERL Reg object works for Citrix Content Switching. I was smiling along, begging to differ as a certified PERL programmer.

At this point, I am somewhat perplexed since it is hard for me to believe it didn't have true regular expression support. My employer opts not to use sub-domains or URL to divide traffic to load-balanced clusters. Instead, the sales/marketing geniuses want the customer to have any URL they want, only to find us engineers and architects scrambling to direct requests to these URL to a proper LB cluster. Sina and Google are using Netscaler in thousands. Does it mean that they don't need wildcards?

Citrix Netscaler NS9000 :: DSR mode server selection based on first packet

it is somewhat crazy for a content switch or application switch or any modern web traffic director to make decision based on first request packet. A content switching load balancer definitely needs to get at least all the headers for a HTTP or SMTP request.

It is my experience that there are quite some proxy servers out there who inserted all these non-sense headers, which push the required and essential HOST header for a HTTP/1.1 request to the very bottom of the header section. For one thing, http request from a Blackberry with BIS or BES account will have a huge Accept-type HTTP header, which pushed the Host: header to the very bottom and outside of first packet for a simple HTTP request. As a result, the Host header shows up in second packet or later. In turn, the server selection based on first packet treats this as no an empty Host header and fails, with the client traffic wrongly directed.

It is problematic with anything other than waiting for all headers or at least till the HOST header shows up or till the all HEADERS you have defined to switch directions against. The latter is not fool-proof, since HTTP doesn't prevent you to have the same header over and over again.

Of course, it'd be even crazier if Citrix Netscaler under non-DSR mode does the same thing. I'll report back.

netscaler EE9000 based on Freebsd 4.9

courtesy of a friend, I borrowed a Netscaler (now Citrix) EE9000 load balancer. An 80G SATA drive at the rear end of the pizza-box and a 256M CompactFlash card too. pretty amusing the 'quick-start guide' doesn't say these network configuration is targeted at which network socket. got to admit, it was quick glance.

connected the enclosed null modem serial cable (DB-9 to DB-9) to a nearby Linux boxen. power it up, logon, and saw 'uname -a' as below:
FreeBSD ns 4.9-NETSCALER-6.1 FreeBSD 4.9-NETSCALER-6.1 .

This motivates me to search for pre-made image for VMware player. unfortunately, most sites only have FreeBSD 6.0 or 6.1. wonder if the difference worthy to do the download/install myself. I mainly want to have FreeBSD installation to test disaster recovery or read some man pages, and some 'dangerous' tricks a good system engineer won't attempt on a production server.