Apache : Troubleshooting Load Balancing, Status with mod_status and mod_info, Monitoring with WebLogicBridgeConfig

Apache : Troubleshooting Load Balancing, Status with mod_status and mod_info, Monitoring with WebLogicBridgeConfig

If you have knowledge of apache or any other web server like IIS, Iplanet etc and any application server like weblogic, websphere, jboss, tomcat etc, and worked or working in a support environment then you must have knowledge on web server and application server integration, and the concept of load balancing.

Most of the very common issue related with same we generally see is uneven load balancing of requests from web server to the application server. almost all of the organizations use a hardware load balancer ( like bigip, cisco ace etc ) either to divert the request to web server and then from web server to application server, or some organizations don't use web server and directly  send request to application server.

So, the basic architecture with hardware load balancer, web server and application server is -

Architecture 1

hardware load balancer => web server => application server ( with a single web server )

Architecture 2

hardware load balancer => web server => application ( more than one web server )

Architecture 3

hardware load balancer => application server ( without web server )

Architecture 4

hardware load balancer => application server ( with two hardware load balancers )

Based on above diagrams, you can imagine we can use n numbers of hardware load balancers, n number of web servers, and n number of application servers combinations to create a proper load balancing environment.

One thing I would like to clear you that there is no relation of cluster algorithms with the web ( http ) requests, means weblogic cluster don't  do the load balancing of http requests, for that you have to use either a web server or harware load balancer for load balancing the http requests.

Now,  the main issue we generally face is the load balancing from web server or hardware load balancer to application servers not working properly. ideal behavior is all nodes in the cluster should get equal number  ( or almost equal )  of requests.

 I am going to demonstrate you what could be possible causes of this, how we can troubleshoot it.

So, the first question is -

What could be the cause of uneven distributions of requests ?

Problem 1

you can define problem as, uneven load balancing of http requests due to uneven load balancing of http sessions.

Basically, proxy send requests to clustered nodes without session cookie, means every time request reach  to web server it divert to next new node on the cluster in round robin fashion, means each node in the cluster should get equal number of http sessions. if this does not happen, your any of the node or nodes in cluster will get more number of http sessions because of sticky session load balancing policy or you can say session affinity based load balancing.

what happened with sticky or session affinity load balancing is, when your http request connect with a server/node in a cluster, all subsequent requests for the same session send to the same server/node every time, that request create a session and maintain it in the request header, thus called uneven load balancing.

So, you can directly go ahead and disable the sticky load balancing / session affinity at hardware load balancer level.

Problem 2

Generally a proxy maintain a in-memory list of clustered nodes in General server List, which contain all the servers in good health at that of time, and proxy send requests to all nodes in round robin fashion. proxy general server list is almost the same as static servers defined in web server file, but sometime if there is some issues with any of the clustered node(s)/server(s) the proxy General Server List shrink to limited number of servers/nodes with good in health and this also cause unequal load balancing.

Troubleshooting - 

There are certain ways to troubleshoot this problems. below troubleshooting ways and steps are based on apache and weblogic configuration.

1. Enabling WebLogicBridgeConfig using DebugConfigInfo parameter at apache

2. Enabling server-status using mod_status module at apache

and with both above options, you have to enable Debug parameter also to enable the debug logging for apache.

Enabling WebLogicBridgeConfig using DebugConfigInfo parameter at apache

Below is my apache configuration

# Enable this with DebugConfigInfo to get detailed information

ExtendedStatus On

# Module to configure apache with weblogic

LoadModule weblogic_module modules/mod_wl_20.so

# configuration to access any of the web application deployed on weblogic

<Location />

SetHandler weblogic-handler

</Location>

# weblogic nodes configuration, with enable Debug and DebugConfigInfo

<IfModule mod_weblogic.c>

WebLogicCluster localhost:7002,localhost:7003

Debug ALL

DebugConfigInfo on

WLLogFile c:/temp/wlproxy.log

WLTempDir c:/temp

Enabling server-status using mod_status module at apache

# Enable server-status module and configuration to access server-status

</IfModule>

LoadModule status_module modules/mod_status.so

ExtendedStatus On

<Location /server-status>

   SetHandler server-status

      Order Deny,Allow

#   Deny from all

   Allow from localhost

   </Location>

Rest Coming Soon...............