LVS feedbackd HOWTO

This document was published using the 0.5 version of the feedbackd package. However, most of the information contained should be relevant to earlier versions

Feedbackd uses the Network Element Control Protocol (NECP) to communicate load data between master and agents. The NECP specification is available at http://www.circlemud.org/~jelson/writings/draft-cerpa-necp-03.txt. An overview of NECP is available at http://www.circlemud.org/~jelson/writings/necp-ietf/.

A typical NECP session proceeds like this:

This procedure outlines a few important concepts:

If configure completes without errors, skip to the next section

Otherwise, if you get the following error:

 configure: error: cannot find net/ip_vs.h in your kernel source. 

then the ipvs headers are not installed in the expected location. There are a few possible reasons:

The master configuration file is installed in /etc/feedbackd-master.conf by default.

Here is the sample configuration file:

# configuration file for feedbackd-master

# logging configuration
#  log file
logfile  = /usr/local/var/log/feedbackd-master.log
#  log level: one of CRIT, ERR, WARN, INFO, DEBUG or VDEBUG
loglevel = WARN

keepalive-interval = 2
keepalive-timeout  = 10
keepalive-retries  = 3
removal-interval   = 30

# Example virtual service definition.
#   name (a symboolic name, used in logging)
#   address (local address to listen on)
#   protocol (tcp or udp),
#   port (numeric or a name in /etc/services)
#   scheduler (eg, rr, wrr - see ipvsadm documentation)
[virtual-service]
name      = http
address   = 127.0.0.1
protocol  = tcp
port      = http
scheduler = wrr

At present, the logging options are the only ones you need to worry about - the scheduler settings are covered in the section called “Adjusting feedbackd parameters”

director:$ feedbackd-master -t

The agent configuration file is installed in /etc/feedbackd-agent.conf by default. You'll need to change (at least) the director IP address, and perhaps the service definitions.

Here is the sample configuration file:

# configuration file for feedbackd-agent

# Where to connect to the director. Either an IP address or hostname
director = $DIP

# How many connection attempts (to the master) to make before giving up.
#  0 = never give up (default)
connection-retries = 0

# How long to wait for the master
master-timeout = 10

# Path to the monitor plugins
moduledir = /usr/local/lib/feedbackd

# logging configuration
#  log file
logfile = /usr/local/var/log/feedbackd-agent.log
#  log level: one of CRIT, ERR, WARN, INFO, DEBUG or VDEBUG
loglevel = WARN


# Example service definition.
# The following directives are required:
#   name (a symboolic name, used in logging)
#   protocol (tcp or udp),
#   port (numeric or a name in /etc/services)
#   forwarding (DR, NAT or TUN)
#   monitor, followed by any configuration for the monitor plugin
[service]
name    = http
protocol = tcp
port    = http
forwarding = NAT

monitor = cpuload.so

# Another service definition
[service]
name    = mail
protocol = tcp
port    = smtp
forwarding = NAT

# Use the exec.so plugin, which requires a command parameter
monitor = exec.so
command = /usr/local/bin/monitor-mailqueue.sh

realserver:$ feedbackd-agent -t

Now you can run the master process. If you have compiled with ipvs support (the default behaviour, without the --without-ipvs option to ./configure), you will need to be root to run it.

director:$ sudo feedbackd-master 

feedbackd-master will parse the configuration, and if all is ok, run in the background. You can view the logs by:

director:$ sudo tail -f /var/log/feedbackd-master.log 

To stop the process, issue a:

director:$ sudo killall feedbackd-master 

Now you can run the agent process. You'll probably be able to do this as a non-root user, as long as your configuration will allow it - (specifically the monitor configuration and logging):

realserver:$ feedbackd-agent 

feedbackd-agent will parse the configuration, and if all is ok, run in the background. You can view the logs by:

realserver:$ tail -f /var/log/feedbackd-agent.log 

After the agent has started, you should see a new realserver added to the ipvs tables on the director. To view these on the director:

director:$ sudo ipvsadm -L 

Depending on the plugin in use, the weights will be updated on a regular basis

To stop the agent, issue a:

realserver:$ killall feedbackd-agent 

monitor = constant.so
value = 32

monitor = cpuload.so

monitor = exec.so
command = /usr/bin/measure-load.sh

monitor = perl.so
file = /usr/lib/feedbackd-agent/monitor.pl

sub get_health() {
	# perform necessary health measurement
        return $health;
}

The frequency of the NECP keepalive messages can be altered on the master. Lower values of the interval will give the director more up-to-date load data, but will increase network and processor load.

The keepalive interval can be changed in the keepalive-interval directive of the feedbackd-master configuration file. The interval is given in seconds.

The default value for the keepalive interval is 2 seconds

If a response is not received for a keepalive message within a predefined amount of time, then the keepalive message is considered lost and resent. After a predefined number of successive retries, the master assumes that the remote host is down, and will quiesce the realserver (so that future requests are not forwarded to the realserver).

These two predefined values are the "keepalive timeout" and "keepalive retries", respectively. These are found in the master configuration file, in the keepalive-timeout (value given in seconds) and keepalive-retries directives.

These values should be adjusted to suit the network connection between the director and realservers. If the realservers are only reachable through high-latency links, then the timeout value should be higher, to prevent active realservers from being inadvertantly quiesced. A timeout that is too high, however, will result in faulty realservers being quiesced after a longer amount of time (giving a larger proportion of errors in the load balanced service).

The default value for the keepalive timeout is 10 seconds, the default number of retries is 3.

(More specifcally, Master Timeout as seen by the agent on the realserver)

The agent expects to receive regular keepalive messages from the master. If the agent receives no data from the master before the timeout, then the agent will immediately close the current connection to the master and reconnect. If you are logging at a level at or above WARN, then the following message will be logged:

Master timed out, attempting reconnection

If these messages occur frequently, you may have the master timeout set too low - the agent is timing out before the master has had a chance to send the keepalive packet. However, large values will result in a slow detection of broken connections.

The master timeout is set in the feedbackd-agent config file, in the master-timeout directive. The value is given in seconds.

The default value of the master timeout is 30 seconds