UAC HAProxy Sample
HAProxy Integration
RealityServer ships with a sample UAC event plugin that integrates with the popular HAProxy load balancer. This plugin will communicate with an HAProxy statistics socket and raise and lower the weight of a given backend according to how many connected users there are. This allows for new users to be forwarded to the RealityServer instance with the lowest load. The plugin comes with complete source code and provides a full example of a UAC event plugin. The source can be found in the src/uac_haproxy directory.
This document will describe a sample load balanced architecture, show how to configure HAProxy to load balance between the servers and how to configure User Access Control on RealityServer and the UAC HAProxy plugin to control server weights.
Network Architecture
In this example we have 3 RealityServer instances running on 3 separate servers. These can be accessed on the local network by the following hostnames:
rshost_1
rshost_2
rshost_3
The HOSTNAME environment variable on each machine is also set to the appropriate hostname above. RealityServer is running on each server on port 8080. Each will be limited to 4 simultaneous users.
We have one instance of HAProxy running on the following server:
haproxy_1
The HAProxy host is exposed to the internet via port 80. The RealityServer hosts are only accessible on the internal network. All user access to RealityServer is mediated through HAProxy.
HAProxy Configuration
HAProxy is configured as follows:
global
stats socket *:10000 level admin
defaults
mode http
option http-server-close
frontend all 0.0.0.0:80
default_backend rs_backend
backend rs_backend
balance roundrobin
option forwardfor
timeout queue 5s
timeout server 5m
timeout connect 4s
option httpchk GET /haproxy_test/empty.html
cookie SERVERID insert nocache indirect maxidle 120s
server rshost_1 rshost_1:8080 weight 255 maxconn 1024 cookie s1 check inter 1000
server rshost_2 rshost_2:8080 weight 255 maxconn 1024 cookie s2 check inter 1000
server rshost_3 rshost_3:8080 weight 255 maxconn 1024 cookie s3 check inter 1000
Global settings
global
stats socket *:10000 level admin
Sets up the statistics socket to listen on all interfaces, port 10000. Set to admin mode so that we change change the server weights.
defaults
mode http
option http-server-close
We're an http proxy.
Frontend
frontend all 0.0.0.0:80
default_backend rs_backend
List on port 80 and we use the RealityServer backend.
Backend
backend rs_backend
balance roundrobin
option forwardfor
timeout queue 5s
timeout server 5m
timeout connect 4s
Setup a round robin backend load balancer with some time out values.
option httpchk GET /haproxy_test/empty.html
Use the given URL to check that given RealityServer instances are connectable. This URL will be exempt from UAC.
cookie SERVERID insert nocache indirect maxidle 120s
HAProxy will use a cookie called SERVERID to persist which particular RealityServer a session is associated with. The cookie has a lifetime of 120s after which it expires and a new connection from the same user will go into the round robin pool. This idle time should match the session timeout in RealityServer.
server rshost_1 rshost_1:8080 weight 255 maxconn 1024 cookie s1 check inter 1000
server rshost_2 rshost_2:8080 weight 255 maxconn 1024 cookie s2 check inter 1000
server rshost_3 rshost_3:8080 weight 255 maxconn 1024 cookie s3 check inter 1000
Backend servers to use. Each has a server name which matches its hostname and forwards requests to port 8080. The cookie keyword specifies the cookie value used to match requests to the server and the servers are checked for connectability every 1000ms.
Each server is set to an initial weight of 255 so they are all selected equally. As sessions start the plugin will modify each servers weight so that less heavily loaded servers will be prioritised for new users.
RealityServer Configuration
We add the following to a default RealityServer configuration:
uac_user_limit 4
uac_session_timeout 120
<user haproxy>
host haproxy_1
stats_port 10000
backend rs_backend
server ${HOSTNAME}
</user>
<url /haproxy_test/.*>
apply_uac off
</url>
<url /favicon.ico>
apply_uac off
</url>
UAC Configuration
uac_user_limit 4
uac_session_timeout 120
A limit of 4 users per RealityServer and a session timeout to match the cookie idle time in HAProxy.
HAProxy plugin Configuration
<user haproxy>
host haproxy_1
stats_port 10000
backend rs_backend
server ${HOSTNAME}
</user>
The HAProxy plugin is configured using a user config block called haproxy. Here we specify the server that HAProxy is running on, the statistics socket port, the name of the backed that the RealityServer instances are found in and the server name used to identify the host. Note that we are using the environment variable substitution system to specify the server name. This way we can share a single realityserver.conf file over all 3 servers.
URL Exemption
<url /haproxy_test/.*>
apply_uac off
</url>
<url /favicon.ico>
apply_uac off
</url>
We exempt the HAProxy test URL from UAC. Note that you will need to make an haproxy_test directory in content_root and place an empty.html file in there. It doesn't need to have any content, it just needs to be there so it can be served up. We also exempt /favicon.ico since many browsers request it by default.
UAC Plugin Implementation
- uac_haproxy_event_handler.cpp - Event handler implementation
- uac_haproxy_plugin.cpp - Plugin implementation to install the event handler
This implementation overview will only cover the RealityServer aspects of the implementation, details such as platform specific networking implementations will be omitted. These can be found in the shipped source code.
UAC Handler Implementation
Initialization
#include "uac_haproxy_event_handler.h"
#include <mi/base/ilogger.h>
#include <sstream>
// include platform specific networking headers
// ...
UAC_HAProxy_event_handler::UAC_HAProxy_event_handler(
const char *host, // The hostname running HAProxy
mi::Uint16 port, // Port that the statistics socket is listening on
const char *server, // Our server name qualified with backend. EG: rs_backend/rshost_1
mi::base::ILogger *logger) : // The logger
m_host(host),
m_port(port),
m_server(server),
m_logger(logger,mi::base::DUP_INTERFACE)
{
// set weight to 100% to start with. do this since it may be set
// to some other value from a previous run.
set_weight(100);
}
const char * UAC_HAProxy_event_handler::get_name() const
{
// The name of our handler
return "uac_haproxy_event_handler";
}
Constructor simply takes the configuration values parsed from the config file and stores them internally. As the constructor is only called once on startup it also sets the initial weight of this server to 100% since it will initially have no users on it.
The Handler
mi::Sint32 UAC_HAProxy_event_handler::handle(
mi::nservices::IEvent_handler_context *context,
mi::nservices::IEvent *event )
{
// Get the arguments from the event
mi::base::Handle<mi::IString> session(event->get_event_data<mi::IString>(0)); // session ID, we ignore this
mi::base::Handle<mi::INumber> curr_users(event->get_event_data<mi::INumber>(1)); // current number of assigned users
mi::base::Handle<mi::INumber> max_users(event->get_event_data<mi::INumber>(2)); // maximum number of assigned users
// extract the actual values from the interfaces
mi::Size curr = curr_users->get_value<mi::Size>();
mi::Size max = max_users->get_value<mi::Size>();
// Calculate new server weight and send command to HAProxy. We'll use the percentage weight since
// it doesn't rely on knowing what the original weight in HAProxy was.
mi::Sint32 weight = static_cast<mi::Sint32>(100*(max-curr)/static_cast<mi::Float32>(max));
// log the weight change
m_logger->printf(mi::base::MESSAGE_SEVERITY_INFO,"UAC_HAP","Update HAProxy server %s weight to %d%%",
m_server.c_str(),weight);
// set the new server weight
set_weight(weight);
return 0;
}
The handler simply calculates the weight as the fraction of free slots in relation to maximum slots. This is converted to a percentage and set as the servers weight
Communication with HAPRoxy
void UAC_HAProxy_event_handler::set_weight( mi::Sint32 weight_percentage )
{
std::stringstream sstr;
sstr << "set weight " << m_server << " " << weight_percentage << "%" << "\n";
// Connect to m_host:m_port via TCP and send sstr.str().c_str() to it to modify the servers weight.
// ...
}
This method constructs the command to modify the servers weight and sends it to the statistics socket.
UAC Plugin Install Implementation
The Handler itself is registered and installed in the IServices_plugin::initialize() implementation
void UAC_HAProxy_plugin::initialize(mi::rswservices::IExtension_context* context)
{
// Get the logger
mi::base::Handle<mi::neuraylib::IPlugin_api> plugin_api(context->get_plugin_api());
mi::base::Handle<mi::neuraylib::ILogging_configuration> log_config(
plugin_api->get_api_component<mi::neuraylib::ILogging_configuration>());
mi::base::Handle<mi::base::ILogger> logger(log_config->get_forwarding_logger());
Gets the logger to provide to the handler and to log our own messages
// Find the config and extract the elements. If any are not provided then we don't
// install the handler.
mi::base::Handle<const mi::rswservices::IConfiguration> config(context->get_configuration());
mi::base::Handle<const mi::rswservices::IUser_configuration> hap_config(config->get_user("haproxy"));
const char* hap_host = NULL;
mi::Uint16 hap_port = 0;
const char* hap_backend = NULL;
const char* hap_server = NULL;
if (hap_config.is_valid_interface()) {
mi::base::Handle<const mi::rswservices::IUser_configuration> hap_config_subitem;
hap_config_subitem = hap_config->get_subitem("host");
if(hap_config_subitem.is_valid_interface()) {
hap_host = hap_config_subitem->get_value();
}
hap_config_subitem = hap_config->get_subitem("stats_port");
if(hap_config_subitem.is_valid_interface()) {
hap_port = atoi(hap_config_subitem->get_value());
}
hap_config_subitem = hap_config->get_subitem("backend");
if(hap_config_subitem.is_valid_interface()) {
hap_backend = hap_config_subitem->get_value();
}
hap_config_subitem = hap_config->get_subitem("server");
if(hap_config_subitem.is_valid_interface()) {
hap_server = hap_config_subitem->get_value();
}
}
if (hap_host == NULL || hap_port == 0 || hap_backend == NULL || hap_server == NULL) {
logger->message(mi::base::MESSAGE_SEVERITY_INFO,"UAC_HAP","Not installing UAC HAProxy "
"manager plugin as it is "
"not configured.");
return;
}
Gets the 'haproxy' user configuration block and extracts all the configuration elements from it. If the block, or any of the elements we require, cannot be found then we log a message and do not continue with installation
// All good, contruct the HAProxy server name
std::string server = hap_backend + std::string("/") + hap_server;
// Create and install the event handler
mi::base::Handle<mi::nservices::IEvent_handler> haproxy_handler(new UAC_HAProxy_event_handler(
hap_host,
hap_port,
server.c_str(),
logger.get()));
context->install_event_handler(haproxy_handler.get());
Generate the full HAProxy server name and instantiate the handler implementation. This is then installed into neuray services so it is available to the event system
// Associate the handler with UAC add and remove session events
mi::base::Handle<mi::nservices::IEvent_context> event_context(context->get_event_context());
event_context->register_handler(mi::rswservices::UAC_SESSION_ADDED(),
haproxy_handler->get_name(),
"haproxy_add",0);
event_context->register_handler(mi::rswservices::UAC_SESSION_REMOVED(),
haproxy_handler->get_name(),
"haproxy_remove",0);
logger->message(mi::base::MESSAGE_SEVERITY_INFO,"UAC_HAP","UAC HAProxy manager plugin initialized.");
logger->printf(mi::base::MESSAGE_SEVERITY_INFO,"UAC_HAP","Managing %s on HAProxy at %s:%u.",
server.c_str(),
hap_host,
hap_port);
}
Finally we obtain the web service event context and register the HAProxy handler with the session add and remove events. This ensures that the handler is called whenever a session is added or removed.