Axional Server is used as a Maven dependency file on other Axional products and can be configure either programmatically or by loading a configuration descriptor from a JAXB XML file.

All products using Axional Server share the same configuration strategy and may add a specific section under application node in configuration file.

1 Parameters

The structure of configuration file is shown below

<server
    engine='engine'
    name='name'
>
    <realm /> !
    <encoding /> !
    <tempDir /> !
    <stop> *
        <port /> !
        <pass /> !
    </stop>
    <http> *
        <queueSize /> !
        <minThreads /> !
        <maxThreads /> !
        <connectors> !
            <connector> +
                <scheme /> !
                <port /> !
                <securePort /> !
                <acceptors /> !
                <selectors /> !
                <acceptQueueSize /> !
                <idleTimeout /> !
                <outputBufferSize /> !
                <requestHeaderSize /> !
                <responseHeaderSize /> !
                <keyStoreFile /> ?
                <keyStorePassword /> ?
            </connector>
            <filters> *
                <filter /> *
            </filters>
        </connectors>
    </http>
    <jdbc> *
        <url /> !
        <initSQL /> ?
        <username /> !
        <password /> !
        <poolType /> ?
        <poolMaxSize /> ?
        <poolExtraSize /> !
        <poolMaxIdle /> ?
        <poolMaxCheckOut /> ?
        <poolAcquireTimeout /> ?
        <poolBornDieTimeout /> ?
        <queryTimeout /> ?
        <queryWarningTime /> ?
        <testOnBorrow /> ?
    </jdbc>
    <mail> !
        <auth /> !
        <host /> !
        <port /> !
        <starttls /> !
        <username /> !
        <password /> !
        <replyTo /> !
    </mail>
    <nexus> ?
        <url /> !
        <username /> !
        <password /> !
        <repository /> !
    </nexus>
    <console> !
        <tcpPort /> !
        <enableWeb /> !
        <enableSwt /> !
        <username /> !
        <password /> !
    </console>
    <logs> !
        <accessLog> !
            <days /> !
            <directory /> !
        </accessLog>
        <debugLog> !
            <enabled /> !
            <numFiles /> ?
            <directory /> !
            <default /> !
            <levels> !
                <entry> +
                    <key /> !
                    <value /> !
                </entry>
            </levels>
        </debugLog>
    </logs>
    <fopConfig /> !
    <externalAuth> ?
        <password /> !
    </externalAuth>
    <jwtAuth> ?
        <key /> !
        <algorithm /> ?
        <expirationMillis /> ?
    </jwtAuth>
    <Application /> ?
</server>

The following example shows a typical Java setup of a server using a custom configuration file

Copy 
OptionsParser options = new OptionsParser("MyServer");

// Add default server options
options.addDefaultHTTPServerOptions("MyServer title", 8080, 8443);

// Add custom options
options.addOption("jdbc.conf.database", 		"Configuration database").setDefault("wic_conf");
options.addOption("jdbc.conf.username", 		"Configuration database user").setRequired();
options.addOption("jdbc.conf.password", 		"Configuration database password").setRequired();

options.parser(new File("myconfig.xml"));
AxionalJettyHttpServerImpl server = (AxionalJettyHttpServerImpl)AbstractHttpServer.create(
	config, 
	AbstractHttpServer.ServerType.JETTY
);

2 Configuration

2.1 Encoding

Set the server encoding for HTTP responses

2.2 Temporary directory

The server uses a temporary directory for BLOB, temporary files, bean caches, etc... and for application specific content (compiled JSP, ...). The default location is the a subfolder of the temporary folder given by Java VM (property "java.io.tmpdir"). The configuration property "tempDir" can be used to override it.

At least 100 MB of free disk space should be required for Axional Server temporary files. Depending on the product being used or the server load much more space will be required.
The path defined in the configuration file should NOT be a system folder ( /tmp, for example) because the server may try to delete or clean it.

2.3 Stop

A port may be configured to allow graceful server shutdown.

Copy

Setting the stop option

...        
   <stop>
        <port>8777</port>;
        <pass>Password2Stop</pass>
   </stop>
...

Once configure you can stop the server by sending the password to the server shutdown socket. For example, using netcat (nc) linux command.

Copy 
$ echo [password] | nc [host] [stop-port]

2.4 License

The license entry allows applications built on Axional Server deploy specific features.

Automatic license servers need to customize:

  • providerUrl, to point to deister automatic license service provider
  • providerKey, a key to access automatic license service
  • customer, to identify the customer name

The license identifies a customer and a computer using a hostid (host identifier). It contains a list of available features by name. Each feature may contain an expiry date and other attributes.

Copy 
<license>
    <notice>(C) Copyright 1996, 2018 deister software</notice>
    <providerUrl>http://www.mydeister.com</providerUrl>
    <providerKey>A20a02303933</providerKey>
    <customer>DEMO</customer>
    <hostid>6e2ffa70</hostid>
    <features>
        <feature>
            <name>studio</name>
            <users>10</users>
            <expires>2017-10-01</expires>
            <signature>a23030aak322w9ss</signature>
        </feature> 
    </features>
</license>

2.5 HTTP

There are two main pools of threads that control the level of concurrent user requests: Acceptor Threads and Server Threads. Acceptor threads receive the HTTPS/SSL requests, and pass those requests on to available Server threads to be processed.

2.5.1 Server Thread Pool

The number of concurrent requests that server is capable of processing is controlled by the server container’s Server Threads pool configuration. When an HTTPS request is received, an Acceptor thread picks an idle Server thread from the pool and dispatches it to process the request.

A default configuration defines a minimum of 15 concurrent threads, with 50 for the "low" and "maximum" number of threads. This means that initially the server creates 15 threads for the pool. If that limit is exceeded, it generates up to 50 threads. This configuration effectively limits the total concurrency of the server to a maximum of 50 requests. If all threads in the pool are processing, additional requests accumulate in the Acceptor Queue and wait until a free Server thread is available from the pool.

For a high reliability system, it should reject the excess requests immediately (fail fast) by using a queue with a bounded capability. The capability (maximum queue length) should be calculated according to the "no-response" time tolerable.

For example, if the webapp can handle 100 requests per second, and if you can allow it one minute to recover from excessive high load, you can set the queue capability to 60*100=6000. If it is set too low, it will reject requests too soon and can't handle normal load spike.

Parameter Key
Queue http.pool.queue.size

It is very important to limit the task queue of the underlying http server engine (Jetty or Tomcat) to the expected load.

The queue is configured via the http.pool.queue.size parameter using the following initialization algorithm based on is's value.

Copy 
BlockingQueue<Runnable> queue = 
	(queueSize < 0 ? new LinkedBlockingQueue<Runnable>()
	: (queueSize == 0 ? new SynchronousQueue<Runnable>()
	: new ArrayBlockingQueue<Runnable>(queueSize)));

Normally, we recommend a LinkedBlockingQueue or an ArrayBlockingQueue with a size equal to the number of processors with a minimum value of 8. Configuring it too low may force server to reject connections.

High load

If queue is unbounded!, under high load in excess of the processing power of the webapp, the http server will keep a lot of requests on the queue. Even after the load has stopped, the http server will appear to have stopped responding to new requests as it still has lots of requests on the queue to handle.
Queue Bounded Notes
LinkedBlockingQueue yes An optionally-bounded blocking queue based on linked nodes. This queue orders elements FIFO (first-in-first-out). Linked queues typically have higher throughput than array-based queues but less predictable performance in most concurrent applications.
The optional capacity bound constructor argument serves as a way to prevent excessive queue expansion but it's not implemented to be used.
SynchronousQueue No A blocking queue in which each insert operation must wait for a corresponding remove operation by another thread, and vice versa. A synchronous queue does not have any internal capacity, not even a capacity of one
ArrayBlockingQueue yes A bounded blocking queue backed by an array. This queue orders elements FIFO (first-in-first-out). The head of the queue is that element that has been on the queue the longest time. The tail of the queue is that element that has been on the queue the shortest time. New elements are inserted at the tail of the queue, and the queue retrieval operations obtain elements at the head of the queue.
Minimum threads http.pool.min.threads
Typically >= 15 or available CPU cores * 25.
Maximum threads http.pool.max.threads or available CPU cores * 75

Configure the max number of threads according to the webapp. That is, how many threads it needs in order to achieve the best performance. Configure with mind to limiting memory usage maximum available. Typically > 50 and < 500.

2.5.2 Acceptor threads and queue size

This connector implementation is the primary connector for the Jetty server over TCP/IP. By the use of various ConnectionFactory instances it is able to accept connections for HTTP, HTTP/2 and WebSocket, either directly or over SSL.

The connector is a fully asynchronous NIO based implementation that by default will use all the commons services (eg Executor, Scheduler) of the passed Server instance, but all services may also be constructor injected into the connector so that it may operate with dedicated or otherwise shared services.

Parameter Key
Acceptor http.connector.acceptors
Acceptor threads accepts new connections from client. They run a loop on a blocking accept() call to accept connections. The right number of acceptor threads is determined by the server load and the traffic shape, in particular by connection open/close rate. The higher this rate, the more acceptors you want.

Acceptors

The number of acceptors should be >=1 <= # CPUs

For example: if you have a 4 CPU/Core machine, it is recommended that you enable 2, 3, or 4 Acceptor threads.
Selector http.connector.selectors
Selectors threads manage events on connected sockets. Every time a socket connects, the acceptor threads accepts the connection and assign the socket to a selector, chosen in a round robin fashion. The selector is responsible to detect activity on that socket (I/O) events such as read availability and write availability) and signal that event to other code in Jetty that will handle the event. One thread runs one selector. This is basic async I/O using selectors.

Selectors

As one selector has a limit o 64K connections, a single selector is enough for most installations. It's default value is -1 and it's automatically tunned on jetty to: $$Math.max(1,Math.min(4,Runtime.getRuntime().availableProcessors()/2)))$$ witch is 4 on a 8 procesors machine.

Accept Queue Size http.connector.acceptQueueSize

Accept queue size, also know as backlog is the maximum number of pending connections on the socket. The exact interpretation is JVM and Operating system specific and may be ignored. Higher values allow more connections to be waiting pending and acceptor thread. Because the exact interpretation is deployment dependent, it is best to keep this value as the default unless there is a specific connection issue for a specific OS that needs to be addressed.

There is no specific formula for sizing the queue. Increasing the queue size has an impact on system resources (i.e., memory footprint). It may also lead to requests being accepted but possibly timing out from the client side if the system is at or beyond capacity. It is only recommended that you increase the queue size if you regularly suffer from HTTPS/SSL connection failures (refused connections) and have already tuned your acceptor and server threads appropriately.

Default settings

If provided thread pool parameters are too low, server will try to adjust them based on the number of processors. For example, for bounded queues, the size will not be less than the number of processors (with a minimum of 8). The following rule is used to ensure maxthreads have a minimum value.
Copy 
int acceptors    = 1;
int selectors    = 4;
int connectors   = 2;
int processors 	 = Math.min(Runtime.getRuntime().availableProcessors(), 8);
maxThreads       = Math.max(maxThreads, processors * connectors * (acceptors + selectors) + (processors * 2));

2.6 JDBC

This section defines the JDBC connection to the bootstrap server. The following is a sample configuration:

Entry Value
url jdbc:informix-sqli:192.168.1.1:1526/${DATABASE}:INFORMIXSERVER=ol_db;DB_LOCALE=en_us.utf8;CLIENT_LOCALE=en_us.utf8
initSQL SET LOCK MODE TO WAIT 15
username [username]
password [password]
poolMaxSize 5
poolExtraSize 0
poolMaxIdle 300
poolMaxCheckOut 1000
poolAcquireTimeout 0
poolBornDieTimeout 0

2.7 Mail

Each server can configure a mail server to allow administrative messaging. For example, the error support dialog may send mails to administrators when users encounters an error. The password recovery feature may send mails to users requesting a password. Or you can automatically enable server state messaging on critical failures.

2.7.1 Configure and send a test mail

Configure mail and use console command to send a mail. The following example shows a mail configuration using Google SMTP service:

Entry Value
host smtp.gmail.com
port 587
starttls true
username [username]
password [password]
from [emailaddress]
replyTo [emailaddress]
For automatic mail services
to [emailaddress]

2.8 Nexus

You can configure Axional Server to check for updates on a nexus repository by using lucene search.

Updates are referred as snapshots on your current version and revision number.

A nexus entry requires the following parameters.

Tag Value
Url http://nexus.deistercloud.com/
username [contact with your provider]
password [contact with your provider]
repository Should be snapshots

2.8.1 How check for updates works ?

The NexusConfig class contains the logic to check on a nexus repository.

  1. Using manifest scanning on runtime jar components, all packages of axional group are checked.
  2. Foreach axional artifact it's version number and revision number is retrieved.
  3. Foreach axional artifact, a lucene search is performed to check for snapshots.
  4. If a new snapshot is found, it is download into libs directory with .new extension.
  5. On reboot, statup shell will check for .new artifacts and replace existing ones for them.

2.8.2 Using check for updates

Check for updates is available from console commands. Select the check for updates option in graphical consoles or use the nexus command in command line console.

2.9 Console

Server support three different console interfaces.

  • A TCP console, enabled specifying a tcp port
  • A WEB console on /console context
  • A SWT console for desktop UI
Either the TCP and WEB console requires a password.

Copy 
<console>
      <tcpPort>4444</tcpPort>
      <enableWeb>true</enableWeb>
      <enableSwt>true</enableSwt>
      <username>admin</username>
      <password>CRYPT-AES128:EEZm5O79PtTrX2klG68S6dep4hs50aP/12B49gAgD2iW+F2jJ0xpVohSlz4M632CYL7S3/VkGzn3</password>
   </console>

2.10 Logs

You can configure access logs and server debug logs. Logs are configured under server/logs element in config.xml.

2.10.1 Access log

The access log file can be configured to trace incoming http requests. If the number of days to log is greater than 0, the access log will be enabled. Logs are placed under the directory specified in directory element.

Copy 
<accessLog>
 <days>2</days>
 <directory>logs</directory>
</accessLog>

2.10.2 Debug log

Logging is broken into three major pieces: the Logger, Formatter and the Handler (Appender). The Logger is responsible for capturing the message to be logged along with certain metadata and passing it to the logging framework. After receiving the message, the framework calls the Formatter with the message. The Formatter formats it for output. The framework then hands the formatted message to the appropriate Appender for disposition.

Server logger is sent by default to ConsoleHandler and MemoryHandler. The console handler output is sent to standard output using ansi color sequences while the memory handler output is stored in a limited memory queue.

Server uses standard Java logging services and it's split in several packages (JAVA, JDBC, HTTP, SOAP, SWT, ...). The standard logging levels in severity order are:

  • SEVERE (highest value)
  • WARNING
  • INFO
  • CONFIG
  • FINE
  • FINER
  • FINEST (lowest value)
In addition there is a level OFF that can be used to turn off logging, and a level ALL that can be used to enable logging of all messages.

To enable logs to be stored in a file, specify a "true" value in the "enabled" property and a directory.

The property "numFiles" sets the number of different log files to keep for the same day (so one file is generated for each server restart, until the specified number is reached). The 0 (zero) value can be used to create only one file per day, with all output in it.

Copy 
<debugLog>
 <enabled>true</enabled>
 <numFiles>0</numFiles>
 <directory>logs</directory>
 <default>CONFIG</default>
 <levels>
    <entry>
       <key>JAVA</key>
       <value>CONFIG</value>
    </entry>
    <entry>
       <key>JDBC</key>
       <value>CONFIG</value>
    </entry>
    <entry>
       <key>HTTP</key>
       <value>CONFIG</value>
    </entry>
    <entry>
       <key>SOAP</key>
       <value>CONFIG</value>
    </entry>
 </levels>
</debugLog>

2.11 FOP

FOP (Formatting Objects Processor) is a print formatter driven by XSL formatting objects (XSL-FO) and an output independent formatter.

A FOP service can be configured using a standard FOP configuration file pointed by configuration variable fop.config

2.12 Application

The application node entry contains application specific information.

3 Security

Configuration file of any server should be protected as it contains critical information about security witch may include passwords.

By default, server will enforce to keep passwords secure using AES encryption and will provide the encryption password of any password option not encrypted.

The parameters security.aes.length and security.aes.shift controls passwords security.

Advanced Encryption Standard

The Advanced Encryption Standard (AES), also known as Rijndael (its original name), is a specification for the encryption of electronic data established by the U.S. National Institute of Standards and Technology (NIST) in 2001. AES is based on the Rijndael cipher developed by two Belgian cryptographers, Joan Daemen and Vincent Rijmen, who submitted a proposal to NIST during the AES selection process. Rijndael is a family of ciphers with different key and block sizes.

For AES, NIST selected three members of the Rijndael family, each with a block size of 128 bits, but three different key lengths: 128, 192 and 256 bits. AES has been adopted by the U.S. government and is now used worldwide. It supersedes the Data Encryption Standard (DES), which was published in 1977. The algorithm described by AES is a symmetric-key algorithm, meaning the same key is used for both encrypting and decrypting the data.

Depending the value of the parameter security.aes.length, server will suggest a AES128, AES196 or AES256 crypt.

Recover password using getPassword() from Java applications

When using the getPassword() function, server will verify password is properly encrypted. If a option password is not encrypted, server will fail providing the encrypted password to be changed in the configuration file. The password should be stored as AES128:hexadecimal-crypted-password or the corresponding AES prefix depending AES length.

Sample password stored in configuration file can be:

  • The server console password
  • A password to be used to connect to a JDBC database
  • A password to be used to connect to a SOAP service
  • A password to be used to connect to another Axional Server acting as backend in a 4 tier configuration

Protection rules for server in DMZ

Even server enforces passwords are keep safe using AES encryption, a server on DMZ should be strongly protected, specially against reverse engineering.

  • Disable root login
  • Disable ping
  • If server requires database passwords, avoid use super privileged users. Instead, use a user with DBA privileges if required.

4 Clustering

Clustering allows us to run applications on several parallel servers (a.k.a cluster nodes). The load is distributed across different servers, and even if any of the servers fails, the application is still accessible via other cluster nodes. Clustering is crucial for scalable enterprise applications, as you can improve performance by simply adding more nodes to the cluster.

As each server runs in a shared nothing environment, a group of servers are setup by pointing them to the same database configuration server.

5 Overload protection

5.1 QoS filter

The Quality of Service Filter (QoSFilter) uses Continuations to avoid thread starvation, prioritize requests and give graceful degradation under load, to provide a high quality of service. When you apply the filter to specific URLs within a web application, it limits the number of active requests being handled for those URLs. Any requests in excess of the limit are suspended.

When a request completes handling the limited URL, one of the waiting requests resumes and can be handled. You can assign priorities to each suspended request, so that high priority requests resume before lower priority requests.

5.1.1 Configure QoS filter

To activate the QoS filter you simply need to define the maximum number of concurrent requests admitted by a server instance.

5.2 DoS filter

The Denial of Service (DoS) filter limits exposure to request flooding, whether malicious, or as a result of a misconfigured client. The DoS filter keeps track of the number of requests from a connection per second. If the requests exceed the limit, server rejects, delays, or throttles the request, and sends a warning message. The filter works on the assumption that the attacker might be written in simple blocking style, so by suspending requests you are hopefully consuming the attacker’s resources.

The DoS filter is related to the QoS filter, using Continuations to prioritize requests and avoid thread starvation.

Parameter Description
maxRequestsPerSec Maximum number of requests from a connection per second. Requests in excess of this are first delayed, then throttled. Default is 25.
delayMs Delay imposed on all requests over the rate limit, before they are considered at all:
  • 100 (ms) = Default
  • -1 = Reject request
  • 0 = No delay
  • any other value = Delay in ms
maxWaitMs Length of time, in ms, to blocking wait for the throttle semaphore. Default is 50 ms.
throttledRequests Number of requests over the rate limit able to be considered at once. Default is 5.
throttleMs Length of time, in ms, to async wait for semaphore. Default is 30000L.
maxRequestMs Length of time, in ms, to allow the request to run. Default is 30000L.
maxIdleTrackerMs Length of time, in ms, to keep track of request rates for a connection, before deciding that the user has gone away, and discarding it. Default is 30000L.
insertHeaders If true, insert the DoSFilter headers into the response. Defaults to true.
trackSessions If true, usage rate is tracked by session if a session exists. Defaults to true.
remotePort If true and session tracking is not used, then rate is tracked by IP+port (effectively connection). Defaults to false.
ipWhitelist A comma-separated list of IP addresses that will not be rate limited.

5.3 Low Resources Monitor

To achieve optimal fair handling for all users of a server, it can be necessary to limit the resources that each user/connection can utilize so as to maximize throughput for the server or to ensure that the entire server runs within the limitations of it’s runtime.

An instance of LowResourcesMonitor may be added to a Server to monitor for low resources situations and to take action to limit the number of idle connections on the server

5.4 JDBC overload protection

The JDBC pool is a protection barrier for overloading a server. The pool can be configured to for various limits like:

  • Number of connections
  • Timeout for long running queries so they will be cancelled after a known period
  • Timeout of connection acquire / destroy, to deal with problematic database server

You can read more in the POOL section

6 JVM parameters

The JVM property "memoryDumpOnThreshold" can be set to "true" to enable the creation of memory dumps when reaching the memory threshold. The default value for this property is "false" (so, memory dumps not enabled).

Copy 
-DmemoryDumpOnThreshold=true

7 High load testing

A tool like siege can be used to perform a stress test on different configurations.

  • Download the latest version of siege
    Copy 
    curl -C - -O http://download.joedog.org/siege/siege-latest.tar.gz
  • Extract the tarball
    Copy 
    tar -xvf siege-latest.tar.gz
  • Change directories to the extracted directory
    Copy 
    cd siege-version
  • Run the following commands (one at a time) to build and install siege
    Copy 
    ./configure
make
make install

This installed siege to /usr/local/bin/ so you need to be root to install

7.1 Running a test

The following sends a 10 requests across 10 concurrent connections for benchmarking (no delay between requests).

Copy 
siege -c 10 -r 10 -b http://localhost:8080/

8 HAProxy

HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic web sites and powers quite a number of the world's most visited ones. Over the years it has become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms.

The most differentiating features are listed below: native SSL support on both sides with SNI/NPN/ALPN and OCSP stapling, IPv6 and UNIX sockets are supported everywhere, full HTTP keep-alive for better support of NTLM and improved efficiency in static farms, HTTP/1.1 compression (deflate, gzip) to save bandwidth, PROXY protocol versions 1 and 2 on both sides, data sampling on everything in request or response, including payload, ACLs can use any matching method with any input sample maps and dynamic ACLs updatable from the CLI stick-tables support counters to track activity on any input sample custom format for logs, unique-id, header rewriting, and redirects, improved health checks (SSL, scripted TCP, check agent, ...), much more scalable configuration supports hundreds of thousands of backends and certificates without sweating.

A sample of configuration file is showed below

Copy 
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
  log 127.0.0.1 local0
  maxconn 4000
  daemon
  uid 99
  gid 99

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
  log     global
  option  tcplog
  option  http-server-close
  mode    http
  timeout server 1d
  timeout connect 1d
  timeout client 1d
  timeout http-request 300s

#---------------------------------------------------------------------
# HTTP (80) main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend http80_frontend
  bind *:80
  option forwardfor
  reqadd X-Forwarded-Proto:\ http

  # WebServer wbs1 in port 8080
  acl mirror2 hdr(host) -i webdomain.mydeister.com
  use_backend wbs1    if mirror2

  default_backend wbsdef


#---------------------------------------------------------------------
# HTTPS (443) main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend https443_frontend
  bind *:443 ssl crt /etc/haproxy/CERT/mydeistercom.pem
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https

  # WebServer wbs1 in port 8080
  acl mirror2 hdr(host) -i mirror2.mydeister.com
  use_backend wbs1    if mirror2

  default_backend wbsdef


#---------------------------------------------------------------------
# WS NODES balancing between the various backends
#
# To uses SSL-Offloading add
# redirect scheme https if !{ ssl_fc }
# 
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# wbs1
#---------------------------------------------------------------------

backend wbs1
  redirect scheme https if !{ ssl_fc }
  balance source
  cookie JSESSIONID prefix
  server wscom1 192.168.10.1:8080 check cookie wscom1
  server wscom2 192.168.10.2:9090 check cookie wscom32


#---------------------------------------------------------------------
# wsdef
#---------------------------------------------------------------------

backend wbsdef
  redirect scheme https if !{ ssl_fc }
  balance source
  cookie JSESSIONID prefix
  server wscom3 192.168.10.3:8080 check cookie wscom3
  server wscom4 192.168.10.4:9090 check cookie wscom4

8.1 HAProxy and WebSockets

Currently there aren’t many options when it comes to proxying WebSockets.

Some potential ways to proxy to a WebSocket backend:

  • proxy based on sub-domain
  • proxy based on a URI
  • proxy using automatic detection

First, let’s get the top portion of our haproxy.cfg file out of the way. This is generally what I use for most configurations:

Copy 
global
  log  127.0.0.1  local0
  log  127.0.0.1  local1 notice
  maxconn  4096
  chroot   /usr/share/haproxy
  uid  99
  gid  99
  daemon

defaults
  log   global
  mode  http
  option  httplog
  option  dontlognull
  retries  3
  option  redispatch
  option  http-server-close
  maxconn  2000
  contimeout  5000
  clitimeout  50000
  srvtimeout  50000

One very important thing to point out in this config is the ‘option http-server-close’ line. Without this, some of the examples below can behave incorrectly. The option tells HAProxy to ignore the servers ‘keepalive’ setting. If it were not specified, then in some cases the conditional rules (used below) would not be re-evaluated every time there is a new request. Instead HAProxy would use the previously established connection for the new request(s) and so therefore would fail to notice that the new request might be a socket request. In short, If you are using WebSockets in a mixed environment, always make sure ‘option http-server-close’ is set.

8.1.1 Proxy based on sub-domain

Some people like to have their WebSocket server running on an entirely separate sub-domain (ie. websockets.example.com). This makes it easy to keep the services on completely separate machines, and also allows you to run your sockets on port 80 directly instead of using a second-proxy on the sub-domain.

Copy 
frontend public
  bind *:80
  acl is_websocket hdr_end(host) -i ws.example.com
  use_backend ws if is_websocket
  default_backend www

backend www
  timeout server 30s
  server www1 127.0.0.1:8080

backend ws
  timeout server 600s
  server ws1 127.0.0.1:8000

This will direct all traffic going to ws.example.com to your websocket server (In this example it’s localhost, but you could easily plug in the IP to a different server).

8.1.2 Proxy based on URI

An alternative to the above setup is to proxy based on the URI (ie. example.com/websockets). This allows you to keep everything operating within the same virtual(or non-virtual) domain.

Copy 
frontend public
  bind *:80
  acl is_example hdr_end(host) -i example.com
  acl is_websocket path_beg -i /websockets
  use_backend ws if is_websocket is_example
  default_backend www

8.1.3 Proxy using WebSocket detection

The final example uses an automatic detection of the websocket request by examining the HTTP header for the Upgrade: WebSocket line. My personal preference is to use this along with a secondary test to make sure we really want to pass the request along to the socket server.

Copy 
frontend public
  bind *:80
  acl is_websocket hdr(Upgrade) -i WebSocket
  acl is_websocket_server hdr_end(host) -i ws.example.com
  use_backend ws if is_websocket is_websocket_server
  default_backend www

This config will ensure that the request not only has the Upgrade: WebSocket header, but also that it’s accessing the correct location for the websocket server (ws.example.com).