The Axional Server SWT console, is built on top of the Eclipse SWT UI interface. It needs the appropriate SWT native library for the target OS architectural to be located under java.library.path. The following artifacts are used for each operating system.

  • macosX64Runtime 'org.eclipse.swt:org.eclipse.swt.cocoa.macosx.x86_64:4.5.2'
  • windowsX64Runtime 'org.eclipse.swt:org.eclipse.swt.win32.win32.x86_64:4.5.2'
  • linuxX64Runtime 'org.eclipse.swt:org.eclipse.swt.gtk.linux.x86_64:4.5.2'
The native library artifact is downloaded as a gradle dependency when using Axional Install

Axional Server SWT console provides a real-time detailed view of Axional Server internals and it's oriented to give either a comprenssive view of server activity and detect potential performance problems.

1 Main

The initial tab shows server general information including JVM overview, thread pool activity and logs.

  • JVM, shows the Java virtual machine version, PID (linux process id), memory and CPU & JVM load.
  • Thread Pool, is the core implementation of a thread pool used to serve http connection requests. The initial parameters like core pool size and maximum pool size, and the current pool size measures the number of threads available for http request.

    Indicator Description
    Core pool size The core number of threads.
    Max pool size The maximum allowed number of threads.
    Current pool size The current number of threads in the pool.
    Completed task count The approximate total number of tasks that have completed execution.
    Rejected count The number of tasks rejected cause pool was full.
    Error count The number of tasks rejected by task error
    Completed / second The number of tasks completed per second
    Rejected / second The number of tasks rejected per second
    Errors / second The number of task errors per second
    Queue size The number of threads in queue
    Active count The approximate number of threads that are actively executing tasks
    Running The approximate number of threads tasks including queued.
    Avg service time $$\frac{getTotalServiceTime}{getNumberOfRequestsRetired}$$
    Avg response time $$getAverageServiceTime + getAverageTimeWaitingInPool$$
    Avg time waiting pool $$\frac{getTotalPoolTime}{getNumberOfRequestsRetired}$$
    Ratio Dead/Response Time $$\frac{getTotalPoolTime}{getTotalServiceTime}$$
    Ratio QueueLength / Cores In this method the getRequestPerSecondRetirementRate rate (RPSRR), or the observed rate at which requests are leaving the system is multiplied by the average time it takes to service a request yielding the average number of requests in the system. The result is divided by the number of cores. $$ RPSRR \times \frac{getAverageServiceTime + getAverageTimeWaitingInPool}{availableProcessors} $$
    Request/s Retirement Rate $$\frac{getNumberOfRequestsRetired}{getAggregateInterRequestArrivalTime}$$

    An important advantage of the fixed thread pool is that applications using it degrade gracefully. If the application simply creates a new thread for every new HTTP request, and the system receives more requests than it can handle immediately, the application will suddenly stop responding to all requests when the overhead of all those threads exceed the capacity of the system. With a limit on the number of the threads that can be created, the application will not be servicing HTTP requests as quickly as they come in, but it will be servicing them as quickly as the system can sustain.

    Threadpool performance can be monitored using built in instrumentation. The parameters shown in the following table can be viewed under Threadpool panel.

  • System monitor shows a timseries graphical view of JVM memory and CPU usage.
  • Logs view shows the last logs reported by server logger.

2 Logs

The logs tab shows most recently logs generated by server. Even logs may be sent to disk or not they can be always viewed from console as they are keep in memory in a short list for at least 1000 items.

Logs have different sources indicating the application module AUTH, JAVA, HTTP, JDBC, POOL, etc. Java standard log levels are shown in a specific color.

  • SEVERE
  • WARNING
  • INFO
  • CONFIG
  • FINE
  • FINER
  • FINEST

When a log has detailed information it can be viewed by clicking on it. For example, an Exception log may show the stack trace in the detailed view.

Debug level can be tuned for each logger in config.xml log section. The available loggers are:

  • AUTH, logs authentication steps including failures using WARNING level.
  • JAVA, logs Java related operations related to caches, virtual machine, etc.
  • POOL, logs object pool related operations.
  • JDBC, logs JDBC related operations.
  • HTTP, logs HTTP related operations.
  • SOAP, logs SOAP related operations.
  • REST, logs REST related operations.
  • OLAP, logs OLAP related operations.
  • FOP, logs Apache FOP processor operations.
  • FTL, logs Apache FreeMarker processor operations.
  • XSL, logs XSL processor operations.
  • XML, logs XML transform operations.
  • TCP, logs TCP operations.
  • SWT, logs SWT (Standard Widgets Tookkit) operations.

Copy
<debugLog>
         <enabled>false</enabled>
         <numFiles>16</numFiles>
         <directory>logs</directory>
         <default>CONFIG</default>
         <levels>
            <entry>
               <key>AUTH</key>
               <value>WARNING</value>
            </entry>
            <entry>
               <key>JAVA</key>
               <value>CONFIG</value>
            </entry>
            <entry>
               <key>POOL</key>
               <value>CONFIG</value>
            </entry>
            <entry>
               <key>JDBC</key>
               <value>CONFIG</value>
            </entry>
            <entry>
               <key>HTTP</key>
               <value>CONFIG</value>
            </entry>
            <entry>
               <key>SOAP</key>
               <value>CONFIG</value>
            </entry>
         </levels>
      </debugLog>
Notice that AUTH level should be configure to WARNING to avoid excessive login messages, specially when using SOAP services. As SOAP is stateles, each connection will perform an auth check and may generate excessive logging. But if you experience login problems and need to debug, switch log level to CONFIG or FINE.

3 Pools

The object pool pattern is a software creational design pattern that uses a set of initialized objects kept ready to use – a pool – rather than allocating and destroying them on demand. A client of the pool will request an object from the pool and perform operations on the returned object. When the client has finished, it returns the object to the pool rather than destroying it; this can be done manually or automatically.

Object pools are primarily used for performance: in some circumstances, object pools significantly improve performance. Object pools complicate object lifetime, as objects obtained from and returned to a pool are not actually created or destroyed at this time, and thus require care in implementation.

Axional Server provides a fast object pool implementation that is mainly used to implement JDBC pooling. Using the pools tab, we can inspect internal pool structure, performance metrics and objects allocated and it's location in either the free or used queue.

Notice that request rate is a meter metric which measures mean throughput as well as one minute, five-minute and one-hour EWMA (exponentially-weighted moving average) throughputs.

Click on a pool to get a detailed pool view.

4 JDBC

The JDBC tab shows information about JDBC connection pools agregated by node, server, database and pool. Each information level agreggates to it's parent from bottom to top.

JDBC uses object pools for it's low level pool implementation.

JDBC pool is built on a hierarchical structure include a node, servers, database and pools.

Component Description
Node Groups all server JDBC information and holds aggrated metrics and statistics for all servers.
Loading...
Server Points to a database server defined by it's URL.
Loading...
Database Points to a database on a server.
Loading...
Pool Points to an specific database pool. A database pool is associated to a database user and a pool configuration.
Loading...

4.1 Node

The node refers to all server JDBC connections of any kind. The node view shows aggregated information for all JDBC views. This means you can see the aggregated metrics of all database servers and it's connections pools.

Loading...

Notice that operation rate is a meter metric which measures mean throughput (sql operations) as well as one minute, five-minute and one-hour EWMA (exponentially-weighted moving average) throughputs.

4.1.1 Servers

Shows aggregated information by JDBC server

4.1.2 Databases

Shows aggregated information by JDBC database

4.1.3 Pools

Shows aggregated information by JDBC pool on all active pools on JDBC node

4.1.4 Connections

Shows information of any JDBC active connection

4.1.5 Statements

Shows information of any JDBC active statement

4.1.6 History

Shows last SQL statements on any JDBC source

4.1.7 Top SQL (used)

Shows top used SQL statements on any JDBC source

4.1.8 Top SQL (time)

Shows top time consuming SQL statements on any JDBC source

4.1.9 Errors

Shows last SQL errors on any JDBC source

4.1.10 Drivers

Shows information about registered JDBC drivers

4.2 Servers

Shows information about JDBC activity by server allowing a hierarchical view of all sub components.

Loading...
When selecting a server you can view detailed information.

4.3 Databases

Shows information about JDBC activity by database allowing a hierarchical view of all sub components.

Loading...
When selecting a server you can view detailed information.

4.4 Pools

Shows information about JDBC activity by pool allowing a hierarchical view of all sub components.

Loading...
When selecting a pool you can view detailed information.

4.4.1 Connections

The connections tab shows active pool connections.

4.4.2 Statements

The statements tab shows active pool statements on all active connections.

4.4.3 History

Shows the statement history (including all connections since pool startup).

4.4.4 TOP sql by use

Shows top SQL operations by use count (including all connections since pool startup).

4.4.5 TOP sql by time

Shows top SQL operations by time consumed (including all connections since pool startup).

4.4.6 Top tables

Shows the most accessed tables (including all connections since pool startup).

4.4.7 Errors

Shows the last errors (including all connections since pool startup).

4.4.8 Cache

Shows the elements in pool cache including table, access count, rows stored, storage type (ArrayList or MemoryMappedFile), and cached SQL.

4.4.9 Blobs

Shows Blob metadata information loaded. This information is used to resolve blobs so they can be stored as references.

4.4.10 Activity

4.4.11 Config

Shows the elements in pool cache including table, access count, rows stored, storage type (ArrayList or MemoryMappedFile), and cached SQL.

5 Cache

Axional Server uses different kind of cache structures to store long living data structures. They are intended to speed up applications by providing a means to manage cached data of various dynamic natures. Like any caching system, Axional Server Cache is most useful for high read, low put applications. Latency times drop sharply and bottlenecks move away from the database in an effectively cached system.

The foundation of Axional Server Cache is the VMTable Cache structure, which is the pluggable controller for a cache type. Four types of caches can be plugged into the VMTable Cache: (1) Maps, (2) Queues, (3) Futures, and (4) Sets.

Caches may belong to several server components types like:

JVM JDBC DISK PROC POOL SOAP MQTT MEMORY THREADS VTABLE JETTY STUDIO

5.1 Maps

Maps are caches with key-value pairs that can have a limit or not. There are several classes of caches bases on maps including LRUHashMap, ConcurrentHashMap, etc.

An LRU is a Map implementation with a fixed maximum size which removes the least recently used entry if an entry is added when full. The least recently used algorithm works on the get and put operations only. Iteration of any kind, including setting the value by iteration, does not change the order. Queries such as containsKey and containsValue or access via views also do not change the order.

Each cache shows it's detailed live information.

Column Description
Id A sequential number on the map collection. The less the older the map is, the greter the newer
Type The server component type it belongs
Class The underlying class implementation
Key type The key type
Value type The value type
Size The current size
Limit The maximum size
Age The time since it was created
Evictable Determines if it can be automatically released during Memory Warning events when memory threshold is reached.

When an cache is selected, its contents can be explored in the detail view. The information varies depending cache content type.

5.2 Queues

A queue is a collection (sized or not) designed for holding elements prior to processing. Server supports several types of queues including:

  • BlockingQueue
  • ConcurrentLimitedQueue
  • ConcurrentLimitedDequeue
  • FixesSizePriorityQueue
  • LinkedBlockingDeque

Examples of queues used in server structures are:

  • BlockingQueuePool - witch holds used objects in object pool
  • SQLStatementHistoryQueue - witch holds a list with last executed SQL statements
  • SQLErrorQueue - witch holds a list with last SQL errors

When an cache is selected, its contents can be explored in the detail view. The information varies depending cache content type.

5.3 Futures

A Future is a data structure used to retrieve the result of some concurrent operation. This result can be accessed synchronously (blocking) or asynchronously (non-blocking).

Future caches hold Future objects. For example, most objects that may be created using a JDBC connection are stored in future caches like JDBCDatabaseMetaData (wrapper of java.sql.DatabaseMetaData).

When an cache is selected, it's contents can be explored in the detail view. The information varies depending cache content type.

5.4 Sets

Caches based on sets uses a NavigableSet implementation based on a TreeMap. The elements are ordered using their natural ordering, or by a Comparator provided at set creation time, depending on which constructor is used.

When an cache is selected, its contents can be explored in the detail view. The information varies depending cache content type.

6 REST

Axional Server includes integrated support for REST applications based on Jersey RESTful Web Services framework. This framework for developing RESTful Web Services in Java provides support for JAX-RS APIs and serves as a JAX-RS (JSR 311 & JSR 339) Reference Implementation.

The REST tab shows deployed applications including details and activity measurement for each endpoint.

Select any rest application tab to se its enpoints and activity.

7 SOAP

The SOAP tab can show either a SOAP server or a SOAP client configuration. Under the SOAP server, each deployed service is shown with its activity measurement.

7.1 SOAP Server

All SOAP services deployed on a server can be monitored from this tab. For each SOAP service method, server keep performance statistics including throughput and transfer size for either input and output.

Notice that request rate and error rate are a meter metric which measures mean throughput as well as one minute, five-minute and one-hour EWMA (exponentially-weighted moving average) throughputs.

7.2 SOAP client

When a server acts as client for remote SOAP services, the client tab shows clients activity.

This does not applies to stand alone direct SOAP calls. It's only applied when SOAP calls are performed by:
deister.axional.server.soap.client.SOAPClient

8 Mqtt

Axional Server provides client implementations of MQTT and MQTT-SN messaging protocols aimed at new, existing, and emerging applications for the Internet of Things (IoT).

The MQTT (Message Queuing Telemetry Transport) tab shows MQTT registered client subscribers on Axional Serverand its activity.

MQTT tab shows registered MQTT clients on server, topics subscribed and the number of messages arrived. To setup a MQTT client you can use AxionalMqttSbuscriber class and override MqttCallback interface methods.

9 Processors

This tab groups template rendering or transformation operations performed by server including Apache FOP transforms, Apache Xalan XSLT processor, HTMLUnit processing and Apache Freemarker template rendering.

Processors (except FTL) may take long to execute and may heavy consume CPU and memory. For example, a long FO document may take seconds to be rendered as a PDF document. To protect server, all those processors are run using a queued executor.

Processors that are subjected to thread pool execution are:

Processor Type #Threads Queue size
FOP Apache FOP Formatting Objects Processor processors / 2 processors * 8
XSL Apache Xalan or Saxon processors / 2 processors * 8
HTML HtmlUnit web scraping processors / 4 processors * 8
Graphviz AT&T Labs Research engine for drawing graphs processors / 4 processors * 8

The Activity section shows the executor queue status of each processor. This section includes:

Indicator Description
Completed Number of executions completed
Rejected Number of executions completed
Running Number of processors running
Queued Number of requests queued
Queue size Queue capacity
#Threads Number of active threads
#Max threads Maximum number of concurrent executions (threads) allowed
 
Total Total executions
Errors Number for failed executions
Min time Minimum process time
Max time Maximum process time
Mean time Mean process time
Sum time Total time consumed
Std dev Standard deviation

9.1 FOP

The FOP tab shows information about FOP rendering processes executed.

When a row is selected, the FOP XML source is shown, and when double-clicked, the generated PDF will be shown. Additional tabs will show rendering events, warning or errors. And the Fonts tab will show font families and triplets allowing generating a PDF will all available fonts.

9.2 XSL

The XSL tab shows information about XSL transformations processes executed. XSL transformations are used for example when building a document from a XML data source.

9.3 HTML

The HTML tab shows server side HTML rendered pages using the HTMLunit processor. This feature it's mainly used during PDF generation of WYSIWSG documents with SVG graphics. During the generation of the XML FO document, server side redering of HTML page is used to extract the SVG image.

You can also enter an URL to do direct web scraping.

This process is heavy CPU consuming.

9.4 Grahviz

Graphviz (short for Graph Visualization Software) is a package of open-source tools initiated by AT&T Labs Research for drawing graphs specified in DOT language scripts. Graphviz consists of a graph description language named the DOT language. DOT is highly customizable and it allows you to control line colors, arrow shapes, node shapes, and many other layout features

To run the Graphviz C code from Java, Axional Servers includes a port to javascript using emscriptem. Emscripten is an LLVM-based project that compiles C and C++ into highly-optimizing JavaScript in asm.js format.

9.5 FTL

The FTL tab shows information about FreeMarker template manager instances active.

FreeMaker instances are shared across REST applications but as template processing is fast it's not attached to any execution queue.

10 Executors

Each server may setup a series of tasks for periodical execution. The Executors console tab show all server active tasks. Executors are divided in three categories according to its nature.

  • EXECUTOR, that performs pooled execution with a limited queue.
  • SCHEDULER, that performs periodic task execution at given intervals.
  • CRON, that performs unix cron style task executions.

Some common Axional Server executors are:

  • MemoryWarning [pool name], listens for memory warning recovery conditions.
  • BloquingQueuePool[Cleanners], cleans pooled objects that are idle for so long.
  • BloquingQueuePool[Passivate], runs a non-blocking passivate process when an object is returned to pool.
  • JDBCTimeoutGuard[database], looks for long living querys that exceed a configuration timeout and fires the SQL cancel if need.
  • FileCleaningTracker, perform the FileReference queue poll of PhantomReferences to fire cleaning of unused files.
  • FOP, XSL & HTML processors, used to queue execution of document rendering.

And some specific of Axional Studio

  • CrontabDaemon[user], holds a unix cron style executor for a given user.
  • SQLBeanCacheRPCBroadcast, performs horizontal replication of cache changes.

Each executor shows detailed information about it's internal state, including fields like:

Indicator Description
Type The executor type that can be EXECUTOR, SCHEDULER or CRON.
Name It's identification name, normlly the task it's doing
State Normally will be Active. Other states can be PAUSED, TERMINATING, TERMINATED and SHUTDOWN
Age Age of the executor
Keep alive Seconds a thread may be idle in pool before destroy it
Threads
#cur Number of current threads
#min Minimum number of threads allowed in pool
#max Maximum number of threads allowed in pool. It's null for SCHEDULER and CRON cause they are unbounded (new threads are created on task arrival, not subject to any limit)
#peak Peak number of threads reached in pool
#active The approximate number of threads that are actively executing tasks
Queue
Type Queue type
Size Size of queue when bounded (null if unbounded)
Used Number of elements in queue
Free Free size of queue when bounded (null if unbounded)
Executions
Completed Number of executions completed
Rejected Number of executions rejected
Timeouts Number of executions that timed out

When a row is selected, detailed information about executor tasks is shown in the bottom window.

11 Java VM

The JVM tab explores the Java VM instance

11.1 JVM Monitor

The main Java view shows information about JVM status

11.2 JVM Memory

Memory console view shows information about heap and non heap memory pools (direct or mapped buffers). It also includes garbage collector statistics.

Since JSR-203/NIO2 brings to JDK a management interface for monitoring the resources associated with pools of buffers. "Buffers" here means direct buffers that are allocated outside of the garbage-collected heap (by ByteBuffer.allocateDirect), and mapped buffers, created by mapping a region of a file into memory (using FileChannel.map). The management interfaces are registered with the platform MBeanServer and exposed to tools that connect to the JMX agent.

11.2.1 JVM Memory Warning

Server includes a Memory Warning Monitor (MWM) that listens for low memory conditions. It is automatically attached to HEAP memory pools with memory threashold supported.

During a low memory condition, server will:

  1. Log a SEVERE message.
  2. Perform a full GC.
  3. Check if low memory condition persists after GC.
  4. If persists, notify to all listeners (caches, pools, ...) that may remove data and perform a full GC again.
  5. Set low memory condition state by checking condition.

If low memory condion is set, it will remain set until a MemoryWarning[Pool] executor checks memory has recovered a good condition. This way, during low memory condition, server may reduce activity to protect overloading.

By default, server enables memory warning monitor at 80% during startup unless changed by Java VM setting from command line flag
-Ddeister.axional.server.java.jvm.mem.JVMMemoryWarningMonitor=value.

Server startup code to setup MWM looks like:

Copy
int threshold = SystemUtils.getInitInt(JVMMemoryWarningMonitor.class, 80);
if (threshold > 0)
	JVMMemoryWarningMonitor.getInstance().setUsageThreshold(threshold / 100.0);
This value is set to 80% to warn a server before it goes to a heavy loaded condition. If your server experiments low memory conditions you should:
  • Increase memory if it's low for the expected load.
  • Review application logic. For example:
    • if handling large memory ResultSets, consider using memory maped pools.
    • If producing large PDF document, consider splitting it.
    • etc.

11.3 JVM Direct memory

Axional Server uses off-heap memory to store long living objects. These objects are stored outside normal memory commonly referred as on-heap memory.

The on-heap store refers to objects that will be present in the Java heap (and also subject to GC). On the other hand, the off-heap store refers to (serialized) objects that are managed by Axional Server, but stored outside the heap (and also not subject to GC). As the off-heap store continues to be managed in memory, it is slightly slower than the on-heap store, but still faster than the disk store.

To solve the performance problem that arise when allocating thousands of little objects using off-heap memory, Axional Server implements a high performance off-heap memory manager. It is provided by MergingByteBufferPoolFactory class.

  • MergingByteBufferPoolFactory, it's a factory to create off-heap memory pools using direct allocated ByteBuffer objects of a desired chunk size
  • MergingByteBufferPool, it's each of the buffer pools (segments) managed by a pool factory. It can be used to allocated and free off-heap memory blocks without the overhead of direct allocation. Allocated blocks are re-linked with collindant blocks to reduce memory fragmentation
  • MergingByteBufferPoolAlocator, shows the internal allocated blocks that mat be Free or Used. When used, its content is displayed.

11.4 JVM Threads

Shows java VM thread information including state, CPU usage and estimated memory usage (on Sun JVM only).

The CPU usage will vary from green to red depending the load of each thread.

The MEMORY usage column, returns an approximation of the total amount of memory, in bytes, allocated in heap memory for the thread. The returned value is an approximation because some Java virtual machine implementations may use object allocation mechanisms that result in a delay between the time an object is allocated and the time its size is recorded.

11.5 JVM Stacktrace

Shows java stacktrace. You can also get a stacktrace from command line by doing:

Copy
$ jps
list of java pid running proceses
Copy
$ jstack [pid]
2016-10-30 12:49:44
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.65-b01 mixed mode):

"Attach Listener" #64 daemon prio=9 os_prio=31 tid=0x00007f814118f800 nid=0x9617 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Thread-11" #63 daemon prio=5 os_prio=31 tid=0x00007f8140e60000 nid=0x6707 waiting on condition [0x00007000046be000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000072401cc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
....

11.6 Computer view

Shows information about underlying hardware and operating system.

11.7 Java properties

Shows java instance runtime properties

11.8 Disk ocupation

Gives information about underlying OS filesystems.

Verify your operating system has enougth free space to allocate temporary data from server applications. Notice that temporary files are need to store cache for SQLResultSet objects when they are memory mapped to files. Also to store temporary blobs and other temporary information.

11.9 Temp folder

The temp folder used by server can be examined from this tab. The temp folder holds several directories, each on referring a server application component.

Directory Description
DiskLRUCache A filesystem cache that can map binary streams of a specific mime type to files. It's mainly used on document processors like FOP to perform FO->XSL->PDF trasformation and keep a copy for system inspection in a LRU queue.
Each application will generte it's own directory like FOPProcessor-fop for FO generation, FOPProcessor-pdf for PDF generation.
FOPProducer Stores temporary FO generated files for standard report PDF rendering (not forms).
JDBCBlobMetaDataDisk Holds a directory for each database to store blobs that have a direct map on file. The file names contains a sequence number and the table and column it's referencing.
webapp Stores webapp applications directories like those for jsp compiled files

11.10 Open files

lsof is a command meaning "list open files", which is used in many Unix-like systems to report a list of all open files and the processes that opened them.

This tab shows the lsof (open files) related to current Java server process.

From Linux command line, you can explore Java VM open files by using:

Copy
$ lsof -p PID
COMMAND   PID USER   FD      TYPE             DEVICE  SIZE/OFF      NODE NAME
java    23416  axs  cwd       DIR                8,6      4096       175 /home/axs/studio
java    23416  axs  rtd       DIR                8,1      4096       128 /
java    23416  axs  txt       REG                8,6      7734  44998460 /home/jdk1.8.0_111/bin/java
java    23416  axs  mem       REG                8,6    207604 105587917 /home/jdk1.8.0_111/jre/lib/amd64/libdcpr.so
java    23416  axs  mem       REG                8,6    492629 105587880 /home/jdk1.8.0_111/jre/lib/amd64/libt2k.so
java    23416  axs  mem       REG                8,6      1024 131591013 /home/axs/studio/tmp/axional-23416/MemoryMappedListStore/[wic_admin@dbsrv1][informix:dpc][RW]/sql6604950430908628032.bin
java    23416  axs  mem       REG                8,6      1024 131585854 /home/axs/studio/tmp/axional-23416/MemoryMappedListStore/[wic_admin@dbsrv1][informix:dpc][RW]/sql2694546414501405258.bin
java    23416  axs  mem       REG                8,6    131072 131585844 /home/axs/studio/tmp/axional-23416/MemoryMappedListStore/[wic_admin@dbsrv1][informix:dpc][RW]/sql8701812905673063554.bin
java    23416  axs  mem       REG                8,6    525453 105587883 /home/jdk1.8.0_111/jre/lib/amd64/libfontmanager.so
java    23416  axs  mem       REG                8,6    413979 105587887 /home/jdk1.8.0_111/jre/lib/amd64/liblcms.so
java    23416  axs  mem       REG                8,6     37420 105587875 /home/jdk1.8.0_111/jre/lib/amd64/libawt_headless.so
java    23416  axs  mem       REG                8,6    772804 105587800 /home/jdk1.8.0_111/jre/lib/amd64/libawt.so
java    23416  axs  mem       REG                8,1    111080      1473 /usr/lib64/libresolv-2.17.so
java    23416  axs  mem       REG                8,1     27776      1455 /usr/lib64/libnss_dns-2.17.so
...

To explore tcp connections to server port 8080

Copy
$ lsof -iTCP@192.168.10.14:8080
COMMAND   PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
haproxy   965 nobody    8u  IPv4 6319987      0t0  TCP web1:51636->web1:webcache (ESTABLISHED)
haproxy   965 nobody   15u  IPv4 6320103      0t0  TCP web1:51720->web1:webcache (ESTABLISHED)
haproxy   965 nobody   16u  IPv4 6316508      0t0  TCP web1:51088->web1:webcache (ESTABLISHED)
haproxy   965 nobody   19u  IPv4 6319614      0t0  TCP web1:51334->web1:webcache (ESTABLISHED)
haproxy   965 nobody   20u  IPv4 6318124      0t0  TCP web1:51224->web1:webcache (ESTABLISHED)
haproxy   965 nobody   22u  IPv4 6319186      0t0  TCP web1:50994->web1:webcache (ESTABLISHED)
haproxy   965 nobody   26u  IPv4 6319629      0t0  TCP web1:51342->web1:webcache (ESTABLISHED)
haproxy   965 nobody   27u  IPv4 6317741      0t0  TCP web1:50646->web1:webcache (ESTABLISHED)
haproxy   965 nobody   30u  IPv4 6316682      0t0  TCP web1:51696->web1:webcache (ESTABLISHED)
haproxy   965 nobody   33u  IPv4 6320145      0t0  TCP web1:51744->web1:webcache (ESTABLISHED)
haproxy   965 nobody   34u  IPv4 6319469      0t0  TCP web1:51182->web1:webcache (ESTABLISHED)
haproxy   965 nobody   38u  IPv4 6320056      0t0  TCP web1:51684->web1:webcache (ESTABLISHED)
haproxy   965 nobody   39u  IPv4 6319634      0t0  TCP web1:51346->web1:webcache (ESTABLISHED)
haproxy   965 nobody   41u  IPv4 6319658      0t0  TCP web1:51362->web1:webcache (ESTABLISHED)
haproxy   965 nobody   42u  IPv4 6316237      0t0  TCP web1:50322->web1:webcache (ESTABLISHED)   
...

A more detailed view of tcp connections to server port 8080

Copy
$ lsof -iTCP@192.168.10.14:8080 -n -P
COMMAND   PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
haproxy   965 nobody    8u  IPv4 6319987      0t0  TCP 192.168.10.14:51636->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   15u  IPv4 6320103      0t0  TCP 192.168.10.14:51720->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   16u  IPv4 6316508      0t0  TCP 192.168.10.14:51088->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   19u  IPv4 6319614      0t0  TCP 192.168.10.14:51334->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   20u  IPv4 6318124      0t0  TCP 192.168.10.14:51224->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   22u  IPv4 6319186      0t0  TCP 192.168.10.14:50994->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   26u  IPv4 6319629      0t0  TCP 192.168.10.14:51342->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   27u  IPv4 6323420      0t0  TCP 192.168.10.14:52530->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   30u  IPv4 6316682      0t0  TCP 192.168.10.14:51696->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   33u  IPv4 6320145      0t0  TCP 192.168.10.14:51744->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   34u  IPv4 6319469      0t0  TCP 192.168.10.14:51182->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   38u  IPv4 6320056      0t0  TCP 192.168.10.14:51684->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   39u  IPv4 6319634      0t0  TCP 192.168.10.14:51346->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   41u  IPv4 6319658      0t0  TCP 192.168.10.14:51362->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   42u  IPv4 6321218      0t0  TCP 192.168.10.14:52218->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   47u  IPv4 6316614      0t0  TCP 192.168.10.14:51366->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   48u  IPv4 6320398      0t0  TCP 192.168.10.14:51912->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   49u  IPv4 6319900      0t0  TCP 192.168.10.14:51550->192.168.10.14:8080 (ESTABLISHED)
haproxy   965 nobody   52u  IPv4 6319742      0t0  TCP 192.168.10.14:51444->192.168.10.14:8080 (ESTABLISHED)    
...

11.11 TCP tunnel

You can add a tcp tunnel for diagnostics. The tunnel can be establish a listen port on local computer that will forward requests to an ip address and port. When established, all incoming traffic to the listen port will be dump in the client section. And all traffic coming as a response will be dump under the server section.

For example, to explore a SOAP request and response we can:

  • Establish a tunnel to our server using the standard port + 1
  • Perform the soap call to the tunnel port.
  • Traffic across tunnel port is redirected to server port while data is captured and logged in tcp console.

Notice that to see SOAP plain text messages body, GZIP should be disabled during client call.

Copy
// we do not accept gzip responses    
call.getSOAPContext().setAcceptGzip(false);
// do not send gzip message call
call.getSOAPContext().setSendGzip(false);

12 Jetty

For Jetty server implementation, this tab show information about Jetty deployed contexts and resources.

12.1 Hierarchy

The hierarchy tab shows a hierarchical view of server handlers.

12.2 Web applications

Shows webapp contexts deployed.

12.3 Servlets

Shows servlet contexts deployed.

12.4 Contexts

Shows contexts deployed.

12.5 Resources

Shows resource handlers deployed.

12.6 Sessions

Shows session managers deployed and for each session manager, the list of active sessions.

12.7 Websockets

Shows websockets per servlet context.

12.8 Statistics

Shows server I/O statistics.

13 Console

The console gives access to the TPC style command line console. Type help to see the list of available commands.

Type any command to see the results in either text console or result table. For example, to see available REST endpoints type:

Copy
$ rest

And to see the details of and endpoint named apps type:

Copy
$ rest apps

To see active JDBC connections type:

Copy
$ jdbc connections