In contrast to the RSS and HDR architecture, the Shared Disk Secondary (SDS) functionality in IDS provides the ability to set up multiple database servers sharing the whole dbspace set defined by a primary database server. The SDS architecture can be used for defining a database server on the same physical machine or different machines with an underlying shared file system.

1 SDS Technology

Unlike HDR Secondary and Remote Standalone Secondary (RSS), the SDS server does not maintain a copy of the database server defined dbspaces on its own disk; rather, it shares the same physical disk with the primary server. Similar to the RSS feature, multiple secondary servers can be attached to the SDS infrastructure. In addition, secondary servers can be attached and detached on demand depending on the actual given hardware infrastructure and the workload.

SDS servers provide increased availability and scalability without the need to maintain multiple copies of the physical layout, which results in lowering data storage costs. Multiple SDS servers also provide the opportunity to dedicate specific SDS servers for specific tasks, such as data warehousing on a DSS oriented server or Web application server with an OLTP approach with the appropriate differences in the configuration for parallel data query (PDQ) and memory requirements. Figure shown a simple SDS infrastructure with three secondary nodes:

Because the SDS servers share the dbspaces, SDS secondary servers are able to access the latest logical log from disk after the SDS primary server has written them out. Other than in a HDR or RSS environment, there is no need to send out the complete log buffer over the network. In an SDS infrastructure, the primary server will only send the Log Sequence Number (LSN) information, which consists of the current log file number and the log position actually processed by the database server to the secondary. The secondary will, after receiving the LSN from the primary, read the associated log entries from the disk and apply the changes. The changes are only applied in the buffer pool and will not be written to disk on a secondary server.

After applying the log entries to the buffer pool, the secondary server will send a acknowledgement (ACK) back to the primary. The primary server will keep track of the ACK messages for the LSN progress. The general message flow between the servers is simplified, and shown in the following figure:

2 Threads

In an SDS environment, the IDS server will create a send and receive thread for SMX once the communication between a newly started secondary and the primary is established. The following threads support the SDS functionality on the primary and secondary server.

  • Primary:
    • SDS_send_<secondary>: Requests a send of all data, such as the LSN.
    • SDS_recv_<secondary>: For incoming messages such as ACKs.
  • Secondary:
    • SDS_LogReader: In case a new LSN arrives, this thread will read the associated log pages from the disk and put them into the work queue for the SDS_apply thread and activate it.
    • SDS_apply: Will convert the log pages from the queue into a readable format for the fast recovery threads that apply the log entries on the pages in the buffer pool.

Use the onstat -g ath command to display information about all threads for a primary or secondary server.

Example

SDS related threads on a primary server

Copy
onstat -g ath
IBM Informix Dynamic Server Version 12.10.FC7ADE -- On-Line -- Up 00:54:07 -- 442536 Kbytes

Threads:
 tid     tcb              rstcb            prty status                vp-class       name

 85       46a2cbe0         45018478         1    sleeping secs: 1        1cpu         SDS_Send_ddemodb
 90       46940028         45018d40         3    sleeping secs: 1        1cpu         SDS_Recv_ddemodb

SDS related threads on a secondary server

Copy
onstat -g ath
IBM Informix Dynamic Server Version 12.10.FC7ADE -- Read-Only (SDS) -- Up 00:00:42 -- 442536 Kbytes

Threads:
 tid     tcb              rstcb            prty status                vp-class       name

 43       461c46a8         450123e0         2    cond wait  SDS_Sec_co   1cpu         SDS_LogReader
 50       462e6948         45014fc8         2    sleeping forever        1cpu         SDS_apply
 51       4656d8b0         45014700         3    sleeping secs: 1        1cpu         SDS_Recv
 52       4656dbf0         45015890         3    sleeping secs: 1        1cpu         SDS_Send

3 Temporary dbspaces

Specific select statements, even those executed on a read only server, require the existence of a temporary dbspace. They need the space for materializing views or table expressions, writing overflow partitions for hash join queries, or to do sorting. In the existing implementation, the secondary is not able to share the temp dbspaces defined by the primary for writing. This restriction requires that the secondary server has to specify its own temporary dbspace, which is visible only in scope of the secondary.

Temporary dbspaces for a secondary SDS server have to be specified with an ONCONFIG parameter named SDS_TEMPDBS. The dbspace will be created by the server at startup time. If the plan is to use a cooked file, unlike the handling of chunk files with onspaces in the primary server, the file does not need to exist. The server will create the file at startup time and add the appropriate permissions.

Copy
SDS_TEMPDBS=<name>,<path>,<pagesize in k>,<offset>,<size>

Temporary dbspaces defined on the primary server are easy to be determined on a secondary due to a new dbspace flag " W".

Example depicts an onstat -d sample output taken from a SDS secondary, which demonstrates how the dbspaces list on a secondary appears.

Copy
onstat -d
IBM Informix Dynamic Server Version 12.10.FC7ADE -- Read-Only (SDS) -- Up 00:00:15 -- 442536 Kbytes

Dbspaces
address          number   flags      fchunk   nchunks  pgsize   flags    owner    name
44f20028         1        0x20801    1        1        2048     NL BA    informix rootdbs
450505b0         2        0x1000801  2        1        2048     NLPBA    informix d_plog
450507f0         3        0x801      3        1        2048     NL BA    informix d_llog
45050a30         4        0x8801     4        1        2048     NLSBA    informix s_sbsys
45050c70         5        0x8801     5        1        2048     NLSBA    informix s_sbstd
46111028         6        0x208001   6        1        2048     N WBA    informix s_sbtmp
46111268         7        0x200001   7        1        2048     N WBA    informix t_tmp1
461114a8         8        0x200001   8        1        2048     N WBA    informix t_tmp2
461116e8         9        0x801      9        1        2048     NL BA    informix d_wics
46111928         10       0x8801     10       2        2048     NLSBA    informix s_wics
46111b68         11       0x801      11       1        2048     NL BA    informix d_conf
46111da8         12       0x8801     12       1        2048     NLSBA    informix s_conf
46112028         13       0x801      13       1        2048     NL BA    informix d_data
46112268         14       0x801      14       1        2048     NL BA    informix i_data
46568750         2047     0x102001   32766    1        2048     N TBA    informix sdstmpdbs1
 15 active, 2047 maximum

Chunks
address          chunk/dbs     offset     size       free       bpages     flags pathname
44f20268         1      1      0          150000     125134                PI-B-D /INFORMIXDEV/rootdbs
461124a8         2      2      0          50000      24947                 PI-BED /INFORMIXDEV/d_plog
46113028         3      3      0          30250      197                   PI-B-D /INFORMIXDEV/d_llog
46114028         4      4      0          50000      46558      46558      PISB-D /INFORMIXDEV/s_sbsys
                                 Metadata 3389       2522       3389
46115028         5      5      0          50000      46558      46558      PISB-D /INFORMIXDEV/s_sbstd
                                 Metadata 3389       2522       3389
46116028         6      6      0          50000      -1         -1         POSB-- /INFORMIXDEV/s_sbtmp
46117028         7      7      0          50000      49947                 PO-B-- /INFORMIXDEV/t_tmp1
46118028         8      8      0          50000      49947                 PO-B-- /INFORMIXDEV/t_tmp2
46119028         9      9      0          750000     539736                PI-BED /INFORMIXDEV/d_wics_01
4611a028         10     10     0          500000     338699     441275     PISB-D /INFORMIXDEV/s_wics_01
                                 Metadata 58672      0          58672
4611b028         11     11     0          119576     99718                 PI-BED /INFORMIXDEV/d_conf_01
4611c028         12     12     0          250000     230450     233099     PISB-D /INFORMIXDEV/s_conf_01
                                 Metadata 16848      12537      16848
4611d028         13     13     0          821920     4984                  PI-BED /INFORMIXDEV/d_data_01
4611e028         14     14     0          250000     237197                PI-B-D /INFORMIXDEV/i_data_01
4611f028         15     10     0          250000     98883      233145     PISB-D /INFORMIXDEV/s_wics_02
                                 Metadata 16852      0          16852
46607028         32766  2047   1          7999       7946                  PO-B-- /home/informix/ifmx-sds/sdstmpdbs1
 16 active, 32766 maximum

NOTE: The values in the "size" and "free" columns for DBspace chunks are
      displayed in terms of "pgsize" of the DBspace to which they belong.


Expanded chunk capacity mode: always

4 Updatable Secondary servers

In a default configuration, SDS servers only allow read-only access to their maintained databases. With IDS version 11.50, there is a new ONCONFIG parameter called UPDATABLE_SECONDARY. After setting this parameter to a value different than 0, the SDS server, similar to RSS and HDR secondaries, supports the execution of DELETE, UPDATE, and INSERT statements, either standalone or in transactions.

5 The sysha database

Once the SDS primary is defined, similar to an RSS environment, a new database in the rootdbs that is named sysha. The primary SDS server will keep track of information about all the involved SDS nodes that are attached to the system in this database.

The primary table for the SDS infrastructure is sds_tab. It is used to keep track of all the attached SDS secondary nodes and their connection time.

Example

Sample output for a select * from sysha:sds_tab

Copy
dbaccess sysha - -

Database selected.

> select * from sds_tab;

sds_servername  ddemodb
state           C
when_created    1504513034

1 row(s) retrieved.

6 Configuration and setup

6.1 Primary server

In a shared disk environment, the primary server is the owner of a disk and has read/write access. In a default configuration, all other SDS servers have read-only access to a disk. To prepare the server to be an SDS primary, there are changes that have to be made. Those changes are discussed in the following sections.

sqlhosts file

Specify in the sqlhosts file at least one network based connection protocol for the primary SDS server, such as sockets or tli. During the setup and configuration of the secondary, more entries can be added to this file for the new secondary server to enable the communication between the primary and each secondary.

  • /etc/hosts file:
    Copy
    192.168.56.101 pdemodb     #Primary server
    192.168.56.112 ddemodb     #Shared disk secondary server
  • /etc/hosts.equiv file:
    Copy
    #Shared disk secondary server
    ddemodb
  • /etc/services file:
    Copy
    sqlexec         9088/tcp                # IBM Informix SQL Interface
    sqlexec         9088/udp                # IBM Informix SQL Interface
  • $ONCONFIG file:
    Copy
    DBSERVERNAME    pdemodb
    DBSERVERALIASES ol_pdemodb
  • $INFORMIXSQLHOSTS file:
    Copy
    #Primary server
    pdemodb         onipcshm        on_pdemodb      on_pdemodb
    ol_pdemodb      onsoctcp        pdemodb         sqlexec
    
    #Shared disk secondary server
    ol_ddemodb      onsoctcp        ddemodb         sqlexec

ONCONFIG parameter

Besides setting an SDS_TIMEOUT parameter, there are no requirements for additional manual SDS related parameters. The server itself automatically adds the settings of an SDS_ENABLE parameter to the ONCONFIG file during SDS enablement for the primary with onmode.

SDS_TIMEOUT specifies the amount of time in seconds that the primary server will wait for a log position acknowledgement to be sent from the SD secondary server. If there is no log position acknowledgement received from the SD secondary server in the specified amount of time, then the primary server will be disconnected from the SD secondary server and continue.

Define the primary server

Use the onmode statement to define the SDS primary server:

Copy
onmode -d set SDS primary ol_pdemodb

Verification

If the onmode statement returns without an error, you can verify the success of the switch to an SDS primary by looking in the log file.

Copy
onstat -m
10:19:01  Building 'sysha' database ...
10:19:02  'sysha' database built successfully.
10:19:02  SDS Source Node Alias Changed from (<null>) to (ol_pdemodb)
10:19:02  Value of SDS_ENABLE has been changed to 1.

A better verification here is to run an onstat -g sds verbose.

Copy
onstat -g sds verbose
IBM Informix Dynamic Server Version 12.10.FC7ADE -- On-Line -- Up 02:31:50 -- 442536 Kbytes

Number of SDS servers:1
Updater node alias name :ol_pdemodb

SDS server control block: 0x45dd4e10
server name: ddemodb
server type: SDS
server status: Active
connection status: Connected
Last log page sent(log id,page):974,84
Last log page flushed(log id,page):974,84
Last log page acked (log id, page):974,84
Last LSN acked (log id,pos):974,344088
Last log page applied(log id,page): 974,84
Approximate Log Page Backlog:0
Current SDS Cycle:72
Acked SDS Cycle:72
Sequence number of next buffer to send: 34
Sequence number of last buffer acked: 31
Time of last ack:2017/09/01 06:45:45
Supports Proxy Writes: N
Time of last received message: 2017/09/01 06:45:49
Time of last alternate write: N/A
Time of last alternate read : N/A

Warning

Do not manually change the ONCONFIG parameter SDS_ENABLE, which was set by the onmode statement to 1 during the lifetime of this server as a SDS primary.

6.2 Secondary server

The SDS server will utilize the definitions of the dbspaces from the primary, so there is no need to create cooked files or configure raw devices for normal dbspaces, blobspaces, or sbspaces again. In a default configuration the SDS secondary server only allows read-only SQL statements. In addition, IDS 11.50 provides the administrator the ability to allow DML statements standalone or in transactions on the secondary server by enabling Updatable Secondary servers in the ONCONFIG file.

In the following sections we describe the configuration steps required to set up the secondary SDS server.

sqlhosts file

Specify in the sqlhosts file at least one network based connection protocol for the primary SDS server, such as sockets or tli. During the setup and configuration of the secondary, more entries can be added to this file for the new secondary server to enable the communication between the primary and each secondary.

  • /etc/hosts file:
    Copy
    192.168.56.101 pdemodb     #Primary server
    192.168.56.112 ddemodb     #Shared disk secondary server
  • /etc/hosts.equiv file:
    Copy
    #Primary server
    pdemodb
  • /etc/services file:
    Copy
    sqlexec         9088/tcp                # IBM Informix SQL Interface
    sqlexec         9088/udp                # IBM Informix SQL Interface
  • $ONCONFIG file:
    Copy
    DBSERVERNAME    ddemodb
    DBSERVERALIASES ol_ddemodb
  • $INFORMIXSQLHOSTS file:
    Copy
    #Shared disk secondary server
    ddemodb         onipcshm        on_ddemodb      on_ddemodb
    ol_ddemodb      onsoctcp        ddemodb         sqlexec
    
    #Primary server
    ol_pdemodb      onsoctcp        pdemodb         sqlexec

ONCONFIG parameter

To enable the SDS, specify the following values in the ONCONFIG:

Copy
SDS_ENABLE              1
SDS_PAGING              /home/informix/ifmx-sds/page_1,/home/informix/ifmx-sds/page_2
SDS_TEMPDBS             sdstmpdbs1,/home/informix/ifmx-sds/sdstmpdbs1,2,0,16000

To enable Updatable Secondary servers on the SDS, specify the following ONCONFIG parameter:

Copy
UPDATABLE_SECONDARY     1

Start up the secondary server

After applying the changes to the ONCONFIG file, start up the SDS secondary simply with the following command:

Copy
oninit

Verification

In contrast to an SDS primary server, the secondary will also report that it is now acting as a secondary server, in the onstat - output string.

Copy
onstat -
IBM Informix Dynamic Server Version 12.10.FC7ADE -- Read-Only (SDS) -- Up 00:00:14 -- 442536 Kbytes

In case the SDS secondary is configured to allow Updatable Secondary servers, the onstat - output appears as shown in example:

Copy
onstat -
IBM Informix Dynamic Server Version 12.10.FC7ADE -- Updatable (SDS) -- Up 00:00:12 -- 442536 Kbytes

7 Monitoring

After a successful setup of the SDS environment, there are several ways to monitor the status of the associated SDS server. Similar to most of the other server features, the onstat utility provides several options to see the status for the SDS on all servers. In addition, the sysmaster database contains some new tables that can be accessed with select statements for further processing in a open source or user defined Web-based administration console.

7.1 Monitoring SDS with onstat

Example shows the output of an onstat -g sds command on the primary as the lowest level of monitoring information. The most interesting information from that output is the column Last LPG sent.

Copy
onstat -g sds
IBM Informix Dynamic Server Version 12.10.FC7ADE -- On-Line -- Up 00:18:09 -- 442536 Kbytes

Local server type: Primary
Number of SDS servers:1

SDS server information

SDS srv      SDS srv      Connection        Last LPG sent        Supports
name         status       status            (log id,page)        Proxy Writes
ol_ddemodb   Active       Connected             977,1527         Y

If you need to have more detailed information, run an onstat -g sds verbose on the primary.

Copy
onstat -g sds verbose
IBM Informix Dynamic Server Version 12.10.FC7ADE -- On-Line -- Up 00:23:33 -- 442536 Kbytes

Number of SDS servers:1
Updater node alias name :ol_pdemodb

SDS server control block: 0x46848028
server name: ol_ddemodb
server type: SDS
server status: Active
connection status: Connected
Last log page sent(log id,page):977,1527
Last log page flushed(log id,page):977,1527
Last log page acked (log id, page):977,1527
Last LSN acked (log id,pos):977,6254660
Last log page applied(log id,page): 977,1527
Approximate Log Page Backlog:0
Current SDS Cycle:12
Acked SDS Cycle:12
Sequence number of next buffer to send: 1715
Sequence number of last buffer acked: 1713
Time of last ack:2017/09/04 05:15:00
Supports Proxy Writes: Y
Time of last received message: 2017/09/04 05:22:18
Time of last alternate write: N/A
Time of last alternate read : N/A

7.2 Monitoring by sysmaster database

The sysmaster database provides a certain set of tables related to SDS, which are named syssrcsds and systrgsds.

The syssrcsds table on the primary SDS server

Copy
echo "select * from syssrcsds" | dbaccess -e sysmaster
Database selected.

select * from syssrcsds



address             1188380712
server_name         ol_ddemodb
server_status       Active
connection_status   Connected
last_sent_log_uniq  977
last_sent_logpage   1654
last_flushed_log_+  977
last_flushed_logp+  1654
last_acked_lsn_un+  977
last_acked_lsn_pos  6774852
seq_tosend          2584
last_seq_acked      2582
timeof_lastack      1504523684
totallsn_posted     0
totallsn_sent       0
totalpageflush_po+  0
totalpageflush_se+  0

1 row(s) retrieved.

Database closed.

The systrgsds table on the secondary SDS server

Copy
echo "select * from systrgsds" | dbaccess -e sysmaster
Database selected.

select * from systrgsds

address             1172130424
source_server       ol_pdemodb
connection_status   Connected
last_received_log+  977
last_received_log+  1654
next_lpgtoread_lo+  977
next_lpgtoread_lo+  1655
last_acked_lsn_un+  977
last_acked_lsn_pos  6774852
last_seq_received   2737
last_seq_acked      2737
cur_pagingfile      /home/informix/ifmx-sds/page_2
cur_pagingfile_si+  10240
old_pagingfile      /home/informix/ifmx-sds/page_1
old_pagingfile_si+  2048

1 row(s) retrieved.

Database closed.

8 Switch off the SDS

the implementation of the SDS feature, also allows the movement of an SDS primary back to a standard server, by using the onmode -d clear SDS primary <primary servername> command. Using this particular command, a transition can only be performed when the SDS primary server is running in a stand-alone fashion. This means there is no additional SDS secondary server up and running in parallel. The transition of an existing SDS primary with a parallel attached SDS secondary, immediately into a stand-alone server, can be accomplished with the -force option of the onmode -d command. This causes the secondary server to stop and the primary to move into a stand-alone server mode. An attempt to stop the SDS on the defined primary with the parallel up and running SDS secondary, without the -force option, results in the message shown in example below.

Copy
onmode -d clear SDS primary ol_pdemodb
Can not clear SDS alias while there are active SDS nodes

To force the command clear SDS:

Copy
onmode -d clear SDS primary ol_pdemodb force

Run same with sysadmin task function

Copy
echo 'EXECUTE FUNCTION task("ha sds clear","ol_pdemodb");' | dbaccess sysadmin
Database selected.

(expression)  Command completed successfully.

1 row(s) retrieved.

Database closed.