SAP HANA scale-out
The following section is an example host setup for SAP HANA scale-out with standby node
on AWS using FSx for ONTAP as the primary storage solution. You can use SAP HANA host auto
failover, an automated solution provided by SAP, for recovering from a failure on your SAP
HANA host. For more information, see SAP
HANA - Host Auto-Failover
Topics
Linux kernel parameters
Create a file named 91-NetApp-HANA.conf with the following configurations
in the /etc/sysctl.d directory on all the nodes.
net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 131072 16777216 net.ipv4.tcp_wmem = 4096 16384 16777216 net.core.netdev_max_backlog = 300000 net.ipv4.tcp_slow_start_after_idle=0 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1
Increase the max sessions slots for NFSv4 to 180.
echo options nfs max_session_slots=180 > /etc/modprobe.d/nfsclient.conf
You must reboot your instance for the kernel parameters and NFS settings to take effect.
Network File System (NFS)
Important
For SAP HANA scale-out systems, FSx for ONTAP only supports NFS version 4.1.
Network File System (NFS) version 4 and higher requires user authentication. You can authenticate with Lightweight Directory Access Protocol (LDAP) server or local user accounts.
If you are using local user accounts, the NFSv4 domain must be set to the same value
on all Linux servers and SVMs. You can set the domain parameter (Domain = <domain
name>) in the /etc/idmapd.conf file on the Linux hosts.
To identify the domain setting of the SVM, use the following command:
nfs show -vserver hana-data -fields v4-id-domain
The following is example output:
vserver v4-id-domain --------- ------------ hana-data ec2.internal
Create subdirectories
Mount the /hana/shared volume and create shared and
usr-sap subdirectories for each host. The following example command applies
to 4+1 SAP HANA scale-out systems.
mkdir /mnt/tmp mount -t nfs -o sec=sys,vers=4.1 <svm-shared>:/HDB-shared /mnt/tmp cd /mnt/tmp mkdir shared mkdir usr-sap-host1 mkdir usr-sap-host2 mkdir usr-sap-host3 mkdir usr-sap-host4 mkdir usr-sap-host5 cd umount /mnt/tmp
Create mount points
On scale-out systems, create the following mount points on all the subordinate and standby nodes. The following example command applies to 4+1 SAP HANA scale-out systems.
mkdir -p /hana/data/HDB/mnt00001 mkdir -p /hana/log/HDB/mnt00001 mkdir -p /hana/data/HDB/mnt00002 mkdir -p /hana/log/HDB/mnt00002 mkdir -p /hana/data/HDB/mnt00003 mkdir -p /hana/log/HDB/mnt00003 mkdir -p /hana/data/HDB/mnt00004 mkdir -p /hana/log/HDB/mnt00004 mkdir -p /hana/shared mkdir -p /usr/sap/HDB
Mount file systems
The created file systems must be mounted as NFS file systems on Amazon EC2. The following table is an example recommendation of NFS options for different SAP HANA file systems.
| File systems | NFS mount options | |||
|---|---|---|---|---|
| Common | NFS version | NFS transfer size | nconnect | |
| SAP HANA data | rw,bg,hard,timeo=600,noatime, | vers=4,minorversion=1,lock, | rsize=262144,wsize=262144, | nconnect=4 |
| SAP HANA log | rw,bg,hard,timeo=600,noatime, | vers=4,minorversion=1,lock, | rsize=262144,wsize=262144, | nconnect=2 |
| SAP HANA shared | rw,bg,hard,timeo=600,noatime, | vers=4,minorversion=1,lock, | rsize=262144,wsize=262144, | nconnect=2 |
| SAP HANA binary | rw,bg,hard,timeo=600,noatime, | vers=4,minorversion=1,lock, | rsize=262144,wsize=262144, | nconnect=2 |
-
Changes to the
nconnectparameter take effect only if the NFS file system is unmounted and mounted again. -
Client systems must have unique host names when accessing FSx for ONTAP. If there are systems with the same name, the second system may not be able to access FSx for ONTAP.
Example - mount shared volumes
Add the following lines to /etc/fstab on all the
hosts to preserve mounted file systems during an instance reboot. You can then run
mount -a to mount the NFS file systems.
<svm-data_1>:/HDB_data_mnt00001 /hana/data/HDB/mnt00001 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=4 <svm-log_1>:/HDB_log_mnt00001 /hana/log/HDB/mnt00001 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 <svm-data_2>:/HDB_data_mnt00002 /hana/data/HDB/mnt00002 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=4 <svm-log_2>:/HDB_log_mnt00002 /hana/log/HDB/mnt00002 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 <svm-data_3>:/HDB_data_mnt00003 /hana/data/HDB/mnt00003 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=4 <svm-log_3>:/HDB_log_mnt00003 /hana/log/HDB/mnt00003 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 <svm-data_4>:/HDB_data_mnt00004 /hana/data/HDB/mnt00004 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=4 <svm-log_4>:/HDB_log_mnt00004 /hana/log/HDB/mnt00004 nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 <svm-shared>:/HDB_shared/shared /hana/shared nfs rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2
Example - mount host-specific volumes
Add the host-specific line to /etc/fstab of each
host to preserve mounted file systems during an instance reboot. You can then run
mount -a to mount the NFS file systems.
| Host | Line |
|---|---|
| Host 1 | <svm-shared>:/HDB_shared/usr-sap-host1 /usr/sap/HDB nfs
rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 |
| Host 2 | <svm-shared>:/HDB_shared/usr-sap-host2 /usr/sap/HDB nfs
rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 |
| Host 3 | <svm-shared>:/HDB_shared/usr-sap-host3 /usr/sap/HDB nfs
rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 |
| Host 4 | <svm-shared>:/HDB_shared/usr-sap-host4 /usr/sap/HDB nfs
rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 |
| Host 5 (standby host) | <svm-shared>:/HDB_shared/usr-sap-host5 /usr/sap/HDB nfs
rw,bg,hard,timeo=600,noatime,vers=4,minorversion=1,lock,rsize=262144,wsize=262144,nconnect=2 |
Set ownership for directories
Use the following command to set the hdbadm ownership on SAP HANA data
and log directories.
sudo chown hdbadm:sapsys /hana/data/HDB sudo chown hdbadm:sapsys /hana/log/HDB
SAP HANA parameters
Install your SAP HANA system with the required configuration, and then set the
following parameters. For more information on SAP HANA installation, see SAP HANA Server Installation and Update Guide
Optimal performance
For optimal performance, set the following parameters in the global.ini
file.
[fileio] max_parallel_io_requests=128 async_read_submit=on async_write_submit_active=on async_write_submit_blocks=all
The following SQL commands can be used to set these parameters on
SYSTEM level.
ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('fileio', 'max_parallel_io_requests') = '128' WITH RECONFIGURE; ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('fileio', 'async_read_submit') = 'on' WITH RECONFIGURE; ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('fileio', 'async_write_submit_active') = 'on' WITH RECONFIGURE; ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('fileio', 'async_write_submit_blocks') = 'all' WITH RECONFIGURE;
NFS lock lease
Starting with SAP HANA 2.0 SPS4, SAP HANA provides parameters to control the
failover behavior. It is recommended to use these parameters instead of setting the
lease time at the SVM level. The following parameters are configured in the
namerserver.ini file.
| Section | Parameter | Value |
|---|---|---|
failover |
normal_retries |
9 |
distributed_watchdog |
deactivation_retries |
11 |
distributed_watchdog |
takeover_retries |
9 |
The following SQL commands can be used to set these parameters on
SYSTEM level.
ALTER SYSTEM ALTER CONFIGURATION ('nameserver.ini', 'SYSTEM') SET ('failover', 'normal_retries') = '9' WITH RECONFIGURE; ALTER SYSTEM ALTER CONFIGURATION ('nameserver.ini', 'SYSTEM') SET ('distributed_watchdog', 'deactivation_retries') = '11' WITH RECONFIGURE; ALTER SYSTEM ALTER CONFIGURATION ('nameserver.ini', 'SYSTEM') SET ('distributed_watchdog', 'takeover_retries') = '9' WITH RECONFIGURE;
Data volume partitions
With SAP HANA 2.0 SPS4, additional data volume partitions allow configuring two or
more file system volumes for the DATA volume of an SAP HANA tenant database in a
single-host or multi-host system. Data volume partitions enable SAP HANA to scale beyond
the size and performance limits of a single volume. You can add additional data volume
partitions at any time. For more information, see Host configuration
Host preparation
Additional mount points and /etc/fstab entries must be created and the
new volumes must be mounted.
-
Create additional mount points and assign the required permissions, group, and ownership.
mkdir -p /hana/data2/HDB/mnt00001 chmod -R 777 /hana/data2/HDB/mnt00001 -
Add additional file systems to
/etc/fstab.<data2>:/data2 /hana/data2/HDB/mnt00001 nfsmount options -
Set the permissions to 777. This is required to enable SAP HANA to add a new data volume in the subsequent step. SAP HANA sets more restrictive permissions automatically during data volume creation.
Enabling data volume partitioning
To enable data volume partitions, add the following entry in the
global.ini file in the SYSTEMDB configuration.
[customizable_functionalities] persistence_datavolume_partition_multipath = true
ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('customizable_functionalities', 'PERSISTENCE_DATAVOLUME_PARTITION_MULTIPATH') = 'true' WITH RECONFIGURE;
Note
You must restart your database after updating the global.ini
file.
Adding additional data volume partition
Run the following SQL statement against the tenant database to add an additional data volume partition to your tenant database.
ALTER SYSTEM ALTER DATAVOLUME ADD PARTITION PATH '/hana/data2/HDB/';
Adding a data volume partition is quick. The new data volume partitions are empty after creation. Data is distributed equally across data volumes over time.
Testing host auto failover
We recommend testing your SAP HANA host auto failover scenarios. For more information,
see SAP HANA - Host Auto-Failover
Some words have been redacted and replaced by inclusive terms. These words may appear
different in your product, system code or table. For additional details, see Inclusive Language at SAP
The following table presents the expected results of different test scenarios.
| Scenario | Expected result |
|---|---|
SAP HANA subordinate node failure using echo b >
/proc/sysrq-trigger |
Subordinate node failover to standby node |
SAP HANA coordinator node failure using HDB kill |
SAP HANA service failover to standby node (other candidate for coordinator node) |
| SAP HANA coordinator node failure while other coordinator nodes act as subordinate nodes | Coordinator node failover to standby node while other coordinator nodes act as subordinate nodes |
Test scenarios
SAP HANA subordinate node failure
Check the status of the landscape before testing.
hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> python landscapeHostConfiguration.py | Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker | | | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | | | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role | Roles | Roles | Groups | Groups | | ------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- | | hana | yes | ok | | | 1 | 1 | default | default | coordinator 1 | coordinator | worker | coordinator | worker | worker | default | default | | hanaw01 | yes | ok | | | 2 | 2 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw02 | yes | ok | | | 3 | 3 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw03 | yes | ok | | | 4 | 4 | default | default | coordinator 3 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw04 | yes | ignore | | | 0 | 0 | default | default | coordinator 2 | subordinate | standby | standby | standby | standby | default | - | overall host status: ok
Run the following command on the subordinate node as root to simulate a
node crash. In this case, the subordinate node is hanaw01.
echo b > /proc/sysrq-trigger
hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> python landscapeHostConfiguration.py | Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker | | | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | | | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role | Roles | Roles | Groups | Groups | | ------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- | | hana | yes | ok | | | 1 | 1 | default | default | coordinator 1 | coordinator | worker | coordinator | worker | worker | default | default | | hanaw01 | no | info | | | 2 | 0 | default | default | subordinate | subordinate | worker | standby | worker | standby | default | - | | hanaw02 | yes | ok | | | 3 | 3 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw03 | yes | ok | | | 4 | 4 | default | default | coordinator 3 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw04 | yes | info | | | 0 | 2 | default | default | coordinator 2 | subordinate | standby | subordinate | standby | worker | default | default | overall host status: info hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support>
SAP HANA coordinator node failure
Check the status of the landscape before crashing the node.
hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> python landscapeHostConfiguration.py | Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker | | | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | | | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role | Roles | Roles | Groups | Groups | | ------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- | | hana | yes | ok | | | 1 | 1 | default | default | coordinator 1 | coordinator | worker | coordinator | worker | worker | default | default | | hanaw01 | yes | ok | | | 2 | 2 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw02 | yes | ok | | | 3 | 3 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw03 | yes | ok | | | 4 | 4 | default | default | coordinator 3 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw04 | yes | ignore | | | 0 | 0 | default | default | coordinator 2 | subordinate | standby | standby | standby | standby | default | - | overall host status: ok hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support>
Use the following command to simulate failure, by interrupting SAP HANA processes,
on the coordinator node. In this case, the coordinator node is hana.
hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> HDB kill
hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> python landscapeHostConfiguration.py nameserver hana:30001 not responding. | Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker | | | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | | | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role | Roles | Roles | Groups | Groups | | ------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- | | hana | no | info | | | 1 | 0 | default | default | coordinator 1 | subordinate | worker | standby | worker | standby | default | - | | hanaw01 | yes | ok | | | 2 | 2 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw02 | yes | ok | | | 3 | 3 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw03 | yes | ok | | | 4 | 4 | default | default | coordinator 3 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw04 | yes | info | | | 0 | 1 | default | default | coordinator 2 | coordinator | standby | coordinator | standby | worker | default | default | overall host status: info hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support>
SAP HANA coordinator node failure while other coordinator nodes act as subordinate nodes
Check the status of the landscape before testing.
hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> python landscapeHostConfiguration.py | Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker | | | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | | | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role | Roles | Roles | Groups | Groups | | ------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- | | hana | yes | ok | | | 1 | 2 | default | default | coordinator 1 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw01 | yes | info | | | 2 | 0 | default | default | subordinate | subordinate | worker | standby | worker | standby | default | - | | hanaw02 | yes | ok | | | 3 | 4 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw03 | yes | ok | | | 4 | 3 | default | default | coordinator 3 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw04 | yes | info | | | 0 | 1 | default | default | coordinator 2 | coordinator | standby | coordinator | standby | worker | default | default | overall host status: info hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support>
Use the following command to simulate failure, by interrupting SAP HANA processes,
on the coordinator node. In this case, the coordinator node is
hana04.
hdbadm@hanaw04:/usr/sap/HDB/HDB00> HDB kill
hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> python landscapeHostConfiguration.py | Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker | | | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | | | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role | Roles | Roles | Groups | Groups | | ------- | -------- | ------- | ---------------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- | | hana | starting | warning | | | 1 | 1 | default | default | coordinator 1 | coordinator | worker | coordinator | worker | worker | default | default | | hanaw01 | starting | warning | | | 2 | 2 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw02 | yes | ok | | | 3 | 3 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw03 | yes | ok | | | 4 | 4 | default | default | coordinator 3 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw04 | no | warning | failover to hana | | 0 | 0 | default | default | coordinator 2 | subordinate | standby | standby | standby | standby | default | - | overall host status: warning hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support> python landscapeHostConfiguration.py | Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker | | | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | | | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role | Roles | Roles | Groups | Groups | | ------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- | | hana | yes | ok | | | 1 | 1 | default | default | coordinator 1 | coordinator | worker | coordinator | worker | worker | default | default | | hanaw01 | yes | ok | | | 2 | 2 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw02 | yes | ok | | | 3 | 3 | default | default | subordinate | subordinate | worker | subordinate | worker | worker | default | default | | hanaw03 | yes | ok | | | 4 | 4 | default | default | coordinator 3 | subordinate | worker | subordinate | worker | worker | default | default | | hanaw04 | no | ignore | | | 0 | 0 | default | default | coordinator 2 | subordinate | standby | standby | standby | standby | default | - | overall host status: ok hdbadm@hana:/usr/sap/HDB/HDB00/exe/python_support>
