Oracle Linux 7 with Oracle databases 12.1
Monitoring
free -h
top
vmstat [options] [delay [count]]
monitoring a particular process
ps -o "ppid pid vsz rss pmem comm args" --pid
top -p
pmap -x
Environment: Oracle Linux Server release 7.3 on Oracle VM 3.4.3
In Oracle VM Manager, edit VM and add a new Virtual Disk, e.g.
In the VM as root, see the new disk:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
├─xvda1 202:1 0 512M 0 part /boot
├─xvda2 202:2 0 8G 0 part /
├─xvda3 202:3 0 4G 0 part /tmp
├─xvda4 202:4 0 1K 0 part
├─xvda5 202:5 0 1.5G 0 part /home
└─xvda6 202:6 0 16G 0 part [SWAP]
xvdb 202:16 0 50G 0 disk
└─xvdb1 202:17 0 50G 0 part /opt/oracle
xvdc 202:32 0 10G 0 disk
└─xvdc1 202:33 0 10G 0 part /opt/netbackup
xvdd 202:48 0 800G 0 disk
└─xvdd1 202:49 0 800G 0 part /data001/oradata/CDRDEV
xvde 202:64 0 200G 0 disk
└─xvde1 202:65 0 200G 0 part /data001/fast_recovery_area
xvdf 202:80 0 800G 0 disk
└─xvdf1 202:81 0 800G 0 part /data001/oradata/CDRSIT
xvdg 202:96 0 100G 0 disk
└─xvdg1 202:97 0 100G 0 part /data001/archivelogs
xvdh 202:112 0 800G 0 disk
└─xvdh1 202:113 0 800G 0 part /data001/oradata/CDRDEV1
xvdi 202:128 0 800G 0 disk
└─xvdi1 202:129 0 800G 0 part /data001/oradata/CDRSIT1
xvdj 202:144 0 800G 0 disk
└─xvdj1 202:145 0 800G 0 part
For persistent device naming, we can configure ASMLIB or set udev rules. We need to use a partition UUID (not a filesystem UUID) in udev rules. For this, use parted to create a GPT partition.
parted /dev/xvdj
mktable gpt
to create a GPT partition:
mkpart PART-LABEL START END
e.g.
mkpart asmpart1 0% 100%
(parted) print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdj: 859GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 859GB 859GB asmpart1
udevadm info --query=property /dev/xvdj1
DEVLINKS=/dev/disk/by-partlabel/asmpart1 /dev/disk/by-partuuid/48940eb5-f6fa-4558-aeba-b12b81199e27 /dev/oracleasm/disk1
DEVNAME=/dev/xvdj1
DEVPATH=/devices/vbd-51856/block/xvdj/xvdj1
DEVTYPE=partition
ID_FS_TYPE=oracleasm
ID_FS_USAGE=filesystem
ID_PART_ENTRY_DISK=202:144
ID_PART_ENTRY_NAME=asmpart1
ID_PART_ENTRY_NUMBER=1
ID_PART_ENTRY_OFFSET=2048
ID_PART_ENTRY_SCHEME=gpt
ID_PART_ENTRY_SIZE=1677717504
ID_PART_ENTRY_TYPE=ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
ID_PART_ENTRY_UUID=48940eb5-f6fa-4558-aeba-b12b81199e27
ID_PART_TABLE_TYPE=gpt
MAJOR=202
MINOR=145
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=800372237892
vi /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="xvd??", ENV{ID_PART_ENTRY_UUID}=="48940eb5-f6fa-4558-aeba-b12b81199e27", SYMLINK+="oracleasm/disk1", OWNER="oracle", GROUP="dba", MODE="0660"
To implement the new rules:
udevadm trigger
ls -la /dev/oracleasm/disk1
lrwxrwxrwx. 1 root root 8 Feb 15 12:32 /dev/oracleasm/disk1 -> ../xvdj1
ls -la /dev/xvd*
brw-rw----. 1 root disk 202, 0 Feb 15 12:32 /dev/xvda
brw-rw----. 1 root disk 202, 1 Feb 15 12:32 /dev/xvda1
brw-rw----. 1 root disk 202, 2 Feb 15 12:32 /dev/xvda2
brw-rw----. 1 root disk 202, 3 Feb 15 12:32 /dev/xvda3
brw-rw----. 1 root disk 202, 4 Feb 15 12:32 /dev/xvda4
brw-rw----. 1 root disk 202, 5 Feb 15 12:32 /dev/xvda5
brw-rw----. 1 root disk 202, 6 Feb 15 12:32 /dev/xvda6
brw-rw----. 1 root disk 202, 16 Feb 15 12:32 /dev/xvdb
brw-rw----. 1 root disk 202, 17 Feb 15 12:32 /dev/xvdb1
brw-rw----. 1 root disk 202, 32 Feb 15 12:32 /dev/xvdc
brw-rw----. 1 root disk 202, 33 Feb 15 12:32 /dev/xvdc1
brw-rw----. 1 root disk 202, 48 Feb 15 12:32 /dev/xvdd
brw-rw----. 1 root disk 202, 49 Feb 15 12:32 /dev/xvdd1
brw-rw----. 1 root disk 202, 64 Feb 15 12:32 /dev/xvde
brw-rw----. 1 root disk 202, 65 Feb 15 12:32 /dev/xvde1
brw-rw----. 1 root disk 202, 80 Feb 15 12:32 /dev/xvdf
brw-rw----. 1 root disk 202, 81 Feb 15 12:32 /dev/xvdf1
brw-rw----. 1 root disk 202, 96 Feb 15 12:32 /dev/xvdg
brw-rw----. 1 root disk 202, 97 Feb 15 12:32 /dev/xvdg1
brw-rw----. 1 root disk 202, 112 Feb 15 12:32 /dev/xvdh
brw-rw----. 1 root disk 202, 113 Feb 15 12:32 /dev/xvdh1
brw-rw----. 1 root disk 202, 128 Feb 15 12:32 /dev/xvdi
brw-rw----. 1 root disk 202, 129 Feb 15 12:32 /dev/xvdi1
brw-rw----. 1 root disk 202, 144 Feb 15 12:32 /dev/xvdj
brw-rw----. 1 oracle dba 202, 145 Feb 15 12:32 /dev/xvdj1
to see all disks and partitions:
lsblk -o name,mountpoint,label,size,uuid,partlabel,partuuid,serial,wwn,fstype
Alternatively, you can use a partition label name ID_PART_ENTRY_NAME:
cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="xvd??", ENV{ID_PART_ENTRY_UUID}=="48940eb5-f6fa-4558-aeba-b12b81199e27", SYMLINK+="oracleasm/disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="xvd??", ENV{ID_PART_ENTRY_NAME}=="asmpart2", SYMLINK+="oracleasm/disk2", OWNER="oracle", GROUP="dba", MODE="0660"
lsblk -o name,partlabel,partuuid
NAME PARTLABEL PARTUUID
xvda
├─xvda1
├─xvda2
├─xvda3
├─xvda4
├─xvda5
└─xvda6
xvdb
└─xvdb1
xvdc
└─xvdc1
xvdd
└─xvdd1
xvde
└─xvde1
xvdf
└─xvdf1
xvdg
└─xvdg1
xvdh
└─xvdh1
xvdi
└─xvdi1
xvdj
└─xvdj1 asmpart1 48940eb5-f6fa-4558-aeba-b12b81199e27
xvdk
└─xvdk1 asmpart2 0b8ba20c-a10d-4564-b582-0692fd9657e8
# cat /proc/sys/vm/nr_hugepages
2054
or
# cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
HugePages_Total: 2054
HugePages_Free: 546
HugePages_Rsvd: 542
HugePages_Surp: 0
Hugepagesize: 2048 kB
checking if transparent huge pages are disabled# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
**********************************************************************Mon Oct 16 12:28:54 2017Dump of system resources acquired for SHARED GLOBAL AREA (SGA)Mon Oct 16 12:28:54 2017 Per process system memlock (soft) limit = 128GMon Oct 16 12:28:54 2017 Expected per process system memlock (soft) limit to lock SHARED GLOBAL AREA (SGA) into memory: 2050MMon Oct 16 12:28:54 2017 Available system pagesizes: 4K, 2048KMon Oct 16 12:28:54 2017 Supported system pagesize(s):Mon Oct 16 12:28:54 2017 PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s) 4K Configured 3 3 NONE 2048K 2054 1025 1025 NONE**********************************************************************
Viewing database 11.2 alert log
************************ Large Pages Information *******************Per process system memlock (soft) limit = 128 GB
Total Shared Global Region in Large Pages = 2050 MB (100%)
Large Pages used by this instance: 1025 (2050 MB)Large Pages unused system wide = 4 (8192 KB)Large Pages configured system wide = 2054 (4108 MB)Large Page size = 2048 KB********************************************************************If number of Huge pages is less than required, than SGA will use both page types. Database 12.1 aler.log: PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s) 4K Configured 3 187704 NONE 2048K 659 1025 658 NONE
Simplified setup1. Have the memlock user limit set in /etc/security/limits.conf file. Set the value (in KB) slightly smaller than total RAM (90%) , at least it must be bigger than HugePages size.* soft memlock 14680064* hard memlock 14680064Atentu! If you use systemd to auto startup databases, systemd ignores limits.conf. You need to add the following to a service unit file[Service]LimitMEMLOCK=infinityLimitNOFILE=65535
2. Get the required number of pages from alert.log (EXPECTED_PAGES) or by running Oracle script hugepages_settings.sh (Doc ID 401749.1)
3. edit vm.nr_hugepages in /etc/sysctl.conf as root4. to reload the parameters, reboot Linux or use sysctl -p
Proper setup and more infoHugePages on Oracle Linux 64-bit (Doc ID 361468.1)HugePages on Linux: What It Is... and What It Is Not... (Doc ID 361323.1)Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration (Doc ID 401749.1)