What is Grid Infrastructure Management Repository (GIMR)
Oracle Grid Infrastructure repository is a container (store) that is used to preserve diagnostic information collected by Cluster Health Monitor (i.e CHM/OS or ora.crf) as well as to store other information related to Oracle database QoS management, Rapid Home provisioning, etcetera.
However, it is primarily used to maintain diagnostic data collected by the Cluster Health Monitor (CHM) which detects and analyzes Operating System (OS) and Clusterware (GI) resource related failure and degradation.
Brief about Cluster Health Monitor (CHM)
Cluster Health Monitor is a Oracle Clusterware (GI) component, which monitors and analyzes the Clusterware as well as Operating System resources and collects information related to any failure or degradation of those resources. CHM runs as Clusterware resource and is identified by the name ora.crf. The status of CHM resource can be queried using the following command
---// syntax to check status of cluster health monitor //---
$GRID_HOME/bin/crsctl status res ora.crf -init
Example:
---// checking status of CHM //---
myracserver2 {/home/oracle}: crsctl status resource ora.crf -init
NAME=ora.crf
TYPE=ora.crf.type
TARGET=ONLINE
STATE=ONLINE on myracserver2
CHM makes use of two services to collect the diagnostic data as mentioned below
- System Monitor Service (osysmond): The system monitor service (osysmond) is a reat-time monitoring and operating system metric collection service that runs on each cluster node. The collected metrics are then forwarded to Cluster logger service (ologgerd) which stores the data in the Grid Infrastructure Management Repository (GIMR) database
- Cluster Logger Service (ologgerd): In a Cluster, there is one cluster logger service (ologgerd) per 32 nodes. Additional logger services are spawned for every additional 32 nodes. As mentioned earlier the cluster logger service (ologgerd) is responsible for persisting the data collected by the system monitor service (osysmond) in the repository database. If the logger service fails and is not able to come up after a fixed number of retries, Oracle Clusterware will relocate and start the service on a different node.
Example:
In the following two node cluster (myracserver1 and myracserver2), we have the system monitor service (osysmond) running on both myracserver1 and myracserver2 where as the cluster logger service (ologgerd) is running just on myracserver2 (since we can have only one logger service per 32 cluster nodes).
---// we have a two node cluster //---
myracserver2 {/home/oracle}: olsnodes
myracserver1
myracserver2
---// system monitor service running on first node //---
myracserver1 {/home/oracle}: ps -ef | grep osysmond
oracle 24321 31609 0 03:23 pts/0 00:00:00 grep osysmond
root 2529 1 0 Aug27 ? 00:07:48 /app/grid/12.1.0.2/bin/osysmond.bin
myracserver1 {/home/oracle}:
---// system monitor service running on second node //---
myracserver2 {/home/oracle}: ps -ef | grep osysmond
oracle 24321 31609 0 03:25 pts/0 00:00:00 grep osysmond
root 2526 1 0 Aug27 ? 00:07:20 /app/grid/12.1.0.2/bin/osysmond.bin
myracserver2 {/home/oracle}:
---// cluster logger service running on second node //---
myracserver2 {/home/oracle}: ps -ef | grep ologgerd
oracle 25874 31609 0 03:27 pts/0 00:00:00 grep ologgerd
root 30748 1 1 Aug27 ? 00:12:31 /app/grid/12.1.0.2/bin/ologgerd -M -d /app/grid/12.1.0.2/crf/db/myracserver2
myracserver2 {/home/oracle}:
---// cluster logger service not running on first node //---
myracserver1 {/home/oracle}: ps -ef | grep ologgerd
oracle 3519 1948 0 03:27 pts/1 00:00:00 grep ologgerd
myracserver1 {/home/oracle}:
Evolution of diagnostic repository with 12c
Prior to the introduction of Oracle database 12c, the Clusterware diagnostic data was managed in a Berkeley DB (BDB) and the related Berkeley database files were stored by default under $GRID_HOME/crf/db location.
Example:
---// Clusterware diagnostic repository in 11g //---
racserver1_11g {/home/oracle}: oclumon manage -get reppath
CHM Repository Path = /app/grid/11.2.0.4/crf/db/racserver1_11g
Done
Oracle has take a step further with Oracle database 12c and replaced the Berkeley DB with a Single instance Oracle 12c Container database (having a single pluggable database) called management database (MGMTDB) with its own dedicated listener (MGMTLSNR). This database is completely managed by the Custerware (GI) and runs as a single instance database regardless of the number of cluster nodes. Additionally, since MGMTDB is a single instance database and managed by Clusterware (GI); in case the hosting node is down the database would be automatically failed over to the other node by the Clusterware (GI).
GIMR in Oracle Database 12.1.0.1
While installing the Clusterware (GI) software in Oracle database 12.1.0.1, it was optional to install the Grid Infrastructure Management Repository database (MGMTDB). If not installed, Oracle Clusterware (GI) features such as Cluster Health Monitor (CHM), etc which depends on it will be disabled.
GIMR in Oracle Database 12.1.0.2
Oracle has now made it mandatory to install the Grid Infrastructure Management Repository database (MGMTDB) as part of the Clusterware (GI) installation starting with Oracle Clusterware version 12.1.0.2. We no longer have the option to opt out of installing MGMTDB during the Clusterware (GI) installation.
Overall framework of GIMR
Following diagram depicts a brief architecture/framework of the Grid Infrastructure Management Repository (GIMR) along with the related components. Considering a "N" node cluster, we have the GIMR (MGMTDB/MGMTLSNR) running only on a single node with one Cluster Logger service (ologgerd) running per 32 nodes and one System Monitor Service (osysmond) running on every node.
Apart from these integrated components, we have the optional RHP (Rapid Home Provisioning) clients which may communicate with the GIMR (MGMTDB/MGMTLSNR) for persisting/querying metadata related to Oracle Rapid Home Provisioning. We also have the Trace File Analyzer (tfactl) which can communicate with the GIMR (MGMTDB/MGMTLSNR) to query the diagnostic data stored (persisted by cluster logger service) in the repository.
When the node hosting the GIMR repository fails, all the repository resources (MGMTDB/MGMTLSNR) automatically fails over to another available cluster node as depicted in the following diagram
Note: Although the diagram shows the repository (MGMTDB/MGMTLSNR) and cluster logger service (ologgerd) relocating to same to node upon failure (for representation purpose), the cluster logger service (ologgerd) relocation is independent of the repository (MGMTDB/MGMTLSNR) and can relocate to any available cluster node.
GIMR space requirement
The average growth size of the repository is approximately 650-750 MB. The space requirement completely depends on the retention desired for the repository. For example a 4 node cluster would lead at the default retention of 3 days to an approximate size of 5.9-6.8 GB
In case, where the Cluster has more than 4 nodes, an additional 500 MB is required for each of the additional cluster node.
Here are few test cases that have been performed against two node cluster to find out the size requirement
| RETENTION (IN DAYS) | SPACE REQUIRED (IN MB) |
|---|---|
| 3 days | 3896 MB |
| 7 days | 9091 MB |
| 10 days | 12986 MB |
| 30 days | 38958 MB |
GIMR database (MGMTDB) location
Starting with Oracle database 12c, the GIMR database (MGMTDB) is by default created within the same Filesystem/ASM-Diskgroup as OCR or VOTING. During the installation of Clusterware (GI) binaries, the OUI will fetch the location (ASM Diskgroup/ File system) of the OCR and VOTING and utilizes the first location to create the datafiles for the MGMTDB database.
For example, if we have the following locations for OCR or VOTING
---// voting disk location //---
myracserver2 {/home/oracle}: crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 38aaf08ea3c74ffabfd258876dd6f97c (/data/clusterfiles/copy1/VOTE-disk01) []
2. ONLINE 97d73bdbe42c4fa4bfa3c3cb7d741583 (/data/clusterfiles/copy2/VOTE-disk02) []
3. ONLINE 00b6b258e0724f6cbf1dc6a03d15fd87 (/data/clusterfiles/copy3/VOTE-disk03) []
Located 3 voting disk(s).
Oracle Universal Installer (OUI) will choose the first location i.e. (/data/clusterfiles/copy1) to create the datafiles for repository database MGMTDB. This can be a problem if we have limited space available on the underlying file system and we want to have a higher retention for the diagnostic data in the repository. It also has a potential to impact the OCR/Voting disk availability.
We can however, relocate the MGMTDB database to a different storage location manually as per the MOS Note 1589394.1 or using the MDBUtil tool as per MOS Note 2065175.1
GIMR Clusterware (GI) components
With the introduction of the repository database MGMTDB, we have now two additional components included in the Clusterware stack and these are ora.mgmtdb (repository database resource) and ora.MGMTLSNR (repository database listener) as shown below:
---// GIMR Clusterware resources //---
myracserver2 {/home/oracle}: crsstat | grep -i mgmt
ora.MGMTLSNR mgmtlsnr ONLINE ONLINE on myracserver2 192.168.230.15 10.205.87.231
ora.mgmtdb mgmtdb ONLINE ONLINE on myracserver2 Open
Unlike a generic Clusterware database or listener resource, these two resources have their own set of Clusterware commands as listed below.
---// list of srvctl commands available to operate on GIMR resources //---
myracserver2 {/home/oracle}: srvctl -h | grep -i mgmt | sort | awk -F ":" '{print $2}'
srvctl add mgmtdb [-domain ]
srvctl add mgmtlsnr [-endpoints "[TCP
srvctl config mgmtdb [-verbose] [-all]
srvctl config mgmtlsnr [-all]
srvctl disable mgmtdb [-node ]
srvctl disable mgmtlsnr [-node ]
srvctl enable mgmtdb [-node ]
srvctl enable mgmtlsnr [-node ]
srvctl getenv mgmtdb [-envs "[,...]"]
srvctl getenv mgmtlsnr [ -envs "[,...]"]
srvctl modify mgmtdb [-pwfile ] [-spfile ]
srvctl modify mgmtlsnr -endpoints "[TCP
srvctl relocate mgmtdb [-node ]
srvctl remove mgmtdb [-force] [-noprompt] [-verbose]
srvctl remove mgmtlsnr [-force]
srvctl setenv mgmtdb {-envs "=[,...]" | -env ""}
srvctl setenv mgmtlsnr { -envs "=[,...]" | -env "="}
srvctl start mgmtdb [-startoption ] [-node ]
srvctl start mgmtlsnr [-node ]
srvctl status mgmtdb [-verbose]
srvctl status mgmtlsnr [-verbose]
srvctl stop mgmtdb [-stopoption ] [-force]
srvctl stop mgmtlsnr [-node ] [-force]
srvctl unsetenv mgmtdb -envs "[,..]"
srvctl unsetenv mgmtlsnr -envs "[,...]"
srvctl update mgmtdb -startoption
myracserver2 {/home/oracle}:
Locating the GIMR database (MGMTDB)
The repository database (MGMTDB) always runs as a single node instance. We can locate the node hosting MGMTDB in any of the following way.
Using SRVCTL commands
To locate MGMTDB database: srvctl status mgmtdb
To locate MGMTLSNR listener: srvctl status mgmtlsnr
Example:
---// use srvctl to find MGMTDB //---
myracserver2 {/home/oracle}: srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node myracserver2
---// use srvctl to find MGMTLSNR //---
myracserver2 {/home/oracle}: srvctl status mgmtlsnr
Listener MGMTLSNR is enabled
Listener MGMTLSNR is running on node(s): myracserver2
Using CRSCTL commands
To locate MGMTDB database: $GRID_HOME/bin/crsctl status resource ora.mgmtdb
To locate MGMTLSNR listener: $GRID_HOME/bin/crsctl status resource ora.MGMTLSNR
Example:
---// use crsctl to find MGMTDB //---
myracserver2 {/home/oracle}: crsctl status resource ora.mgmtdb
NAME=ora.mgmtdb
TYPE=ora.mgmtdb.type
TARGET=ONLINE
STATE=ONLINE on myracserver2
---// use crsctl to find MGMTLSNR //---
myracserver2 {/home/oracle}: crsctl status resource ora.MGMTLSNR
NAME=ora.MGMTLSNR
TYPE=ora.mgmtlsnr.type
TARGET=ONLINE
STATE=ONLINE on myracserver2
Using OCLUMON utility
To locate the node hosting repository: $GRID_HOME/bin/oclumon manage -get master
Example:
---// use oclumon utility to locate node hosting GIMR //---
myracserver2 {/home/oracle}: oclumon manage -get master
Master = myracserver2
myracserver2 {/home/oracle}:
On the hosting node, we can identify the processes associated with these repository database resources as follows
---// locating MGMTDB on the master node //---
myracserver2 {/home/oracle}: ps -ef | grep pmon | grep MGMT
oracle 2891 1 0 06:35 ? 00:00:01 mdb_pmon_-MGMTDB
---// locating MGMTLSNR on the master node //---
myracserver2 {/home/oracle}: ps -ef | grep tns | grep MGMT
oracle 17666 1 0 05:23 ? 00:00:00 /app/grid/12.1.0.2/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
myracserver2 {/home/oracle}:
The repository database (MGMTDB) is by default associated with SID"-MGMTDB" and an equivalent entry can be located in /etc/oratab as shown below.
---// oratab entry for GIMR database MGMTDB //---
myracserver2 {/home/oracle}: grep -i mgmt /etc/oratab
-MGMTDB:/app/grid/12.1.0.2:N
myracserver2 {/home/oracle}:
Explore the GIMR database (MGMTDB)
As mentioned earlier in the introductory section, the repository database MGMTDB is created during the Clusterware installation process. This database is a single instance CONTAINER database (CDB) and has only one PLUGGABLE database (PDB) associated with it apart from the SEED database. The pluggable database is the actual repository holding all the diagnostic information. The container (CDB) database is named as _MGMTDB where as the pluggable (PDB) database is named after the Cluster name (with hyphen "-" replaced as underscore "_" in the cluster name)
Example:
---// GIMR container database MGMTDB information //---
SQL> select name,db_unique_name,host_name,cdb from v$database,v$instance;
NAME DB_UNIQUE_NAME HOST_NAME CDB
--------- ------------------------------ -------------------- ---
_MGMTDB _mgmtdb myracserver2 YES
---// pluggable database holding the actual data //---
SQL> select CON_ID,DBID,NAME,OPEN_MODE from v$containers;
CON_ID DBID NAME OPEN_MODE
---------- ---------- ------------------------------ ----------
1 1091149818 CDB$ROOT READ WRITE --> root container CDB
2 1260861561 PDB$SEED READ ONLY --> seed database3 521100791 MY_RAC_CLUSTER READ WRITE--> actual repository---// actual repository is named after the cluster name //---
myracserver2 {/home/oracle}: olsnodes -c
my-rac-cluster
Note: In case where the Cluster name has hyphen (-) in between the name it gets replaced by an underscore (_) while naming the pluggable (PDB) database.
The management repository database (MGMTDB.[pdb_name]) is comprised of the following tablespaces.
TABLESPACE_NAME FILE_NAME SIZE_MB MAX_MB AUT
------------------------------ ---------------------------------------------------------------------- ---------- ---------- ---
SYSAUX /data/clusterfiles/_MGMTDB/datafile/o1_mf_sysaux__2318922894015_.dbf 150 32767.9844 YES
SYSGRIDHOMEDATA /data/clusterfiles/_MGMTDB/datafile/o1_mf_sysgridh__2318922910141_.dbf 100 32767.9844 YES
SYSMGMTDATA /data/clusterfiles/_MGMTDB/datafile/o1_mf_sysmgmtd__2318922860778_.dbf 2048 0 NO
SYSMGMTDATADB /data/clusterfiles/_MGMTDB/datafile/o1_mf_sysmgmtd__2318922925741_.dbf 100 0 NO
SYSTEM /data/clusterfiles/_MGMTDB/datafile/o1_mf_system__2318922876379_.dbf 160 32767.9844 YES
USERS /data/clusterfiles/_MGMTDB/datafile/o1_mf_users__2318922940656_.dbf 5 32767.9844 YES
Where:-
| TABLESPACE | DESCRIPTION |
|---|---|
| SYSMGMTDATA | This is the primary tablespace and the repository which is used to store the diagnostic data collected by the Cluster Health Monitor (CHM) tool. |
| SYSMGMTDATADB | There is not much details available about this tablespace and by default, it doesn't contain any object. However, I assume it has something to do with the Change Assistant. |
| SYSGRIDHOMEDATA | This tablespace is used to store data related to Rapid Home provisioning (in cloud database context). By default, it doesn't contain any objects in it |
Note: In this example, the repository datafiles are not using the default location (OCR/Vote-disk filesystem). I had relocated the repository to different storage location
These set of tablespaces are mapped to the following list of users
---// database users owning repository objects/data //---
SQL> select username,account_status,default_tablespace from dba_users
2 where default_tablespace in ('SYSGRIDHOMEDATA','SYSMGMTDATA','SYSMGMTDATADB');
USERNAME ACCOUNT_STATUS DEFAULT_TABLESPACE
-------------- -------------------------------- ------------------------------
GHSUSER EXPIRED & LOCKED SYSGRIDHOMEDATA
CHM OPEN SYSMGMTDATA --> User mapped to the Cluster Health Monitor (CHM)
CHA EXPIRED & LOCKED SYSMGMTDATADB
By default, only CHM database account is unlocked and is in turn used by the Cluster Health Monitor (CHM) to store the Clusterware diagnostic data in the database. The user account GHSUSER is used for Rapid Home provisioning (in cloud database context) and comes in to picture only when Rapid Home provisioning is used. The CHA user account is related to Cluster Health Adviser (CHA), which is an improvised version of Cluster Health Monitor (CHM) and would be available in the upcoming release of Oracle Clusterware.
Conclusion
Clusterware diagnostic repository has evolved a lot with Oracle Database 12c. Having a dedicated Oracle database for repository also brings more clarity in terms how the diagnostic data is stored by Oracle as well as it opens up multiple ways to query those diagnostic data. Oracle is likely to leverage the GIMR to store a variety of diagnostic and management data in the upcoming releases.
With Organizations rapidly moving to Oracle cloud infrastructure, this repository database is also going to be extensively used for storing the provisioning metadata used by the cloud deployments.