2012-10-25

OEM Jobs Report


Environment
Oracle Enterprise Manager (OEM) Grid Control 11.1

Report
This report shows backup jobs scheduled in the given time interval. It also shows HOST name and IP address of the database target which is handy for preparing for network, server or co-location outages.
Modify parameters in the end of the SQL.
Attention! The report looks for NEXT execution time (not all future ones). If a job is supposed to run before the given interval as well as in the interval, it will not appear in the report.

Run it in OEM repository (OMR) as any user who can select SYSMAN tables.

SELECT
        j.job_name,
        t.target_name, --t.target_type, t.target_guid
        --j.job_owner,  j.job_type,
        --j.job_id,
        --e.execution_id,  e.start_time, e.end_time,
        DECODE(status,
                1, 'Scheduled',
                2, 'Running',
                3, 'Error',
                4, 'Failed',
                5, 'Succeeded',
                6, 'Suspended By User',
                7, 'Suspended: Agent Unreacheable',
                8, 'Stopped',
                9, 'Suspended on Lock',
               10, 'Suspended on Event',
               11, 'Stop Pending',
               13, 'Suspend Pending',
               14, 'Inactive',
               15, 'Queued',
               16, 'Failed Retried',
               17, 'Suspended',
               18, 'Skipped', status) status,
        e.scheduled_time,
        DECODE(s.timezone_info, 3, to_char(s.timezone_offset), s.timezone_region) timezone_region,
        DECODE (s.timezone_info,
                    1, 'Repository',
                    2, 'Agent',
                    3, 'Specified Offset/Region',
                    4, 'Specified Offset/Region',
                    s.timezone_info) timezone_type,
        DECODE (s.frequency_code, 1, 'One Time',
                                     2, 'Interval',
                                     3, 'Daily',
                                     4, 'Weekly',
                                     5, 'Monthy',
                                     6, 'Yearly') schedule_type,
        s.interval,
        TH.TARGET_NAME host_name, TP.PROPERTY_VALUE IP_address
      FROM
        sysman.MGMT_JOB j,
        sysman.MGMT_JOB_EXEC_SUMMARY e,
        sysman.MGMT_JOB_TARGET jt,
        sysman.MGMT_TARGETS t,
        sysman.MGMT_TARGET_ASSOCS ta,
        sysman.MGMT_TARGETS th,
        sysman.MGMT_TARGET_PROPERTIES tp,
        sysman.MGMT_JOB_SCHEDULE s
      WHERE
        j.schedule_id = s.schedule_id AND
        J.IS_LIBRARY = 0 and
        j.system_job = 0 AND
        j.nested=0 AND
        j.job_id=e.job_id AND
        j.is_corrective_action=0 AND
        e.job_id=jt.job_id (+) AND
        e.execution_id=jt.execution_id (+) AND
        jt.target_guid=t.target_guid (+)
        and jt.target_guid=TA.SOURCE_TARGET_GUID
        and TA.ASSOC_TARGET_GUID=Th.TARGET_GUID and Th.TARGET_TYPE='host'
        and TA.ASSOC_TARGET_GUID=TP.TARGET_GUID and TP.PROPERTY_NAME='IP_address'
        --
        -- you can modify the following parameters
        --
        and Job_name like 'BACKUP%'
        --and status = 1 -- Scheduled
        and scheduled_time between to_date('2012-10-27 16:00','yyyy-mm-dd hh24:mi') and to_date('2012-10-28 10:00','yyyy-mm-dd hh24:mi');

2012-10-05

High temp space consumption by OEM agent DBSNMP

Environment
Solaris 10, Oracle 11.2.0.2, OEM agent 11.1.0.1.0

Symptoms
Agent (DBSNMP) is consuming CPU and huge amount of temp space by selecting from v$lock

    /* OracleOEM */ WITH blocked_resources AS
(select id1 ,id2 ,SUM(ctime) as blocked_secs ,MAX(request) as max_request ,COUNT(1) as blocked_count
from v$lock
where request > 0 group by id1,id2 ) ,blockers AS
(select L.* ,BR.blocked_secs ,BR.blocked_count
from v$lock L ,blocked_resources BR
where BR.id1 = L.id1 and BR.id2 = L.id2 and L.lmode > 0 and L.block <> 0 )
select B.id1||'_'||B.id2||'_'||S.sid||'_'||S.serial# as id ,'SID,SERIAL:'||S.sid||','||S.serial#||',LOCK_TYPE:'||B.type||',PROGRAM:'||S.program||',MODULE:'||S.module||',ACTION:'||S.action||',MACHINE:'||S.machine||',OSUSER:'||S.osuser||',USERNAME:'||S.username as info ,B.blocked_secs ,B.blocked_count
from v$session S ,blockers B
where B.sid = S.sid;

Solution
Check and gather statistics for fixed objects.
select OWNER, TABLE_NAME, object_type, LAST_ANALYZED from dba_tab_statistics where table_name IN
('X$KSUSE',
'X$KDNSSF',
'X$KSQEQ',
'X$KTADM',
'X$KTATRFIL',
'X$KTATRFSL',
'X$KTATL',
'X$KTSTUSC',
'X$KTSTUSS',
'X$KTSTUSG',
'X$KTCXB',
'X$KSQRS',
'X$KSLWT',
'X$KSLED');

exec dbms_stats.GATHER_FIXED_OBJECTS_STATS;

Useful Info
Check the following Oracle Support documents
Query Against v$lock Run from OEM Performs Slowly [ID 1328789.1]
High temp space consumption by DBSNMP [ID 1389377.1]
EM Agent DBSNMP Using Up Excessive Temp Space In Database [ID 748251.1]

2012-09-27

Cloning Oracle 11.2 database installation on Solaris


Environment

Oracle 11.2.0.2, Solaris 10

Packing Source Oracle Home

export CLONE=/oracle/stage/clone11.2.0.2
echo $CLONE
cd $ORACLE_HOME

Optionally you can create Exclude List for tar command to exclude some files from source home.
Log files:
find . -name "*.log" >$CLONE/ExcludeList.txt
find . -name "*.err" >>$CLONE/ExcludeList.txt
Database templates:
find ./assistants/dbca/templates >>$CLONE/ExcludeList.txt
Network files:
find ./network/admin >>$CLONE/ExcludeList.txt

Check your Exclude file:
cat $CLONE/ExcludeList.txt

tar -cvfX - $CLONE/ExcludeList.txt . | gzip -c >$CLONE/oracle_home11.2.tar.gz

If source and target are on the same host you can use
cp -Rp $ORACLE_HOME
Ignore errors for files owned by root (nmb, nmo, etc.)

Cloning Oracle Home

Check or create oraInst.loc
cat /var/opt/oracle/oraInst.loc

Optionally J check known issues in
Master Note For Cloning Oracle Database Server ORACLE_HOME's Using the Oracle Universal Installer (OUI) [ID 1154613.1]

Navigate to new Oracle home:
cd /oracle/product/11.2.0.2/db_2
or if you have set the variables
cd $ORACLE_HOME
pwd
Unpack Oracle home to new location
gunzip < /oracle/stage/clone11.2.0.2/oracle_home11.2.tar.gz | tar xvf –

unset LD_LIBRARY_PATH
unset LD_LIBRARY_PATH_64
Check that other variables do not point to different Oracle home or unset them
echo $PATH
unset ORACLE_HOME
unset ORACLE_BASE
unset TNS_ADMIN

cd clone/bin

perl clone.pl ORACLE_HOME="/oracle/product/11.2.0.2/db_2" ORACLE_HOME_NAME="OraDB11g_home2" ORACLE_BASE="/oracle" OSDBA_GROUP=dba OSOPER_GROUP=oper

as root run
/oracle/rdbms/11.2.0.2/ora10R4/root.sh

Update
listener.ora
tnsnames.ora

Useful Links

Master Note For Cloning Oracle Database Server ORACLE_HOME's Using the Oracle Universal Installer (OUI) [ID 1154613.1]
How To Clone An Existing RDBMS Installation Using OUI [ID 300062.1]
FAQs on RDBMS Oracle Home Cloning Using OUI [ID 565009.1]


2012-05-17

Host preferred credentials failure after cloning Agent OEM 11g

Environment
AIX 6.1
Oracle Enterprise Manager (OEM) Grid Control 11.1

Problem
There is an issue with host preferred credentials in Grid Control .
Agent cannot connect to host .
Agent home and database home are owned by different UNIX users.

Cause
OUI is run as part of agent cloning procedure, e.g.:
cd $ORACLE_HOME/oui/bin
./runInstaller -clone -forceClone ORACLE_HOME=/oracle/oem/agent11g ORACLE_HOME_NAME=OH_agent11g -noconfig -silent OMS_HOST=oms.domain.com OMS_PORT=4889

During the cloning OUI creates about a hundred backup files (*.ouibak).
You can see and delete them as:
find $ORACLE_HOME -name "*.ouibak" -ls
find $ORACLE_HOME -name "*.ouibak" -exec rm {} \;

There is a catch here. OUI creates new files with different permissions.
For example in $ORACLE_HOME/perl/bin, all files have 755 before cloning, and 700 after cloning.
If oracle agent and oracle database homes are owned by different UNIX users, even the same group (oinstall), Oracle agent cannot connect to host when you test host preferred credentials in OEM.

When switching the agent into DEBUG mode, you can see in the trace file $ORACLE_HOME/sysman/log/emagent.trc, that the user (database owner) cannot execute perl.
In emagent.nohup, you can see “ERROR: Execution failed due to binary missing or permission issues”.

Solution
At least grant 750 (if owners in the same group - oinstall) or 755 to $OH/perl/bin/perl
Or for all files in $OH/perl
find $ORACLE_HOME/perl -perm 700 -exec chmod g+rx {} \;

Useful info
How to debug agent:
Troubleshooting Host Credentials Issues in Grid Control Console [ID 565414.1]
About perl permission issue:
Testing Host Credentials fails with ERROR: Execution failed due to binary missing or permission issues [ID 1294136.1]

Configuring standby OMS (OEM 11g)


Environment
Oracle Enterprise Manager 11.1
UNIX AIX 6.1

This is a simple configuration. There are two OMS hosts in different co-locations (host1, host2) and common DNS alias (oms).
There is no hardware SLB in this example.

Tips
When you configure OMS with SLB, DNS alias (oms.domain.com) should have different IP address from local hostname.

If they are the same, the result is:
Console Server Host : oms.domain.com
OMS is not configured with SLB or virtual hostname

If they are different, the result is:
Console Server Host : host1.domain.com
SLB or virtual hostname: oms.domain.com

After securing OMS with SLB alias, restart OMS (I didn’t see it in Oracle documentation), otherwise agents will not be able to register with the alias:
emctl secure oms -host oms.domain.com
emctl stop oms -all
emctl start oms

After configuring OMS as SLB/alias, you can point the alias oms.domain.com to host1.

Here is Oracle documentation how to Setup Standby Management Server.