Friday 23 December 2011

Connection 1,***,6201 connect (Connection refused)

PROBLEM:Con­nec­tion 1,***,6201 con­nect (Con­nec­tion refused) where *** is the server’s ip address might occur on Ora­cle Appli­ca­tion Server 10g Infra­struc­ture or MiddleTier. I faced this problem when trying to apply patch on OID infrastructure.


SOLUTION: Try to understand where you problem is and then try this solution
This prob­lem is due to file per­mis­sions on the Appli­ca­tion Server direc­tory , try a chmod –R oracle:dba on the direc­tory and then run root.sh on both infra­struc­ture and portal. For my specific problem I ran root.sh from OID home which resolved the problem.




Thursday 24 November 2011

RMAN backup details require to restore a database

 When you have to restore a database from RMAN backup, last thing you want to do is to identify which backups, tapes and dates you need to use for database restore. There are few sql commands that can be very handy in order to extract those details.


column handle format a60
column media format a10
column tag format a20
column start_time format a20
column name format a10
set lines 200
    select d.name,    p.tag,   p.handle , p.media ,    s.incremental_level "LEVEL",
     to_char(s.start_time, 'DD-MON-YY HH24:MI') start_time,
     s.elapsed_seconds/60 "MINUTES",   p.backup_type
      from  RC_DATABASE d, RC_BACKUP_PIECE p, RC_BACKUP_SET s  
      where
        d.name='ELPM'  and
        d.db_key = p.db_key  and
        s.db_key = p.db_key     and
        p.bs_key = s.bs_key      and
       s.start_time between to_date('08-SEP-11 00:00','dd-MON-yy hh24:mi')
       and to_date('10-SEP-11 00:00','dd-MON-yy hh24:mi')
      Above script can give you backup details by providing database name and dates after connecting to RMAN repository.
      Another method is to connect to RMAN prompt using control file

      $>rman connect target / nocatalog

      RMAN> restore database preview summary;





      Changing VIP address in Oracle RAC 11g R1

      I just want to share my experience with changing ip addresses of running Oracle RAC.
      Before I go into details , let me explain my environment. This is real-life experience on a client site which was running 11G R1 ASM and CRS but databases are running on 10G R2 on Red Hat release 4 (RHEL4). So ip address changes require on CRS level.

      First of all I just go through the steps that I have taken according to the issue i was facing. Later I'll try to add other steps/command to cover as much as possible around this topic.


      This case was simple as only thing that was changed was ip address and subnet and LAN remain the same.

      1. Stop all databases on all RAC nodes by using this command
      srvctl stop database -d poem
      srvctl stop database -d poid
      srvctl stop database -d prman

      2.Stop all nodeapps on every node
      NODE 1

      srvctl stop nodeapps -n dbm1301

      NODE 2
      srvctl stop nodeapps -n dbm1401


      3. Make sure that ip addresses has been changed in dns for vip.
      ping dbm1301-vip

      checks that it return new ip address.

      4. Check that which bond this vip is using by runing ifconfig

      /sbin/ifconfig

      if it is using bond0:1 then in CRS configuration you have to specify bond0 which will create an alias of bond0:1. In my case bond0 is bonded with eth0 which is using production LAN.

      5. Run the following command just in case to enforce the new ip address

      /ora/product/11.1.0/crs/bin/srvctl modify nodeapps -n dbm1301 -A dbm1301-vip/255.255.255.0/bond0


      6. restart CRS to pick up new ip address
      crsctl stop crs

      crsctl start crs


      7. Restart listener so that it picks up new ip address too.
      srvctl stop listener -n dbm1301
      srvctl start listener -n dbm1301

      8. repeat the same steps on other node

      9. Start nodes apps and databases
      srvctl start nodeapps -n dbm1301

      srvctl start database -d poem
      srvctl start database -d prman
      srvctl start database -d poid

      10. check that all services are running
      srvctl status service -d poem
      if need to start it then use srvctl start service -d -s  

      This has changed ip address on a node which was running RAC successfully.



      Another brilliant article is here
      http://www.db-nemec.com/RAC_IP_Change.html







      Monday 4 July 2011

      RESTORING A PHYSICAL STANDBY TO DIFFERENT LOCATION

      Restoring a PHYSICAL STANDBY database to a different location …

      First make memory usage of your system
      You do this, in the case we need to remove shared memory or semaphore.
      # ipcs -b

      Assuming that you have already use RMAN to backup your standby database into flashback recovery area (db_recovery_file_dest).
                                   
      So, if your  db_recovery_file_dest=’/recovery_area’, then your backup will be at the following location by default. On the new server create the

      recovery area and restore the backup.

      Default:
      /recovery_area//backupset
      /recovery_area//autobackup

      On the empty server, we assume you have already cloned the Oracle binaries and set your global variables such as ORACLE_HOME, ORACLE_SID, etc.

      Create external password:
      # orapwd file=@ORACLE_HOME/dbs/orapw password=your-sys-pass entries=3

      Run the RMAN tool and login as sys. Then set your DBID to restore spfile and controlfiles.  If you don’t know where your controlfiles are located

      first restore spfile then create pfile and check the location and then build it if they are not exist.

      # rman target /
      RMAN> SET DBID xxxxxxxxxxxx;
      RMAN> STARTUP FORCE NOMOUNT;
      RMAN> show all;    --  to display default rman configurations.

      To restore spfile:
      RMAN> restore spfile from autobackup
                  db_recovery_file_dest=’/recovery_area’ db_unique_name=’xxxx’_stdby;
      --or—
      RMAN> restore spfile from autobackup
                  db_recovery_file_dest=’/recovery_area’ db_name=’xxxx’;
      Now, you can goto SQLPLUS and create your parameter file.
      --or—
      If your spfile and controlfile were backup on $ORACLE_HOME/dbs, then do the following:
      RMAN> restore spfile from autobackup;
      SQL> create pfile from spfile;


      To restore controlfiles:
      Shutdown, make sure all the directories including audit file and then startup with no mount.
      RMAN> shutdown immediate;
      --Make sure all directories exist including “adump.”
      RMAN> startup nomount;
      RMAN> restore controlfile from autobackup
                 db_recovery_file_dest=’/recovery_area’ db_unique_name=’xxxx_stdby’;
      --or--
      RMAN> restore controlfile from autobackup
                db_recovery_file_dest=’/recovery_area’ db_name=’xxxx’;
      --or--
      If your spfile and controlfile were backup on $ORACLE_HOME/dbs, then do the following:
      RMAN> restore controlfile from autobackup;

      Mout the instance:
      SQL> shutdown immediate;
      SQL> startup mount;

      Locate the database files destinations:
      Note: if your controlfiles’ destination were not set, rman will copy one to $ORACLE_HOME/dbs/cntrl.dbf.
      Now, from parameter file, check the diagnostic_dest, audit_file_dest, db_recovery_file_dest, etc.
      Use the trace file by backing controlfile to trace to locate other data files destinations.
      SQL> ALTER DATABASE BACKUP controlfile TO TRACE; -- located at udump.

      Restore the database:
      If the database is not mounted:
      SQL> alter database mount;
      RMAN> restore database;

      If you have missing logs do the following:
      Now go to your archivelog location and check your last archivelog sequence number. It should be more than value from the following query.
      SQL> SELELCT * FROM v$log;

      Then recover until the last sequence number;
      RMAN> recover database until sequence your-number;

      If you have get flashback recovery on/off error message, try to turn it off.
      SQL> ALTER DATABASE flashback OFF;
      SQL> shutdown immediate;
      SQL> startup mount;
      --or--
      SQL> recover database;  --if you have all logs
      --or--
      SQL> recover database until sequence 52;
      --or--
      SQL> recover database using backup controlfile until sequence 52;
      --or--
      SQL> recover database using backup controlfile until cancel;

      Important:
      After the recover make sure that you have all your redo logs plus standby.
      SQL> SELECT * FROM gv$logfile;
      To drop it we you are on the mount mode:
      SQL>ALTER DATABASE DROP LOGFILE group nnn;

      Add them if there are not there in directory. You may see them on query but not physically on disk.
      Example:
      SQL> alter database add standby logfile
        2  '/oradata/your-db_name/stndby_redo/stndby04.log' size 50m reuse;


      SQL> ALTER DATABASE OPEN resetlogs;

      If you hav all your archivelogs:
      Go to primary database and switch logfile and copy all archivelogs to the new server. Then create trace file with noresetlogs option.
      # scp arch*.log oracle@yournew-server:$DB_RECOVERY_FILE_AREA//archivelog/.

      On SQLPLUS:
      SQL> ALTER DATABASE BACKUP CONTROLFILE TO TRACE as '/home/oracle/myprim-cntl.sql' NORESETLOGS;

      Copy the myprim-cntl.sql file to your new server. And edit it to replace the existing controlfiles.

      Shutdown the database.
      SQL> shutdown immediate;
      SQL> @myprim-cntl.sql

      Notice:
      If you get hange, make sure you check the memory usage:
      # ipcs –b

      Use the following command to remove them from shared memory or semaphores if needed.
      # ipcrm –m
      # ipcrm –s

      Friday 1 July 2011

      SOME ASM COMMANDS


      ASM packages installed a service called oracleasm which can be called by running these commands

      # /etc/init.d/oracleasm
      # service oracleasm



      /etc/init.d/oracleasm script takes the following options:
      configure: Configure ASM.
      start: Start the ASM service.
      stop: Stop the ASM service.
      status: Print the status of ASM service.
      createdisk: Create an ASM disk.
      scandisks: Search for ASM disks.
      listdisks: List ASM disks.
      querydisk: Query the current status of an ASM disk.
      deletedisk: Delete an ASM disk.
      restart: Restart the ASM service.
      link: Load the oracleasm module.
      enable: Enable the ASM service.
      disable: Disable the ASM service.



      Starting ASM service

      # /etc/init.d/oracleasm start

      To stop ASM service

      # /etc/init.d/oracleasm stop

      To check status of ASM service
      # /etc/init.d/oracleasm status

      CREATEDISK
      # /etc/init.d/oracleasm createdisk VOL1 /dev/sdb1
      Marking disk "/dev/sdb1" as an ASM disk [ OK ]
       # /etc/init.d/oracleasm createdisk VOL2 /dev/sdc1
      Marking disk "/dev/sdc1" as an ASM disk [ OK ]
       # /etc/init.d/oracleasm createdisk VOL3 /dev/sdd1
      Marking disk "/dev/sdd1" as an ASM disk [ OK ]


      SCANDISK
      # /etc/init.d/oracleasm scandisks


      LISTDISK to verify the disks are available
      # /etc/init.d/oracleasm listdisks

      To query the status of existing disk
      # /etc/init.d/oracleasm querydisk VOL1

      DELETEDISK
      # /etc/init.d/oracleasm deletedisk VOL1


      Deleting an ASM Instance Manually
      delete the ASM instance from the OCR:

       $ srvctl remove asm -n DB1 -i +ASM1
       $ srvctl remove asm -n DB2 -i +ASM2


      On each node, delete the ASM instance from /etc/oratab. Then, on each node, delete the
      parameter file, as in the following example:
      [oracle@london1 oracle] $ rm $ORACLE_HOME/dbs/init+ASM.ora

      Finally, on each node, delete the password file:
      [oracle@london1 oracle] $ rm $ORACLE_HOME/dbs/orapw+ASM1


      Col name form a30
      Col failgroup form a30
      Select name,failgroup from v$asm_disk
      Where group_number=(
      Select group_number from v$asm_diskgroup
      Where name=’G3_RECOVERY_AREA’)
      /

      Checking the operation of rebalancing
      Select * from gv$asm_operation;



      ################################################################
      # Adding/Removing/Managing configuration about existing ASM instances################################################################

      --Use the following syntax to add configuration information about an existing ASM instance:
      srvctl add asm -n node_name -i +asm_instance_name -o oracle_home

      --Use the following syntax to remove an ASM instance:
      srvctl remove asm -n node_name [-i +asm_instance_name]

      --Use the following syntax to enable an ASM instance:
      srvctl enable asm -n node_name [-i ] +asm_instance_name

      --Use the following syntax to disable an ASM instance:
      srvctl disable asm -n node_name [-i +asm_instance_name]

      --Use the following syntax to start an ASM instance:
      srvctl start asm -n node_name [-i +asm_instance_name] [-o start_options]

      --Use the following syntax to stop an ASM instance:
      srvctl stop asm -n node_name [-i +asm_instance_name] [-o stop_options]

      --Use the following syntax to show the configuration of an ASM instance:
      srvctl config asm -n node_name

      --Use the following syntax to obtain the status of an ASM instance:
      srvctl status asm -n node_name

      --P.S.:
      --For all of the SRVCTL commands in this section for which the
      --option is not required, if you do not specify an instance name, then -i
      --the command applies to all of the ASM instances on the node.

      ###################################
      # Managing DiskGroup inside ASM:
      ###################################

      --Note that adding or dropping disks will initiate a rebalance of the data on the disks.
      --The status of these processes can be shown by selecting from v$asm_operation.


      --Quering ASM Disk Groups
      col name format a25
      col DATABASE_COMPATIBILITY format a10
      col COMPATIBILITY format a10
      select * from v$asm_diskgroup;
      --or
      select name, state, type, total_mb, free_mb from v$asm_diskgroup;

      --Quering ASM Disks
      col PATH format a55
      col name format a25
      select name, path, group_number, TOTAL_MB, FREE_MB, READS, WRITES, READ_TIME,
      WRITE_TIME from v$asm_disk order by 3,1;
      --or
      col PATH format a50
      col HEADER_STATUS  format a12
      col name format a25
      --select INCARNATION,
      select name, path, MOUNT_STATUS,HEADER_STATUS, MODE_STATUS, STATE, group_number,
      OS_MB, TOTAL_MB, FREE_MB, READS, WRITES, READ_TIME, WRITE_TIME, BYTES_READ,
      BYTES_WRITTEN, REPAIR_TIMER, MOUNT_DATE, CREATE_DATE from v$asm_disk;

      ###################################
      #TUNING and Analisys###################################
      --Only Performance Statistics
      --N.B Time in Hundredseconds!
      col READ_TIME format 9999999999.99
      col WRITE_TIME format 9999999999.99
      col BYTES_READ format 99999999999999.99
      col BYTES_WRITTEN  format 99999999999999.99
      select name, STATE, group_number, TOTAL_MB, FREE_MB,READS, WRITES, READ_TIME,
       WRITE_TIME, BYTES_READ, BYTES_WRITTEN, REPAIR_TIMER,MOUNT_DATE
      from v$asm_disk order by group_number, name;


      --Check the Num of Extents in use per Disk inside one Disk Group.
      select max(substr(name,1,30)) group_name, count(PXN_KFFXP) extents_per_disk,
      DISK_KFFXP, GROUP_KFFXP from x$kffxp, v$ASM_DISKGROUP gr
      where GROUP_KFFXP=&group_nr and GROUP_KFFXP=GROUP_NUMBER
      group by GROUP_KFFXP, DISK_KFFXP order by GROUP_KFFXP, DISK_KFFXP;


      --Find The File distribution Between Disks
      SELECT * FROM v$asm_alias  WHERE  name='PWX_DATA.272.669293645';

      SELECT GROUP_KFFXP Group#,DISK_KFFXP Disk#,AU_KFFXP AU#,XNUM_KFFXP Extent#
      FROM   X$KFFXP WHERE  number_kffxp=(SELECT file_number FROM v$asm_alias
      WHERE name='PWX_DATA.272.669293645');

      --or

      SELECT GROUP_KFFXP Group#,DISK_KFFXP Disk#,AU_KFFXP AU#,XNUM_KFFXP Extent#
      FROM X$KFFXP WHERE  number_kffxp=&DataFile_Number;

      --or
      select d.name, XV.GROUP_KFFXP Group#, XV.DISK_KFFXP Disk#,
      XV.NUMBER_KFFXP File_Number, XV.AU_KFFXP AU#, XV.XNUM_KFFXP Extent#,
      XV.ADDR, XV.INDX, XV.INST_ID, XV.COMPOUND_KFFXP, XV.INCARN_KFFXP,
      XV.PXN_KFFXP, XV.XNUM_KFFXP,XV.LXN_KFFXP, XV.FLAGS_KFFXP,
      XV.CHK_KFFXP, XV.SIZE_KFFXP from v$asm_disk d, X$KFFXP XV
      where d.GROUP_NUMBER=XV.GROUP_KFFXP and d.DISK_NUMBER=XV.DISK_KFFXP
      and number_kffxp=&File_NUM order by 2,3,4;


      --List the hierarchical tree of files stored in the diskgroup
      SELECT concat('+'||gname, sys_connect_by_path(aname, '/')) full_alias_path FROM
      (SELECT g.name gname, a.parent_index pindex, a.name aname,
      a.reference_index rindex FROM v$asm_alias a, v$asm_diskgroup g
      WHERE a.group_number = g.group_number)
      START WITH (mod(pindex, power(2, 24))) = 0
      CONNECT BY PRIOR rindex = pindex;
      ################################################################

      ###################################
      #Create and Modify Disk Group
      ###################################

      create diskgroup FRA1 external redundancy disk '/dev/vx/rdsk/oraASMdg/fra1'
      ATTRIBUTE 'compatible.rdbms' = '11.1', 'compatible.asm' = '11.1';

      alter diskgroup FRA1  check all;

      --on +ASM2 :
      alter diskgroup FRA1 mount;


      --Add a second disk:
      alter diskgroup FRA1 add disk '/dev/vx/rdsk/oraASMdg/fra2';


      --Add several disks with a wildcard:
      alter diskgroup FRA1 add disk '/dev/vx/rdsk/oraASMdg/fra*';


      --Remove a disk from a diskgroup:
      alter diskgroup FRA1 drop disk 'FRA1_0002';


      --Drop the entire DiskGroup
      drop diskgroup DATA1 including contents;

      --How to DROP the entire DiskGroup when it is in NOMOUNT Status
      --Generate the dd command which will reset the header of all the
      --disks belong the GROUP_NUMBER=0!!!!
      select 'dd if=/dev/zero of=''' ||PATH||''' bs=8192 count=100' from v$asm_disk
      where GROUP_NUMBER=0;


      select * from v$asm_operation;

      ---------------------------------------------------------------------------------------

      alter diskgroup FRA1 drop disk 'FRA1_0002';
      alter diskgroup FRA1 add disk '/dev/vx/rdsk/fra1dg/fra3';

      alter diskgroup FRA1 drop disk 'FRA1_0003';
      alter diskgroup FRA1 add disk '/dev/vx/rdsk/fra1dg/fra4';
      --When a new diskgroup is created, it is only mounted on the local instance,
      --and only the instance-specific entry for the asm_diskgroups parameter is updated.
      --By manually mounting the diskgroup on other instances, the asm_diskgroups parameter
      --on those instances are updated.

      --on +ASM1 :
      create diskgroup FRA1 external redundancy disk '/dev/vx/rdsk/fradg/fra1'
      ATTRIBUTE 'compatible.rdbms' = '11.1', 'compatible.asm' = '11.1';

      --on +ASM2 :
      alter diskgroup FRA1 mount;


      --It works even for on going balances!!!
      alter diskgroup DATA1 rebalance power 10;
      ################################################################
      # New ASM Command Line Utility (ASMCMD) Commands and Options################################################################

      ASMCMD Command Reference:

      Command Description
      --------------------
      - cd Command Changes the current directory to the specified directory.
      - cp Command Enables you to copy files between ASM disk groups on a local instance and remote instances.
      - du Command Displays the total disk space occupied by ASM files in the specified
      - ASM directory and all of its subdirectories, recursively.
      - exit Command Exits ASMCMD.
      - find Command Lists the paths of all occurrences of the specified name (with wildcards) under the specified directory.
      - help Command Displays the syntax and description of ASMCMD commands.
      - ls Command Lists the contents of an ASM directory, the attributes of the specified
      - file, or the names and attributes of all disk groups.
      - lsct Command Lists information about current ASM clients.
      - lsdg Command Lists all disk groups and their attributes.
      - lsdsk Command Lists disks visible to ASM.
      - md_backup Command Creates a backup of all of the mounted disk groups.
      - md_restore Command Restores disk groups from a backup.
      - mkalias Command Creates an alias for system-generated filenames.
      - mkdir Command Creates ASM directories.
      - pwd Command Displays the path of the current ASM directory.
      - remap Command Repairs a range of physical blocks on a disk.
      - rm Command Deletes the specified ASM files or directories.
      - rmalias Command Deletes the specified alias, retaining the file that the alias points to.

      --------
      -- kfed tool From Unix Prompt for reading ASM disk header.
      kfed read /dev/vx/rdsk/fra1dg/fra1


      ################################################################
      # CREATE and Manage Tablespaces and Datafiles on ASM################################################################

      CREATE TABLESPACE my_ts DATAFILE '+disk_group_1' SIZE 100M AUTOEXTEND ON;

      ALTER TABLESPACE sysaux ADD DATAFILE '+disk_group_1' SIZE 100M;

      ALTER DATABASE DATAFILE '+DATA1/dbname/datafile/audit.259.668957419' RESIZE 150M;


      -------------------------
      create diskgroup DATA1 external redundancy disk '/dev/vx/rdsk/oraASMdg/fra1'
      ATTRIBUTE 'compatible.rdbms' = '11.1', 'compatible.asm' = '11.1';


      select 'alter diskgroup DATA1 add disk ''' || PATH || ''';' from v$asm_disk
      where GROUP_NUMBER=0 and rownum<=&Num_Disks_to_add;

      select 'alter diskgroup FRA1 add disk ''' || PATH || ''';' from v$asm_disk
      where GROUP_NUMBER=0 and rownum<=&Num_Disks_to_add;


      --Remove ASM header
      select 'dd if=/dev/zero of=''' ||PATH||''' bs=8192 count=100' from v$asm_disk
      where GROUP_NUMBER=0;







      SET LINESIZE  200
      SET PAGESIZE  9999
      SET TRIMSPOOL on
      SET VERIFY    off

      COLUMN disk_group_name        FORMAT a10           HEAD 'Disk Group Name'
      COLUMN disk_file_path         FORMAT a30           HEAD 'Path'
      COLUMN disk_file_name         FORMAT a10           HEAD 'File Name'
      COLUMN disk_file_fail_group   FORMAT a10           HEAD 'Fail Group'
      COLUMN total_mb               FORMAT 999,999,999   HEAD 'File Size (MB)'
      COLUMN used_mb                FORMAT 999,999,999   HEAD 'Used Size (MB)'
      COLUMN pct_used               FORMAT 999.99        HEAD 'Pct. Used'

      break on report on disk_group_name skip 1
      compute sum label ""              of total_mb used_mb on disk_group_name
      compute sum label "Total: "       of total_mb used_mb on report

      SELECT
        a.group_number,
        a.name,
        a.state
      FROM
          v$asm_diskgroup a
      ORDER BY
          a.group_number
      ;

      SELECT
        b.group_number                                   disk_group_number,
        b.path                                           disk_file_path,
        b.name                                           disk_file_name,
        b.failgroup                                      disk_file_fail_group,
        b.total_mb                                       total_mb,
        (b.total_mb - b.free_mb)                         used_mb,
        decode(b.total_mb,0,0,(ROUND((1- (b.free_mb / b.total_mb))*100, 2)))      pct_used
      FROM
          v$asm_disk b
      WHERE b.header_status = 'MEMBER'
      ORDER BY
        b.group_number
      ;

      Grid Control Repository: Steps to Migrate the 10g Repository from One Machine to Another on the Same Platform

      Grid Control Repository: Steps to Migrate the 10g Repository from One Machine to Another on the Same Platform [Oracle Doc ID 1068841.1]

      Modified 14-APR-2011     Type HOWTO     Status PUBLISHED
      In this Document
        Goal
        Solution
           Pre-Migration EM-Specific Steps
           Repository Database Migration
           Post-Migration EM-Specific Steps
        References




      Applies to:

      Enterprise Manager Grid Control - Version: 10.2.0.1 to 10.2.0.5 - Release: 10.2 to 10.2
      Information in this document applies to any platform.

      Goal

      This note describes the steps to be followed for migrating the Grid Control 10G Repository Database from one machine to another machine, both having the same Operating System and version.

      The scenario outlined assumes the following:

      - The source and destination Operating System (platform) and version are the same.
      - Valid Operating System level backups are taken for the , and the (on the OMS machine).
      - Adequate down-time is scheduled for the Grid Control setup during the migration.
      - The following steps have been pre-attempted on a Test setup before being tried with a Production Grid Control setup.

      • For Cross-platform migration of the 10g Repository, refer to:
        Note 388090.1: Grid Control Repository: Steps for Cross Platform Migration of the Grid Control Repository/OMS
      • For the 11g Grid Control repository migration, refer to
        Note 1302281.1: 11g Grid Control: Steps for Migrating the 11g Grid Control Repository from One Database to Another.


      Solution

      Pre-Migration EM-Specific Steps

      • Stop the HTTP Server component of the OMS, to stop the Agents from uploading any further xml files during the Repository Database migration:
        cd /opmn/bin
        opmnctl stopproc process-type=HTTP_Server

        If there are multiple OMS, then the above step should be performed on all OMS machines.

        Wait for sometime and check whether there are any pending xml files in the /sysman/recv directory or the value of the ReceiveDir parameter (if this has been manually modified from the default value) in the <>/sysman/config/emoms.properties file
        Wait for all the xml files to be cleared up. A small number of xml backlog is acceptable but having large number of files may cause performance issues when the OMS is re-started after the repository database migration (pending backlog before migration + backlog created during migration).

        After this, stop the OMS:

        cd /opmn/bin
        opmnctl stopall

      • Login to the repository database and run the following SQL statements:

        SQL> sqlplus "/as sysdba"
        SQL> show parameter job_queue_processes
        SQL> alter system set job_queue_processes=0;
                 Make a note of the value that was set for the job_queue_processes initialization parameter.
                 Verify from the database alert.log file that the database is running fine without any major issues.
      • Login to the database as the sysman user and execute:

        SQL> exec emd_maintenance.partition_maintenance; (This will drop the old partitions);
        SQL> exec emd_maintenance.remove_em_dbms_jobs;


      Repository Database Migration

      • Install the RDBMS software on destination machine. Ensure sure that the RDBMS software version and one-off patches applied are same on both the destination and source machines. Also, install a grid Agent installed on this machine if not present already.

      • Clone / Migrate the repository database to the new machine using any of database cloning methods (using RMAN, DBCA etc).

        References:
        Note 259694.1: Oracle10G RMAN Database Duplication
        Note 388431.1: Creating a Duplicate Database on a New Host.

        Note 733530.1: Grid Control Repository: How to Move Grid Control Repository from One Machine to Another Using DBCA (on the same Platform)
        Note 954615.1: Grid Control Repository: How to use RMAN to Move and Upgrade the Repository Database in a RAC Cluster Configuration?
        Note 602955.1: Grid Control Repository: Steps To Move the Repository Using an Inconsistent (Hot) Database Backup

      • If DBCA is used for the new database, then all the schema users will be locked and their password expired. So, the accounts need to be unlocked and their password reset manually.
        Execute the following for EM-specific users :

        SQL> sqlplus "/as sysdba"
        SQL> alter user SYSMAN identified by account unlock ;
        SQL> alter user DBSNMP identified by account unlock ;
        SQL> alter user MGMT_VIEW account unlock ;

        Note:
        - The password for the sysman and dbsnmp users should be same as the password in the original repository database.
        - For list of other users run "select username from dba_users;"
        - Unlock and re-set the passwords for the other users using similar 'alter user' command as the above.

      Post-Migration EM-Specific Steps

      • Login to new Repository database as sysman and execute:

        SQL> exec emd_maintenance.analyze_emd_schema('SYSMAN');

      • Login to repository database as sysdba, restore the original value of job_queue_processes initialization parameter and submit the EM jobs:
        SQL> alter system set job_queue_processes=10 scope=both;
        SQL> connect sysman/
        SQL> exec emd_maintenance.submit_em_dbms_jobs;

      • On the OMS machine:
        - Take a backup of the /sysman/config/emoms.properties
        - In the emoms.properties file, look for the parameter oracle.sysman.eml.mntr.emdRepConnectDescriptor.
        - Update the details: host name / listener port / Repository SID in this parameter value with the connection details of the new repository. 
        - The above steps should be performed on all the OMS connecting to the repository database.

      • Startup OMS

        cd /opmn/bin
        opmnctl startall

      • Connect to the Grid Console from a browser. Navigate to setup -> Management Services and Repository -> Monitoring Configuration. Update the connection details for the "Management Services and repository" target to point the new Repository.

      • From a machine, where emcli is configured, relocate the repository database target from the old machine to the new database machine:

        emcli relocate_targets -src_agent=""
        -dest_agent="" -target_name="" -target_type=oracle_database -copy_from_src -force

        Reference:
        Note 577443.1: How to Setup and Configure Target Relocate using EMCLI 

      References

      NOTE:781314.1 - Grid Control Repository: Steps to Rename the Repository Database Name using NID Utility
      NOTE:1302281.1 - 11g Grid Control: Steps for Migrating the 11g Grid Control Repository from One Database to Another
      NOTE:388090.1 - Grid Control Repository: Steps for Cross Platform Migration of the Grid Control Repository

      Creating a Duplicate Database on a New Host

      Creating a Duplicate Database on a New Host. [Oracle Document ID 388431.1]

        Modified 01-FEB-2011     Type HOWTO     Status PUBLISHED  
      In this Document
        Goal
        Solution
           Duplicate database from server A to server B (Non ASM)
           1. Backup of the primary database.
           2. Determine how much disk space will be required.
           3. Ensuring you have enough space on your target server.
           4. Making the backup available for the duplicate process.
           5. Creating the init.ora & administration directories for the duplicate database.
           6. Ensuring SQL*NET connections to primary database and RMAN catalog are working.
           7. Prepare RMAN duplicate script.
           8. Execute the RMAN script.
        References





      Applies to:

      Oracle Server - Enterprise Edition - Version: 9.0.1.4 to 11.2.0.0 - Release: 9.0.1 to 11.2
      Information in this document applies to any platform.
      ***Checked for relevance on 06-Aug-2009***
      ***Checked for relevance on 01-Feb-2011***

      Goal

      The following note will guide a user in creating a duplicate database on a new host (non ASM).  The scenario refers to a recovery catalog however a catalog is not mandatory. The following example has been completed on a linux operating system.

      The note is applicable for 9i, 10g and 11g. However 11g has new features related to the DUPLICATE.
      See for more details on the 11g DUPLICATE new features : Note 452868.1 RMAN 'Duplicate Database' Feature in 11G

      Solution

      Duplicate database from server A to server B (Non ASM)


      Assumed database names:
      Primary Database SID:       PROD
      Duplicate Database SID:    AUX
      RMAN Catalog SID:          RMAN


      ====================================================================================
      Steps

      1.  Backup the primary database.

      2.  Determine how much disk space will be required.

      3.  Ensuring you have enough space on your target server.

      4.  Making the backup available for the duplicate process.

      5.  Creating the init.ora & administration directories for the duplicate database.

      6.  Ensuring SQL*NET connections to primary database and RMAN catalog are working.

      7.  Prepare RMAN duplicate script.

      8.  Execute the RMAN script.

      =====================================================================================

      1. Backup of the primary database.


      Host A (Target)

      Prior to creating a duplicate database you must have a backup of the target database. In this example we assume the backup media is disk. Find appropriate disk space on host A and take a full backup including archive logs and current controlfile (Figure 1a). If you are planning to duplicate a subset of the database refer to (Figure 1b) this illustrates the RMAN command to backing up certain tablespaces.

      [oracle@linux] export ORACLE_SID=PROD

      [oracle@linux] rman target=/ catalog=rman/rman@RMAN

      RMAN> run {
              allocate channel d1 type disk;
              backup format '/backups/PROD/df_t%t_s%s_p%p' database;
              sql 'alter system archive log current';
              backup format '/backups/PROD/al_t%t_s%s_p%p' archivelog all;
              release channel d1;
            }

             Figure 1a - This command will perform a full database backup including archivelogs and the current controlfile.


      [oracle@linux] export ORACLE_SID=PROD

      [oracle@linux] rman target=/ catalog=rman/rman@RMAN

      RMAN> run {
              allocate channel d1 type disk;
              backup format '/backups/PROD/df_t%t_s%s_p%p'
                     tablespace SYSTEM, SYSAUX, UNDO, USERS;
              sql 'alter system archive log current';
              backup format '/backups/PROD/al_t%t_s%s_p%p' archivelog all;
              release channel d1;
           }
          Figure 1b- This command will perform a tablespace backup ( SYSTEM, SYSAUX, UNDO & USERS) including archive logs and the current controlfile.

      2. Determine how much disk space will be required.


      Host A(Target) -

      After deciding what you will be duplicating, calculate the entire space this will require on the new host. The full database calculation (Figure 2a) will calculate the entire space required whereas (figure 2b) allows you to enter the tablespace names in order to calculate the space required. Figure 2c provides a sample output.

      Note: sql valid for version 10g only.
      SQL> select DF.TOTAL/1048576 "DataFile Size Mb",
                  LOG.TOTAL/1048576 "Redo Log Size Mb",
                  CONTROL.TOTAL/1048576 "Control File Size Mb",
                  (DF.TOTAL + LOG.TOTAL + CONTROL.TOTAL)/1048576 "Total Size Mb"
           from dual,
          (select sum(a.bytes) TOTAL from dba_data_files a) DF,
          (select sum(b.bytes) TOTAL from v$log b) LOG,
          (select sum((cffsz+1)*cfbsz) TOTAL from x$kcccf c) CONTROL;
           Figure 2a - Calculate total space for all datafiles within database.

      SQL> select DF.TOTAL/1048576 "DataFile Size Mb",
                  LOG.TOTAL/1048576 "Redo Log Size Mb",
                  CONTROL.TOTAL/1048576 "Control File Size Mb",
                  (DF.TOTAL + LOG.TOTAL + CONTROL.TOTAL)/1048576 "Total Size Mb" from dual,
          (select sum(a.bytes) TOTAL from dba_data_files a
           where tablespace_name in('SYSTEM','UNDOTBS1', 'SYSAUX', 'USERS')) DF,
          (select sum(b.bytes) TOTAL from v$log b) LOG,
          (select sum((cffsz+1)*cfbsz) TOTAL from x$kcccf c) CONTROL;
           Figure 2b - Calculate space for list of tablespaces within primary database.

      DataFile Size Mb Redo Log Size Mb Control File Size Mb Total Size Mb
      ---------------- ---------------- -------------------- -------------
                   900              150             20.34375    1070.34375
           Figure 2c - Sample output of space calculation.

      3. Ensuring you have enough space on your target server.


      Host B (Aux)

      Prior to starting the duplicate process you must ensure you have enough disk space within file system to hold the database.

      [oracle@linux] df -kh
      Filesystem         Size     Used     Avail     Use%     Mounted on
      /dev/vg01/root     9.9G     2.8G      6.6G      30%              /
      /dev/sda1          145M      16M      122M      12%          /boot
      none               999M        0      999M       0%       /dev/shm
      /dev/vg01/tmp      2.0G     383M      1.5G      20%           /tmp
      /dev/vg01/u01       20G      12G      7.0G      62%           /u01
      /dev/vg01/u02      4.9G    1010M      3.6G      22%           /u02
      /dev/vg01/backups  5.9G     1.2G      4.4G      22%       /backups
      /dev/vg01/oradata   15G      13G      2.0G      87%       /oradata
           Figure 3a - Sample output showing the space available on your filesystem.

      Compare the results received from this query with the output from 2a or 2b and ensure you have enough diskspace for your duplicate database.


      4. Making the backup available for the duplicate process.


      If your backup resides on disk you will need to copy this back up from host A to host B. Ensure you place it in the same directory as where it was created. In the example below (figure 4a) the backup piece resides in ‘/backups/PROD’ these files need to be copied into the same directory on host B.

      Furthermore also ensure than all archive log backups have also been moved and any archive logs in the archive log directory that may be require for the recovery. The archive logs required will depend on the point in time of the recovery.

      RMAN> list backup;

      BS Key Type LV Size Device Type Elapsed Time Completion Time
      ------- ---- -- ---------- ----------- ------------ -------------------
      22 Full 529M DISK 00:00:51 2006/05/16 11:12:54
      BP Key: 22 Status: AVAILABLE Compressed: NO Tag: TAG20060516T111203
      Piece Name: /backups/PROD/df_t590584323_s23_p1
      List of Datafiles in backup set 22
      File LV Type Ckp SCN Ckp Time Name
      ---- -- ---- ---------- ------------------- ----
      1 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/system01.dbf
      2 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/undotbs01.dbf
      3 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/sysaux01.dbf
      4 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/users01.dbf


      BS Key Size Device Type Elapsed Time Completion Time
      ------- ---------- ----------- ------------ -------------------
      24 48M DISK 00:00:06 2006/05/16 11:13:07
      BP Key: 24 Status: AVAILABLE Compressed: NO Tag: TAG20060516T111301
      Piece Name: /backups/PROD/al_t590584381_s25_p1

      List of Archived Logs in backup set 24
      Thrd Seq Low SCN Low Time Next SCN Next Time
      ---- ------- ---------- ------------------- ---------- ---------
      1 78 1344750 2006/05/15 04:55:34 1353208 2006/05/15 10:00:19
      1 79 1353208 2006/05/15 10:00:19 1353213 2006/05/15 10:00:20
      1 80 1353213 2006/05/15 10:00:20 1371564 2006/05/15 22:00:11
      1 81 1371564 2006/05/15 22:00:11 1373291 2006/05/15 22:00:59
      1 82 1373291 2006/05/15 22:00:59 1381066 2006/05/16 03:00:05
      1 83 1381066 2006/05/16 03:00:05 1390685 2006/05/16 09:03:00
      1 84 1390685 2006/05/16 09:03:00 1393870 2006/05/16 11:13:00
      1 85 1393870 2006/05/16 11:13:00 1393929 2006/05/16 11:13:00
           Figure 4a - A list backup showing the backup pieces that need to be copied across Host B

      5. Creating the init.ora & administration directories for the duplicate database.


      Host B(AUX)


      Create the pfile [initAUX.ora] parameter file in the $ORACLE_HOME/dbs directory for the auxiliary database. An example is shown below (figure 5a) with bare minimum settings. Note you may require to set parameters which are the same as your production database (refer to you primary init.ora)

      # +----------------------------------------+
      # | FILE : initAUX.ora                     |
      # | DATABASE NAME : AUX                    |
      # +----------------------------------------+
      # Set the below to location of the clone Duplicate database / name of
      # clone database.

      audit_file_dest =/oradata/AUX/adump
      background_dump_dest =/oradata/AUX/bdump
      core_dump_dest =/oradata/AUX/cdump
      user_dump_dest =/oradata/AUX/udump
      db_name ="AUX"
      instance_name =AUX

      cluster_database=false (in case the production is a rac environment)


      # Set the below to the location of the duplicate clone control file.

      control_files =('/oradata/AUX/control01.ctl','/oradata/AUX/control02.ctl','/oradata/AUX/control03.ctl')



      # Set the below for the from and to location for all data files / redo
      # logs to be cloned.

      db_file_name_convert =("/oradata/PROD/", "/oradata/AUX/")
      log_file_name_convert =("/oradata/PROD/", "/oradata/AUX/")
      #Set the below to the same as the production target

      undo_management =AUTO
      undo_retention =10800
      undo_tablespace =UNDOTBS1
      db_block_size = 8192
      compatible = 10.2.0.1.0
           Figure 5a - Sample initAUX.ora with minimal settings

      Following the creation of the initAUX.ora startup nomount the auxiliary instance.

      [oracle@linux]export ORACLE_SID=AUX

      [oracle@linux] sqlplus '/as sysdba'

      SQLPLUS> startup nomount;
      Figure 5b - startup nomount the AUX instance.


      6. Ensuring SQL*NET connections to primary database and RMAN catalog are working.


      Host B(AUX)

      Ensure the production target database is open or at least mounted. If using a catalog database this must also be open. Test your SQL*NET connections: From Host B you must be able to connect to the primary DB as sysdba and make an rman connection to RMAN. Perform a basic test using sqlplus. When you have successfully connected exit sqlplus and move onto the next step.

      [oracle@linux]% sqlplus ‘sys/oracle@PROD as sysdba’

      [oracle@linux]% sqlplus rman/rman@RMAN   (not mandatory)
           Figure 6a - SQL*NET connections

      7. Prepare RMAN duplicate script.


      In a working directory on Host B create an RMAN script file duplicate.rcv. The example below (figure 7a) shows the command for a complete duplicate (figure 7b) skips the tablespaces which are not required in the duplicate and (figure 7b) provide the syntax required for a point in time duplicate.

      run {
        allocate auxiliary channel C1 device type disk;
        duplicate target database to AUX;
      }
           Figure 7a - Sample duplicate command.

      run {
        allocate auxiliary channel C1 device type disk;
        duplicate target database to AUX skip tablespace ABC, XYZ;
      }
           Figure 7b- Sample duplicate script omitting optional tablespaces;

      run {
        set until time "to_date('Jan 01 2000 12:00:00','Mon DD YYYY HH24:MI:SS')";
        allocate auxiliary channel C1 device type disk;
        duplicate target database to AUX;
      }
            Figure 7c- Sample duplicate script to a point in time.

      8. Execute the RMAN script.


      Start RMAN, connect to the production target, the catalog instance and also the auxiliary clone. Run the RMAN duplicate script as shown below. Before doing this ensure that the Oracle SID environment variable is set to the duplicate clone database.

      [oracle@linux] export ORACLE_SID=AUX

      [oracle@linux] rman target sys/sys@PROD catalog rman/rman@rman auxiliary /

      RMAN> @duplicate.rcv
          Figure 8a - Execute the RMAN duplicate script.

      References

      NOTE:452868.1 - RMAN 'Duplicate Database' Feature in 11G

      Thursday 9 June 2011

      Instantiation by the oracle Data Pump for both Golden Gate Replication and Streams

      This is one of my favourite method of instantiation as most of the time I worked on the environment where database is quite big in size and replicated database was created from the backup of primary database.

      As database backup and restore can take hours and during that time it might not be feasible to leave capture or extract process running. In this case, Oracle data pump can be used to export tables or schema(s) from the source database. The falshback_scn export parameter is specified in the expdb command line. The current SCN of the source database is specified as the flashback_scn. Key point is to find the SCN number from the point when backup ends and use it to export data from that point onwards.

      This information can also be found through RMAN while doing backup or last SCN applied at target database before resetting the log files i.e. SCN number.



      Wednesday 4 May 2011

      Monitor Table stats in Oracle

      This script helps to find out how old the table stats are in oracle database. This will also return value 0 if stats are less than 24 hrs old, 1 if stats are between 24 to 48 hrs and 2 if they are older than 48 hrs.
      0 = OK
      1= Warning
      2= Critical

      Select CASE WHEN Time < 24 then 0 when Time between 24 and 48 then 1 else 2 end as return_code,' '||
      'Table Name is : ' ||table_name||
      'Object Type is : ' ||object_type||
      'Time in Hours : ' || Time
      as return_statement
      from
      (Select table_name,object_type,floor(((sysdate - last_analyzed)*24*50*50)/3500) Time
      from all_tab_statistics
      where owner='SCOTT')
      UNION ALL
      SELECT 0 as return_code ,'There is no row returned' as return_statement from dual

      This script is can be easily modified to monitor stats of whole database or limit to specific objects.




      How to monitor Long Running Sql statements

      This script is to find out long running sql statement in database. It is also returning values 0,1 and 2 to use with Nagios monitoring tools to show normal,warning, critical alerts.


      set long 5000
      set pages 1000
      select  case when max_sql_time is null then 0
          when max_sql_time < 1800 then 0
          when max_sql_time between 1800 and 3600 then 1
          else 2 end
          as return_code, 'OpName is : ' || opname||
          '. The max long running SQL in v$session_longops ran for '|| max_sql_time ||
          ' seconds. Message is ' || message ||
          '. Username is ' || l.username ||
          '. OSuser is ' || sess.osuser||   
          '. Program is '|| sess.program ||
          '. Machine is ' || sess.machine ||
          '. SQL Text is ' || s.sql_fulltext
          as return_statement
          from (select max(elapsed_seconds)
                as max_sql_time,
                username,
                opname,
                message,
                sql_address
                from v$session_longops
                where
                 last_update_time > sysdate - interval '15' minute
                and message not like 'RMAN: aggregate input: backup%'
                and message not like 'RMAN: aggregate output: backup%'
                and message not like 'RMAN: incremental datafile backup%'
               group by username, opname, message,sql_address) l,v$sql s,v$session sess
              where l.sql_address=s.address
               AND  sess.sql_address=s.address
             UNION ALL
      SELECT 0 as return_code ,to_clob('There are no rows returned') as return_statement   from dual

      Tuesday 19 April 2011

      Redo logs switching frequency

      To monitor redo switching below script will generate a report for last two weeks activity of redo switching. It is quite helpful



      set linesize 300
      set pagesize 5000
      set trimspool on
      column 00 format 99 heading "00:00"
      column 01 format 99 heading "1am"
      column 02 format 99 heading "2am"
      column 03 format 99 heading "3am"
      column 04 format 99 heading "4am"
      column 05 format 99 heading "5am"
      column 06 format 99 heading "6am"
      column 07 format 99 heading "7am"
      column 08 format 99 heading "8am"
      column 09 format 99 heading "9am"
      column 10 format 99 heading "10am"
      column 11 format 99 heading "11am"
      column 12 format 99 heading "12:00"
      column 13 format 99 heading "1pm"
      column 14 format 99 heading "2pm"
      column 15 format 99 heading "3pm"
      column 16 format 99 heading "4pm"
      column 17 format 99 heading "5pm"
      column 18 format 99 heading "6pm"
      column 19 format 99 heading "7pm"
      column 20 format 99 heading "8pm"
      column 21 format 99 heading "9pm"
      column 22 format 99 heading "10pm"
      column 23 format 99 heading "11pm"
      column 24 format 99 heading "12pm"
      column "Day" format a3

      prompt
      prompt Redo Log Switches
      prompt

      SELECT trunc (first_time) "Date",
      to_char (trunc (first_time),'Dy') "Day",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 0, 1)) "00",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 1, 1)) "01",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 2, 1)) "02",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 3, 1)) "03",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 4, 1)) "04",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 5, 1)) "05",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 6, 1)) "06",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 7, 1)) "07",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 8, 1)) "08",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 9, 1)) "09",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 10, 1)) "10",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 11, 1)) "11",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 12, 1)) "12",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 13, 1)) "13",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 14, 1)) "14",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 15, 1)) "15",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 16, 1)) "16",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 17, 1)) "17",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 18, 1)) "18",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 19, 1)) "19",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 20, 1)) "20",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 21, 1)) "21",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 22, 1)) "22",
      sum (decode (to_number (to_char (FIRST_TIME, 'HH24')), 23, 1)) "23"
      from v$log_history
      where trunc (first_time) >= (trunc(sysdate) - 14) -- last X days. 0 = today only. 1 = today and yesterday
      group by trunc (first_time)
      order by trunc (first_time) DESC
      /





      Second script is to find out average log switch per minute

      COLUMN avg_log_switch_min format 9,999.99 heading "Average
      Log
      Switch
      Times
      (Min)"

      SET lines 132 pages 60 feedback off verify off echo off
      TTITLE 'Average Log Switch Time Report'
      PROMPT 'Note that high rates of Redo Log switching can cause performance problems'
      PROMPT 'If this report indicates average switch times of less than 10 minutes'
      PROMPT 'you should consider increasing the size of your redo logs.'
      WITH redo_log_switch_times AS
      (SELECT sequence#, first_time,
      LAG (first_time, 1) OVER (ORDER BY first_time) AS LAG,
      first_time
      - LAG (first_time, 1) OVER (ORDER BY first_time) lag_time,
      1440
      * (first_time - LAG (first_time, 1) OVER (ORDER BY first_time)
      ) lag_time_pct_mins
      FROM v$log_history
      ORDER BY sequence#)
      SELECT AVG (lag_time_pct_mins) avg_log_switch_min
      FROM redo_log_switch_times;


      Third script is to monitor Redo latch statistics

      COLUMN name FORMAT a30 HEADING Name
      COLUMN percent FORMAT 999.999 HEADING Percent
      COLUMN total HEADING Total
      rem
      SELECT
      l2.name,
      immediate_gets+gets Total,
      immediate_gets "Immediates",
      misses+immediate_misses "Total Misses",
      DECODE (100.*(GREATEST(misses+immediate_misses,1)/
      GREATEST(immediate_gets+gets,1)),100,0) Percent
      FROM
      v$latch l1,
      v$latchname l2
      WHERE
      l2.name like '%redo%'
      and l1.latch#=l2.latch# ;

      In my opinion by using the combination of these scripts you should be able to identify problem with redo logs.

      Deleting Duplicate Rows

      It is not very common to delete duplicate rows but when you do then it has to be quick.

      Here is an example of finding duplicate rows and then script to delete them.

      To find the Duplicate rows

      select article,art_version,medasset,med_version,count(*) from articleasses
      group by article,art_version,medasset,med_version
       having count(*) >1

      Delete Duplicate Rows

      Once you are happy with the results of above query and backup the table then use this procedure to delete duplicate rows.

      declare
      v_rowid     varchar2(100);
      i_count     number(3);
      cursor c_dup
      is
      select medasset,med_version ,count(*) from medasset
      group by medasset,med_version
      having count(*) >1;
      begin
          i_count := 0;
          i_count := 0;
          for r_dup in c_dup loop
              select max(rowid)
              into v_rowid
              from medasset
              where medasset = r_dup.medasset
       and med_version = r_dup.med_version;
              delete from medasset
              where medasset = r_dup.medasset
        and med_version=r_dup.med_version
              and rowid = v_rowid;
              i_count := i_count + 1;
              dbms_output.put_line('Record = '||i_count);
          end loop;
       end;
      /

      Check the results again.

      Auditing and Restricting User Connections



      This tip was published on Oracle website in 2003. It looks like it is archived now by oracle but some other website has copied it to their own websites. I have now copied it to my blog.

      Tip of the Week
      Tip for Week of June 14, 2004



      This tip comes from Ajwat Ul-Hasan, Database Administrator/System Analyst, The Carphonewarehouse in London, United Kingdom.



      In Oracle9i, this script can restrict users logging into the database. In one table you provide information about which users are allowed to log into the database. You can modify this script to allow or block a user with respect to application, ip address, username, osuser, terminal, etc. It is a very easy and efficient way of implementing security in database.







      create table valid_users (

      valid_ora VARCHAR2(30),

      valid_os varchar2(30)

      );

      and another table

      CREATE TABLE USER_AUDIT

      (

      SID NUMBER NULL,

      SERIAL# NUMBER NULL,

      TIMESTAMP DATE NULL,

      USERNAME VARCHAR2(30) NULL,

      OSUSERID VARCHAR2(30) NULL,

      MACHINENAME VARCHAR2(64) NULL,

      PROGRAM VARCHAR2(48) NULL

      )





      And then create this trigger:







      CREATE OR REPLACE TRIGGER test_audit

      AFTER LOGON ON database

      DECLARE

      machinename VARCHAR2(64);

      osuserid VARCHAR2(30);

      ora_username VARCHAR2(30) DEFAULT NULL;

      os_username VARCHAR2(30);

      v_sid NUMBER;

      v_serial NUMBER;

      v_program VARCHAR2(48);

      v_numuser NUMBER;

      CURSOR c1 IS

      SELECT sid, serial#, osuser, machine, program

      FROM v$session

      WHERE audsid = userenv('sessionid');

      BEGIN

      OPEN c1;

      FETCH c1 INTO v_sid, v_serial, osuserid, machinename,

      v_program;

      CLOSE c1;

      SELECT count(*)

      INTO v_numuser

      FROM valid_users

      WHERE valid_ora = user;



      IF v_numuser > 0 then

      INSERT INTO user_audit

      VALUES (v_sid, v_serial, sysdate,

      user, osuserid, machinename,v_program);

      ELSE

      INSERT INTO user_audit

      VALUES (v_sid, v_serial, ! sysdate,

      user, osuserid, machinename,'IN ELSE');

      raise_application_error(-20001,'You are not allowed to connect

      to the database');

      END IF;

      END;

      /





      In the valid_user table you have to insert information about valid users, otherwise no user can log in to the database. You can generate information about user connection by using information in user_audit table.

      Monday 18 April 2011

      UNDO TABLESPACE MANAGEMENT

      This document intends to provide information about UNDO tablespace that is not commonly available. I have not included syntax of tablespace creation, dropping etc commands as this can be easily available in books, manuals and websites.


      WHAT IS UNDO?

      Word UNDO means to reverse or erase the change. In oracle world, UNDO allows to reverse the transaction so that database looks like before the transaction started. Also it provides read consistent image of data i.e. if during a transaction if one session is changing a data where as at the same time other session wants to read that data then UNDO will provide the state of data before the transaction starts.


      USE OF UNDO

      There is few main usage of UNDO in oracle database.

      * UNDO is used to reverse the uncommitted transaction on the issue of ROLLBACK command.
      * It is used to provide read consistent image of a record
      * It is used to during database recovery to apply any uncommitted transaction in redo logs to datafiles.
      * Flashback Query also uses UNDO to get the image of data back in time.

      UNDO MANAGEMENT

      Oracle needs UNDO tablespace to create undo segments. If undo tablespace is not created then it will use SYSTEM tablespace as UNDO and not recommended. To create UNDO tablespace you have to use UNDO keyword in create tablespace command. Most common configuration of UNDO_MANAGEMENT is AUTO (Default is MANUAL in 8i & 9i).

      One can have multiple UNDO tablespace in a database but only one can be active at any given time.

      In AUTOMATIC undo management oracle create and manages undo itself and DBAs don’t have to worry about its management. Oracle attempts to assign one undo segment to each transaction. When it cannot, it creates and assigns additional undo segments. Multiple transactions can write to one undo segment if space in undo tablespace is depleted which can cause contention. In order to avoid this situation add more space in undo tablespace. Later this article will provide scripts to monitor fragmentation in UNDO tablespace and method to overcome it.

      From Oracle documentation

      The following is a summary of the initialization parameters for automatic undo management mode:

      Initialization Parameter Description
      UNDO_MANAGEMENT If AUTO, use automatic undo management mode. If MANUAL, use manual undo management mode. The default is MANUAL.


      UNDO_TABLESPACE An optional dynamic parameter specifying the name of an undo tablespace to use. This parameter should be used only when the database has multiple undo tablespaces and you want to direct the database instance to use a particular undo tablespace.

      UNDO_RETENTION A dynamic parameter specifying the minimum length of time to retain undo. The default is 900 seconds. The setting of this parameter should take into account any flashback requirements of the system.


      TYPES OF AUTOMATIC UNDO MANAGEMENT

      When oracle introduced UNDO management it was based on segment management. Segment management is expensive on CPU, memory and IO so oracle add a new feature in later version of oracle 10g to do undo management in memory and called it in-memory Undo (IMU).

      a) IN-MEMORY UNDO

      Main benefit of IMU is that as there is no block level change of UNDO segment involves here so IMU will not generate any redo entry. Hence IMU improve the Undo Header contention and undo block contention by doing UNDO management in memory called IMU Node. Rather than writing a change to undo buffer ,undo is written to IMU Node. On the other hand memory structure require latches to maintain serial execution so make sure that enough IMU latches are created by changing processes parameter. Oracle use ‘In memory Undo Latch’ to access IMU structures in shared pool. So if you have high waits on this latch then you can increase the number of latches by increasing processes parameter or switch in-memory undo by setting in_memory_undo to false.

      The following is a summary of the initialization parameters for automatic undo management mode:

      Initialization Parameter Description

      _in_memory_undo Default is TRUE and enabled. To disable it change parameter to FALSE.

      _imu_pools Default is 3 on some system. This sets the number of IMU pools. It is not related to memory allocation for IMU.

      _recursuve_imu_transactions This enables Oracle’s own SQL to use IMU. Default is FALSE.

      _db_writer_flush_imu Allows Oracle the freedom to artificially age a transaction for increased automatic cache management. Default is

      Initialization Parameter Description TRUE


      There is no parameter that allow us to change memory allocation for IMU Node but changing shared_pool_size parameter can help adjusting the IMU memory allocation.

      To findout how much memory is allocated run the command

      On  UAT database

      SQL> select * from v$sgastat where name like 'KTI-UNDO';


      POOL NAME BYTES
      ------------ -------------------------- ----------
      shared pool KTI-UNDO 8522272

      SQL> select
      2 n.name statistic,
      3 v.value statvalue,
      4 v.statistic# statnum
      5 from
      6 v$statname n,
      7 v$sysstat v
      8 where
      9 n.statistic# = v.statistic# and
      10 n.name like '%commit%'
      11 order by
      12 1 asc
      13 /

      Statistic Value Stat #
      ------------------------------------------------------- ---------------- ------
      IMU commits 987,170 267



      WARNING : Understand before taking action

      As mentioned earlier that huge wait on “in memory Undo Latch “ and high amount of cpu usage shows that you have contention latch contention with IMU. As they are very CPU hungry if waiting for latch so you would like to resolve this issue as early as possible.

      There are few reasons that you might have contention with IMU

      * Few latches than actually require
      * Not enough memory allocated for IMU Node. Check v$latch_children view.
      * Not enough CPU core available i.e. if your system is already suffering from high CPU utilisation.

      Few suggestions to try in these cases

      * Increase the number of processes parameter so that it you can increase number of latches. Be careful and understand the other impact of increasing processes parameter.

      * Increase the shared_pool_size. Again also understand the impact of increasing shared_poo_size on over database performance.

      * Increase the number of CPU core if possible.

      * Last resort is to disable the IMU and force oracle to use segment management by changing _in_memory_undo to FALSE.

      b) SEGMENT BASED UNDO

      Segment based Undo management is same as any table or index segment management and normally costly than IMU but on the other hand Oracle has more experience in handling segment based undo management than IMU. IMU is released in later version of oracle 10g.

      Oracle also provides a tool cal UNDO advisor which checks the information in AWR and advice and helps you on setting up undo environment. Sample example of Undo advisor from oracle documentation is mentioned below

      DECLARE
      tid NUMBER;
      tname VARCHAR2(30);
      oid NUMBER;
      BEGIN
      DBMS_ADVISOR.CREATE_TASK('Undo Advisor', tid, tname, 'Undo Advisor Task');
      DBMS_ADVISOR.CREATE_OBJECT(tname, 'UNDO_TBS', null, null, null, 'null', oid);
      DBMS_ADVISOR.SET_TASK_PARAMETER(tname, 'TARGET_OBJECTS', oid);
      DBMS_ADVISOR.SET_TASK_PARAMETER(tname, 'START_SNAPSHOT', 1);
      DBMS_ADVISOR.SET_TASK_PARAMETER(tname, 'END_SNAPSHOT', 2);
      DBMS_ADVISOR.execute_task(tname);
      end;
      /

      Where start_snapshot is 1 and end_snapshot is 2. This information is also available in DBA_ADVISOR_* data dictionary views.

      Oracle document further discusses the formula of calculating undo space requirement.

      UndoSpace = UR * UPS + overhead
      where:
      UndoSpace is the number of undo blocks
      UR is UNDO_RETENTION in seconds. This value should take into consideration long-running queries and any flashback requirements.
      UPS is undo blocks for each second
      overhead is the small overhead for metadata (transaction tables, bitmaps, and so forth)
      As an example, if UNDO_RETENTION is set to 2 hours, and the transaction rate (UPS) is 200 undo blocks for each second, with a 4K block size, the required undo space is computed as follows:

      (2 * 3600 * 200 * 4K) = 5.8GBs
      Such computation can be performed by using information in the V$UNDOSTAT view. In the steady state, you can query the view to obtain the transaction rate. The overhead figure can also be obtained from the view.

      DEFRAGMENT UNDO TABLESPACE

      I have already touched this topic in other white paper “ Free Space Fragmentation by Tablespace” but I feel that it is appropriate to mention it here again.

      As oracle create different size segments in UNDO tablespace normally and also extend and shrink according to the need of system, this cause fragmentation of UNDO tablespace. Oracle manages defragmentation of UNDO tablespace by SMON processes but if system is highly transactional then it is unlikely that SMON will have a chance to properly defragment UNDO tablespace. It is a common observation then whenever we receive error message of snapshot too old or run out of space from undo tablespace, DBAs add more space as resolution. I have done one test to defragment undo tablespace and happy to share the results.


      On one UAT database , UNDO tablespace was fragmented by measuring FSFI and size of tablespace was 60Gb. I created secondary undo tablespace and switch new undo tablespace (UNDO2) as default undo tablespace. This allow oracle (SMON) to clear down all unused segments down on the first undo tablespace (UNDO). Also I found that we are now using 20Gb of space for undo compare to 60Gb this is 2/3rd of space back to the system. Once original undo tablespace (UNDO) is clear down then you can switch it back to default undo tablespace and drop the other undo tablespace (UNDO2).

      To switch undo tablespace use
      ALTER SYSTEM SET UNDO_TABLESPACE = ‘UNDO2’;


      If you are using spfile then you can add spfile clause also. If you are using pfile then change it manually and bounce the database.

      To monitor tablespace fragmentation
      rem
      rem Name: free_space.sql
      rem
      rem FUNCTION: Provide data on tablespace extent status
      rem FUNCTION: Use 0 = OK , 1= Warning , 2= Critical
      rem FUNCTION: includes fsfi from DBA Handbook
      rem
      SET FEED OFF
      SET FLUSH OFF
      SET VERIFY OFF
      set pages 58 LINES 132

      COLUMN tablespace HEADING Name FORMAT a25
      COLUMN files HEADING '#Files' FORMAT 9,999
      COLUMN pieces HEADING 'Frag' FORMAT 9,999
      COLUMN free_bytes HEADING 'Free
      Byte' FORMAT 9,999,999,999,999
      COLUMN free_blocks HEADING 'Free
      Blk' FORMAT 999,999,999
      COLUMN largest_bytes HEADING 'Biggest
      Bytes' FORMAT 9,999,999,999,999
      COLUMN largest_blks HEADING 'Biggest
      Blks' FORMAT 999,999,999
      COLUMN ratio HEADING ' Percent' FORMAT 999.999
      COLUMN average_fsfi HEADING 'Average
      FSFI' FORMAT 999.999
      COLUMN alert_level HEADING 'Alert
      level' FORMAT 99
      SELECT
      tablespace,
      files,
      pieces,
      free_bytes,
      free_blocks,
      largest_bytes,
      largest_blks,
      ratio,
      average_fsfi,
      case when average_fsfi < 30 then 2
      when average_fsfi between 30 and 40 then 1
      else 0
      end alert_level
      FROM
      (SELECT
      tablespace,
      COUNT(*) files,
      SUM(pieces) pieces,
      SUM(free_bytes) free_bytes,
      SUM(free_blocks) free_blocks,
      SUM(largest_bytes) largest_bytes,
      SUM(largest_blks) largest_blks,
      SUM(largest_bytes)/sum(free_bytes)*100 ratio,
      SUM(fsfi)/COUNT(*) average_fsfi
      FROM
      (SELECT tablespace_name tablespace, file_id, COUNT(*) pieces,
      SUM(bytes) free_bytes, SUM(blocks) free_blocks,
      MAX(bytes) largest_bytes, MAX(blocks) largest_blks,
      SQRT(MAX(blocks)/SUM(blocks))*(100/SQRT(SQRT(COUNT(blocks)))) fsfi
      FROM sys.dba_free_space
      GROUP BY tablespace_name, file_id, relative_fno)
      GROUP BY tablespace)
      /

      CONCLUSION

      UNDO tablespace tuning is one of the areas commonly missed by DBAs as most of the time tuning is fire fighting activity. This document will help you to analyse your undo tablespace management and tuning scientifically on the basis of some analysis. There is so much more to add on this topic that maybe I’ll add in later version of this document.

      Key point is when you do tuning is to “Understand what you doing” and look at the bigger picture.