Tuesday, February 11, 2020

Exadata - RAC TFA Collector - TFA with Database Support Tools Bundle

TFA Collector - TFA with Database Support Tools Bundle

Oracle Trace File Analyzer (TFA) provides a number of diagnostic tools in a single bundle, making it easy to gather diagnostic information about the Oracle database and clusterware, which in turn helps with problem resolution when dealing with Oracle Support.

My Oracle Support note 1513912.1 "TFA Collector - Tool for Enhanced Diagnostic Gathering" at https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1513912.1

Trace File Analyzer (TFA) Collector simplifies diagnostic data collection on Oracle Cluster Ready Services (CRS), Oracle Grid Infrastructure (Oracle GI), and Oracle RAC systems. TFA behaves in a similar manner to the ion utility packaged with Oracle Clusterware. Both tools collect and package diagnostic data. However, TFA is much more powerful than ion because TFA centralizes and automates the collection of diagnostic information.

Please refer the below MOS document for more details:
TFA Collector - TFA with Database Support Tools Bundle (Doc ID 1513912.1)

Patch download, unzip and Install/upgrade:

HOST01:(root)-/opt
>mkdir oracle.tfa

HOST01:(root)-/opt
>ls -ltrh

HOST01:(root)-/opt
>cp /u01/patches/TFA/TFA-LINUX_v18.1.1.zip /opt/oracle.tfa/

HOST01:(root)-/opt
>cd /opt/oracle.tfa/

HOST01:(root)-/opt/oracle.tfa
>ls -ltrh
total 172M
-rw-r--r-- 1 root root 172M Apr  5 04:12 TFA-LINUX_v18.1.1.zip

HOST01:(root)-/opt/oracle.tfa
>unzip TFA-LINUX_v18.1.1.zip
Archive:  TFA-LINUX_v18.1.1.zip
  inflating: README.txt
  inflating: installTFA-LINUX
HOST01:(root)-/opt/oracle.tfa

TFA installation:

HOST01:(root)-/opt/oracle.tfa
>./installTFA-LINUX
TFA Installation Log will be written to File : /tmp/tfa_install_385107_2018_04_05-04_15_23.log

Starting TFA installation

TFA Version: 181100 Build Date: 201802010159

TFA HOME : /u01/app/12.2.0.1/grid/tfa/HOST01/tfa_home

Installed Build Version: 122120 Build Date: 201709270025

TFA is already installed. Patching /u01/app/12.2.0.1/grid/tfa/HOST01/tfa_home...
TFA patching CRS or DB from zipfile extracted to /tmp/.385107.tfa
TFA patching typical install from zipfile is written to /u01/app/12.2.0.1/grid/tfa/HOST01/tfapatch.log

TFA will be Patched on:
HOST01
HOST02

Do you want to continue with patching TFA? [Y|N] [Y]:

Checking for ssh equivalency in HOST02
HOST02 is configured for ssh user equivalency for root user

Using SSH to patch TFA to remote nodes :

Applying Patch on HOST02:

TFA_HOME: /u01/app/12.2.0.1/grid/tfa/HOST02/tfa_home
Stopping TFA Support Tools...
Shutting down TFA
oracle-tfa stop/waiting. . . . .
Killing TFA running with pid 101177. . .
Successfully shutdown TFA..
Copying files from HOST01 to HOST02...

Current version of Berkeley DB in  is 5 or higher, so no DbPreUpgrade required
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
Moving Properties.bkp to Properties
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
Running commands to fix init.tfa and tfactl in HOST02...
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
Updating init.tfa in HOST02...
Starting TFA in HOST02...
Starting TFA..
oracle-tfa start/running, process 23827
Waiting up to 100 seconds for TFA to be started... . . . .
Successfully started TFA Process... . . . .
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
TFA Started and listening for commands
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.

Enabling Access for Non-root Users on HOST02...

Applying Patch on HOST01:

Stopping TFA Support Tools...

Shutting down TFA for Patching...

Shutting down TFA
oracle-tfa stop/waiting. . . . .
Killing TFA running with pid 240432. . .
Successfully shutdown TFA..

No Berkeley DB upgrade required

Copying TFA Certificates...
Moving Properties.bkp to Properties

Running commands to fix init.tfa and tfactl in localhost

Starting TFA in HOST01...

Starting TFA..
oracle-tfa start/running, process 105565
Waiting up to 100 seconds for TFA to be started... . . . .
Successfully started TFA Process... . . . .
TFA Started and listening for commands

Enabling Access for Non-root Users on HOST01...

WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
.-------------------------------------------------------------------.
| Host        | TFA Version | TFA Build ID         | Upgrade Status |
+-------------+-------------+----------------------+----------------+
| HOST01 |  18.1.1.0.0 | 18110020180201015951 | UPGRADED       |
| HOST02 |  18.1.1.0.0 | 18110020180201015951 | UPGRADED       |
'-------------+-------------+----------------------+----------------'
HOST01:(root)-/opt/oracle.tfa


TFA post verification:

HOST01:(root)-/u01/app/12.2.0.1/grid/tfa/bin
>./tfactl  print status
.----------------------------------------------------------------------------------------------------.
| Host        | Status of TFA | PID    | Port | Version    | Build ID             | Inventory Status |
+-------------+---------------+--------+------+------------+----------------------+------------------+
| HOST01 | RUNNING       | 105843 | 5000 | 18.1.1.0.0 | 18110020180201015951 | COMPLETE         |
| HOST02 | RUNNING       |  24075 | 5000 | 18.1.1.0.0 | 18110020180201015951  | COMPLETE         |
'-------------+---------------+--------+------+------------+----------------------+------------------'
HOST01:(root)-/u01/app/12.2.0.1/grid/tfa/bin


HOST01:(root)-/u01/app/12.2.0.1/grid/tfa/bin
>./tfactl print repository
.-----------------------------------------------------.
|                     HOST01                     |
+----------------------+------------------------------+
| Repository Parameter | Value                        |
+----------------------+------------------------------+
| Location             | /u01/app/grid/tfa/repository |
| Maximum Size (MB)    | 10240                        |
| Current Size (MB)    | 758                          |
| Free Size (MB)       | 9482                         |
| Status               | OPEN                         |
'----------------------+------------------------------'

.-----------------------------------------------------.
|                     HOST02                     |
+----------------------+------------------------------+
| Repository Parameter | Value                        |
+----------------------+------------------------------+
| Location             | /u01/app/grid/tfa/repository |
| Maximum Size (MB)    | 10240                        |
| Current Size (MB)    | 604                          |
| Free Size (MB)       | 9636                         |
| Status               | OPEN                         |
'----------------------+------------------------------'
HOST01:(root)-/u01/app/12.2.0.1/grid/tfa/bin


TFA collection:

utility: tfactl

Below are the steps to collect TFA:

[root@HOST01]locate tfactl
/u01/app/12.2.0.1/grid/bin/tfactl
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/tfactl
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/tfactl.bat.tmpl
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/tfactl.pl
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/tfactl.tmpl
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/common/tfactlglobal.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/common/tfactlshare.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/common/tfactlwin.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/common/exceptions/tfactlexceptions.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlaccess.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactladmin.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlanalyze.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlbase.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlcell.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlcollection.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactldiagcollect.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactldirectory.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlexttools.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlips.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlmineocr.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/bin/modules/tfactlprint.pm
/u01/app/12.2.0.1/grid/suptools/tfa/release/tfa_home/resources/tfactlhelp.xml
/u01/app/12.2.0.1/grid/tfa/bin/tfactl
HOST01:(grid):(+ASM1)- /home/grid

HOST01:(grid)$/u01/app/12.2.0.1/grid/tfa/bin/tfactl diagcollect -all -from "MON/23/2018 06:30:00" -to "MON/23/2018 07:30:00"

WARNING: User 'grid' is not allowed to run Collections on Storage Cells (Run diagcollect as root user to collect files from Storage Cells).

Collecting data for all components using above parameters...
Collecting data for all nodes
Scanning files from Feb/03/2018 04:00:00 to Feb/03/2018 06:30:00
Creating ips package in master node ...
Trying ADR basepath /u01/app/oracle
Trying to use ADR homepath diag/rdbms/obiprod/OBIPROD1 ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/obiprod/OBIPROD1
Trying to use ADR homepath diag/rdbms/obiprod/OBIPROD ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/obiprod/OBIPROD
Trying to use ADR homepath diag/rdbms/dbm01/DBM011 ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/dbm01/DBM011
Trying to use ADR homepath diag/rdbms/ebsprod_delete/EBSPROD1 ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/ebsprod_delete/EBSPROD1
Trying to use ADR homepath diag/rdbms/obipaos/OBIPAOS1 ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/obipaos/OBIPAOS1
Trying ADR basepath /u01/app/grid
Trying to use ADR homepath diag/asm/+asm/+ASM1 ...
Submitting request to generate package for ADR homepath /u01/app/grid/diag/asm/+asm/+ASM1
Trying to use ADR homepath diag/crs/HOST01/crs ...
Submitting request to generate package for ADR homepath /u01/app/grid/diag/crs/HOST01/crs
Master package completed for ADR homepath /u01/app/oracle/diag/rdbms/obiprod/OBIPROD1
Master package completed for ADR homepath /u01/app/oracle/diag/rdbms/obiprod/OBIPROD
Master package completed for ADR homepath /u01/app/oracle/diag/rdbms/obipaos/OBIPAOS1
Master package completed for ADR homepath /u01/app/oracle/diag/rdbms/ebsprod_delete/EBSPROD1
Created package 3 based on time range 2018-02-03 04:00:00.000000 -05:00 to 2018-02-03 06:30:00.000000 -05:00, correlation level basic
Master package completed for ADR homepath /u01/app/oracle/diag/rdbms/dbm01/DBM011
Created package 3 based on time range 2018-02-03 04:00:00.000000 -05:00 to 2018-02-03 06:30:00.000000 -05:00, correlation level basic
Master package completed for ADR homepath /u01/app/grid/diag/asm/+asm/+ASM1
Created package 3 based on time range 2018-02-03 04:00:00.000000 -05:00 to 2018-02-03 06:30:00.000000 -05:00, correlation level basic
Master package completed for ADR homepath /u01/app/oracle/diag/rdbms/ebsprod/EBSPROD1
Created package 3 based on time range 2018-02-03 04:00:00.000000 -05:00 to 2018-02-03 06:30:00.000000 -05:00, correlation level basic
Master package completed for ADR homepath /u01/app/grid/diag/crs/HOST01/crs
Created package 3 based on time range 2018-02-03 04:00:00.000000 -05:00 to 2018-02-03 06:30:00.000000 -05:00, correlation level basic
Master package completed for ADR homepath /u01/app/oracle/diag/rdbms/ebsprod/EBSPROD
Created package 3 based on time range 2018-02-03 04:00:00.000000 -05:00 to 2018-02-03 06:30:00.000000 -05:00, correlation level basic
Non matching adrhomepath for diag/rdbms/ebsprod_delete/EBSPROD1 during the creation of ips package in remote node HOST02.
Non matching adrhomepath for diag/rdbms/ebsprod_delete/EBSPROD1 during the creation of ips package in remote node HOST03.
Non matching adrhomepath for diag/rdbms/dbm01/DBM011 during the creation of ips package in remote node HOST02.
Non matching adrhomepath for diag/rdbms/ebsprod/EBSPROD during the creation of ips package in remote node HOST02.
Non matching adrhomepath for diag/rdbms/ebsprod/EBSPROD during the creation of ips package in remote node HOST03.
Non matching adrhomepath for diag/rdbms/ebsprod/EBSPROD1 during the creation of ips package in remote node HOST02.
Non matching adrhomepath for diag/rdbms/ebsprod/EBSPROD1 during the creation of ips package in remote node HOST03.
Remote package completed for ADR homepath(s) /diag/crs/HOST02/crs,/diag/crs/HOST03/crs
Remote package completed for ADR homepath(s) /diag/asm/+asm/+ASM2,/diag/asm/+asm/+ASM3

Collection Id : 20180203063237HOST01

Detailed Logging at : /u01/app/grid/tfa/repository/collection_Sat_Feb_03_06_32_37_EST_2018_node_all/diagcollect_20180203063237_HOST01.log
2018/02/03 06:33:30 EST : Collection Name : tfa_Sat_Feb_03_06_32_37_EST_2018.zip
2018/02/03 06:33:30 EST : Collecting diagnostics from hosts : [HOST03, HOST01, HOST02]
2018/02/03 06:33:30 EST : Scanning of files for Collection in progress...
2018/02/03 06:33:30 EST : Collecting additional diagnostic information...
2018/02/03 06:39:37 EST : Completed collection of additional diagnostic information...
2018/02/03 06:42:30 EST : Getting list of files satisfying time range [02/03/2018 04:00:00 EST, 02/03/2018 06:30:00 EST]
2018/02/03 06:42:47 EST : Collecting ADR incident files...
2018/02/03 06:42:48 EST : Completed Local Collection
2018/02/03 06:42:48 EST : Remote Collection in Progress...
.--------------------------------------.
|          Collection Summary          |
+------------+-----------+------+------+
| Host       | Status    | Size | Time |
+------------+-----------+------+------+
| HOST02 | Completed | 98MB | 297s |
| HOST03 | Completed | 91MB | 355s |
| HOST01 | Completed | 70MB | 558s |
'------------+-----------+------+------'

Logs are being collected to: /u01/app/grid/tfa/repository/collection_Sat_Feb_03_06_32_37_EST_2018_node_all
/u01/app/grid/tfa/repository/collection_Sat_Feb_03_06_32_37_EST_2018_node_all/HOST01.tfa_Sat_Feb_03_06_32_37_EST_2018.zip
/u01/app/grid/tfa/repository/collection_Sat_Feb_03_06_32_37_EST_2018_node_all/HOST03.tfa_Sat_Feb_03_06_32_37_EST_2018.zip
/u01/app/grid/tfa/repository/collection_Sat_Feb_03_06_32_37_EST_2018_node_all/HOST02.tfa_Sat_Feb_03_06_32_37_EST_2018.zip
HOST01:(grid):(+ASM1)- /home/grid


[root@na2drdbadm01 ~]# /u01/app/12.2.0.1/grid/tfa/bin/tfactl diagcollect -all

WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
The -all switch is being deprecated as collection of all components is the default behavior. TFA will continue to collect all components.

Collecting data for the last 12 hours for all components...
Collecting data for all nodes and cells
Creating ips package in master node ...
Trying ADR basepath /u01/app/oracle
Trying to use ADR homepath diag/rdbms/dbm01/dbm011 ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/dbm01/dbm011
Trying to use ADR homepath diag/rdbms/obixprod/DROBIXP ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/obixprod/DROBIXP
Trying to use ADR homepath diag/rdbms/drobixp/DROBIXP ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/drobixp/DROBIXP
Trying to use ADR homepath diag/rdbms/drebsxp/DREBSXP ...
Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/drebsxp/DREBSXP
Trying ADR basepath /u01/app/grid
Trying to use ADR homepath diag/crs/na2drdbadm01/crs ...
Submitting request to generate package for ADR homepath /u01/app/grid/diag/crs/na2drdbadm01/crs
Collection Id : 20180405093216HOST03

Detailed Logging at : /u01/app/grid/tfa/repository/collection_Thu_Apr_05_09_32_16_EDT_2018_node_all/diagcollect_20180405093216_HOST03.log
2018/04/05 09:32:53 EDT : NOTE : Any file or directory name containing the string .com will be renamed to replace .com with dotcom
2018/04/05 09:32:53 EDT : Collection Name : tfa_Thu_Apr_05_09_32_16_EDT_2018.zip
2018/04/05 09:32:53 EDT : Collecting diagnostics from hosts : [HOST03, HOST01, HOST02]
2018/04/05 09:32:53 EDT : Scanning of files for Collection in progress...
2018/04/05 09:32:53 EDT : Collecting additional diagnostic information...
2018/04/05 09:33:03 EDT : Getting list of files satisfying time range [04/04/2018 03:51:00 EDT, 04/04/2018 06:00:00 EDT]
2018/04/05 09:33:19 EDT : Collecting ADR incident files...
2018/04/05 09:38:01 EDT : Completed collection of additional diagnostic information...
2018/04/05 09:38:06 EDT : Completed Local Collection
2018/04/05 09:38:06 EDT : Remote Collection in Progress...
.--------------------------------------.
|          Collection Summary          |
+------------+-----------+------+------+
| Host       | Status    | Size | Time |
+------------+-----------+------+------+
| HOST01 | Completed | 95MB | 311s |
| HOST02 | Completed | 80MB | 311s |
| HOST03 | Completed | 79MB | 313s |
'------------+-----------+------+------'
Logs are being collected to: /u01/app/grid/tfa/repository/collection_Thu_Apr_05_09_32_16_EDT_2018_node_all
/u01/app/grid/tfa/repository/collection_Thu_Apr_05_09_32_16_EDT_2018_node_all/HOST01.tfa_Thu_Apr_05_09_32_16_EDT_2018.zip
/u01/app/grid/tfa/repository/collection_Thu_Apr_05_09_32_16_EDT_2018_node_all/HOST02.tfa_Thu_Apr_05_09_32_16_EDT_2018.zip
/u01/app/grid/tfa/repository/collection_Thu_Apr_05_09_32_16_EDT_2018_node_all/HOST03.tfa_Thu_Apr_05_09_32_16_EDT_2018.zip
[root@HOST03 ~]$


Please do follow me and support me on,

Regards,
Mallikarjun Ramadurg
Mobile: +91 9880616848
WhatsApp: +91 9880616848


Diagcollection - Clusterware diagcollection information

Diagcollection - Clusterware diagcollection information 

Diagcollection is a script that allows you to collect diagnostic information in the cluster. The output is very useful to analyze any critical issues with the cluster.

Collect Cluster Health Monitor data from any node in the cluster by running Grid_home/bin/diagcollection.pl script on the node.

When an Oracle Clusterware error occurs, run the diagcollection.pl diagnostics collection script to collect diagnostic information from Oracle Clusterware into trace files.

Run the diagcollection.pl script as root from the Grid_home/bin directory.

[root@wd01db01]# /u01/app/11.2.0.3/grid/bin/diagcollection.sh
Production Copyright 2004, 2010, Oracle.  All rights reserved
Cluster Ready Services (CRS) diagnostic collection tool
The following CRS diagnostic archives will be created in the local directory.
crsData_wd01db01_20140615_2310.tar.gz -> logs,traces and cores from CRS home. Note: core files will be packaged only with the --core option. 
ocrData_wd01db01_20140615_2310.tar.gz -> ocrdump, ocrcheck etc 
coreData_wd01db01_20140615_2310.tar.gz -> contents of CRS core files in text format

osData_wd01db01_20140615_2310.tar.gz -> logs from Operating System
Collecting crs data
/bin/tar: log/wd01db01/cssd/ocssd.log: file changed as we read it
Collecting OCR data 
PROT-302: Failed to initialize ocrdump
/bin/tar: wd01db01_OCRDUMP: Cannot stat: No such file or directory
/bin/tar: Error exit delayed from previous errors
Collecting information from core files

warning: exec file is newer than core file.
Cannot access memory at address 0xf8758966f07d8948

warning: exec file is newer than core file.
Cannot access memory at address 0xf8758966f07d8948

warning: core file may not match specified executable file.

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffdb5fd000

warning: core file may not match specified executable file.

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff033d2000

warning: exec file is newer than core file.
Cannot access memory at address 0x2b8ac494ec38

warning: exec file is newer than core file.
Cannot access memory at address 0x2b267a1a0c38

warning: exec file is newer than core file.
Cannot access memory at address 0x2afdffaa2c38

warning: exec file is newer than core file.
Cannot access memory at address 0x2add0a05dc38

warning: exec file is newer than core file.
Cannot access memory at address 0x2b325bc04c38
The following diagnostic archives will be created in the local directory.
acfsData_wd01db01_20140615_2310.tar.gz -> logs from acfs log.
Collecting acfs data
acfsutil log: CLSU-00100: Operating System function: open64 failed with error data: 2
acfsutil log: CLSU-00101: Operating System error message: No such file or directory
acfsutil log: CLSU-00103: error location: OOF_1
acfsutil log: CLSU-00104: additional error information: open64 (/dev/ofsctl)
acfsutil log: ACFS-00502: Failed to communicate with the ACFS driver.  Verify the ACFS driver has been loaded.
Collecting OS logs
Collecting sysconfig data
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/dm-2 doesn't contain a valid partition table
Disk /dev/dm-3 doesn't contain a valid partition table
[root@wd01db01]#

Regards,
Mallik

Exadata - How To Collect Sosreport on Oracle Linux?

How To Collect Sosreport on Oracle Linux?

When you are working with Oracle support engineer on OS issue, Most commonly asked report from oracle support engineer is sosreport.

What is sosreport?

“The “sosreport” is a tool to collect troubleshooting data on an Oracle Linux system. It generates a compressed zip of debugging information that gives an overview of the most important logs and configuration of a Linux system.

The sosreport includes information about the installed rpm versions, syslog, network configuration, mounted filesystems, disk partition details, loaded kernel modules and status of all services

Please refer the below MOS document for more details:
How To Collect Sosreport on Oracle Linux (Doc ID 1500235.1)

[root@HOST03 ~]# locate sosreport
/etc/selinux/targeted/modules/active/modules/sosreport.pp
/opt/oracle.cellos/validations/init.d/sosreport
/usr/lib/python2.6/site-packages/sos/sosreport.py
/usr/lib/python2.6/site-packages/sos/sosreport.pyc
/usr/lib/python2.6/site-packages/sos/sosreport.pyo
/usr/sbin/sosreport
/usr/share/man/man1/sosreport.1.gz
/usr/share/selinux/devel/include/system/sosreport.if
/usr/share/selinux/targeted/sosreport.pp.bz2
[root@HOST03 ~]#

[root@HOST03 ~]# sosreport 

sosreport (version 3.2)

This command will collect diagnostic and configuration information from
this Oracle Linux system and installed applications.

An archive containing the collected information will be generated in
/tmp/sos.BUdkox and may be provided to a Oracle USA support
representative.

Any information provided to Oracle USA will be treated in accordance
with the published support policies at:

  http://linux.oracle.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [HOST03.r02.xlgs.local]: 
Please enter the case id that you are generating this report for []: 3-20516635001

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Running 75/75: yum...                      
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-HOST03.r02.xlgs.local.3-20516635001-20190711082835.tar.xz

The checksum is: b721f41c5d69454c92915b1168a0c3e7

Please send this file to your support representative.

[root@HOST03 ~]# 

[root@HOST03 ~]# ls -ltrh /tmp/sosreport-HOST03.r02.xlgs.local.3-20516635001-20190711082835.tar.xz
-rw------- 1 root root 15M Jul 11 08:29 /tmp/sosreport-HOST03.r02.xlgs.local.3-20516635001-20190711082835.tar.xz
[root@HOST03 ~]# 

Regards,
Mallik

Monday, February 10, 2020

Infomratica How to Capture Load / Batch start time and update into database batch table?

Infomratica how to capture load / batch start time and update into database batch table?

Lots of informatica jobs or loads or batch jobs we run on everyday but to make sure each job ran successfully or not and capturing the load start time and load ID is very difficult job. 

In case if the load fails due to some network issue or connection issue these batch table is very use full which contains Batch name, Batch ID and Batch start time, based on these information we can cleanup the incomplete data and restart the load. 

Question is capturing these information into batch table is tedious job, This blog will help you in automating this job using linux sheel script.

This script will capture batch start time

ods_batch_start.sh
#!/bin/ksh
#===============================================================================================#
# Name          : ods_batch_start.sh                                                            #
# Written by    : Mallik                                                                        #
# Purpose       : This script will capture batch start time                                     #
# Pending       : NONE                                                                          #
# Creation Date : 10-FEB-2019                                                                   #
# Version       : 1                                                                             #
# ==============================================================================================#
TODAY_DATE=`date +%d%m%Y_%H%M%S`

StartTime=`/u01/Informatica/10.2.0/server/bin/pmcmd getworkflowdetails -sv INFADEV_INT -d INFADEV_Domain -u dev_test -p dev_test wf_TEST_LOAD_01 |grep "Start time" |cut -d"[" -f2 | cut -d"]" -f1`

sqlplus -s ETL_USER/ETL_USER123@TESTDB << EOF

insert into CTL_OWNER.T_CTL_BATCH values (sysdate, nvl((select max(BATCH_ID) from BTT1318.T_CTL_ODS_BATCH where BATCH_NAME= wf_TEST_LOAD_01'),0)+1, 'wf_TEST_LOAD_01', (SELECT to_char(to_date('$StartTime','Dy Mon dd HH24:MI:SS YYYY'),'DD/MON/YY HH12:MI:SS') from dual), NULL);
commit;
exit
EOF

Regards,
Mallik

Exadata - What is Exadata sundiag or sundiag.sh - Collecting sundiag Information

What is Exadata sundiag or sundiag.sh 

Oracle Exadata Diagnostics Collection Tool sundiag.sh

Very often when creating a Support Request (SR) for an issue on an Oracle Exadata Database Machine, you’ll need to run the script “sundiag.sh“.  

Which is the “Oracle Exadata Database Machine – Diagnostics Collection Tool“.

The tool collects a lot of diagnostics information that assist the support analyst in diagnosing your problem, such as failed hardware like a failed disk, etc.

More information Please refer the below MOS document:

SRDC – EEST Sundiag (Doc ID 1683842.1)

Oracle Exadata Diagnostic Information required for Disk Failures and some other Hardware issues (Doc ID 761868.1)

Below is the technical steps for running or collecting sundiag report.

[root@HOST1]# locate sundiag.sh
/opt/oracle.SupportTools/sundiag.sh

 [root@HOST1]#/opt/oracle.SupportTools/sundiag.sh
Oracle Exadata Database Machine - Diagnostics Collection Tool
Gathering Linux information

error: "Input/output error" reading key "net.ipv6.conf.all.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.bond0.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.bondeth0.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.bondeth1.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.default.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.eth0.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.eth1.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.eth2.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.eth3.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.eth4.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.eth5.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.ib0.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.ib1.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.lo.stable_secret"
Skipping collection of OSWatcher/ExaWatcher logs, Cell Metrics and Traces
Skipping ILOM collection. Use the ilom or snapshot options, or login to ILOM
over the network and run Snapshot separately if necessary.

/var/log/exadatatmp/sundiag_pexdbadm01_1707NM10AX_2018_02_04_01_04
Gathering dbms information
Generating diagnostics tarball and removing temp directory
==============================================================================
Done. The report files are bzip2 compressed in /var/log/exadatatmp/sundiag_pexdbadm01_1707NM10AX_2018_02_04_01_04.tar.bz2
==============================================================================
[root@HOST1]#

Regards,
Mallik

Exadata - HCC for Exadata Servers (HCC)

HCC for Exadata servers ( HCC)

What is HCC?

Hybrid Columnar Compression on Exadata enables the highest levels of data compression and provides enterprises with tremendous cost-savings and performance improvements due to reduced I/O. HCC is optimized to use both database and storage capabilities on Exadata to deliver tremendous space savings AND revolutionary performance. Average storage savings can range from 10x to 15x depending on which Hybrid Columnar Compression level is implemented – real world customer benchmarks have resulted in storage savings of up to 204x

HCC Technology overview

Oracle’s Hybrid Columnar Compression technology is a new method for organizing data within a database block. As the name implies, this technology utilizes a combination of both row and columnar methods for storing data. This hybrid approach achieves the compression benefits of columnar storage, while avoiding the performance shortfalls of a pure columnar format. 

A logical construct called the compression unit is used to store a set of hybrid columnar compressed rows. When data is loaded, column values for a set of rows are grouped together and compressed. After the column data for a set of rows has been compressed, it is stored in a compression unit.

4 different type of HCC available are 

Query low
Query high
Archive low
Archive High

Please run reports for all 4 type of compression using script mentioned on TECH_STEP1 by mentioned any one of the below compression which will give you compression ratio.

Query low 
Query high 
Archive low 
Archive High


Technical details on gather statistic ratio and enabling HCC

TECH_STEP1: compression_stats

Compression_Stats_Script for TRANSACTIONS_STG table which will actual tells you how much compression it is going to give you.

TRANSACTIONS_STG_Stats_Query_Low.sql

spool TRANSACTIONS_STG_Query_Low.log
select name from v$database ;
set serveroutput on;
set time on
set timing on
select to_char(sysdate,'DD-MM-YYYY:HH24:MM:SS') time_now from dual;
declare
 v_blkcnt_cmp     pls_integer;
 v_blkcnt_uncmp   pls_integer;
 v_row_cmp        pls_integer;
 v_row_uncmp      pls_integer;
 v_cmp_ratio      number;
 v_comptype_str   varchar2(60);
begin
 dbms_compression.get_compression_ratio(
 scratchtbsname   => 'XXLGARCH',                          -- Tablespace Name
 ownname          => 'XXLGARCH',                          -- USER NAME
 tabname          => 'TRANSACTIONS_STG',          -- TABLE NAME
 partname         => NULL,
 comptype         => dbms_compression.comp_for_query_low, --compression type
 blkcnt_cmp       => v_blkcnt_cmp,
 blkcnt_uncmp     => v_blkcnt_uncmp,
 row_cmp          => v_row_cmp,
 row_uncmp        => v_row_uncmp,
 cmp_ratio        => v_cmp_ratio,
 comptype_str     => v_comptype_str);
 dbms_output.put_line('Estimated Compression Ratio: '||to_char(v_cmp_ratio));
 dbms_output.put_line('Blocks used by compressed sample: '||to_char(v_blkcnt_cmp));
 dbms_output.put_line('Blocks used by uncompressed sample: '||to_char(v_blkcnt_uncmp));
end;
/
select to_char(sysdate,'DD-MM-YYYY:HH24:MM:SS') time_now from dual;
spool off;
exit;

TECH_STEP2: compression_stats_script log

Below is the log collected from the above compression stat script. Which will give you compression prediction ration

XLG_AED_TRANSACTIONS_STG_Query_Low.log 

NAME                                                                            
---------                                                                       
TESTDB                                                                         


TIME_NOW                                                                        
-------------------                                                             
13-12-2017:01:12:16                                                             

Elapsed: 00:00:00.00
Compression Advisor self-check validation successful. select count(*) on both   
Uncompressed and EHCC Compressed format = 1000001 rows                          
Estimated Compression Ratio: 5.5 ----------- Compression Ratio                                    
Blocks used by compressed sample: 23527                                         
Blocks used by uncompressed sample: 130140                                      

PL/SQL procedure successfully completed.

Elapsed: 00:12:50.58

TIME_NOW                                                                        
-------------------                                                             
13-12-2017:01:12:07                                                             

Elapsed: 00:00:00.00


TECH_STEP3: Actual compression_scripts

Actual compression_scripts for TRANSACTIONS_STG table, below is the script which will do actual compression on the table.

TRANSACTIONS_STG_Compress_archive_low.sql 

set serveroutput on;
spool TRANSACTIONS_STG_Compress_archive_low.log
set timing on

select name from v$database;

col SEGMENT_NAME format a40
select segment_name, round(sum(bytes)/1024/1024/1024,2) total_space_GB  from dba_segments  where  owner='XXLGARCH' and segment_name='TRANSACTIONS_STG' 
group by segment_name;

select to_char(sysdate,'DD-MM-YYYY:HH:MI:SS') from dual;

alter table XXLGARCH.TRANSACTIONS_STG move compress for archive low;

select to_char(sysdate,'DD-MM-YYYY:HH:MI:SS') from dual;

col SEGMENT_NAME format a40
select segment_name, round(sum(bytes)/1024/1024/1024,2) total_space_GB  from dba_segments  where  owner='XXLGARCH' and segment_name='TRANSACTIONS_STG' 
group by segment_name;

set lines 300
select owner,table_name,index_name from dba_indexes where status='UNUSABLE';

spool off
exit;

Regards,
Mallik

Exadata - How to collect ILOM snapshots?

How to collect ILOM snapshots?

The purpose of the Oracle ILOM Service Snapshot utility is to collect data for use by Oracle Services personnel to diagnose system problems.

When working with Exadata, you will eventually have to provide an ILOM snapshot to support SR#.

Try to collect a full ILOM snapshot (Oracle support team will always ask for the ILOM snapshots)

Please refer the below MOS document for more details:

How to run an ILOM Snapshot on a Sun/Oracle X86 System (Doc ID 1448069.1)

Technical Steps:

STEP1: Connect to ILOMS for which ILOM you want to collect ILOM snapshots

[root@DBNODE01 ~]# ssh CELL01-ilom
Password: 

Oracle(R) Integrated Lights Out Manager

Version 4.0.3.24 r126093

Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: CELL01-ilom

STEP2: Set snapshot dataset as normal 

-> set /SP/diag/snapshot dataset=normal
Set 'dataset' to 'normal'

STEP3: Provide the DB nodes username/passwords/DB node IP address and location where the ILOM snapshots get dumped

Example: set /SP/diag/snapshot dump_uri=sftp://root:welcome1@10.89.0.13/tmp/

-> set /SP/diag/snapshot dump_uri=sftp://root:welcome1@10.11.12.1/tmp
Set 'dump_uri' to 'sftp://root:welcome1@10.11.12.1/tmp'

STEP4: Check the ILOM snapshot status as beelow. If the status shows running mean ILOM snapshot collection will be still running

-> cd /SP/diag/snapshot
/SP/diag/snapshot

-> show

 /SP/diag/snapshot
    Targets:

    Properties:
        dataset = normal
        dump_uri = (Cannot show property)
        encrypt_output = false
        result = Running ----------- Snapshot running 

    Commands:
        cd
        set
        show

-> 

STEP5: ILOMS snapshot statis shows completed means, ILOM snapshots are collected. Please upload the dumpfiles to SR#

-> show

 /SP/diag/snapshot
    Targets:

    Properties:
        dataset = normal
        dump_uri = (Cannot show property)
        encrypt_output = false
        result = Collecting data into sftp://root@10.11.12.1/tmp/CELL01-ilom_1842XC204A_2019-08-05T10-49-56.zip
                 Snapshot Complete. ------ Snspshot collected successfully 
                 Done.
                 

    Commands:
        cd
        set
        show

-> 

Regards,
Mallik

How to Check Port Status and How to Check Whether port is listening or not on the server?

How check port status and port is listening or not on the server?

There are several methods to verify particular port is opened on the server or not.

There are several methods to verify particular port listening or not in the server

Purpose of this blog is to

1. How to verify the port is opened on the server or not?
2. How to verify the particular port is listening or not?

***** Using nmap command we can check the port status

Check the port 22 status on the server whether it has opened or not:

[mramadur@CT-mnn-01 ~]$ nmap -p 22 -Pn ex01db01
Starting Nmap 5.51 ( http://nmap.org ) at 2016-03-17 06:38 GMT
mass_dns: warning: Unable to determine any DNS servers. Reverse DNS is disabled. Try using --system-dns or specify valid servers with --dns-servers
Nmap scan report for ex01db01 (192.168.1.10)
Host is up (0.21s latency).
rDNS record for 192.168.1.10: ex01db01.prod.mnn.net
PORT   STATE SERVICE
22/tcp open  ssh ------------------- which means post is opened 
Nmap done: 1 IP address (1 host up) scanned in 0.25 seconds


[oracle@ CT-mnn-01 bin]$ nmap -p 22 -Pn ex01db02
Starting Nmap 5.51 ( http://nmap.org ) at 2016-03-17 06:02 GMT
mass_dns: warning: Unable to determine any DNS servers. Reverse DNS is disabled. Try using --system-dns or specify vali                                                 d servers with --dns-servers
Nmap scan report for dal-ep01dbadm01.prod.nmm.com (192.168.1.11)
Host is up.
PORT   STATE    SERVICE
22/tcp filtered ssh ------------------- which means post is not opened
Nmap done: 1 IP address (1 host up) scanned in 2.07 seconds


***** Using telnet command we can check the port status

[infa@ETLD01 Scripts]$ telnet 192.168.1.100 22
Trying 10.24.6.26...
Connected to 192.168.1.100. ------- Which means port 22 is opened
Escape character is '^]'.
SSH-2.0-OpenSSH_7.4
^C
[infa@ETLD01 Scripts]

[infa@ETLD01 Scripts]$ telnet 192.168.1.100 25
Trying 10.24.6.26...
telnet: connect to address 192.168.1.100: Connection refused ------- Which means port 22 is not opened
[infa@ETLD01 Scripts]$


***** Check particular port is listening or not using netstat 

Port 8015 is not listening

[root@s616455s233 ~]# netstat -an | grep LISTEN | grep 8015
[root@s616455s233 ~]# netstat -anp |grep 8015


Port 7741 is not listening

[root@s616455s233 ~]# netstat -an | grep LISTEN | grep 7741
[root@s616455s233 ~]# netstat -anp |grep 7741

Port 7777 is not listening

[root@s616455s233 ~]# netstat -an | grep LISTEN | grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      24179/httpd.worker
tcp        0      0 127.0.0.1:9428          127.0.0.1:7777          TIME_WAIT   -
tcp        0      0 127.0.0.1:9370          127.0.0.1:7777          TIME_WAIT   -
tcp        0      0 127.0.0.1:9312          127.0.0.1:7777          TIME_WAIT   -
[root@s616455s233 ~]# netstat -anp |grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      24179/httpd.worker
tcp        0      0 127.0.0.1:9428          127.0.0.1:7777          TIME_WAIT   -
tcp        0      0 127.0.0.1:9370          127.0.0.1:7777          TIME_WAIT   -
tcp        0      0 127.0.0.1:9312          127.0.0.1:7777          TIME_WAIT   -

Regards,
Mallik

How to Keep your Linux Session Active?

How to keep you linux session active?

Frequently our linux terminal sessions are getting expiring due to idle timeout.

Biggest challenge is how to keep you terminal session active. There are several technics or methods to keep your session active.

1. Some people will set idle session timeout before the launch terminal.
2. Some people will launch the terminal in screen mode so that we can reconnect it back. 
3. Many other ways you can keep you session active 

Question is all above technics works if you are connecting to net terminal session, suppose you have already connected to terminal session and you forget to enable the screen or idle timeout options, in this case below are the few technics to keep your session alive.


1. Using watch command keep you session alive 

watch -n 2 date 
--- which will display date on every 2 seconds, Please read more on watch command usage

[infa@ETLD01 Scripts]$ watch -n 2 date 
Every 2.0s: date Sun Feb  9 20:06:19 2020
Sun Feb  9 20:06:19 +03 2020

2. Using vmstat command keep you session alive 

vmatat 100
--- which will keep count of the virtual memory usage for 100 times 

[infa@ETLD01 Scripts]$ vmstat 100
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0   3808 47567996 198912 14133300    0    0     2  1963   22   14 15 36 46  3  0
^C
[infa@ETLD01 Scripts]$


3. Using top command keep you session alive 

top
--- top command displays processor activity of your Linux box and also displays tasks managed by kernel in real-time and keeps session active

[infa@ETLD01 Scripts]$ top
top - 20:03:44 up 5 days,  8:07,  2 users,  load average: 3.17, 3.55, 3.63
Tasks: 248 total,   5 running, 243 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.2 us, 40.9 sy,  0.0 ni, 40.9 id,  1.5 wa,  0.0 hi,  1.5 si,  0.0 st
KiB Mem : 65789768 total, 47562896 free,  3888520 used, 14338352 buff/cache
KiB Swap: 12578812 total, 12575004 free,     3808 used. 58083008 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  1471 root      16  -4   55528   1092    652 R  87.5  0.0   6547:36 auditd
  1475 root      16  -4   33972   3976   1076 R  56.2  0.0   3926:51 audisp-remote


Regards,
Mallik

Informatica Housekeeping or Move or Archive Files Which are Older than 30 days

Informatica Housekeeping or Move or Archive files which are older than 30 days

As a admin you have to do housekeeping activity frequently and which is tedious. In fact developers runs thousands of Informatica jobs every day and each jobs are creating lots log files like sessions logs and workflow logs and many other logs files.

We need to clean these logs files regularly and cleaning of these log file manually is time taking and need lots efforts.

This blogs explain you how smartly you can do this housekeeping activity.

Below the shell script which will do Housekeeping or Move or Archive files which are older than 30 days.

ods_archive.sh #!/bin/ksh
#===============================================================================================#
# Name          : ods_archive.sh                                                                #
# Written by    : Mallik                                                                        #
# Purpose       : Move files which are older than 30 days                                       #
# Pending       : NONE                                                                          #
# Creation Date : 09-FEB-2019                                                                   #
# Version       : 1                                                                             #
# ==============================================================================================#
TODAY_DATE=`date +%d%m%Y_%H%M%S`

ARCHIVE=/u01/Informatica/10.2.0/server/infa_shared/Archive

BACKUP=/u01/Informatica/10.2.0/server/infa_shared/Backup
BAD_FILES=/u01/Informatica/10.2.0/server/infa_shared/BadFiles
LKP_FILES=/u01/Informatica/10.2.0/server/infa_shared/LkpFiles
LOG=/u01/Informatica/10.2.0/server/infa_shared/log
SESS_LOGS=/u01/Informatica/10.2.0/server/infa_shared/SessLogs
SRC_FILES=/u01/Informatica/10.2.0/server/infa_shared/SrcFiles
STRORAGE=/u01/Informatica/10.2.0/server/infa_shared/Storage
TEMP=/u01/Informatica/10.2.0/server/infa_shared/Temp
TGT_FILES=/u01/Informatica/10.2.0/server/infa_shared/TgtFiles
WORKFLOW_LOGS=/u01/Informatica/10.2.0/server/infa_shared/WorkflowLogs
RECON_DATA=/u01/Informatica/10.2.0/server/infa_shared/SrcFiles/Recon_Data
REF_DATA=/u01/Informatica/10.2.0/server/infa_shared/SrcFiles/Ref_Data
INVALID=/u01/Informatica/10.2.0/server/infa_shared/SrcFiles/Ref_Data/Invalid
PROCESSED=/u01/Informatica/10.2.0/server/infa_shared/SrcFiles/Ref_Data/Processed

find $BACKUP -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $BAD_FILES -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $LKP_FILES -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $LOG -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $SESS_LOGS -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $SRC_FILES -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $STRORAGE -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $TEMP -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $TGT_FILES -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $WORKFLOW_LOGS -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $RECON_DATA -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $REF_DATA -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $INVALID -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;
find $PROCESSED -type f -mtime +30 -exec mv '{}' $ARCHIVE/ \;


Regards,
Mallik

Automation Script | Archivelog Generation Hourly Monitoring

1. List out all the running databases and pic one database where we want to monitore the archive log generation from last 1 month. [oracle@o...