Informatica Source System Flat Files Validations

Informatica Source System Flat Files Loading Pre file validations and Post file validations.

While processing/loading the source system flat files we need to do couple of Prechks before processing the file.

I have written 2 shell scripts pre and post file validations.

Pre File Validation
--- If file size is 0 KB then reject it
--- If file contain only header then reject it
--- If file contain some data then process it

Post File Validation
--- Once the file successfully processed then archive it.

How it works:
The script works based on the return code from the shell script.

[infa@ETD01]$ ls -ltrh
total 20K
drwxr-xr-x. 2 infa oinstall   98 Feb  5 15:28 files
-rwxr-xr-x. 1 infa oinstall 1.3K Feb  5 15:33 ods_post_file_validation.sh
-rwxr-xr-x. 1 infa oinstall  204 Feb  5 15:38 test.sh
-rwxr-xr-x. 1 infa oinstall   63 Feb  5 15:38 3_File_with_data.csv
-rwxr-xr-x. 1 infa oinstall    0 Feb  5 15:38 2_Empty_file.csv
-rwxr-xr-x. 1 infa oinstall   49 Feb  5 15:38 1_Only_header.csv
-rwxr-xr-x. 1 infa oinstall 1.3K Feb  5 22:13 ods_pre_file_validation.sh

[infa@ETD01]$ ./ods_pre_file_validation.sh
Usage: ./ods_pre_file_validation.sh filename
[infa@ETD01]$ echo $?
1

[infa@ETD01]$ ./ods_pre_file_validation.sh File_Not_Exists.csv
Error: ./ods_pre_file_validation.sh file not found.
[infa@ETD01]$ echo $?
2

[infa@ETD01]$ ./ods_pre_file_validation.sh 1_Only_header.csv
1_Only_header.csv contains only header
[infa@ETD01]$ echo $?
3

[infa@ETD01]$ ./ods_pre_file_validation.sh 2_Empty_file.csv
2_Empty_file.csv is empty.
[infa@ETD01]$ echo $?
4

[infa@ETD01]$ ./ods_pre_file_validation.sh 3_File_with_data.csv
3_File_with_data.csv contains some data and ready for processing
[infa@ETD01]$ echo $?
0

Return codes:
0 - Source File contains some data and ready for processing
1 - Source File mentioned does not exists
2 - Source File size is 0 KB
3 - Source File contains only headers
4 - Source File is empty 

ods_pre_file_validation.sh – for pre file validation

#!/bin/ksh
#==================================================================#
# Name       : ods_pre_file_validation.sh                                                                   #
# Written by : Mallik                                                                                       #
# Purpose    : Precheks File Validation                                                                     #
# Pending    : NONE                                                                                                       #
# Version    : 1                                                                                           # ===================================================================#
TODAY_DATE=`date +%d%m%Y`

INVALID=/u01/Informatica/10.2.0/server/infa_shared/ODS/SrcFiles/Ref_Data/Invalid
REF_DATA=/u01/Informatica/10.2.0/server/infa_shared/ODS/SrcFiles/Ref_Data

cd $REF_DATA

_file="$1"
[ $# -eq 0 ] && { echo "Usage: $0 filename"; return 1; }
[ ! -f "$_file" ] && { echo "Error: $0 file not found."; return 2; }

if [ -s "$_file" ]
then
        count=`wc -l "$_file" | awk '{print $1}'`;
        if [[ ${count} -gt "1" ]]
        then
                echo "$_file contains some data and ready to process"
                return 0
        else
                if [[ ${count} -eq "1" ]]
                then
                        echo "$_file contains only headers"
                        name=${_file%.*}
                        mv $_file $INVALID/${name}_${TODAY_DATE}.csv
                        return 3
                fi
        fi
else
        echo "$_file is empty, does not contaian any data"
        name=${_file%.*}
        mv $_file $INVALID/${name}_${TODAY_DATE}.csv
        return 4
fi

ods_post_file_validation.sh – for pre file validation

#!/bin/ksh
#===============================================================#
# Name       : ods_post_file_validation.s                                                            #
# Written by : Mallik                                                                                              #
# Purpose    : Postcheks File Validation                                                                #
# Pending    : NONE                                                                                          #
# Version    : 1                                                                                    #
# ===============================================================#
TODAY_DATE=`date +%d%m%Y`

PROCESSED=/u01/Informatica/10.2.0/server/infa_shared/ODS/SrcFiles/Ref_Data/Processed
REF_DATA=/u01/Informatica/10.2.0/server/infa_shared/ODS/SrcFiles/Ref_Data

cd $REF_DATA

_file="$1"

echo "$_file contains some data"
name=${_file%.*}
mv $_file $PROCESSED/${name}_${TODAY_DATE}.csv

Regards,

Mallik