Latest Updates

Reply to comment

Script: Check No Missing Files After Reorganisation of Directory Trees

Sat, 08/01/2009 - 16:43 - jmarki | |

Here's another script I did when I had to reorganised a folder hierarchy of years of data. Basically to ensure files are not missing, or corrupted.


#!/bin/bash                                                                                                                              

#########################
#                        
# checkNoMissingFiles    
# ===================    
#                        
# This script checks that no files are missing after folders are reorganised.
# Basic algorithm is to checksum all files in both old and new folders, then 
# checking through both lists of checksums to ensure all checksums are present
# in both lists.

#                                                                             
# Changelog                                                                   
# =========                                                                   
#                                                                             
# 18 Oct 2007 - Junhao                                                        
# * Initial commit                                                            
#                                                                             
# 11 Dec 2007 - Junhao                                                        
# * Tidied style                                                              
# * Fixed bug with spaces in filenames                                        
# * added option to save generated checksums                                  
# * changed md5sum to sha1sum                                                 
# * changed checksum to general algorithm                                     
#########################                                                     

PATH=/bin:/usr/bin
export PATH       

## Program Locations
awk=/bin/awk        
cat=/bin/cat        
echo=/bin/echo      
find=/usr/bin/find  
grep=/bin/grep      
checksum=/usr/bin/sha1sum
mktemp=/bin/mktemp       
rm=/bin/rm               
tee="/usr/bin/tee -a"    
touch=/bin/touch         
## End Program Locations 

## Start Script

## Script parameters
f_logFile=/dev/null 
d_orgLoc=/dev/null  
d_newLoc=/dev/null  
v_oldFileName=      
v_oldFileChksum=    
f_oldChksumLog=     
f_newChksumLog=     
v_missingFilesCount=0
v_missingFiles=""    
v_output=            
v_f1flag=1           
v_f2flag=1           
## End Script parameters

function print_usage () {
    ${echo} "            
$0                       
Usage: $0 [-L logfile] [-f1 filename] [-f2 filename] [oldDir] [newDir]
 or    $0 -h                                                          
Description: Checks that there are no missing files after reorganising a directory.
Options:                                                                           
  -L logfile    (Optional) Path to log file                                        
  -h            (Optional) This help text                                          
  -1            (Optional) Filename to save checksum for old directory             
  -2            (Optional) Filename to save checksum for new directory             
  oldDir        Location of old directory                                          
  newDir        Location of new directory                                          
"                                                                                  
}                                                                                  

if [ $# -lt 2 ]; then
    print_usage      
    exit 1           
else                 
    while getopts hL:1:2: options; do
        case "${options}" in         
            h)  print_usage          
                exit 1               
                ;;                   
            L)  f_logFile=${OPTARG}  
                ;;                   
            1)  f_oldChksumLog=${OPTARG}
                v_f1flag=0              
                ;;                      
            2)  f_newChksumLog=${OPTARG}
                v_f2flag=0              
                ;;                      
            *)  f_logFile=/dev/null     
                ;;                      
        esac                            
    done                                
    shift $((${OPTIND} - 1))            

    if [ -d "$1" ]; then
        d_orgLoc="$1"   
    else                
        ${echo} "Error: Original directory does not exist!"
        print_usage                                        
        exit 1                                             
    fi                                                     
                                                           
    if [ -d "$2" ]; then                                   
        d_newLoc="$2"                                      
    else                                                   
        ${echo} "Error: New directory does not exist!"     
        print_usage                                        
        exit 1                                             
    fi                                                     

    if [ -z ${f_oldChksumLog} ]; then
        f_oldChksumLog=`${mktemp}`   
    elif [ -f ${f_oldChksumLog} ]; then
        ${echo} "Error: File ${f_oldChksumLog} exists! Please give another filename."
        exit 2                                                                       
    else                                                                             
        ${touch} ${f_oldChksumLog}                                                   
        if [ ! -f ${f_oldChksumLog} ]; then                                          
            ${echo} "Error: ${f_oldChksumLog} cannot be created!"                    
            exit 4                                                                   
        fi                                                                           
    fi                                                                               

    if [ -z ${f_newChksumLog} ]; then
        f_newChksumLog=`${mktemp}`   
    elif [ -f ${f_newChksumLog} ]; then
        ${echo} "Error: File ${f_newChksumLog} exists! Please give another filename."
        exit 3                                                                       
    else                                                                             
        ${touch} ${f_newChksumLog}                                                   
        if [ ! -f ${f_newChksumLog} ]; then                                          
            ${echo} "Error: File ${f_newChksumLog} cannot be created!"               
            exit 5                                                                   
        fi                                                                           
    fi                                                                               
fi                                                                                   

${echo} "${find} \"${d_orgLoc}\" -type f -exec ${checksum} \"\\{\\}\" \\;" | ${tee} ${f_logFile}
${find} "${d_orgLoc}" -type f -exec ${checksum} "{}" \; | ${tee} ${f_oldChksumLog}              
${echo} "${find} \"${d_newLoc}\" -type f -exec ${checksum} \"\\{\\}\" \\;" | ${tee} ${f_logFile}
${find} "${d_newLoc}" -type f -exec ${checksum} "{}" \; | ${tee} ${f_newChksumLog}              


while read -r v_oldFileChksum v_oldFileName; do
    if [[ `${grep} ${v_oldFileChksum} ${f_newChksumLog}` ]]; then
        v_output="Okay:  ${v_oldFileName} -> "                   
        v_output="${v_output} `${grep} \"${v_oldFileChksum}\" \"${f_newChksumLog}\" | ${awk} '{print $2}'`"
    else                                                                                                   
        v_output="ERROR: ${v_oldFileName} is missing"                                                      
        v_missingFiles="${v_missingFiles} ${v_oldFileName}"                                                
        v_missingFilesCount=$((v_missingFilesCount+1))                                                     
    fi                                                                                                     
    ${echo} "${v_output}" | ${tee} ${f_logFile}
done < ${f_oldChksumLog}

#### cleanup ####
if [ "1" == ${v_f1flag} ]; then
    ${rm} ${f_oldChksumLog}
fi
if [ "1" == ${v_f2flag} ]; then
    ${rm} ${f_newChksumLog}
fi


if [ ${v_missingFilesCount} -gt 0 ]; then
    ${echo} "ERROR: ${v_missingFilesCount} files are missing:" | ${tee} ${f_logFile}
    ${echo} "ERROR:   ${v_missingFiles}" | ${tee} ${f_logFile}
    exit 99
else
    ${echo} "Success: ${v_missingFilesCount} files are missing" | ${tee} ${f_logFile}
    exit 0
fi

Reply

Smileys
:):D:bigsmile:;):p:O:|:(:~:((8):steve:J):glasses::party::love:
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <pre> <img> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Textual smileys will be replaced with graphical ones.
  • E-Mail addresses are hidden with reCAPTCHA Mailhide.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.