Day 14: Shell Scripting Challenge - Log Analyzer and Report Generator

Welcome to Day 14 of DevOps adventurous journey. I know I am running behind in publishing blogs but today we will do a new challenge. So let's dive in with real life scenarios.

Challenge

Our today's challenge is as follows:

• Create a bash script that automates the task to analyze log files generated on different servers as a SA, identify specific events, and generate a summary report.

Script Breakdown

Here’s the bash script to accomplish this:

#!/bin/bash

#Input Validation - Function to tell the usage of the script
function display_usage {
    echo "Usage: $0 <Enter the Path on the server where the log file is present>"
}

#Condition to check if user has passed the valid arguments or not
if [ $# -eq 0 ] || [ ! -f "$1" ]; then
    echo "Please enter a valid argument"
    display_usage
    exit 1
fi

path_to_logfile=$1 #A variable passed while running the script to know the path of log file which needs to be analyzed

#1. Count total number of lines in Log File
total_lines=$(wc -l < "$path_to_logfile")

#2. Count the lines with word "ERROR" or "failed"
total_error_count=$(grep -Eic "error | failed" < "$path_to_logfile")

#3.Search for critical events (lines containing the keyword "CRITICAL") and store them in array
mapfile -t critical_events < <(grep -n -i "CRITICAL"  "$path_to_logfile")

#4. Identify the top 5 most common error messages and their occurrence count using associative arrays
declare -A error_messages #Syntax in Bash to declare assosiative arrays
while IFS= read -r line; do
    # Use awk to extract the error message (fields are space-separated)
    error_msg=$(awk '{for (i=3; i<=NF; i++) printf $i " "; print ""}' <<< "$line")
    ((error_messages["$error_msg"]++))
done < <(grep -i "ERROR" "$path_to_logfile")

# Sort the error messages by occurrence count (descending order)
sorted_error_messages=$(for key in "${!error_messages[@]}"; do
    echo "${error_messages[$key]} $key"
done | sort -rn | head -n 5)

#5. Generate the summary report in a separate file
summary_report="log_summary_$(date +%Y-%m-%d).txt"
{
    echo "Date of analysis: $(date)"
    echo "Log file: $path_to_logfile"
    echo "Total lines processed: $total_lines"
    echo "Total error count: $total_error_count"
    echo -e "\nTop 5 error messages:"
    echo "$sorted_error_messages"
    echo -e "\nCritical events with line numbers:"
    for event in "${critical_events[@]}"; do
        echo "$event"
    done
} > "$summary_report"

echo "Summary report generated: $summary_report"

Benefits

1. Enhanced Visibility: The script provides a comprehensive view of system logs, making it easier to identify patterns, anomalies, and trends over time.

2. Time Efficiency: Automating the analysis and report generation process saves time, allowing you to focus on more critical tasks instead of manually sifting through log files.

3. Improved Troubleshooting: The script can quickly highlight errors, warnings, or specific events, speeding up the troubleshooting process and helping to resolve issues faster.

4. Customizable Reports: You can tailor the reports to meet specific needs, such as filtering logs by time, type, or severity, making the information more relevant and actionable.

5. Proactive Monitoring: Regularly generated reports can help in proactive monitoring, allowing you to catch potential issues before they escalate into bigger problems.

6. Compliance and Auditing: For industries with strict compliance requirements, a log analyzer report generator script ensures that logs are consistently reviewed and archived, aiding in audits and compliance checks.

7. Resource Optimization: By analyzing logs, the script can help identify resource bottlenecks, unused resources, or inefficiencies, leading to better system optimization.

8. Security Enhancement: The script can detect unusual patterns or unauthorized access attempts, contributing to stronger security by enabling quick responses to potential threats.

9. Error Reduction: Automation reduces the risk of human error in log analysis, ensuring more accurate and consistent reporting.

10. Scalability: As your system grows, manually analyzing logs becomes impractical. A script can scale with your infrastructure, handling large volumes of data without added manual effort.

Conclusion 🌟

Implementing a log analyzer report generator is a powerful, cost-effective, and scalable solution for managing your data. By using shell scripts, you can automate tasks, making real-life industry scenarios more efficient and streamlined. This approach reduces manual effort, minimizes errors, and accelerates delivery times, all while offering customizable reporting options.