Day 14: Shell Scripting Challenge - Log Analyzer and Report Generator
#Day11ofDevOpsBlog
Photo by Markus Spiske on Unsplash
Table of contents
Welcome to Day 14 of DevOps adventurous journey. I know I am running behind in publishing blogs but today we will do a new challenge. So let's dive in with real life scenarios.
Challenge
Our today's challenge is as follows:
- Create a bash script that automates the task to analyze log files generated on different servers as a SA, identify specific events, and generate a summary report.
Script Breakdown
Hereβs the bash script to accomplish this:
#!/bin/bash
#Input Validation - Function to tell the usage of the script
function display_usage {
echo "Usage: $0 <Enter the Path on the server where the log file is present>"
}
#Condition to check if user has passed the valid arguments or not
if [ $# -eq 0 ] || [ ! -f "$1" ]; then
echo "Please enter a valid argument"
display_usage
exit 1
fi
path_to_logfile=$1 #A variable passed while running the script to know the path of log file which needs to be analyzed
#1. Count total number of lines in Log File
total_lines=$(wc -l < "$path_to_logfile")
#2. Count the lines with word "ERROR" or "failed"
total_error_count=$(grep -Eic "error | failed" < "$path_to_logfile")
#3.Search for critical events (lines containing the keyword "CRITICAL") and store them in array
mapfile -t critical_events < <(grep -n -i "CRITICAL" "$path_to_logfile")
#4. Identify the top 5 most common error messages and their occurrence count using associative arrays
declare -A error_messages #Syntax in Bash to declare assosiative arrays
while IFS= read -r line; do
# Use awk to extract the error message (fields are space-separated)
error_msg=$(awk '{for (i=3; i<=NF; i++) printf $i " "; print ""}' <<< "$line")
((error_messages["$error_msg"]++))
done < <(grep -i "ERROR" "$path_to_logfile")
# Sort the error messages by occurrence count (descending order)
sorted_error_messages=$(for key in "${!error_messages[@]}"; do
echo "${error_messages[$key]} $key"
done | sort -rn | head -n 5)
#5. Generate the summary report in a separate file
summary_report="log_summary_$(date +%Y-%m-%d).txt"
{
echo "Date of analysis: $(date)"
echo "Log file: $path_to_logfile"
echo "Total lines processed: $total_lines"
echo "Total error count: $total_error_count"
echo -e "\nTop 5 error messages:"
echo "$sorted_error_messages"
echo -e "\nCritical events with line numbers:"
for event in "${critical_events[@]}"; do
echo "$event"
done
} > "$summary_report"
echo "Summary report generated: $summary_report"
Benefits
Enhanced Visibility: The script provides a comprehensive view of system logs, making it easier to identify patterns, anomalies, and trends over time.
Time Efficiency: Automating the analysis and report generation process saves time, allowing you to focus on more critical tasks instead of manually sifting through log files.
Improved Troubleshooting: The script can quickly highlight errors, warnings, or specific events, speeding up the troubleshooting process and helping to resolve issues faster.
Customizable Reports: You can tailor the reports to meet specific needs, such as filtering logs by time, type, or severity, making the information more relevant and actionable.
Proactive Monitoring: Regularly generated reports can help in proactive monitoring, allowing you to catch potential issues before they escalate into bigger problems.
Compliance and Auditing: For industries with strict compliance requirements, a log analyzer report generator script ensures that logs are consistently reviewed and archived, aiding in audits and compliance checks.
Resource Optimization: By analyzing logs, the script can help identify resource bottlenecks, unused resources, or inefficiencies, leading to better system optimization.
Security Enhancement: The script can detect unusual patterns or unauthorized access attempts, contributing to stronger security by enabling quick responses to potential threats.
Error Reduction: Automation reduces the risk of human error in log analysis, ensuring more accurate and consistent reporting.
Scalability: As your system grows, manually analyzing logs becomes impractical. A script can scale with your infrastructure, handling large volumes of data without added manual effort.
Conclusion π
Implementing a log analyzer report generator is a powerful, cost-effective, and scalable solution for managing your data. By using shell scripts, you can automate tasks, making real-life industry scenarios more efficient and streamlined. This approach reduces manual effort, minimizes errors, and accelerates delivery times, all while offering customizable reporting options.