|
Avoid complicated, expensive & unfocused tools - Install LOGROBOT specifically to Generate Reliable Notification Alerts on an unlimited number of log files on as many UNIX hosts & servers as is necessary! |
|
|
Screenshots for Nagios Users: Set up notification alerts on all log files - Monitor time stamps, log contents & size, Alert if log file suddenly no longer exists (if deleted or not created), Alert if log lacks proper permissions...etc
|
|
|
|
|
|
Increase awareness to network infrastructure problems with fast detection of outages, Generate actionable alerts on Application specific issues / events, Database errors, System problems, Abnormal log behavior and more! |
|
|
|
|
|
Screenshots for Cacti Users: Graph & Beautify your log file activities - Visually inspect the graphs of all your monitored logs - Easily identify Rises & Drops in the frequency of entries
|
|
|
For these particular graphs, LOGROBOT scans a unique log file every 5 minutes. The log file is written to by an application several times a minute. LOGROBOT scrapes the log at five minute intervals and then pulls out all information written to it within that time frame. From the 5 minute worth of data retrieved after scanning, LOGROBOT graphs the number of lines it finds that contains "hostname.*SUCCESS" or "hostname.*FAILURE" ("hostname" = name of a server). The GREEN graph show the number of lines found containing "hostname.*SUCCESS". The RED graph show the Failures. The middle graph shows an Aggregate of both graphs. |
|
|
Scan / Monitor log files for user-defined entries & EXCLUDE specific lines from the results Case Scenario: Within the last 30 minutes, find out how many lines in the log file [ /var/log/app.log ] contain both entries of "ERROR" and "Client". If any lines are found containing these two strings (ERROR.*Client), take note of that. From the list of lines found, see if there are any lines that also contain the keywords "error 404" OR "updateNumber". If there are, remove them from the list. After removing them, show me what is left. If the number of lines left is between 5 and 9, alert as WARNING. If equal to or over 10, alert as CRITICAL. If below 5, do not alert! Command: logrobot autofig /var/log/app.log 30 ‘ERROR.*Client’ '(error 404|updateNumber)' 5 10 -show
Monitor log files for certain entries - ALERT IF those entries are NOT found Case Scenario: For instance, within the last 30 minutes, if LOGROBOT does not find at least 2 lines containing the words "Success" and "Client" and "returned 200" OR "update:OK" in the log file, it must alert. So in other words, the lines to search for MUST contain both words of Success & Client (Success.*Client) AND one or both of the strings returned 200 and update:OK. Command: logrobot autofig /var/log/app.log 30 ‘SUCCESS.*Client’ '(returned 200|update:OK)' 2 2 -notfoundn
Scan Log files for specific entries & display results to the screen
This is particularly helpful in cases where you might want to see the actual lines that contain the patterns you instructed the tool to search for.
Example: logrobot autofig /var/log/app.log 30 ‘ERROR.*Client’ '(error 404|updateNumber:OK)' 5 10 -show
Example: logrobot autofig /var/log/app.log 30 ‘SUCCESS.*Client’ '(returned 200|update:OK)' 5 10 -show
Scan log files for minutes, hours, days, weeks or months worth of data
For instance, to pull out 2 weeks of information from within a large log file and to find out how many lines contain certain strings and patterns, you can run a command similar to this:
Example: logrobot autofig /var/log/app.log 2w ‘ERROR|error|panic|fail’ ‘ERROR|error|panic|fail’ 5 10 -foundn
Notice the [ 2w ]. And also, notice the strings being searched for. I repeated the strings ‘ERROR|error|panic|fail’ twice because there is no need to specify different search terms to look for. You don't have to repeat the first string. You can just enter a dot in its place for the second string..i.e:
logrobot autofig /var/log/app.log 2w ‘ERROR|error|panic|fail’ ‘.’ 5 10 -foundn
From this specific example, I'm telling LOGROBOT that I care about EVERY single line that contains any of the keywords I provided. The [ 2w ] of course means 2 weeks.
See below for the different ways of specifying the date range:
5m = 5 minutes (changeable to any number of minutes) 10h = 10 hours (changeable to any number of hours) 2d = 2 days (changeable to any number of days) 2w = 2 weeks (changeable to any number of weeks) 3mo = 3 months (changeable to any number of months)
Suppose you inherited a UNIX environment at your new job and don't know what to search for in the logs, here's an idea; instead of worrying about what to watch for, why not force the logs to reveal their hidden contents?
In the example below,
LOGROBOT was instructed to search the entire messages file (denoted
with the '.'). Then, it is to
ignore every line
that contains any one of these specific strings: 'nagios-primary
nagios' OR 'not responding' OR 'synchronized to'.
Whatever lines are left after these THREE patterns are ignored should
be outputted to the screen.
The logic here is; if you can
identify which entries in the logs are of NO importance to you, you
can exclude them from being monitored. Therefore, if a log file
is stripped of the familiar/unwanted, whatever is left will be
unfamiliar, thus requiring investigation.
General LOGROBOT Syntax: /bin/logrobot autofig (logfile) (time-in-minutes) '(string1)' '(string2)' (warn) (critical) (-foundn) Basic Usage:
[root@monitor
jbowman]# 2---240---108---ATWFILF---(Apr/13)-(03:35)---(Apr/14)-(03:35:23)
So now lets break this down: logrobot is the tool name. autofig is an option that is passed to the logrobot tool to tell it what to do. In this particular case, autofig is instructing logrobot to "automatically figure out" what type of log file /var/log/messages is, and if the format of the log file is supported, perform the remaining functions. /var/log/messages is of course the log file. 1440 is the amount of previous minutes you want to search the log file for. 1440 = last 24 hours. "ntpd" is one of the strings that is in the lines of logs that you're interested in. "stratum" is another string on the same line that you expect to find the "ntpd" string on. Specifying these two strings (luance and Err1310) isolates and processes the lines you want a lot quicker, particularly if you're dealing with a huge log file. 5 specifies Warning. By specifying 5, you're telling the program to alert as WARNING if there are at least 5 occurrences of the search strings you specified, in the log file within the last 60 minutes. 10 specifies Critical. By specifying 10, you're telling the program to alert as CRITICAL if there are at least 10 occurrences of the search strings you specified, in the log file within the last 60 minutes. -foundn specifies what type of response you'll get. By specifying -foundn, you're saying if anything is found that matches the specified strings within the 60 minute time frame, then that should be regarded as a problem and outputted out. Summarized Explanation: As you can see, the logrobot tool is monitoring a log file. The arguments that are passed to the tool instructs it to do the following: Within the last 60 minutes, if the tool finds less than 5 occurrences of the specified strings in the log file, DO NOT alert. If the tool finds between 5 to 9 occurrences of the specified strings in the log, it'll alert with a WARNING. If the tool discovers 10 or more instances of the strings in the log within the last 60 minutes, it'll alert with a CRITICAL. Now, let us look at the result of the command: 2---240---108---ATWFILF---(Apr/13)-(03:35)---(Apr/14)-(03:35:23) There are 6 columns which are separated by 3 hyphens (---). The first column shows the exit code of the command you just ran. 0 means all is well. 1 means WARNING, which means, LOGROBOT discovered conditions that fell under the WARNING specification you provided. 2 means CRITICAL, which means, the worst case scenario has been reached. In this particular example, here's what the output is telling us: You requested to have the /var/log/messages file scanned as far back as 24 hours ago (1440 minutes). The timeframe that was scanned was from [ April 13, 03:35 ] to [ April 14, 03:35 ]. After scanning through the records that were written to the log in that time frame, LOGROBOT found 108 lines that contained both strings of "ntpd" and "stratum 2". Also, as an FYI, the last date and time those specific strings were found in the log file was 240 seconds ago.
Monitor Any Log regardless of log format:
Alert if a
specific pattern is found in a log file. In the alert, show
offending/matching log lines:
Command:
NagiosLogMonitor
hostA logrobot autonda /var/log/syslog 60m 'kernel|panic' '.' 3
5 syslog_check -ndshow
Explanation:
Taking
simplicity to a New level:
Instead of forcing
users to have to read complex documentations, LOGROBOT provides real
life examples of its usage right from the command line. Yes, REAL LIFE
EXAMPLES! No guessing, no confusion, no scratching of the head. We
strongly believe in simplicity and we take the extra steps many
utilities refuse to take. Example: [root@nagios-primary ~]# ./logrobot autofig
----------------------------------------------------------------------------------------------------------------------------------------------
|
Who needs LOGROBOT?
Download LOGROBOT if you wish to:
More Features:
Monitors database logs for any error (mysql/oracle/etc) - Get notified
on issues quickly
Monitors custom application Log Files on any Unix
host, regardless of flavor
Alerts on the size / disk usage of files on your UNIX hosts -
Prevent log file(s) from taking up space
Monitor timestamps of files (ensure specific logs are being
updated regularly / frequently)
Scan Log files by time frames (i.e. previous 20 minutes, 60 minutes,
1 day, 1 week
etc) - Precise!
Alert when expected record of events are NOT found in a
log within a set period of time
In-Depth Analysis: scan logs for instances of unusual entries - Easily
Identify bizarre log content
Monitor different log formats (including
custom logs, not just the typical system files)
Scan logs for specific entries and exclude a list of user specified
patterns from the result
Includes PHP Web Interface for viewing of log files
on local and remote hosts
Allows for monitoring of all log files, even Windows files (mounted through
NFS on a UNIX server)
Ability to specify multiple strings / patterns
to watch for using Command line or Configs
Identify & Inform users on how long ago a particular string/pattern/keyword
was last found
Remote Agent Included for the monitoring of logs on several hosts
FROM ONE master
Automatically figures out conditions on which to recover alerts, based on log content
& other variables
Simple, pluggable command-line parameters (no need
for any confusing configuration files)
Automatically
detects log file type and format without user intervention
Get notified via email of all
events related to your log of choice on any server(s) you specify
Updated / Upgraded regularly to meet new demands from various clients,
different companies
Other Specific
Features:
Scan / Scrape / Monitor log files for any error
Monitor all logs in a specific directory
Point logrobot to ANY directory with just one check!
Avoid having to define separate checks for each log file
Specify the type of files to exclude / include in monitoring
Automate log checks via Nagios or CRONTAB
Get email alerts & notifications on all log checks
Access documentation directly from the CLI
Monitor log files for abnormal behavior/activity
Manage log file checks from a central location
Eliminate tedious administration
Avoid cumbersome maintenances
Adapts seamlessly to any custom scenario
Supported Log Files
Will all my logs be supported?
Yes, all log types / log formats are supported.
Some of the supported log files are listed below:
Tomcat Catalina.out logs
Apache Maxclient logs
Apache access logs
Apache error logs
OutOfMemory logs
JBoss log files
Java log files
Weblogic logs
Glassfish logs
Syslog log monitor
Maillog / Postfix / Syslog log files
Mysqld / Oracle Alert logs
Log4j
NEW FEATURE: Monitor any type of
log file regardless of format
Monitor File Timestamps:
In the example below, the LOGROBOT tool is asked to check the timestamp of the /var/log/messages file. If this file is not updated in 10 minutes, alert as WARNING. If the file isn't updated in 20 minutes, alert as CRITICAL. If the latest update time of the file is under 10 minutes, do not ALERT at all. Just abort with a 0. The '290' tells (in seconds) how long ago the file was last written to.
[root@monitor
jbowman]#
Monitor File Size:
In the next example below, LOGROBOT is being asked to monitor the size of the /var/log/messages file. If the size of the file is between 60 and 120 megabytes, it will alert as WARNING. If the size of the file is equal to or above 120 MB, it will alert as CRITICAL. Of course, if the reported file size is below 60, there are no issues to alert on. It will just abort with a 0. The 19:52 MB below denotes the current/latest size of the specified size.
[root@monitor
jbowman]#
[root@monitor
jbowman]# logrobot autodoc /var/log/messages
60 120 -filesize
Log Analysis:
Analyze Log Files Minute by Minute or Hour by Hour
If you wish to quickly identify, from your logs, what time period an issue occurred, logrobot can accomplish that for you effortlessly. The options to use are "-exceldm" and "-exceldh". These two options are designed specifically for minute by minute or hour by hour analysis.
EXAMPLE 1:
If you wanted to find out how many instances of your search patterns occurred each hour for the last 1 day, examine the below. The very first line of the output here tells us there are 2551 instances of the occurrence "CRITICAL" at the 6:00 hour. This immediately tells us it was in the 6:00 hour something serious happened.
[root@nagios-master ]# logrobot autofig /var/log/messages 1d 'CRITICAL' '.' 1 2 -exceldh
Example 1: Using Cacti, a log file on 6 different servers is being graphed for 500 related errors Each server below is tagged with a different color for easy identification:
Example 2: Using Cacti, a log file on 8 different servers is being graphed for 500 related errors Each server below is tagged with a different color for easy identification:
|
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
