When monitoring a system that has a web interface like this one, a good high-level metric to track is how many API requests are returning with erroneous 40X or 50X HTTP status codes. This is useful, but to understand health we need to see the traffic that is “failing”: Parse the Load Balancer or Firewall logs to view traffic Starting at the highest level, we first count requests to the load balancer, sorted by request.

Having a basic indication of how much activity is occurring is crucial.Ĭan tell you if there has been a massive surge of work, or else if your applications abruptly went idle. Now, lets make some graphs! Count the highest level metrics of “Activity” Simple string searches (“Error”, “timeout”) are useful high level tools, but grouping the results in turn by application, server and type is what makes log aggregation awesome.įollow up visualizations with data tables that contain the data shown, for easy save-to-clipboard and drill-down. Metrics that are not positioned in time are not useful – when is as important as what. Unless we are measuring overall activity, we don’t graph anything except for evidence of problem(s). Start with the highest-level metric of uptime/success at the top, and then drill downwards. Try to get every metric you might want to view onto a single board for easy comparisons. When designing an Operations Dashboard for a specific platform like this, we follow these broad principles: This is a useful capability, but we can do better – it’s dashboard time! Designing an Operations Dashboard OK – How can we make better sense of all these disparate indicators, using Graylog?Īt the most basic level, if we pull all the logs from this architecture into Graylog using collectors, we can search for key terms like “ERROR” across all the different log types at once and review the results on a timeline.

In the circumstances, a notepad search through a few log files for “error” is not going to cut the mustard. There can be thousands of different error events in a log, many benign and regularly occurring long before the issue was reported. Working like this, it is difficult to figure out even basic facts such as when an issue started, which types of workflow are affected or even which part of the infrastructure an error is originating from. Reading the log files directly like this is slow going even with the search tools of modern text editors. There are 14 such log files being actively written, logging a total of around 380,000 events daily. Spoiler: we hope you will not be too shocked to discover that the correct answer is a Graylog Operations Dashboard Performance metrics, looking for a resource bottleneck? The App Server logs, looking for errors? Which ones are relevant to this problem? The Web Server logs, looking for 50X status codes? Which requests are affected? The Load Balancer logs, looking for strange traffic? The on-call engineer must open their laptop, connect to the VPN and through one interface or another and seek to build a picture of what is happening. It’s a customer phoning, and they report that sometimes, maybe a tenth of the time, the web front end is returning a generic error as they try to complete a workflow. Picture a simple E-commerce platform with the following components, each generating logs and metrics. Imagine now the on-call Engineer responsible for this platform, feet up on a Sunday morning watching The Lord of The Rings with a coffee, when suddenly the on-call phone starts to ring!

Capsy study