Setup Kibana Dashboards for Nginx Log Data to Understand the Behaviour

What do we want to achieve?

  • Geographical map to see where people are connecting from
  • Piechart that represents the percentage of cities accessing my blog
  • Top 10 Most Accessed Pages
  • Top 5 HTTP Status Codes
  • Top 10 Pages that returned 404 Responses
  • The Top 10 User Agents
  • Timeseries: Status Codes Over Time
  • Timeseries: Successfull Website Hits over time
  • Counter with Website Hits
  • Average Bytes Returned
  • Tag Cloud with the City Names that Accessed my Blog

Pre-Requirements

Setup Kibana Visulizations

Geomap: Map to see where people are connecting from

-> Metrics, Value: Count.  
Buckets, Geo Coordinates,
Aggregation: Geohash,
Field: nginx.access.geoip.location

Piechart: Cities

-> Metrics: Slice Size, Aggregation: Count
-> Buckets: Split Slices, Aggregation: Terms,
Field: nginx.access.geoip.city_name,
Order by: metric: count,
Order: Descending, Size: 20

Top 10 Accessed Pages

-> Metrics: Y-Axis, 
Aggregation: Count
-> Buckets: X-Axis,
Aggregation: Terms,
Field: nginx.access.url,
Ordery by: Metric count,
Order: Descending, Size 10
NOT (nginx.access.url:"/" OR nginx.access.url:"/rss/" OR nginx.access.url:"/subscribe/" OR nginx.access.url:*.txt)

Top 5 HTTP Status Codes

-> Metrics: Y-Axis, 
Aggregation: Count
-> Buckets: X-Axis,
Aggregation: Terms,
Field: nginx.access.response_code,
Ordery by: Metric count,
Order: Descending, Size 5

Top 404 Pages

-> Metrics: Y-Axis, 
Aggregation: Count
-> Buckets: X-Axis,
Aggregation: Terms,
Terms, Field: nginx.access.url,
Order by: Metric count,
Order: Descending,
Size 20

Top 10 User Agents

-> Metrics: Y-Axis, 
Aggregation: Count
-> Buckets: Split Rows,
Aggregation: Terms,
Field: nginx.access.user_agent.name,
Order by: Metric count,
Order: Descending, Size 10

Timeseries: Status Codes over Time

.es(index=filebeat*, timefield='@timestamp', q=nginx.access.response_code:200).label('OK'), .es(index=filebeat*, timefield='@timestamp', q=nginx.access.response_code:404).label('Page Not Found')

Timeseries: Successfull Website Hits over Time

.es(index=filebeat*, timefield='@timestamp', q=nginx.access.response_code:200).label('200')

Count Metric: Website Hits

-> Search Query: fields.blog_name:sysadmins AND nginx.access.response_code:200  
-> Metrics: Y-Axis,
Aggregation: Count

Average Bytes Transferred

-> Metrics: Y-Axis, 
Aggregation: Average,
Field: nginx.access.body_sent.bytes
-> Buckets: X-Axis,
Aggregation: Date Histogram,
Field: @timestamp

Tag Cloud with Most Popular Cities

-> Metrics: Tag size, 
Aggregation: Count
-> Buckets: Tags,
Aggregation: Terms,
Field: nginx.access.geoip.city_name,
Order by: Metric count,
Order: Descending, Size 10

Create the Dashboard

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ruan Bekker

Ruan Bekker

DevOps Engineer and Open Source Enthusiast