Understanding and fixing "too many script compilations" errors in Elasticsearch

Introduction When using Elasticsearch, in some rare instances you may see an error such as “Too many dynamic script compilations within X minutes”. Such an error may be caused by a poor script design where parameters are hard-coded. In other cases this may be due to the script cache being too small or the compilation limit being too low. In this article, I will show how to determine if these default limits are too low, and how these limits can be modified. ...

October 21, 2020

Using Logstash and Elasticsearch to calculate transaction duration in a microservices architecture

September 16, 2020 Introduction Elasticsearch allows you to unify your observability data in a powerful datastore so you can search and apply interactive analytics in real time to a huge number of use cases. In one such use case, you may be using Elasticsearch to monitor a system that is composed of multiple microservices that process a given transaction. For such a system, you may be collecting an event corresponding to when the first microservice in the system starts processing the transaction, and another event corresponding to when the last microservice in the system finishes processing the transaction. In such an approach, each event should include a field with the transaction identifier, which will allow multiple events corresponding to a single transaction to be combined for analysis. ...

September 16, 2020

Using Grok with Elasticsearch to add structure to your data

This article is available on Elastic’s blog as a 3-part series. Please check it out at the following URLs: https://www.elastic.co/blog/structuring-elasticsearch-data-with-grok-on-ingest-for-faster-analytics https://www.elastic.co/blog/slow-and-steady-how-to-build-custom-grok-patterns-incrementally https://www.elastic.co/blog/debugging-broken-grok-expressions-in-elasticsearch-ingest-processors

July 13, 2020

Storing ingest time and calculating ingest lag in Elasticsearch

This article is available on Elastic’s blog at: https://www.elastic.co/blog/calculating-ingest-lag-and-storing-ingest-time-in-elasticsearch-to-improve-observability

June 2, 2020

Using boolean queries to improve search relevancy in Elasticsearch

Page moved to https://alexmarquardt.com/using-boolean-queries-to-improve-search-relevancy-in-elasticsearch/

May 12, 2020

Using boolean queries to improve search relevance in Elasticsearch

This article is available at: https://www.elastic.co/blog/how-to-improve-elasticsearch-search-relevance-with-boolean-queries

May 1, 2020

Using slow logs in Elastic Cloud Enterprise

April 26, 2020 Authors Tom Schreiber Alex Marquardt Version This blog article is based on ECE 2.4.3. Introduction Elastic Cloud Enterprise (ECE) is a platform designed to ease the management, deployment, and configuration of multiple Elasticsearch clusters through a single administrative user interface. ECE, is the same product that powers the Elasticsearch Service hosted offering, and is available for installation on customer-managed servers. ECE can be deployed anywhere - on public or private clouds, virtual machines, or even on bare metal hardware. Once installed, ECE allows Elasticsearch clusters to be created, upgraded, or deleted with the click of a button. ...

April 26, 2020

Using Elastic machine learning to detect anomalies in derivative values

April 21, 2020 Introduction In this blog, we use Elastic machine learning (ML) and derivative aggregations to detect sudden unexpected increases or decreases in the rate-of-change of CPU load on servers that are monitored by Metricbeat. In order to make this blog easier to follow and the results easy to recreate, we abstract away the requirement for driving data from Metricbeat, and instead generate “fake” Metricbeat data using a Python script that drives data into Elasticsearch. ...

April 21, 2020

Using the Elasticsearch Enrich Processor with CSV data

This article is available at: https://www.elastic.co/blog/how-to-enrich-logs-and-metrics-using-an-elasticsearch-ingest-node

March 21, 2020

Converting CSV to JSON in Filebeat

Introduction Many organisations use excel files for creating and storing important data. For various reasons it may be useful to import such data into Elasticsearch. For example, one may need to get Master Data that is created in a spreadsheet into Elasticsearch where it could be used for enriching Elasticsearch documents. Or one may wish to use Elasticsearch and Kibana for analysing a dataset that is only available in a spreadsheet. In such cases, one option is to use Filebeat for uploading such CSV data into an Elasticsearch cluster. ...

March 17, 2020