Category: Uncategorized

Automating the Import and Export of Kibana Saved Objects

Introduction

Kibana is an open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features that allow users to visualize data from Elasticsearch in various formats such as charts, tables, and maps.

While Kibana offers a robust user interface for managing many tasks, certain operations can become tedious and time-consuming when done manually, especially for operations teams managing large and complex environments. One such operation is the migration of Kibana spaces and objects between environments—a task that can be critical in scenarios where clients cannot utilize the snapshot/restore functionality provided by Elasticsearch.

Options for Migrating Kibana from one Environment to Another

Traditionally, moving Kibana objects (dashboards, visualizations, saved searches, etc.) and spaces between different environments or instances requires either using the snapshot and restore features of Elasticsearch or manually exporting and importing JSON files through the Kibana UI.

If you can make use of snapshot/restore for migrating Kibana feature states, that should be the preferred approach. However, snapshot/restore might not always be available due to various constraints such as corporate security policies or other restrictions. Meanwhile, manual exports and imports are labor-intensive. In this blog, we present a third option – scripts that will automate the import and export of Kibana Saved Objects.

Scripts available on github

If you would like to automate the process of exporting and importing Kibana objects, you may find the kibana-import-export repository on github to be useful.

This repository has two simple to use scripts: ‘export_kibana.py’ which extracts your Kibana objects from all of your Kibana spaces to a local file. And ‘import_kibana.py’ which can be used to recreate the extracted Kibana spaces and objects on a different Kibana instance.

These scripts make use of the the following APIs:

After downloading the scripts from https://github.com/alexander-marquardt/kibana-import-export, type the following commands to get information about the command line parameters that are available:

  • export_kibana.py -h
  • import_kibana.py -h

Disclaimer and Usage Notes

  • Use at Your Own Risk: These scripts are provided as is, without warranty of any kind. Users should use the scripts at their own risk.
  • Validation Required: Always validate the script in a realistic test environment before applying it on production systems. This ensures that any unforeseen errors can be caught and corrected in a safe manner.
  • Intended Use: These scripts are generally intended for setting up a new system to which you are migrating, and which is empty. The default setting in the import script is to overwrite existing Kibana objects in the destination. If the destination system is not empty and you wish to preserve existing objects, you should modify the command line parameters.

Conclusion

For Kibana users unable to leverage snapshot/restore, these scripts for exporting and importing spaces are a feasible alternative.

Barcelona: A Personal Guide

  1. Motivation
  2. Welcome to Barcelona!
  3. Language and Navigation
  4. Choosing Your Base
  5. Getting Around
  6. Must-See Attractions
  7. Dining
  8. Nightlife in Barcelona
  9. Safety
  10. Cultural Insights and Local Etiquette
  11. Biking and Hiking Around Barcelona
    1. Biking Adventures:
    2. Hiking Escapes:
  12. Day and Weekend Trips
  13. Conclusion

Motivation

Originally from Canada, I have lived in Barcelona on and off since 2003. Over the years, many friends and acquaintances have asked for my recommendations on what to see and do in this beautiful city. To save time and ensure no detail is missed, I’ve compiled this comprehensive guide based on my personal experiences and research I’ve done over the years.

Welcome to Barcelona!

Barcelona offers a unique travel experience that caters to every kind of adventurer. From enchanting streets lined with timeless architecture to warm, sandy beaches, the city celebrates life at every corner. Whether you’re a history enthusiast, a shopping aficionado, or a lover of fine food, Barcelona will not disappoint.

Language and Navigation

Barcelona is a bilingual city where both Spanish and Catalan are spoken. They are distinct languages, each with its own nuances.

Street names and neighborhood signs may appear in either language, which can sometimes confuse visitors who may be looking for a street name that is written in one language on their map, but appears in the other language on the street signs. A basic understanding of both languages can enhance your experience.

Choosing Your Base

For a central stay, consider areas around Plaza Catalunya or Paseo de Gracia (“Passeig de Gracia” in Catalan). The neighbourhoods of Gracia, Eixample, or El Born are also great choices. Note that Barcelona is quite lively and many older buildings lack good soundproofing, which can a make it difficult to sleep if you’re not used to the noise!

Getting Around

Barcelona is compact and densely packed. You can walk from “one side” to the “other” (e.g. Plaça d’Espanya to Plaça de les Glòries Catalanes) in about an hour. Additionally, the public transportation system is excellent. If you are planning to spend most of your time in Barcelona, a car is absolutely not needed, and will be more of a hinderance than a benefit.

For those wishing to explore beyond the city, Catalunya offers well-maintained highways. If you’re renting a car for a few days road trip, remember that most vehicles are manual transmission.

Must-See Attractions

Barcelona offers a rich tapestry of attractions that appeal to a wide range of interests, from awe-inspiring historical landmarks to bustling markets. To ensure a seamless experience, it’s wise to plan your visits in advance, particularly for popular sites like the Sagrada Familia, where tickets can sell out weeks ahead.

Here are some essential attractions to include on your Barcelona itinerary:

  • Sagrada Familia: An iconic church by Gaudi that’s been under construction for over a century, showcasing amazing architectural details. This masterpiece is known for its intricate facades and ambitious interior, which combine Gothic and Art Nouveau forms.
  • Park Guell: Gaudi’s imaginative park is a public space that welcomes visitors with colorful, mosaic-covered buildings, serpentine benches, and the famous salamander sculpture. It offers fantastic views of the city and a glimpse into Gaudi’s artistic mind.
  • Sant Antoni Market: Recently renovated, this market is a hub for locals seeking fresh produce, meats, and seafood. It’s also a popular spot for brunch, offering a variety of food stalls and casual dining options.
  • The Borne Neighborhood: A lively area known for its mix of historic charm and modern culture. It features narrow streets filled with boutique shops, trendy bars, and art galleries, plus the Picasso Museum.
  • Barrio Gotico (Gothic Quarter): This is the old heart of Barcelona, featuring narrow winding streets, historic architecture, and the Barcelona Cathedral, known for its stunning Gothic facade and peaceful cloister.
  • The Port (Port Vell): Once a historic harbor, now transformed into a leisure area with the Maremagnum shopping mall, IMAX cinema, and the largest aquarium in Europe, making it a family-friendly destination.
  • Barceloneta: Known for its stretch of (fake and somewhat dirty/dusty) sand beach and promenade lined with seafood restaurants and bars. It’s a popular spot for both relaxation and recreation, and during summer is packed with tourists.
  • Las Arenas Shopping Centre: A unique shopping center created from a converted bullring, offering a variety of shops, dining options, and a circular rooftop terrace that provides panoramic views of the city.
  • Montjuic: This scenic hill features a fortress at its peak, the Magic Fountain below, and several museums including the National Art Museum of Catalonia. It’s also a great place for peaceful walks and enjoying nature within the city.
  • Tibidabo: Standing on Barcelona’s tallest hill, Tibidabo Amusement Park offers traditional rides and attractions, along with fantastic views of the city and coastline from the Church of the Sacred Heart that tops the park.
  • Palau de la Musica: An exquisite example of Modernista architecture, this concert hall is renowned for its lavish interior that includes a stunning stained glass skylight and detailed ceramic decorations.
  • Plaza de Catalunya: As the city’s central and largest square, it connects the old city with the Eixample district and is surrounded by several notable buildings, fountains, and sculptures. It’s a common meeting point and hosts various public events. I recommend going to the restaurant at the top of the Corte Ingles (which is a large department store) to take in the views of the centre of Barcelona. The food is served in a cafeteria, and is acceptable (but you are there for the views, not the food).
  • Ancient Roman Ruins near the City Hall (MUHBA): Situated beneath the city’s streets, this part of the Barcelona History Museum offers an extensive archaeological site that showcases the ancient Roman city of Barcino. This includes streets, houses, and public baths, providing a deep dive into the historical layers of Barcelona.
  • Barcelona Cathedral (Catedral de Barcelona): Located in the Gothic Quarter, this cathedral is famous for its magnificent Gothic architecture and a beautifully serene cloister where 13 white geese are kept.
  • Church in the Borne Area (Basilica of Santa Maria del Mar): An excellent example of Catalan Gothic architecture, known for its stark, majestic interior and impressive stained glass windows.
  • Plaza del Ayuntamiento (Plaça de Sant Jaume): This square is the political heart of Barcelona, housing both the City Hall and the Palace of the Generalitat of Catalonia. It often hosts cultural events and political gatherings.

Dining

The local food scene in Barcelona starts late, with dinner typically around 9 PM. To immerse yourself in authentic local dining, avoid tourist traps by dining when the locals do. It is not customary or expected to leave tips in Barcelona. For a meal costing around 50 euros, a small tip of 2 or 3 euros is sufficient, as service staff do not rely solely on tips, which is sometimes reflected in the service quality.

Customer service in Spain, and particularly in Barcelona, may not always meet the expectations set by other countries. Service can be slow, and it’s not uncommon for it to take 15 minutes to take your order and up to half an hour for your food to be served. Remember, you are on vacation—try to relax and enjoy the slower pace of life that is part of the local charm.

You may wish to test out TheFork while you are in Spain – it is an app that allows you to make reservations at many restaurants, and often offers discounts of up to 50% off the food portion of your meal

Below are some must-try restaurants that offer a variety of flavors and experiences:

  • Ciutat Comtal: Renowned for its tapas and bustling atmosphere, this restaurant offers a wide array of Spanish dishes in a central location. It is perhaps my favourite restaurant in Barcelona. It doesn’t accept reservations, to you may wish to try it out in non-peak hours.
  • Alba Granados: Upper-middle/fine dining – great food and a nice atmosphere. With drinks and food, it will probably cost €60 per person or more. The food is very good and worth it. Appetizers that I have enjoyed include octopus, clams, and tuna tartar. The paellas and black rice are a good choice of main, as are the meat dishes that come with a hot stone.
  • Can Fisher Restaurante playa Bogatell: Nice location along the beach and good seafood.
  • La Pubilla: This eatery offers traditional Catalan cuisine with a modern twist. Located in the Gracia district, it’s known for its fresh, market-driven menu. If you want to go, you’ll likely need to make a reservation a week in advance.
  • Bodega Sepulveda: Known for its authentic Catalan dishes and extensive selection of local wines, this bodega provides a warm, rustic setting perfect for a relaxed meal.
  • El Rebost de Sant Antoni: A charming deli and eatery offering a selection of homemade Catalan specialties. It’s a great place to experience the flavors of the region.
  • Maison Carne: Specializing in steak, Maison Carne offers a unique concept where diners can enjoy high-quality meat dishes, making it a hit among meat lovers. The menu is very limited, but very good.
  • Teatro Kitche & Bar: If you’re looking for high-end (expensive) dining, give this place a try. I haven’t been here yet, but its on my list. (let me know if you go!)

Each of these restaurants provides a unique dining experience that reflects the diversity and richness of Barcelona’s culinary scene. Whether you’re in the mood for traditional Catalan dishes, fresh seafood, or international flavors, Barcelona has something to offer every palate.

Nightlife in Barcelona

Barcelona’s nightlife has options ranging from laid-back bars to high-energy nightclubs. As the sun sets, the city transforms into a lively hub of entertainment, catering to all tastes and preferences.

  • Starting the Night: Begin your evening in the El Born neighborhood, known as ‘The Borne’ by locals. This area comes alive around 10 PM, offering an array of trendy bars that are perfect for pre-club drinks. While it’s popular with tourists, its charm and lively atmosphere make it a great starting point for anyone looking to experience Barcelona’s nightlife.
  • Clubs and Late-night Venues: For those looking to dance the night away, the beachfront has many option including Opium Mar, CDLC, and Shôko. Known for attracting a large tourist crowd, this area offers a mix of electronic and European music and often gets busy. It’s advisable to arrive by 10 or 11 PM to avoid long queues.Several of these clubs remain open until 6 AM, making it ideal for night owls looking to party until dawn.
  • Language Considerations: If you’re not fluent in Spanish, sticking to tourist-friendly clubs might be more enjoyable, as staff and patrons are more likely to speak English. More local clubs often see less English spoken, which can be challenging for those not well-versed in Spanish.
  • Dress Code: Dress to impress if you’re planning a night out in Barcelona’s clubs. Many clubs enforce a dress code, be sure to check how you need to dress before you go. Smart casual with black or brown shoes is usually a safe bet to ensure entry.
  • Exclusive Spots: The W Hotel, known locally as ‘Hotel Vela’ due to its sail-like structure, offers upscale nightlife options with two exclusive bars/clubs. One is located on the top level, offering spectacular views of the Mediterranean, while the other is on the 2nd level and features a chic poolside vibe.
  • Cost of Partying: Expect nightlife in Barcelona to come with a price tag. Typical costs include around 7 euros for a beer and between 10 to 14 euros for a mixed drink at more fashionable spots. Prices at bars, especially those busy from 11 PM to 2 AM, may be slightly lower, but not by much.
  • Local Interaction: Keep in mind that due to the high volume of tourists, locals in Barcelona may not always seem overly friendly in nightlife settings. It’s a city that has experienced the impacts of tourism extensively, so while you’re out, maintaining a respectful and polite demeanor can help foster more positive interactions.

Exploring Barcelona’s nightlife offers an exciting glimpse into the city’s culture, with diverse options that cater to all tastes. Whether you’re in the mood for a chill evening or a night of dancing, Barcelona has something to offer every nocturnal adventurer.

Safety

Barcelona has one of the highest pickpocket rates globally, so vigilance is crucial. Be cautious with your belongings and wary of overly friendly strangers. Do not leave your purse hanging on the back of your chair when dining out or at bars, as it is likely to get stolen. Instead, follow the local practice of keeping your purse on your knee. Despite the presence of petty crimes, Barcelona is not considered dangerous. Keep an eye on your wallet, phone, or any items that might be targeted for theft.

The city’s appearance might be misleading in some areas; there is a lot of graffiti on the metal doors that people pull down over their shops when they are closed—this does not indicate that you are in a bad neighborhood.

Cultural Insights and Local Etiquette

While exploring Barcelona, it’s important to be mindful of the local culture and how tourism impacts the community. Barcelona is an exiting city with a rich history and culture that attracts millions of tourists each year. However, the influx of visitors has led to a sense of ‘overtourism’ among some locals, which can sometimes affect their desire to interact with tourists.

  • Interacting with Locals: Many residents in Barcelona may seem reserved or less willing to engage with tourists. This attitude partly stems from the city’s overwhelming number of visitors, which can put a strain on local life. As a visitor, being respectful of people’s space and understanding that not everyone may want to interact can make for more positive experiences.
  • Language and Communication: Attempting to speak a few words in Catalan or Spanish can go a long way in showing respect for the local culture. Even simple phrases like “Bon dia” (Good morning in Catalan) or “Gracias” (Thank you in Spanish) can help create a friendly rapport.

Biking and Hiking Around Barcelona

Barcelona is not only a hub for rich culture and busy city life but also offers numerous opportunities for outdoor enthusiasts to explore its stunning natural surroundings through biking and hiking.

Biking Adventures:

  • Barcelona’s Coastline: One of the most scenic bike routes is along the Barcelona coastline. Starting from Barceloneta Beach and extending towards the Forum area, this route offers breathtaking sea views and is perfect for a leisurely ride. The path is well-paved and suitable for cyclists of all levels.
  • Collserola Park: For a more challenging ride, head to Collserola Park, the largest metropolitan park in the world. It’s a haven for mountain bikers with its extensive network of trails that weave through lush forest and offer panoramic views of the city from various lookout points. Rentals are available near the entrances, and guided tours can help you discover the best trails.
  • Montjuïc: This iconic hill in Barcelona is not only a cultural landmark but also a great place for cycling. The relatively gentle slopes and well-maintained paths make it ideal for family outings or solo rides. The climb to the top rewards cyclists with stunning views of the city and the Mediterranean.

Hiking Escapes:

  • Montjuïc Hill: Apart from biking, Montjuïc is also a fantastic place for hiking. There are several walking paths that lead you past the Montjuïc Castle, the Olympic Stadium, and various gardens, each offering its own charm and history.
  • The Carretera de les Aigües: Just a short trip from Barcelona, this flat trail runs along the Tibidabo mountain ridge and offers spectacular views over the city. It’s perfect for a relaxing hike or a run, especially at sunset.
  • Garraf Natural Park: Located about 30 minutes from Barcelona, this park features rugged limestone hills, deep valleys, and a variety of trails that cater to both casual walkers and serious hikers. The park’s monastery, situated in the heart of Garraf, is a peaceful spot to visit during your hike.
  • Sant Jeroni Trail, Montserrat: For those willing to venture a bit further, the Sant Jeroni trail in Montserrat provides a more challenging hike. It takes you to the highest point of the Montserrat massif, where you can enjoy awe-inspiring views of the monastery and surrounding landscape.

Day and Weekend Trips

Barcelona’s strategic location offers exceptional opportunities to explore the rich landscapes and diverse cultures of Catalonia and beyond. Whether you’re interested in historical sites, nature, or unique local gastronomy, there is a nearby destination to suit every taste. Here are many places I’ve visited, along with a few still on my to-do list:

  • Puigcerdà: Nestled in the Pyrenees near the French border, known for its stunning mountain landscapes and as a hub for winter sports. The town features a lovely lake and a nice market every Sunday.
  • Perpignan: Located in Southern France, this city is infused with French and Catalan cultures. Key attractions include the Palace of the Kings of Majorca and areas showcasing distinct Catalan architecture.
  • Carcassonne: Just across the French border, about three hours from Barcelona by car. This medieval fortified hilltop town is ringed by massive double walls and over 50 towers, with cobbled streets, a Gothic basilica, and beautiful views across the Aude Valley. A doable overnight if you’re looking for a change of country without a flight.
  • Andorra: A small principality in the Pyrenees known for ski resorts and duty-free shopping. It’s also great for summer hiking, offering breathtaking mountain scenery and a blend of Catalan and French cultural influences.
  • Palamós: A coastal town on the Costa Brava noted for its fishing industry, especially prawns. It offers beautiful beaches, a charming old town, and a lively port area perfect for seafood lovers.
  • Tarragona: A historic seaside town famous for its well-preserved Roman ruins, including an amphitheater by the sea. It offers a mix of ancient history and lively beach culture.
  • Sitges: Known for beautiful beaches, nightlife, and cultural festivals. This town combines the charm of a Mediterranean fishing village with cosmopolitan sophistication.
  • Girona: Offers a journey back in time with its medieval walls, narrow streets, and well-preserved Jewish Quarter. The city’s majestic cathedral and colorful houses along the Onyar River are must-sees.
  • Vic: Renowned for its medieval architecture and lively market days in the main square, serving as a cultural hub with a rich history.
  • Montserrat: A stunning mountain range with peculiar rock formations and a Benedictine monastery, known for spiritual significance and breathtaking views.
  • Olot: Located within the La Garrotxa volcanic zone, surrounded by natural parks with extinct volcanoes and lush forests, ideal for nature lovers.
  • Sant Feliu de Guixols: Offers sandy beaches and a historical monastery, providing a quieter alternative to the busier Costa Brava spots.
  • Baix Empordà: Features rolling landscapes, medieval villages, and vineyards, perfect for exploring rural Catalonia.
  • Cadaqués: A picturesque village with whitewashed houses and cobblestone streets, beloved by artists like Salvador Dalí.
  • Tossa de Mar: Features an intact medieval castle overlooking the beach, combining historical intrigue with coastal relaxation.
  • Ullastret: Home to significant Iberian archaeological sites, offering insights into ancient civilizations.
  • Besalú: Famous for its Romanesque bridge and well-preserved medieval architecture, offering a deep dive into Catalonia’s medieval past.
  • Monells: Known for its medieval square and stone archways, providing a tranquil escape.
  • Peratallada: Renowned for its ancient stone buildings and fortifications, a living museum of medieval history.
  • Pals: A Gothic town known for its towered walls and panoramic views from its hilltop setting.
  • Castellfollit de la Roca: Perched atop a basalt cliff, offering dramatic views and a unique geological setting.
  • Beget: Features a stunning Romanesque church and beautifully preserved medieval stone houses.
  • Peralada: Known for its castle, wine production, and a summer music festival held in its gardens.
  • Madremanya: A beautifully preserved stone village, offering serene and winding streets.
  • PortAventura: One of Europe’s largest amusement parks near Tarragona, with thrill rides and themed areas.
  • Figueres: Birthplace of Salvador Dalí, home to the Dalí Theatre-Museum which houses the largest collection of his works.
  • Platja d’Aro: A modern resort town on the Costa Brava known for its nightlife and sandy beaches.
  • Begur: A town with a well-preserved medieval castle, surrounded by coves and clear waters, ideal for history and beach enthusiasts.
  • Castelldefels: Just south of Barcelona, known for its vast beach perfect for water sports and a historic castle overseeing the town.
  • Gavà: Famous for its prehistoric mines and a beautiful beach that’s less crowded than those in Barcelona.

These destinations showcase the diversity and beauty of Catalonia and the broader region, providing ample opportunities for exploration

Conclusion

Barcelona is an extraordinary city with more to do than can typically be fit into one vacation. Embrace the spontaneous, stay alert, and savor every moment in this magnificent city.

Re-directing Elasticsearch documents with out-of-range timestamps that (would) fail to get written into Time Series Data Streams

Introduction

Elasticsearch Time Series Data Streams (TSDS) are designed to provide an efficient and scalable way to handle time-based data within the Elasticsearch ecosystem. This feature is specifically optimized for storing, searching, and managing time-series data such as metrics, and events, where data is continuously indexed in chronological order. However, if events arrive with timestamps that fall outside of a pre-defined range, they will be lost.

In this blog I will demonstrate logic that can be added to an Elasticsearch ingest pipeline which can be used to intercept documents that would be rejected by the TSDS index due to timestamp range issues, and to instead redirect them to a “failed” index. The documents that are redirected to the “failed” index may (for example) be used to raise alerts and examined.

Motivation

As discussed in a previous blog on Storing ingest time and calculating ingest lag in Elasticsearch, there are several reasons why events may have incorrect timestamps. If such events are to be sent into a TSDS, they may be rejected due to falling outside of the range that is allowed by the look_back_time or the look_ahead time. Additionally, if ILM is used then older indices may already be set to read-only, which may cause events with old timestamps to silently fail to be written.

It is often useful to know if documents that should have been indexed into a TSDS are rejected, so that they can be further investigated. This blog presents a simple ingest pipeline script that can be used to redirect such documents to a “failed” index for further investigation.

Design

In order to ensure that no documents with incorrect timestamps disappear, you may wish to set the look_ahead_time and look_back_time intervals to a slightly larger time range than what we use in the script presented below. This will ensure that the script catches all timestamp range issues, rather than the indexer failing and the event disappearing. Additionally, if you know that due to ILM running, older indices become read-only, you would want to ensure that the time range defined in the script below would send these documents to a separate index rather than silently failing to write to the read-only index.

Below is a script that gives a general demonstration of the concept of how to redirect events to a different index based on their timestamp. The script should be adjusted so that the time ranges are relevant to your particular situation.

First, lets setup an index template for a data stream as follows:


PUT /_index_template/my-data-stream-template?pretty
{
  "index_patterns": [
    "my-data-stream*"
  ],
  "data_stream": {},
  "template": {
    "settings": {
      "index": {
        "mode": "time_series",
        "routing_path": [
          "host"
        ],
        "number_of_replicas": 0,
        "number_of_shards": 2
      }
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "bytes": {
          "type": "long",
          "time_series_metric": "gauge"
        },
        "host": {
          "type": "keyword",
          "time_series_dimension": true
        }
      }
    }
  }
}

You can then define an ingest pipeline that validates time ranges on the incoming events as follows:

PUT _ingest/pipeline/my-timestamp-pipeline
{
  "description": """If a document falls outside of the time ranges that would result in it being correctly written into a time series data stream, then send it to a special index for further evaluation""",
  "processors": [
    {
      "set": {
        "field": "ingest_time",
        "value": "{{_ingest.timestamp}}"
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": """
          def future_hours = 2;
          def past_hours = 2;
          
          // Parse the ingest time and original timestamp once
          ZonedDateTime ingestTime = ZonedDateTime.parse(ctx["ingest_time"]);
          ZonedDateTime eventTimestamp = ZonedDateTime.parse(ctx["@timestamp"]);
          
          // Check if the original timestamp is more than 2 hours earlier of the ingest time 
          // or more than 2 hours later than the ingest time
          if (eventTimestamp.isBefore(ingestTime.minusHours(past_hours)) || eventTimestamp.isAfter(ingestTime.plusHours(future_hours))) {
            ctx['_index'] = "timeseries_failures_index";
          } 
        """
      }
    }
    ]
}

And finally you can test the above script by passing in documents such as the following (be sure to update the timestamp to fall within a few hours of the current time that you are testing the script):

PUT /my-data-stream/_bulk?refresh&pipeline=my-timestamp-pipeline&pretty
{"create": {}}
{"@timestamp":"2024-04-16T18:11:30Z","host":"host_c", "bytes":1234}
{"create": {}}
{"@timestamp":"2024-04-16T19:11:30Z","host":"host_c", "bytes":2345}

Conclusion

In this article, I have showed how a simple ingest pipeline can be written which will detect documents that have timestamps which would cause them to fail to index in an Elasticsearch Time Series Data Stream (TSDS). This is a proof of concept that can be extended and adapted to your ingest requirements.

Using Logstash to scan inside event contents to replace sensitive data with a consistent hash

Introduction

Logstash is commonly used for transforming data before it is sent to another system for storage, and so it is often well positioned for finding and replacing sensitive text, as may be required for GDPR compliance.

Therefore, in this blog I show how Logstash can make use of a ruby filter to scan through the contents of an event and to replace each occurrence of sensitive text with the value of its hash.

This is done by making use of Ruby’s gsub functionality, providing a definition of a regular expression pattern that will be replaced (in this blog, we demonstrate with an email address regex pattern), and by executing a hash function to calculate the replacement value.

Note that the use case addressed in this blog is different than the use cases for the fingerprint filter. The fingerprint filter can combine and hash one or more fields, but it does not analyze or replace substrings inside the field(s).

This blog also demonstrates several other Logstash concepts, including:

  1. Use of a generator in Logstash to automatically create new events to make testing of your filters quick and easy.
  2. Defining custom ruby code inside your Logstash pipeline.
  3. Use of the stdout output to easily debug your Logstash pipeline.
  4. Automatic reload functionality to allow you to immediately validate any code changes that you make in your Logstash pipeline.

Acknowledgement

Thanks to my co-worker Joao Duarte at Elastic for coming up with the custom ruby filter that is presented in this blog.

Code description

The code given below demonstrates an entire Logstash pipeline that creates a simulated input event that contains a message with multiple email addresses in it. It then processes the event with a custom ruby filter which finds and replaces each email addresses in the “message” field with its SHA1 digest, and then writes the modified event to stdout.

Logstash pipeline

input {
    generator {
        lines => [
            '{"message": "Someone had an email address foobar@example.com and sent mail to foobaz@another-example.com"}'
        ]
        count => 1
        codec => "json"
    }
}

filter {
  ruby {
    init => "require 'digest'; @email_regex = /([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})/"
    code => "str = event.get('message'); event.set('message', str.gsub(@email_regex) {|v| Digest::SHA1.hexdigest(v) })"
  } 
}

output {
    stdout { codec => "rubydebug" }
}

Executing the pipeline

The above pipeline can be executed with the following command line, which will automatically restart the pipeline each time it is modified: 

./bin/logstash -f <your pipeline config file> --config.reload.automatic

This will generate the following output, which confirms that the email addresses in the message field (of the event created in the generator) have been replaced with the SHA1 hash value, as desired:

{
      "@version" => "1",
          "host" => "New2020MacBook.lan",
       "message" => "Someone had an email address 6f25d1a16b65ee184e83d06a268af7f44d4e8a10 and sent mail to f1377593215966404efb0f42c6ce48017c2c5522",
    "@timestamp" => 2022-01-20T17:50:10.598Z,
      "sequence" => 0
}

Conclusion

In this brief post, I have demonstrated the following concepts:

  1. How to use a generator to create a custom event to help easily verify the functionality of your Logstash filters and to help debug your Logstash pipeline.
  2. How to define a custom Ruby filter in your Logstash pipeline.
  3. How to make use of Ruby’s gsub functionality along with a regular expression and a call to a SHA1.hexdigest function, for replacing sensitive text.
  4. How to view the resulting modified events on stdout.
  5. How to automatically reload your pipeline as your make edits.