A summary of recent research on increasing lifespan

Authors

  • Ronald R. Marquardt
  • Suzhen Li
  • Alexander Marquardt

In association with All Natural Nutritional Products (ANNP) Inc.

Introduction

This blog provides an overview of recent research in the field of ageing.

A brief overview of the book: “Lifespan: Why we age — and why we don’t have to”

David A. Sinclair and M. D. LaPlante have written a New York Times Bestseller called Lifespan: Why we age — and why we don’t have to. David Sinclair has also given many interviews and talks which can be found online.

Below we summarize some of the highlights from this book and from his online discussions.

  • David’s theory is that ageing is quite simply a loss of information over time.
  • Our cells store both digital and analog information. The main difference between them is that analog information is continuous whilst digital is discrete.
  • Our genetic code is digital, while the epigenetic code that determines and dictate which genes are expressed in each cell is analog. For example, even though skin cells and nerve cells have the same DNA, it is the epigenetic code that expresses different genes in a skin cell than would be expressed in a nerve cell.
  • Analog information is prone to the accumulation of noise and disruption over time.
  • David suggests that noise accumulation in epigenetic information disrupts gene expression and other nuclear processes within a cell leading to malfunctioning and inevitably ageing of the organism.
  • In other words, ageing doesn’t involve a change in our DNA sequences but can change how our body reads the DNA sequences which unlike genetic changes, may be reversible. Most epigenetic changes involve the reversible binding of certain compounds such as methyl groups to DNA.

Metabolic changes associated with ageing

  • Autophagy is an essential catabolic process that promotes the clearance of waste cellular component (cellular garbage). Loss of autophagy results in cell death and a number of human diseases. Caloric restriction or starvation promotes autophagy and therefore, longevity and healthy living. Excess caloric intake results in the opposite effect, obesity, etc.
  • A critical metabolic compound required for the autophagic destruction of waste cellular material (garbage) is NAD+ (oxidized nicotinamide adenine dinucleotide).
  • The enzyme sirtuin uses NAD+ to activate lysozymes which autophagically degrades waste protein.
  • Sirtuin is a very important enzyme and is activated by resveratrol and some other nature compounds.
  • Essential requirements for autophagy and longevity:
    1. NAD+ and not NADH (oxidized vs reduced form)
    2. NAD+ is increased by fasting (caloric restriction), NAD+ precursor supplementation and activation of sirtuins by compounds such as resveratrol.
    3. The net effect is an enhancement of autophagy and life expectancy.

Suggestions for life style regime for longevity and healthy living (What does David do?)

David Sinclair does not make recommendations for any products or supplements, however he has openly discussed his daily regime which includes the following:

  • 1 gram of NMN: NMN is converted into NAD+ which is required for autophagy. NMN is expensive. Two new NMN forms have been very recently discovered which are 5 to 10 times more effective then NMN. They should be available in 2 to 5 years.
  • 1 gram of resveratrol. It enhances sirtuin activity which uses NAD+ to enhance autophagy. Health Canada recommends one 250 mg capsule containing 98% resveratrol, twice per day.
  • 1 gram of metformin (a prescription drug for diabetes but has autophagy benefits)
  • Takes a daily dose of vitamin D, vitamin K2 and 83 mg of aspirin (baby aspirin.
  • David has given up on desserts (he doesn’t want to consume sucrose, a metabolic poison).
  • David tries to skip a meal per day.
  • Our opinion of additional supplements that David could add to his regimen:
    1. Psyllium (Metamucil): A plant based soluble fiber or some other soluble fiber such as beta glucans or arabinoxylans. They reduce nutrient up take and favorably modifies gut microflora which produce beneficial short chain fatty acids such as butyric acid.
    2. Grape seed extract, high in anthocyanidins (antioxidants).
    3. Daidzein, a phytoestrogen in soy isoflavones. Important for extension of life.
    4. Apigenin, luteolin or quercetin as inhibitors of the CD 38 and CD 157 NADase pathway. The net effect of these supplements is to increase NAD+ which can be used by sirtuins.

Autophagy A requirement for healthy living and longevity

Autophagy is an essential process in which damaged cellular proteins are degraded by lysosomes. Autophagy is an essential for homeostatic maintenance. Disruption of autophagy results in many metabolic diseases including a greatly shortened life span.

The sirtuins enzymes play an essential role in regulating autophagy including aging and various age-related disease (including diabetes, cancer, and heart, liver, kidney diseases). [See review by Lee, Experimental and Molecular Medicine 511:102, 2019].

The sirtuins enzymes react with NAD+ to remove acetyl groups from protein, DNA, and histones which is an essential autophagic step. Tissue concentration of NAD+ can be increased by calorie restriction or supplementation of NAD+ precursors such NR or NMN. Resveratrol, a natural product, greatly increases the activity of sirtuins.

Supplements that can increase sirtuin activity and autophagy

This section is based on a review of “The rise of NAD+ and sirtuin – activating compounds.” [Bonkowski and Sinclair, Natural Reviews, Molecular Cell Biology 17:679-690. Important supplements include the following:

  • NAD+- precursors: NMN and NR are currently the best available supplements. NAD+ is an essential substrate for the sirtuin enzyme.
  • Sirtuin -activity compounds (STACs):
  • Resveratrol: It is the most potent natural activator of the essential enzyme. It is found in red wine and supplements.
  • Metformin: Metformin is a prescription drug for the control of type 2 diabetes. It also increases sirtuin activity.
  • Resveratrol and Metformin: A combination of two compounds when added to the diet synergistically enhanced SIRT1 (sirtuin one) activity. They have been shown to alleviate insulin resistance in rodents fed high fat diets via amelioration of hyperlipidemia, hyperglycemia, inflammation and increased oxidative mechanisms. A potentially powerful antiaging combination. [E1 Agamy and Ahmed, Benha Medical Journal 37(3):561-577, 2020.]

Inhibition of enzymes that compete with sirtuin for NAD+:

  • There are three main classes of enzymes that consume NAD+ ; PARPs enzymes, sirtuins, and CD 38 and CD 157 enzymes. CD 38 and CD157 enzymes are upregulated in ageing tissue. The age-dependent decline of NAD+ levels with increased activity of CD 38 and CD 137 range from 10 to 65% depending on organ and age. CD 38 is one of the main NADases in mammals and plays a key role in age-related decline of NAD+. Several inhibitors of CD 38 have been identified including apigenin, quercetin and luteolin (naturally occurring flavonoids). Supplement of these compounds results in an inhibition of NAD+ utilization by CD 38 resulting in an increased amount for sirtuins and therefore an increased level of autophagy.
  • Inhibitor of CD 38: Current information on inhibition of NADases. Please see publication by Ogura et al., Aging (Albany NY) 12(12):11325-11336, 2020; and Covarrubias et al., Nature Reviews, Molecular Cell Biology 27:119-141, 2021.
  • Naturally occurring CD 38 inhibitors of NADases other than sirtuins that are flavonoids, such apigenin, quercetin and luteolin.

Summary of NAD+ and its precursors:

NAD+ is a coenzyme involved in the control of metabolism. It is involved in redox reactions carrying electrons from one reaction to another. NAD+ (nicotinamide dinucleotide) decreases with age. The following list of precursors can increase NAD+ levels.

  • Niacin (vitamin B3, nicotinic acid): Niacin is direct precursor of the coenzymes NAD+ and NADH.
  • NADH: NADH is the reduced form of NAD+ and is involved in the production of ATP, the energy-factory in the cell.
  • NR (nicotinamide riboside): It is niacin with a riboside group. NR is a precursor of NAD+ and is a more potent supplement than either niacin or NAD+. Niagen (NR) is commercially available supplement that is more potent than either niacin or NAD+.
  • NMN (nicotinamide mononucleotide): It is also a direct precursor of NAD+. It can be synthesized from NR by the addition of a phosphate group. It is much more potent as a supplement than either niacin or NAD+. NMN supplements are expensive but have antiaging benefits.
  • NRH (the reduced form of NR): NRH is a powerful newly discovered metabolite that increases NAD+ intercellular levels 50-fold greater than NR (Giroud – Gerbetant et al., Mol Metabolism 30:192-202, 2019). NRH supplements will probably prove to be highly effective antiaging compounds which should be available after approval by FDA and HC, probably in several years.
  • NMNH (the reduced form of NMN): NMNH is another newly discovered metabolite. It is a potent NAD+ precursor. NMNH increases NAD+ levels in the body to a much higher extent and faster than NMN or NR (Zapata -Perez et al., The FASEB Journal, 2021:35e21456).
  • NMNH and NRH are powerful newly discovered compounds: NMNH like NRH are exciting newly discovered metabolites that have considerable potential to increase cellular NAD+ levels and probably autophagy and, therefore, to prevent metabolic diseases. These compounds may prove to have powerful antiaging properties but may not be commercially available for several years. Can life expectancy be greatly increased when these compounds are consumed?

E. Pathways for synthesis of different precursors of NAD+

The potency of NRH and NMNH are much greater (~50 fold) than NR and NMN. NR and NMN are more potent than niacin and NAD+ but much less potent than NRH and NMNH. All of the precursor compounds are converted into NAD+ in the cell. Currently, NR and NMN are commercially available supplements.

A model for the connection between NAD+, sirtuins and autophagy

Nutrient limitation increases the level of the essential cellular metabolite NAD+, which is utilized by sirtuins to increase autophagy within cells and tissues. The activity of sirtuins and autophagy decreases during normal aging as well as due to some diseases and correlates with the known reduction in levels of tissue NAD+. From Lee 2019, Experimental & Molecular Medicine.

Overview on procedures to increase NAD+ in the body

  1. Calorie restriction and exercise:
  2. Natural product supplements to increase tissue NAD+ levels:
    • NAD+ precursor supplements (NMN OR NR) will increase cellular NAD+ levels,
    • Resveratrol increases sirtuin activity thereby enabling its use of NAD+ for autophagy.
    • Apigenin, quercetin or luteolin will inhibit the activity of the CD 38 pathway particularly in the elderly. This will result in an increased level of NAD+ which can be used by sirtuins.
    • The combined benefits of taking the three compounds should provide greatly enhanced synergistic anti-aging benefits. Unfortunately, research on the combined safety and benefits of taking the combined supplements has not been reported.

A summary of therapeutic approaches to restore NAD+ levels and their impact on health


Ageing is associated with decreased nicotinamide adenine dinucleotide (NAD+) levels in tissues that promote or exacerbate ageing-related diseases. Thus, restoring NAD+ levels has emerged as a therapeutic approach to prevent and treat ageing-related diseases and to restore health and vigour during the ageing process. Some potential strategies that boost NAD+ levels include lifestyle changes, such as increasing exercise, reducing caloric intake, eating a healthy diet and following a consistent daily circadian rhythm pattern by conforming to healthy sleeping habits and mealtimes. Another approach is the use of small-molecule inhibitors or activators to boost NAD+ biosynthesis and the use of dietary supplements, including NAD+ precursors, such as nicotinamide mononucleotide (NMN) and nicotinamide riboside (NR). All of these approaches promote increased tissue NAD+ levels and are beneficial for health. These include improved tissue and organ function, protection from cognitive decline, improved metabolic health, reduced inflammation and increased physiological benefits, such as increased physical activity, which may collectively extend patient health span and potentially lifespan [Covarrubias et al., Nature Reviews, Molecular Cell Biology 22:119-141, 2021.]

Conclusion

In this blog we provided an overview of some of the recent research in ageing, as well as a list of supplements and lifestyle changes which theoretically may slow the rate of ageing.

Combining Elasticsearch stemmers and synonyms to improve search relevance

The article called The same, but different: Boosting the power of Elasticsearch with synonyms gives a great introduction to why and how you can incorporate synonyms into your Elasticsearch-powered application. Here I build upon that blog and show how you can combine stemmers and multi-word synonyms to take the quality of your search results to the next level.

Motivation

Imagine that you are using Elasticsearch to power a search application for finding books, and in this application you want to treat the following words as synonyms:

  • brainstorm
  • brainstorming
  • brainstormed
  • brain storm
  • brain storming
  • brain stormed
  • envisage
  • envisaging
  • envisaged
  • etc.

It is tedious and error prone to explicitly use synonyms to define all possible conjugations, declensions, and inflections of a word or of a compound word.

However, it is possible to reduce the size of the list of synonyms by making use of a stemmer to extract the stem of each word before applying synonyms. This would allow us to get the same results as the above synonym list by specifying only the following synonyms:

  • brainstorm
  • brain storm
  • envisage

Custom analyzers

In this section, I show code snippets that define custom analyzers that can be used for matching synonyms. Later on in this blog I show how to submit the analyzers to Elasticsearch.

The blog called The same, but different: Boosting the power of Elasticsearch with synonyms goes into details on the difference between index-time and search-time synonyms. In the solution presented here, I make use of search-time synonyms.

We will create a synonym graph token filter that matches multi-word synonyms and will be called “my_graph_synonyms” as follows:

        "filter": {
          "my_graph_synonyms": {
            "type": "synonym_graph",
            "synonyms": [
              "mind, brain",
              "brain storm, brainstorm, envisage"
            ]
          }
        }

Next we need to define two separate custom analyzers, one that will be applied to text at index-time, and another that will be applied to text at search-time.

We define an analyzer called “my_index_time_analyzer” which uses the standard tokenizer and the lowercase token filter and the stemmer token filter as follows:

      "my_index_time_analyzer": {
        "tokenizer": "standard",
        "filter": [
          "lowercase",
          "stemmer"
        ]
      }

We define an analyzer called “my_search_time_analyzer”, which also makes use of the standard tokenizer and the lowercase token filter and the stemmer token filter (as above). However, this also includes our custom token filter called “my_graph_synonyms”, which ensures that synonyms will be matched at search-time:

      "my_search_time_analyzer": {
        "tokenizer": "standard",
        "filter": [
          "lowercase",
          "stemmer",
          "my_graph_synonyms"
        ]
      }

Mappings

Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. Each document is a collection of fields, which each have their own data type. In this example we define the mapping for a document with a single field called “my_new_text_field”, which we define as “text”. This field will make use of “my_index_time_analyzer” when documents are indexed, and will make use of “my_search_time_analyzer” when documents are searched. The mapping looks as follows:

  "mappings": {
    "properties": {
      "my_new_text_field": {
        "type": "text",
        "analyzer": "my_index_time_analyzer",
        "search_analyzer": "my_search_time_analyzer"
      }
    }
  }

Bringing it together

Below we bring together our custom analyzers and mappings and apply it to an index called “test_index” as follows:

PUT /test_index
{
  "settings": {
    "index": {
      "analysis": {
        "filter": {
          "my_graph_synonyms": {
            "type": "synonym_graph",
            "synonyms": [
              "mind, brain",
              "brain storm, brainstorm, envisage"
            ]
          }
        },
        "analyzer": {
          "my_index_time_analyzer": {
            "tokenizer": "standard",
            "filter": [
              "lowercase",
              "stemmer"
            ]
          },
          "my_search_time_analyzer": {
            "tokenizer": "standard",
            "filter": [
              "lowercase",
              "stemmer",
              "my_graph_synonyms"
            ]
          }
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "my_new_text_field": {
        "type": "text",
        "analyzer": "my_index_time_analyzer",
        "search_analyzer": "my_search_time_analyzer"
      }
    }
  }
}

Testing our custom search-time analyzer

If we wish to see how an analyzer is tokenizing and normalizing a given string, we can directly call the _analyze api as follows:

POST test_index/_analyze
{
  "text" : "Brainstorm",
  "analyzer": "my_search_time_analyzer"
}

Testing on real documents

We can use the _bulk API to drive several documents into Elasticsearch as follows:

POST test_index/_bulk
{ "index" : { "_id" : "1" } }
{"my_new_text_field": "This is a brainstorm" }
{ "index" : { "_id" : "2" } }
{"my_new_text_field": "A different brain storm" }
{ "index" : { "_id" : "3" } }
{"my_new_text_field": "About brainstorming" }
{ "index" : { "_id" : "4" } }
{"my_new_text_field": "I had a storm in my brain" }
{ "index" : { "_id" : "5" } }
{"my_new_text_field": "I envisaged something like that" }

After driving the sample documents into “test_index”, we can execute a search that will correctly respond with document #1, #2, #3 and #5, as follows:

GET test_index/_search
{
  "query": {
    "match": {
      "my_new_text_field": "brain storm"
    }
  }
}

We can execute the following search which correctly returns only documents #2 and #4, as follows:

GET test_index/_search
{
  "query": {
    "match": {
      "my_new_text_field": "brain"
    }
  }
}

We can execute the following search which will correctly respond with document #1, #2, #3 and #5, as follows:

GET test_index/_search
{
  "query": {
    "match": {
      "my_new_text_field": "brainstorming"
    }
  }
}

We can execute the following search which correctly returns documents #2 and #4, as follows:

GET test_index/_search
{
  "query": {
    "match": {
      "my_new_text_field": "mind storm"
    }
  }
}

And finally, we can execute the following search which correctly returns only documents #2 and #4 as follows:

GET test_index/_search
{
  "query": {
    "match": {
      "my_new_text_field": {
        "query": "storm brain"
      }
    }
  }
}

Conclusion

In this blog I demonstrated how you can combine stemmers and multi-word synonyms in Elasticsearch to improve the quality of your search results.

Driving Filebeat data into separate indices (uses legacy index templates)

Introduction

When driving data into Elasticsearch from Filebeat, the default behaviour is for all data to be sent into the same destination index regardless of the source of the data. This may not always be desirable since data from different sources may have different access requirements , different retention policies, or different ingest processing requirements.

In this post, we’ll use Filebeat to send data from separate sources into multiple indices, and then we’ll use index lifecycle management (ILM), legacy index templates, and a custom ingest pipeline to further control that data.

If you are interested in using the newer composable templates and data streams to control your Filebeat data, see the blog How to manage Elasticsearch data across multiple indices with Filebeat, ILM, and data streams.

Driving different data types into different destinations

You may have several different types of data that may be collected by filebeat, such as “source1”, “source2”, etc. As these may have different access/security requirements and/or different retention requirements it may be useful to drive data from different sources into different filebeat indices.

Keep in mind that splitting the filebeat data into different destinations adds complexity to the deployment. For many customers, the default behaviour of driving all filebeat data into a single destination pattern is acceptable and does not require the custom configuration that we outline below.

Step 1 – Create alias(es)

Each destination “index” that we will specify in Filebeat will actually be an alias so that index lifecycle management (ILM) will work correctly.

We can create an alias that will work with the Filebeat configuration that we give later in this blog, as follows:

PUT filebeat-7.10.2-source1-000001
{
  "aliases": {
    "filebeat-7.10.2-source1": {
      "is_write_index": true
    }
  }
}

You would want to do the same for other data sources (eg. source2, etc).

In the above alias, by naming the index filebeat-7.10.2-source1, which includes the version number after the word filebeat, we ensure that the default template that is pushed into the cluster by filebeat will be applied to the index.

Step 2 – Define an ILM policy

You should define the index lifecycle management policy (see this link for instructions).

A single policy can be used by multiple indices, or you can define a new policy for each index. In the next section, I assume that you have created a policy called “filebeat-policy”.

Step 3 – Ensure that ILM will use the correct rollover alias

In order for ILM to automate rolling over of indices, we define the rollover alias that will be used. This can be done by creating a high-order template (that overwrites the lower-order templates) so that each unique data type will have a unique rollover alias. For example, the following template can be used to ensure that the source1 data rolls over correctly:

PUT _template/filebeat-7.10.2-source1-ilm
{
  "order": 50,
  "index_patterns": [
    "filebeat-7.10.2-source1-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "filebeat-policy",
        "rollover_alias": "filebeat-7.10.2-source1"
      }
    }
  }
}

Step 4 – (Optional) Define a custom ingest pipeline

Below is an example of a very simple ingest pipeline that we can use to modify documents that are ingested into Elasticsearch. As we will see in the next section, this can selectively be applied to data from different sources depending on the destination index name.

If the ingest pipeline has a failure in it, then the document that triggered the failure is rejected. This is likely undesirable, and may be enhanced by including on_failure error handling into the pipeline code as shown below:

PUT _ingest/pipeline/my_custom_pipeline
{
  "description": "Put your custom ingest pipeline code here",
  "processors": [
    {
      "set": {
        "field": "my-new-field",
        "value": "some new value",
        "on_failure": [
          {
            "set": {
              "field": "error.message",
              "value": "my_custom_pipeline failed to execute set - {{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "Uncaught failure in my_custom_pipeline: {{ _ingest.on_failure_message }}"
      }
    }
  ]
}

Step 5 – (Optional) Execution of the custom ingest pipeline

The following template will ensure that our ingest pipeline will be executed against all documents arriving to indices that match the specified index patterns.

Be sure to modify the index patterns below to include all of the index names that should have our custom pipeline applied.

PUT _template/filebeat-template-to-call-my-custom-processors
{
  "order": 51,
  "index_patterns": [
    "filebeat-*-source1-*",
    "filebeat-*-source2-*"
  ],
  "settings": {
    "index": {
      "final_pipeline": "my_custom_pipeline"
    }
  },
  "mappings": {},
  "aliases": {}
}

Notice that we have used final_pipeline rather than default_pipeline. This is to ensure that calling our custom pipeline does not override any of the default Filebeat pipelines that may be called by various Filebeat modules.

Step 6 – Filebeat code to drive data into different destination indices

The following filebeat code can be used as an example of how to drive documents into different destination index aliases. Note that if the alias does not exist, then filebeat will create an index with the specified name rather than driving into an alias with the specified name, which is undesirable. Therefore, defining the alias as shown in Step 1 of this blog should be done before running filebeat with this configuration.

The example below would drive data into an alias (or index) called filebeat-7.10.2-source1 (assuming we are running Filebeat version 7.10.2).

filebeat.inputs:
- type: log
 enabled: true
 paths:
   - /tmp/<path to your input>.txt
 fields_under_root: true
 fields:
   data_type: "source1"


setup.template.enabled: true
setup.template.name: "filebeat-%{[agent.version]}"
setup.template.pattern: "filebeat-%{[agent.version]}-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.ilm.enabled: false # we handle ILM in the cluster, so not defined here


output.elasticsearch:
 hosts: ["localhost:9200"]
 indices:
   - index: "filebeat-%{[agent.version]}-%{[data_type]}"

In the above example, there are several setup.template settings which will ensure that the default filebeat templates are loaded correctly into the cluster if they do not already exist. See configure elasticsearch index template loading for more information.

Upgrading filebeat

Once you have implemented the above, then when upgrading to a new version of filebeat you will have to ensure that a new index alias is pointing to the correct underlying indices (re-execute step 1), and that ILM will use the correct alias (re-execute step 3). If these steps are done, then the new version of filebeat should be able to execute the same filebeat.yml that we have defined above in step 6 without modification.

Appendix 1 – code for testing the ingest pipeline

The pipeline below will send two documents into the pipeline given in Step 4. This can be used for validating that the pipeline is behaving as expected.

POST _ingest/pipeline/my_custom_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": "this is doc 1"
      }
    },
    {
      "_source": {
        "message": "this is doc 2"
      }
    }
  ]
}

Conclusion

In this post, I showed how to use Filebeat to send data from separate sources into multiple indices, and how to use index lifecycle management (ILM), legacy index templates, and a custom ingest pipeline to further control that data.

Using Kibana’s Painless Lab (Beta) to test an ingest processor script

Introduction

In several previous blog posts I have shown how a Painless script can be used to process new documents as they are ingested into an Elasticsearch cluster. In each of these posts I have made use of the simulate pipeline API to test the Painless scripts.

While developing such scripts, it may be helpful to use Painless Lab (Beta) in Kibana to debug Painless scripts. In this blog I will show how to use Painless Lab to develop and debug custom scripts, and then show how these can be then easily copied into ingest pipelines.

Example

In the blog post titled Using Elasticsearch Painless scripting to recursively iterate through JSON fields, we demonstrated how to iterate over all elements in a document, and then delete each field where the value is an empty string. The code was written as a script processor in an ingest pipeline, and then simulated.

When developing this Painless script (before putting the code into an ingest pipeline), Painless Lab can be used to catch syntax errors in real time. The code from that blog can be tested in Painless Lab as demonstrated below.

There are a few modifications to the ingest pipeline code from the previous blog to get it to execute correctly in Painless Lab.

When used in an ingest processor (which is where this will ultimately execute after it is debugged), the script will expect the “ctx” variable to contain the source of the document that is currently being processed. However, because Painless Lab does not (yet) provide a way of directly passing “ctx” to the script, this can be faked by setting “Parameters” to a JSON document with a field called “ctx” (line 2 in the above diagram on the right) that contains the “real” document as its value. We then create a variable “ctx” in the script which is set to “params.ctx” (line 17 in the above diagram on the left).

You can easily view the output by clicking on the “Output” tab as follows.

Notice that in the above “Output” the result is as expected – the “key3” has been removed as it contained an empty string. Now that we have confirmed that the script is behaving as expected, it will require some modification to get it into a state that can be used in an ingest pipeline. In the above example, line 17 and line 19 would need to be removed. You will then end up with the same script as the one that was demonstrated and verified in Using Elasticsearch Painless scripting to recursively iterate through JSON fields. It is therefore quite straightforward to copy code that has been built in Painless Lab using this technique into an ingest pipeline.

Conclusion

In this blog, I have shown how you can use the Painless Lab for debugging scripts that are used in an ingest processor. This provides real-time syntax verification, and immediate feedback on what the output document will look like.

Acknowledgemenet

Thanks to Honza Kral for pointing out the trick of setting ctx = params.ctx.