Enterprise Framework

PROBLEM:  ElasticSearch - POST JSON to Remove hits .total, .max_score, ._index, ._type, _id, ._score

I have to be honest, this one really upset and frustrated me because I couldn't find an easy solution/answer using Google or ElasticSearch Documentation on how to only return back _source from a search POST json body.  When I finally figured it out, I think it wasn't implemented correctly. 

SOLUTION using QUERYSTRING:

First, off if you finally find in it in the ElasticSearch documentation, you can use the query string to remove extra metadata fields like _id, max_score, _index, _type, etc...  Correction, you can tell it which fields to include through a querystring:

GET /_search?q=elasticsearch&filter_path=took,hits.hits._id,hits.hits._score

reference:  https://www.elastic.co/guide/en/elasticsearch/reference/5.6/common-options.html

You can see from the example that you need to set the filter_path and pass in the fields you want returned back.

HOWEVER!!!!, if you are using the ElasticSearch-JS client for Node/browser, it doesn't really tell you how to do it in the API examples or in a POST json body example.

If this is documented, I sure as heck couldn't find it.

THE POST BODY SOLUTION

I'm going to tell you how to do it and there is a bug.

This is the normal client json request

            this.client.search({
                index: index,
                type: type,
                from: (page - 1) * pageSize,
                size: pageSize,
                body: body
            })

            this.client.search({
                index: index,
                type: type,
                from: (page - 1) * pageSize,
                size: pageSize,
                body: body,
                filter_path: [
                    'hits.total,hits.max_score,hits.hits._id,hits.hits._score,hits.hits._source'                    
                ]
            })

How to get ElasticSearch to return only specific source fields.  FYI, this is called Source Filtering on elastic.co's website.

Example: 1

in the request payload, specify 

       _source: ["fieldname1", "fieldname2"]

Example:

Sample Data

POST /library/books/_bulk

{ "index": { "_id": 1 } }

{ "title": "The quick brown fox", "price": 5, "colors": ["red","green","blue"] }

{ "index": { "_id": 2 } }

{ "title": "The quick brown fox jumps over the lazy dog", "price": 6, "colors": ["blue","yellow"] }

{ "index": { "_id": 3 } }

{ "title": "The quick brown fox jumps over the quick dog", "price": 7, "colors":  ["green", "purple"] }

{ "index": { "_id": 4 } }

{ "title": "The quick brown fox jumps over the stupid dog", "price": 8, "colors": ["green","yellow"] }

{ "index": { "_id": 5 } }

{ "title": "The quick brown fox jumped and farted", "price": 9, "colors": ["yellow", "purple"] }

Search with _source - return back only title and price

GET /library/books/_search

{

    "_source": [ "title", "price" ]

}

Example: 2

Sample Data

POST /library/books/6
{ 
  "title": "The quick brown fox", 
  "price": 5, 
  "colors": ["red","green","blue"],
  "contacts": [
    {
      "id": 1,
      "name": "john doe"
    },
    {
      "id": 2,
      "name": "john doe"
    },
    {
      "id": 3,
      "name": "john doe"
    }
  ]
}

Search with _source Example

GET /_search

{

    "_source": {

        "includes": [ "title", "price" ],

        "excludes": [ "colors" ]

    },

    "query" : {

        "term" : { "user" : "kimchy" }

    }

}

Reference Documentation:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html

Official Elastic.co ElasticSearch client for NodeJS, Browser client

$ npm install elasticsearch

Resources:

• Github: https://github.com/elastic/elasticsearch-js
• Doc:  https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html

Get graphql-compose-elasticsearch

$ npm install graphql graphql-compose elasticsearch graphql-compose-elasticsearch --save

• Github:  https://github.com/nodkz/graphql-compose-elasticsearch
• Doc:  https://graphql-compose.herokuapp.com/elasticsearch/

GraphQL

http://graphql.org/

Great Video on Learning GraphQL

https://www.lynda.com/GraphQL-tutorials/Welcome/595829/642132-4.html?autoplay=true

Cheat Sheet

https://github.com/sogko/graphql-schema-language-cheat-sheet

Relay

https://facebook.github.io/relay/

Yarn install issue on Mac

After trying to install Yarn from command line:

$ brew update

$ brew install yarn

Warning: Building gcc from source: 

The bottle needs the Xcode CLT to be installed. xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun Error: gcc cannot be built with any available compilers. Install GNU's GCC brew install gcc

After researching, I need to install the latest xcode command line tools.

$ xcode-select --install

Then I was able to run Yarn install

$ brew install yarn

Problem:  Have an ElasticSearch Dynamic Template that will apply Templates and/or new properties to any new Index or Type created.  

Anytime an Index is created with a new type it should

• Have a default DynamicTemplate
• Add the properties "timestamp" and "creation_date". 

First use the following resources:

https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/default-mapping.html

That only gets you part way.  

Solution:

This was not easy to find, But what you will need to is set (Examples below):

• "index_patterns": ["*"]  /* this will match all indexes created */
• "mappings"."{type}".  This should be set to "_default_"

{ 
    "index_patterns": ["*"],
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
        "_default_": {
            "_source": {
                "enabled": true
            },
            "dynamic_templates": [
                {
                    "id_integers": {
                        "match": "*_id",
                        "match_mapping_type": "long",
                        "mapping": {
                            "type": "long"
                        }
                    }
                },
                {
                    "date_fields": {
                        "match": "*_date",
                        "match_mapping_type": "date",
                        "mapping": {
                            "type": "date"
                        }
                    }
                },
                {
                    "json_fields": {
                        "match": "*_json",
                        "match_mapping_type": "*",
                        "mapping": {
                            "type": "text"
                        }
                    }
                },
                {
                    "int_template": {
                        "match": "_*",
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "keyword"
                        }
                    }
                }                
            ],
            "properties": {
                "creation_date": {
                    "type": "date",
                    "format": "yyyy-MM-dd"
                },              
                "timestamp": {
                    "type": "date",
                    "format": "epoch_millis"
                }
            }
        }
    }
}

{ 
  "title": "The quick brown fox", 
  "price": 5, 
  "colors": ["red","green","blue"],
  "do_something":"hello world"
}

{
  "school": {
    "mappings": {
      "student": {
        "dynamic_templates": [
          {
            "id_integers": {
              "match": "*_id",
              "match_mapping_type": "long",
              "mapping": {
                "type": "long"
              }
            }
          },
          {
            "date_fields": {
              "match": "*_date",
              "match_mapping_type": "date",
              "mapping": {
                "type": "date"
              }
            }
          },
          {
            "json_fields": {
              "match": "*_json",
              "mapping": {
                "type": "text"
              }
            }
          },
          {
            "int_template": {
              "match": "_*",
              "match_mapping_type": "string",
              "mapping": {
                "type": "keyword"
              }
            }
          }
        ],
        "properties": {
          "creation_date": {
            "type": "date",
            "format": "yyyy-MM-dd"
          },
          "timestamp": {
            "type": "date",
            "format": "epoch_millis"
          }
        }
      }
    }
  }
}