Ship Logs from CloudWatch Logs to Elasticsearch

[0x4D31]

First, I'm not sure if the following script is correct for shipping logs from CloudWatch Logs to Elasticsearch. After I run the lambda script, it logs "Handling event for CloudWatch logs", but then I get this error: "Event did not match any mappings".

shipper = require('lambda-stash');

exports.handler = function(event, context, callback) {
  var config = {
    elasticsearch: {
      host: 'something.us-west-2.es.amazonaws.com',
      index: 'logs',
      region: 'us-west-2',
      useAWS: true
    },
    mappings: [
      {
       processors: [
         'formatCloudwatchLogs',
         'shipElasticsearch' 
       ],
       elasticsearch: {
          type: 'test'
        }
      }
    ]
  };
  shipper.handler(config, event, context, callback);
};

Then I'd like to know if there is a way to ship the logs to different Elasticsearch indexes, based on their LogGroup.

[arithmetric]

Hi @0x4D31,

The mappings configuration for CloudWatch is missing this identifier: type: 'CloudWatch'.

For example, the mappings array should be:

[
  {
    type: 'CloudWatch',
    processors: [
      'formatCloudwatchLogs',
      'shipElasticsearch' 
    ],
    elasticsearch: {
      type: 'test'
    }
  }
]

Currently there is not a way to provide different indexes by log group. I think this would be a good feature to add.

[0x4D31]

Thanks Joe. Seems it works well.

Another thing which is really needed is dynamic indexes (like the Logstash's elasticsearch output plugin, which you can provide indexes like "lambdastash-%{+YYYY.MM.dd}". This is something that I need for using lambda-stash in production.

[arithmetric]

Regarding dynamic indexes, see #11.