我想从ClouldTrail登录到ElasticSearch这样我们就可以看到什么是我们的AWS回事占比较好。

I am trying to get logs from ClouldTrail into ElasticSearch so that we can see what is going on in our AWS account better.

我已经建立了两个Logstash和ElasticSearch我的机器（Ubuntu的14.04）上，并且可以将文本从标准输入来ElasticSearch。然而，当我尝试使用S3输入没有被添加到ElasticSearch。

I have set up both Logstash and ElasticSearch on my machine (Ubuntu 14.04), and can push text from stdin to ElasticSearch. However when I try to use the S3 input nothing is added to ElasticSearch.

下面是即时通讯使用conf文件，我已经打消了我的亚马逊键

Here is the conf file Im using, I have removed my amazon keys

input {
  s3 {
    bucket => 'ko-cloudtrail-log-bucket'
    secret_access_key => ''
    access_key_id => ''
    delete => false
    interval => '60'
    region => 'eu-west-1'
    type => 'CloudTrail'
    codec => cloudtrail {}
    }
}

output {
    stdout {}
    elasticsearch {
            host => '127.0.0.1'
     }
}

我安装 logstash- codeC- cloudtrail codeC但文档是pretty的稀疏。

I have install the logstash-codec-cloudtrail codec but the documentation is pretty sparse.

我与-v并没有什么运行Logstash即使在我的终端没有错误打印到标准输出。有我丢失的东西？

I get no errors in my terminal even when running Logstash with -v and nothing is printed to stdout. Is there something I am missing?

推荐答案

下面是我的cloudtrail输入。它有一个轻微的发出─它重复记录的伟大工程。由于 preFIX 表示，我把cloudtrail日志在S3：// bucketname / cloudtrail，不是根

Here's my cloudtrail input. It works great with one minor issue- it duplicates records. As the prefix indicates, I put the cloudtrail logs at s3://bucketname/cloudtrail, not the root.

的突变是可选的即可。该EventSource的突变是使日志更易读一点，和红宝石ingest_time给了我一个日期记录在ELK-出现了，否则，它只有事件的时间。最后，我放弃，只是增加了噪音，我的系统中一个非常普遍的纪录。

The mutations are optional. The eventSource mutation is to make the logs a little more readable, and the ruby ingest_time gives me a date the record showed up in ELK- otherwise, it only has the time of the event. Finally, I drop a very common record that just adds noise to my system.

input {
  s3 {
    bucket => "bucketname"
    delete => false
    interval => 60 # seconds
    prefix => "cloudtrail/"
    type => "cloudtrail"
    codec => "cloudtrail"
    credentials => "/etc/logstash/s3_credentials.ini"
    sincedb_path => "/opt/logstash_cloudtrail/sincedb"
  }
}

filter {
  if [type] == "cloudtrail" {
    mutate {
      gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
      add_field => {
        "document_id" => "%{eventID}"
      }
    }
    if ! [ingest_time] {
      ruby {
        code => "event['ingest_time'] = Time.now.utc.strftime '%FT%TZ'"
      }
    }


    if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "secret_username" {
      drop {}
    }
  }
}

该credentials.ini格式的 S3输入页面解释;它只是这样的：

The credentials.ini format is explained on the s3 input page; it's just this:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=