====== La suite Elastic : Beat ElasticSearch Logstash Kibana (BELK) ====== Documentation fonctionnelle mais non terminée ====== Projet ELK : La Suite Elastic ====== Elastic Stack : https://www.elastic.co/fr/blog/elastic-stack-5-0-0-released Retour d'expérience : * http://blog.d2-si.fr/2016/08/23/bonnes-pratiques-elastic-stack/ * INSA : http://capitoul.org/ProgrammeReunion20170622?action=AttachFile&do=view&target=ELK_rsyslog_kafka_pres-capitoul-20170622.pdf Retour d'installation : http://www.alasta.com/bigdata/2016/05/05/elasticstack-alpha-decouverte.html (2016) - http://magieweb.org/2017/04/tutoriel-mise-en-place-dun-serveur-de-monitoring-avec-elastic-stack-elk/ (2017) - http://blog.kinokocorp.com/?p=191 (2017 - Centos7 ) ===== Partie 1 : Le Besoin ===== * Recensement * Indexation * Correlation * Actions * Securité des données * Service en continue * Authentification * Sauvegarde * Alerte XMPP ? https://elastalert.readthedocs.io/en/latest/ ===== Partie 2 : Les outils ===== * Tableau avec les outils gratuit et payant : https://www.elastic.co/fr/subscriptions#request-info * Les gratuits : * **ElasticSearch** : Recherchez, analysez et stockez vos données https://www.elastic.co/products/elasticsearch * **Logstash** : Intégrer les données https://www.elastic.co/fr/products/logstash * pipeline qui ingère et traite simultanément des données provenant d'une multitude de sources, puis les transforme. On préférera la solution Beats * **Kibana** : Visualisez los données https://www.elastic.co/fr/products/kibana * **Beats ** : Intégrer les données https://www.elastic.co/fr/products/beats * Filebeat : log fichier * Metricbeat : indicateur * Packetbeat : données réseau * Winlogbeat : logs windows * Heartbeat : heartbeat * X-pack : Search Profiler * X-pack : Monitoring * Palier au module LDAP uniquement payant, faire une authentification apache/nginx : https://mapr.com/blog/how-secure-elasticsearch-and-kibana/ ===== Tools ===== ==== LibBeats ==== * Beats de la communautée : https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html * systemd.journald / http / apache / mysql /ping / openconfig / nagios * Exemple de journald : https://github.com/mheese/journalbeat ==== Kafka ==== * Kafka en complément d’ElasticSearch, afin de faire tampon entre ElasticSearch et ceux qui envoient les messages. Cela permet par exemple d’arrêter ElasticSearch le temps d’une mise à jour, Kafka se chargeant de stocker les messages et de les transmettre une fois que le serveur ElasticSearch est à nouveau disponible * https://kafka.apache.org/ * Kafka-manager : Afin d'avoir une interface web de gestion du cluster ==== elasticsearch-HQ ==== * elasticsearch-HQ est un outil web permettant l'administration d'un cluster ElasticSearch. Il permet de voir l'état des nœuds, voir les différents documents. * https://github.com/royrusso/elasticsearch-HQ ===== Architecture ===== * kafka1.domaine.fr * kafka2.domaine.fr * elasticstack.domaine.fr elasticsearch.domaine.fr kibana.domaine.fr ( même machine ) * clientweb1.domaine.fr ( filebeat ) * clientdns1.domaine.fr ( logstash ) ===== Installation ===== ==== Pré-requis ==== * 4vcpu; 6Go * selinux : disable * firewalld : disable === openJDK === * Il faut installer openjdk sur les noeuds elasticsearch et les noeuds kafkayum install java-1.8.0-openjdk === Clés RPM & folder === * rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch mkdir -p /local/rpm cd /local/rpm ---- ==== ELASTICSEARCH ==== * https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.2.rpm rpm --install elasticsearch-5.2.1.rpm === Configuration === * (https://www.elastic.co/guide/en/elasticsearch/reference/5.5/important-settings.html ) * vi /etc/elasticsearch/elasticsearch.yml cluster.name: cluster-test node.name: ${HOSTNAME} bootstrap.memory_lock: true path.data: /local/elasticsearch/data path.logs: /local/elasticsearch/logs network.host: localhost http.port: 9200 #memory_lock = Désactiver le swap pour Elasticsearch : ( pour la gestion java des gros traitement) * vi /usr/lib/systemd/system/elasticsearch.service #Décommenter la ligne suivante : LimitMEMLOCK=infinity #Supprimer l'option --quiet du paramètre ExecStart pour voir les évènements elasticsearch dans journalctl : --quiet * vi /etc/sysconfig/elasticsearch #Décommente la ligne suivante : MAX_LOCKED_MEMORY=unlimited === Start : === * systemctl daemon-reload systemctl enable elasticsearch systemctl start elasticsearch === Check === * netstat -ltpn tcp6 0 0 127.0.0.1:9200 :::* LISTEN 2344/java curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty' nodes :{......} curl -XGET 'localhost:9200/?pretty' { "name" : "8Y5O47R", "cluster_name" : "elasticsearch", "cluster_uuid" : "2tt8eL_2TKuUsHVzflH6xQ", "version" : { "number" : "5.5.2", "build_hash" : "b2f0c09", "build_date" : "2017-08-14T12:33:14.154Z", "build_snapshot" : false, "lucene_version" : "6.6.0" }, "tagline" : "You Know, for Search" } ---- ==== Kafka ==== * mkdir /local/kafka cd /local/kafka wget http://apache.crihan.fr/dist/kafka/0.11.0.0/kafka_2.12-0.11.0.0.tgz tar -xvf kafka_2.12-0.11.0.0.tgz cd kafka_2.12-0.11.0.0 groupadd kafka useradd kafka -d "/local/kafka/" -s "/bin/sh" -g "kafka" -M === Configuration === * Import des certificats : # Conversion au format pkcs12 openssl pkcs12 -export -in /etc/pki/certs/cert.crt -inkey /etc/pki/certs/cert.key -chain -CAfile /etc/pki/certs/certCA.crt -name "elasticstack" -out elasticstack.p12 # import dans le keystore keytool -importkeystore -deststorepass hhjjkk -destkeystore server.keystore.jks -srckeystore elasticstack.p12 -srcstoretype PKCS12 # Lister le keystore: keytool -list -keystore server.keystore.jks # Autorité de certification : keytool -keystore server.truststore.jks -alias CARoot -import -file /etc/pki/certs/certCA.crt * vim config/server.properties # Ecoute du port + fix problème fqdn/certificat listeners=PLAINTEXT://:9092,SSL://:9093 advertised.host.name=kafka1.domaine.fr advertised.listeners=PLAINTEXT://kafka1.domaine.fr:9092,SSL://kafka1.domaine.fr:9093 # Replications sur les deux noeud offsets.topic.replication.factor=2 transaction.state.log.replication.factor=2 transaction.state.log.min.isr=2 default.replication.factor=2 offsets.topic.replication.factor=3 # SSL ssl.keystore.location=/local/kafka/kafka_2.12-0.11.0.0/server.keystore.jks ssl.keystore.password=hhjjkk ssl.key.password=hhjjkk ssl.truststore.location=/local/kafka/kafka_2.12-0.11.0.0/server.truststore.jks ssl.truststore.password=hhjjkk * vim config/server.properties dataDir=/tmp/zookeeper clientPort=2181 tickTime=2000 initLimit=10 syncLimit=5 server.1=kafka1.domaine.fr:2888:3888 server.2=kafka2.domaine.fr:2888:3888 === Creation des services systemd === * vim /etc/systemd/system/kafka-zookeeper.service [Unit] Description=Apache Zookeeper server (Kafka) Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple User=kafka Group=kafka Environment=JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk ExecStart=/local/kafka/kafka_2.12-0.11.0.0/bin/zookeeper-server-start.sh /local/kafka/kafka_2.12-0.11.0.0/config/zookeeper.properties ExecStop=/local/kafka/kafka_2.12-0.11.0.0/bin/zookeeper-server-stop.sh [Install] WantedBy=multi-user.target * vi /etc/systemd/system/kafka.service [Unit] Description=Apache Kafka server (broker) Documentation=http://kafka.apache.org/documentation.html Requires=network.target remote-fs.target After=network.target remote-fs.target kafka-zookeeper.service [Service] Type=simple User=kafka Group=kafka Environment=JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk ExecStart=/local/kafka/kafka_2.12-0.11.0.0/bin/kafka-server-start.sh /local/kafka/kafka_2.12-0.11.0.0/config/server.properties ExecStop=/local/kafka/kafka_2.12-0.11.0.0/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target === Start === * systemctl daemon-reload systemctl start kafka-zookeeper.service systemctl start kafka.service === Monitoring du cluster === * git clone https://github.com/yahoo/kafka-manager.git cd kafka-manager/ ./sbt clean dist cd target/universal/ unzip kafka-manager-1.3.3.13.zip cd kafka-manager-1.3.3.13 ZK_HOSTS=localhost:2181 ./bin/kafka-manager * Go http://kafka1:9000/ ---- ==== LOGSTASH ==== * wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.2.rpm rpm -ivh logstash-5.5.2.rpm === Configuration === * Import du certification dans /etc/pki/TERENA * Convertir la clé pour que logstash puisse l'utiliser : erreur : [2017-09-04T15:44:06,011][ERROR][logstash.inputs.beats ] Looks like you either have an invalid key or your private key was not in PKCS8 format. {:exception=>java.lang.IllegalArgumentException: File does not contain valid private key: /etc/pki/certs/cert.key} solution : 15:45:45 root@elasticstack:/local/rpm# openssl pkcs8 -topk8 -inform PEM -outform PEM -in /etc/pki/certs/cert.key -out /etc/pki/certs/cert.pem -nocrypt * cd /etc/logstash/conf.d/ * webtest.conf (log apache) input { kafka { bootstrap_servers => 'kafka1.domaine.fr:9092,kafka2.domaine.fr:9092' topics => ["WEB-TEST_APACHE"] auto_offset_reset => "earliest" /* pour que logstash recupère les logs manquant */ codec => json {} } } filter { grok { match => { "message" => ["%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?", "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"-\" %{NUMBER:[apache2][access][response_code]} -" ] } remove_field => "message" } mutate { add_field => { "read_timestamp" => "%{@timestamp}" } } date { match => [ "[apache2][access][time]", "dd/MMM/YYYY:H:m:s Z" ] remove_field => "[apache2][access][time]" } useragent { source => "[apache2][access][agent]" target => "[apache2][access][user_agent]" remove_field => "[apache2][access][agent]" } geoip { source => "[apache2][access][remote_ip]" target => "[apache2][access][geoip]" } } output { elasticsearch { index => "webtest-logs-%{+YYYY.MM.dd}" hosts => ["localhost:9200"] sniffing => false } stdout { codec => rubydebug } } * Tester la configuration et la syntax /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/web-test.conf -t === Start === * systemctl enable logstash systemctl start logstash ---- ==== FILEBEAT ==== * wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.2-x86_64.rpm rpm -vi filebeat-5.5.2-x86_64.rpm === Configuration === * Import du certification dans /etc/pki/TERENA * filebeat.prospectors: - input_type: log paths: - /var/log/httpd/*log document_type: apache - input_type: log paths: - /var/log/*.log ............. * Exemple vers logstash output.logstash: # The Logstash hosts hosts: ["elasticstack.domaine.fr:5443"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications ssl.certificate_authorities: ["/etc/pki/certs/certCA.crt"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" template.name: "filebeat" template.path: "filebeat.template.json" template.overwrite: false * Exemple vers kafka output.kafka: output.kafka: # initial brokers for reading cluster metadata #hosts: ["kafka1.domaine.fr:9092","kafka2.domaine.fr:9092"] hosts: ["kafka1.domaine.fr:9093","kafka2.domaine.fr:9093"] # message topic selection + partitioning topic: WEB-TEST_APACHE #topic: '%{[type]}' partition.round_robin: reachable_only: false required_acks: 1 compression: gzip max_message_bytes: 1000000 ssl.certificate_authorities: ["/etc/pki/certs/certCA.crt"] ssl.certificate: "/etc/pki/certs/cert.crt" ssl.key: "/etc/pki/certs/cert.key" === Start === * systemctl enable filebeat systemctl start filebeat ---- ==== KIBANA ==== * wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-x86_64.rpm rpm -ivh kibana-5.5.2-x86_64.rpm === Configuration === * vim /etc/kibana/kibana.yml server.port: 5601 server.host: "localhost" elasticsearch.url: "http://localhost:9200" === Start === * systemctl enable kibana systemctl start kibana === ProxyPass === * yum install httpd vim /etc/httpd/conf.d/kibana.conf ProxyPass "http://localhost:5601/" ProxyPassReverse "http://localhost:5601/" # Ajouter authentification de votre choix (htpasswd, ldap, ... ) === Utilisation === * Selectionner les indexs ( pour le faire après configuration initiale : Management > Inde Patterns ) * exemple logstash : **filebeat-*** * exemple kafka : **webtest-logs-*** * Time Filter field name : @timestamp ---- ==== ELASTICSEARCH-HQ ==== * cd /local/ git clone https://github.com/royrusso/elasticsearch-HQ.git === Configuration === * vim /etc/elasticsearch/elasticsearch.yml ..... http.cors.allow-origin: "*" #Mettre ip autorisé à faire l'admin http.cors.enabled: true * vim /etc/httpd/conf.d/proxypass.conf ProxyPass "!" Options Indexes FollowSymLinks AllowOverride None Require all granted ==== AUTRE ==== === packetbeat === Attention ! Beaucoup de CPU quand beaucoup de requêtes yum install libpcap wget https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.6.0-x86_64.rpm rpm -vi packetbeat-5.6.0-x86_64.rpm * Importer les dashboard dans kibana /usr/share/packetbeat/scripts/import_dashboards -es http://elasticsearch.domaine.fr:9200 ################# grok debugger http://grokdebug.herokuapp.com/ [[systemes:start|Retour]]