A lot of very good experience here, thanks for sharing. Luckily for us (ok, not just luck), we are using ES to index Hadoop (both HDFS and YARN), and we are using MySQL Cluster as our source of truth. For HDFS, we create logging tables that store mutations to inodes and we have a program that transactionally periodically pulls mutation batches and applies to the ES. Failures means ES falls behind but we have no data loss. For YARN, however, we just want to index recent stats - what containers are being allocated right now to which apps. So, we use the streaming Event API for MySQL Cluster (native C++ api) that we push into ES. Failures mean lost data, but we get near instaneous updates in ES, which is crucial as it is used as part of an interactive service (monitor YARN applicaion progress).