Elasticsearch index management for paas style logging system

06/12/2018 - 11:00 to 11:40
Moon Lounge
long talk (40 min)

Session abstract: 

In this session, we will introduce large-scale log management system called NELO used in Naver corp and mainly discuss how to maintain elasticsearch indices for paas style logging system. Naver Corporation is an Internet content service company which operates Korea's top search engine Naver and manages global mobile services such as the mobile messenger LINE, video messenger Snow, and group communication service BAND. NELO is handling various different kinds of logs and more than 3 billions of logs are incoming every day. As backend storage and search engine, we are heavily depending on elasticsearch. Because the number of logs and variety of logs is increasing, managing indices in elasticsearch clusters are more and more complicated. In the beginning, we only created one index every day, but as scales are growing, we suffered mapping explosion issues and performance issues. By introducing index management service inside NELO, now we have resolved mapping explosion issues and supported custom type and custom retention time, etc. In this session we will explain our first and recent index model and how to resolve mapping explosion and how to support custom type. From this information, users will be able to understand difficulties of maintaining large scale elasticsearch cluster and index model for multi-tenant log management system which can cover many different kinds of logs with different mappings.