Whitepaper: OpenSearch and its log agents

1. Brief introduction: the importance of logs

In a technology-driven digital world, the logs generated by systems, applications and services have become a vital source of information.

As organizations grow and diversify their infrastructure, managing and analyzing these logs efficiently can become a daunting challenge.

It is in this scenario that OpenSearch emerges: a comprehensive solution that enables the centralization and analysis of logs from various sources. OpenSearch is a tool that supports observability: an evolution of monitoring that allows users to efficiently manage complex IT infrastructures.

In this guide we will explore how to achieve this centralization using OpenSearch’s specialized agents.

2. Fundamentals: What is OpenSearch?

OpenSearch is an open source, community-based, Apache 2.0 licensed search and analysis suite.

It is used for a wide range of needs, including real-time application monitoring, log analysis, and web site search. 

It provides a highly scalable system for quick access and response to large volumes of data. Its integrated visualization tool, OpenSearch Dashboards, facilitates data analysis by users. 

In addition, OpenSearch is powered by the Apache Lucene search library and supports a variety of search and analysis capabilities. These include k-nearest neighbor (KNN) search, SQL, anomaly detection and Machine Learning Commons. Also, crawl analysis or full text search.

3. What are logs and why is centralization important?

Logs are text files that contain chronological information about changes or events in an application. It does not matter whether there are normal behavior or errors: these files are recorded for later analysis.

It is an idea that seemed to be under control for many years and that has recently gone overboard. This was due to the increase in the number of applications and technologies such as Docker and Kubernetes.

The result? The number of logs grew – and continues to grow – exponentially.

In this context, centralizing them offers advantages such as simplified administration, early detection of problems and the possibility of performing a comprehensive analysis. Opensearch allows us to collect logs and process them so they can be queried and analyzed by machine learning tools.

For all this to happen, an agent that transforms this “raw” data into data that can be queried is needed. In this way, users obtain the information that allows them to understand what is happening in each of their applications.

4. Types of agents and their uses

OpenSearch and AWS provide specialized agents for collecting and sending logs from various sources. These are optimized for different types of data and sources. Some of them are:

• Amazon Kinesis Data Streams. Serverless streaming data service that simplifies the capture, processing and storage of data streams at any scale.
• AWS Services Logs. Allows ingesting logs produced by AWS within the OpenSearch service.
• Beats OSS. A set of agents designed to read logs from files on servers and send them to OpenSearch. It is useful for collecting logs from applications and operating systems.

5. Log collection and processing

Once configured, the agents start collecting logs and metrics according to the set configuration. Additionally, they can perform parsing and normalization to extract relevant information from the logs, such as dates, criticality levels or specific details.

In general, logs can be generated in different formats, such as single-line text, multi-line text, JSON or Syslog, among others.

6. Indexing and analysis in OpenSearch

The creation of indexes and schemas allows to define how the logs will be indexed in OpenSearch, including the field structure and the types of analysis to be applied.

Then, the logs sent by the agents are processed, indexed and stored in OpenSearch according to the defined configuration.

8. Search, visualization and alerts

Once the ingestion is done, OpenSearch, through OpenSearch Dashboard, applies filters and queries allowing users to find specific information related to the problem that is needed to be analyzed.

The integrated tools can also be used to create graphs and visualizations from the log data, or configure alerts in OpenSearch to receive notifications when certain predefined conditions are met.

9. Intelligent log extraction

A cloud infrastructure administrator for a critical application has an ally in OpenSearch.

First, because it allows users to collect and store the logs and metrics of their server instances, databases and other resources.

Then, by implementing artificial intelligence algorithms it is possible to detect out-of-range patterns, such as excessive resource usage, and trigger alerts in OpenSearch enabling automated decisions to automatically scale or adjust cloud resources.

AI and Generative AI applied on the logs stored in OpenSearch allow anomalies to be detected and problems in systems and applications to be predicted. Algorithms identify unusual patterns in logs and alert on potential problems or unauthorized changes.

AI can also predict failures by analyzing log histories, enabling preventive measures. With natural language processing, it can automatically classify incidents and even improve log understanding.

Moreover, the generative component of these models is capable of generating synthetic data for testing and analysis, which will be used to simulate critical situations in the infrastructure and evaluate how it responds.

9. Conclusions

To sum up, the combination of observability technologies with AI and generative AI increases the organization’s ability to keep IT events managed and secure.

In this context, OpenSearch creates a complete system that in addition to extracting and storing logs, allows users to make smart decisions or have a proactive response to changing situations, which is key to improving operational efficiency and business responsiveness to technical challenges.

Our team of experts is available to answer questions or expand any aspect related to OpenSearch and its log agents, learn more!