Elastic News

• 

  Monitoring and securing a geo-dispersed data center with Elasticsearch
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  About Hill AFB

  Hill Enterprise Data Center (HEDC) provides hosting services for more than 100 information systems of the US Air Force logistics center at Hill Air Force Base (AFB), Utah. As Doug Babb, Chief Architect at Hill AFB, explains, the amount of data these logistic centers produce in their responsibility for the maintenance of aircraft and intercontinental ballistic missiles “is just beyond belief.”

  The HEDC team collects, monitors, and analyzes logs from Hill AFB’s numerous information systems — as well as its overarching supporting infrastructure — to provide National Institute of Standards and Technology (NIST)-compliant monitoring of their multi-tenant hosting platform in real time. They need to continuously innovate their own Platform as a Service (PaaS) to deliver logging and Department of Defense (DoD) compliance monitoring for the lifecycle of their hosted information systems. They accomplish all this with support from Elastic Cloud Enterprise (ECE).

  Log aggregation

  Log aggregation takes on a whole new meaning at HEDC. Through building Portable Operating Databases (PODs), their private cloud can be linked across more than a dozen separate sites, which they use to communicate with other PODs at bases around the US. This results in a lot of data transfer and aggregation. The team needed a way to correlate all their systems of record. ECE has all the backings of Elastic Cloud, but is self deployed to operate on any public or private infrastructure. This was the clear choice for HEDC in their need to connect geo-distanced PODs with remote access.

  For HEDC, it all starts with an effective ingest process. Raw data streams in through established information system partitions — each comprised of a collection of servers. Filebeat ships the data from these individual information systems through a load balancer and then to Logstash, where it receives node designation. Once Logstash has applied a translate filter through a memcache plugin, the bundled and polished data is dropped into HEDC’s corresponding Info Sys cluster within ECE. A catch-all HEDC infrastructure cluster collects raw logs to function as a backend that the team can use for systems of record in the event that they need to comprehensively correlate data together to capture a picture in time.

  And with that, their data is ready for the analytical power of Elasticsearch. HEDC can then perform all their necessary analytics, metrics findings, and compliance upkeep — displaying these reports through Kibana dashboards. But these dashboards only serve their purpose if they can be readily and securely shared across other PODs and within the DoD. That’s where feature parity and security across these multiple systems becomes a major priority.

  Seamless role-based authentication

  Authorization through role-based access control (RBAC) is an essential function for HEDC to securely grant access — from data to dashboards — to the appropriate PODs and DoD users. An included security feature within Elastic’s Gold and Platinum subscriptions (ECE comes with Platinum features), RBAC revolves around 5 constructs:

  • Secured Resource: the resource to which access is restricted (i.e., indices, documents, fields, or clusters)
  • Privilege: a specified course of action a user can execute against a selected resource
  • Permissions: a set of one or more permissions against a secured resource
  • Role: a named set of permissions
  • User: the authenticated alias

  RBAC provides the team with a consistent, cloud-shared UI that allows them to seamlessly access and corroborate data across different privileges and locations. HEDC can customize roles to be very specific. They may provide certain levels of information system access (with necessary machine learning plugins) for an engineer, application administration access, or create an access role with specific functions for the information system security manager (ISSM) — a DoD requirement.

  “[Role-based authentication] is a requirement of NIST … rather than using some kind of index or tag to split out people’s data and deliver it to them within an index, which we don’t want to do.” - Nate Benson, Cloud Hosting SME | Hill AFB

  This role-based authentication also extends into their PaaS infrastructure deployment. “We can allow our security compliance engineers and all engineers [into] infrastructure-based roles,” Cloud Hosting SME at Hill AFB Nate Benson explains. Now analysts in SecOps, operations, hyper_infra, etc. can all have different levels of access to HEDC’s vast data store.

  The Elastic benefit

  So why go through all this trouble? Why the hefty ingest routing and role designation? “We’re doing this because again and again, we see people in the DoD frustrated,” Benson explains. “They’re frustrated with paper compliance — the inefficacy of it. They’re frustrated that they get all these silos and it doesn’t mesh or talk … there’s no way to actually time sequence it well together.”

  ECE provides HEDC with a streamlined ingestion process, secure data transfer and permissions protocol, and a (compliance mandated) correlated sequence of record. Providing all of this cloud-based data transfer and access across PODs is a task that ECE is uniquely qualified to handle quickly, securely, and at scale.

  Want to learn more about how ECE helped HEDC securely monitor and secure its geo-dispersed data center? Check out their Elastic{ON} Tour Washington, D.C. presentation from October 2018.
  
  https://www.elastic.co/blog/monitoring-and-securing-a-geo-dispersed-data-center-with-elasticsearch
• 

  Equal Pay Day @ Elastic
  • sorangutan  

  1
  1
  Posts
  25
  Views

  

  Today, April 2, is Equal Pay Day in the United States. Why? The current date is a symbol of how far into 2019 American women on average must work to earn what white men earned in 2018. The date varies around the world but the headlines are unfortunately consistent. In a perfect world, Equal Pay Day would be January 1 (or better yet, go away entirely).

  While it frustrates me that these pay gaps exist, progress is being made, and Equal Pay Day is an opportunity to have a candid conversation about what pay equity looks like at Elastic.

  Two and a half years and nearly 1,000 employees ago, we took a close look at our salaries globally to ensure that we didn’t have unnatural pay gaps based on gender. At the time, it was a manual, somewhat ad hoc process. We had yet to build out our pay bands and compensation structures, to have a comprehensive “leveling” system. We dug in and found a handful of roles where we identified unexpected gaps between men and women in similar roles and we made pay adjustments to fix those issues.

  Following this review, we also looked at our recruiting process. We knew that basing offers on past pay has played a significant part in perpetuating gender pay gaps so we stopped asking for prior pay long before jurisdictions started passing mandates making it unlawful.

  Fast forward to 2019. We have nearly 1,400 employees in nearly 40 countries. Given our hyper-growth and much more structured recruiting and pay processes, we wanted to make sure that we truly understood where we stood on pay equity. So, this year, we partnered with Economists Incorporated (EI), an external consulting firm, to undertake a statistical pay equity analysis.

  Erik Mitchem, our partner at EI, provided an overview of the process and outcome:

  “The review was rigorous examining various types of compensation and grouping employees in a number of ways. Our analysis included multiple regression analysis of base salary, variable pay, and on-target earnings controlling for grade, job family, years of service, years in current job, and age at hire (as a proxy for prior work experience). The analysis was conducted by department, job family, headcount rollup, and across all employees to identify potential pay disparities that may exist.”

  Candidly, I was anxious to see our results. So where did we land? Again, Erik provided a summary:

  “We looked for unexplained differences in pay between men and women and found zero statistically significant pay disparities. For every dollar earned by men, female employees in the U.S. on average earn 99.6 cents in the same grade, job, department, and with similar experience. Globally, female employees earn 98.8 cents for every dollar earned by comparable male employees.
  
  Female employees in Engineering, a profession where gender gaps are often persistent, are paid $1.03 for every dollar earned by men in similar roles and with similar experience.”

  Are we perfect? Certainly not. Like many companies, we are on a journey with our approach to diversity, inclusion and belonging. And it is critical that the most basic of our employees’ needs, their compensation, is approached equitably. To ensure that we maintain what we’ve achieved, we will resubmit our pay recommendations after our upcoming annual compensation cycle and then again on an annual basis. Over time, we hope to include further demographic data to deepen our understanding of where other potential gaps might exist, particularly among people of color. I’m incredibly proud that we’ve kept to the path we set out on and were able to validate that doing the right thing is making a difference to pay equity across Elastic globally.

  
  
  https://www.elastic.co/blog/equal-pay-day-at-elastic
• 

  Announcing the new Elasticsearch JavaScript client (RC1)
  • sorangutan  

  1
  1
  Posts
  145
  Views

  

  This is not an April Fools' Day joke! In the past months, we have worked hard to refactor the current JavaScript client, and we couldn't be happier to announce that we now have the first release candidate of a new JavaScript client! In this blog post, we'll explain why we refactored the client and the technical decisions that have been made. We plan to release the final version of the client a few days after the final release of Elasticsearch 7.0.

  The new client will be released under the elastic npm organization, so you can install it with:

  npm install @elastic/elasticsearch
  

  Why a new Elasticsearch npm module?

  From as far back as 5.0, Elastic has released its software using the same version number across the whole stack, so if you install Elasticsearch v6.4.0, you will use Kibana v6.4.0 and so on. All of the official Elasticsearch clients follow the same release train, except for elasticsearch-js, which currently is at version v15.4.1.

  The versioning problem.

  One of the main issues with the old JavaScript client was the versioning strategy and the fact that we were shipping the entire API in a single package. Many solutions were considered, here are a few and why they weren’t a good fit:

  • Restart publishing old versions of the module and force users to use them.Needless to say, it wasn’t a good idea. There would have been a lot of cognitive overhead and dozens of issues related to version handling.
  • Start publishing version 50.x, 60.x of the module, where the major is kind of similar to the major version of Elasticsearch (50.x maps to 5.x, 60.x maps to 6.x and so on).It may have fixed the issue, but it does not feel right. The versioning scheme would have been strange to newcomers, and it feels weird to use such large version numbers.
  • Publish many modules with the major number in the module name, elasticsearch-js-6 for example.This would have worked, but it would have created a dispersion of packages on npm and the overall user experience wouldn’t have been very good.
  • Publish a new scoped package inside the npm organization, so @elastic/elasticsearch, resulting in a brand new version handling.This fixes the versioning problem, but scoped packages are harder to find, and the current elasticsearch module has many downloads every week, so people would continue to use that.

  None of those solutions would have worked well enough.

  We had an Engineering All-Hands meeting in Dublin in October 2018, and we discussed at length how to fix this. You can ask the rest of the clients team about how much I personally mulled over this!

  Then, one night, after the right amount of Guinness, I woke up with an idea. I wrote it down and went back to sleep. The following day, I presented the solution to the rest of the team, and everyone agreed that it was the winner!

  The Package Reorganization.

  Given that we couldn’t get rid of the elasticsearch module on npm because of how npm works and because it would have broken too many users, the solution we agreed to adopt is the following:

  Publish a new scoped package under the @elastic organization on npm, @elastic/elasticsearch with the new codebase and the correct version numbers. Then, the current unscoped package will directly depend on the scoped module, therefore transforming it to a “pointer” to the latest and greatest version of it.

  Note, this last step will not happen immediately to avoid confusion and help users transition from the old client to the new one.

  Why should I use the new JS client?

  Great question! There are many reasons why you should use the new client. Here are seven:

  1. The client has been rewritten from scratch using a new, modern syntax. It’s thus more intuitive and easier to use.
  2. The API has been redesigned to offer a consistent developer experience between callbacks and promises. It offers different utilities to handle complex situations, such as an extend method to build tailor-made APIs, and a child method, that allows you to have multiple instances of the client, with different options but that share a connection pool.
  3. The overall performance has been improved, including startup time and execution time. Since more and more services are moving to Serverless we thought that is important to offer the best startup performance, to help our users save time and money.
  4. All of the public APIs of Elasticsearch are supported out of the box, this means that no matter which API you are using you will not be required to use external plugins.
  5. You can handle any Elasticsearch response as a Node.js stream, in case you need to pipe it directly into an HTTP response.
  6. Automatic node resurrection by default. If a node is marked as dead, the client will try automatically to revive it.
  7. TypeScript is supported out of the box. The client offers type definitions for all its internals and request parameters. Request bodies and responses are shipped as any because there is not an official spec yet (but we are working on it). Furthermore, because they are shipped as any, you can easily override them with your own typing definitions.

  Breaking changes are coming

  If you were already using the previous version of this client — i.e. the one you used to install with npm install elasticsearch — you will encounter some breaking changes.

  Don’t panic!

  We understand that breaking changes are not ideal, but every breaking change has been carefully considered and we believe it was right to thing to do, to advance the client. Furthermore, the new codebase has been rewritten with modern JavaScript in mind, has been carefully designed to be easy to maintain, and to offer a more intuitive developer experience.

  Breaking changes

  • The minimum supported version of Node.js is v6.
  • Everything has been rewritten using ES6 classes to help users extend the defaults more easily.
  • There is no longer an integrated logger. The client now is an event emitter that emits the following events: request, response, sniff, and resurrect.
  • The code is no longer shipped with all the versions of the API, but only that of the package’s major version, This means that if you are using Elasticsearch v6, you will be required to install @elastic/elasticsearch@6, and so on.
  • The internals are completely different, so if you used to tweak them a lot, you will need to refactor your code. The public API should be almost the same.
  • No more browser support. This module is intended for Node.js only.
  • The returned value of an API call will no longer be the body, statusCode, and headers for callbacks and just the body for promises. The new result object contains the body, statusCode, headers, warnings, and meta, for both callback and promises.
    
    

    // before
    const body = await client.search({
      index: 'my-index',
      body: { foo: 'bar' }
    })
    client.search({
      index: 'my-index',
      body: { foo: 'bar' }
    }, (err, body, statusCode, headers) => {
      if (err) console.log(err)
    })
    
    // after
    const { body, statusCode, headers, warnings } = await client.search({
      index: 'my-index',
      body: { foo: 'bar' }
    })
    client.search({
      index: 'my-index',
      body: { foo: 'bar' }
    }, (err, { body, statusCode, headers, warnings }) => {
      if (err) console.log(err)
    })
        

  • Errors: there is no longer a custom error class for every HTTP status code (such as BadRequest or NotFound). There is instead a single ResponseError. Each error class has been renamed, and now each is suffixed with Error at the end.
  • Errors that have been removed: RequestTypeError, Generic, and all the status code specific errors (such as BadRequest or NotFound).
  • Errors that have been added: ConfigurationError (in case of bad configurations) and ResponseError, which contains all the data you may need to handle the specific error, such as statusCode, headers, body, and message.
  • Errors that have been renamed:
    • RequestTimeout (408 statusCode) ⇒ TimeoutError
    • ConnectionFault ⇒ ConnectionError
    • NoConnections ⇒ NoLivingConnectionsError
    • Serialization ⇒ SerializationError
    • Serialization ⇒ DeserializationError
  • You must specify the port number in the configuration. In the previous version you can specify the host and port in a variety of ways, with the new client there is only one way, via the node parameter.
  • The plugins option has been removed, if you want to extend the client now you should use the client.extend API.
    
    

    // before
    const { Client } = require('elasticsearch')
    const client = new Client({ plugins: [...] })
    
    // after
    const { Client } = require('@elastic/elasticsearch')
    const client = new Client({ ... })
    client.extend(...)
        

  • There is a clear distinction between the API related parameters and the client related configurations, the parameters ignore, headers, requestTimeout and maxRetries are no longer part of the API object, and you should specify them in a second option object.
    
    

    // before
    const body = await client.search({
      index: 'my-index',
      body: { foo: 'bar' },
      ignore: [404]
    })
    client.search({
      index: 'my-index',
      body: { foo: 'bar' },
      ignore: [404]
    }, (err, body, statusCode, headers) => {
      if (err) console.log(err)
    })
    
    // after
    const { body, statusCode, headers, warnings } = await client.search({
      index: 'my-index',
      body: { foo: 'bar' }
    }, {
      ignore: [404]
    })
    client.search({
      index: 'my-index',
      body: { foo: 'bar' }
    }, {
      ignore: [404]
    }, (err, { body, statusCode, headers, warnings }) => {
      if (err) console.log(err)
    })
        

  • The transport.request method will no longer accept the query key, but the querystring key instead (which can be a string or an object), furthermore, you need to send a bulk-like request, instead of the body key, you should use the bulkBody key. Also in this method, the client specific parameters should be passed as a second object.
    
    

    // before
    const body = await client.transport.request({
      method: 'GET',
      path: '/my-index/_search',
      body: { foo: 'bar' },
      query: { bar: 'baz' }
      ignore: [404]
    })
    client.transport.request({
      method: 'GET',
      path: '/my-index/_search',
      body: { foo: 'bar' },
      query: { bar: 'baz' }
      ignore: [404]
    }, (err, body, statusCode, headers) => {
      if (err) console.log(err)
    })
    
    // after
    const { body, statusCode, headers, warnings } = await client.transport.request({
      method: 'GET',
      path: '/my-index/_search',
      body: { foo: 'bar' },
      querystring: { bar: 'baz' }
    }, {
      ignore: [404]
    })
    client.transport.request({
      method: 'GET',
      path: '/my-index/_search',
      body: { foo: 'bar' },
      querystring: { bar: 'baz' }
    }, {
      ignore: [404]
    }, (err, { body, statusCode, headers, warnings }) => {
      if (err) console.log(err)
    })
        

  Release Candidate

  As said at the beginning of this post, the currently available client is a release candidate. This means that if you find a bug or you want to propose a new feature, please don’t hesitate!

  If you want to know more, you can open an issue in the client repository, post a question in Discuss, or ping @delvedor on Twitter. Happy coding!

  
  
  https://www.elastic.co/blog/announcing-the-new-elasticsearch-javascript-client-rc1
• 

  A Sip of Elastic RUM (Real User Monitoring)
  • sorangutan  

  1
  1
  Posts
  131
  Views

  

  Sorry if I lured you into the mood of having a sip of a wonderful cocktail made with rum and you realized that the RUM I’m talking about is not the rum you are craving. But, be assured that Elastic RUM is equally wonderful! Let’s take a sip! I do want to warn you that it will take a bit of time to go through the amount of detail I will cover in this blog!

  What is RUM?

  Elastic real user monitoring captures user interactions with the web browser and provides a detailed view of the “real user experience” of your web applications from a performance perspective. Elastic’s RUM Agent is a JavaScript Agent, which means it supports any JavaScript-based application. RUM can provide valuable insight into your applications. Some of the common benefits of RUM include:

  • RUM performance data can help you identify bottlenecks and discover how site performance issues affect your visitors’ experience
  • User agent information captured by RUM enables you to identify the browsers, devices, and platforms most used by your customers so that you can make informed optimizations to your application
  • Together with location information, individual user performance data from RUM helps you understand regional performance of your website worldwide
  • RUM provides insight and measurement for your application’s service level agreements (SLA)

  Getting started with RUM using Elastic APM

  In this blog, I will take you through the complete process of instrumenting a simple web application made of a React frontend and a Spring Boot backend, step by step. You will see how easy it is to use the RUM agent. As a bonus, you will also see how Elastic APM ties the frontend and the backend performance information together with a holistic, distributed trace view. Please see my previous blog for an overview of Elastic APM and distributed tracing if you are interested in knowing more details.

  To use Elastic APM real user monitoring, you have to have the Elastic Stack with APM server installed. You can of course download and install the latest Elastic Stack with APM server locally on your computer. However, the easiest approach would be creating an Elastic Cloud trial account and have your cluster ready in a few minutes. APM is enabled for the default I/O Optimized template. From now on, I’ll assume you have a cluster ready to go.

  Sample application

  The application we are going to instrument is a simple car database application made of a React frontend and a Spring Boot backend that provides API access to an in-memory car database. The application is purposely kept simple. The idea is to show you detailed instrumentation steps starting from zero so that you can instrument your own applications following the same steps.

  

  Create a directory called CarApp anywhere on your laptop. Then clone both the frontend and the backend application into that directory.

  git clone https://github.com/adamquan/carfront.git
  git clone https://github.com/adamquan/cardatabase.git
  

  As you can see, the application is extremely simple. There are only a couple of components in the React frontend and a few classes in the backend Spring Boot application. Build and run the application following the instructions in GitHub for both the frontend and backend. You should see something like this. You can browse, filter cars, and perform CRUD options on them.

  

  Amazed by how much information is captured by the RUM agent by default? Pay special attention to the markers like timeToFirstByte, domInteractive, domComplete and firstContentfulPaint. Mouse over the black dots to see the names. They provide you with great details about content retrieval and browser rendering of these contents. Also pay attention to all the performance data about resource loading from the browser. By just initializing your RUM agent, without any custom instrumentation, you get all these detailed performance metrics, out of the box! When there is a performance issue, these metrics enable you to easily decide whether the issue is due to slow backend services, a slow network, or simply a slow client browser. That is very impressive!

  For those of you who need a refresher, here is a quick explanation of the web performance metrics. Do keep in mind that for modern web application frameworks like React, these metrics might only represent the “static” part of the web page, due to the async nature of React. For example, dynamic contents might still be loading after domInteractive, as you will see later.

  • timeToFirstByte is the amount of time a browser waits to receive the first piece of information from the web server after requesting it. It represents a combination of network and server-side processing speed.
  • domInteractive is the time immediately before the user agent sets the current document readiness to “interactive,” which means the browser has finished parsing all of the HTML and DOM construction is complete.
  • domComplete is the time immediately before the user agent sets the current document readiness to “complete,” which means the page and all of its subresources like images have finished downloading and are ready. The loading spinner has stopped spinning.
  • firstContentfulPaint is the time the browser renders the first bit of content from the DOM. This is an important milestone for users because it provides feedback that the page is actually loading.

  Flexible custom instrumentation

  The RUM agent provides detailed instrumentation for your browser interaction out of the box, as you just saw. You can also perform custom instrumentations when needed. For example, because the React application is a single-page-application and deleting a car will not trigger a “page load,” RUM does not by default capture the performance data of deleting a car. We can use custom transactions for something like that.

  With our current release (APM Real User Monitoring JavaScript Agent 4.x), users have to manually create transactions for AJAX calls and Single-Page-Application (SPA) calls that do not trigger a page load. In some frameworks, like JSF, you have very little control over the JavaScript. So, manually creating transactions for button clicks which initiate AJAX requests is not viable. Even if the developer has direct control over the AJAX requests, instrumenting a big application would be a lot of effort. We are planning to enhance the RUM agent so that it will automatically create a transaction for these requests in case there is not an active one currently. This would make the auto-instrumentation cover a lot more of the application without developers having to programmatically add tracing logic to their applications.

  The "New Car" button in our frontend application allows you to add a new car to the database. We will instrument the code to capture the performance of adding a new car. Open the file Carlist.js in the components directory. You will see the following code:

  // Add new car
  addCar(car) {
      var transaction = apm.startTransaction("Add Car", "Car");
      var httpSpan = transaction.startSpan('Add Car', 'Car')
      apm.addTags(car);
      fetch(SERVER_URL + 'api/cars',
          {
              method: 'POST',
              headers: {
                  'Content-Type': 'application/json',
              },
              body: JSON.stringify(car)
          })
          .then(res => this.fetchCars())
          .catch(err => console.error(err))
      httpSpan.end()
      transaction.end();   
  }
  

  The code basically created a new transaction and a new span called “Add Car” of “Car” type. Then, it tagged the transaction with the car to provide contextual information. The transaction and span ended at the end of the method.

  Add a new car from the application web UI. Click on the APM UI in Kibana. You should see an “Add Car” transaction listed. Make sure you select “Car” in the “Filter by type” dropdown. By default, it displays “page-load” transactions.

  

  Click on the “Tags” tab. You will see the tags we added. Tags and logs add valuable contextual information to your APM traces.

  

  See the big picture with distributed tracing

  As a bonus point, we are going to also instrument our backend Spring Boot application so that you have a complete view of the overall transaction from the web browser all the way to the backend database, all in one view. Elastic APM distributed tracing enables you to do so.

  Configuring distributed tracing in RUM agents

  Distributed tracing is enabled by default in the RUM agent. However, it only includes requests made to the same origin. In order to include cross-origin requests you must set the distributedTracingOrigins configuration option. You will also have to set the CORS policy in the backend application, as we will discuss in the next section.

  For our application, the frontend is served from http://localhost:3000. To include requests made to http://localhost:8080, we need to add the distributedTracingOrigins configuration to our React application. This is done inside rum.js. The code is already there. Simply uncommenting the line will do.

  var apm = initApm({
    ...
    distributedTracingOrigins: ['http://localhost:8080']
  })
  

  This effectively tells the agent to add the distributed tracing HTTP header (elastic-apm-traceparent) to requests made to http://localhost:8080.

  To use the default instrumentation out of the box on the server side, you need to download the Java agent and start your application with it. Here is how I configured my Eclipse project to run the backend Spring Boot application. You will have to configure this using your own APM URL and APM token.

  -javaagent:/Users/aquan/Downloads/elastic-apm-agent-1.4.0.jar 
  -Delastic.apm.service_name=cardatabase 
  -Delastic.apm.application_packages=com.packt.cardatabase
  -Delastic.apm.server_urls=https://aba7c3d90b0b4820b05b0a9df44c096d.apm.us-central1.gcp.cloud.es.io:443 
  -Delastic.apm.secret_token=jeUWQhFtU9e5Jv836F
  

  

  For readers who are really paying attention to the timeline visualization above, you might be wondering why the “Car List” page-load transaction ends at 193 ms, which is the domInteractive time, while data is still being served from the backend. Great question! This is due to the fact that the fetch calls are async by default. The browser “thinks” it finished parsing all the HTML and DOM construction is complete at 193 ms because it loaded all the “static” HTML contents served from the web server. On the other hand, React is still loading data from the backend server asynchronously.

  Cross-origin resource sharing (CORS)

  The RUM agent is only one piece of the puzzle in a distributed trace. In order to use distributed tracing, we need to properly configure other components too. One of the things that you will normally have to configure is cross-origin resource sharing, the “notorious” CORS! This is because the frontend and the backend services are typically deployed separately. With the same-origin policy, your frontend requests from a different origin to the backend will fail without properly configured CORS. Basically, CORS is a way for the server side to check if requests coming in from a different origin are allowed. To read more about cross-origin requests and why this process is necessary, please see the MDN page on Cross-Origin Resource Sharing.

  What does that mean for us? It means two things:

  1. We must set the distributedTracingOrigins configuration option, as we have done.
  2. With that configuration, the RUM agent also sends an HTTP OPTIONS request before the real HTTP request to make sure all the headers and HTTP methods are supported and the origin is allowed. Specifically, http://localhost:8080 will receive an OPTIONS request with the following headers:
     
     

     Access-Control-Request-Headers: elastic-apm-traceparent
     Access-Control-Request-Method: [request-method]
     Origin: [request-origin]
         

     And APM server should respond to it with these headers and a 200 status code:
     
     

     Access-Control-Allow-Headers: elastic-apm-traceparent
     Access-Control-Allow-Methods: [allowed-methods]
     Access-Control-Allow-Origin: [request-origin]
         

  The MyCorsConfiguration class in our Spring Boot application does exactly that. There are different ways of configuring Spring Boot to do this, but here we are using a filter based approach. It’s configuring our server-side Spring Boot application to allow requests from any origin with any HTTP headers and any HTTP methods. You may not want to be this open with your production applications.

  @Configuration
  public class MyCorsConfiguration {
      @Bean
      public FilterRegistrationBean<CorsFilter> corsFilter() {
          UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
          CorsConfiguration config = new CorsConfiguration();
          config.setAllowCredentials(true);
          config.addAllowedOrigin("*");
          config.addAllowedHeader("*");
          config.addAllowedMethod("*");
          source.registerCorsConfiguration("/**", config);
          FilterRegistrationBean<CorsFilter> bean = new FilterRegistrationBean<CorsFilter>(new CorsFilter(source));
          bean.setOrder(0);
          return bean;
      }
  }
  

  Let’s close the blog with one more powerful feature of the RUM agent: source maps. Source maps make debugging errors with your application much easier by telling you exactly where the error happened in your original source code, instead of the cryptic minified code.

  Easy debugging with source maps

  It is a common practice to minify JavaScript bundles for production deployment, for performance reasons. However, debugging minified code is inherently difficult. The screenshot below shows an error message captured by the RUM agent for a production build. As you can see, the exception stack trace does not make much sense, because it’s minified code. All the error lines show javascript files like 2.5e9f7401.chunk.js, and it always points to “line 1” because of the way minification was done. Wouldn’t it be nice if you could see exactly your source code as you developed it here?

  

  Summary

  Hopefully this blog made it clear that instrumenting your applications with Elastic RUM is simple and easy, yet extremely powerful. Together with other APM agents for backend services, RUM gives you a holistic view of application performance from an end user perspective through distributed tracing.

  Once again, to get started with Elastic APM, you can download Elastic APM server to run it locally, or create an Elastic Cloud trial account and have a cluster ready in a few minutes.

  As always, reach out on the Elastic APM forum if you want to open up a discussion or have any questions. Happy RUMing!

  
  
  https://www.elastic.co/blog/performing-real-user-monitoring-rum-with-elastic-apm
• 

  This Week in Elasticsearch and Apache Lucene - 2019-03-29
  • sorangutan  

  1
  1
  Posts
  89
  Views

  

  ### Elasticsearch #### 6.7.0 released We [released Elastic Stack 6.7.0](https://www.elastic.co/blog/elastic-stack-6-7-0-released) in which we made [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/index-lifecycle-management.html), [CCR](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/xpack-ccr.html) and [SQL](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/xpack-sql.html) GA features. You can read about these features and all the other awesome things that went into Elasticsearch 6.7.0 in the [release blog post](https://www.elastic.co/blog/elasticsearch-6-7-0-released). [This blog post on CCR](https://www.elastic.co/blog/follow-the-leader-an-introduction-to-cross-cluster-replication-in-elasticsearch) is also worth checking out for an overview of the feature. #### Snapshot Lifecycle Management Snapshot Lifecycle Management is a lot like Index Lifecycle Management, but instead of managing indices, it manages Snapshots. This past week we have been busy discussing the scope of the work and working on a branch so that that can now actually take snapshots! Follow the progress along at [#40383](https://github.com/elastic/elasticsearch/pull/40383) #### Ingest Node Lookup Processor We have spent time this past week developing various prototype implementations for the ingest node lookup processor. We are narrowing down the technical design and then we will re-visit some of the functional conversations and officially kick off the development. #### Snapshot and Restore UI There is a [PR open for creating and editing repositories](https://github.com/elastic/kibana/pull/34020), and another to [refactor our file organisation and approach towards TypeScript](https://github.com/elastic/kibana/pull/33690). ![Snapshot and Restore UI][1] #### Packaging For the 6.7 and 7.0 releases of Elasticsearch, [we will use JDK 12 as the base JDK in the Docker images, and the bundled JDK in our distributions (7.0 only).](https://github.com/elastic/elasticsearch/pull/40229). Additionally the 7.0 docker images will now use the [bundled JDK](https://github.com/elastic/elasticsearch/pull/40238). #### Performance We published a [blog post on the real memory circuit breaker](https://www.elastic.co/blog/improving-node-resiliency-with-the-real-memory-circuit-breaker) in Elasticsearch 7.0.0. It is definitely worth checking out to understand more about this alternative that will help avoid OutOfMemoryErrors. #### Templated Role Mappings We merged a change that [allows role mappings to use Mustache templates](https://github.com/elastic/elasticsearch/pull/39984). This is intended to support some edge cases for mapping from SAML attributes to Elasticsearch roles, and also situations where users are authenticated by a builtin realm (such as Active Directory) but their roles need to be defined in a custom provider. #### Consistent Secure Settings Elasticsearch has a way to store settings that need to have the same value across the whole cluster (cluster settings) and a way to store settings that are secret and need to be stored securely (keystore settings) but we don't currently have a way to get *both* of those properties. This feature is important for things like local encrypytion keys where all nodes need to be encrypting data using the same set of keys. We are working on [Consistent Secure Settings](https://github.com/elastic/elasticsearch/pull/40416) that validate that, for specific keystore settings, all nodes have been configured with an identical value for that setting. #### Snapshot resiliency Snapshot deletions can leave orphaned files behind in case where the master fails during the deletion process. We are looking for ways of making this process more resilient as well as generally speeding up the deletion process, which currently triggers the file deletions in a sequential way. One step is moving the [repository-internal deletion APIs to an async interface](https://github.com/elastic/elasticsearch/pull/40144), which will allow some sections of the snapshot deletion logic to be executed concurrently by multiple threads. Another change here is [adding support](https://github.com/elastic/elasticsearch/pull/40322) for [S3's bulk deletion API](https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html), which will dramatically reduce the number of S3 API requests for deletes and has the potential to massively speed up deleting large snapshots from S3. #### Replicated closed indices We investigated how many replicated closed indices can be allocated on a small node instance with 1GB heap and explored ways to reduce the memory overhead. As replicated closed indices can't be indexed into or searched, we can [avoid creating a mapper service](https://github.com/elastic/elasticsearch/pull/40423) and skip the initialization of the caching infrastructure that preallocates a fixed amount of memory per index. #### Search As You Type We have merged a new field type called [search_as_you_type](https://www.elastic.co/guide/en/elasticsearch/reference/master/search-as-you-type.html). It uses the inverted lists to provide an infix document based suggester. Coupled with the new ability to skip non-competitive documents during search this new field type is a good alternative to the completion suggester: * Any fields in the mapping can be used to filter the result so no need to add specific contexts in the suggest inputs. * The scoring of documents can be modified at query time using the ranking and boosts DSL like other regular queries. We also added a new variant for the match query called[ `match_bool_prefix`](https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-match-bool-prefix-query.html) that is particularly suited for this new field type. The match_bool_prefix query analyzes its input and constructs a bool query from the terms except that the last term is considered as a prefix. #### Script Score query We added [random score](https://github.com/elastic/elasticsearch/pull/40186) function to the script_score query. This PR is the last piece of work to make `script_score` feature in parity with `function_score`. #### Rounding in SQL We have [rewritten](https://github.com/elastic/elasticsearch/pull/40242) the ROUND and TRUNCATE functions in SQL. A particular characteristic of these two functions is that one of their parameters is optional. And the way this parameter was handled in the constructor before the rewrite, made SQL not able to correctly compare two ROUND functions inside the same SQL query (think of SELECT ROUND(salary, 2) FROM emp GROUP BY ROUND(salary, 2) where each ROUND "instance" in the query actually means two separate not "equal" Java instances). Because of this bug the sample query above would have triggered an exception. ### Lucene #### Bugfixes A bug in the bridge code between ValueSource and DoubleValuesSource was [fixed](https://issues.apache.org/jira/browse/LUCENE-8741) An intervals issue is being investigated where an unordered search without overlaps can miss hits but hasn't there is no issue yet. We found a pretty serious issue where FilteredDirectory didn't delegate pending deletes correctly causing corrupted indices in certain situations. So far we haven't seen anything in the wild but only in test scenarios. The same [issue](https://issues.apache.org/jira/browse/LUCENE-8734) was also chased but this fix was simpler. #### Enhancements The constant score query can [now](https://issues.apache.org/jira/browse/LUCENE-8732) early terminate queries if the minimum score is greater than the constant score and the number of hits that match the query is not tracked We [continued](https://issues.apache.org/jira/browse/LUCENE-8701) to work on improving the block join query (nested query) to add early termination when the score of children is ignored and to fix a regression when scores are requested. Lucene can now [automatically](https://issues.apache.org/jira/browse/LUCENE-8477) pull up disjunctions in intervals queries where the internal gaps of an interval are important (ie in phrase queries or queries wrapped with a MAXGAPS source). Take the disjunction OR(ORDERED("a", "b"), "a"); given the document string `a b c`, this will only return the interval 'a', as 'a -> b' gets minimised away; which further means that BLOCK(OR(ORDERED("a", "b"), "a"), "c") won't match the document, because the resulting interval 'a -> c' contains a gap. The way round this is to rewrite to OR(BLOCK("a", "b", "c"), BLOCK("a", "c")), which is now done automatically. Because this can end up being less efficient (intervals for "c" get pulled twice in the rewritten source here), there is an option to prevent a disjunction from being rewritten. We are currently [working](https://issues.apache.org/jira/browse/LUCENE-8730) on a small change to WordDelimiiterGraphFilter that will make removing the deprecated WordDelimiterFilter easier ### Changes #### Changes in Elasticsearch Changes in 7.1: * Updates max dimensions for sparse_vector and dense_vector to 1024. [#40597](https://github.com/elastic/elasticsearch/pull/40597) * Add start and stop time to cat recovery API [#40378](https://github.com/elastic/elasticsearch/pull/40378) * Add randomScore function in script_score query [#40186](https://github.com/elastic/elasticsearch/pull/40186) * Move top-level pipeline aggs out of QuerySearchResult [#40319](https://github.com/elastic/elasticsearch/pull/40319) * SQL: Polish behavior of SYS TABLES command [#40535](https://github.com/elastic/elasticsearch/pull/40535) * Improve error message for absence of indices [#39789](https://github.com/elastic/elasticsearch/pull/39789) * Get node ID from nodes info in REST tests [#40052](https://github.com/elastic/elasticsearch/pull/40052) * search as you type fieldmapper [#35600](https://github.com/elastic/elasticsearch/pull/35600) * ignore 409 conflict in reindex responses [#39543](https://github.com/elastic/elasticsearch/pull/39543) * Fix an off-by-one error in the vector field dimension limit. [#40489](https://github.com/elastic/elasticsearch/pull/40489) * No mapper service and index caches for replicated closed indices [#40423](https://github.com/elastic/elasticsearch/pull/40423) * SQL: Adjust the precision and scale for drivers [#40467](https://github.com/elastic/elasticsearch/pull/40467) * Remove String interning from `o.e.index.Index`. [#40350](https://github.com/elastic/elasticsearch/pull/40350) * [ML] Data Frame HLRC Get API [#40209](https://github.com/elastic/elasticsearch/pull/40209) * Adding a soft limit to the field name length. Closes #33651 [#40309](https://github.com/elastic/elasticsearch/pull/40309) * Support mustache templates in role mappings [#39984](https://github.com/elastic/elasticsearch/pull/39984) * Resolve JAVA_HOME at windows service install time [#39714](https://github.com/elastic/elasticsearch/pull/39714) * SQL: Polish parsing of CAST expression [#40428](https://github.com/elastic/elasticsearch/pull/40428) * [ML] Data Frame HLRC Get Stats API [#40327](https://github.com/elastic/elasticsearch/pull/40327) * SQL: Fix classpath discovery on Java 10+ [#40420](https://github.com/elastic/elasticsearch/pull/40420) * SQL: Spec tests now use classpath discovery [#40388](https://github.com/elastic/elasticsearch/pull/40388) * Add implicit this for class binding in Painless [#40285](https://github.com/elastic/elasticsearch/pull/40285) Changes in 7.0: * Optimise rejection of out-of-range `long` values [#40325](https://github.com/elastic/elasticsearch/pull/40325) * Use default discovery implementation for single-node discovery [#40036](https://github.com/elastic/elasticsearch/pull/40036) * Deprecate types in `_graph/explore` calls. [#40466](https://github.com/elastic/elasticsearch/pull/40466) * Remove timeout task after completing cluster state publication [#40411](https://github.com/elastic/elasticsearch/pull/40411) Changes in 6.7: * Parse composite patterns using ClassicFormat.parseObject [#40100](https://github.com/elastic/elasticsearch/pull/40100) * Correct ILM metadata minimum compatibility version [#40569](https://github.com/elastic/elasticsearch/pull/40569) * Handle null retention leases in WaitForNoFollowersStep [#40477](https://github.com/elastic/elasticsearch/pull/40477) * SQL: add "fuzziness" option to QUERY and MATCH function predicates [#40529](https://github.com/elastic/elasticsearch/pull/40529) * Wrap Dockerfile yum operations in retries [#40349](https://github.com/elastic/elasticsearch/pull/40349) * Only run retention lease actions on active primary [#40386](https://github.com/elastic/elasticsearch/pull/40386) * Fix major version in 6.7 branch [#40507](https://github.com/elastic/elasticsearch/pull/40507) * Store Pending Deletions Fix [#40345](https://github.com/elastic/elasticsearch/pull/40345) * Update feature aware check ASM to 7.1 [#40389](https://github.com/elastic/elasticsearch/pull/40389) * SQL: Fix RLIKE bug and improve testing for RLIKE statement [#40354](https://github.com/elastic/elasticsearch/pull/40354) * SQL: CAST supports both SQL and ES types [#40365](https://github.com/elastic/elasticsearch/pull/40365) * Geo Point parse error fix [#40447](https://github.com/elastic/elasticsearch/pull/40447) Changes in 6.6: * SQL: Fix getTime() methods in JDBC [#40484](https://github.com/elastic/elasticsearch/pull/40484) * SQL: Fix metric aggs on date/time to not return double [#40377](https://github.com/elastic/elasticsearch/pull/40377) * SQL: Add missing handling of IP field in JDBC [#40384](https://github.com/elastic/elasticsearch/pull/40384) * SQL: JLine upgrade and polishing [#40321](https://github.com/elastic/elasticsearch/pull/40321) #### Changes in Elasticsearch Management UI Changes in 7.1: * Instrument ILM with user action telemetry [#33089](https://github.com/elastic/kibana/pull/33089) #### Changes in Elasticsearch SQL ODBC Driver Changes in 7.1: * Fail quickly if hostname starts with a HTTP(S) scheme [#138](https://github.com/elastic/elasticsearch-sql-odbc/pull/138) * Enable testing with all Kibana sample data [#137](https://github.com/elastic/elasticsearch-sql-odbc/pull/137) * Add request params for multi-value fields and timezone [#132](https://github.com/elastic/elasticsearch-sql-odbc/pull/132) Changes in 6.7: * Advertise supported current_date and current_timestamp [#136](https://github.com/elastic/elasticsearch-sql-odbc/pull/136) [1]: https://www.elastic.co/assets/blt6e3f7ff5ddb702b3/snapshot-and-restore-ui.png
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-03-29
• 

  Follow the Leader: An Introduction to Cross-Cluster Replication in Elasticsearch
  • sorangutan  

  1
  1
  Posts
  239
  Views

  

  ## The Needs of the Many The ability to natively replicate data to an Elasticsearch cluster from another Elasticsearch cluster is our most heavily requested feature, and one that our users have [long been asking](https://discuss.elastic.co/t/cross-data-center-replication/11670) for. After years of engineering effort [laying the necessary foundations](https://www.elastic.co/blog/elasticsearch-sequence-ids-6-0), [building new fundamental technology into Lucene](https://issues.apache.org/jira/browse/LUCENE-8198), and [iterating and refining our initial design](https://github.com/elastic/elasticsearch/issues/30086), we are excited to announce that cross-cluster replication (CCR) is now available and production-ready in Elasticsearch 6.7.0. In this post, the first in a series, we will provide some brief introduction to what we have implemented, and some technical background on CCR. In future posts we will deep-dive into specific CCR use cases. Cross-cluster replication in Elasticsearch enables a variety of mission-critical use cases within Elasticsearch and the Elastic Stack: * **Disaster Recovery (DR) / High Availability (HA)**: Tolerance to withstand a datacenter or region outage is a requirement for many mission-critical applications. This requirement was previously solved in Elasticsearch with additional technologies, which added additional complexity and management overhead. Satisfying cross-datacenter DR / HA requirements can now be solved natively in Elasticsearch, utilizing CCR, with no additional technologies. * **Data Locality**: Replicate data in Elasticsearch to get closer to the user or application server, reducing latencies that [cost you money](https://developers.google.com/web/fundamentals/performance/why-performance-matters/). For example, a product catalog or reference dataset may be replicated to twenty or more data centers around the world, to minimize the distance between the data and the application server. Another use case may be a stock trading firm with offices in London and New York. All trades in the London office are written locally and replicated to the New York office, and all trades in the New York office are written locally and replicated to London. Both offices have a global view for all trades. * **Centralized Reporting**: Replicate data from a large number of smaller clusters back to a centralized reporting cluster. This is useful when it may not be efficient to query across a large network. For example, a large global bank may have 100 Elasticsearch clusters around the world, each within a different bank branch. We can use CCR to replicate events from all 100 banks around the world back to a central cluster where we can analyze and aggregate events locally. Prior to Elasticsearch 6.7.0, these use cases could be partially addressed with third-party technologies, which is cumbersome, carries a lot of administration overhead, and has significant drawbacks. With cross-cluster replication natively integrated into Elasticsearch, we free our users of the burden and drawbacks of managing complicated solutions, can offer advantages above what existing solutions can provide (e.g., comprehensive error handling), and provide APIs within Elasticsearch and UIs in Kibana for managing and monitoring CCR. Stay tuned for our follow-up posts to dive into each of these use-cases in greater detail. ## Getting Started with Cross-Cluster Replication Hop on over to our [downloads page](https://www.elastic.co/downloads/) to obtain the latest releases of Elasticsearch and Kibana, and dive in with our [getting started guide](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/ccr-getting-started.html). CCR is a [platinum level feature](https://www.elastic.co/subscriptions), and is available through 30-day trial license that can be activated through the [start trial API](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/start-trial.html) or [directly from Kibana](https://www.elastic.co/guide/en/kibana/6.7/managing-licenses.html). ## A Technical Introduction to Cross-Cluster Replication [CCR is designed around an active-passive index model](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/ccr-overview.html). An index in one Elasticsearch cluster can be configured to replicate changes from an index in another Elasticsearch cluster. The index that is replicating changes is termed a “follower index” and the index being replicated from is termed the “leader index”. The follower index is passive in that it can serve read requests and searches but can not accept direct writes; only the leader index is active for direct writes. As CCR is managed at the index level, a cluster can contain both leader indices and follower indices. In this way, you can solve some active-active use cases by replicating some indices one way (e.g., from a US cluster to a European cluster), and other indices the other way (from a European cluster to a US cluster). [Replication is done at the shard level](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/ccr-overview.html#_the_mechanics_of_replication); each shard in the follower index will pull changes from its corresponding shard in the leader index, which means that a follower index has the same number of shards as its leader index. All operations are replicated by the follower, so that operations to create, update, or delete a document are replicated. Replication is done in near real-time; as soon as the [global checkpoint on a shard advances](https://www.elastic.co/blog/elasticsearch-sequence-ids-6-0) an operation is eligible to be replicated by a following shard. Operations are pulled and indexed efficiently in bulk by the following shard, and multiple requests to pull changes can be in flight concurrently. These read requests can be served by the primary and its replicas, and aside from reading from the shard, do not put any additional load on the leader. This design allows CCR to scale with your production load, so that you can continue to enjoy the high-throughput indexing rates you have come to appreciate (and expect) in Elasticsearch. CCR supports both newly-created indices and existing indices. When a follower is initially configured, [it will bootstrap itself from the leader index](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/remote-recovery.html) by copying the underlying files from the leader index in a process similar to how a replica recovers from the primary. After this recovery process completes, CCR will replicate any additional operations from the leader. Mappings and settings changes are automatically replicated as-needed from the leader index. From time to time, CCR might encounter error scenarios (e.g., a network failure). CCR is able to automatically classify these errors into recoverable errors, and fatal errors. When a recoverable error occurs, CCR enters a retry loop so that as soon as the situation that led to the failure is resolved, CCR will resume replication. Status of replication can be [monitored through a dedicated API](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/ccr-overview.html#_inspecting_the_progress_of_replication). Via this API, you can monitor how closely the follower is tracking the leader, see detailed statistics about the performance of CCR, and track any errors that require your attention. We have integrated CCR with the monitoring and management apps within Kibana. The monitoring UI gives you insight into CCR progress and error reporting. ![Elasticsearch CCR Monitoring UI in Kibana][1]

  Elasticsearch CCR Monitoring UI in Kibana

  The management UI enables you to configure remote clusters, configure follower indices, and manage auto-follower patterns for automatic index replication. ![Elasticsearch CCR Management UI in Kibana][2]

  Elasticsearch CCR Management UI in Kibana

  ## Simon Says, Follow Today’s Indices Many of our users have workloads that create new indices on a periodic basis. For example, daily indices from log files that Filebeat is pushing or indices rolled over automatically by [index lifecycle management](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/index-lifecycle-management.html). Rather than having to manually create follower indices to replicate these indices from a source cluster, we have built [auto-follow functionality](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/ccr-auto-follow.html) directly into CCR. This functionality allows you to configure patterns of indices to automatically be replicated from a source cluster. CCR will monitor source clusters for indices that match these patterns, and configure following indices to replicate those matching leader indices. We have also integrated CCR and ILM so that time-based indices can be replicated by CCR, and managed in both source and target clusters by ILM. For example, ILM understands when a leader index is being replicated by CCR and so carefully manages destructive operations like shrinking and deleting indices until CCR has finished replicating. ## Those Who Don’t Know History For CCR to be able to replicate changes, we require a history of operations on the shards of the leader index, and pointers on each shard to know what operations are safe to replicate. This history of operations is governed by sequence IDs and the pointer is known as the [global checkpoint](https://www.elastic.co/blog/elasticsearch-sequence-ids-6-0). There is a [complication](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents) though. When a document is updated or deleted in Lucene, Lucene marks a bit to record that the document is deleted. The document is retained on disk until a future merge operation merges the deleted documents away. If CCR replicates this operation before the delete is merged away, then all is well. Yet, merges occur on their own lifecycle, and this means that a deleted document could be merged away before CCR has had the opportunity to replicate the operation. Without some ability to control when deleted documents are merged away, CCR could miss operations and be unable to fully replicate the history of operations to the follower index. Early in the design of CCR we planned to use the [Elasticsearch translog](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/index-modules-translog.html) as a source for the history of these operations; this would have side-stepped the problem. We quickly realized that the translog was not designed for the access patterns that CCR needed to perform effectively. We considered putting additional data structures on top of and alongside the translog to achieve the performance that we needed, but there are limitations to this approach. For one, it would add more complexity to one of the most critical components of our system; this is simply not consistent with our engineering philosophy. Further, it would tie our hands for [future changes](https://github.com/elastic/elasticsearch/issues/1242) that we intend to build on top of the history of operations where we would be forced to either limit the types of searches that can be done on the history of operations, or reimplement all of Lucene on top of the translog. With this insight, we [realized that we needed to build natively into Lucene](https://issues.apache.org/jira/browse/LUCENE-8198) functionality that would give us control over when a deleted document is merged away, effectively pushing the history of operations into Lucene. We call this technology “soft deletes”. This investment into Lucene will pay off for years to come as not only is CCR built on it, but we are reworking our replication model on top of soft deletes, and the forthcoming [changes API](https://github.com/elastic/elasticsearch/issues/1242) will be based on them too. [Soft deletes are required to be enabled on leader indices](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/ccr-requirements.html#ccr-requirements). What remains then is for a follower to be able to influence when soft deleted documents are merged away on the leader. To this end, we introduced [shard history retention leases](https://github.com/elastic/elasticsearch/issues/37165). With a shard history retention lease, a follower can mark in the history of operations on the leader where in history that follower currently is. The leader shards know that operations below that marker are safe to be merged away, but any operations above that marker must be retained for until the follower has had the opportunity to replicate them. These markers ensure that if a follower goes offline temporarily, the leader will retain operations that have not yet been replicated. Since retaining this history requires additional storage on the leader, these markers [are only valid for a limited period](https://www.elastic.co/guide/en/elastic-stack-overview/6.7/ccr-requirements.html#ccr-overview-soft-deletes) after which the marker will expire and the leader shards will be free to merge away history. You can tune the length of this period based on how much additional storage you are willing to retain in case a follower goes offline, and how long you’re willing to accept a follower being offline before it would otherwise have to be re-bootstrapped from the leader. ## Summary We are delighted for you to try out CCR and share with us your feedback on the functionality. We hope you enjoy this functionality as much as we enjoyed building it. Stay tuned for future posts in this series where we will roll up our sleeves and explain in more detail some of the functionality in CCR and the use-cases that CCR is aimed at. And if you have any CCR questions, reach out on the [Discuss](https://discuss.elastic.co/c/elasticsearch) forum. ---------- The thumbnail image associated with this post is copyright by NASA, and is licensed under the CC BY-NC 2.0 license. The banner image associated with this post is copyright by Rawpixel Ltd, is licensed under the CC BY 2.0 license, and is cropped from the original [1]: https://www.elastic.co/assets/blt5f4199278fe0732e/ccr-monitoring.png [2]: https://www.elastic.co/assets/blt52c5ee3f73f080a1/ccr-management.png
  
  https://www.elastic.co/blog/follow-the-leader-an-introduction-to-cross-cluster-replication-in-elasticsearch
• 

  Creating frozen indices with the Elasticsearch Freeze index API
  • sorangutan  

  1
  1
  Posts
  285
  Views

  

  First, some context

  Hot-Warm architectures are often used when we want to get the most out of our hardware. It is particularly useful when we have time-based data, like logs, metrics, and APM data. Most of these setups rely on the fact that this data is read-only (after ingest) and that indices can be time(or size)-based. So they can be easily deleted based on our desired retention period. In this architecture, we categorize Elasticsearch nodes into two types: 'hot' and 'warm'.

  Hot nodes hold the most recent data and thus handle all indexing load. Since recent data is usually the most frequently queried, these nodes will be the most powerful in our cluster: fast storage, high memory and CPU. But that extra power gets expensive, so it doesn’t make sense to store older data that isn’t queried as often on a hot node.

  On the other hand, warm nodes will be the ones dedicated to long-term storage in a more cost-efficient way. Data on the warm nodes is not as likely to be queried often and data within the cluster will move from hot to warm nodes based on our planned retention (achieved through shard allocation filtering), while still being available online for queries.

  Starting with Elastic Stack 6.3, we've been building new features to enhance hot-warm architectures and simplify working with time-based data.

  

  Data rollups were first introduced in version 6.3, to save storage. In time-series data, we want fine-grained detail for the most recent data. But it is very unlikely that we will need the same for historical data, where we will typically look at datasets as a whole. And this is where rollups came in, since starting at version 6.5 we can create, manage and visualize rollup data in Kibana.

  Shortly after, we added source-only snapshots. This minimal snapshots will provide a significant reduction of snapshot's storage, with the tradeoff of having to reindex data if we want to restore and query. This has been available since version 6.5.

  In version 6.6, we released two powerful features, Index Lifecycle Management (ILM) and Frozen Indices.

  ILM provides the means to automate your indices management over time. It simplifies moving indices from hot to warm, allows deletion when indices are too old, or automates force merging indices down to one segment.

  And for the rest of this blog, we’ll talk about frozen indices.

  Why freeze an index?

  One of the biggest pain points with “old” data is that, regardless of age, indices still have a significant memory footprint. Even if we place them on cold nodes, they still use heap.

  A possible solution could be to close the index. If we close an index, it won’t require memory, but we will need to re-open it to run a search. Reopening indices will incur an operational cost and also require the heap it was using before being closed.

  On each node, there is a memory (heap) to storage ratio that will limit the amount of storage available per node. It may vary from as low as 1:8 (memory:data) for memory intensive scenarios, to something close to 1:100 for less demanding memory use cases.

  This is where frozen indices come in. What if we could have indices that are still open — keeping them searchable — but do not occupy heap? We could add more storage to data nodes that hold frozen indices, and break the 1:100 ratio, understanding the tradeoff that searches might be slower.

  When we freeze an index, it becomes read-only and its transient data structures are dropped from memory. In turn, when we run a query on frozen indices, we will have to load the data structures to memory. Searching a frozen index doesn’t have to be slow. Lucene heavily depends on the filesystem cache which might have enough capacity to retain significant portions of your index in memory. In such a case searches are comparable in speed per shard. Yet, a frozen index is still throttled such that only one frozen shard is executing per node at the same time. This aspect might slow down searches compared to unfrozen indices.

  How freezing works

  Frozen indices are searched through a dedicated, searched-throttled threadpool. This by default uses a single thread, to ensure that frozen indices are loaded into memory one at a time. If concurrent searches are happening, they will get in the queue to add additional protections to prevent nodes from running out of memory.

  So, in a hot-warm architecture, we will now be able to transition indices from hot to warm, and then be able to freeze them before archiving or deleting them, allowing us to reduce our hardware requirements.

  Before frozen indices, to reduce infrastructure cost we had to snapshot and archive our data, which adds a significant operational cost. We would have to restore data if we needed to search again. Now, we can keep our historical data available for search, without a significant memory overhead. And if we need to write again on an already froze index, we can just unfreeze it.

  

  How to freeze an Elasticsearch index

  Frozen indices are easy to implement in your cluster, so let’s get started on how to use the Freeze index API and how to search on frozen indices.

  First, we'll start by creating some sample data on a test index.

  POST /sampledata/_doc
  {
      "name": "Jane",
      "lastname": "Doe"
  }
  POST /sampledata/_doc
  {
      "name": "John",
      "lastname": "Doe"
  }
  

  And then check that our data has been ingested. This should return two hits:

  GET /sampledata/_search
  

  As a best practice, before you freeze an index it’s recommended to first run a force_merge. This will ensure that each shard has only a single segment on disk. It will also provide much better compression and simplifies the data structures we will need when running an aggregation or a sorted search request on the frozen index. Running searches on a frozen index with multiple segments can have a significant performance overhead up to multiple orders of magnitude.

  POST /sampledata/_forcemerge?max_num_segments=1
  

  The next step is to just invoke a freeze on our index via de Freeze index API endpoint.

  POST /sampledata/_freeze
  

  Searching frozen indices

  Now that it’s frozen, you’ll see that regular searches won’t work. The reason for this is that, to limit memory consumption per node, frozen indices are throttled.  Since we could target a frozen index by mistake, we’ll prevent accidental slowdowns by specifically adding ignore_throttled=false to the request.

  GET /sampledata/_search?ignore_throttled=false
  {
   "query": {
     "match": {
       "name": "jane"
     }
   }
  }
  

  Now we can check the status of our new index, by running the following request:

  GET _cat/indices/sampledata?v&h=health,status,index,pri,rep,docs.count,store.size
  

  This will return a result similar to the following, with the index status being ‘open’:

  health status index      pri rep docs.count store.size
  green  open   sampledata   5   1          2     17.8kb
  

  As mentioned above, we must protect the cluster from running out of memory, thus there is a limit in the number of frozen indices we can concurrently load for search on a node. The number of threads in the search-throttled threadpool defaults to 1, with a default queue of 100. This means that if we run more than one request, they will be queued up to a hundred. We can monitor the threadpool status, to check queues and rejections, with the following request:

  GET _cat/thread_pool/search_throttled?v&h=node_name,name,active,rejected,queue,completed&s=node_name
  

  Which should return a response similar to:

  node_name             name             active rejected queue completed
  instance-0000000000   search_throttled      0        0     0        25
  instance-0000000001   search_throttled      0        0     0        22
  instance-0000000002   search_throttled      0        0     0         0
  

  Frozen indices might be slower, but they can be pre-filtered in a very efficient manner. It is also recommended to set the request parameter pre_filter_shard_size to 1.

  GET /sampledata/_search?ignore_throttled=false&pre_filter_shard_size=1
  {
   "query": {
     "match": {
       "name": "jane"
     }
   }
  }
  

  This will not add a significant overhead to the query and will allow us to take advantage of the usual scenario. For example, when searching on a date range on time-series indices, not all shards will match.

  How to write to a frozen Elasticsearch index

  What will happen if we try to write on an already frozen index? Let’s go for it and find out.

  POST /sampledata/_doc
  {
    "name": "Janie",
    "lastname": "Doe"
  }
  

  What happened? Frozen indices are read-only, so writing is blocked. We can check these in the index settings:

  GET /sampledata/_settings?flat_settings=true
  

  Which will return:

  {
   "sampledata" : {
     "settings" : {
       "index.blocks.write" : "true",
       "index.frozen" : "true",
       ....
     }
   }
  }
  

  We have to use the Unfreeze index API, invoking the unfreeze endpoint on the index.

  POST /sampledata/_unfreeze
  

  And now we'll be able to create a third document and search for it.

  POST /sampledata/_doc
  {
   "name": "Janie",
   "lastname": "Doe"
  }
  GET /sampledata/_search
  {
   "query": {
     "match": {
       "name": "janie"
     }
   }
  }
  

  Unfreezing should be done only under exceptional situations. And remember to always run a `force_merge` before freezing the index again to ensure optimal performance.

  Using frozen indices in Kibana

  To begin with, we will need to load some sample data, like the sample flight data.

  Click on the “Add” button for Sample flight data.

  

  We should now be able to see the loaded data by clicking the “View data” button. The dashboard will be similar to this one.

  

  Now we can test freezing the index:

  POST /kibana_sample_data_flights/_forcemerge?max_num_segments=1
  POST /kibana_sample_data_flights/_freeze
  

  And if we go back to our dashboard, we’ll notice that the data has apparently “disappeared”.

  

  We need to tell Kibana to allow searches on frozen indices, which is disabled by default.

  Go to Kibana Management, choose Advanced Settings. In the Search section, you will find that “Search in frozen indices” is disabled. Toggle to enable and save the changes.

  

  And the flight's dashboard will show the data again.

  Wrapping up

  Frozen Indices are a very powerful tool in hot-warm architectures. They enable a more cost-effective solution for increased retention while retaining online search. I recommend that you test your search latency with your hardware and data, to come with the right sizing and search latency for your frozen indices.

  Check out the Elasticsearch documentation to learn more about the Freeze index API. And as always, if you have any questions, reach out on our Discuss forums. Happy freezing!

  
  
  https://www.elastic.co/blog/creating-frozen-indices-with-the-elasticsearch-freeze-index-api
• 

  Elastic Stack 7.0.0 RC1 Released
  • sorangutan  

  1
  1
  Posts
  29
  Views

  

  Hot on the heels of a [feature-rich 6.7 release](/blog/elastic-stack-6-7-0-released), we are thrilled to announce that the first release candidate for 7.0 is available today. Say heya to 7.0.0-rc1. The first release candidate builds on the work done in the [alpha 1](/blog/elastic-stack-7-0-0-alpha1-released), [alpha2](/blog/elastic-stack-7-0-0-alpha2-released) and [beta 1](/blog/elastic-stack-7-0-0-beta1-released) versions, and adds new layers of polish and bug fixes across the stack. Here is a quick recap of all the features going into 7.0. - Improved [cluster coordination](/blog/a-new-era-for-cluster-coordination-in-elasticsearch) for improved resiliency - Faster queries for top k hits - Intervals query - the next-gen span query for legal and patent search - Script score query - a.k.a. function score 2.0 - Geotile grid aggregations for smoother zoom on maps - Nanosecond precision support - Brand new design and navigation in Kibana (including a pervasive dark mode) - And so much more … ### Try it out on Elasticsearch Service Yes, that’s correct. We are excited to offer our first 7.0 release candidate on our [Elasticsearch Service on Elastic Cloud](/cloud/elasticsearch-service) - the only hosted Elasticsearch and Kibana solution from the creators of the Elastic Stack. Log in (or sign up for a free trial), create a cluster with a few clicks, and, and you are good to go. It is the easiest way to experience 7.0 during this preview period. And if you prefer to run it yourself, you can always [download the stack](/downloads). Before you get too excited, keep in mind that this is still a release candidate, so don’t put it into production. There is no guarantee that any of the 7.0.0 pre-release versions will be compatible with other pre-releases, or the 7.0.0 GA. ### Still time to join the Pioneer Program As with earlier preview releases, the [7.0 pioneer program](/blog/elastic-pioneer-program-7-0) continues with this release. A big thank you to everyone in the Elastic community who took the time to try out early versions and give us feedback. It’s your inputs that help us strengthen and improve the release over the preview period, and get it ready for prime time. The pioneer program continues through the RC period, and we welcome further inputs. Try it out, give feedback, and become an Elastic pioneer for a chance to win some limited edition Elastic swag. ### Get It Now! Deploy a cluster on our [Elasticsearch Service](https://www.elastic.co/cloud/elasticsearch-service/signup) or [download the stack](/downloads) to take these latest features for a spin.
  
  https://www.elastic.co/blog/elastic-stack-7-0-0-rc1-released
• 

  Helper Node Series | Imma Valls: Teaching Kids to Read at LECXIT
  • sorangutan  

  1
  1
  Posts
  31
  Views

  

  Elastic believes in giving back, both by supporting the communities in which we live and work, and recognizing that participating in these activities can inspire and enrich our lives in unexpected ways. That’s why we offer our Elasticians volunteer time off (VTO) to foster a culture of caring.

  The Helper Node blog series shines a light on the inspiring ways our Elasticians use their volunteer hours to help their community and the world at large. In this edition of the Helper Node series we meet Imma Valls, who volunteers her time helping kids learn the joy of reading.

  ---

  Can you tell us a little bit about yourself, what do you do at Elastic?

  Imma Valls: My name is Imma Valls and I’m an engineering consultant at Elastic. I’m based out of Barcelona and have been at Elastic for 9 months.

  

  Imma Valls

  How do you use your VTO?

  Imma: There is a program in my region in Spain (Catalonia) called LECXIT that mentors kids between ages eight and twelve who have difficulties reading. Schools and libraries throughout the region participate in this program and organize it on their premises.

  I have been volunteering with LECXIT for a couple of years at a local library, not far from where I live. They assign me a kid every school year, and I do weekly one-hour sessions with them. Usually from October to June. This is my third year with the program.

  Tell me about how you became involved with this organization?

  Imma: Well, I was looking to volunteer in general. One of my best friends, David, is involved in a myriad of causes and I asked him to forward me anything I might find of interest. When he sent me the LECXIT opportunity, I instantly knew this was it. I’ve always loved reading. I even think it saved my life as a kid, being able to fly anywhere with a book. So I knew I would love teaching kids the same joy.

  How did you prepare for being a tutor?

  Imma: In the beginning, I didn’t know whether I would be an effective reading mentor. I’d never tutored before, and I was a little worried. Thankfully, LECXIT gave us a few hours training before I was allowed to begin with a pupil. Regardless of my doubts, the kids always make it so easy. LECXIT attracts kids that are super cool and willing to collaborate, even if they might not love reading. And when they arrive on time each session, or when they smile at you or ask you if you are coming back next Friday, it feels like a gift.

  How do you come up with your lessons?

  Imma: During the week I try to think of what to do during the session. That could be building a Lego robot that helps teach reading through a set of instructions, or coding a game with Scratch. Or, I will prepare reading or writing materials. I look for resources and ideas everywhere I can. One of my favorites activities I do with the kids is using a set of instructions they have to read to make an origami jumping frog or chattering teeth.

  
  
  https://www.elastic.co/blog/helper-node-series-imma-valls-teaching-kids-to-read-at-lecxit

• 

  Elasticsearch 6.7.0 released
  • sorangutan  

  1
  1
  Posts
  45
  Views

  

  We are pleased to announce the release of Elasticsearch 6.7.0, based on Lucene 7.7.0. This is the latest stable release, and is already available for deployment via our Elasticsearch Service on Elastic Cloud.

  Latest stable release in 6.x:

  • Download Elasticsearch 6.7.0
  • Elasticsearch 6.7.0 release notes
  • Elasticsearch 6.7 breaking changes

  You can read about all the changes in the release notes linked above, but here are our highlight:

  If it’s a 3-letter acronym in Elasticsearch, odds are it’s now Generally Available (GA) in 6.7!

  CCR (Cross Cluster Replication) is now GA!

  We introduced Cross Cluster Replication (CCR) as a beta feature in Elasticsearch 6.5. CCR was one of the most heavily requested features for Elasticsearch. We’re excited to announce CCR is now generally available and ready for production use in Elasticsearch 6.7! CCR has a variety of use cases, including cross-datacenter and cross-region replication, replicating data to get closer to the application server and user, and maintaining a centralized reporting cluster replicated from a large number of smaller clusters.

  In addition to maturing to a GA feature, there were a number of important technical advancements in CCR for 6.7. Previous versions of CCR required replication to start on new indices only: existing indices could not be replicated. CCR can now start replicating existing indices that have soft deletes enabled in 6.7. We introduced new technology to prevent a follower index from falling fatally far behind its leader index. We’ve added a management UI in Kibana for configuring remote clusters, indices to replicate, and index naming patterns for automatic replication (e.g. for replicating metricbeat-* indices). We’ve also added a monitoring UI for insight into CCR progress and alerting on errors. Check out the Getting started with cross cluster replication guide, or visit the reference documentation to learn more. Cross cluster replication is a platinum level feature, and is available through a 30-day trial license that can be activated either via an API or within Kibana.

  
  
  https://www.elastic.co/blog/elasticsearch-6-7-0-released
• 

  Kibana 6.7.0 released
  • sorangutan  

  1
  1
  Posts
  69
  Views

  

  Each release of Kibana brings new features and product improvements. Here are the highlights of the new features and user experience changes we’ve introduced in 6.7.

  Refer to the Kibana 6.7.0 Release Notes for a list of bug fixes and other changes.

  Kibana localization and support for Simplified Chinese

  Kibana now supports Simplified Chinese, in addition to English. With 6.7, we’ve introduced a new localization framework to support localized text. If you’d like to use Kibana with Simplified Chinese, set i18n.locale to zh-CN in the kibana.yml file. For more information on the supported languages and i18n tooling, check out the localization docs.

  

  Maps for geospatial analysis (beta)

  We are excited to introduce a brand new application built for geospatial analysis: Maps.  “Where” is a critical question for most data points, and Maps enables you to ask (and answer) meaningful questions for your location-based data. Our Maps beta doubles down on geo and  introduces all new features such as support for multiple layers and data sources, mapping individual geo points and shapes, dynamic client-side styling, and global search for ad-hoc analysis. To get started, check out our Maps beta release blog.

  

  Follower indices in cross-cluster replication

  Elasticsearch has the ability to follow an index in cross-cluster replication (CCR), and we’ve added the new Add Follower Index UI in Kibana Management.  With this UI, you can replicate data from existing indices, or set up local followers on a case-by-case basis. CCR is a Platinum feature and is ready for production use in 6.7.  CCR is available through a 30-day trial license,  which you can activate in Kibana License Management.

  

  Frozen indices in Index Management

  We’ve made several enhancements related to frozen indices in Kibana Index Management. You can now freeze and unfreeze indices from the Manage menu. Kibana also now marks frozen indices, follower indices, and rollup indices with badges, so you can distinguish them at-a-glance.

  

  Also in 6.7, you can now download the workpad directly from the Edit view. Simply click the link in the workpad header to download a JSON file. To get started with Canvas, check out our Getting starting with Canvas blog or hop into our demo environment.

  Infrastructure is now GA

  We’re happy to announce that the Infrastructure app in Kibana is ready for production use in 6.7. This app is a turnkey solution for monitoring infrastructure metrics using the Elastic Stack. You can configure indices and fields associated with the Infrastructure app via the Source Configuration UI, specify neutral color gradient, group the Map view by any field, and switch to an alternative Table view. For more information about the Infrastructure app, check out our Infrastructure release blog.

  

  Introducing Uptime

  In 6.7, we’re introducing, Uptime, which enables you to easily navigate the status of your active application checks powered by Heartbeat. See our Uptime release blog for all the details.

  

  Try it now

  Deploy a cluster on our Elasticsearch Service or download the stack to take these latest features for a spin. As always, if you have any questions or want to share feedback, reach out on our Kibana forums.

  
  
  https://www.elastic.co/blog/kibana-6-7-0-released
• 

  Logstash 6.7.0 released
  • sorangutan  

  1
  1
  Posts
  45
  Views

  

  This release introduces Logstash 6.7.0 has been released with new features and enhancements. This release introduces Java 11 support for Logstash, and the move from experimental to beta of native support for Java Plugins.
  

  Java 11 Support

  With this release, we are happy to announce enhanced support for different versions of Java, so that we now support Logstash running on Java 11! Now Logstash will run on Java 8 and 11.  This change also includes a bump of the version of JRuby that Logstash uses from jruby-9.1.13.0 to jruby-9.2.6.0, which includes support for Ruby 2.5, and includes many bug fixes.

  Native support for Java plugins

  After the experimental release of this feature as experimental in 6.6.0, this feature is now moving to beta. With this API, Java developers should be able to write Logstash plugins using idiomatic Java code free of any dependencies on Ruby, which can seamlessly inter-operate with Ruby plugins and incur lower overhead in the Java execution engine.

  Plugin Improvements

  Logstash 6.7.0 includes a number of improvements to some of our bundled plugins. Some highlights:

  • The KV filter now has a timeout enforcer to help prevent slow events from blocking the pipeline, and also introduces a new `tag_on_failure` directive to make error handling configurable.
  • The Mutate filter can now directly convert from integer and float to boolean.
  • The Elasticsearch input now has an added `slices` option to manage slice scrolling.
  • The HTTP input now has an option to supply a configurable response code 
  • The TCP input fixes a race condition where data would be accepted before the queue was configured
  • The Elasticsearch output now has an option to automatically enable Index Lifecycle Management depending on whether the Elasticsearch cluster supports it.

  Download and Release Notes
  

  You can find the full list of the changes in the release notes.

  • Download 6.7.0
  • Release Notes for 6.7.0

  Feedback

  If you want to try the new features added in Logstash 6.7.0, please download it, install it, and let us know what you think on Twitter (@elastic) or in our forum. Enjoy!
  

  
  
  https://www.elastic.co/blog/logstash-6-7-0-released
• 

  Elastic Uptime Monitoring solution released
  • sorangutan  

  1
  1
  Posts
  54
  Views

  

  We are happy to announce the general availability of the Elastic Uptime solution, new with Elastic Stack 6.7. To try it out, spin up a cluster in our Elasticsearch Service (on Elastic Cloud) or download the Elastic Stack locally.

  What is Uptime Monitoring?

  Service uptime is one of the most important success metrics in service delivery. If the service is down users cannot be successful, no matter how great the user experience is when the application is up. In the world of interconnected applications and microservices, operation teams face a wider footprint of critical endpoints to monitor, not all in their control. As a result, actively measuring, reporting, and alerting on service uptime and response time at scale is the cornerstone of any observability initiative.

  Measuring service availability starts with the basics. Are the hosts or containers underlying the applications responding to network pings? Is the application port responsible for communication accessible? Uptime checks get more intricate quickly. Is the website responsible for user interaction accessible by the users, and does the HTTP response delivered by the server contain the expected content? Is the set of internal APIs in a microservices architectures available and returning correct API responses? Are the 3rd party services the application rely on responding quickly?

  Elastic Uptime

  The Elastic Uptime solution answers these questions easily and elegantly across all tiers and infrastructure components. Based on Heartbeat, an open source uptime monitor, Elastic Uptime is transparent, flexible, and developer-friendly. It can be deployed both inside and outside your organization's network to understand service uptime and response time characteristics from multiple network vantage points.

  
  

  Combininguptime monitoring with logs, metrics, and APM trace data, with Elastic Stack as the operational datastore, organizations can bridge data silos and enable analyst workflows from high-level KPIs to detailed events. This speeds up troubleshooting and root cause analysis and further empowers observability initiatives. Elastic alerting and machine learning features provide proactive notification and anomaly detection capabilities on top of all this data in a consistent and data source agnostic way.

  In addition to ensuring the health of production applications, uptime monitoring can also help with quality assurance and continuous integration workflows. As your applications evolve and move through a set of planned changes, you can find out about problems in QA and Production before user traffic ever hits them. This speeds up product development cycles, while reducing risk of negative impact to your customers.

  Collecting uptime information using Heartbeat

  Heartbeat is an open source, lightweight shipper which was originally developed in close collaboration with our SRE team to measure the response time and availability of our Elastic Cloud services across multiple regions. The product was such a useful addition to our stack for logging and metrics analytics that we decided to release it, first as beta and then as a GA in 6.5.

  Heartbeat can operate from inside or outside of your network; all it needs is network access to the desired HTTP, TCP, or ICMP endpoint. Configuration is as simple as supplying Heartbeat with the list of URLs you want to monitor. Heartbeat executes periodic checks to verify whether the endpoint is up or down, then reports this information, along with other useful metrics, to Elasticsearch.

  In a typical customer deployment, we normally see Heartbeat deployed in multiple locations, to better simulate the end user experience coming from multiple vantage points. As soon as Heartbeat starts to run, it will check the list of endpoints you have configured, send the information to the Elastic Stack.

  
  

  

  Exploring monitor data in Elastic Uptime

  Now that we have the data, we can look at it in the Uptime application in Kibana.

  Uptime Overview page

  Endpoint status

  It really doesn't matter how many Heartbeat instances you've got. As long as they report their information to Elasticsearch, you will see an aggregated view across all your monitored endpoints. If you want to know how many of your endpoints are up or down this is the place to look.

  

  Status over time

  This chart provides the status across all monitored endpoints, over a period of time. The Y-axis is the number of the monitored endpoint, while the X-axis represent a period of a selected time.

  

  Monitor status

  This table provides an up-to-date status on each individual endpoint that is monitored by Heartbeat

  Every line represents a single endpoint, with its related information, such as status (is the endpoint up or down), Last update (when did it report the latest information), Monitor name (if one was defined, otherwise the URL will be used), full URL, IP and status over time.

  
  

  

  If you want to see the up-to-date status of each monitored endpoint, this is the where to look.

  Error list

  If one of Heartbeat's monitors detects an error, Heartbeat will report it to Elasticsearch, and the error information will be presented in the error list table below. You will be able to see error code, error message, number of time this error occurred, and the error type.

  

  Uptime Monitor page

  When you click on a specific monitor – in the "Monitor Status" table, you will be redirected to the second page on this solution, the "Monitor page". This page would help you check a specific endpoint thoroughly.

  Monitor duration and Check status

  The "Monitor duration" chart represent the response time over time (in milliseconds) for a specific endpoint, while the "Check status" chart represent the endpoint's availability over time.

  

  Check History
  

  This table provides a detailed view on each individual check.  Every line in this table represents a single check that Heartbeat executes against a specific endpoint, you can filter to examine all checks or filter by up/down.

  

  Ad hoc analysis of uptime data using Kibana

  Heartbeat data is just another index in Elasticsearch. You can visualize Heartbeat data in Kibana for ad-hoc analysis. We provide a default dashboard to help you get started. To learn more about using Kibana for ad hoc analysis of Heartbeat data, please refer to this blog.

  Alerting on top of Uptime data

  You can create alerts on top of Uptime data using Watcher. Watcher is a powerful alerting and actions framework on top of Elasticsearch. Most simple threshold based alerts are easily created using the Watcher UI. For more advanced conditions and actions, you can also build a watch by working directly with the Watcher API. For a non-trivial example of an alert using uptime data, see this blog that alerts on invalid SSL certificates.

  Observability using the Elastic Stack

  The 6.7 release also brings general availability of the Infrastructure and Logs solutions. In addition to APM, these solutions are a part of the broader 'observability' story using the Elastic Stack.

  

  Done? We are just getting started!

  This is only the first phase of Uptime solution and we are not stopping here. In our upcoming releases, we will provide additional capabilities in terms of managing and configuring monitors. Until then, I hope you will use and enjoy this first release of Uptime monitoring, please feel free to reach out and to provide more feedback on our forum discuss.elastic.co.

  Uptime is available on Elastic Cloud Elasticsearch Service -- you can get started in just minutes with monitoring the uptime of your apps. Don't have an account? Sign up for a free 14-day trial today.
  
  https://www.elastic.co/blog/elastic-uptime-monitoring-solution-released
• 

  Elastic Logs app released
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  We are excited to announce general availability of Elastic Logs, a console-like experience for working with logs using the Elastic Stack. If you can't wait to try it out, spin up the latest version of the Elastic Stack in Elasticsearch Service (on Elastic Cloud) or download and run it locally.

  Elastic Logs expects very little in a way of data input --  a @timestamp and  message fields are all that is required for the application to stream logs in real time or search them historically. However, if you extract fields from your logs, the Logs UI will help you compose structured queries using auto-complete, helping you get started with your analysis faster. The best way to receive structured logs from known log files is Filebeat, an open source log shipper (check out existing modules here).

  We released Elastic Logs as beta in 6.5, and based on feedback from you, we've been hard at work to improve it. Thank you to all the users that have tried out the product and engaged with us on Discuss and Github.

  

  
  

  Below we describe some of the key improvements we made on the road to GA.

  Source Configuration UI

  The Logs UI works out of the box with Filebeat default index and field names. If you change those defaults, it was previously possible to adjust the settings using kibana.yml. The 6.7 release makes that process even easier by adding an in context flyout that allows you to configure these settings from within the Logs application. Just change what you need, and click "Update Source".

  Log event details flyout

  While looking at the log message field is most actionable as you search and filter your logs, once you've found the event you are looking for, you will want to examine metadata and structured fields behind it. The "Log event document details" flyout allows you to do that with a click of a button, without losing context. In the flyout, the icons next to the field values will enable you to filter the logs per that value, further honing the workflow of working with logs.

  Support for serverless log shipping with Functionbeat

  In 6.5, we added Functionbeat to support shipping logs in a ‘serverless' way. Functionbeat runs as a function (initially in AWS Lambda) and supports a number of inputs, including AWS Cloudwatch Logs, SQS, and Kinesis. In instances where logs are shipped by default in the schema the Logs UI recognizes, with @timestamp and message fields present, you can use the Logs UI to browse these logs right away! For other inputs, where the schema may diverge from the expected default, you can use the Source Configuration UI to adjust settings to view your logs.

  Observability using the Elastic Stack

  The 6.7 release also brings general availability of the Infrastructure and Uptime solutions. In addition to APM, these solutions are a part of the broader ‘observability' story using the Elastic Stack.

  

  What's next

  To experience Elastic Logs, spin up the latest version of the Elastic Stack in Elastic Cloud Elasticsearch Service or download and run it locally.

  This GA release means you can use this solution in production, but that does not mean that we are done! We are going to continue evolving and improving Elastic Infrastructure. Please engage with us on Discuss with questions and feedback.

  Also, do not hesitate to jump into Github and look at issues labeled with ":Logs UI". All of our discussions are in the open, and we'd like you, our users, to weigh in! Some important enhancements we're looking at for near term releases include support for custom structured logs with pin field as column and view surrounding logs.  

  
  
  https://www.elastic.co/blog/elastic-logs-app-released
• 

  Elastic Infrastructure app released
  • sorangutan  

  1
  1
  Posts
  51
  Views

  

  We are excited to announce general availability of Elastic Infrastructure, a turnkey solution for monitoring infrastructure metrics using the Elastic Stack. To experience it, spin up the latest version of the Elastic Stack in Elasticsearch Service (on Elastic Cloud) or download and run it locally.

  Elastic Infrastructure works in concert with Metricbeat, an open source shipper for metrics data, to provide an easy way to see a high-level view of hosts and containers in the context of key metrics coming from these infrastructure components. Users can group these entities by key fields and then drill down into metrics, logs, and APM traces in the context of the infrastructure component they select.

  We released the Infrastructure app in 6.5 as beta, and based on the feedback from you, we’ve been hard at work to improve it. Thank you to all the users that have tried out the product and engaged with us on Discuss and Github.

  

  Below we describe some of the key improvements we made on the road to GA.
  

  Source Configuration UI

  The Infrastructure UI works out of the box with Metricbeat default index and field names. If you change those defaults, it was previously possible to specify custom index names in kibana.yml. The 6.7 release makes that process easier by adding an in-context flyout for configuring these settings from within the Infrastructure application.

  

  Neutral color gradient

  The traditional red-yellow-green color scheme for indicating the range of metric values represented in the Map view works for many scenarios. But in some cases, the metric you are plotting is not necessarily good or bad, so a neutral color gradient makes more sense.

  

  
  

  Group by any field

  When looking at infrastructure, grouping by custom fields and tags is as important as having sensible defaults. Now you can easily group your servers or containers in the Map view by ad hoc fields, and have those fields appear in the drop-down for easy access in the future.  

  

  
  

  Infrastructure table view

  The Map view is great, but sometimes you just want to see a table and sort your infrastructure components by key attributes. To do that, switch over to the Table view with just one click, without losing context.

  

  Newly supported metrics data types

  Demand for turnkey metrics collection using the Elastic Stack continues to grow. In 6.7 we added support for MariaDB and Percona as part of our MySQL database module.

  In addition to that, we’ve been hard at work to bring many of the existing Metricbeat modules to general availability. Among these are a number of modules from the Kubernetes ecosystem -- Traefik, etcd, Envoy Proxy -- as well as modules for Windows, RabbitMQ, Couchbase, golang, php_fpm, Memcached, and ceph.

  See summary table below and read details at individual module pages in the Metricbeat documentation:

  Module	Status
  MariaDB support in MySQL module	New in 6.7
  envoyproxy, traefik, etcd	Moved to GA
  Aerospike, Ceph, Couchbase, Golang, Memcached, Munin, PHP_FPM, RabbitMQ, uwsgi, Windows	Moved to GA

  Observability using the Elastic Stack

  The 6.7 release also brings general availability of the Logs and Uptime solutions. In addition to APM, these solutions are a part of the broader ‘observability’ story using the Elastic Stack.

  

  What’s next

  To experience Elastic Infrastructure, spin up the latest version of the Elastic Stack in Elastic Cloud Elasticsearch Service or download and run it locally.

  This GA release means you can use this solution in production, but that does not mean that we are done! We are going to continue evolving and improving Elastic Infrastructure. Please do not hesitate to engage with us on Discuss with questions and feedback.

  
  
  https://www.elastic.co/blog/elastic-infrastructure-app-released
• 

  Elastic Maps beta released
  • sorangutan  

  1
  1
  Posts
  46
  Views

  

  We are excited to introduce Elastic Maps, a brand new way to map, visualize and query your location data in Kibana.

  Whether you're monitoring the source location of a DDoS attack in real time, mapping top cities driving your web traffic or stringing together queries like "How many taxis were available in the Chicago Financial District in the past 5 minutes?", location is an integral component of most search experiences powered by the Elastic Stack. Elastic Maps significantly expands on the existing region maps and coordinate maps visualizations in Kibana, and introduces a tailored solution with a suite of new features for geospatial analysis. Elastic Maps is launching as a beta feature in version 6.7, and ships in the free default distribution of the Elastic Stack.

  New features in Elastic Maps

  Let's take a quick look at all of the great new geo functionality that ships with Elastic Maps.

  Multiple layers, multiple data sources

  Add any number of layers to your map from a variety data sources.

  Data sources include:

  • Documents and grid aggregations from Elasticsearch
  • Files and tiles served by the Elastic Maps Service
  • Self-hosted or 3rd party hosted vector layers and image services, including custom GeoJSON, tile services, or OGC Web Map Services

  Multiple layer types are supported, each bringing its own set of functionality:

  • Use vector layers for vector data like GeoJSON, Elasticsearch documents, or grid aggregations. Style features in these layers based on properties of the data and enrich those features with metrics derived from other data sources.
  • Use heatmap layers to show densities of your Elasticsearch data at different resolutions.
  • Use raster layers for background layers or custom overlays, like weather maps or satellite imagery

  
  

  Map individual documents

  As long as your index pattern has a geo_point or geo_shape field, with Elastic Maps, you can put individual documents as features on a map. These documents are fully filterable with KQL or Lucene queries. Another advantage is that, unlike the region maps visualization, you no longer need to setup external services to deliver contextual map data, like boundary layers. Just index your features into Elasticsearch, and bring them in as a layer.

  

  Data-driven styling

  All properties of the vector features on a map, like lines and fills, can be styled individually. Line and fill colors as well as line and symbol sizes can also be driven by properties of your features. You can map numerical values to a color ramp, in the case of line and fill colors, or to a pixel-size range in the case of line size or symbol size.

  Overview first, detail on demand

  You can set scale-ranges on the visibility of individual layers. Take this approach to show the correct visual representation of your data at all scales.

  A common example is to show a heatmap with density plots when the user is zoomed out at small scales. As the user zooms in, start showing clusters of documents. Then, as the user reaches a level where the clusters get smaller and smaller, start showing the actual individual features. While the user is zooming, they can use the tooltip functionality to read out either the aggregate metrics for a cluster, or, when zoomed in all the way, the field values of the individual documents that make up the cluster.

  

  Real-time Kibana goodness

  Elastic Maps is fully integrated with the existing functionality of Kibana and Elasticsearch. You can filter your data in real time using full text search, and run queries using the KQL and Lucene query languages.

  

  Looking towards GA

  Elastic Maps is in beta, but we are already looking towards the GA release. The team is actively working on two main features in order to make that milestone happen:

  • Full embeddability: Embed maps in a dashboard, and have them behave just like any other visualization on a dashboard. Send filters from the map to the Kibana global filter bar, and have global filters and queries sent from other visualizations apply to your map.

  • GeoJSON upload: Ingesting geo-features with the Elastic Stack should be easy, and upload support for a GeoJSON file is the first step in making that happen. Just drag and drop into your map to index those features into Elasticsearch.

  Learn more about Maps

  To learn more, consult the Elastic Maps documentation and the getting started page.

  We also encourage you to attend the free Elastic Maps for Geospatial Analysis webinar on on April 10th, 2019. If you're reading this after the 10th, don't worry, all webinars are recorded and saved to the same URL as they're presented from.

  Feedback encouraged

  We look forward to hearing your thoughts and comments on the Elastic Maps beta. Please feel free to submit feature requests and bugs in the Kibana GitHub repo, and talk how-why-what on the Kibana discuss forum.

  
  
  https://www.elastic.co/blog/elastic-maps-beta-released
• 

  Beats 6.7.0 released
  • sorangutan  

  1
  1
  Posts
  49
  Views

  

  We are pleased to announce the Beats 6.7.0 release. This is the latest stable version and it comes with a GA version release for Functionbeat and a migration tool for Beats Central Management.

  • Download
  • Release notes

  Functionbeat GA

  A few months ago, we were announcing the first version of Functionbeat as beta, a serverless ingestion for Elasticsearch. Functionbeat is a new addition to the Beats product suite that can easily ingest data in a scalable, reliable and cost efficient fashion.

  

  Besides getting the logs from AWS Cloudwatch and Simple Queue Service (SQS), the GA version also brings support for Kinesis.

  Upgrade Beats Central Management (beta) to the 6.7 version

  In the 6.7 version of the Beats Central Management, the schema of the configuration documents used by Kibana has changed. As a result of the template changes, enrolled Beats and tags created prior to 6.7 will not work with the new UI until they are moved to the new format. For this, we have created a one time tool to help you migrate your data from 6.6 to 6.7.

  Before running the migration tool, you would need to configure your cluster details in the migrate.yml file under the url, username and password along with ssl.* settings.

   ./migrate-management-beats
  

  Please check the documentation for more details.

  New datasets in the Auditbeat system module

  The Auditbeat system module, which was introduced in 6.6, got more functionality in this release:

  • A login dataset that collects log in information by reading /var/log/wtmp and /var/log/btmp.
  • A package dataset that collects the installed DEB, RPM, and Homebrew packages. Like most data sets in the Auditbeat system module, it can send the full list of installed packages but also differential updates (packages added or removed).
  • The process dataset is now enhanced with user information, so it's easy to correlate between users and their processes.
  • A new entity_id is added to the various datasets to enable visualising the number of unique processes, sockets, users, etc. See this ticket for how this field is defined.

  Feedback

  If you want to try the new features added in Beats  6.7.0, please download it, install it, and let us know what you think on Twitter (@elastic) or in our forum. Enjoy!

  
  
  https://www.elastic.co/blog/beats-6-7-0-released
• 

  Using the Elastic Stack for Business Intelligence at Liefery
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  Founded in 2014, Liefery is a delivery company based in Berlin. They provide transparent and plannable same day and next day delivery services. They are also developing urban delivery concepts for the future. By entering the parcel delivery market, their goal is to create the modern customer experience that people expect. This means predictable, plannable parcel delivery at the terms and at the time that suits the receiver.

  We recently met with Simon Stemplinger, CTO at Liefery. He walked us through the company’s Elastic  journey that helped enable their business analytics capabilities with Kibana as a core component. As he and Leifery found out, the Elastic Stack is applicable to more than just log analytics.

  ---

  Before using Elastic, what issues were you looking at solving?

  Simon Stemplinger: We develop and run our own software platform to manage and support our delivery services. Our software spans multiple separate apps and services on multiple application cluster nodes. Looking through log files was quite painful, and therefore it was something we only did when we really had to.

  Furthermore, we were looking for a business intelligence (BI) software to help our colleagues in the operational teams (where we manage our drivers and the parcels) make sense of our business data and keep track and improve our operational metrics.

  We looked at a BI tool, as well as several hosted cloud-based logging apps, before coming across the Elastic Stack as a self-hosted solution for log analytics. I was familiar with the Elasticsearch DB from a previous project where we used it for a full-text product search and knew that it performed well and could handle large amounts of data. We gave the Elastic Stack a test drive and were immediately excited by the combination of the power of the Elasticsearch database and the flexibility and ease of use of the Kibana visualizations. Kibana has quickly become the go-to tool for our answering all questions about our business data.

  So you started using the Elastic Stack for log analytics. What other Elastic Stack use cases have you carried out?

  Stemplinger: When we started centralizing our logs using the Elastic Stack, several initiatives were also launched around reporting and service quality management. Most of these initiatives were using Excel. It was very tedious and the user adoption was slow. At some point our engineering team thought: "Kibana is great for data visualisation. Why don’t we just try throwing our business data in there and see how far we can get?" 

  I sat down with our Head of Operations and presented him in a demo how to create visualizations and dashboards using Kibana. He was immediately hooked. Despite not being a technical person, he managed to build in a short time quite in-depth and detailed data visualizations and dashboards. We created system health dashboards for IT and operational quality dashboards for our business teams. We then set up large flat-screen TVs with Raspberry PIs attached showing these dashboards during normal work hours. Kibana became our new de-facto quality and reporting BI tool. 

  Then we started using the commercial features, as we wanted to widen our user base to all parts of our company. The security feature of the Elastic Stack lets us give certain users read-only access or limited access to certain indices. This allows us to give read-only access to a wider part of the company while giving write access only to certain trained users. In the future we might extend this to allow access to certain data (e.g. financial data) only to privileged users. 

  Our Kibana based reporting solution is still widely popular in the organization. In addition we use the opportunity of having both all application log and business data in one database to do all kinds of business alerting (detection of brute force attacks, API issues, pathologic business data, etc.).

  What are the future projects you’re looking at developing with the Elastic Stack?

  Stemplinger: Having made a name for ourselves as an innovative software-driven company in the urban logistics space, we are starting to offer our software as SaaS software platform to everyone. A number of key players in the industry have already expressed interest and are starting POC projects in various stages. The Elastic Stack enables us to extend our service offering with a powerful analytics add-on that we can pre-configure with help reports and visualizations out of the box, but that is also highly customizable to our clients’ needs.

  Can you tell me more about the impact Elastic has had on your company and the team?

  Stemplinger: We experienced tons of process improvements. Today, using the Elastic Stack, the business team can save time on their daily review calls on service quality, they don’t need to allocate more time to actually prepare the report and can just at the quality reports in Kibana. In a similar way, the developers can also quickly detect issues using performance dashboards on which they monitor key metrics like the response time, they also enhanced the data with users IDs to track what’s being done by who. 

  In a way, Elastic has helped our teams become more responsive and agile. The Elastic Stack helped us overall to get more insight into our data. The result manifests first and foremost in better service quality, less shipping delays, better predictions, as well as more informed meetings and discussions. It allows us to quickly troubleshoot technical problems and gives both tech and business teams a very powerful tool in their arsenal.

  ---

  

  Simon Stemplinger is the CTO at Liefery. He is a hands-on developer with a passion for databases, and is a big fan of the Elastic Stack. He spent the last 5 years building up the software of the company; from the first lines of code and fully featured logistics to a last-mile software platform and SaaS product. 

  
  
  https://www.elastic.co/blog/using-the-elastic-stack-for-business-intelligence-at-liefery
• 

  This Week in Elasticsearch and Apache Lucene - 2019-03-22
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  Highlights

  Scripting

  We have added a REST API that will return information about script contexts and associated whitelists - this is the first step toward automation of whitelist APIs in the documentation, and may possibly be used for other internal features as well.

  Snapshot and Restore UI

  We merged PR for repositories list and details UI. Next for the app is the create repository form.

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2019-03-22

• 

  Elasticsearch as a time series database for telemetry data at NS1
  • sorangutan  

  1
  1
  Posts
  28
  Views

  

  NS1 is a leading Domain Name System (DNS) host and web traffic management company that equips customers with resilient, redundant, and multi-purpose DNS solutions for various application delivery stacks. Providing industry-leading web redundancy and speed means NS1’s time series database (TSDB) needs to collect and analyze incredible amounts of telemetry data — nominal throughput spikes can reach 700k data points per second — in as close to real time as possible.

  This telemetry data comes from a globally distributed infrastructure that spans across systems, providers, and third-party integrations that all dynamically scale and emit logs, time series data, packet captures, sflow, and more. As NS1 continued to grow, establishing an architecture that could support real-time telemetry became a major need — one that NS1 DevOps and Software Engineer Christian Saide describes as being “difficult to solve elegantly.”

  The team’s initial solution was a 13-server OpenTSDB cluster for handling OS and application metrics, as well as a three-server Elasticsearch cluster for collecting log data and general unstructured telemetry. This dual system worked well enough … until NS1’s terabytes-per-day operations continued to scale.

  “These systems started to fail. Our telemetry systems started requiring more time to manage than the infrastructure it was intended to monitor.” - Christian Saide, DevOps and Software Engineer, NS1

  As the team deliberated options for restructuring their TSDB, they established some core requirements:

  • A single system for metrics and unstructured telemetry
  • Granular control over what data is stored and retained
  • Easy scalability
  • Large, diverse, responsive community for support
  • Sufficient analysis capabilities

  NS1 decided to go all in with Elastic. Not only would their primary need for a single, scalable TSDB be solved with Elasticsearch, but they could now smoothly transition to the Kibana UI to make sense of the mountains of diverse data being ingested by a variety of Elastic Beats — Metricbeat, Filebeat, Packetbeat, and Heartbeat. Product features aside, it was in large part Elastic’s community that won over the team and influenced their decision.

  “The community and all of the plugins all fit together and make a cohesive environment for you to utilize,” Saide explains. “There is a myriad of blog posts and different organizations and companies out there that are explaining how they use Elasticsearch and providing insight to others on how they might be able to utilize it … [discuss.elastic.co] has been an absolutely amazing tool for us.”

  They also found value in data introspection and data structure support. Gone were the days of simple keyword and wildcard matching; they could now perform searches for everything from IP range and numeric lookups to actual full-text queries and tokenization. “These are incredibly powerful tools to allow your operations team to actually take a look at what you’re collecting and use that in interesting ways to solve problems or understand what’s happening in an infrastructure.”

  [Pull quote:] “This has been an incredibly effective solution for us at NS1.”

  NS1’s biggest triumph by far has been their ability to manage a single operational telemetry cluster. “I cannot tell you how much time, money, and just cognitive brainspace that this has already saved us,” Saide says. “[W]e get to focus on one and really learn one cluster. And it has been an absolutely amazing opportunity for us to spend more time working on other systems that are more directly related to our company.”

  Elastic’s clustering capabilities quickly became a hit at NS1. The automatic and manual multicast and unicast clustering options in Elasticsearch — and inherent fault tolerance therein — meant the team could readily scale up and down as needed without worrying about their cluster. They discovered they could take down nodes for major upgrades or maintenance with zero downtime.

  Pioneering a major TSDB architecture overhaul doesn’t come without its learning curve, though. In the Leveraging Elasticsearch as a Time Series Database webinar, Saide details not only the successes, but also the challenges that came with NS1’s rebuild and even dives into suggestions for adopting Elasticsearch as a primary cluster source.

  

  
  
  https://www.elastic.co/blog/elasticsearch-as-a-time-series-database-for-telemetry-data-at-ns1