Elastic News

• 

  Hosted Elasticsearch Services Roundup: Elastic Cloud and Amazon Elasticsearch Service
  • sorangutan  

  1
  1
  Posts
  67
  Views

  

  Our Elasticsearch Service on Elastic Cloud, which runs Elasticsearch and Kibana as a service, has become popular. But there's been some confusion. Elastic's Elasticsearch Service and the Amazon Elasticsearch Service are different offerings, and neither is the same as running vanilla Elasticsearch on AWS.

  To be very clear, the Amazon Elasticsearch Service is not related to Elastic. Elastic does not partner with, participate in, or support, the Amazon Elasticsearch Service.

  Instead, we offer Elastic Cloud, our platform for hosted Elasticsearch, Kibana, and X-Pack service that runs on AWS and Google Cloud Platform (GCP). We also support a variety of customers who self-manage their own clusters on AWS.

  And, again, Elastic's Elasticsearch Service is not the same as the Amazon Elasticsearch Service. In fact, we believe our Elastic Cloud platform brings unique value to mission-critical applications, validating the importance of our offering.

  We'd like to explain why.

  Features That Delight with the Projects You Love, Hosted

  When we talk about the Elasticsearch Service built on Elastic Cloud, we often dive straight into the details about our open source products in the Elastic Stack — Elasticsearch, Kibana, Beats, and Logstash. While they're fantastic, we'd like to start from a different angle: X-Pack.

  X-Pack is an extension that bundles together security, alerting, monitoring, reporting, and graph exploration capabilities that extend what you can do with the Elastic Stack. Furthermore, Elastic's Elasticsearch Service is the only offering that ships with free features like Search Profiler and the Elastic Maps Service (with zoom levels you can dial up to 18).

  Access to these features opens up brave new worlds and possibilities with Elasticsearch and Kibana. And X-Pack features, along with the open source products, get better and better with every release.

  This is why we take pride in being able to offer the latest versions of Elasticsearch, Kibana, and X-Pack on Elasticsearch Service the day a release drops. When Elasticsearch 5.4 or 6.0 or 10.3 ships, you can count on the Elasticsearch Service having the option to upgrade to it at the same time. Similarly, if there is a security vulnerability, we patch user clusters as soon as the fix is released.  

  Our ability to get you the latest and greatest the moment it's available is critical to your success. We also know there's more to it than that, whether you need to make a change to your elasticsearch.yml (specific settings are whitelisted in Elastic Cloud), click a single button to upgrade your cluster to the most recent version, restore from an automated snapshot that occurs every 30 minutes, scale your cluster by moving a slider, or, recently, file a ticket in the support portal provided all Elastic Cloud customers.

  All of this matters because it makes good sense, but also because you've told us it matters. And we will always strive to honor that and listen to you.

  There Is No Compression Algorithm for Experience

  As an Elasticsearch Service user you are, in a way, benefiting from the wisdom of the relative ancients. Elastic Cloud has been around since 2012 (known as Found in its early life). Amazon's Elasticsearch Service was introduced in 2015.

  Operational excellence is not something to take lightly or undervalue. There is a lot of work that goes into ensuring that the lights stay on and customers are getting a high level of service. There is no compression algorithm for experience. (Amazon, of course, has been offering services via AWS since 2006. In the context of managing and supporting hosted Elasticsearch, however, our team has a few years head start.)

  The Elastic Cloud is a mature platform. It supports customers like IBM, Fandango, Activision Blizzard, Unilever, Shopify, and more. They put their trust in us to keep their mission-critical systems firing on all cylinders.

  To make sure that we are providing the uptime our Elastic Cloud premium customers require, we now provide an SLA for Elastic Cloud. And, we do so by measuring cluster-level availability instead of infrastructure-level availability.

  We Eat, Sleep, and Breathe the Elastic Stack

  The team that builds the Elastic Stack works alongside the team that builds and runs Elastic Cloud. No one, and we mean no one, understands the nature of issues like the Elastic team. We strive for a bug-free, problem-free experience. But it is software and things do, indeed, happen. Support from the source can make the difference between a pleasant day with a resolved interruption and one that is downright unpleasant.

  Amazing Performance, Very Little Cost

  As of today, Elastic Cloud uses only I-series instances on AWS (currently I2). As such, even the smallest cluster in Elastic Cloud runs on best-in-class AWS instances. Given a production need for I/O, network, and CPU guarantees, T-series instances aren't terribly applicable. And, of course, I3 instances are coming soon.

  That said, simple price comparison is a race to the bottom. The Amazon Elasticsearch Service is cheaper in some circumstances. In an apples-to-apples comparison (i2 instances with similar memory to disk ratios, retention periods that match, matching deployment across AZs, and ignoring support and commercial products), the Amazon Elasticsearch Service is actually ~20% more expensive. Learn how much your deployment will cost on the Elasticsearch Service with our interactive pricing calculator.
  

  As you can see, Elastic's Elasticsearch Service is price competitive and you also get the benefit of current releases, security patches, and X-Pack.

  "But wait," you may say. "Amazon already has my credit card number." Or sometimes, "I have to use my Amazon credentials for payment."

  We've got you covered.  Now you can pick between having your bill come through Elastic or AWS via the hosted Elasticsearch service on the AWS Marketplace!

  Running Elasticsearch on AWS

  We have the perfect post for you if you're interested in running and managing your own instances of Elasticsearch on AWS.

  The Details Matter

  "I get it," you say, "Elastic knows their products better than anyone else. But what about the details?" They are below and we will keep the table current as things change.

  The beauty of modern software is that there is flexibility in deployment and consumption models. If you desire a cloud deployment, make sure you understand what you are getting and from whom. As the creators of Elasticsearch, Kibana, Beats, and Logstash, we want you to have the best possible experience with the Elastic Stack. That's why we created and stand behind only one hosted offering — Elastic's Elasticsearch Service on the Elastic Cloud.

  Last updated on November 14, 2018.

  	Amazon Elasticsearch Service	Elastic Elasticsearch Service
  Elastic Stack - Open Source Features	Some	All
  Current Version	6.3.0	6.5.0
  Same Day Elastic Stack Version Release	No	Yes
  X-Pack (Elastic Commercial Plugins)	No	Yes Security, Alerting, Monitoring, Graph, Reporting, Machine Learning
  Canvas	No	Yes
  Kibana Spaces	No	Yes
  Elasticsearch SQL Support	No	Yes
  Data Rollups (API and UI)	No	Yes
  Centralized Management for Beats & Logstash	No	Yes
  Elastic Technical Support	No	Yes Elastic Cloud Subscriptions
  One-Click Upgrades	Yes	Yes
  Hot-Warm Deployment Template, with Index Curation	No	Yes
  Underlying Cluster Hardware	Single instance type for all roles	Multiple instance types (via deployment templates)
  Default Snapshots	1 time per day	48 times per day Every 30 minutes Stored for 48 hours _source only snapshot also available
  Instant Rollout of Elasticsearch and Kibana Security Patches	No	Yes
  Custom Plugin Support	Not supported	Supported
  Java Transport Client	Not supported	Supported
  Cross Zone Replication	Support for up to 2 availability zones	Support for up to 3 availability zones
  SLA-Based Support	General level support, not specific to AWS ES	Yes
  Uptime SLA	No	Yes 99.95% cluster uptime in a given month as long as the cluster is deployed across 2 or more zones
  Elastic Maps Service (geo-visualizations in Kibana)	Does not work out of the box	Yes
  Security	Only perimeter-level security and standard IAM policies	Transport encryption Authentication Role-based access control Field- and document-level security Encryption at rest
  Alerting	Need to build and manage your own system to create alerting functionalities. This depends on Amazon Cloudwatch, which comes with predefined, simple metrics. If you want something more sophisticated, or related to your data, you'll need to build a custom metric and alerts.	Allows you to create scheduled queries, conditions, and actions on your data in Elasticsearch. UI to create, manage, and take actions on alerts.
  Monitoring	Depends on Amazon Cloudwatch, that covers a few metrics including cluster state, node information, etc.	Feature-rich and complete monitoring product specifically designed for Elasticsearch and Kibana. Captures a wider range of metrics including search/index rate and latency, garbage collection count and duration, thread pool bulk rejection/queue, Lucene memory breakdown, and more with 10-second data granularity to ensure that clusters are running healthy. Robust tools to diagnose, troubleshoot and keep your cluster healthy including automatic alerts on cluster issues.
  Graph	No	Yes
  Reporting	No	Yes

  If you haven't already, we invite you to take a 14-day trial of the Elasticsearch Service (no credit card required). Enjoy.

  
  
  https://www.elastic.co/blog/hosted-elasticsearch-services-roundup-elastic-cloud-and-amazon-elasticsearch-service
• 

  Self-Managed Elastic App Search Beta Released
  • sorangutan  

  1
  1
  Posts
  69
  Views

  

  Many of you are well familiar with Elasticsearch, which has been downloaded over 100,000,000 times, providing search within the most popular applications the world over. But you might not be familiar with Elastic App Search. Today — for the first time — you can download it and run it within your own infrastructure.
  

  Elasticsearch Born

  Elasticsearch — you know, it’s for search. Elastic App Search is also for search, and it is built atop Elasticsearch. While Elasticsearch is found at the heart of searching, logging, and data storing infrastructure, it is seldom exposed to the public internet.

  To allow public access to your Elasticsearch data, you might put it behind a proxy server, or wrap its APIs in your own middleware. But then you’ll also need to ponder what to expose, keep it secure, manage authentication, and write your own insulating logic to allow clients to reach your data.

  If you just want to provide search within your application — to index your data, then construct an application or product from it — you will be most interested in Elastic App Search. It is the intelligent, optimized, API wrapper in front of Elasticsearch, and much more.

  Search, Dance

  We can best demonstrate Elastic App Search by experimenting with its key operations — first, there is the creation of your Engine, the conduit that represents your search indexes.

  We will make an Engine called:

  walk-in-the-park
  

  While we can choose from over a dozen different languages, we will choose English, the language that matches our data set:
  

  curl -X POST 'http://localhost:3002/api/as/v1/engines' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer private-xxxxxxxxxxxxxxxxxxxx' \
  -d '{
    "name": "walk-in-the-park"
  }'
  

  An Engine is nothing without documents, so we shall provide some. How about two objects that represent National Parks?

  curl -X POST 'http://localhost:3002/api/as/v1/engines/walk-in-the-park/documents' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer private-xxxxxxxxxxxxxxxxxxxx' \
  -d '[
      {
    "description": "This park and the co-managed state parks protect almost half of all remaining coastal redwoods, the tallest trees on earth. There are three large river systems in this very seismically active area, and 37 miles (60 km) of protected coastline reveal tide pools and seastacks. The prairie, estuary, coast, river, and forest ecosystems contain a wide variety of animal and plant species.",
    "nps_link": "https://www.nps.gov/redw/index.htm",
    "states": [
      "California"
    ],
    "title": "Redwood",
    "id": "park_redwood",
    "visitors": 536297,
    "world_heritage_site": true,
    "location": "41.3,-124",
    "acres": 138999.37,
    "square_km": 562.5,
    "date_established": "1968-10-02T05:00:00Z"
  },
  {
    "description": "Bisected north to south by the Continental Divide, this portion of the Rockies has ecosystems varying from over 150 riparian lakes to montane and subalpine forests to treeless alpine tundra. Wildlife including mule deer, bighorn sheep, black bears, and cougars inhabit its igneous mountains and glacial valleys. Longs Peak, a classic Colorado fourteener, and the scenic Bear Lake are popular destinations, as well as the historic Trail Ridge Road, which reaches an elevation of more than 12,000 feet (3,700 m).",
    "nps_link": "https://www.nps.gov/romo/index.htm",
    "states": [
      "Colorado"
    ],
    "title": "Rocky Mountain",
    "id": "park_rocky-mountain",
    "visitors": 4517585,
    "world_heritage_site": false,
    "location": "40.4,-105.58",
    "acres": 265795.2,
    "square_km": 1075.6,
    "date_established": "1915-01-26T06:00:00Z"
  }
  ]'
  

  In response, a simple set of JSON objects have now become indexed, searchable documents, optimized by default for full-text search. The documents exist within your Elasticsearch cluster at the heart of App Search.

  But we have more than text fields, and so we should adjust our schema. Fields can also be of type number, date, or geolocation.

  Want to generate results given proximity to geo-coordinates or a specific date? Or search through a range of numbers? If the schema is right, you can do so at query time.

  curl -X POST 'http://localhost:3002/api/as/v1/engines/walk-in-the-park/schema' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer private-xxxxxxxxxxxxxxxx' \
  -d '{
    "description": "text",
    "nps_link": "text",
    "location": "geolocation",
    "states": "text",
    "title": "text",
    "visitors": "number",
    "world_heritage_site": "text",
    "acres": "number",
    "square_km": "number",
    "date_established": "date"
  }'
  

  Ahh-ha, good good. We are now ready to search through this data, and for that we are going to use a JavaScript client. But first, how about we craft a unique key…

  curl -X POST 'http://localhost:3002/api/as/v1/credentials' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer admin-xxxxxxxxxxxxxxxx' \
  -d '[
  {
    "name": "limited-javascript-search-key",
    "type": "search",
    "read": true,
    "write": false,
    "access_all_engines": false,
    "engines": [
      "walk-in-the-park"
    ]
  }
  ]'
  

  Key acquired: search-lpaewu2xf6uc52dr8med54v8. Along with this key, we have an Engine - which contains an index with multiple data types — and two lush documents over which we can search. And now, to search.

  We will use a JavaScript client as promised, and we will request only the title and a limited description:

  const client = SwiftypeAppSearch.createClient({
    searchKey: 'search-soaewu2ye6uc45dr8mcd54v8',
    engineName: 'walk-in-the-park'
  })
  const query = 'rocky mountain'
  const options = {
    filters: {
      result_field: {
        title: {
          raw: {}
        },
        description: {
          raw: {}
        }
      }
    }
  }
  client
    .search(query, options)
    .then(function(response){ console.log(response) })
    .catch(function(error){ console.log(error) })
  

  Look what we found!

  {
    "meta": {
      "warnings": [],
      "page": {
        "current": 1,
        "total_pages": 4,
        "total_results": 38,
        "size": 10
      },
      "request_id": "ce1a5d2a70113fab5d361b335a76e0ad"
    },
    "results": [
      {
        "title": {
          "raw": "Rocky Mountain"
        },
        "description": {
          "raw": "Bisected north to south by the Continental Divide, this portion of the Rockies has ecosystems"
        },
        "id": {
          "raw": "park_rocky-mountain"
        },
        "_meta": {
          "score": 48.68962
        }
      }
    ]
  }
  

  Stop! Query Time

  At query time, one can adjust the weight of fields, giving greater precedence to specific fields. One can also provide boosts depending on the result values of those fields, receive facet counts, sort, create groups, apply tags, and more — all while collecting analytics data in realtime.

  The Analytics API can be used in conjunction with search; are your top 3 search results often changing? Perhaps your front-end code can request your most popular search results and then re-configure its content or product listings in real-time. Or, perhaps you just want a tidy way to view what users are searching for, finding, and not finding.

  More than just APIs, team members of all kind can add value from the Elastic App Search dashboard. You can view analytics, curate results, add synonyms, configure weights and boosts, and more, all from within an intuitive user interface.

  All Packed Up, Somewhere to Go

  During the beta period, you can spin Elastic App Search up on your own hardware at no charge. To do so, download App Search.

  This self-managed release of Elastic App Search requires that you have Java 8 installed and Elasticsearch 6.4+ running. Elasticsearch requires at least a Basic license. You will receive a licence for free when downloading Elasticsearch from Elastic.co.

  Once the dependencies have been taken care of, enter the decompressed App Search directory and run:

  bin/app-search
  

  And now access http://localhost:3002, login with the default credentials and explore. You can now host and configure App Search however you like.

  Wondering where to start? Index some documents, then see how your data looks within the Reference UI, a foundational, React based open source search experience that you can build on top of, or use from within the dashboard for search experimentation. From there, we have ample documentation that can help you fine tune search relevance and build the next great search experience.

  We’d Love Your Feedback

  Please let us know how you are feeling about Elastic App Search. If you run into any bugs, have any comments or questions, please reach us at: [email protected].

  Elastic App Search, Managed

  If this sounds intriguing, but you are more interested in Elastic App Search as a managed service, you can sign up for a free 14 day trial.

  
  
  https://www.elastic.co/blog/elastic-app-search-beta-released
• 

  Monitoring Airport Security Operations with Canvas & Elasticsearch
  • sorangutan  

  1
  1
  Posts
  83
  Views

  

  At Crimson Macaw, we recently took on an interesting project with Manchester Airport Group (MAG). We were to create and implement real-time dashboards for Airport Security Operations at Stansted Airport.

  This would give the control room and security staff a better view of the passenger flow and security performance, allowing them to quickly make decisions based on real-time data. Data needed to be ingested from a number of on-premises systems and external data sources and then visualised on a number of large screens.

  Data Ingestion Challenges

  After deciding on Elasticsearch as the data storage layer, we needed to determine what data would be ingested and how we would do it. The information was available from a variety of sources. On-premises database systems, files frequently put into AWS S3 Buckets, as well from external API data sources. An example being National Rail data, which we loaded into Elasticsearch using the STOMP (Streaming Text Oriented Messaging Protocol) interfaces.

  This in itself presented some initial challenges that we had to overcome:

  • Polling for data from databases had to be more frequent than 1 minute
  • Incoming data from STOMP was gzip compressed

  Polling Databases More Frequently than Once per Minute

  We addressed the first problem with a simple patch to the existing logstash-input-jdbc plugin. Before the patch, schedules could only be expressed in cron format. It used a Job Scheduler known as rufus-scheduler, which supported supplying schedule expressions as either cron or number of seconds. The only change was single line of code to use the repeat method instead of cron:

  if @schedule   @scheduler = Rufus::Scheduler.new(:max_work_threads => 1)   @scheduler.cron @schedule do   @scheduler.repeat @schedule do     execute_query(queue)   end

  To be able to make use of this, our patch is available on GitHub.

  Handling Gzip Compressed Messages

  In order to handle compressed messages arriving from STOMP interfaces, we had to decompress them to filter the data using Logstash. Although there is an existing codec for reading lines of data from a gzip compressed message, in our use case the gzip decompressed messages were multi-line XML. To overcome this, we created our own plugin — logstash-codec-gzip — which we’ve made available on GitHub.

  Visualising the Data using Canvas

  We started to use Kibana to visualise the data that was available in Elasticsearch hosted in Elastic Cloud, but we couldn’t quite do what we wanted. It felt like a more granular level of control was needed. We came across Canvas by chance, when the Head of BI from Manchester Airport Group and I were at the AWS Summit at London May 2018. We spoke to the folks at the Elastic stand who mentioned Canvas and how they felt it was a better fit for what we wanted to achieve.

  “We’re having difficulty in trying to visualise our data in Kibana as it doesn’t give the low-level control that we need.”
  “Have you seen Canvas?”
  “No, what’s that?”
  “Our new visualisation tool; it’s in technology preview at the moment, but it sounds like it might give you what you want”

  Installing Canvas and First Impressions

  Canvas was available as a plugin for Kibana and installed using the normal process of adding any Kibana plugin. Since Canvas was not available on Elastic Cloud during the technology preview phase, we hosted Kibana in the Manchester Airport Group AWS environment — fully scripted, using Terraform.

  Canvas has a nice simple expression language for controlling how each element is visualised. It reminds me of shell programming where the output of one command is piped into the next and sub-expressions are achieved by declaring an expression inside braces.

  

  My first attempt at a Canvas Dashboard for trains arrivals (see below for the current version)

  After a couple of days of familiarising ourselves with Canvas, we were able to visualise the train arrivals into Stansted Airport. This dashboard was created using the Markdown element for the text, Image Repeat for the icons and images in the footer.

  Things were looking great and we had control of the individual elements, but there was still something missing. For our use case, we needed more control to provide:

  • Format timestamps based on a time zone – data was being stored as UTC, but the time needed to adjust based on daylight saving
  • Change the colour of the text and/or images based on known thresholds

  Expanding Canvas

  Writing plugins for Canvas is very similar to writing plugins for Kibana. They are written in NodeJS and utilise registries which allow you to add additional functions and even new elements that can be selected inside the Canvas UI. We created 3 plugins to achieve the level of control needed.

  Using Time Zones when Formatting Timestamps

  This was the first plugin we wrote, it’s a very simple expansion of the built-in formatdate:

  import moment from 'moment';
  import 'moment-timezone/builds/moment-timezone-with-data';
  export const formatdatetz = () => ({
    name: 'formatdatetz',
    type: 'string',
    help: 'Output a ms since epoch number as a formatted string according to a given timezone',
    context: {
      types: ['number'],
    },
    args: {
      _: {
        types: ['string'],
        help: 'MomentJS Format with which to bucket (See https://momentjs.com/docs/#/displaying/)',
        required: true
      },
      timezone: {
        types: ['string'],
        help: 'The timezone',
        required: true,
        default: 'UTC'
      }
    },
    fn: (context, args) => {
      if (!args._) return moment.utc(new Date(context)).tz(args.timezone).toISOString();
      return moment.utc(new Date(context)).tz(args.timezone).format(args._);
    },
  });
  

  The formatdatetz plugin is available in GitHub for you to install on top of Canvas. Installation is as simple as:

  ./bin/kibana-plugin install https://github.com/crimsonmacaw/nodejs-canvas-plugin-formatdatetz/releases/download/v1.0.2/canvas-plugin-formatdatetz-1.0.2.zip
  

  Controlling Colour of Text and Images

  We couldn’t use Markdown to control the style of text and images (though you can provide cascade style sheets within canvas, Markdown syntax does not have support for setting HTML class attributes for which style to apply).

  The simplest approach was for us to create a plugin that would allow direct coding of HTML and in a similar way to the Markdown element, allow handlebars expressions for data binding. As SVG images can be embedded directly inside HTML, we could apply the same level of control to dynamically change the images based on data retrieved from Elasticsearch.

  Creating the Visualisations

  Once we had the plugins we wanted, having the control to create something simple as a table with different colours on the rows was now possible.

  

  The first pass design for the security hub dashboard based on the user wireframes (the current dashboard for this is below, and it looks very different)

  The visualisations were now coming together, data was being populated into each of the relevant placeholders, but the look and feel still didn’t feel quite right and a request came in that reflected that.

  “Can we make them look sexier?”

  It was time to take inspiration from the internet, searching for “modern dashboard” on platforms such as Pinterest and Dribbble gave some interesting approaches to designing a dashboard and a different direction was taken.

  

  The security hub dashboard again, it’s now a dark theme, with use of gauges and colour gradients (still not finished, it will change more)

  A number of elements were added to the screen, mostly using the HTML plugin we had created which had a mixture of simple HTML and some complex SVG images dynamically generated.

  The initial feedback was positive, however users that hadn’t been part of the design didn’t understand all the metrics without an explanation.

  “Wow! That looks a lot better, but what do the bottom of the bars mean?”

  Going back to the original requirement of the Airport Security Operations being able to quickly make decisions, meant that although you could produce something that may look impressive, it’s not always the best way to visualise the information. A user should be able to instantly know what is being displayed in front of them without the dashboard having to be explained.

  An Iterative Approach

  Having become proficient in the Canvas expression language, we held a number of workshops with live editing of dashboards with the stakeholders in the room. It was clear that we had moved on from the wireframe constructed before people had seen the tool or based on their current reporting, to inspired approaches on how best to display the information on the dashboards. Stakeholders took the latest versions of the dashboards to end users for feedback and changes were made based on the feedback given.

  Final Dashboards

  The following dashboards were the result of collaboration and creative thinking.

  “The dashboards that Crimson Macaw have created for us using Elasticsearch and Canvas have given Stansted Airport a modern real-time display of our data that was previously held in individual systems. This has created a number of efficiencies in our internal processes that would not have been possible before.” – Stuart Hutson, Head of BI and Analytics, Manchester Airport Group

  Note, the data in all of the dashboards has been randomised for presentational purposes. We created a randomise plugin for Canvas to do this!

  Security Hub

  This dashboard shows the current status of the Airport Security, from people entering and leaving the Airport Security to compliance rates, as well as information about the individual lanes. The shape chosen for the security lanes reflects the physical layout within the Airport Security, so that the operation staff can immediately relate the dashboard to a physical lane.

  

  Security Pod

  This dashboard shows trend information, displaying people entering the Airport Security area plotted against a predicted forecast of passenger turnup and queue wait times in different areas.

  

  Flight Departures

  Very similar to a normal FIDS (Flight Information Display Screen), but with the additional detail of matching the percentage of passengers who have entered Airport Security against the predicted passenger for the upcoming flights.

  

  Train Arrivals

  A large percentage of passenger arrive into Stansted Airport via train and delays to their journeys can have a significant effect on the number of people arriving at Airport Security. A single train will have hundreds of travelers and multiple trains can arrive in a short period after issues have been resolved.

  The dashboard not only displays a typical table of train information, but also shows train arrival time on a timeline. You can see this has changed substantially from the first iteration when we were experimenting with Canvas.

  

  Conclusion

  Canvas is a brilliant tool to enable very impressive, real-time visualisation of data. Although Canvas is currently in beta and may need some additional features, having the ability to expand its core functionality with plugins does feel like the Elastic approach to software.

  Lots of BI tools limit you to graphs and tables with a sprinkling of gauges and other standard visualisations. The simplicity of creating dashboards with Canvas allows Data Engineers and Data Visualisation Specialists the freedom to create visualisations only limited by their imagination.

  ---

  Robert Bruce is a founding partner and Director of Engineering at Crimson Macaw, a Cloud/Data IT Consulting company based in Manchester, UK. Robert has worked in the Data Engineering and Web industries for over 20 years and now works in the cloud technology space.

  
  
  https://www.elastic.co/blog/monitoring-airport-security-operations-with-canvas-and-elasticsearch
• 

  This Week in Elasticsearch and Apache Lucene - 2018-11-23
  • sorangutan  

  1
  1
  Posts
  73
  Views

  

  Elasticsearch

  Remote clusters UI

  Work continues on the remote clusters UI. We have finished the UI layer, and have been working on the app layer to integrate the UI with endpoints. So far the work has been done based on the assumption that add/edit/delete are all persistent settings. Next, we will work on handling transient settings as well as settings that come from elasticsearch.yml.

  Index lifecycle management UI

  Work on this UI is progressing nicely. The flows from the index management app to add a policy to and remove a policy from an index are completed. We also added the ability to see the error stack trace for lifecycle errors and to see the phase definition. Remaining tasks include adding the ability to disable the UI, adding a button to add a policy to an index template from the policy CRUD UI, adding filter buttons in index management to filter on managed/unmanaged and lifecycle phase, and updating autocomplete rules for console to include new ILM APIs.

  Query Script Query

  There is a new query called script_score that allows users to manipulate the score of documents through a Painless script. The query also includes a script context which exposes a number of useful methods including the Decay functions of the function_score query. This new query is the first part of the work to deprecate and remove the function_score query in favour of easier to use alternatives.

  Rare terms aggregation

  We have a PR for a new aggregation that is designed to surface rare terms. This is designed to be used instead of ordering a terms agg by count ascending which has an unbounded error. The rare terms aggregation uses a different strategy to merge the results from different shards to ensure a better precision.

  Build

  As every PR against our repo is built on our CI infrastructure as part of the review process, it is very important to have fast feedback cycles. By splitting the build and running it on multiple worker nodes we were able to reduce the total PR build time by 50% down to 90 minutes. We have also enabled Gradle build scans so we can find more room for improvements.

  Backup of Security

  It’s currently possible to backup the .security index, but it’s not an ideal experience. We have a plan of tasks we want to complete to improve this experience and we have started working through this list.

  We have opened a PR for a “snapshot” privilege and have been thinking about how to allow some low-privileged users the ability to list the .security index, and potentially see some metadata about it, but not necessarily be able to search it, or get documents from it.

  Zen 2

  We now have a storage layer that can atomically write cluster states, and the remaining follow-up work will focus on integrating the new storage layer with the existing Zen2 components.

  We also fixed a long-standing issue where nodes miss applying a cluster state update, indefinitely lagging behind the master in terms of applied cluster states. This can potentially impact other operations in the cluster, for example block indexing operations to wait on mapping updates. The solution is for the master to track applied cluster state versions of each node and to kick a node out of the cluster if it doesn't apply a published cluster state or any of its follow-up published states within some timeout, giving us a strong bound on how stale the cluster state applied on every node in the cluster can be.

  Cross Cluster Replication

  We are also continuing to investigate and benchmark update- and refresh-heavy use cases, to see if the number of refreshes (and merges) can be improved without ruining the indexing throughput. This work has led to two Lucene changes to improve lock contention when indexing happens concurrently to refresh / open reader calls: 1) Improve merge performance in the common case where soft- and hard-deletes are not mixed and 2) prevent blocking on resolving deletes.

  The work to implement "recovery from remote" by bootstrapping the creation of the follower index through the Snapshot/Restore functionality has started by prototyping a CCR Snapshot/Restore repository that gets registered based on available remote clusters.

  Changes

  Changes in 6.4:

  • [CI] Reactivate 3rd party tests on CI [branch 6.4] #35733

  Changes in 6.5:

  • Avoid NPE in follower stats when no tasks metadata #35802
  • [Scripting] Use Number as a return value for BucketAggregationScript #35653
  • SQL: Improve validation of unsupported fields #35675

  Changes in 6.6:

  • BREAKING: Forbid negative scores in functon_score query #35709
  • SQL: Implement NVL(expr1, expr2) #35794
  • TransportResyncReplicationAction should not honour blocks #35795
  • GEO: More robust handling of ignore_malformed in geoshape parsing #35603
  • SQL: Implement ISNULL(expr1, expr2) #35793
  • HLRC: ML Delete event from Calendar #35760
  • [HLRC] Fix issue in equals impl for GlobalOperationPrivileges #35721
  • [GEO] Add support to ShapeBuilders for building Lucene geometry #35707
  • SQL: Implement IFNULL variant of COALESCE #35762
  • SQL: Perform lazy evaluation of mismatched mappings #35676
  • [HLRC] Add support for get application privileges API #35556
  • [Tests] Fix SimpleQueryStringBuilderTests #35784
  • manage_token privilege for kibana_system #35751
  • SQL: Introduce INTERVAL support #35521
  • [HLRC] Added support for CCR Resume Follow API #35638
  • Clean up PutLicenseResponse #35689
  • Clean up StartBasicResponse #35688
  • Add a prebuilt ICU Analyzer #34958
  • [HLRC] Added support for CCR Unfollow API #35693
  • Fix problem with MatchNoDocsQuery in disjunction queries #35726
  • HLRC: ML Get Calendar Events #35747
  • HLRC ML Add Event To Calendar API #35704
  • HLRC: ML Delete job from calendar #35713
  • Add a _freeze / _unfreeze API #35592
  • HLRC: Add "_has_privileges" API to Security Client #35479
  • HLRC for _mtermvectors #35266
  • HLRC: ML Add Job to Calendar API #35666
  • Build: Fix official plugins list #35661
  • HLRC: Add ML delete filter action #35382
  • SQL: Move internals from Joda to java.time #35649
  • Fix phrase_slop in query_string query #35533

  Changes in 7.0:

  • Add a new query type - ScriptScoreQuery #34533
  • Deprecate types in validate query requests. #35575
  • Deprecate types in count and msearch. #35421

  Lucene

  BM25F

  A first implementation of BM25F has been pushed to sandbox. BM25F is an extension to BM25 that supports scoring across multiple fields with different degrees of importance. The way it works is quite simple, for instance say that you want to give title a weight of 5 and body a weight of 1 then BM25F will score across these two fields as if there was a single field that consisted of the concatenation of the body field and 5 times the title field. While this first implementation is already helpful, this is currently implemented via a custom query that ignores the similarity that is configured on IndexSearcher. There is still work needed in order to make it more flexible, eg. by accepting different values of b (the BM25 parameter that controls length normalization) per field and making it easier to use. We hope to eventually leverage this functionality to improve scoring in Elasticsearch when using the cross_fields way of scoring across multiple fields.

  Benchmarks for LatLonShape

  While we would eventually want to benchmark LatLonShape - our new BKD-based support for shapes - with realistic datasets of shapes (polygons, lines, etc), we started simple by running our current benchmark for points with LatLonShape. This already gives us a good indication of how much overhead LatLonShape adds compared with LatLonPoint when only indexing points. See the purple line at http://people.apache.org/~mikemccand/geobench.html if you are curious. LatLonShape is still at an early stage of its development and we hope to bridge some of the gap with LatLonPoint in the near future.

  Faster merging when indexing geo shapes

  We recently enhanced BKD trees so that only a subset of dimensions is indexed, which in-turn helps use BKD trees as a R-tree. We noticed that we are still doing work on non-indexed dimensions which is not necessary. Fixing it is expecting to make merging faster for fields that index shapes. We will hopefully soon see a bump in the indexing speed charts (see above paragraph).

  Less contention when applying deletes

  Deletes may be applied either while indexing or refreshing. Today this is a blocking call, which might block some indexing threads. However blocking is only required when refreshing, in order to make sure that all deletes are reflected in the new point-in-time view of the index. an optimization that only makes it blocking for refreshes has been implemented to solve this.

  Normalization as a first-class citizen

  The ability for analysis chains to only normalize content and skip tokenization, stemming and all other components that don't work with partial terms (think of wildcard queries) always felt a bit like a second-class citizen as you had to check for whether a component implemented a special marker interface to know whether it should be applied for normalization. This was fixed by moving normalization directly to the CharFilter and TokenFilter factory classes.

  Resources around Lucene

  As highlighted in a recent Twitter conversation, the Lucene wiki has a useful page that collects various resources that either describe Lucene's internals or have been influential.

  Other

  - The Lucene 7.6 release process is on pause during Thanksgiving.

  - Nori, the korean tokenizer, could hit a NullPointerException if the part-of-speech attribute is not set.

  - Nori doesn't do a good job with non-CJK tokens.

  - The "flexible" query parser can fail with index-out-of-bounds exceptions with non-default locales.

  - In the general case, counting soft deletes upon merge requires a linear scan. Simon optimized the common special case that a segment consists only of soft deletes, as in that case the number of soft deletes is already known via IndexReader#numDocs.

  
  
  https://www.elastic.co/blog/this-week-in-elasticsearch-and-apache-lucene-2018-11-23
• 

  Keeping up with Kibana: This week in Kibana for Nov 12th, 2018
  • sorangutan  

  1
  1
  Posts
  66
  Views

  

  Here is a quick update on what Kibana team has been working on last week.
  

  Oh by the way, we are hiring!

  • Senior Product Manager, Kibana (Data Visualization)
  • Senior Product Manager, Kibana (Security)
  • Kibana UI Technical Lead
  • Kibana Platform - Principal JavaScript Engineer
  • Kibana - Senior JavaScript Engineer
  • Kibana - Senior Security Developer (JavaScript)
  • UI Engineer - Design
  • Kibana - Visualisations & Vega Engineer

  
  

  Key theme updates

  Security

  Granular Application Privileges progress continues! The PR to allow features to define their own privileges has merged; and the PR to allow features to be hidden per application with a Basic license is going through review. A PR to disable certain UI capabilities based on privileges is also going through review.

  Alerting Service

  After reviewing our notes from other teams we talked to we updated our phase 1 plan with necessary capabilities for developers in order to make the service usable. You can critique the changes to phase 1 through the meta issue on github.

  K7 design

  This week Dave merged ML conversion to SASS (that was a BIG PR!). We are also continuing work on converting filter bar to EUI:

  Geo

  GIS App

  • Continue to work on styling
  • Improvements to join-functionality
  • Added UX controls for setting extent

  

  

  
  

  Canvas Integration

  Planning week! We re-evaluated, but mostly reaffirmed our feature plans for 6.6 and GA in 6.7, but it’s not like we did nothing but meetings. That just isn’t our style…
  

  URL Parameter support

  Joe introduced URL parameter function support and wildly simplified the code around our router. You can once again pass parameters directly from the URL anywhere into a Canvas expression. Oh look, there’s Internet Explorer. Hi there Internet Explorer.

  

  Persistent Grouping

  Robert is getting closer to supporting fully persisted groups of elements. Elements are selected by holding down shift, clicking a bunch of them, and hitting the `g` key. 

  Workpad Templates

  Workpad templates are coming along. The first phase will allow for plugging in new templates on the filesystem. For example, a Kibana administrator could drop a JSON file on the disk and others would simply be allowed to clone it into Elasticsearch.

  We’re also working to add thumbnails now, and we anticipate adding support for auto-generated thumbnails to saved workpads in the near future.

  
  

  New Visualization Editor

  This week the editor design incorporated field statistics similar to those found in discover and machine learning. This will help end-users get a quick glance of their data without the need to swap between discover and visualize.

  

  
  

  Team summaries

  
  

  Design

  The team continues the ever-lasting SASS conversion, working our way into the ui/public and ui_framework (KUI) folders. TSVB got a small facelift during its conversion #24250 (though there’s a small style bug with combo boxes that has fix in the next EUI release). The global date picker in EUI got it’s final round of visual patterns implemented. Hopeful that it will get logic hookups in the coming weeks. We are continuing to test EUI for Framer by adding more components. Typescript in EUI continues.

  Spaces and the upgrade checker are getting some copy changes. 6.5.1 release notes are being finalized along with a blog post for rollups.

  EUI is at 5.1.0

  PRs

  • Added support for nodes as "Action" column headers in EuiBasicTable, which was overlooked in the original change in 4.5.0 (#1312)
  • Updated GlobalDatePicker example to include all Kibana features (#1219)

  • Fixed fullWidth size of EuiComboBox, a regression introduced in 4.7.0 (#1314)

  • Improve TypeScript definitions for EuiTableRowCellProps (#1310)
  • TypeScript types are now published to a eui.d.ts top-level file (#1304)

  • Introduce TypeScript support into the dev & build systems (#1317)

  Operations
  

  PRs
  

  • Run jenkins:unit task with dev flag in order to run license check (#19832)
  • Fix recent non-conforming licenses (#25560)
  • Remove Husky (#24887

  KibanaApp (Visualizations, Sharing, Discover)

  The team focused on finishing several bug fixes for 6.5.1. Other work included fixing reporting tests, working towards using canvas expressions for all visualizations, localization code reviews, and the EUIfication of the filter bar. Here is a screenshot of the progress so far:

  PRs

  • changing vis.API.events (#25280)
  • [I18n] Rename 'i18n-context' to 'i18n-description' in Timelion help (#25637)
  • [I18n] Inject Intl Polyfill to PhantomJS (#25465)
  • fixing map bounds (#25040)
  • Timefilter should return strings instead of moments (#25625)
  • Ensure custom set axis titles are preserved when loading a saved vis. (#24176)
  • Remove experimental flag from Visual Builder (#25634)
  • Add href option in addition to onClick (#25233)
  • Combine common pdf png functions (#25152)
  • [Bug] Fixing flash of unstyled dashboard content (#25559)
  • [I18n] Translate Timelion (#23880)
  • [Flaky test fix] Turn back on data_shared_attributes test (#25156)
  • Fix timelion issue when elasticsearch.shardTimeout is 0 (#25461)

  
  
  https://www.elastic.co/blog/keeping-up-with-kibana-2018-11-12
• 

  Uptime Monitoring with Heartbeat and the Elastic Stack
  • sorangutan  

  1
  1
  Posts
  312
  Views

  

  With the 6.5 release of the Elastic Stack, we are happy to announce the general availability of Heartbeat. Initially introduced in beta in January 2017, Heartbeat is a lightweight shipper — similar to the rest of our other Beats (Metricbeat, Filebeat, Winlogbeat, etc.) — but it's designed for uptime monitoring.

  Heartbeat can operate from inside or outside of your network. All it needs is network access to the desired HTTP, TCP, or ICMP endpoint. Configuration is as simple as supplying Heartbeat with the list of URLs you want to monitor. Heartbeat executes periodic checks to verify whether the endpoint is up or down, then reports this information, along with other useful metrics, to Elasticsearch. This information is automatically displayed in pre-built Kibana dashboards for turnkey uptime monitoring.

  Let's take a closer look at how to set up and use Heartbeat with the Elastic Stack.

  Installing Heartbeat

  To install Heartbeat, you will need to download and extract a pre-built executable for your platform. In this example, I will use macOS, however we support many other operating systems, so you should use the commands that work with your system.

  Download and extract the latest version of Heartbeat:

  curl -l -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-6.5.0-darwin-x86_64.tar.gz
  tar xzvf heartbeat-6.5.0-darwin-x86_64.tar.gz
  

  Configuring Uptime Monitors

  In order for Heartbeat to know which service to check, it needs a list of URLs. This configuration is specified the heartbeat.yml file under the /heartbeat folder. Here is an example for multiple HTTP checks with Heartbeat, which run every 10 seconds:

  # Configure monitors
  heartbeat.monitors:
  - type: http
    # List or urls to query
    urls:
          - "https://www.elastic.co"
          - "https://discuss.elastic.co"
    # Configure task schedule
    schedule: '@every 10s'
  

  In addition to the HTTP/S monitors, Heartbeat can perform TCP and ICMP checks so you can get better insight across different layers of your services. In Heartbeat we can also define additional layers of checks, for instance, with HTTP/S monitors we can check the response code, body and header. With TCP monitors we can define port checks and strings checks.

  heartbeat.monitors:
  - type: http
  # List or urls to query
    urls: ["http://localhost:9200"]
    # request details:
    check.request:
         method: GET
    check.response:
         body: "You Know, for Search"
  # Configure task schedule
    schedule: '@every 10s'
  

  Here is an example for an HTTP body check, where Heartbeat is looking for the string “You Know, for Search” at http://localhost:9200 (the only URL specified in the config).

  On all Heartbeat monitors, we can define additional parameters such as name, timeout and schedule. You can find complete configuration instructions on the Configuring Heartbeat documentation.

  The final step in the configuration is to setup Heartbeat outputs (where to send the data to). Among the supported outputs are self-managed Elasticsearch cluster, Elastic Cloud, Logstash and more. In this example, I am sending Heartbeat data into my local Elasticsearch instance  (“localhost:9200”):

  output.elasticsearch:
  # Array of hosts to connect to.
    hosts: ["localhost:9200"]
   # Optional protocol and basic auth credentials.
   #protocol: "https"
   username: "elastic"
    password: "changeme"
  

  You can find an example file with the full configuration, in the heartbeat.reference.yml file.

  Starting Heartbeat for the First Time

  Heartbeat comes with prebuilt dashboards that provide a lot of value out of the box. Set up the dashboards and run Heartbeat using these commands:

  To set up Heartbeat dashboards in Kibana: (optional, and only needs to be run once)

  ./heartbeat setup --dashboards
  

  To run Heartbeat:

  ./heartbeat -e
  

  As soon as Heartbeat starts to run, it will check the list of URLs you have configured, send back the information to the Elastic Stack, and pre-populate Kibana dashboards.

  Visualizing Uptime Data in Kibana

  When you open Kibana, you will see Heartbeat information in the Discover tab when you select the Heartbeat index (heartbeat-* by default):

  

  Among dashboards, you will find the “Heartbeat HTTP monitoring” dashboard that is populated with all of the information Heartbeat had sent:

  

  This is the default Heartbeat dashboard:

  

  In the upper left corner you will see a table with a list of all Heartbeat-monitored endpoints, along with their response times. In the upper right, you will see the summary of the monitor status codes, and below it additional graphs which represent the percentage of monitors which are up or down, response time over time and number of checks over time.

  

  Heartbeat dashboards are interactive. With the default Heartbeat dashboard, you can easily detect if a specific endpoint is up or down, by clicking on the monitor down section (that is the red slice in the HTTP monitor status piechart located on the upper right corner). Clicking on it will apply a filter on the dashboard, which will display only the monitors which are down.

  

  You can customize Heartbeat Kibana dashboards and visualizations in order to meet your needs. To get started with that, click the “Edit” button on the dashboard top right, click the gear on an individual visualization, and select “Edit Visualization”. You can also create new visualizations based on Heartbeat data, combine Heartbeat metrics with other log, metrics, and APM data sources, and add them to any operational dashboard you like.

  Combining Heartbeat Data with Other Operational Data Sources

  Combing Heartbeat data with additional data sources such as Metricbeat, Filebeat, and APM can provide important information when triaging an outage. Heartbeat will notify you when a specific service is down, Metricbeat and APM can provide you with the potential suspects, it can significantly improve the MTTR and will focus everyone to look at the right place, instead of playing the “blame game” in a typical war room.

  Visit our solution page to learn more about our logging, metrics, and APM solutions.

  Here is an example for a Kibana dashboard which brings together metrics from Heartbeat (top two panels), and Metricbeat (bottom two panels).

  • On the left hand side you can see that whenever Heartbeat detected that our endpoint is down, Metricbeat detects an increasing number of errors via the Nginx module.
  • On the right hand side we can clearly see that whenever our HTTP ping time is increasing, Metricbeat detect high request rate in the NGINX server.

  Observing this coincidence, we can conclude that a massive spike in request rate is triggering Nginx errors, which, in turn, is causing Heartbeat checks to fail:

  

  Alerting and Anomaly Detection

  You can use the alerting capabilities to get alerts and notifications on an outage or a performance degradation based on Heartbeat monitor data. You can also use the Elastic Stack machine learning capabilities to detect any anomalies based on time-series analytics of Heartbeat data. More on that in my next Heartbeat blog.

  Summary

  Monitoring can be a tough job. You need to identify, alert on, and resolve issues quickly. If today you are using multiple disparate tools to gain visibility across your applications and infrastructure, your task is likely even harder.

  Heartbeat adds effective uptime monitoring functionality on top of existing logs, metrics, and APM features in the Elastic Stack. Use it by itself, or combine it with other valuable sources of operational data in Elasticsearch — so you won't miss a beat!

  Heartbeat Resources

  That’s it! Hope you found this overview useful. To get started, download Heartbeat and read its documentation for more in-depth coverage of the functionality. One of the fastest ways to experience Heartbeat is to ship your uptime data to the Elasticsearch Service on Elastic Cloud — the hosted Elasticsearch and Kibana offering from the creators of these projects.

  If you have questions, please engage with us on the Heartbeat Discuss forum, and if you find any issues or would like to file an enhancement request, please do so in Github.
  
  https://www.elastic.co/blog/uptime-monitoring-with-heartbeat-and-the-elastic-stack
• 

  Kickstart Search with the Elastic App Search Reference UI
  • sorangutan  

  1
  1
  Posts
  75
  Views

  

  Elastic App Search provides you with a robust set of API endpoints to build any type of search experience. While modern tools and a blank canvas can be inspiring, we want you to craft functional and valuable things as quickly as possible. And so, we bring you the App Search Reference UI.
  

  Open Sourcery

  The Elastic App Search Reference UI is written in React. It is an open source developer tool that has been woven into the App Search dashboard. Its goal is twofold:

  1. Provide a clean and dynamic search preview to experiment with features like facets, pagination, and sorting.
  2. Generate a quality design and developmental foundation for new search experiences.  

  You can see a live Reference UI by visiting: https://parks.swiftype.info.

  Field Day

  Before we dig deeper into the Reference UI, we should consider a key goal of App Search: provide relevant search. Relevance is the key to the value of search, for both your end-user and your business goals. But what does it mean, to be relevant?

  Relevance is how well a result can fulfill the needs of the user or system that requested it. In technical terms, a query is analyzed against document fields and the most relevant document is returned given how well the field values match the query.

  Code and national parks feel like a nice pair, and so we will take a practical look at the Reference UI through the eyes of the national parks data set. It has a schema containing the following fields:

  {
    "description": "text",
    "nps_link": "text",
    "states": "text",
    "title": "text",
    "visitors": "number",
    "world_heritage_site": "text",
    "location": "geolocation",
    "acres": "number",
    "square_km": "number",
    "date_established": "date"
  }
  

  Within this data set, a park is less vast and wood-y, like this…

  

  When we click Generate a Preview, it appears just like our sample application. And when we search for “old growth"...

  
  
  https://www.elastic.co/blog/kickstart-search-with-the-elastic-app-search-reference-ui

• 

  Elastic APM Java Agent is Generally Available
  • sorangutan  

  1
  1
  Posts
  149
  Views

  

  We are thrilled to announce that our Java Agent is now generally available (GA)! After a successful beta phase, with lots of downloads and plenty of feedback, we are ready to release the production-ready version, 1.0. Note that this version requires at least APM Server 6.5.

  The Java agent is the new member of the Elastic APM agent family, joining Node.js, Python, Ruby, JavaScript/RUM, and Go (a new GA sibling as well). Elastic APM is the Application Performance Monitoring solution from Elastic that helps you to gain insight into the performance of your application and track errors.

  Main Features

  What you get out of the box

  Installing the Elastic APM Java agent is as simple as adding the -javaagent flag and a couple of system properties to your JVM startup command. As long as you are using supported frameworks and platforms, such as Spring MVC/Spring Boot, Servlet API, and JAX-RS, your application will be instrumented and monitored automatically.

  Web Request Visibility

  Get a high-level overview of your Java applications, and drill down to see how your Servlets or Spring Controllers are doing individually:

  

  

  Data Store Visibility

  Using JDBC or Elasticsearch? Learn where your Java application spends time on DB calls, which queries take longest and find out what errors they produce. You might be surprised what kind of queries your ORM mapper generates!

  

  Networking Visibility

  The Java agent also records how long your application is spending on external HTTP calls, so you can tell how much time is spent waiting for services out of your control.

  Distributed Tracing

  If your Java application is just one component in a multi-service setup, your traces will include data collected from any caller and callee that have an APM agent installed on them as well. As long as you are using one of the supported networking frameworks, the agent makes sure that it passes correlation IDs along so that you can see which services are involved in serving a request and which one is to blame when the request was slow. You can read more about distributed tracing in the Elastic APM 6.5.0 release announcement.

  Enrich and Extend

  For anything you want and don’t get out of the box

  Is your Java application not using any of the supported technologies? Do you want better visibility to specific parts of your internal code? No worries, we provide the infrastructure that can help you trace virtually anything.

  Customize your APM solution using our API

  You can trace any method you wish using our public API. This API lets you customize and manually create spans and transactions, as well as track errors.

  For example, our annotations API can be used as follows:

  import co.elastic.apm.api.CaptureSpan;
  import co.elastic.apm.api.ElasticApm;
  import co.elastic.apm.api.Span;
  
  public class MyImportantBusinessLogic {
      
      @CaptureSpan("Main-business-logic")
      public void myVeryImportantMethod(String someInfo) {
          ElasticApm.currentSpan().addTag("info", someInfo);
          // My important logic comes here
      }
  }
  

  To get your code show up as a span:

  
  
  https://www.elastic.co/blog/elastic-apm-java-agent-is-generally-available