New Relic News

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you may not be able to execute some actions.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

New Relic News

How to Run an Adversarial Game Day
• sorangutan

1

1
Posts

94
Views

Adversarial game days help New Relic apply chaos engineering to our reliability best practices—here’s how they work.

http://feedproxy.google.com/~r/NewRelic/~3/LDJ5nd456SE/
Best Software Developer Events and Conferences: A Global Guide
• sorangutan

1

1
Posts

87
Views

When it comes to industry events for developers and other tech professionals, the options can be overwhelming. Here’s our list of essential conferences, hackathons, and more from all over the world.

http://feedproxy.google.com/~r/NewRelic/~3/FhcEq2xVJc4/
Building An Inventory Management Database with The Kubernetes API Server And Custom Resources
• sorangutan

1

1
Posts

80
Views

The New Relic Container Fabric team's search for an inventory management solution points to new ways to get value from Kubernetes core components.

http://feedproxy.google.com/~r/NewRelic/~3/JgKr6xvqDgs/
How the New Relic Infrastructure Team Automated Its Windows Build-and-Deploy Pipeline
• sorangutan

1

1
Posts

76
Views

Here’s how we reduced toil by changing the release process of our Windows agent from half-day-long manual process to a five-minute Jenkins job.

http://feedproxy.google.com/~r/NewRelic/~3/VNOPfOmi4dc/
Getting Started with New Relic Mobile: Best Practices That Set You up for Success
• sorangutan

1

1
Posts

56
Views

Set up your team to get the most from New Relic Mobile with this set of field-tested best practices.

http://feedproxy.google.com/~r/NewRelic/~3/td97p4TV_pM/
Introducing New Relic’s Ignite Program for Early Career Engineers
• sorangutan

1

1
Posts

76
Views

There are significant barriers to hiring and supporting new engineers—the Ignite program is New Relic's answer to these challenges.

http://feedproxy.google.com/~r/NewRelic/~3/zUrwAsoSLlU/
Breaking to Learn: Chaos Engineering Explained
• sorangutan

1

1
Posts

64
Views

What chaos engineering is, and how you use it to make your complex systems more resilient¬.

http://feedproxy.google.com/~r/NewRelic/~3/z4GWRkmXB3U/
3 Ways New Relic SREs Use the New Relic Platform
• sorangutan

1

1
Posts

71
Views

Our SREs use the New Relic platform to operationalize services, experiment on services with load testing, and lead teams through incident responses.

http://feedproxy.google.com/~r/NewRelic/~3/cwuaSgqvb3U/
FutureStack Berlin Video—Insights on Digital Transformations from Flixbus, OneFootball, and AWS
• sorangutan

1

1
Posts

64
Views

Watch customer presentations from Flixbus and Onefootball from our FutureStack18 Berlin conference.

http://feedproxy.google.com/~r/NewRelic/~3/7jGRBMy_9f8/
6 Things New Relic Believes About Modern Monitoring
• sorangutan

1

1
Posts

62
Views

What’s really important in the world of application performance management, and how New Relic addresses them.

http://feedproxy.google.com/~r/NewRelic/~3/8lyxUbkFHEY/
10 Secrets You Should Have Learned With Your Software Engineering Degree—or at Coding BootCamp—but Probably Didn’t
education developers culture software development coding tips dev engineers • • sorangutan

1

1
Posts

100
Views
Originally published on June 3, 2014, this post was expanded and updated on January 2, 2019.

It’s an all too common story in many professions: You go to school for years and walk out with a freshly printed diploma. You snag your first job—and immediately, you’re buried under a pile of unwritten rules and day-to-day complexities that no one bothered to warn you about.

Programming jobs are no exception. In fact, programming may be the poster child for this phenomenon. That’s one reason for the proliferation of coding boot camps: those accelerated schools that promise to teach you everything you need to know about coding in a matter of months.

As it turns out, whether you’ve earned a shiny new Computer Science degree or a boot camp certificate, odds are you’re still not 100% ready for your first real job writing software. I’m still continually surprised, for example, to see how many newbies, and even some experienced programmers, still edit their code in Notepad or Pico and don’t know how to use version control.
If you don’t want to be “that person”—or don’t want to be that person anymore—the 10 basic skills listed here can help you get started doing real work with less hand-holding and fewer boneheaded mistakes. We’ve divided the list into three categories: technical skills, habits and mindset, and soft skills.

Technical skills

These five technical skills should be part of any beginning developer’s toolbox. If you didn’t learn them in school or boot camp, it’s time to get up to speed now:

1. Version control systems

This may be the biggest omission in a typical computer-science curriculum. Every graduate of any school should know the basics of using Git or another version-control system to track computer source code—just as she should know how to create programs in languages like Ruby, Python, or Java. Every programmer should know how to create repositories, edit and commit code, and branch and merge effectively as part of a project workflow using Git.

Acquiring this skill is not just a practical issue; having your work backed by a good workflow affects how you write code. Continuous integration (CI) relies on version control. You write code in discrete, more manageable tasks. Without version control, refactoring of code can be dangerous.

Version control systems are ubiquitous within software development teams these days, and a new employee joining a company without understanding the basic mechanics of Git is going to be at a disadvantage. Knowing about version control systems also means that you’ll know better than to work for an organization that doesn’t use one.

For an introduction to Git, check out Pro Git by Scott Chacon, available in print and free online. For a wider survey of version control concepts and systems, see Version Control by Example, by Eric Sink, also available free online.

2. HTML and CSS basics

If you’re working in app development these days, chances are you’re working in web apps, whether or not they’re on the public internet. The ability to create functional HTML documents is an assumed skill—similar to making a Microsoft Word document (or a Google Doc). And these days, working knowledge of cascading style sheets (CSS) goes along with that HTML expertise.

The key knowledge here, besides the mechanics of well-formed HTML documents, is not understanding arcane browser-specific tricks and tags, but rather thinking in terms of semantic document markup and knowing how to separate document structure from presentation.

3. Developer toolkit: IDEs, editors, and CLI tools

A carpenter would never complete an apprenticeship knowing how to use only a hacksaw, so it’s astonishing that computer science programs and coding boot camps can turn out graduates who know only Notepad or Pico.

Programming tools exist to manipulate source code and other development resources to make a programmer’s life easier—often much easier. The Unix command line and standard Unix tools like the bash shell, find, grep, and sed should be part of every programmer’s knowledge set. Heck, even programmers using Windows can use these tools starting with Windows 10.

For an introduction to working with the command line, I recommend Brian Hogan’s Small Sharp Software Tools.

4. Regular expressions

Regular expressions constitute a powerful mini-language that programmers can use to find patterns in text strings, in ways that simple string searching can’t match. Unfortunately, regexes (as the term is abbreviated), get a bad rap in the programming world from developers who seem to be afraid of the syntax.

Don’t listen to the haters. If part of your program, for example, must ensure that a part number in a database has five letters, a dash, and a digit, then using a regex like /^[A-Z]{5}-\d$/ to validate the pattern is how you’re going to do the job. Regular expressions can be hard to read if you’re not familiar with them—but you didn’t come out of the womb knowing what f = (9/5) * c + 32 meant, either.

5. SQL

The era of storing data in flat files is over. Everything goes into and out of a database—and SQL, or Structured Query Language, is the language that’s used to retrieve it.

As someone said to me at a meetup once, “All the SQL I know I learned on the job. Why are databases an elective in computer science programs? What doesn’t use a database?”

SQL is a declarative language, not a procedural language, which means that it requires you to learn a new way to think about solving problems. Every programmer should understand the basics of database normalization and be able to do SELECTs (including basic INNER and OUTER JOINs), INSERTs, UPDATEs and DELETEs.

The process of learning SQL and regular expressions has another huge benefit: It requires you to think in terms of different metaphors. SQL is all about operations on sets of data, not looping and branching the way imperative programming works. Regexes are about finding patterns in text and considering what parts of the data you want to use and transform. Adding new paradigms and different ways of thinking help sharpen a developer’s problem-solving skills.

There are many SQL tutorials online, but SQL Zoo is excellent, and I also like Allen Taylor’s SQL For Dummies. For next steps in your SQL learning, I suggest Stephane Faroult’s The Art of SQL.

Mindsets, habits, and soft skills

Being a competent programmer isn’t just about languages and tools. New programmers need to know more than just specific pieces of software and languages. They also need to understand how to communicate with fellow software developers, how to stand on the shoulders of their forebears, and how to minimize errors in their work. That’s where these soft skills loom large:

6. Clear and effective written communication

Working as a programmer is more than just coding in a programming language. You also have to write release notes for your projects. You write commit messages for version control. You write tickets for bugs in the system. All of these tasks, and many more, require clear and effective English (or some other human language) communication—a skill that computer science programs seldom emphasize.

As more jobs become remote, and more teams connect via email, collaboration platforms like Slack, and ticketing systems, the ability to communicate clearly and effectively in written English becomes even more important.

7. Using libraries and other existing code

New developers typically learn how to solve problems by writing their own code, but a wise developer knows when to leverage existing code to solve common software problems. No developer should waste time trying to write code to extract the host name from a URL, for example, when libraries of proven, debugged, and thoroughly tested code are readily available.

(For a look at some of these common coding tasks, see my post on Seven Things You Should Never Code Yourself.)

8. Defensive programming

Even the best programmers are fallible; much of the world is out of our control, and things continue to have a habit of going wrong one way or another. Defensive programming is about coming to terms with these simple truths.

If things didn’t go wrong, we wouldn’t have to check file opens for success, for example, or assert that customer IDs are valid integers, or even test our code to make sure that it works properly.

Programmers need to grasp that compiler warnings are helpful tools that make life easier—not nuisances to avoid. In fact, wise developers enable as many automated quality checkpoints as they can to find potential problems in their code. Every programmer should also know why every PHP program should start with error_reporting(E_ALL), and why all JavaScript programs should incorporate the “use warnings” directive.

Another important aspect of defensive programming is having a security mindset. In a world where most apps are web-facing, programmers must consider the possibility of input that’s not just unexpected, but actively malicious. SQL injection attacks, for example, are predicated on programs that blindly use untrustworthy outside data.

9. Debugging

All programmers should be able to debug with an interactive debugger or by sprinkling print statements liberally throughout the code. The ability to track down a problem through stepwise refinement is too important to be left for programmers to learn by the seat of their pants.

Notably, the ability to narrow down problems to find root causes goes hand-in-hand with the ability to communicate effectively. Writing an effective bug report requires finding the core problem, but it also demands the ability to communicate clearly so others can understand what you’re talking about.

10. Teamwork

Few programming jobs allow you to work entirely on your own, and even those programmers who do work on their own don’t have the benefit of learning from others. In almost all cases, your code must interact with other programmers’ code, and in many cases your code will actually be intermingled with code from others. No matter how talented they may be, programmers who can’t collaborate on projects with others will have drastically reduced or even negative productivity, dramatically lowering their value to their organizations.

Bonus secret: Working on existing code

In Computer Science school, every class assignment is a new, greenfield project. That’s not how it works in the real world. The first thing that happens to new software engineer hires is an assignment to fix ticket #8347 in the bug tracking system. After that, they have to add a small new complementary feature to an existing system with an established codebase. Writing new code to solve new problems will come months later—if you’re lucky.

Here’s one way that schools—whether in a Computer Science degree program or at a programming boot camp—could make assignments more like the real world of software development: After the class does assignment #5, Alice would review Bob’s code as assignment #6, discussing what’s good about it and what could be improved. Then, assignment #7 would be Alice extending the functionality of Bob’s code.

If the original program is fragile or broken, then it falls on Alice to fix it up—and of course, badmouthing Bob or his code is forbidden.

Another way to work on these skills is by contributing to open-source projects. Working in open source involves many of the same skills discussed in this post:
- Version control
- Working on remote teams
- Written communication
- Release-based projects
- Taking advantage of existing code
- Building atop existing code
- Test-driven development
For ideas on how to get involved with open source, even as a newbie, see my post on The Beginner’s Guide To Contributing to Open Source Projects.

Double-bonus secret: We’re all learning as we go!

Think of this last point as a tidbit for a Computer Science commencement speech: New developers need to know that we’re all learning as we go, and there’s nothing wrong with asking for help. I’ve seen new software engineers who are embarrassed to look in manuals or on the internet for arcane syntax or specific calling conventions for functions.

Don’t be embarrassed. As a software developer, your job is to solve problems, not to memorize the sequence of arguments to the substr function. And don’t worry, over time and with enough practice, you’ll find that you need to look in the manual less and less.

However, it’s not all about finding answers online and pasting them into your code. Looking up answers to questions on StackOverflow is good as far as it goes, but it’s important that you understand exactly what you’re cutting and pasting into your code. One trick is to avoid direct cut-and-paste operations from online sources. Instead, retype it by hand. This helps ensure that you read every character that you put into your program.

If you’re a new developer, whether you’re just getting out of school or making a career change later in life, I hope this post has sparked some ideas and areas for further exploration. If you’re an experienced software engineer, then I hope that this list has reminded you of something important, or that it inspires you to help out junior developers in your organization or in your community.

Finally, I want to recommend two essential books for all software developers: Code Complete, 2nd Edition by Steve McConnell; and The Pragmatic Programmer: From Journeyman to Master, by Andy Hunt and Dave Thomas. Both contain timeless advice about the practice of software construction and belong on every programmer’s bookshelf.
Looking Back at New Relic’s Amazing 2018
kubernetes community aws lambda new relic news and produc customers distributed tracing futurestack total economic impact • • sorangutan

1

1
Posts

78
Views

Does a blog post dedicated to New Relic’s amazing 2018 qualify as bragging? Maybe, but New Relic really did have a remarkable year in almost every way, from business growth and industry kudos to rolling out a series of innovative new products and features designed to help our customers get the real-time insights they need to innovate faster.

Muhammad Ali once said it’s not bragging if you can back it up, and that’s good enough for us. So let’s get to work—there’s a lot of news to recap!

Success by the numbers: 17,000 customers and counting

Let’s start with a quick look at a fundamental measure of New Relic’s success as a business: Today, more than 17,000 companies—including half of the Fortune 500—rely on New Relic to solve some of their most urgent and sensitive business technology challenges. That group includes more than 2,000 new customers added during 2018; it also includes more than 700 firms with at least a $100K investment in the New Relic platform this year, compared to 517 firms in this key enterprise customer cohort the year before.

Kubernetes Cluster Explorer makes its debut

New Relic is always working to make our set of applications even better. And in 2018 we were constantly making waves—innovating, listening to customers, and continually improving our products.

Take our new Kubernetes Cluster Explorer, announced earlier in December: It’s an intuitive yet powerful tool that helps our customers monitor and manage the health and performance of their Kubernetes environments.

The rollout of New Relic’s Kubernetes Cluster Explorer was one highlight among many during a year packed with innovative upgrades to the New Relic Platform

Kubernetes Cluster Explorer makes it easier to engage with Kubernetes performance data—whether scaling up to assess the overall health of a cluster or drilling down to inspect and troubleshoot a specific pod, node, or container. As our customers grow and scale their Kubernetes environments, they need better ways to manage complexity and to troubleshoot problems before they impact users: Kubernetes Cluster Explorer offers an elegant yet practical solution.

Distributed tracing anomaly detection

But that’s only the latest innovation. In November, we rolled out distributed tracing anomaly detection to customers using New Relic APM Pro. Distributed tracing gives customers a powerful new way to address the growing complexity of their distributed application architectures—tracking the performance of a single request across all of the services and microservices within a system, pinpointing sources of latency, and revealing opportunities to fix errors and to optimize the underlying code.

Support for AWS Lambda

New Relic also addressed the growth of serverless, announcing APM support for AWS Lambda in November. Many New Relic customers are moving to add serverless architectures to support their application modernization efforts, but traditional system and application metric solutions don’t always work well in serverless environments. Adding APM support for AWS Lambda—the market’s most significant Functions-as-a-Service (FaaS) offering—is a key step in extending modern monitoring techniques to serverless environments.

More 2018 rollouts

Of course, those weren’t the only new capabilities New Relic rolled out this year. For New Relic Mobile customers, there was backend API monitoring—giving users new abilities to resolve HTTP issues for business-critical endpoints, and supporting closer collaboration between mobile development and backend teams.

New Relic Infrastructure added an integration with StatsD—giving users the ability to monitor metrics sent to StatsD from New Relic Insights. Infrastructure customers also got stepped-up multi-cloud monitoring support with new integrations for Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform, and another long-awaited set of capabilities with on-host integration for Kubernetes.

Additions to New Relic Browser included the rollout of a JavaScript error analytics beta designed to help customers cut the guesswork out of JavaScript troubleshooting, so they can find and fix errors faster.

New Relic Insights added a Dashboard API designed to give users programmatic control over their organizations’ dashboards—the starting point for a wide range of unique and often very powerful capabilities.

New Relic Alerts got a host of enhancements—ranging from outlier detection for NRQL conditions and a new NRQL-condition UI to beefed-up incident context and improved preview charts—that take full advantage of the platform’s increasingly versatile intelligence capabilities.

Last but not least, New Relic Synthetics customers got acquainted with containerized private minions (CPMs), designed to simplify the process of configuring, deploying, upgrading, and auto-remediating Synthetics minions—including the ability to run CPMs as a non-root user.

Whew! You may call it a long list of new and improved capabilities— we call it business as usual for New Relic’s product and engineering teams.

Global growth—and a new acquisition

Global growth is also business as usual for New Relic, which has long served customers on six continents—give us a call, Antarctica! But 2018 turned out to be a huge year for New Relic’s European presence, as we opened a new EMEA HQ in Dublin, Ireland and a branch office in Paris; and partnered with IBM Cloud to open a dedicated EU region data hosting option located in Frankfurt, Germany.

New Relic executives preside over the official opening of New Relic’s EMEA HQ in Dublin, Ireland.

New Relic also turned to Europe to acquire technology and team members from Belgium-based CoScale in October. The CoScale team brings a cutting-edge body of work around deep Docker, Kubernetes, and OpenShift monitoring, and members of the CoScale team—including co-founders Stijn Polfliet and Frederick Ryckbosc—are joining our growing European Development Center in Barcelona.

Europe wasn’t the only location for New Relic’s 2018 expansion. We also celebrated the opening of New Relic’s East Coast HQ in Atlanta, where the New Relic team grew to more than 30 employees—with room for up to 50 more—to be closer to some of our most important customers.

On the other side of the world, New Relic expanded its presence in Japan by launching a Japanese joint venture with Japan Cloud.

New resources for our developer community

New Relic has always maintained a special relationship with application developers and DevOps practitioners. In 2018, we took that relationship to the next level with the brand new (and already extremely popular) New Relic developer program. Intended to be a journey towards openness, simplification, and engagement, the program gives developers and DevOps practitioners access to a rich catalog of reference code, documentation, examples, tutorials, and other resources. It’s all designed to help devs ease the process of moving data into and out of New Relic, automating New Relic into a team’s existing workflows, and other essential tasks.

FutureStack goes global—and stays local

In 2018, it was especially gratifying to see New Relic’s FutureStack18 event series emerge as a must-attend event for so many software development experts and practitioners around the world.

In fact, this year’s roster of FutureStack events—much like New Relic’s customer community—expanded into a truly global affair. It included flagship events in cities on three continents, including San Francisco, London, and Sydney/Melbourne. At all these events, DevOps teams from some of the world’s most innovative businesses shared how they leverage New Relic to drive innovation and to tap into the full potential of their companies’ digital transformations.

New Relic Founder and CEO Lew Cirne addresses the FutureStack18 opening keynote audience.

For many New Relic customers, however, our FutureStack Roadshow events turned out to be exactly what they needed: compact events held in host cities near them throughout North America, Europe, Asia, the Middle East, and the Pacific Region. The Roadshows offered the same, high-quality FutureStack content and New Relic University training courses, along with the same great networking opportunities—but with far fewer travel hassles.

Of course, FutureStack wasn’t the only place where New Relic was able to connect with our customers. New Relic team members and customers spoke at breakout sessions, gave product demonstrations, hosted lightning talks, and answered questions about the platform at industry events such as AWS re:Invent in Las Vegas; KubeCon + CloudNativeCon North America 2018 in Seattle; DockerCon 2018 in San Francisco; and our own 2018 Well Instrumented World Tour, which covered 17 cities in partnership with this year’s AWS Global Summit series.

But wait, there’s more … isn’t there?

Crazy as it sounds, we probably left out more highlights than we managed to squeeze into this blog post—being named to Gartner’s Magic Quadrant for Application Performance Monitoring Suites for the sixth consecutive time, for instance. But you get the point: A lot of important things happened at New Relic in 2018, and we couldn’t resist sharing them
OOO, It’s Time For New Relic Out-of-Office Replies
culture holidays travel vacation • • sorangutan

1

1
Posts

83
Views
Well, it’s mid December—already!—and that can mean only one thing: ‘Tis the season of the “out of office” reply.

We’re into our fourth year of gathering the top “OoO” messages from New Relic employees. This year, the best have reached new levels of snark and wit. Hopefully, you’ll find some inspiration for writing your own when you finally check out for the rest of the year or take your next vacation.

Don’t miss our previous OoO message coverage!

A real puzzler

Senior Graphic Designer Leah Purvis, didn’t make it easy for us to figure out how to reach her during her vacation. In fact, she made it downright difficult.

Hi, hi! I’m not here right now. If by some strange phenomenon there is a design emergency, my number is below. The highlighted squares, when solved, will give you the last nine digits. The first digit? It would help to know that I am from the land of crop circles, mister glass, and dead people.

Into the wild

Henry Shapiro, VP of product management, left us wondering if we should send out a search party.

Hi there,

I’m on vacation, motorcycling through the high deserts of Oregon and likely getting bitten by snakes and/or contracting diphtheria as I am forced to abandon my party and ford the river with just one of several remaining oxen. Should I survive, I will return to the office Wednesday, July 18th.

Wishing and hoping

Associate Communications Specialist Natalie Wiser desired nothing more this year than to make the cut for this post. Her drive to do so exceeded her excitement for her actual European vacation. She had nothing to worry about, though; including her message was an easy choice for our editors.

Hola,

I am out of office embarking a daring (and delicious) European adventure! I wanted to make a very clever OoO message so I would make the New Relic annual OOO blog post, but I couldn’t think of anything. Instead, I shall list all of the Spanish words I have learned in preparation for this trip:
- Empanada
- Cerveza
- Vino
- Queso
- El menú
…yup, seems like all the essentials to me!

An explosive good time

We were a bit worried when Campaign Manager Shay Loanzon went on a Hawaiian vacation during the Kilauea volcano eruption. But she was kind enough to address that in her message:

Thanks for reaching out. I am unable to respond to your message right away … because I’m probably on the beach, on a zipline, or in a food coma from all the Hawaiian food.

But don’t worry, I’m far from the erupting volcano and will be able to get back to you as soon as I return.

For any urgent explosions, please contact my manager.

You only live once

It can be hard pull away from the office, but we commend Ali Gerrard, Senior Product Marketing Manager, for making time for herself:

Hey,

Kicking off 2018, I’ve decided it’s time to be #bold and try new things. To that extent, I am currently out of the office on this thing the HR department calls “vacation.” (Don’t worry, I had to Google it, too).

My coworkers have given me explicit instruction that “vacation” requires vacating from email, Slack, and all-things work-related during this respite.

Spriiiiiiiiiiing break!

Senior Technical Marketing Engineer Adam Larson embarked on a Mexican getaway, and he proved that he may never willingly grow up:

Pretending I’m not turning 30 and going to Cabo to live like a teenager on Spring Break. Back on June 1st.

O Canada

In an OoO blog first, Henry Shapiro graced us with a second entry in this year’s round-up. This time around, he couldn’t resist poking fun at our neighbors to the north.

Hi there,

I’m in Canada!

For those of you familiar with Canada, you know as well as I that both electricity and telecommunication are in limited supply here. Your best bet for getting in touch with me is to send a horse-drawn carriage across the border with a note attached to its saddle.

Upon receiving the note, I can send a ceremonial French-Canadian dove to communicate my reply. However, this process (or so I am told) takes anywhere from 4-6 weeks, and by that time both the dove and horse may have perished.

Poutine dreams

And finally, it wouldn’t be our annual OoO blog post if I didn’t close with my own message. Just hours after FutureStack18 in San Francisco, I split for a much-needed break in Montreal. (And yes, despite Henry’s last message, we do love Canada.)

I am currently suffering from post-FutureStack-itis. Symptoms include: Wanderlust and cheese and wine cravings. My doctor has recommended I take some time off in a locale that can cure these two very specific needs, so I am self-medicating in Montreal for a few days.

I pray you don’t suffer my fate. Your thoughts and prayers are feverishly appreciated.

We know there’s more where these came from! Send us your favorite OoO messages at @NewRelic using #OutofOffice
Getting Started with New Relic Insights: Best Practices That Set You Up for Success
new relic news and produc new relic insights best practices dashboards nrql • • sorangutan

1

1
Posts

171
Views
The following is adapted from a series of best practice guides that reflect the New Relic Community Team’s work with thousands of customers—identifying challenges and coming up with actionable solutions for organizations getting started with New Relic Browser. Also check out our guides to getting started with New Relic APM, New Relic Infrastructure, and New Relic Browser, and discover the New Relic Community Team’s latest Best Practice Guide offerings.

The New Relic platform generates astounding quantities of data about your applications and environments. The insights within those data flows can transform your ability to make decisions, to stay competitive, and to keep your applications and infrastructure running at peak performance.

New Relic Insights is your key to unlocking the full value and potential within all of this data. Insights doesn’t just collect and manage data; it performs a wide range of analytical tasks, and it presents insights using visualizations and other highly intuitive, easy to use tools.

Working with Insights can be a complex task—but it doesn’t have to be. In this guide, we’ll get you started with our most popular and useful tips for working with New Relic Insights, based on our work helping thousands of actual users. We’ll also point the way to documentation and other resources you can use to continue your path to mastery with Insights and other New Relic applications.

(If you’re completely new to New Relic, we suggest starting with this primer and series of short videos that give New Relic newbies a quick but thorough introduction to our family of applications, the technology that powers them, and their place in modern software development.)

Getting started and organized with New Relic Insights

NRQL -> Widgets -> Dashboards -> Data Apps

There are four basic building blocks to the New Relic Insights user experience. It’s important to understand how these work together to make Insights powerful and flexible, while also helping Insights stay intuitive and relatively easy to use:
- NRQL (New Relic Query Language) is a SQL-like query language that is optimized for use with the New Relic platform. (We strongly recommend taking the time to learn more about creating NRQL queries—a step that will help you to extract a wide range of business-critical insights from your data within New Relic Insights.)
- When you run a NRQL query, it can return the results in the form of a data visualization, such as a chart, table, or histogram, among many other options. We refer to these as widgets in the context of the New Relic Insights.
- Widgets can be combined to create dashboards that display tailored information about your application’s performance, your users’ actions, or a combination of them.
- Finally, data apps are collections of linked dashboards that create a curated, application-like experience for Insights users. You can also use the FACET widget to link to a pre-filtered dashboard—giving users an intuitive (and easily implemented) way to navigate between your dashboards.
Learn more about data apps and dashboards in New Relic Insights

This video tutorial offers a good introduction to how data apps and dashboards work together in New Relic Insights, along with tips on how to create dashboards:

The New Relic Insights documentation provides detailed guidance on how to Create and Edit Insights Dashboards as well as how to use Insights data apps to build a collection of linked dashboards.

Get familiar with the Insights dashboard API

Most DevOps teams depend on APIs to distribute and share information between the systems they manage and across their organizations. The Insights dashboard API serves the same purposes, as it:
- Gives you programmatic control over your dashboards
- Makes it easier for other users within your organization to see and configure dashboard and widget data
- Ensures that teams can benefit from a shared understanding of your Insights data.
These capabilities are especially important since DevOps teams in different parts of an organization often need to monitor and solve the same problems. Due to the structure of their New Relic accounts, teams can’t necessarily access or even see how other teams are working with their Insights dashboards—a situation that can create confusion and needless duplication of work. Teams can close this gap by using the API to create JSON “templates” of their dashboards, which they can store in a shared repository and push to any other team or account in the organization.

When you start working with the Insights dashboard API, keep one important point in mind: Create shared dashboards that reference default attributes, not custom data. Other accounts may not have visibility into your custom data attributes, so they may not get value from shared dashboards that use custom data.

The New Relic Insights Dashboard API—Get More Programmatic Control Over Your Dashboards offers an excellent introduction of the Insights dashboard API and its capabilities. In addition, New Relic University has a course on Managing Insights Dashboards with the New Relic REST API that can sharpen your dashboard management skillset.

Add Dashboard descriptions with notes

Dashboard notes are a great way to add comments to your dashboards, so you can help other users understand the purpose of a dashboard or explain how to use it. Dashboard notes support Markdown, so you can add titles, headings, images, and other formatting elements.

Understanding your New Relic Insights data

Use the data formatter to set default numeric value formats

Insights automatically formats the data returned by your NRQL queries. For example, when a query returns long integers that can be read as a unix timestamp, Insights will convert them to human readable values. The Insights data formatter gives you control over the default formats for attributes with numeric values, including rounding, comma usage, and the number of digits visible after a decimal point.

Keep in mind that changes you make with the data formatter always apply on an account-wide basis.

Get a detailed introduction to using the New Relic Insights data formatter with this New Relic University tutorial.

Explore and understand attributes

New Relic agents collect and send a wide variety of data to New Relic Insights. The Attribute Dictionary helps you to understand these data types so that you can build more useful and efficient queries, and understand the results of your queries.

Using the Attribute Dictionary is easy: Select the event type or the data source you want to explore, and you’ll get a detailed definition of every attribute that is collected from that event or source.

Metric explorer and event explorer

Sometimes you need to drill into specific metrics, like the CPU percent utilization per host for a single application. In such cases, you’d use the Insights metric explorer to pick an application and dig deeper into each of the metrics that it reports. Add these charts to your dashboards for a centralized view of metric data alongside your NRQL event charts.

New Relic maintains an API explorer – metric names application that will return a list of known metrics and their value names for a given set of parameters. You can also learn about requesting and using metric values in the documentation.

Using Insights to make your data actionable

Create NRQL alerts

When customers begin working with NRQL, they often discover that changes in query results may point to underlying application or infrastructure issues that require immediate attention. Use NRQL to set an alert on any query that returns a single numeric value —giving your team another channel for monitoring and making use of these important insights.

New Relic support documents include detailed guidance on creating NRQL alert conditions.

Send custom data

Insights is extremely powerful thanks to its flexible and highly extensible architecture. While the New Relic platform collects a huge variety of standard data attributes, New Relic Insights also supports the ability to collect and process custom data attributes. Insights gives you several ways to send custom data—through an agent, for example, or directly through the Insert API. Once you load your custom data into Insights, you can use it to build dashboards and to take advantage of most other standard analysis and reporting capabilities.

Begin with a thorough overview of how to send custom data to New Relic Insights. You can also learn more about using the Event API to send custom data to Insights.

Ready to expand your knowledge of New Relic Insights?

Looking for more best practices and tips for New Relic Insights? Check out the Insights level up category pages in the New Relic Explorers Hub, and discover a wealth of community-generated resources on these and related topics. When you’re ready, test your Insights knowledge by taking our Insights Best Practices Quiz and earning your proficiency badge!
How Kafka’s Consumer Auto Commit Configuration Can Lead to Potential Duplication or Data Loss
apache kafka troubleshooting software engineering • • sorangutan

1

1
Posts

121
Views
At New Relic, we’ve built some critical parts of our pipeline with Apache Kafka. Over the years, we’ve hit plenty of issues and devised best practices for managing our Kafka clusters. And we’ve learned a lot about how Kafka works—both effectively and not so effectively—along the way.

One fundamental problem we’ve encountered involves Kafka’s consumer auto commit configuration—specifically, how data loss or data duplications can occur when the consumer service experiences an out of memory (OOM) kill or some other type of hard shutdown.

Let me explain this behavior and what it means for Kafka users.

(Note: This post assumes familiarity with the basics of Kafka, including producer and consumer groups, topic partitions, and offsets.)

A look at the problem

Every message your producers send to a Kafka partition has an offset—a sequential index number that identifies each message. To keep track of which messages have already been processed, your consumer needs to commit the offsets of the messages that were processed.

Committed offsets in a topic partition

Unless you’re manually triggering commits, you’re most likely using the Kafka consumer auto commit mechanism. Auto commit is enabled out of the box and by default commits every five seconds.

For a simple data transformation service, “processed” means, simply, that a message has come in and been transformed and then produced back to Kafka. For any other scenario, we’d consider the message to be unprocessed. This is important because if you consume <InputMessage>, but your service dies before managing to produce <TransformedMessage>, you don’t want to commit the offset for that input message—you need to let it get picked up again.

This gets to the root of our problem because, unfortunately, that’s not how auto commits work. The Kafka consumer has no idea what you do with the message, and it’s much more nonchalant about committing offsets. As far as the consumer is concerned, as soon as a message is pulled in, it’s “processed.”

So now imagine that your consumer has pulled in 1,000 messages and buffered them into memory. Then an auto commit fires, committing the offsets for those 1,000 messages. However, say your service has now used too much memory and is promptly shut down by an OOM kill signal before all the messages are processed. Hundreds of those messages will never be processed. That’s data loss.

The inverse situation is also possible. You could successfully process those 1,000 messages and then experience a hard shut down before committing the offsets. In this scenario, you’ll re-process hundreds of messages on another instance after your consumers rebalance. That’s data duplication.

Ok, so what can I do about it?

The bad news is that there aren’t any easy fixes here. Fundamentally, this is a problem of weak consistency guarantees. What all Kafka users want is exactly-once processing—a guarantee that you will consume and process messages exactly once. Kafka version 0.11 attempts to solve this problem and has made things slightly better. It’s possible to write an exactly-once pipeline with Kafka 0.11, but to do exactly-once consumption, you need to implement your own transactional semantics in your consumers to tell Kafka when you’re done processing (or roll back if things go wrong).

Additionally, in our testing, we found that the transactional model for message production in Kafka 0.11 didn’t process messages as quickly as we needed it to, taking up to 10 – 100 milliseconds per message. That’s a lot of additional latency that we can’t afford to incur.

You do have some options, though. Several streaming solutions, like Flink or Kafka Streams, offer exactly-once processing as long as you stay within the constraints of those frameworks.

Another option would be to “roll your own” exactly-once strategy that would automatically commit offsets only for messages that had reached the end of the processing pipeline. This is a recommendation only for the bravest types out there, as getting this right is pretty hard, and you may cause more problems than you solve.

And there’s always the option to accept the risk of data loss or duplication. Sadly, this is the option that many Kafka users choose, but it’s not always an unreasonable choice. The amount of data you’d actually lose or duplicate in one of these scenarios is relatively small; the auto commit should only be a few seconds off from the actual last committed message. (Your throughout levels are probably the best indicator of how much duplication or loss you’d see). The frequency of occurrence should also be low—ideally your Kafka consumer services aren’t getting regular OOM kill signals.

So, compared to the engineering cost of solving this problem in all of your Kafka consumer services, this may just be a risk that you live with if the the data you work with permits a small amount of loss. Of course, you can always mitigate the risk by building reliable services.

Moving forward

As I said, we have a lot of experience at New Relic managing Kafka clusters. We’ve discovered that building highly reliable services gets tougher as we scale to handle massive data volumes, and we’ve made some practical decisions about how we handle this particular flaw in Kafka should one of our services ever experience a hard shutdown.

In the cases where you accept the risk of data loss, take the following steps to minimize it:
- Be aware of this problem, and document it in your risk matrix.
- Make sure your services are stable. Alert on your service’s SIGKILLs and OOMs.
- When building new services, consider using a streaming system that solves this problem for you right out of the box.
Five Steps for an Incident Response Workflow, Featuring the Jira Software and New Relic Integration
new relic news and produc alerting incident response jira software • • sorangutan

1

1
Posts

113
Views
There are many ways to set up a successful incident response system, but the end result should always be the same: You need to discover incidents—and ideally fix them—before they affect your customers. Successful incident response—also referred to as incident orchestration—requires the alignment of teams, tools, and processes to prepare for and react to incidents and outages in your software.

You can’t afford to wait until an incident occurs to figure out a plan. You need to act quickly and efficiently to:
- Maximize efficiency in communication and effort
- Minimize the overall impact to your business
Modern software teams can do a lot more than just write a plan for handling incidents; they can connect real-time performance data and alerts to their incident response systems and automate much of the process using tools like the Jira Software and New Relic integration. Jira Software, developed by Atlassian, is a wildly popular tool used by Agile teams to plan, track and release software. The seamless inclusion of New Relic data in Jira, however, lets teams easily file Jira tickets about incidents and include all the context stakeholders need to resolve them.

Let’s highlight five steps for building an incident response workflow using the Jira Software and New Relic integration.

(Note: This is an abridged version of “Incident orchestration: Align teams, tools, processes,” part of our Guide to Measuring DevOps Success.)

1. Assign owners for team dashboards

Team dashboards in New Relic Insights provide, in a single view, the performance status of major components in your applications. But more importantly, they also allow you to visualize the service level indicators (SLIs) and other key performance indicators (KPIs) for your applications.

Each team dashboard should have an owner who assumes responsibility for the health of the applications and features that the dashboard monitors. And there should be no ambiguity about who is responsible for attending to and resolving an alert condition.

How you set up such policies will vary depending on the size, structure, and culture of your organization. For example, some teams may prefer to assign dashboards and alerts based on de facto features or application ownership. Other teams may prefer to adopt an on-call rotation. In on-call rotations, designated team members handle all first-line incident responses, and they resolve or delegate responsibilities based on predetermined incident thresholds.

2. Determine incident thresholds for alert conditions

The term “alerting” often carries negative connotations; for too many developers, alerting correlates too closely with errors, mistakes, and ongoing issues. However, developers who are proactive about alerting know they don’t have to stare at their dashboards all day, because effective alerts will tell them when they need to check in. For instance, a certain alert condition may be dismissible during low-traffic periods but require active remediation during peak hours.

For each of your applications, set a proactive alerting policy:
1. Identify the thresholds for what is officially considered an incident.
2. As you create alert policies with New Relic Alerts, make sure each set of threshold criteria is context-dependent.
3. Document incident evaluation and known remediation procedures in runbooks.
4. Include links to your runbooks when you define conditions and thresholds for your alert policies.
3. Ensure alerts have auditable notification channels

A key part of your incident response process is communication, which should take place in easily accessible and highly visible channels. A group chat room dedicated to incident communication allows all stakeholders to participate or observe and provides a chronology of notifications, key decisions, and actions for postmortem analysis.

New Relic Alerts integrates with several communication platforms, including Slack and Opsgenie. In fact, at New Relic we use a designated Slack channel for incidents; the channel is automated by a bot, called Nrrdbot (a modified clone of GitHub’s Hubot), that guides our incident responders through the communication process during an incident.

4. Create faster triage and resolution times with New Relic and Jira Software

When you discover problems in your application, using New Relic, that you need your developers to fix, you can push the details directly into your ticketing system in your Jira account. We built our integration to work with any combination of required custom fields.

The Jira Software and New Relic integration improves your incident response feedback cycle and helps seamlessly transition from error trace to hot fix. Specifically, you can correlate incidents detected by New Relic with Jira tickets directly from the New Relic UI, and easily file and assign issues to responsible teams for tracking or remediation.

File a Jira ticket directly from New Relic with the Jira Software and New Relic integration.

From within the corresponding Jira ticket, you can track the current status and severity of an incident from creation to resolution.

When an incident occurs in your system, there’s a broad set of stakeholders—from first responders to devs and ops folks leading the remediation effort to support engineers relaying information to your customers—who need to understand the issue and its status. You need a single source of truth from which you can convey the details. New Relic brings the issue to your attention, and Jira makes sure everyone can track your remediation efforts in real time.

5. Hold a retrospective and document your findings

After you’ve resolved an incident, key stakeholders and participants should hold a retrospective (also known as a “post-mortem”) to capture accurate and thorough documentation about the incident. In the documentation, include:
- A root-cause analysis
- A chronology and summary of remediation steps and their result, whether they were successful or not (including links to relevant Jira tickets)
- A measure of the impact to the business in terms of user experience and financial losses, if possible
- Recommendations for system or feature improvements to prevent a recurrence
- Recommendations for process and communication improvements
Store postmortem reports in a highly visible, searchable repository, like an internal wiki. Culturally, it is essential that in this process, you focus on constructive learning and improvement rather than punishment or blame.

Get started by visiting the Atlassian Marketplace to install and set-up the Jira Software and New Relic integration, and join the the discussion at the New Relic Online Technical Community. For additional information, check out the documentation page.
DevOps Weekly’s Gareth Rushgrove on the History and Evolution of DevOps
devops technology industry sre • • sorangutan

1

1
Posts

109
Views

If you want to keep tabs on what’s happening in the ever-changing world of DevOps, Gareth Rushgrove’s DevOps Weekly newsletter has been a must-read for the past nine years. By day, Gareth is a product manager for Docker, but every Sunday, from his offices in Cambridge, UK, he curates an eclectic mix of DevOps stories and resources that keep his almost 30,000 subscribers informed and up-to-date.

In the course of gathering content for 400 or so email newsletters (with an estimated 5,000 links to DevOps content), DevOps Weekly has kept Gareth’s finger on the evolution of DevOps since its early days. New Relic was thrilled to sit down with him to discuss the changes he’s seen over the years, and to learn how he expects the filed oew

New Relic: You’ve been publishing the DevOps Weekly for some nine years now… how has DevOps evolved during that time? How are things different now compared to when you started?

Gareth: It’s a broader conversation now. Not because the people who were involved early were myopically interested in only a very specific subset of the conversation, but because there were fewer people. And frankly, we were all younger. Some of those people are doing broadly the same things, while others are in a lot more senior positions, dealing with bigger challenges.

DevOps is wholly community oriented—it’s about the people in the room.

In some cities, DevOps got a developer flavor because the people there who were passionate about organizing something were developers. In other places operations is more central. Sometimes it’s more balanced between the two. And it’s important to remember that DevOps is still growing, still finding new areas that fit into its original brief of working better as software professionals.

If you look a bit further out, the term “DevOps” isn’t the most important bit. It’s allowed us to come together under the DevOps banner, but that visibility is pure marketing. DevOps is basically a marketing success story in that sense—it’s one reason why people get emotionally attached to DevOps in the same way as they get attached to brands.

Sometimes people like the friction between things that are different. Sports teams are a classic example, but also picking your software tools or programming languages. You can’t possibly like Java and PHP, can you? It’s the tribal nature of communities, people finding things to bond over. And sometimes that’s the thing, and sometimes that’s the thing you’re against.

New Relic: Where does something like site reliability engineering (SRE) fit into this discussion?

Gareth: SRE has come out of Google, and not many of those people were in the room when DevOps was born because they were at Google. Google already had rooms filled with people talking about some of these things. One of the things that’s changed about Google is they’re now much more actively involved in external communities. At one point Google was known for sharing what it was doing via academic papers, but they’ve broken that down to where they’ll often share via open source projects and other means.

The SRE book is a fantastic example. The previous generation would probably have shared that information via targeted academic papers, not in a mass audience book.

More specifically, DevOps and SRE is an apples to oranges comparison. SRE has had a parallel evolution to DevOps, but they’re definitely not the same thing. I see SRE as an implementation, a specific collection of practices, where DevOps is more of a community conversation.

Serverless has fans, too

New Relic: Given this tribalism that you talked about, are we starting to see newer approaches and branding for the DevOps mindset—in the way that DevOps challenged earlier mindsets?

Gareth: Definitely! The best example is what’s happening under the serverless community banner. A lot of conversations about tools and practices are tied up with current assumptions, especially the adoption of cloud. It’s not that DevOps is only about cloud computing, but a lot of the API-driven infrastructure and automation has gone hand-in-hand with the adoption of public cloud and cloud practices.

If you look at it from the point of view of the new serverless community, there are fundamental differences: “We’re new. It’s all different!” From the outside, though, you may say, “Oh, no. It’s all the same.” And there’s a bit of truth in both viewpoints.

DevOps drivers vs. DevOps barriers

New Relic: What do you see as the biggest drivers for DevOps adoption right now. You mentioned the cloud, but what other items are pushing organizations towards DevOps?

Gareth: Speed. Organizations are now fundamentally understanding that they need to move quickly. Not moving fast is such a detriment to your competitive position, whether against existing competitors or against new competitors that may have a radically different, and much faster, business model. That fear of disruption has driven a lot of organizations to appreciate that they’re not moving quickly enough.

Look at the State of DevOps Report from one of my former employers, Puppet, as well as DORA’s Accelerate: State of DevOps 2018 report. There’s so much data and solid research showing that it is possible to go faster without sacrificing stability or security. The argument against DevOps was always, “No, we can’t do that. We won’t be secure. We won’t be compliant. We won’t be safe.” But the data shows that’s simply not the case.

New Relic: On the flip side, what do you see as the biggest barriers to DevOps adoption? Is it primarily a technological issue, a cultural issue, or mixed?

Gareth: I think technology plays a part, but first and foremost it’s a people, process, and cultural issue. Especially in large organizations where you have such a broad range of existing technologies, existing people, existing processes, existing partners, as well as possibly different regions, locales, offices, different cultures, and different teams in different places.

Trying to change that is really difficult. It’s even harder when you’re trying to move toward a model of constant change. People want digital transformation to be something that they can do—a process with a beginning and an end. Ultimately, though, DevOps is about embracing a world where change is constant. Lots of organizations are not set up for that; they’re set up for discrete, periodic changes, with success measured between those changes.

Another issue is procurement, purchasing, and budgeting. The importance of the financial side, moving past a project mentality to create incentives that lead to structures that can adopt these practices easily.

Getting started with DevOps

New Relic: So—if I’m in a really large organization, and I want to explore the brave new world of DevOps, how do I get started?

Gareth: Listen to people outside your walls. My government experience is a good example; in many large organizations, people end up in an echo chamber claiming that, “Oh, no. Government is different.” Or “Banking is different.” And they don’t want to talk to people outside their organization or industry.

Attending a local session of DevOpsDays can be hugely eye-opening. I’ve been to quite a lot of these, and there’s nearly always a few people there who have never really been to anything like that before. And they’re blown away by how open people are.

Lots of people come back and bring that enthusiasm to their own organizations. Enthusiasm goes an awfully long way when you’re trying to change an organization from the inside. If you’re not enthusiastic, why are you even doing this in the first place?

There’s also a wealth of DevOps resources to dive into. The recent Accelerate book presents useful data and evidence that people can [use] to make their case for DevOps. The DevOps Cookbook, as well. There are books targeting CXO-type folks and books targeting technical practitioners. There are books for people who like data.

The critical role of monitoring in DevOps

New Relic: Let’s talk for a minute about the critical role that monitoring plays in DevOps—an issue that we care about here at New Relic. What’s your take on the current state of the relationship between monitoring and DevOps?

Gareth: Earlier we talked about cultural automation, measurement, and sharing. Measurement was always part of the DevOps mindset. And not just measuring infrastructure or applications, but measuring everything: people and practices and processes.

Monitoring was a fundamental thing. In the early days of DevOps you had a lot of conversation about deployment tooling, and you had a lot of conversations about monitoring. It was something that the community was interested in and everyone was able to talk about. The explosion of monitoring tools has gone hand-in-hand with the product and process things—especially the rise of deployment automation.

Ten years ago, the idea that Flickr was doing five deployments a day was crazy. Now, it’s, “Oh, yeah. Of course we’re doing five deployments a day.” Even in large organizations with scale and compliance issues, this is demonstrably possible and demonstrably positive. But you cannot do it without two things.

One is comprehensive monitoring. It’s not good enough to know what your servers are doing. You need monitoring of the whole system and how your users experience it. Otherwise, you can’t react to issues before they affect people.

The other thing you need is continuous integration and continuous deployment (CI/CD). Again, that’s a sea change from 10 years ago when it comes to deployment tooling. The deployment pipeline is now a critical part of your infrastructure. Before, you might have said, “I need my deployment tools once every six months.” But, “Now I need my deployment tools to be 24/7.” It’s the evolution of the monitoring tools that give us the confidence to do that without simply throwing people at the problem.

The DevOps Weekly—there from almost the beginning

New Relic: Now that we’ve talked about how DevOps has evolved, let’s delve into you got involved, and how the DevOps Weekly came to be.

Gareth Rushgrove: My involvement in DevOps goes back to when I used to work on the development and operations side of things for large organizations, building out continuous integration tools and helping get software into production. Probably most formatively, I worked for the UK government for about four years on what would now be regarded as “digital transformation,” although we weren’t using those words then.

At the same time, I started doing a lot of things with small teams, growing an operations team to help other government departments adopt modern infrastructure practices. So my background includes everything from business software development to operations work to classic developer things, and also the consulting and people side. I think it’s the broad spread of things I’ve done that drew me towards what eventually became DevOps.

New Relic: So how did that transition happen? How did you discover DevOps?

Gareth: Really by proximity to someone who was there at the start. The DevOps term began with Patrick Debois and organizing the first DevOpsDays event in Ghent, Belgium, back in 2009. I wasn’t there, but one of my colleagues at the time was. He came back and said, “Oh, you should have been at that. They were talking about all of the things that we’re doing and you’re interested in.”

I was at the second one, in Hamburg, and I thought: “We now have a word for this thing!” I ended up sending an email out to a few of the people I met and staying in contact with them. And I’m still doing that nine years later.

New Relic: So that was the genesis of the DevOps Weekly newsletter?

Gareth: DevOps didn’t invent the practices that came to be attached to it. But it gave us a forum to talk about those things. I was already interested in build tools, I was interested in automation, I was interested in the crossover between operations and development. But it wasn’t until a bunch of people who were also interested in those things got in a room and started talking about it, and realized other people were interested too, that suddenly it got a lot more interesting.

This was in the early days of Twitter. And at that time if you searched for the term DevOps, just about anyone using it was an interesting person, which created the opportunity for high-value conversations. So I met a lot of friends who were some of the first people who realized this was what they had in their head but didn’t have a word for it before. Once we had a word for it, everything sort of coalesced.

Seeing that grow over time has given me a super-interesting vantage point, and DevOps Weekly is just one of those side projects that got out of hand.

New Relic: How you decide what to put in the newsletter each week? Do you put in everything that you come across that’s DevOps related, or just the best stuff? And how has the content changed over time?

Gareth: It’s always been a curated thing. I put something in if I find it interesting enough. I read a lot, and I’ve always collected links, and I’ve found that if I think something is interesting, other people do as well. I’m pretty happy if I can hit one or two things that people find interesting in every issue.

Certainly, the topics that people write about have changed over the years. But so have the types of things I’m interested in. The newsletter has evolved along with the DevOps movement, which helps keep it fresh for me but also for some of the subscribers who have literally been reading from day one.

Is DevOps cool?

New Relic: I have one more question. This comes out of what we were talking about earlier: Is DevOps still cool?

Gareth: I don’t think DevOps was ever cool. It was a bunch of geeks talking about service management under a marketing banner, and we were definitely not the cool people. I think it’s interesting how DevOps emerged into the wider world, but I’m not sure it was ever cool.

On the other had, a lot of cool things have happened under the DevOps banner. A lot of people spend an awful lot of time and effort making people feel welcome in the DevOps community—even if that’s realistically just made up of a word and the internet. And I think that’s cool.
Trying Out New Relic APM for Lambda With an Image Recognition Prototype App
aws aws lambda software engineering new relic insights apm for lambda • • sorangutan

1

1
Posts

101
Views
Lately, I’ve been playing around with Amazon Rekognition, which you can use to analyze images and video to do real-time analysis to recognize things like faces, objects, and text. The video below is an example of how I used Rekognition to pick up the New Relic logo on a coffee mug:

But that wasn’t enough. I wanted to see what else I could do with it … and I decided to see if I could use Rekognition—and New Relic—to track the exact number of seconds that the New Relic logo appeared on screen during the telecast of the 2018 Masters Tournament. You see, we sponsor golfer Mark Leishman, who was paired with Tiger Woods in the tournament’s opening round and was just two strokes off the lead heading into Saturday. We estimate that 10 million people could have seen the New Relic logo on Mark’s shirt, and in fact, this visibility correlated with an approximate 34% traffic spike in our web properties over that weekend.

To calculate the number of seconds that the New Relic logo appeared on screen, I architected a prototype app using footage of the Masters and several AWS services—Rekognition, AWS Lambda, Amazon API Gateway, Amazon Kinesis, Amazon CloudWatch, and Amazon S3. And, of course, I included some instrumentation to gain visibility and send custom data to New Relic. Hopefully, my project will spark ideas on other ways to use New Relic to monitor images and video.

Inside my Amazon Rekognition prototype app

Here’s how it works: An Android mobile app streams the relevant video from my phone to a Kinesis video stream. When I’m ready to process part of the stream, I manually invoke the API Gateway to kick off my Lambda function (in a real-world, practical application, I’d script this). The Lambda function grabs the most recent footage from the video stream, uses the Rekognition service to recognize any “New” and “Relic” text from the footage, writes the resultant image to an S3 bucket, and returns the image in base64 encoding as the API response. It’s all pretty slick.

Of course, no project would be complete without some New Relic instrumentation. This time, I had the pleasure of using an alpha release of New Relic’s APM for Lambda, now available in preview. Using APM for Lambda, I was able to gather and visualize some custom data (in this case, how many seconds the New Relic logo appeared in the video stream) and have enhanced monitoring of my main Lambda function processing the stream. To gather this data, I used the New Relic log ingestion function (described below) to pipe a stream of CloudWatch logs generated by my main Lambda function into New Relic.

A closer look at each part of the pipeline

The Kinesis app

Luckily, I didn’t have to develop my own application to stream video from my Android phone to the Kinesis video stream. Instead, I used the Android Producer Library, which provides a sample app with which you can set up a Kinesis Video Stream producer client in an Android application to send data to a Kinesis Video Stream. In other words, the client ran on my phone and sent the relevant footage to the Kinesis service.

I had to set up a Cognito user pool to be able to set up the sample app, but once I did that things worked “right out of the box.”

The Kinesis Video Stream

I created the Kinesis Video Streams in the AWS Management Console. The console provides a dashboard that shows what’s currently flowing into my Kinesis stream. I was able to scroll backwards and forwards through any part of the stream until it was 24 hours old, at which point the system purges it.

The AWS Lambda processing function

I consider this function, written in Python, to be the “brain” of the pipeline, as it processes the video stream and moves it through the rest of the application—a lot of heavy lifting. Specifically, the function:
1. Gathers about one second of video from the Kinesis video stream
2. Writes that video to a temporary file
3. Grabs the first frame from the temporary video file, creates a grayscale frame, and encodes the grayscale frame to PNG format
4. Sends the grayscale frame to Rekognition for text recognition
5. Uses text recognition responses to draw a bounding box on the frame and encodes the resulting frame
6. Writes the new frame to a temporary file and saves it in an S3 bucket
7. Encodes the saved file to base64 for transmission and returns the results through the API Gateway in a JSON-formatted response
The function’s code could use some refactoring and optimization, but it worked for my prototype.
```
import cv2
import boto3
import tempfile
import os
import json
from base64 import b64encode
import re
import time
import newrelic

print('Loading function')

@newrelic.monitor_lambda
def lambda_handler(event, context):
	# Get about a second of video from the Kinesis video stream
	video_client = boto3.client('kinesis-video-media',endpoint_url='https://s-5xxxxxx4.kinesisvideo.eu-west-1.amazonaws.com',region_name='eu-west-1')
	stream = video_client.get_media(StreamARN='arn:aws:kinesisvideo:eu-west-1:36xxxxxxx4:stream/my-stream/55555555555',StartSelector={'StartSelectorType': 'NOW'})
	streamingBody = stream["Payload"]
	datafeed = streamingBody.read(40000)
	
	# Write video to a temporary file
	new_file, filename = tempfile.mkstemp()
	os.write(new_file, datafeed)
	os.close(new_file)
	
	# Grab the first frame for the temporary video file, create grayscale frame and encode to png
	cap = cv2.VideoCapture(filename)
	ret, frame = cap.read()
	gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
	rekog_frame = cv2.imencode('.png',gray)[1]
	
	# Send grayscale frame to Rekognition for text recognition
	client=boto3.client('rekognition','eu-west-1')
	response = client.detect_text(Image={'Bytes': rekog_frame.tobytes()})
	
	# Use text recognition response to draw bounding box on frame and encode resulting frame
	mod_frame = bounding_box(response,'new relic', frame)
	return_frame = cv2.imencode('.png',mod_frame)[1]
	
	# Write resulting frame to temporary file and save the file to s3
	cv2.imwrite('/tmp/image1.png',mod_frame)
	s3_file_path = save_to_s3('/tmp/image1.png')
	
	# Encode resulting image to base64 for transmission
	base64_bytes = b64encode(return_frame.tobytes())
	base64_string = base64_bytes.decode()
	
	# Return results in API Gateway accepted format
	return({
    	"isBase64Encoded": True,
    	"statusCode": 200,
    	"headers": { "s3_file_path": s3_file_path },
    	"body": base64_string
	})

def bounding_box(response, word_array, frame):
	width, height = frame.shape[:2]
	x1_coord=y1_coord=x2_coord=y2_coord=None
	for entry in response['TextDetections']:
    	if entry['Type'] == "LINE" and re.search('new relic',entry['DetectedText'], flags=re.IGNORECASE):
        	x1_coord = entry['Geometry']['Polygon'][3]['X']
        	y1_coord = entry['Geometry']['Polygon'][3]['Y']
        	x2_coord = entry['Geometry']['Polygon'][1]['X']
        	y2_coord = entry['Geometry']['Polygon'][1]['Y']
        	break
    	if entry['Type'] == "WORD":
        	if re.search('New',entry['DetectedText'], flags=re.IGNORECASE):
            	print('Detected "new"')
            	x1_coord = entry['Geometry']['Polygon'][1]['X']
            	y1_coord = entry['Geometry']['Polygon'][1]['Y']
            	if x2_coord == None and y2_coord == None:
                	x2_coord = entry['Geometry']['Polygon'][3]['X']
                	y2_coord = entry['Geometry']['Polygon'][3]['Y']
        	if re.search('Relic',entry['DetectedText'], flags=re.IGNORECASE):
            	print('Detected "relic"')
            	if x1_coord == None and y1_coord == None:
                	x1_coord = entry['Geometry']['Polygon'][1]['X']
                	y1_coord = entry['Geometry']['Polygon'][1]['Y']
            	x2_coord = entry['Geometry']['Polygon'][3]['X']
            	y2_coord = entry['Geometry']['Polygon'][3]['Y']
	try:
    	cv2.rectangle(frame, (round(x1_coord*height), round(y1_coord*width)),(round(x2_coord*height), round(y2_coord*width)), (255,0,0),5)
    	newrelic.record_custom_event('rekogEvent', {'logo_detection': 1})
	except Exception as e:
    	print(e)
	return frame
	
def save_to_s3(filename):
	bucket = 'rekognitionseantest'
	timestr = time.strftime("%Y%m%d-%H%M%S")
	key_name = timestr + '-rekog.png'
	directory = "Test_images/"
	s3 = boto3.client('s3')
	s3.upload_file(filename, bucket, directory + key_name, ExtraArgs={'ContentType': "image/png", 'ACL': "public-read"} )
	return(directory + key_name)
```
The Amazon Rekognition service

This part of the pipeline is essentially an API to which I send an image and receive a response on whether or not it recognized the specific text for which I asked it to search—in this case, the words “New” and “Relic.” Unfortunately, Rekognition can’t recognize text in video, but I got around that by extracting frames from the Kinesis Video Stream (asI mentioned above when I described the Lambda function).

New Relic log ingestion (from CloudWatch)

The New Relic log ingestion function—the second function in my app—is an AWS Lambda function that consumes the CloudWatch logs generated when my main Lambda function executes. Specifically, it watches for CloudWatch log entries that contain the NR_LAMBDA_MONITORING marker. This marker is produced by a wrapper (@newrelic.monitor_lambda) that I added to my main Lambda function. The custom data I collect with my function (newrelic.record_custom_event('rekogEvent', {'logo_detection': 1}) is also tracked by the CloudWatch logs.

Here’s an example log that returns details about my function and the custom event:
```
{
	"marker": "NR_LAMBDA_MONITORING",
	"version": "Alpha-4",
	"context": {
    	    "functionName": "cloud9-python3videostream-python3videostreamtodata-7xxxxxx5",
    	    "functionArn": "arn:aws:lambda:eu-west-1:36xxxxxxx4:function:cloud9-python3videostream-python3videostreamtodata-6VL57MEK2E26",
    	    "functionVersion": "$LATEST",
    	    "functionRuntime": "python3.6",
    	    "functionRegion": "eu-west-1",
    	    "awsRequestId": "f3xxxxxxxxxxxxxxxxxxf",
    	    "sourceInfo": null,
    	    "containerInfo": {
        	"isColdStart": true,
        	"startTime": 1528975664117,
        	"executions": 1,
        	"guid": "30xxxxxxxxxxxx8"
    	    }
	},
	"traceEvent": {
    	    "startTime": 1528975664118,
    	    "endTime": 1528975666044,
    	    "failed": false,
    	    "memoryMB": 63.15625
	},
	"operationEvents": [
    	    {
        	"service": "kinesis-video-media",
        	"operation": "GetMedia",
        	"startTime": 1528975664206,
        	"endTime": 1528975664291,
        	"failed": false
    	},
    	{
        	"service": "rekognition",
        	"operation": "DetectText",
        	"startTime": 1528975664650,
        	"endTime": 1528975665621,
        	"failed": false
    	},
    	{
        	"service": "s3",
        	"operation": "PutObject",
        	"startTime": 1528975665925,
        	"endTime": 1528975666040,
        	"failed": false,
        	"target": "rekognitionseantest"
    	}
	],
	"customEvents": [
    	    {
        	"eventType": "rekogEvent",
        	"logo_detection": 1,
        	"timestamp": 1528975665632
    	    }
	]
}
```
In New Relic, I set up a New Relic Insights dashboard that captures my custom events (length of logo appearance, in seconds) and metrics about my Lambda function, such as memory usage, function execution counts, and the duration of each function execution.

In some cases, the New Relic logo was on screen for up to 90 seconds at a time. Apparently, that was enough to make plenty of people curious enough to visit our web properties.

AWS and APM—always raising the bar

For every image Rekognition analyzes, AWS charges about $0.01 . That may not look expensive at first glance, but depending on the video format your stream uses, you could have 25 – 50 frames per second; multiply that by minutes or hours, and a project like this could really add up. (The pricing for video analysis is more reasonable—$0.10 per minute of footage).

AWS is setting a high bar as it uses Rekognition and related services to bring AI and related technology “to the masses.” And the company is making it easier than ever to add various types of automation capabilities to homegrown applications.

AWS Lambda makes it easier than ever to write and quickly execute your code, since you don’t have to worry about infrastructure management. However, removing a layer of abstraction (or, one could argue, adding another one!) doesn’t make it easier to monitor what’s going on in the code while it’s running. I found that APM for Lambda opened the “black box” surrounding my Lambda function and gave me real-time, code-level APM insights—and even allowed me to track custom metrics.

If you’re interested in getting early access to New Relic APM for Lambda and want to contribute feedback, sign up here. Unfortunately, not everyone who signs up will be invited to the private beta, but everyone will receive updates as we hit critical milestones on the way to general availability.

Now I wonder: What should I use this app to track next?
Edge Computing vs. Cloud Computing: Which One Is Right for Your Application?
edge computing technology industry cloud computing • • sorangutan

1

1
Posts

89
Views
On a farm in rural Oregon, a set of gauges monitor weather and drought conditions to promote optimal crop production.

An automated drone soars through the Grand Canyon taking photographs, and gathering environmental and geographical data.

A semi-truck barreling down I-5 transmits its location, load weight, and operating condition to a central transportation system thousands of miles away.

En route to your favorite coffee shop, you place your latte order from a mobile app so it’s ready and waiting when you arrive.

What do these four scenarios have in common? They all rely on edge computing.

Edge computing happens any time you move part of your application closer to where the “action” is. That “action” may be a source of data you want to process (weather conditions on a farm) or track (the semi-truck on the highway), the end user of your application (a person ordering their latte), or a system you’re controlling (an automated drone).

But why should you perform some computing tasks in an edge environment and others in a centralized cloud environment? What’s the difference between the two, and why would you use one instead of the other? What are the benefits and challenges of each approach?

The edge, the cloud, or both? The case of the autonomous car

Edge computing is all about putting time-sensitive operations closer to where they will have an impact. It puts computation—data collection and analysis—where it can operate most efficiently, as opposed to where it’s most convenient for developers or operators.

In cloud computing, data is collected and analyzed in some centralized location where developers and operators have more control over processing and system communications.

Putting computation at the edge is harder and riskier than keeping it together in the cloud, so you want to do it only when there’s a good reason to do so. That said, how do you decide whether to put computation in the cloud, at the edge, or somewhere in between?

To help figure it out, let’s consider the case of the driverless car.

Today’s autonomous cars are chock full of cameras, sensors, and controls that detect the road and various obstacles. The car uses these data feeds and controls to determine what’s really happening around it: Is that blob over there a human crossing the street or a road-closed barrier?

Autonomous cars have controls for steering, braking, and applying power. But they also have controls and sensors for monitoring the health of the car itself: Is the motor operating efficiently? Is the passenger compartment a comfortable temperature? Does the car need to deploy an airbag right now? Some of this computation must occur in the car, but much of it can occur in the cloud.

In many cases, performance considerations mean computations must occur in the car, including image recognition (is that another car near me or some other type of object?); threat detection (is that car in front of me suddenly applying its brakes?); road management (am I staying in my lane?); collision control (quick—swerve right to avoid a crash!). These kinds of time-sensitive calculations must be as fast and accurate as possible. The results of these computations can’t suddenly lag or go offline because of a bad internet connection. This is why autonomous cars need edge computing.

On the other hand, many driverless-vehicle processes can—and should—occur farther away, in the cloud. For instance, the cloud is a suitable place to run navigation systems that address such questions as: How do I get from point A to point B? What’s the most optimal route? Is there construction or traffic on this route that makes taking another preferable?

These systems need access to centralized data (such as maps and traffic information), and they need to correlate information from other sources to complete the computations. But these tasks are rarely time sensitive, at least not down to the level of microseconds it takes to communicate with the cloud.

Considering when and how to separate computation between the edge and the cloud is important because computation at the edge typically requires much more effort to properly manage. Upgrading software, troubleshooting bugs, monitoring performance—all of these things are easier to do at a centralized location rather than within a distributed and remote system. Just remember: When your software needs to scale, you have to remember that scaling has a different meaning for edge and cloud computing.

The edge’s advantages and challenges

Let’s look at some of the advantages of edge computing, and let’s also look at some of the challenges it presents—especially when dealing with large numbers of nodes.

First, consider the advantages of edge computing:
- The edge provides more time-sensitive and responsive processing.
- The edge is less dependent on network connectivity, which increases reliability in the face of unknown connectivity.
- The edge enables dedicated processing for a single, specific task.
- The edge provides ready access to highly individualized data.
But edge computing also poses unique challenges, typically centered around the need for a large number of nodes distributed over a wide geographic area. Edge computing challenges include:
- Managing deployments across a fleet of edge nodes. For example, how do you ensure all edge devices are running the same version of your software? Additionally, the edge has variable and unique provisioning issues (more on that below).
- Monitoring the usage and performance of edge software becomes increasingly difficult as you add nodes.
- Remote debugging and troubleshooting becomes increasingly difficult as you add nodes.
- Determining if the root cause of an incident lies at the system level or at the single-node level becomes increasingly difficult as you add nodes.
- The edge isn’t ideal for use cases that require unpredictable or “bursty” amounts of CPU. Such tasks are better suited to the dynamic computation available in the cloud.
- The edge isn’t ideal for use cases that require ready access to global data. This data is more accessible from cloud-based services.
Seven keys to successfully build edge computing into your application

Fortunately, careful implementation of edge computing can dramatically increase your odds of success. These seven best practices are a good place to start:
1. Be smart about what you manage at the edge versus in the cloud. Do a thorough analysis of your use case and make an active decision. Both the cloud and the edge have unique advantages and disadvantages, but as general rule of thumb: When in doubt, use the cloud. The edge is optimized for computations that require immediate, individualized results, which does include not the majority of today’s use cases.
2. Don’t throw away DevOps principles at the edge. It’s tempting to abandon established best practices when you adopt new technologies, and edge computing is no exception. Yes, edge computing represents a highly specialized situation, and it often requires new processes and procedures. But don’t discount the utility of existing DevOps principles and best practices. Process and tools may change, but your teams still require ownership, accountability, and distributed decision making—just the kinds of things that DevOps provides.
3. Automate distributed deployments. All too often, developers ship new applications and leave deployment automation as something to “fix later.” Automated and repeatable deployments are critical for all applications, but they are even more important for edge applications, due to the remote nature of the applications and the huge number of nodes involved.
4. Reduce versioning as much as possible. At the same time, you need to limit the number of deployments in edge computing architectures. This may seem antithetical to the DevOps emphasis on continuous integration/continuous deployment (CI/CD), but the high number of distributed nodes in an edge computing environment mean you can’t always deploy to the edge as fast or as often as you can deploy to the cloud. Similarly, to ease software management on a large number of nodes, each node should run the same hardware, firmware, and software configurations. Of course, this isn’t always possible; mobile applications often run on a plethora of hardware and software configurations. But the more you can reduce the number of variables, the easier it is to manage the software for these nodes.
5. Understand the difference between scaling at the edge and in the cloud. Edge software often runs thousands of instances of the software in a highly distributed manner, but each instance typically performs a single task or manages one device. These instances are often distributed geographically in many different locations. Cloud software, on the other hand, runs a few instances—perhaps on multiple distributed servers—but each instance typically manages actions for thousands of users, and the instances run in a small number of locations. Scaling in the cloud is about addressing how much each cloud instance can handle. This is typically done via vertical scaling—increasing the size of individual nodes as demand increases. Edge scaling, on the other hand, is dictated by how many edge nodes you can handle at a give time. This is horizontal scaling: adding more nodes to meet demand, rather than making individual nodes larger.
6. Deploy monitoring and analytics all along the edge. As you deploy more distributed nodes at the edge, it becomes increasingly important to understand how each node is performing at any given time. To properly manage a highly scaled and distributed system of edge nodes, you need a continuous view into the health of every node. If you have a fleet of autonomous vehicles, for example, you can’t predict the behavior of the entire fleet based on metrics gathered from just one car.
7. The edge is not magic. Edge computing is not new; it’s just a rebranding of what we’ve been doing for years. What we used to call a “browser application” or a “mobile application” or a “point of sale” device has always been the edge. The edge is just a new label on an existing class of computation, but the increasing popularity of the term will encourage a future with better, edge-focused tooling and custom-tailored services.
Remember: Edge computing is really no more than a new way to think about what we’re already doing with many of our applications. Successful edge computing is all about managing modern applications and their components, whether these reside in the cloud or on the edge.
Monitoring Application Performance in Kubernetes: How New Relic Sets You Up for Success
kubernetes container orchestration software engineering new relic apm new relic insights • • sorangutan

1

1
Posts

115
Views
This post was originally published on November 27, 2017. It was updated with information on the New Relic cluster explorer and using the New Relic platform to monitor Kubernetes environments.

Over the last few years, Kubernetes has emerged as the de facto standard for orchestrating and managing application containers. Virtually all the significant players in the cloud-native application space—many of them fierce competitors—have thrown their support behind Kubernetes as an industry standard. Measured on developer support, Kubernetes’ journey over the last three years—from first public appearance to serving as the main topic at an event attracting more than 7,000 attendees (KubeCon + CloudNativeCon)—has been simply spectacular.

Here at New Relic, the Kubernetes story boils down to a couple of very strategic points. First, we believe that for all of its success to date, Kubernetes is actually—and amazingly—still in its early journey towards achieving its full potential. Tomorrow’s Kubernetes environments will operate at a scale that will dwarf what we see today. As Kubernetes environments continue to scale, they will also continue to get more complex—and to present bigger challenges for efforts to monitor their performance and health.

Second, many of our customers tell us they see tremendous value in leveraging the New Relic platform to transition to and run their application workloads in Kubernetes. So we have been working hard to expand and upgrade the New Relic platform with a new generation of monitoring tools designed for Kubernetes environments. These capabilities will help our customers to keep moving quickly and to act with confidence when they adopt Kubernetes to manage and orchestrate their containerized workloads.

New Relic and Kubernetes: A powerful pairing

Most organizations that deploy Kubernetes environments today share the same basic goal: To maximize the benefits of running containerized applications, at virtually any scale, through the use of automation, workload orchestration, and other enabling technologies. It’s difficult to overstate the importance of orchestration in general, and Kubernetes in particular, in achieving this goal. Without these capabilities, the sheer complexity of managing and optimizing containerized workloads, microservices, and related innovations—especially when deployed at scale—might easily outweigh their benefits.

While Kubernetes excels at abstracting away much of the toil associated with managing containerized workloads, it also introduces some complexities of its own. The Kubernetes environment includes an additional layer that comes into play between the application and the underlying infrastructure. The challenge of monitoring and maintaining the performance and health of these Kubernetes environments, or of troubleshooting issues when they occur, can be daunting—especially as organizations deploy these environments at massive scale.

This is where the New Relic platform enters the picture as a powerful and uniquely capable complement for today’s Kubernetes environments. New Relic is all in on Kubernetes, which is why we have been steadily expanding and upgrading our ability to instrument, monitor, and support full observability across Kubernetes environments. This includes giving you the ability to move quickly and easily from high-level, holistic views of the performance and health of Kubernetes clusters, down to the node, container, and application-level visibility required to identify and troubleshoot performance issues. It also includes the ability to instrument and monitor Kubernetes infrastructure alongside applications, which critical for understanding dependencies and isolating potential performance issues.

Using New Relic to achieve end-to-end Kubernetes monitoring

With that in mind, we’d like to introduce you to some of the most powerful capabilities currently being introduced in the New Relic platform for monitoring Kubernetes applications and resolving performance issues in these environments. In particular, we’ll demonstrate how to use the Kubernetes Downward API to surface Kubernetes-specific metadata; and then how to use the New Relic APM agent API to tie this metadata into APM in order to explore performance issues and troubleshoot application transaction errors.

The importance of an end-to-end view

By design, applications typically aren’t aware of whether they’re running in a container or on an orchestration platform. A Java web application, for example, doesn’t have any special code paths that get executed only when running inside a Docker container on a Kubernetes cluster. This is a key benefit of containers (and container orchestration engines): your application and its business logic are decoupled from the specific details of its runtime environment. If you ever have to shift the underlying infrastructure to a new container runtime or Linux version, you won’t have to completely rewrite the application code.

Similarly, New Relic APM’s language agent—which instruments the application code to track rich events, metrics, and traces—doesn’t care where the application is running. It could be running in an ancient Linux server in a forgotten rack or on the latest 72-CPU Amazon EC2 instance. When monitoring applications managed by an orchestration layer, however, being able to relate an application error trace, for instance, to the container, pod, or host it’s running in can be very useful for debugging or troubleshooting.

All of these are must-have capabilities for a Kubernetes application monitoring solution because they help our customers answer an all-too-common question: Does a performance problem really reside in the code, or is it actually tied to the underlying infrastructure?

Application instrumentation with the Kubernetes Downward API

Traditionally, for application transaction traces or distributed traces you collect using New Relic APM, our agent can tell you exactly which server the code was running on. In many container environments, though, this gets more complex: the worker nodes (hosts) where the application runs (in the containers/pods) are often ephemeral—they come and go. It’s fairly common to configure policies for applications running in Kubernetes to automatically scale their host count in response to traffic fluctuations, so you may investigate a slow transaction trace, but the container or host where that application was running no longer exists. Knowing the containers or hosts where the application is currently running is not necessarily an indication of where it was running 5, 15, or 30 minutes ago—when the issue occurred.

Fortunately, the Kubernetes Downward API allows containers to consume information about the cluster and pod they’re running in using environment variables instead of external API calls. We can use a small sample app to demonstrate how this works in a Node.js application. (Feel free to fork this repo for your own use.)

To integrate with the Downward API, we edited the environment section of the application manifest; and we defined environment variables that would give a containerized application access to the Kubernetes node name, host IP, pod name, pod namespace, pod tier, and pod service account in which it’s running.

A few examples of these variables include:
```
- name: K8S_NODE_NAME
  valueFrom:
    fieldRef:
      fieldPath: spec.nodeName
- name: K8S_HOST_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP
- name: K8S_POD_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.name
```
From there, we use the New Relic Node.js Agent API to expose those environment variables as custom parameters for all application transaction traces in an express.js application:
```
// Middleware for adding custom attributes
// These map to environment variables exposed in the pod spec
var CUSTOM_PARAMETERS = {
    'K8S_NODE_NAME': process.env.K8S_NODE_NAME,
    'K8S_HOST_IP': process.env.K8S_HOST_IP,
    'K8S_POD_NAME': process.env.K8S_POD_NAME,
    'K8S_POD_NAMESPACE': process.env.K8S_POD_NAMESPACE,
    'K8S_POD_IP': process.env.K8S_POD_IP,
    'K8S_POD_SERVICE_ACCOUNT': process.env.K8S_POD_SERVICE_ACCOUNT,
    'K8S_POD_TIER': process.env.K8S_POD_TIER
};

app.use(function(req, res, next) {
  newrelic.addCustomParameters(CUSTOM_PARAMETERS);
  next();
});
```
Exploring application performance in Kubernetes with New Relic

To get the instrumentation in place, we deployed the updated application code to the Kubernetes cluster, and then the custom parameters from Kubernetes began to appear in the New Relic UI.

In the following screenshot, we see some error details; the transaction attributes shows us, among other details, the Kubernetes hostname and IP address where the error occurred:

We also see the same metadata show up under Distributed Trace attributes.

Next, we used New Relic Insights to look at the same instrumentation to see the performance of application transactions based on pod names. To do this in Insights, we simply wrote the following custom New Relic Query Language (NRQL) query:
```
SELECT percentile(duration, 95) from Transaction where appName='newrelic-k8s-node-redis' and name='WebTransaction/Expressjs/GET//'FACET K8S_POD_NAME TIMESERIES auto
```
And here’s the result:

The Downward API exposes a good deal of useful metadata for monitoring solutions. You can use it to gain useful information about performance outliers and track down individual errors. In aggregate, APM error profiles also surfaces any correlations between errors in transactions that have been instrumented with data you’ve gathered with the Downward API.

For instance, in this example, APM error profiles automatically notices that nearly 57% of errors come from the same pods and pod IP addresses:

APM error profiles automatically incorporates the custom parameters and uses different statistical measures to determine if an unusual number of errors is coming from a certain pod, IP, or host within the container cluster. From there, you can zero in on infrastructure or cluster-specific root causes of the errors (or maybe you’ll just discover some bad code).

APM and the Kubernetes cluster explorer

The discussion above shows how to leverage Kubernetes metadata within APM to help troubleshoot issues. Conversely, if you are starting your investigation into an issue from the infrastructure side of the house, you can also visualize application performance metrics within the newly announced New Relic Kubernetes cluster explorer. If you identify issues within a node or a pod but don’t see any issues with the infrastructure itself, you can quickly check application metrics (throughput, response time, error rate and violations) for the application instance running on that specific node/pod.

The Kubernetes cluster explorer lets teams drill down to get visibility into applications and infrastructure metrics side-by-side in a rich, curated UI that simplifies complex environments.

Effectively monitoring applications running in Kubernetes requires not just code-level visibility into the applications but also the ability to correlate applications with the containers, pods, or hosts in which they’re running. By identifying pod-specific performance issues for an application’s workload, you can troubleshoot performance issues faster and have confidence that your application is always available, running fast, and doing what it is supposed to do.

Using the Kubernetes cluster explorer, you can get an understanding of how your application is performing. Additionally, clicking on the application (in this example ‘Order-Packaging’) sends you to the APM overview page where you can troubleshoot and dissect that specific application.

Your customers may not care about Kubernetes … but you must

Your customers likely don’t care if you’re using traditional virtual machines; a bleeding-edge, multi-cloud federated Kubernetes cluster; or a home-grown artisanal orchestration layer. They just want the applications and services you provide to be available, reliable, and fast.

With the rise of Kubernetes, however, along with new technical concepts like orchestration layers, teams need new contexts for understanding and exploring performance in their applications and code. We believe that to efficiently build and run modern software, you need visibility into the application layer—especially when those applications are running inside Kubernetes clusters. We look forward to continuing our work with customers and the broader Kubernetes community, and to creating and providing the solutions they need to succeed.

Learn more about monitoring Kubernetes and the new cluster explorer by visiting New Relic at KubeCon + CloudNativeCon North America 2018, in Seattle, on December 11-13, at booth #P20.

52 / 53

How to Run an Adversarial Game Day • sorangutan

Best Software Developer Events and Conferences: A Global Guide • sorangutan

Building An Inventory Management Database with The Kubernetes API Server And Custom Resources • sorangutan

How the New Relic Infrastructure Team Automated Its Windows Build-and-Deploy Pipeline • sorangutan

Getting Started with New Relic Mobile: Best Practices That Set You up for Success • sorangutan

Introducing New Relic’s Ignite Program for Early Career Engineers • sorangutan

Breaking to Learn: Chaos Engineering Explained • sorangutan

3 Ways New Relic SREs Use the New Relic Platform • sorangutan

FutureStack Berlin Video—Insights on Digital Transformations from Flixbus, OneFootball, and AWS • sorangutan

6 Things New Relic Believes About Modern Monitoring • sorangutan

10 Secrets You Should Have Learned With Your Software Engineering Degree—or at Coding BootCamp—but Probably Didn’t education developers culture software development coding tips dev engineers • • sorangutan

Technical skills

Mindsets, habits, and soft skills

Looking Back at New Relic’s Amazing 2018 kubernetes community aws lambda new relic news and produc customers distributed tracing futurestack total economic impact • • sorangutan

Success by the numbers: 17,000 customers and counting

Kubernetes Cluster Explorer makes its debut

Distributed tracing anomaly detection

Support for AWS Lambda

More 2018 rollouts

Global growth—and a new acquisition

New resources for our developer community

FutureStack goes global—and stays local

But wait, there’s more … isn’t there?

OOO, It’s Time For New Relic Out-of-Office Replies culture holidays travel vacation • • sorangutan

A real puzzler

Into the wild

Wishing and hoping

An explosive good time

You only live once

Spriiiiiiiiiiing break!

O Canada

Poutine dreams

Getting Started with New Relic Insights: Best Practices That Set You Up for Success new relic news and produc new relic insights best practices dashboards nrql • • sorangutan

Getting started and organized with New Relic Insights

NRQL -> Widgets -> Dashboards -> Data Apps

Learn more about data apps and dashboards in New Relic Insights

Get familiar with the Insights dashboard API

Add Dashboard descriptions with notes

Understanding your New Relic Insights data

Use the data formatter to set default numeric value formats

Explore and understand attributes

Metric explorer and event explorer

Using Insights to make your data actionable

Create NRQL alerts

Send custom data

Ready to expand your knowledge of New Relic Insights?

How Kafka’s Consumer Auto Commit Configuration Can Lead to Potential Duplication or Data Loss apache kafka troubleshooting software engineering • • sorangutan

A look at the problem

Ok, so what can I do about it?

Moving forward

Five Steps for an Incident Response Workflow, Featuring the Jira Software and New Relic Integration new relic news and produc alerting incident response jira software • • sorangutan

1. Assign owners for team dashboards

2. Determine incident thresholds for alert conditions

3. Ensure alerts have auditable notification channels

4. Create faster triage and resolution times with New Relic and Jira Software

5. Hold a retrospective and document your findings

DevOps Weekly’s Gareth Rushgrove on the History and Evolution of DevOps devops technology industry sre • • sorangutan

Serverless has fans, too

DevOps drivers vs. DevOps barriers

Getting started with DevOps

The critical role of monitoring in DevOps

The DevOps Weekly—there from almost the beginning

Is DevOps cool?

Trying Out New Relic APM for Lambda With an Image Recognition Prototype App aws aws lambda software engineering new relic insights apm for lambda • • sorangutan

Inside my Amazon Rekognition prototype app

A closer look at each part of the pipeline

AWS and APM—always raising the bar

Edge Computing vs. Cloud Computing: Which One Is Right for Your Application? edge computing technology industry cloud computing • • sorangutan

The edge, the cloud, or both? The case of the autonomous car

The edge’s advantages and challenges

Seven keys to successfully build edge computing into your application

Monitoring Application Performance in Kubernetes: How New Relic Sets You Up for Success kubernetes container orchestration software engineering new relic apm new relic insights • • sorangutan

New Relic and Kubernetes: A powerful pairing

Using New Relic to achieve end-to-end Kubernetes monitoring

The importance of an end-to-end view

Application instrumentation with the Kubernetes Downward API

Exploring application performance in Kubernetes with New Relic

APM and the Kubernetes cluster explorer

Your customers may not care about Kubernetes … but you must

How to Run an Adversarial Game Day
• sorangutan

Best Software Developer Events and Conferences: A Global Guide
• sorangutan

Building An Inventory Management Database with The Kubernetes API Server And Custom Resources
• sorangutan

How the New Relic Infrastructure Team Automated Its Windows Build-and-Deploy Pipeline
• sorangutan

Getting Started with New Relic Mobile: Best Practices That Set You up for Success
• sorangutan

Introducing New Relic’s Ignite Program for Early Career Engineers
• sorangutan

Breaking to Learn: Chaos Engineering Explained
• sorangutan

3 Ways New Relic SREs Use the New Relic Platform
• sorangutan

FutureStack Berlin Video—Insights on Digital Transformations from Flixbus, OneFootball, and AWS
• sorangutan

6 Things New Relic Believes About Modern Monitoring
• sorangutan

10 Secrets You Should Have Learned With Your Software Engineering Degree—or at Coding BootCamp—but Probably Didn’t
education developers culture software development coding tips dev engineers • • sorangutan

Looking Back at New Relic’s Amazing 2018
kubernetes community aws lambda new relic news and produc customers distributed tracing futurestack total economic impact • • sorangutan

OOO, It’s Time For New Relic Out-of-Office Replies
culture holidays travel vacation • • sorangutan

Getting Started with New Relic Insights: Best Practices That Set You Up for Success
new relic news and produc new relic insights best practices dashboards nrql • • sorangutan

How Kafka’s Consumer Auto Commit Configuration Can Lead to Potential Duplication or Data Loss
apache kafka troubleshooting software engineering • • sorangutan

Five Steps for an Incident Response Workflow, Featuring the Jira Software and New Relic Integration
new relic news and produc alerting incident response jira software • • sorangutan

DevOps Weekly’s Gareth Rushgrove on the History and Evolution of DevOps
devops technology industry sre • • sorangutan

Trying Out New Relic APM for Lambda With an Image Recognition Prototype App
aws aws lambda software engineering new relic insights apm for lambda • • sorangutan

Edge Computing vs. Cloud Computing: Which One Is Right for Your Application?
edge computing technology industry cloud computing • • sorangutan

Monitoring Application Performance in Kubernetes: How New Relic Sets You Up for Success
kubernetes container orchestration software engineering new relic apm new relic insights • • sorangutan