The Sysdig Container Intelligence Platform is now offered as a Red Hat Certified Image
The Red Hat OpenShift Container Platform helps developers easily and quickly develop, build, and deploy container-native applications in nearly any infrastructure, public or private. But as you move from development to a large scale production environment, monitoring and security take center stage.
The ephemeral nature of containers and growing scale make it difficult to track the health of the services from the individual containers. Monitoring at the cluster level — rather than single containers — is a better way to understand how they interact with each other, which is vital when using microservices that spans multiple containers and services.
Unfortunately, legacy monitoring and security tools were not built for containers and cannot see, analyze or protect containers and microservices. Containers are like black boxes, useful for moving applications from development into production. They provide a great level of portability and isolation, but makes it harder to understand what’s happening inside, monitor and secure it. Microservices help to develop applications faster and orchestration tools allow admins to deploy and dynamically scale easily but now we have different pieces moving around and monitoring and security tools need to understand the state of the infrastructure and the applications running on it. Fortunately, the Sysdig Container Intelligence Platform was purpose-built to monitor and secure container environments like the Red Hat OpenShift Container Platform. The Sysdig Container Intelligence Platform recently achieved Red Hat OpenShift Certification and is available in the Red Hat OpenShift Container Catalog.
In this post we’ll review the challenges of managing container-native applications in large scale deployments, and give a number of examples of how you can monitor and secure the Red Hat OpenShift Container Platform and other Kubernetes environments with Sysdig:
- How troubleshooting, post-mortem analysis and forensics change in containerized environments.
- Best practices for leveraging Kubernetes metadata to monitor and secure your infrastructure and applications.
- How Sysdig can help you implement monitoring and security with a single container agent per host, fine-tuned for OpenShift.
Why unify monitoring & security?
“The purpose and intent of DevSecOps is to build on the mindset that 'everyone is responsible for security' with the goal of safely distributing security decisions at speed and scale to those who hold the highest level of context without sacrificing the safety required.” - DevSecOps
The rise of DevSecOps has created new paradigm of “Continuous Security”, where security professionals work to add security and compliance across the entire the software supply chain. By working side-by-side with developers, a set of security services as tooling can be infused into agile development processes, to make sure the applications are healthy, as well as secure.
This new “Continuous Security” practice, focuses workflows around three main concepts:
- Provenance – what’s the origin of my application and container properties (source, build, test status).
- Visibility – what’s the health of my service? Is my infrastructure secure.
- Forensics – what happened to my service that crashed? What unexpected outbound connection was spawned, and what data was written to disk?
Sysdig is executing on this vision by providing deep insight into the provenance and health of your infrastructure and container-native applications. Sysdig’s unique approach uses a single monitoring agent per OpenShift node to provide both security and monitoring. Container performance metrics and security events are enriched with service-oriented metadata from your cloud platform and orchestration tools enabling you to aggregate metrics at the service level or drill into your data to troubleshoot a specific issue by application, container image, host or network.
Getting started with Red Hat OpenShift & Sysdig
If you don’t have an OpenShift cluster, check out How to deploy OpenShift on AWS to get step by step instructions for deploying and get your environment running in minutes.
To get started instrumenting your environment, we will run the Sysdig agent on each of our hosts in the OpenShift cluster. Sysdig will automatically identify hundreds of application metrics out-of-the-box. It will also discover and tag custom metrics, including Prometheus metrics.
For more info on how to get started with Sysdig for OpenShift and how Sysdig collects data from your environment, check out Sysdig installation for OpenShift.
Visibility into Kubernetes services
One of the best parts of Kubernetes is how comprehensive the internal labeling is. Sysdig ServiceVision provides data enrichment with service-oriented metadata that increases the value of your information. You can quickly visualize the health of all infrastructure components - grouped or individually - in graph, table, and map formats, from a variety of host and cloud services ‘views’.
You’re able to explore your containers based on their physical hierarchy (for example, Host > Pod > Container) or based on their logical microservice hierarchy (for example, Namespace > Deployment > Pod > Container).
Difference between a physical and a logical grouping to monitor your containers with Kubernetes context.
If you’re interested in the utilization of your underlying physical resource (eg. identifying noisy neighbors) then the physical hierarchy is great. But if you’re looking to explore the performance of your applications and microservices, then the logical hierarchy is often the best place to start.
In general, the ability to regroup your infrastructure on the fly is a more powerful way to troubleshoot your environment as compared to the typical static dashboards. To learn more about this approach, check out this Kubernetes monitoring guide.
Securing Kubernetes services
This same metadata can be used to apply security policies and protect your Kubernetes services - if you are collecting the right security and compliance events. Luckily, Sysdig does that alongside monitoring.
Using a label like kubernetes.deployment.name, we can enforce a policy to protect a logical service regardless of how many containers, hosts or regions the deployment is running in. Other approaches only enforce a policy by image name... leading to all sorts of complications in any environment larger than a test cluster.
What we’re looking at below is a policy to protect my Redis Kubernetes deployment from an exfiltration event by detecting an unexpected outbound connection from that logical service. From there, we can also take actions on any policy violation to quarantine the container before any data has left our Redis service.
If you are looking into more information about security on OpenShift and Kubernetes we would like to recommend this Kubernetes security guide.
Forensics in container environments
Doing forensics and incident response in container environments is challenging: containers are ephemeral and the data we want is often long gone. Also, they’re essentially black boxes and it’s often hard to tell what’s actually running inside of them.
We don’t have time to ssh into the host and run a core dump if Kubernetes is killing our containers. Also, the required troubleshooting and analysis tools are not often installed and doing so is not probably a good idea. Our system needs to proactively capture and alert on all activity with the ability to analyze that data outside of production. If you need to look at historical data - say from a week ago during an unexpected run-time compliance violation - you likely won’t have any meaningful data in your health dashboards in OpenShift or Kubernetes.
Sysdig’s unique instrumentation allows us to capture all activities from users, processes, and even contents written to file or passed over to the network, pre and post policy violation, and then store it indefinitely. This is something so powerful that it’s best explained over a quick one minute video: check out this analysis of what can happen when a user spawns a shell in a container and all the data we can collect about their subsequent actions.
For more details on container troubleshooting and forensics, read Sysdig Inspect: A GUI for system call analysis.
In this post you have learned how adopting containers and microservices as part of your infrastructure stack requires new layers and new processes. OpenShift provides excellent capabilities to implement your pipeline from development into production. But you shouldn’t stop with deployment. You still need to make sure your operations are agile, healthy and secure.
Sysdig Container Intelligence platform provides the perfect OpenShift companion to extend your pipeline beyond deployment: monitoring and securing your infrastructure and applications running on OpenShift. Sysdig goes one step further and thanks to its troubleshooting, post-mortem analysis and forensic capabilities can close the loop and give back to your DevOps team, developers and security analyst the required information of what happened, where, why and who was involved.
The views, opinions, and positions expressed within guest posts are those of the author and do not represent those of Red Hat or OpenShift. The accuracy, completeness, and validity of any statements made within this article are not guaranteed. The copyright of this content belongs to the author and any liability with regards to infringement of intellectual property rights remains with them.