Kubernetes is becoming the de facto standard when it comes to application deployment and management, just like AWS did for Cloud Computing and VMware did for Enterprise Virtualisation. There is therefore a fair assumption that when your team starts designing a new developer platform for the dev teams to use, it will be backed by Kubernetes. Whether this is running on Cloud, in your Datacenter or both will depend on your organisation’s circumstances and requirements.
And while Kubernetes is a very capable platform, it requires additional components to make it a great platform for dev teams to use. While these components can be implemented after the platform has been designed and deployed, it becomes increasingly time consuming and costly to do so after the fact. In some circumstances it may even require a complete redesign/rebuild which is often not an ideal experience for dev teams. By keeping these 4 components at the forefront during the design and build stages of your platform you are setting yourself up for a great developer experience from Day One. Let’s go through each of these 4 components in a bit more detail.
Logging
When moving to a container based platform the first thing teams will notice is that it becomes more difficult to troubleshoot when and what happened on an application. Because of the inherent nature of containers being ephemeral, making sure that adequate logging is in place should be a top priority from the beginning. It will help you find issues in your apps faster, helping your dev teams to spend less time troubleshooting/bug fixing and more time developing new cool features for your customers.
Logs become incredibly important when auditing who done what where in your environment. Kubernetes is backed by an API which by default keeps track of all requests, once your environment starts growing this can easily generate millions of audit events per day.
It therefore becomes incredibly important that you choose a solution for storing and managing these events and logs. And while starting out, one solution may be sufficient to store and manage all of these logs, you will most likely end up with multiple solutions for each set of logs. The 3 sets of logs you should store at a minimum are:
Application logs
Application logs will assist you in troubleshooting application problems
System component logs
System component logs will assist you to troubleshoot issues on the platform itself
Audit events
Audit events will assist you when someone comes to ask you to let them know who deleted a namespace or scaled down the wrong deployment. It will also assist when the security teams come knocking after a possible system breach.
Some tools that you can look into include Elastic, Kibana, Fluentd, Graylog, Splunk, Datadog, Logstash, etc.
Monitoring
Moving to a cloud-native/microservice application architecture is difficult at the best of times, doing so without a monitoring solution that’s capable of assisting teams to troubleshoot effectively is a disaster waiting to happen.
Adopting cloud-native applications that could be running across 10s or even 100s of micro-services and possibly across multiple geographical regions is no mean feat. While this design can make applications extremely resilient it can also make it extremely difficult to troubleshoot when things go pear shaped, regardless of whether this was introduced via a change or a component failure.
As with the logging components, when starting out, one monitoring solution may be sufficient to cover these requirements. However, as your environment starts to grow you’ll most likely realise that certain solutions shine in only a few areas and you’ll require multiple solutions to make sure you have the best coverage. Make certain your solutions cover all components of your platform, from the application requests down to your cloud or on-prem infrastructure (yes, Public Clouds do not have 100% uptime).
Some open source products you could look into include Grafana, Kiali, Prometheus, Kuberhealthy, Thanos, OpenMetrics, etc.
Security
With the increased adoption of Kubernetes for application deployment and management, you are almost certainly guaranteed that it will receive a high level of focus from bad actors. This will also extend to any tool sets that interact with Kubernetes such as CI/CD platforms used to build and initiate deployments of applications.
While most of these security features and functions can be enabled/added onto after the fact, they are often time consuming tasks when doing so on a live environment and may even require a rebuild of a platform. Security standards for these platforms are constantly changing, sometimes even monthly, so building your platform from the beginning with a strong security baseline can help ease the burden of keeping it as secure as possible.
Moving to a micro-services architecture means that your attack surface increases exponentially. Where previously you had to make sure that your operating systems were patched and the software you were using didn’t have any vulnerabilities, you now also have to contend with the thousands and later tens or hundreds of thousands of containers running in your environment that could each potentially expose you to a system breach.
Some products you should look at using include Cert-Manager, Clair, Falco, Kube-Warden, Kyverno, Neuvector, Notary, OPA, Trivy, etc.
Training
While this topic is not a component you can deploy on your platform, it is in my opinion the most overlooked part when moving onto a Kubernetes platform.
At the end of the day, you are building a platform that will be used by a multitude of teams. It is therefore incredibly important that they understand the fundamentals of what makes it work as well as its capabilities and features.
Training should not just cover the platform support teams and as well as development teams, it should cover any team that interacts with the platform in any way. Whether this be the security teams that may be responsible for maintaining the security posture of the platform, to the network team that may become involved in troubleshooting some application connectivity issues.
Training is possibly the cheapest and easiest of these 4 components to implement. Training up teams of SMEs who can then cross-skill and then provide internal training can help keep costs to a minimum. There are also thousands of hours of free training resources available online for teams to consume. These resources constantly get updated as new tools and components get released.
Youtube is a great source of free professional grade training materials, you can also check out the below providers
https://www.cncf.io/certification/training/
https://www.youtube.com/c/cloudnativefdn
Summary
In my opinion, these 4 components are often overlooked when building the first development platform. Teams then have to spend an incredible amount of resources to make sure that they get implemented as soon as possible. They are critical parts of the system and should be part of your solution before going live.
Tom's Blog 
You must be logged in to post a comment.