Follow us
Breaking
Software

Platform teams avoiding Kubernetes resource and security traps

Platform engineers face resource starvation and security risks when misconfiguring Kubernetes limits or RBAC. Managed OpenShift provides integrated security and CI/CD, while Nomad offers simpler scaling up to 10,000 nodes for smaller IT departments.

Share

Platform teams often stumble when they ignore resource constraints in Kubernetes. Forgetting resource limits turns a pod into a thirsty shark that devours CPU and memory, which triggers evictions for every other pod on the same node. If a container requires more memory than a node holds, the scheduler cannot place the pod, and if memory usage exceeds the limit, the kubelet terminates the container with an exit code 137. I find that configuring resource requests to the steady-state p95 level helps avoid these issues. Setting memory requests to the steady-state plateau while setting limits to cover the startup peak prevents these failures. If you request more memory than the available memory, pods fail to schedule, and if memory requests fall below the limit, Kubernetes terminates the pod and triggers a restart via kubelet. For CPU, a request of 1000m equals one core, while 250m equals 1/4 of a core. If a container hits its CPU limit, the kernel’s CFS quota slows down the application rather than stopping the container. Platform engineers use pod priority and preemption to manage these imbalances, where a higher priority pod in a queue evicts a lower priority pod. You can also use ResourceQuota to cap the total resources all pods in a namespace can request. To prevent one object from using all resources and causing starvation, you can use LimitRange to set minimum and maximum limits on CPU, RAM, and storage requests. This complexity differs from HashiCorp’s Nomad, which follows UNIX design principles of small scope and cooperability. Nomad uses HCL for configuration, while Kubernetes uses YAML or JSON. Nomad also scales to 10,000 nodes and 2 million containers, whereas Kubernetes scales to 5,000 nodes and 300,000 containers.

Platform engineers often increase operational costs by granting broad cluster roles to move fast. Kubernetes uses Role-Based Access Control where permissions are purely additive, meaning the system lacks deny rules to block specific actions. If you grant the pod exec subresource permission, that user can open a shell and run any command inside the container. This capability allows a user to read local files or query databases, which makes manual log review a massive labor expense. OpenShift uses Kubernetes RBAC but adds stricter defaults, such as running containers as non-root users by default. It also includes image vulnerability scanning and network policy enforcement. A Role sets permissions inside one namespace, while a ClusterRole sets permissions across all namespaces. You attach these rules to users with a RoleBinding or ClusterRoleBinding. To mitigate risk, tools like hoop.dev run as a sidecar for the Kubernetes API to check every request and response. This tool can redact matched keys in returned JSON responses and log streams in memory. You might want to check your current resource usage before making these changes. Do these automated policies truly stop a rogue script from querying a database after a user gains shell access?

Choosing between these platforms requires weighing immediate setup speed against long-term maintenance. OpenShift provides a ready-to-use environment with built-in CI/CD pipelines and image registries, whereas Kubernetes requires you to piece together external tools like Jenkins, Prometheus, or Fluentd. I suggest you look at the total cost, because software licenses account for only one part of the spend. Engineering hours spent managing roles, audits, and access tickets add huge hidden costs because administrators must manually review long audit trails to identify which users ran specific commands inside a pod. Managed OpenShift on a CSP for 24/7 availability costs 10,100 CHF per month. This price includes worker capacity and multi-AZ design.

Feature Nomad Kubernetes
Max Nodes 10,000 5,000
Max Containers 2,000,000 300,000
Config Language HCL YAML or JSON
External Services Built-in Third-party

Nomad supports virtualized, containerized, and standalone apps, while Kubernetes focuses on Linux containers. Nomad provides simplicity and flexibility, so it is a good choice for organizations with smaller IT departments. In contrast, Kubernetes provides fine-grained control through abstractions like Pods and Services. Organizations with smaller IT departments should choose Nomad for its simplicity, while large enterprises requiring integrated security and CI/CD should select OpenShift.

Share

Technewsdaily

Senior tech writer covering AI, gadgets and cybersecurity. Breaking down the news that matters, every day.