• 2 Posts
  • 63 Comments
Joined 1 year ago
cake
Cake day: June 19th, 2023

help-circle
  • Certs are a waste of time tbh. If you have 8 years of experience, you should have more than enough to fill out a resume already.

    An AWS cert is almost certainly even more useless for you specifically unless you wanted to get into devops/sre and do systems design. I have been in sre for a very long time and have never even heard of anyone writing tooling in Java. That section of the industry is entirely dominated by go, python, and (more often than anything else) bash for really quick automation.



  • Hated Windows. TechTV had a download of day that “works on both Windows and Linux!”

    “I don’t know what Linux is but it can’t be worse that Windows.”

    I’ve been on it ever since. That was 20+ years ago.

    I honestly don’t know how windows works… I only ever used it for about a year and some change when I was a teenager in the 90s.


  • It’s easier to think about Linux on the context of what an individual application needs to run. Pretty much everything you do will have these components.

    • configuration
    • an executable
    • a communication mechanism (dbus, networking, web server, etc)
    • something that decides if the application runs or not (systemd, monit, docker/docker compose, kubernetes scheduler, or you as the user)
    • a way of accepting input (keyboard and mouse, web requests, database queries, etc)
    • a way of delivering an output (logging to unique log files, through syslog, or to stdout/stderr, showing something on a screen, playing a sound, returning a message to the client, etc)
    • storage (optional)
    • some cpu and memory capacity

    That’s really it. If something isn’t working, it’s pretty much exclusively going to fall into one of those categories. What that means is going to vary significantly from app to app but understanding this is how literally everything works makes the troubleshooting process a lot easier.


  • My personal laptop is whatever the first gen Framework is called. After many, many years doing the “cool” distros, I’ve settled on Mint and don’t really have any motivation to do anything else… I have real work I need to do and can’t be bothered to deal with figuring out weird shit. I just need it to work.

    TBH, the only things I use my laptop for anymore is a browser, vim, git, and kubernetes tooling… I barely have any interest in running Linux on a workstation at this point. The only things that really interest me anymore are being run in distributed clusters. Desktop Linux is kinda boring and tedious for me.





  • As a counter balance to that though, interviewers need to understand what they are hiring for and tailor the questions asked to those requirements.

    For example, there is genuinely very little coding required of an SRE these days but EVERY job interview wants you to do some leetcode style algorithm design… Since containers took over, the times I have used anything beyond relatively unremarkable bash scripts is exceptionally small. It’s extremely unlikely that I will be responsible for a task that is so dependent on performance that I need to design a perfect O(1) algorithm. On terraform though, I’m a fucking surgeon.

    SRE specifically should HEAVILY focus on system design and almost all other things should have much much less priority… I’ve failed plenty of skill assessments just because of the code though.





  • Most resistance I have seen mostly comes down to a misunderstanding in the benefits that kubernetes offers. The assumption is that kube is used for autoscaling and that, if the inbound traffic is predictable then the added complexity is unnecessary. When that happens the “kube isn’t right for all situations” turns into “kube isn’t right for any situation” whether the person in question would ever admit that or not…

    All of this ignores the MASSIVE reliability enhancement kube delivers and the huge amount of effort currently going into modern tool development surrounding the kube ecosystem.


  • Real talk, you don’t have the luxury of being an idealist right out of university. Your goal is to get a job. When you’re in that job you will likely not have the luxury of being an idealist either.

    When you have enough experience making practical, reasoned decisions, then you can stand on principals.

    For context, I have been in this business for nearly 20 years. The people I have personally worked with who have resisted things on philosophical grounds ALWAYS get left behind. I’ve seen it with systemd, the cloud, and now I’m seeing it again with kubernetes. You cannot escape the collective inertia of an entire industry.

    Obviously there are still thresholds… I would never work for someone like Raytheon. You have to draw lines somewhere but saying you aren’t going to work for a company that does user behavior tracking is short sighted and impractical.




  • We had a service that compiles a dataset once per quarter. The total size is ~30gb. We were starting a container, storing it on an EFS volume, and mounting like any other disk.

    Every time a pod started it would need to read this data into memory so we would get quick initial start-up time but the time to be ready for traffic still took a while.

    Since we didn’t need to update it very often, we decided to just package the compiled dataset into the container and skip the EFS volume. We updated the image pull policy to ifNotPresent so it cut egress traffic pricing from EFS to zero. Now there is a cost to pull the image from ECR but that’s only if the pod is being scheduled onto a node it hasn’t been run on before. There was no noticable change in behavior or performance and we saved a bunch on cost.

    Sometimes the big, dumb option is the right choice.




  • We focus a lot more on production than the average developer. It’s our job to make sure whatever devs build is run quickly, efficiently, safely, and scalably.

    You will work with a lot of kubernetes, Argo, terraform, Prometheus, grafana. You’ll design build pipelines and software rollout strategies. You plan for zero downtime migrations and upgrades, database maintenance… You’ll have your hands in everything from capacity planning to security to cost optimization to developer support… User permissions, infrastructure, networking, observability… You will write RFCs and setup POCs for new tools. You define and track error budgets and figure out how to keep your org under those projections. When there is an outage you will be involved in writing post mortems.

    The days are so varied and unpredictable that it keeps things interesting. The landscape changes so often you’re never really stuck doing the same thing over and over.

    I genuinely love it.

    EDIT: The SRE Podcast from Google is actually really great for learning about this world. The first season talks about what you’ll be doing and why (based around the SRE O’Riley book). The second season talks about what to expect in different stages of your career progression.