Hey Codereliant friends!

Our team was busy creating the new AI-powered Kubernetes UI project, born from our experience managing 5,000 node clusters. Introducing Kubeintel!

TLDR;

🚀 Try the live demo (no registration required): https://demo.kubeintel.io/search

🔗 Learn more: https://kubeintel.io/

🔔 Join the waitlist: https://kubeintel.io/waitlist

🎥 Video introduction:

Many current Kubernetes UIs struggle with large clusters due to:

• Poor scalability: Not designed for 1000+ node clusters, leading to performance issues for both control plane and client-side operations. Issues include inefficient querying, excessive data fetching, lack of caching, and high API server load.

• Limited search capabilities: Requiring precise knowledge (e.g., namespace or deployment name) or resorting to slow and heavy cluster-wide queries (kubectl get pods -A sound familiar?).

• The lack of AI-powered troubleshooting tools creates substantial additional work for infrastructure, platform, & dev teams.

Consequently, many teams fall back to using terminals for cluster and workload navigation. We began developing Kubeintel a few months ago to address these challenges.

Key features:

• Lightning-fast UI and resource search - scaling seamlessly from 10 to 5000+ node clusters

• Intelligent full-text search - typo-tolerant for effortless resource discovery

• Near-zero control plane overhead - utilizing an internal embedded database instead of constant K8s API queries

• AI-powered insights and troubleshooting - leveraging popular LLMs (OpenAI, Anthropic, Gemini, Ollama) to diagnose resource issues, such as pod crashes.

This is an early preview, and we want to hear from you. We'd love your feedback:

• What specific pain points in your K8s workflow would you like addressed?

• How does Kubeintel compare to your current cluster navigation tools?

• What features would you like to see added?

Honey wake up, the new SSH vulnerability just dropped.

A new high-severity vulnerability in OpenSSH, dubbed "regreSSHion" (CVE-2024-6387), has been discovered couple days ago. This flaw affects OpenSSH servers on glibc-based Linux systems and could allow unauthenticated remote code execution with root privileges.

What you need to know:

• The vulnerability impacts OpenSSH versions 8.5p1 to 9.7p1, as well as versions earlier than 4.4p1 (if not patched for older CVEs).

• Over 14 million potentially vulnerable instances are exposed to the internet.

• While there's no known exploitation in the wild yet, a working exploit has been developed by researchers. The complexity of the exploit might delay widespread attacks, but it's important to act quickly.

Action items:

1. Update all OpenSSH instances to version 9.8p1 or later ASAP.

2. If immediate patching isn't possible, set LoginGraceTime to 0 in the sshd config file as a temporary mitigation.

3. Reduce the number of internet-facing sshd servers where possible to be less prone to Zero-day issues like this one.

If you want to know more you can check out the deep dive by Qualys Threat Research Unit, who discovered and reported the issue. Stay secure!

In our security analysis, we identified that this vulnerability is a regression of the previously patched vulnerability CVE-2006-5051, which was reported in 2006. A regression in this context means that a flaw, once fixed, has reappeared in a subsequent software release, typically due to changes or updates that inadvertently reintroduce the issue.

The good old make command has been around for ages. It helps us compile code, build projects, run tests and automate all sorts of tasks. As projects grow, Makefiles can become hard to use. The syntax, while simple, can get cryptic with complex rules and conditionals. Debugging Makefiles is no fun either. Error messages from make can be hard to decipher, making it tough to figure out what went wrong. A single space in the wrong place can result in Makefile:42: *** commands commence before first target. Stop. 😬

Meet Taskfile - a task runner tool that aims to be simpler and easier to use than Makefile

What's Taskfile?

A Taskfile is essentially a configuration file for the Task tool. It uses a YAML-based syntax, which is both human-readable and easy to write. Instead of the somewhat cryptic syntax of Makefiles, Taskfiles are structured and clear. This makes them easier to maintain and understand, especially for new team members.

Let’s take a look at some of the key features:

• Cross-Platform Compatibility - one of the coolest things about Taskfiles is their cross-platform support. You write a Taskfile once, and it works on any system that has the Task tool installed, even Windows!

• Built-in Dependency Management - similar to Makefile, Taskfiles offer built-in support for dependencies. You can define which tasks depend on others, ensuring that tasks run in the correct order.

• Parallel Execution - you can easily specify tasks to run in parallel, which can significantly speed up your build and deployment processes

• Templating and Variable Substitution - you can define variables and templates directly in your Taskfile, making your task definitions more dynamic and reusable.

• Simplicity and Readability - YAML readability makes it easy for anyone on your team to understand and modify the Taskfile.

Let's try it

Grab the project you are currently working on and let's try Taskfile in action

First we install it and create a simple Taskfile.yml

$ brew install go-task
$ vim Taskfile.yml # or task --init

As mentioned above it uses simple Yaml schema. Let's start simple by defining a few basic tasks. Here’s an example:

version: '3'

tasks:
  build:
    cmds:
      - go build ./...
  test:
    cmds:
      - go test ./...

Here we basically define build and test tasks similar to how you will do in a Makefile. Next you likely would need to define some dependencies, like so

version: '3'

tasks:
  build:
    desc: Build the project
    cmds:
      - go build ./...
  test:
    desc: Test the project
    deps: [build]
    cmds:
      - go test ./...

Note: dependencies run in parallel, so dependencies of a task should not depend on one another.

Let's run it! You can view all available tasks by executing task --list-all

$ task --list-all
task: Available tasks for this project:
* build: Build the project
* test: Test the project

$ task build
task: [build] go build ./...

$ task test
task: [build] go build ./...
task: [test] go test ./...

Your first taskfile is ready to go!

Let's extend it more

Now that we have hello world Taskfile, let's extend it further. In the example below we add:

• vars - which can be interpolated and reused across all the tasks

• env - which are env variables passed to all the tasks

• sources - when given, task will compare the checksum of the source files to determine if it's necessary to run the task (to avoid unnecessary build execution for example)

version: '3'

env:
  GOFLAGS: -mod=mod

vars:
  goTestArgs: -v

tasks:
  build:
    desc: Build the project
    cmds:
      - go build ./...
  test:
    desc: Test the project
    deps: [build]
    cmds:
      - go test {{.goTestArgs}} ./...

env and vars here are set on the global level, but you can also do the same on the task level for better flexibility.

What else can you do with Taskfiles:

• Including other Taskfiles (e.g. OS specific)

• Loop over variables/tasks and even use Go's template engine

• Do task cleanups using defer functionality

• Use a dry-run mode

and more!

Conclusion

Go ahead, give Taskfiles a try. It's powerful, flexible and can simplify your build and automation processes. Want to see how others in open source use it? Here is sourcegraph link to find good examples!

One of the things I love about engineering is the endless puzzles it throws our way. I can easily be carried away by a single unexpected performance degradation of a service, because it's so fun!

Recently we experienced one of these cases with ArgoCD, a continuous delivery tool for Kubernetes clusters.

Problem

As a part of the ArgoCD adoption we deployed it to the huge Kubernetes cluster, with ~5,000 physical nodes and hundreds of thousands of resources. Upon using it we noticed that ArgoCD status updates and reconciliation performance were noticeably slower than on smaller clusters.

It was visible on the metrics as well, internal workqueue was always high:

The reconciliation performance heatmap showed consistently high reconciliation times for all of our applications:

Upon closer look we figured out that the actual resource sync goes pretty fast, but the status updates in the UI and in the CRD was taking minutes to complete. Based on the code it's done by using so-called "status processors", like so

	for i := 0; i < statusProcessors; i++ {
		go wait.Until(func() {
			for ctrl.processAppRefreshQueueItem() {
			}
		}, time.Second, ctx.Done())
	}

This would be an important detail later.

Fix Attempt #1

As a first attempt we relied on community knowledge to configure ArgoCD for larger scale. Some great material you can find here and here. Things we tried:

• Bumped controller.status.processors and controller.operation.processors to recommended higher values

• Bumped controller/repo-server/server resource requests/limits to the point where it no longer was throttled or OOMed

• Bumped clientQPS and burstQPS

• Bumped repo-server/server replicas

We use one argocd instance per cluster, so we left sharding capabilities untouched, because it would have no impact on our setup. Unfortunately nothing from the list above helped and arguably made things slightly worse 🤔

Let's look closer

If you get stuck with anything performance related in golang, pprof is your best friend. We discussed it in details in our Debug Golang Memory Leaks newsletter issue some time ago - how to run it, how to read flamegraphs, available profiling data, etc. Check it out if you need a refresher.

ArgoCD has pprof endpoint enabled by default, which is convenient (but make sure to not expose it to untrusted networks, as your infosec team might knock on your door)

First I wanted to look at why it uses ~4 CPU cores and ~8 Gbs of memory to operate. Depending on how your argocd is setup, you should be able to do something like this to get access to the CPU profiling information.

$ kubectl port-forward pod/argocd-application-controller-0 8082  -n argocd
$ go tool pprof -http=localhost:8080 localhost:8082/debug/pprof/profile?seconds=5

It will open the web UI and by navigating to the flamegraphs I found this

This flamegraph basically shows that most of the CPU cycles spent in processAppRefreshQueueItem, which is exactly the functionality we see lagging. Looking further down the stack we can see that CPU is wasted on iterating through the hierarchy of resources that are located in ArgoCD cache implementation.

Fix Attempt #2

Looking at the code we can see that controller builds a cache of all the resources that are in the cluster and maintains the "internal view" of a cluster in order to be able to accurately compare the state of resources in the cluster with the desired state in Git. That comes with the caveat - even though you might have a handful of applications and couple hundreds resources in them, ArgoCD will cache all resources that exist in your cluster, even if it won't ever use them. To confirm this we can look into the "Resource Objects Count" in cache:

As you can see it has ~370k resources and ArgoCD spends a lot of time processing through these, so it can explain the slow processing, high CPU and high memory.

So what if we reduce the amount of resources ArgoCD keeps in cache by removing things we don't need? Then the status update performance and resource utilization should improve.

You can configure resource exclusion like so:

    resource.exclusions: |
      - apiGroups:
          - autoscaling.k8s.io
        kinds:
          - '*'
        clusters:
          - '*'
      - apiGroups:
          - discovery.k8s.io
        kinds:
          - EndpointSlice
        clusters:
          - '*'
...

After doing the above and excluding resources we don't care about, here are the results:

• The amount of resources in cache reduced 3x to ~100k

• Memory usage reduced 5x

• Avg CPU usage reduced by 2x

• Reconciliation speed improved, but only slightly, maybe by 10% 😢

We were back to square one

Let's dig more

Another useful feature pprof has is ability to take goroutine dumps. You can do so by querying the pprof endpoint like so:

# full goroutine dump
$ curl 'localhost:8082/debug/pprof/goroutine?debug=2'

# "grouped" view
$ curl 'localhost:8082/debug/pprof/goroutine?debug=2'

I was looking for processAppRefreshQueueItem to understand where it gets stuck.

The goroutine dump would look something like so for this call:

#	0x46dba4	sync.runtime_SemacquireRWMutexR+0x24									/usr/local/go/src/runtime/sema.go:82
#	0x1eec619	sync.(*RWMutex).RLock+0x99										/usr/local/go/src/sync/rwmutex.go:71
#	0x1eec5f4	github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).IterateHierarchy+0x74			/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/cache/cluster.go:972
#	0x2259aa4	github.com/argoproj/argo-cd/v2/controller/cache.(*liveStateCache).IterateHierarchy+0xc4			/go/src/github.com/argoproj/argo-cd/controller/cache/cache.go:604
#	0x22c43c2	github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).getResourceTree+0x782		/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:535
#	0x22c3675	github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).setAppManagedResources+0x95		/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:455
#	0x22d04b6	github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem+0x14b6	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1596
#	0x22c83c4	github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3+0x24			/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:819
#	0x12fda52	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x32						/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:157
#	0x12fd90e	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xae							/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:158
#	0x12fd7fe	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x7e							/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:135
#	0x12fd73d	k8s.io/apimachinery/pkg/util/wait.Until+0x1d								/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:92

As you can see this goroutine is stuck waiting on sync.(*RWMutex).RLock at the top of the stack.

By taking look at grouped view the problem I had 💡 moment: 49/50 status processors were waiting on the lock, so it was a lock contention problem:

• 24 goroutines blocked on github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).IterateHierarchy

• 14 goroutines blocked on github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).GetAPIResources

• 6 goroutines blocked on github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).GetManagedLiveObjs

• 5 goroutines blocked on github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).FindResources

• 1 active on github.com/argoproj/gitops-engine/pkg/cache.(*Resource).iterateChildren

Unfortunately in golang there is no easy way to understand who is holding the lock, but going through the code it's likely related to the watch, which updates the cache on k8s resources updates. In large clusters, where resource updates occur frequently, this lock contention could explain the performance issues.

Fix Attempt #3

We were running ArgoCD 2.5 and I found two issues related to locks (1 and 2). The first one was to likely improve the lock contention cause it removes unnecessary lock from GetAPIResources call, which we saw occupying significant amount of goroutines.

So we upgraded to ArgoCD 2.10 and confirmed that GetAPIResources no longer waits for a lock 🚀

The reconcile speed improved... somewhat 😔. The workqueue visibly reduced on the graphs during low pod churn periods, but still took minutes when a lot of pods being scheduled.

Taking a fresh goroutine dump:

• 36 goroutines blocked on github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).IterateHierarchy

• 10 goroutines blocked on github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).GetManagedLiveObjs

• 3 goroutines blocked on github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).FindResources

• 1 active on github.com/argoproj/gitops-engine/pkg/cache.(*Resource).iterateChildren

So the lock contention was just skewed towards IterateHierarchy call. Ugh!

Next Steps

So after trying all of the above solutions we are down to two options at this point, which were a last resort but will address the performance issues we were seeing

• Significantly bump reconciliation timeout from 45s to 5 minutes or more. This will reduce the workqueue size, since applications would be time queued less often

• Filter out resource updates not related to the "change in desired state of application". By default, an ArgoCD application is refreshed every time a resource that belongs to it changes. So if we filter out not-so-important, but frequently updated fields like /status we can also reduce the workqueue size and improve speed.

In conclusion, troubleshooting performance issues in complex systems can be challenging, but with the right tools and approach, it can also be a rewarding learning experience. Hope you enjoyed this deep dive! 🕵️

SRECon has gathered SRE and systems engineers in March this year, but if you missed it - talks were recently uploaded to youtube on Usenix channel.

Here is Codereliant's Top 5 recommendations of what to watch:

📖 Product Reliability for Google Maps
Micah Lerner and Joe Abrams

Learn how Google Maps, the app with over 1 billion monthly users, tackled subtle user experience issues that traditional server monitoring couldn't catch. Learn about their journey towards user-centric reliability and the lessons they learned along the way.

📖 Sharding: Growing Systems from Node-scale to Planet-scale
Adam Mckaig

This talk goes deep into workload sharding: when it makes sense, what problem it has, and how it can solve common scalability and reliability issues in your systems. You will learn fundamental sharding patterns like worker pools, read replicas, horizontal and vertical sharding, and multi-cell/multi-region architectures.

📖 Teaching SRE
Mikey Dickerson

This talk has an interesting insights into training a new wave of system engineers. The author designed and taught a class for computer science majors called "Managing Complex Systems". Students work on a project, progressing through a series of assignments that cover key SRE topics such as setting up infrastructure, monitoring, on-call rotations, capacity planning, incident response, etc.

📖 Deciphering Cache Inconsistencies in a Distributed Environment
Akashdeep Goel and Prudhviraj Karumanchi

Probably every SRE has their "caching" story. Engineers from Netflix share their experience debugging a critical issue with their distributed, replicated key-value store called EVCache, which threatened to delay a major global product launch. The incident walkthrough highlights the challenges and rewards of operating and troubleshooting large-scale distributed systems in a high-stakes production environment.

📖 Meeting the Challenge of Burnout
Christina Maslach

While this is not a technical talk, I think we don't talk about this topic enough. A pioneering researcher on job burnout, clarifies the concept of burnout, its causes, and potential solutions. She emphasizes that burnout is an occupational phenomenon resulting from chronic workplace stressors that have not been successfully managed, rather than an individual medical condition.

Hey there, reader! 👋

You know what they say, sharing is caring! If you found this newsletter helpful please take a moment to share it with your friends, colleagues and social media followers. Thanks for being a part of this community! 🚀

Subscribe now

Whether you're automating processes, managing server chores, or just experimenting; you often need a way to keep your scripts running after disconnect or indefinitely. I see people using mostly nohup & screen for this, but there are some other tools available as well.

Let's discuss the less well-known but convenient ways to run scripts or commands on a Linux machine forever—even after logging out—in this blog post.

But first, let's us do a refresher of known methods:

1️⃣ nohup

nohup while true; do dig codereliant.io NS; sleep 1; done &

This command uses nohup to ignore the HUP (hangup) signal, which is sent to the process when the user logs out. The output of the command will be redirected to a file named nohup.out by default, unless you specify otherwise.

The command while true; do dig codereliant.io NS; sleep 1; done is a loop that continuously runs the dig command to retrieve the NS records for the domain codereliant.io, with a 1-second delay between each query. And we will use this as an example throughout the post.

2️⃣ screen or tmux

Alternatively, using screen or tmux can be very effective. These tools allow you to start a session, run the command, and then detach from the session while leaving it running on the server.

1. Start a new screen or tmux session:

screen

2. Run your command

while true; do dig codereliant.io NS; sleep 1; done

3. Detach from the screen session by pressing Ctrl-A followed by D.

This way, the process will keep running in the background inside the screen session, even if you log out, and you can reattach to it later if needed.

For tmux, you use tmux to start a new session, and Ctrl-B followed by D to detach from the screen.

3️⃣ disown

Sometimes in a machine or mostly in a running docker container you might not have nohup, screen, or tmux installed, you can use another approach to keep your command running in the background even after you log out or detach from the container by using the shell's job control features along with redirection of outputs to avoid the terminal sending a HUP (hangup) signal.

The disown command in bash removes a shell job from the shell's job table, making it immune to the HUP signal when you log out. Here’s how you can use it:

1. Start the process in the background as usual:

while true; do dig codereliant.io NS; sleep 1; done &

2. Then you can use jobs -l to see the running jobs and their job numbers.

3. Use disown with the job number to remove the job from the shell’s job table:

disown %1

4. Replace %1 with the actual job number if it's different.

4️⃣ setsid

Another tool you might have available is setsid, which runs a program in a new session. This means the program will not be tied to the terminal session and will continue running after logout:

setsid sh -c 'while true; do dig codereliant.io NS; sleep 1; done' &

This command wraps your loop in a sh -c command string, ensuring it can handle the loop syntax correctly, and starts it in a new session using setsid.

Redirecting Outputs

In all these cases, it’s a good idea to redirect stdout and stderr to a file or /dev/null to prevent the process from trying to write to a non-existent terminal, which could cause it to stop:

while true; do dig codereliant.io NS; sleep 1; done >/dev/null 2>&1 &

Adding >/dev/null 2>&1 redirects both the standard output and the standard error to /dev/null, effectively discarding all output.

Using any of these methods, you should be able to run your command in the background effectively without having traditional utilities like nohup or screen.

If you liked this post and want to learn more about similar concepts, subscribe to our newsletter! We'll keep you updated with more tips, tricks, and in-depth guides to ensure you're always a step ahead in managing your technology.

Hey there, reader! 👋

You know what they say, sharing is caring! If you found this newsletter helpful please take a moment to share it with your friends, colleagues and social media followers. Thanks for being a part of this community! 🚀

Subscribe now

"Prodverbs" (or production proverbs) is a cool collection of sayings maintained by Google's SRE team, which all of us who write or maintain distributed systems should understand. Let's explore each one:

1️⃣ If two systems must agree for them to work, someday they will inevitably disagree

It captures an important challenge in distributed systems: the need for agreement among multiple components. In a distributed system, processes often need to reach consensus on a single value or state to maintain consistency and coordinate their actions. However, relying on perfect agreement is unrealistic in the face of network partitions, split-brain situations, hardware failures, etc. The prodverb serves as a reminder that building reliable distributed systems requires careful design and the use of proven consensus algorithms, such as Paxos and Raft, which ensure that all processes in the group will eventually converge on a single agreed-upon value, even in the presence of failures and unreliable networks.

2️⃣ Decrease variance, increase mean

You should be focusing on improving the typical user experience rather than over-optimizing for edge cases. Reducing variability leads to more predictable performance for majority of the users.

3️⃣ Backups are only as good as the last restore

Backups are a critical part of our reliability toolkit, but they're not something we can just set and forget. This saying emphasizes the importance of regularly testing our restore processes. Automating and exercising restores helps ensure that when we need our backups, they'll actually work.

4️⃣ If you have no SLOs, toil is your job

Without clear reliability targets, SRE work can become an endless cycle of reactive firefighting. Service Level Objectives align teams around key metrics and help prioritize work that meaningfully improves reliability. This prodverb reminds us that if we don't set our own agenda with SLOs, toil will rule the day.

5️⃣ Hope is not a strategy

This prodverb reminds us that wishful thinking is not enough to build reliable systems. We can't just hope that things will work out or that failures won't happen.

Some key aspects of having a solid strategy include:

• Designing for failure

• Capacity planning

• Security and access controls

• Incident response planning

• etc

This prodverb encapsulates mindset shift from passively wishing for the best to actively working to make the system as robust as possible.

6️⃣ Scale maintenance sublinearly with the growth of the service

What worked for 10 servers may not cut it for 10,000. Plan for the future, explore ways to reduce maintenance overhead - automate tasks, simplify architectures, and empower dev teams to share the load.

7️⃣ May all your incidents be novel

In a perfect world, we'd never have repeat incidents. While that may not be realistic, this saying suggests us to strive to learn from every incident. By performing blameless postmortems and addressing the core of the problem, we can work to ensure that the same type of incident won't happen again.

Hey there, reader! 👋

You know what they say, sharing is caring! If you found this newsletter helpful please take a moment to share it with your friends, colleagues and social media followers. Thanks for being a part of this community! 🚀

Subscribe now

💡 A Step-by-Step Tutorial on how to make an AI chatbot for your website

💡 We will learn and use:
• Python uv: blazing fast pip replacement
• Scrapy: Web scraper toolkit
• OpenAI: embeddings api & LLM
• Langchain: Semantic Chunking
• Qdrant: Vector DB to store embedding & query them

Did you see lately an increase in the number of websites that has an AI chatbot capability? That allows you to ask question about pricing, documentation, advanced questions, etc....

If you've noticed this trend, you're not alone. AI chatbots have become increasingly popular on websites across various industries, as they provide a convenient and efficient way for users to get the information they need quickly. These chatbots are powered by advanced technologies such as natural language processing (NLP), machine learning, and deep learning, which enable them to understand and respond to user queries in a human-like manner.

We will look into the process of creating your own AI chatbot for your website using Python and a combination of powerful tools and libraries. We'll cover web scraping using Scrapy to gather content from your site, creating embeddings from the scraped content using the OpenAI API, and utilizing Qdrant as a vector database to store and query the embeddings. Additionally, we'll explore how to use Langchain for semantic chunking and integrate the queried content with OpenAI's GPT-4 to provide accurate and context-aware responses to user queries.

Set Up & Env

We will make a twist and use uv as our blazing fast python installer, so go ahead and install if you don't have it:

# On macOS and Linux.
curl -LsSf https://astral.sh/uv/install.sh | sh

# On Windows.
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Then, we will create a new directory and install all the needed libraries:

mkdir chatbot && cd chatbot
uv venv
uv pip install scrapy
uv pip install beautifulsoup4
uv pip install openai
uv pip install qdrant-client
uv pip install langchain
uv pip install langchain_experimental
uv pip install langchain_openai
uv pip install gradio

Website Scrapping

Now let's create a new scrapy project for our website, in our use case, we will use yours truly codereliant.io:

scrapy startproject codereliant

In the file items.py modify the class CodereliantItem like below:

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class CodereliantItem(scrapy.Item):
    url = scrapy.Field()
    pass

Let's do a simple test to see if our scraping is working as intended:

cd codereliant
scrapy crawl codereliant -L ERROR -O output.json

From within the codereliant folder we run the scrapping command, which will result in a file output.json that contains all of our website urls:

less output.json
[
{"url": "https://www.codereliant.io"},
{"url": "https://www.codereliant.io/sre-interview-prep-plan-week-6/"},
{"url": "https://www.codereliant.io/free-goodies/"},
{"url": "https://www.codereliant.io/about/"},
{"url": "https://www.codereliant.io/the-most-tragic-bug/"},
{"url": "https://www.codereliant.io/pod-doctor/"},
{"url": "https://www.codereliant.io/from-reactive-to-proactive-transforming-software-maintenance-with-auto-remediation/"},
{"url": "https://www.codereliant.io/topics/"},
{"url": "https://www.codereliant.io/the-2038-problem/"},
{"url": "https://www.codereliant.io/14-years-of-go/"},
......

We now that we will need more than just the urls if we want to create the chatbot.

Data Extraction & Preprocessing

We already collected the url, now we need to other interesting data: page title, description, & page body.

Let's add these items to our CodereliantItem class in items.py:

class CodereliantItem(scrapy.Item):
    url = scrapy.Field()
    title = scrapy.Field()
    description = scrapy.Field()
    body = scrapy.Field()
    pass

Now our item class has these fields defined, lets' actually grab the data in parse_item method that is defined in CodereliantSpider within spiders/codereliant.py:

def parse_item(self, response):
    item = CodereliantItem()
    item['url'] = response.url
    item['title'] = response.css('title::text').get()
    item['description'] = response.css(
        'meta[name="description"]::attr(content)').get()
    item['body'] = response.css('body').get()
    return item

💡 We used css selector to collect the data needed. You can read more about Scrapy data selectors in here.

In order to either preprocess or post process our data, we need to write pipelines. the first pipeline we will write is the pipeline that will convert our page body content from html to just plain text.

First let's add a data field to CodereliantItem, which will hold the plain body:

class CodereliantItem(scrapy.Item):
    url = scrapy.Field()
    title = scrapy.Field()
    description = scrapy.Field()
    body = scrapy.Field()
    plain_body = scrapy.Field() # add this line
    pass

All of our pipelines are located in pipelines.py. we will write our first simple BodyTxtConverterPipeline:

class BodyTxtConverterPipeline:
    # use beatifulsoup to convert html to text
    def process_item(self, item, spider):
        item['plain_body'] = BeautifulSoup(item['body'], 'html.parser').get_text()
        return item

Now that we wrote our pipeline we need to enable it in settings.py, we need to uncomment the variable ITEM_PIPELINES and make it match:

ITEM_PIPELINES = {
   "codereliant.pipelines.BodyTxtConverterPipeline": 300,
}

To test that everything is working, we just need to try the scrapy command like above:

scrapy crawl codereliant -L ERROR -O output.json

by looking inside the file output.json we can see that we have successfully collected the data we are interested in + the plain_body.

Semantic Chunking

Now the more interesting part, which is now that we have collected all the data we need, what's next?

💡 For this part, you will need an OpenAI API account with sufficient credits.

Let's add a field in CodereliantItem to hold the chunks:

class CodereliantItem(scrapy.Item):
    chunks = scrapy.Field() # add this line
    ......

Our pipeline will leverage the abstractions of langchain, and it will also live in pipelines.py:

class TextSplitterPipeline:
    def open_spider(self, spider):
        self.text_splitter = SemanticChunker(
            OpenAIEmbeddings(), breakpoint_threshold_type="percentile"
        )

    def process_item(self, item, spider):
        docs = self.text_splitter.create_documents([item['plain_body']])
        item['chunks'] = []
        for doc in docs:
            text = doc.page_content.replace("\n", " ")
            item['chunks'].append(text)
        return item

Let's add this tosettings.py like the previous pipeline:

ITEM_PIPELINES = {
   "codereliant.pipelines.BodyTxtConverterPipeline": 300,
   "codereliant.pipelines.TextSplitterPipeline": 400,
}

The number here signifies the order, we run pipelines from smallest to largest.

VectorDB Storage

Now we need create embeddings from these chunks and store them in our database. Let's start a standalone docker container of qdrant in a new terminal:

docker run -p 6333:6333 -p 6334:6334 \
    -v $(pwd)/qdrant_storage:/qdrant/storage:z \
    qdrant/qdrant

Our pipeline this time is a bit involved, but we will explain the code below:

class VecorDBPipeline:
    def open_spider(self, spider):
        # constants
        self.COLLECTION_NAME = "codereliant"
        self.EMBEDDING_MODEL = "text-embedding-3-small"
        self.EMBEDDING_MODEL_VECOTR_SIZE = 1536 # hardcoded, but can be obtained from the model

        # clients
        self.openai_client = OpenAI()
        self.client = QdrantClient(url="http://localhost:6333")

        # create collection in qdrant
        self.client.recreate_collection(
            collection_name=self.COLLECTION_NAME,
            vectors_config=models.VectorParams(
                size=self.EMBEDDING_MODEL_VECOTR_SIZE,  
                distance=models.Distance.COSINE,
            ),
        )

    def process_item(self, item, spider):
        unique_ids = [str(uuid.uuid4()) for _ in item['chunks']]
        result = self.openai_client.embeddings.create(input=item["chunks"], model=self.EMBEDDING_MODEL)

        self.client.upload_points(
            collection_name=self.COLLECTION_NAME,
            points=[
                models.PointStruct(
                    id=unique_id,
                    vector=data.embedding,
                    payload={
                        "url": item['url'],
                        "title": item['title'],
                        "description": item['description'],
                        "chunk": chunk,
                    }
                )
                for unique_id, chunk, data in zip(unique_ids, item['chunks'], result.data)
            ],
        )
        return item

In summary, the open_spider method is responsible for initializing the necessary clients and creating the collection in the Qdrant database, while the process_item method handles the creation of embeddings for each chunk using the OpenAI API and uploads the embeddings and associated metadata to the Qdrant database.

and here is the explanation in great details:

1. open_spider(self, spider) method:

   • This method is called when the spider is opened and is used for initialization purposes.

   • It sets up some constants:

     • COLLECTION_NAME: The name of the collection in the Qdrant database.

     • EMBEDDING_MODEL: The name of the OpenAI embedding model to be used.

     • EMBEDDING_MODEL_VECOTR_SIZE: The vector size of the embedding model (hardcoded as 1536 in this case).

   • It initializes two clients:

     • openai_client: An instance of the OpenAI client for interacting with the OpenAI API.

     • client: An instance of the QdrantClient for connecting to the Qdrant database.

   • It recreates a collection in the Qdrant database using the recreate_collection method of the QdrantClient. The collection is created with the specified COLLECTION_NAME and a VectorParams configuration that includes the vector size (EMBEDDING_MODEL_VECOTR_SIZE) and the distance metric (models.Distance.COSINE).

2. process_item(self, item, spider) method:

   • This method is called for each item yielded by the spider.

   • It generates unique IDs for each chunk in the item['chunks'] list using uuid.uuid4().

   • It creates embeddings for each chunk using the OpenAI API's embeddings.create method, specifying the chunks and the EMBEDDING_MODEL.

   • It uploads the embeddings and associated metadata to the Qdrant database using the upload_points method of the QdrantClient.

     • For each chunk, it creates a PointStruct object containing:

       • id: The unique ID generated for the chunk.

       • vector: The embedding vector obtained from the OpenAI API.

       • payload: A dictionary containing metadata associated with the chunk, such as the URL, title, description, and the chunk itself.

Finally, we need to add the pipeline VecorDBPipeline to settings.py:

ITEM_PIPELINES = {
   "codereliant.pipelines.BodyTxtConverterPipeline": 300,
   "codereliant.pipelines.TextSplitterPipeline": 400,
   "codereliant.pipelines.VecorDBPipeline": 500,
}

Now before we test it we need to set our OpenAI token

export OPENAI_API_KEY='sk-...........'
scrapy crawl codereliant -L ERROR -O output.json

We can verify by looking at our Qdrant dashboard UI which is located at: http://localhost:6333/dashboard .

From there we can see the info of the collection and some useful stats:

Chatbot

Finally we got to the actual chatbot. We will create a separate python file outside the codereliant and within out chatbot directory:

cd ../
touch chat.py
uv pip install gradio

we just created a python file that will contain our chat interface and install gradio ui toolkit.

Below is the code for chat.py:

import json
import gradio as gr
from openai import OpenAI
from qdrant_client import QdrantClient

# constants
COLLECTION_NAME = "codereliant"
EMBEDDING_MODEL = "text-embedding-3-small"

# clients
openai_client = OpenAI()
client = QdrantClient(url="http://localhost:6333")

def search_similar_text(text):
    hits = client.search(
        collection_name=COLLECTION_NAME,
        query_vector=openai_client.embeddings.create(input=[text],model=EMBEDDING_MODEL,)
        .data[0]
        .embedding,
        limit=3,
    )
    return hits

def generate_response(query, history):
    hits = search_similar_text(query)
    # create the context based on the hits
    context = ""
    for hit in hits:
        context += json.dumps(hit.payload) + "\n"
    prompt = generate_prompt(context, query)
    completion = openai_client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[
            {"role": "system", "content": "You are codereliant chatbot, skilled in answering questions about codereliant.io."},
            {"role": "user", "content": prompt}
        ]
    )
    
    return completion.choices[0].message.content

def generate_prompt(context, query):
    prompt = f"""
Context:
{context}

Query: {query}

Based on the provided context, answer the query. If the context does not contain enough information to answer the query, respond with "I'm sorry, but the provided context does not have sufficient information to answer your query. Also, answer like a customer service representative would."

Answer:
"""
    return prompt

demo = gr.ChatInterface(fn=generate_response, examples=[ "what is this website about?","what can I learn from codereliant?", "Do you have any SRE interview preparation posts?"], title="CodeReliant Bot", multimodal=False)
demo.launch()

Let's go through each function and briefly explain what they do:

1. search_similar_text(text):

   • This function takes a text parameter and searches for similar texts in the Qdrant database.

   • It creates an embedding of the input text using the specified EMBEDDING_MODEL and the OpenAI API.

   • It then searches the Qdrant collection (COLLECTION_NAME) for the top 3 (limit=3) most similar vectors to the input text's embedding.

   • The function returns the search results (hits) containing the similar texts and their associated metadata.

2. generate_response(query, history):

   • This function generates a response to a given query based on the search results from the Qdrant database.

   • It calls the search_similar_text(query) function to retrieve the most similar texts to the query.

   • It then constructs a context string by concatenating the payload (metadata) of each hit retrieved from the database.

   • It generates a prompt using the generate_prompt(context, query) function, which combines the context and the query.

   • Finally, it uses the OpenAI API to create a chat completion using the gpt-4-turbo model, passing the generated prompt as the user's message.

   • The function returns the generated response obtained from the chat completion.

3. generate_prompt(context, query):

   • This function generates a prompt by combining the context and the query.

   • The prompt includes the context, the query, and instructions on how to answer the query based on the provided context.

   • If the context does not contain enough information to answer the query, it instructs the model to respond with a specific message indicating the lack of information and to answer like a customer service representative.

   • The function returns the generated prompt as a string.

4. The last part of the code sets up a Gradio ChatInterface for interacting with the chatbot:

   • It creates an instance of gr.ChatInterface called demo.

   • The fn parameter is set to the generate_response function, which will be called whenever a user sends a message.

   • The examples parameter is set to a list of example queries to showcase the chatbot's capabilities.

   • The title parameter sets the title of the chat interface to "CodeReliant Bot".

   • The multimodal parameter is set to False, indicating that the chat interface will only handle text input and output.

   • Finally, the launch() method is called on the demo instance to start the chat interface.

These functions work together to create a chatbot that can search for similar texts in our Qdrant database based on user queries, generate prompts combining the retrieved context and the query, and provide responses using the OpenAI API's chat completion feature. The chatbot is then made interactive using Gradio's ChatInterface.

To run it, make sure the OpenAI token is set, and just run:

python chat.py

#Running on local URL:  http://127.0.0.1:7860

#To create a public link, set `share=True` in `launch()`.

Below is a screenshot of trying the examples with our ChatBot and getting responses like we do from other sites:

The Github link contains the full code of this tutorial!

Hey there, reader! 👋

You know what they say, sharing is caring! If you found this newsletter helpful please take a moment to share it with your friends, colleagues and social media followers. Thanks for being a part of this community! 🚀

Subscribe now

VSCode has become the default code editor for many developers. And as many developers, we are always on the lookout for tools that can boost our productivity and workflows.

Let's explore 5 must-have VSCode extensions that can take your coding productivity to the next level. I'll skip the Copilot, as I hope you already use it 😄

1️⃣ GitLens

GitLens is a powerful extension that improves the built-in Git capabilities of VSCode. It provides a set of features for visualizing and exploring Git repositories. With GitLens, you can easily view the history of a file, see who made changes and when, and even compare different versions of the code side by side. It also offers advanced blame annotations, allowing you to quickly identify the author of each line of code.

2️⃣ Find It Faster

FindItFaster is an extension that enhances your search capabilities within VSCode. It provides a more intuitive and efficient way to search for files, symbols, and text across your project. With this extension, you can use fuzzy search to quickly locate files based on partial matches, search for symbols within files, and even perform global searches across multiple directories. This extension can save you valuable time when navigating through large codebases. Under the hood it uses a combination of fzf, rg and bat, tools we discussed in one the previous articles here.

3️⃣ CodeSnap / Live Share

CodeSnap & Live Share is a combination of two powerful extensions that enhance collaboration and code sharing capabilities in VSCode.

CodeSnap allows you to easily create beautiful code snippets and share them with others. It captures your code as an image, complete with syntax highlighting and customizable themes. You can add custom backgrounds and annotations to make your code snippets more visually appealing. It's perfect for sharing code snippets on social media, in documentation, or in presentations.

Live Share enables real-time collaboration within VSCode. It allows you to share your code with others and work together in real-time. You can invite teammates to join your coding session, share your editor, terminal, and even debugging sessions. Live Share makes paired programming seamless and efficient.

4️⃣ Remote - SSH

Remote - SSH is an extension that allows you to connect to remote machines and work with files and folders as if they were local. You can develop on remote servers, virtual machines, or even containers seamlessly within VSCode.

This extension enables you to open any folder on a remote machine and take advantage of all the features and extensions of VSCode, including IntelliSense, debugging, and version control. It provides a secure and efficient way to work on remote projects without the need for local setup or file synchronization.

5️⃣ Project Manager

With Project Manager, you can define a list of projects, each with its own set of settings, such as the project's root directory, mark favorites, autodetect projects from Git repositories and more. Once your projects are set up, switching between them becomes very easy. You can use the keyboard shortcuts to open a project, and Project Manager will handle the rest, instantly loading the project's files and applying the associated settings.

One of the standout features of Project Manager is its ability to remember the state of your workspace for each project. It preserves the opened files, layout, and even the active file and line position, ensuring that you can quickly resume work exactly where you left off.

💡 Bonus

Nothing improves the productivity more than a good IDE theme 💅 😀

I've been mostly using One Dark Pro and Night Owl with some customizations.

Happy Coding!

Have you ever been on call and you get woken up by the same alert, an alert that you can't do anything about it, and the only way to make it go away is to restart the system responsible.

Auto remediation is an approach that SREs apply for these types of alerts and many other different ones. it makes our life easier, and it helps us focus on more critical tasks. Auto remediation is the process of automatically identifying and resolving issues in software systems without human intervention. It enables our systems to self-heal and recover from failures, minimizing downtime and ensuring continuous operation.

In this blog post, we'll explore what auto remediation is, its benefits, and how to implement it effectively.

What is Auto Remediation?

Auto remediation refers to the automated process of identifying and resolving issues in software systems without human intervention. It ensures that systems stays up and have a smooth continuous operation through auto recovering from failures.

Common Scenarios for Auto Remediation

Auto remediation can be applied to various scenarios, including:

• Infrastructure failures: Automatically provisioning backup resources or restarting failed components.

• Application errors and exceptions: Identifying and resolving common application issues.

• Security vulnerabilities: Automatically patching known vulnerabilities.

• Performance issues: Dynamically scaling resources based on workload.

Benefits of Auto Remediation

Implementing auto remediation offers several benefits:

• Faster recovery times: Issues are resolved quickly, minimizing downtime.

• Reduced manual intervention: Automated remediation reduces the need for manual troubleshooting.

• Improved system reliability and availability: Systems become more resilient and self-healing.

• Cost savings: Automated remediation reduces the cost associated with manual intervention and downtime.

Key Components of an Auto Remediation System

An effective auto remediation system consists of the following components:

1. Monitoring and Alerting: Continuously monitor system metrics and logs to detect anomalies and trigger alerts.

   1. For example tools like Prometheus and AlertManger.

2. Incident Detection and Analysis: Analyze alerts and correlate data to identify the root cause of incidents.

3. Remediation Policies and Rules: Define a set of predefined policies and rules to determine the appropriate remediation actions.

4. Automation and Orchestration Tools: Utilize automation tools to execute remediation actions and orchestrate the overall process.

Here's a simple example of a remediation webhook for AlertManager using Golang:

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "time"
)

type AlertMessage struct {
    Alerts []Alert `json:"alerts"`
}

type Alert struct {
    Status      string            `json:"status"`
    Labels      map[string]string `json:"labels"`
    Annotations map[string]string `json:"annotations"`
    StartsAt    time.Time         `json:"startsAt"`
    EndsAt      time.Time         `json:"endsAt"`
}

func handleWebhook(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
        return
    }

    var alertMessage AlertMessage
    err := json.NewDecoder(r.Body).Decode(&alertMessage)
    if err != nil {
        http.Error(w, "Invalid request payload", http.StatusBadRequest)
        return
    }

    for _, alert := range alertMessage.Alerts {
        if alert.Status == "firing" {
            if alert.Labels["alertname"] == "HighCPUUsage" || alert.Labels["alertname"] == "HighMemoryUsage" {
                duration := time.Since(alert.StartsAt)
                if duration >= 30*time.Minute {
                    nodeName := alert.Labels["instance"]
                    restartNode(nodeName)
                }
            }
        }
    }

    w.WriteHeader(http.StatusOK)
}

func restartNode(nodeName string) {
    // Implement the logic to restart the node based on the nodeName
    fmt.Printf("Restarting node: %s\n", nodeName)
    // ...
}

func main() {
    http.HandleFunc("/webhook", handleWebhook)
    fmt.Println("Webhook server is running on :8080")
    http.ListenAndServe(":8080", nil)
}

In this code:

1. We define the necessary structs (AlertMessage and Alert) to represent the JSON payload received from Alertmanager.

2. The handleWebhook function is the HTTP handler for the webhook endpoint. It expects a POST request with the alert payload.

3. We decode the JSON payload into the AlertMessage struct.

4. We iterate over each alert in the Alerts array.

5. For each alert, we check if the alert status is "firing" and if the alert name is either HighCPUUsage or HighMemoryUsage.

6. If the alert has been firing for at least 30 minutes (we can adjust the duration as needed), we extract the instance label, which represents the node name.

7. We call the restartNode function, passing the node name as an argument. This function should contain the logic to restart the node (e.g., making an API call to a cloud provider or executing a system command).

8. Finally, we set up the webhook server to listen on port 8080 and handle incoming requests.

To use this webhook, we need to configure Alertmanager to send alerts to the webhook URL (e.g., http://your-server:8080/webhook) when the specified conditions are met.

Implementing Auto Remediation

To implement auto remediation effectively:

1. Identify critical systems and failure scenarios.

2. Define remediation workflows and playbooks.

3. Select appropriate automation tools and technologies.

4. Test and validate auto remediation processes.

By following these guidelines and leveraging the power of automation, we can build resilient and self-healing software systems that minimize downtime and ensure optimal performance.

Conclusion

Auto remediation is crucial for every engineer that run and operate systems. It empowers SREs to focus on more strategic tasks while ensuring the reliability and availability of software systems. Keep in mind that you should continuously refine and optimize your processes to build truly resilient and self-healing systems.

Hey devs,
As many of you, I’ve been exploring how AI can help me be more productive with my day to day tasks. So “Pod Doctor” was born. This is a little tool (just 180 lines!) which helps me with Kubernetes Pod troubleshooting faster than if I were to do it myself. You can find the full code on github and in this beginner tutorial we’ll go through the steps required to be able build similar tools for yourself.

We’ll learn how to create Gradio's chatbot UI, integrate the Kubernetes Python client, and leverage GPT-4's language for understanding and reasoning.

The user interface of the Pod Doctor application is built using Gradio, which is a Python library for creating customizable UI components and deploying machine learning models. The interface consists of the following elements:

1. Title: A label displaying the name of the application, "Pod Doctor."

2. Chatbot: A chatbot-style interface for displaying the conversation between the user and the language model.

3. Namespace Dropdown: A dropdown menu that allows users to select the Kubernetes namespace they want to interact with.

4. Pod Dropdown: A dropdown menu that populates with the list of pods available in the selected namespace.

5. Include Events Checkbox: A checkbox that allows users to include pod events in the information provided to the language model.

6. Include Logs Checkbox: A checkbox that allows users to include pod logs in the information provided to the language model.

7. Message Input: A text input field where users can type their messages or queries.

with gr.Blocks(theme=gr.themes.Soft()) as demo:
    with gr.Row():
        title = gr.Label("Pod Doctor")
    with gr.Row():
        with gr.Column(scale=4):
            chatbot = gr.Chatbot(height="600px")
        with gr.Column(scale=1):
            namespace = gr.Dropdown(
                list_namespaces(),
                label="Namespace",
                info="Select the namespace you want to check the pod in.",
                interactive=True,
            )
            pod = gr.Dropdown(
                [],
                label="Pod",
                info="Select the pod you want to interact with.",
                interactive=True,
            )
            include_events = gr.Checkbox(
                label="Include Events",
                info="Include the pod events.",
                interactive=True,
            )
            include_logs = gr.Checkbox(
                label="Include Logs",
                info="Include the pod logs.",
                interactive=True,
            )
            namespace.change(namespace_change, inputs=namespace, outputs=pod)
    with gr.Row():
        msg = gr.Textbox()

    def respond(message, chat_history, namespace, pod, include_events, include_logs):
        if not namespace or not pod or not message:
            chat_history.append((message or "Error", "Please select a namespace and pod."))
            return "", chat_history
        bot_message = call_llm(message, namespace, pod, include_events, include_logs)
        new_message = f"Namespace {namespace}, Pod: {pod}, Include Events: {include_events}, Include Logs: {include_logs} \n {message}"
        chat_history.append((new_message, bot_message))
        return "", chat_history
    msg.submit(respond, [msg, chatbot, namespace, pod, include_events, include_logs], [msg, chatbot])

As we progress through the tutorial, you may encounter function calls that are not yet defined. However, these functions will be explained and implemented as we move forward.

Let's define a function that is responsible for updating the dropdown of pods once a namespace is selected:

def namespace_change(value):
    """
    Returns a Dropdown widget populated with a list of pods based on the given value.

    Args:
        value (str): The namespace to filter the pods.

    Returns:
        gr.Dropdown: A Dropdown widget with choices populated from the list of pods.
    """
    pods = list_pods(value)
    return gr.Dropdown(choices=pods)

Kubernetes Integration

The Pod Doctor application integrates with a Kubernetes cluster using the kubernetes Python library. The following functions handle the interaction with the Kubernetes API:

first we need to import the k8s library and load the config :

from kubernetes import client, config

config.load_kube_config()

list_namespaces(): Retrieves a list of namespaces from the Kubernetes cluster.

def list_namespaces():
    """
    Retrieves a list of namespaces from the Kubernetes cluster.

    Returns:
        A list of namespace names.
    """
    v1 = client.CoreV1Api()
    namespaces = v1.list_namespace()
    return [namespace.metadata.name for namespace in namespaces.items]

list_pods(namespace): Lists all pods in the specified namespace.

def list_pods(namespace):
    """
    List all pods in the specified namespace.

    Args:
        namespace (str): The namespace to list pods from.

    Returns:
        list: A list of pod names.
    """
    v1 = client.CoreV1Api()
    pods = v1.list_namespaced_pod(namespace)
    return [pod.metadata.name for pod in pods.items]

get_pod_info(namespace, pod, include_events, include_logs): Retrieves detailed information about a specific pod, including events and logs if requested.

def get_pod_info(namespace, pod, include_events, include_logs):
    """
    Retrieves information about a specific pod in a given namespace.

    Args:
        namespace (str): The namespace of the pod.
        pod (str): The name of the pod.
        include_events (bool): Flag indicating whether to include events associated with the pod.
        include_logs (bool): Flag indicating whether to include logs of the pod.

    Returns:
        dict: A dictionary containing the pod information, events (if include_events is True), and logs (if include_logs is True).
    """
    v1 = client.CoreV1Api()
    pod_info = v1.read_namespaced_pod(pod, namespace)
    pod_info_map = pod_info.to_dict()
    pod_info_map["metadata"]["managed_fields"] = None
    
    info = {
        "PodInfo": pod_info_map,
    }
    if include_events:
        events = v1.list_namespaced_event(namespace)
        info["Events"] = [
            {
                "Name": event.metadata.name,
                "Message": event.message,
                "Reason": event.reason,
            }
            for event in events.items
            if event.involved_object.name == pod
        ]
    if include_logs:
        logs = v1.read_namespaced_pod_log(pod, namespace)
        info["Logs"] = logs
    
    return info

LLM & GPT-4

The heart of the Pod Doctor application is the integration with the GPT-4 language model from OpenAI. The call_llm function is responsible for generating a response based on the user's message, the selected namespace, pod, and the options for including events and logs.

Here's how the call_llm function works:

1. The get_pod_info function is called to retrieve the pod information, including events and logs if requested.

2. The create_prompt function generates a prompt message by combining the user's message with the pod information.

3. The OpenAI API is called using the openAiClient.chat.completions.create method, passing the prompt message and the GPT-4 model.

4. The response from the language model is returned and displayed in the chatbot interface.

from openai import OpenAI
import yaml

openAiClient = OpenAI()

def create_prompt(msg, info):
    """
    Creates a prompt message with the given message and pod information.

    Args:
        msg (str): The main message for the prompt.
        info (dict): A dictionary containing pod information.

    Returns:
        str: The generated prompt message.
    """
    prompt = f"{msg}\n"
    prompt += f"Pod Info: \n {yaml.dump(info['PodInfo'])} \n"
    if info.get("Events"):
        prompt += f"Here are the last few events for the pod: \n"
        for event in info["Events"]:
            prompt += f"Event: {event['Name']}, Reason: {event['Reason']}, Message: {event['Message']}\n"
    if info.get("Logs"):
        prompt += f"Here are the last few logs for the pod: \n"
        prompt += info["Logs"]
    return prompt


def call_llm(msg, namespace, pod, include_events, include_logs):
    """
    Calls the Language Model to generate a response based on the given message, namespace, pod, and options.

    Args:
        msg (str): The user's message.
        namespace (str): The namespace of the pod.
        pod (str): The name of the pod.
        include_events (bool): Whether to include events in the pod information.
        include_logs (bool): Whether to include logs in the pod information.

    Returns:
        str: The generated response from the Language Model.
    """
    info = get_pod_info(namespace, pod, include_events, include_logs)
    prompt = create_prompt(msg, info)
    
    completion = openAiClient.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a kubernetes expert, and will help the user with request below based on the context and info provided"},
            {"role": "user", "content": prompt}
        ]
    )
    return completion.choices[0].message.content

Usage

We need one piece of code to wrap everything together:

if __name__ == "__main__":
    demo.launch()

export OPENAI_API_KEY='sk-xxxxxxxxxx'
python app.py

To use the Pod Doctor application, follow these steps:

1. Make sure your local kubernetes config is setup try kubectl cluster-info

2. Select the desired Kubernetes namespace from the "Namespace" dropdown.

3. Select the pod you want to interact with from the "Pod" dropdown.

4. Optional: Check the "Include Events" and "Include Logs" checkboxes if you want to include pod events and logs in the information provided to the language model.

5. Type your message or query in the text input field and press Enter.

6. The application will display the response from the GPT-4 language model in the chatbot interface.

Let's try a deploying a pod with the wrong port in the probes configuration and ask Pod Doctor to troubleshoot it for us.

Below is the bad nginx pod we will apply:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
      livenessProbe:
        httpGet:
          path: /
          port: 81
      readinessProbe:
        httpGet:
          path: /
          port: 81

After we apply this yaml we go to our app, select the namespace where we aplied it, select include events and logs, and type `troubleshoot` in the textbox, hit enter and watch the magic:

Conclusion

By combining the power of Gradio for building user interfaces, the Kubernetes Python library for interacting with Kubernetes pods, and the advanced language understanding capabilities of GPT-4, the Pod Doctor application provides a powerful and intuitive way to interact with and troubleshoot your Kubernetes pods.

You can find the full code in github.

In the early 1980s, a groundbreaking piece of medical technology emerged, promising to revolutionize cancer treatment.

The Therac-25, a computer-controlled radiation therapy machine, was a leap forward, promising to deliver precise radiation doses to patients battling cancer. It was one of the first radiation therapy machines controlled primarily by software rather than hardwired logic.

Hospitals eagerly adopted this cutting-edge device, trusting its software to accurately calibrate and deliver calculated dosages of high-energy radiation to destroy cancerous cells without harming surrounding healthy tissue. Little did they know, the Therac-25's software carried lethal bugs that would turn its precision medicine promise into a real-life nightmare.

Fatal Flaws

Between 1985-1987, a series of horrifying incidents occurred where the Therac-25 massively overdosed patients with radiation levels hundreds of times higher than safe limits. The impacts were catastrophic - causing serious radiation injuries, permanent disabilities, and multiple deaths.

Malfunction 54

It started at a cancer center in East Texas in 1986. An experienced technician was setting up a radiation treatment, which some training and use of a machine would only take seconds to breeze through. After initially entering the wrong radiation mode code, she quickly course-corrected using the cursor keys as was common practice. The machine then showed an error message "Malfunction 54". The technician had seen these unclear error messages many times before during normal use. She didn't think it was a serious issue, so she went ahead with the treatment anyway.

But this bypassed an important safety step. It meant the machine's powerful scanning magnets were out of sync with the new radiation settings the technician had entered. As a result, the patient ended up getting blasted with a massive radiation dose - over 16,000 times more than he was supposed to receive. It was radiation exposure on the same level as the Chernobyl nuclear disaster. The patient said it felt like a "burning, shocking" sensation, as if hot coffee had been poured on him. After a few days the patient unfortunately died from the complications of this exposure.

Root Cause Analysis

The fatally flawed software behavior was eventually recreated during another patient treatment by the same technician who had dealt with the initial "Malfunction 54" error. Tragically, this second patient also died from a massive radiation overdose. The hospital physicist became convinced there was an underlying issue with how the machine's powerful magnets were controlling and shaping the radiation beam.

After trying different scenarios, the physicist finally managed to trigger the glitch by rapidly entering treatment settings in a very specific sequence at maximum speed. To his alarm, the amount of radiation being delivered was so high that it went beyond the maximum his testing equipment could measure. Recalibrating the detectors revealed a dose in the range of 10,000 to 20,000 rads - over 100 times the intended therapeutic level and lethally excessive.

Here's what was happening: multiple routines were running concurrently in the software, including ones for data entry and keyboard input handling. These routines shared a single variable that recorded when the technician finished entering commands.

Once data entry completed, the beam calibration and magnet setting phase would begin. However, if the technician made a specific sequence of rapid edits during that 8-second magnet setting window, the new settings wouldn't actually apply to the hardware due to the shared program variable.

The user interface would then display the wrong treatment mode to the technician, who would proceed to confirm and start the potentially lethal treatment, unaware of the inconsistent machine state. This race condition defect was present in the previous Therac-20 model as well. But hardware interlocks on that older system prevented the flaw from actually causing radiation overdoses.

The Aftermath

It took over 2 years and multiple serious overdoses before the connection was made to the race condition causing these catastrophic failures in the Therac-25's operation. This resulted in at least 6 incidents, where patients were given massive overdoses of radiation altering their life forever.

The aftermath investigations found several major contributing causes beyond just the specific race condition software bug. These included:

• Poor software development practices at the manufacturer AECL, with lack of independent code review, testing, and safety analysis. The software was developed by a single unidentified programmer over the course of several years

• Overconfidence and assurances given to operators that overdoses were impossible, causing them to discount safety warnings and hardware safeties.

• Engineering issues like cryptic error messages, lack of hardware interlocks, reusing software from older models without verifying safety.

In addition to Malfunction 54 there was a number of other bugs potentially costing lives. The tragedy exposed widespread lapses in regulatory oversight and quality control processes for medical device software at the time. It prompted an overhaul of FDA regulations and safety standards, including:

• Mandating rigorous hazard analysis, risk assessment, and testing protocols for safety-critical code.

• Stricter requirements around software design, development lifecycles, and quality assurance.

• Enabling better traceability through design history file documentation.

Not many of us work on software projects like this, but the Therac-25 incidents serves as a wake-up call that software bugs could have catastrophic real-world consequences when proper development rigor and safety processes were not followed.

Links to learn more:

• http://sunnyday.mit.edu/papers/therac.pdf

• https://www.youtube.com/watch?v=Ap0orGCiou8

• https://www.bugsnag.com/blog/bug-day-race-condition-therac-25/

Series Overview:

• Week 1: Fundamentals of SRE

• Week 2: Automation & Scripting

• Week 3: Monitoring, Logging, and Alerting

• Week 4: Incident Management Lifecycle

• Week 5: Scalability, Performance, & System Design

• Week 6: Mock Interviews and Revision (This post)

As we step into the final week of our SRE interview preparation plan, it's time to focus and polish our skills.

Week 6 is designed to simulate the real interview experience, combining technical and behavioral mock interviews with a focused revision period. Through technical mock interviews on Days 1-3, our focus would be to sharpen our problem-solving and communication skills.

Days 6-7 offers a critical opportunity for revision and personalized feedback, allowing you to refine your approach and improve your understanding based on the insights gained. This final push is about more than just acing your interviews; it's about laying a solid foundation for a thriving career in Site Reliability Engineering.

🚀 For the mocking interviews you can use ChatGPT or any other suitable AI chat bot. If you want to go through a more authentic experience with a real person you can look for services that provide it as a paid service, e.g. https://interviewing.io/ (not sponsored)

Days 1-3: Technical Mock Interviews

Day 1-2: LeetCode Coding Challenges

The first day kicks off with a deep dive into LeetCode problems, focusing primarily on data structures and algorithms. Your tasks for the day will include a balanced mix of easy to medium difficulty questions, covering arrays, linked lists, and binary trees. If you know you will be interviewing for a more dev focused roles throw in some hard challenges as well. You should approach problems methodically, emphasizing not just the correctness of your solutions but also their efficiency and scalability. Talk the solutions out loud to illustrate your train of thought, because that's what interviewers would expect from you. After completing the challenges, spend some time reviewing your solutions, identifying areas for optimization and alternative approaches.

Day 3: System Design Fundamentals

Transitioning from coding challenges, spend Day 3 on System Design problems, a critical component of the SRE interview process. You should already be confident with it, but find some scenarios that require you to design scalable, reliable, and efficient systems, reflecting real-world problems SREs often face. Start with fundamental concepts such as load balancing, caching, and database selection, moving towards designing a complete system by the end of the day. Ensure you consider scalability, resilience, and maintenance, and not just the immediate requirements. Engage in mock interviews where you'll need to explain your design choices, trade-offs, and how you'd monitor and troubleshoot the systems you design.

Resources:

• Leetcode

• AlgoExpert

• HackerRank

• Grokking Modern System Design Interview

• Interviewing.io

These three days of intensive technical mock interviews are structured to cover a broad spectrum of topics and question types you're likely to encounter.
Remember, the goal is not just to find the right answers but to demonstrate clear, logical thinking, and effective communication of complex ideas.

Days 4-5: Behavioral Mock Interviews Focusing on SRE Scenarios

Day 4: Behavioral Interview Fundamentals

The fourth day of the week shifts the focus towards behavioral aspects, crucial for any SRE role. This day explore your past experiences, focusing on scenarios that highlight your problem-solving skills, teamwork, leadership, and resilience. The STAR method (Situation, Task, Action, Result) will be your guiding principle, helping you structure your responses in a clear, concise, and compelling manner. Ask yourself questions like, "Describe a time when you had to troubleshoot a critical issue under tight deadlines," or "Tell us about a moment when you proposed a change that significantly improved system reliability." The aim is to not just recount your experiences but to demonstrate your thought process, decision-making skills, and the impact of your actions.

Day 5: Incident Management and Collaborative Problem-Solving

Building on the behavioral foundations, Day 5 goes into SRE-specific scenarios, particularly focusing on incident management, system failures, and collaborative problem-solving under pressure. You'll face questions designed to uncover your ability to manage crises, coordinate with cross-functional teams, and communicate effectively during high-stakes situations. Prepare for questions like, "Explain how you managed a major service outage," or "Describe your approach to resolving a conflict within your team during a critical system update." Utilizing the STAR method, your responses should clearly articulate how you identified the issue, formulated a strategy, collaborated with others, and implemented a solution that mitigated the impact of the incident.

Resources:

• Star Interview Method

• The Star Method for Behavioral Interview

• 10 Incident Response SRE Interview Questions

• Behavioral Interview for Software Engineers

These two days of behavioral mock interviews are crucial for demonstrating your soft skills, which are as important as your technical skills in an SRE role. Employers look for candidates who not only have the technical expertise to manage and improve systems but also the interpersonal skills to work effectively within teams and handle stress and conflict gracefully.

Days 6-7: Revision and Feedback

Day 6: Comprehensive Revision

On Day 6, it's time to consolidate your learnings from the week. Begin with a thorough review of the technical concepts and system design principles you tackled in the first three days. Revisit the LeetCode problems and system design scenarios, paying special attention to the areas where you felt challenged or received constructive feedback. This is also a good opportunity to refine your understanding of SRE fundamentals, including reliability patterns, performance metrics, and incident management protocols. Use this day for focused study sessions, perhaps revisiting key resources or tutorials that address your areas for improvement. The goal is to reinforce your knowledge base and sharpen your skills, ensuring you're well-prepared for any technical question that might come your way.

Day 7: Reflecting on Feedback and Setting Goals

The final day of your interview preparation plan is dedicated to reflection and forward planning. Start by reviewing the feedback received during your mock interviews. Identify common themes or areas for improvement, whether they relate to technical skills, communication, problem-solving approaches, or behavioral responses. Reflect on this feedback critically, acknowledging your strengths and recognizing areas where you can grow.

After this reflection, set actionable goals for continued improvement. These might include continue mastering specific technical skills, practicing the STAR method to refine your storytelling abilities, or engaging in more real-world problem-solving exercises to build your confidence and competence in SRE scenarios. Consider scheduling follow-up mock interviews with peers or mentors to assess your progress against these goals.

Additionally, use this day to practice self-care and mental preparation. Interviewing can be stressful, and ensuring you're mentally and physically ready is as important as the technical and behavioral preparation.

Days 6 and 7 wrap up your intensive interview preparation plan by emphasizing the importance of self-assessment and continuous improvement. By taking the time to review, reflect, and set future goals, you're not just preparing to succeed in your interviews but also laying the groundwork for a thriving career.

Conclusion

As we conclude our six-week SRE interview preparation journey, remember that the essence of success lies in the balance between both technical and behavioral proficiency.

🚀 Stay committed to continuous learning, and approach your interviews with confidence. Don't stop practicing preparation topics while you are hunting for jobs, talking to recruiters and engineering managers. Good luck!

The "Harmless" Y2K

Remember the "Year 2000" problem (or Y2K) in the late 90s? This referred to the fact that many older computer systems represented years using only two digits. This meant that when the year 2000 arrived, many systems would incorrectly display it as "00" instead of "2000", causing errors with dates and calculations.

While I was a teenager at the time, I was already into computers so it fascinated me and I was glued to the screen reading about it. When the New Year eve hit... nothing happened. Life continued as normal, so all the non-technical people around me said this "threat" was blown out of the proportion to make some news headlines. What I didn't know is that behind the scenes there was a massive efforts to remediate systems to avoid major disruptions when the actual date arrived.

This is a great example of "Preparedness paradox". When a potential disaster is properly prepared for, it won't seem like a big deal after the fact. People will think the precautions were unnecessary. But Y2K could have been a major catastrophe if nothing was done. Some estimates suggested failures in key systems could lead to blackouts, air travel shutdowns, supply chain issues and even nuclear missile system failures.

The End of Time

"The 2038 problem" relates to an issue with how Unix-based systems store dates and timestamps. Most Unix systems use a 32-bit signed integer to represent the number of seconds since January 1st, 1970. This is known as the Unix epoch or Unix time.

The core of the issue is that a 32-bit variable can only store integers up to 2147483647. Once the system clock ticks past this at 03:14:07 UTC on January 19, 2038, it will integer to overflow, setting its value to −(2³¹) , wreaking havoc from there:

When this overflow happens, any application relying on the correct Unix time stamp will fail or behave erratically. Dates will suddenly show as being in December 1901, calculations will be off, comparisons won't work properly, and overall system functionality will break.

The Impact

What systems can be impacted by this issue if we don't act accordingly:

• Legacy Systems and Embedded Devices: many older systems, especially those in critical infrastructure like nuclear power plants, airplanes, healthcare devices, telecommunications equipment, and embedded systems in consumer electronics are using 32-bit time representations. These systems are often not updated or replaced frequently, making them the biggest threat.

• Operating Systems: some older or less maintained operating systems that still rely on 32-bit time representations will face issues. Although many modern operating systems have transitioned to 64-bit time representations, legacy versions and systems might still be in use in certain environments.

• Databases: some databases still operate with 32-bit time fields and apps rely on UNIX_TIMESTAMP() type of queries.

• File systems: older file system versions still use 32 bits to represent times in the inodes.

• Networking Equipment: networking protocols and systems that use 32-bit time stamps for scheduling, logging, timeout management, etc could fail.

• Financial Systems: banking and financial systems that use 32-bit timestamps for transaction processing, logging, or future date calculations may also be impacted.

Essentially any software, service, business or personal function that relies on accurate recording of dates and times is at risk if steps are not taken to address the 2038 problem. And the risks go beyond just computer failures - they extend to legal, financial, scientific and other integrity issues. I don't consider myself a prepper, but proactive remediation is needed to avoid catastrophe.

Let's do a little experiment. Set your system date past January 19, 2038 and see if the software you are working on is still functions properly. Chances are it might break.

The Mitigation

We are only 14 years away. The time is running out fast, but the good news the tech community is already working on mitigation, in order to address the risks before 2038.

Some OSS examples I found of what being done proactively:

• Linux is ready to the end of time

• Debian efforts to address Y2K38

• LWN coverage of 2038 progress

• XFS addresses 2038 problem

• MySQL functions to support 64-bit unix timestamp

• Memcached fixes log timestamps

Implementing the following actions today can significantly reduce your chances of hitting the bugs in your systems:

• Identify the inventory of vulnerable systems by thorough inspections and testing to uncover any sections of code, libraries, or dependencies susceptible to the 2038 issue.

• Transition to a 64-bit system for the storage and management of date and time information, greatly increasing the range of dates that can be accommodated.

• Adopt alternative formats for date and time, such as ISO 8601, to enhance the robustness and compatibility of data handling.

• Perform required software, firmware or hardware upgrades and migrations to ensure your systems are future proof. Easier said than done, as some of this would require significant capital/operating investments and multi-year journey to finish.

Closing Thoughts

The Year 2038 bug is real and will bite us if systems are not properly updated. While it may seem distant, the task of updating legacy infrastructure is monumental and time-consuming. We cannot afford to delay.

And I don't know about you, but I'm definitely not flying or driving anywhere that day 😄

Hey there! Ever wished you could just chat with your Kubernetes cluster instead of wrestling with endless commands? Well, guess what? Now you can, thanks to a bit of magic from OpenAI's GPT.

In this post, I'm gonna show you a super simple Python script that lets you do just that. Whether you're a coding newbie or a seasoned pro, you'll see how easy it is to make your Kubernetes cluster understand plain English.

So, grab your favorite snack, and let's get this coding party started. It's gonna be fun, I promise!

Design

Here is a diagram illustrating the workflow of the chatbot system, including interactions between the User, Chatbot, OpenAI API, and the Kubernetes cluster:

• User to Chatbot: The user inputs a command or query.

• Chatbot to OpenAI API: The chatbot sends the query to the OpenAI API.

• Conditional Interaction with Kubernetes:

  • If the OpenAI API requests the execution of a Kubernetes command, the chatbot executes it on the Kubernetes System and returns the output back to the OpenAI API.

  • If no Kubernetes command is needed, the OpenAI API generates a response based on AI.

• Final Response to User: The OpenAI API sends the final response to the chatbot, which then displays it to the user.

Environment Set Up

Before we go into the tutorial, ensure you have Python 3 installed and accessible in your terminal. We'll begin by creating a virtual environment and installing two essential libraries: openai and colorama. This setup is crucial for a streamlined development experience.

mkdir k8s-chat
cd k8s-chat

# set up env
python3 -m venv .venv
source .venv/bin/activate

# install libraries
pip install openai
pip install colorama

# create the python file
# for our code
touch chat_with_k8s.py

Code

Let's start by writing a simple python function that will be able to execute any arbitrary kubectl commands:

Executing Kubectl commands:

import subprocess

def execute_kubectl_cmd(cmd):
    # add kubectl prefix if it's not already there
    if not cmd.startswith("kubectl"):
        cmd = f"kubectl {cmd}"

    if cmd.startswith("kubectl delete"):
        return "I'm sorry, Deleting resources is disabled."
    
    try:
        output = subprocess.check_output(cmd, shell=True)
        return output.decode('utf-8')
    except subprocess.CalledProcessError as e:
        return f"Error executing kubectl command: {e}"

This function is a utility for safely executing Kubernetes commands from a Python script. It ensures that commands are correctly formatted, prevents potentially destructive delete operations, and provides feedback on the success or failure of command execution.

Tool Definition:

Let's then define the using the openai function calling documentation,

import json
import subprocess

from colorama import Back, init, Fore
from openai import OpenAI

client = OpenAI()
model_name = "gpt-3.5-turbo-0125"

tools = [
    {
        "type": "function",
        "function": {
            "name": "execute_kubectl_cmd",
            "description": "execute the kubeclt command against the current kubernetes cluster",
            "parameters": {
                "type": "object",
                    "properties": {
                        "cmd": {
                            "type": "string",
                            "description": "the kubectl command to execute",
                        },
                    },
                    "required": ["cmd"],
            },
        },
    }
]

The client is simply initializing the openai api, and the model_name is just a variable for what model we want to use with our system.

A list named tools is defined with a single tool (execute_kubectl_cmd), detailing its purpose and required parameters. This tool is intended to execute Kubernetes commands.

💡 You are not limited to one function when using gpt function calling

Processing Chat Input:

This function contains the chat completion and the interaction with the openai api

def chat_completion(user_input):
    messages = [{"role": "user", "content": user_input}]
    response = client.chat.completions.create(
        model=model_name,
        messages=messages,
        tools=tools,
        tool_choice="auto",
    )
    response_message = response.choices[0].message
    tool_calls = response_message.tool_calls
    if tool_calls:
        available_functions = {
            "execute_kubectl_cmd": execute_kubectl_cmd,
        }
        messages.append(response_message)
        for tool_call in tool_calls:
            function_name = tool_call.function.name
            function_to_call = available_functions[function_name]
            function_args = json.loads(tool_call.function.arguments)
            function_response = function_to_call(
                cmd=function_args.get("cmd"),
            )
            messages.append(
                {
                    "tool_call_id": tool_call.id,
                    "role": "tool",
                    "name": function_name,
                    "content": function_response,
                }
            )
        second_response = client.chat.completions.create(
            model=model_name,
            messages=messages,
        )
        return second_response.choices[0].message.content
    else:
        return response_message

Below are the steps the function takes.

• The chat_completion function simulates processing user input through an AI model. It prepares the input message, sends it to the OpenAI client, and retrieves a response.

• If the response includes a call to the execute_kubectl_cmd tool, it executes the command and appends both the AI's response and the command's output to the conversation.

• It then sends this extended conversation back to the model for further processing, if necessary, and returns the final message content to be displayed to the user.

Running the Chatbot:

The run_conversation function initiates an interactive chat session. It welcomes the user and enters a loop to accept user input.

def run_conversation():
    print("Welcome to the Kubernetes chatbot!")
    print("You can ask me anything about your Kubernetes cluster.")
    while True:
        print(Back.CYAN + Fore.BLACK + " You: ", end="")
        print(" > ", end="")
        user_input = input()
        if user_input.lower() == "exit" or user_input.lower() == "q":
            print("Goodbye!")
            break
        else:
            resp = chat_completion(user_input)
            print("")
            print(Back.GREEN + Fore.BLACK + " Assitant: ", end="")
            print(" > ", end="")
            print(f"{resp}")
            print("")

If the user types "exit" or "q", the chatbot ends the session. Otherwise, it processes the user's input through the chat_completion function.

Main Execution

Finally, the script initializes Colorama's auto-reset feature (to prevent color codes from leaking into unrelated terminal output) and starts the conversation loop.

def main():
    init(autoreset=True)
    run_conversation()


if __name__ == "__main__":
    main()

To run the code, just make sure that you have OPENAI_API_KEY secret in the env and that you are running from the venv env created at the beginning of the post.

💡 In order to obtain the OpenAI api key secret, you need to create an account with OpenAI, and add $$ to it. You can find more on this page.

export OPENAI_API_KEY="xxxxxxxxxx"
python chat_with_k8s.py

Demo

We will use Kind to run a local kubernetes cluster, and chat with it.

Utilizing the openai library, we seamlessly interact with the GPT AI model, particularly through function calling, streamlining the process of communicating with functions or APIs. This demonstrates the practicality and efficiency of integrating AI into operational scripts.

for the full code, you can check the git repository.

And don't forget to sign up to get fresh content delivered to your inbox.

At the closing talk of GopherConAU 2023, Rob Pike, one of the original creators of Go, shared a retrospective on the 14-year journey of the Go programming language. Celebrating the anniversary of Go's launch as an open-source project, Pike offered a personal, rather than an official Google or Go team perspective. Check it out! But if you are short on time check our notes below.

Upd: a stranger from hackernews pointed out that Pike also posted a detailed text version of this talk here.

Beyond a Programming Language

Pike told the audience that Go's inception was driven not by a desire to create a new programming language but to address the complexities of modern server software development. Highlighting goals like improved build times, effective dependency management, and efficient multicore CPU utilization, he argued that Go's real triumphs extend into tooling, deployment, and community engagement rather than just language design.

Successes Celebrated

Pike emphasized several aspects where Go notably succeeded:

• Specification and Multiple Implementations: Go's formal specification from the outset allowed for multiple compatible compiler implementations, providing a diverse ecosystem.

• Portability: Go was designed to support easy cross-compilation, allowing developers to write code on any platform and compile it for any other.

• Compatibility: Go team has been committed to a compatibility promise, ensuring that code written for earlier versions continues to compile and run correctly with newer versions. This commitment has been crucial for developers relying on Go for long-term projects, providing them with confidence that their codebase will remain stable over time.

• Concurrency Model: Go's approach to concurrency, leveraging goroutines and channels, was highlighted as a significant advancement, making concurrent programming more accessible and safer.

• Standard Library: The comprehensive standard library was praised for providing a solid foundation for developing modern server software, unifying the community and preventing many libraries from rising at early stages.

• Interfaces: Pike emphasized that the way interfaces integrate into Go, enabling polymorphism through composition, is one of the most distinctive and successful aspects of the language. The interface mechanism aligns well with Go's focus on simplicity, readability, and organic code design.

• Tooling: The reliability, speed, and ease of use of the Go toolchain, including the compiler, linker, and package manager, were highlighted as key factors in Go's adoption. Tools like go fmt and go vet ensure code consistency and quality, aiding in the maintenance and collaboration of large codebases.

• Community and Documentation: The vibrant and supportive Go community was acknowledged for its contributions to enriching the Go ecosystem with libraries, tutorials, and blog posts. Pike also commended the extensive and clear documentation that has significantly facilitated the onboarding of new developers.

Addressing Criticisms

While acknowledging Go's widespread adoption and success in various industries, Pike did areas of contention, such as error handling, the initial absence of generics, and the limitations in reflection capabilities and performance. He expressed personal regrets and areas for improvement, including the standard library and safety measures in the compiler.

Looking Forward

Pike's talk also looked to the future, suggesting areas where Go could continue to evolve, such as improving compile-time safety checks and exploring memory management strategies like arenas for specific use cases. He underscored the importance of balancing innovation with Go's core principles of simplicity and efficiency.

As Go continues to mature, its journey offers lessons in language design, community building, and the iterative process of software development.

Series Overview:

• Week 1: Fundamentals of SRE

• Week 2: Automation & Scripting

• Week 3: Monitoring, Logging, and Alerting

• Week 4: Incident Management Lifecycle

• Week 5: Scalability, Performance, & System Design (This post)

• Week 6: Mock Interviews and Revision

Welcome to Week 5 of our SRE interview preparation guide, focusing on scalability, performance tuning, and systems design. This week is crucial for understanding how to ensure that systems can handle growth and maintain performance under different loads. As systems grow in users and data, they must scale effectively to meet demand without compromising on performance. This part will equip you with the foundational knowledge needed to tackle these challenges.

Throughout this week, we'll cover the key concepts of scalability and performance, understand various tuning techniques, and explore real-world case studies to see how theory is applied in practice. Whether you're preparing for an interview or looking to strengthen your skills, this will be valuable for designing and optimizing systems that are robust, efficient, and scalable.

Days 1-3: Understanding Scalability and Performance Concepts

Day 1: Introduction to Scalability

• What is Scalability? Learn the difference between vertical and horizontal scaling and when to use each.

• Challenges of Scaling: Discover common challenges in scaling applications, including database bottlenecks, cache invalidation, and the CAP theorem.

Day 2: Performance Metrics and Tools

• Key Metrics: Understand the importance of latency, throughput, and error rates in measuring system performance.

• Monitoring and Observability Tools: Get acquainted with tools like Prometheus, Grafana, and Elastic Stack for monitoring system health and performance.

Day 3: Architectural Patterns for Scalability

• Microservices Architecture: Explore how decomposing applications into microservices can improve scalability.

• Database Sharding: Learn about database sharding techniques for distributing data across multiple servers to reduce load.

• Caching Strategies: Understand various caching strategies and their impact on performance.

Sample Interview Questions:

1. What is the difference between vertical scaling and horizontal scaling, and when would you use each?

2. How does the CAP theorem affect the design of distributed systems?

3. What are some common challenges you might face when scaling an application, and how would you address them?

4. Explain the importance of latency, throughput, and error rates in measuring the performance of a system.

5. Discuss how you would use a tool like Prometheus or Grafana for monitoring system performance.

6. How can a microservices architecture improve the scalability of an application? What are the potential trade-offs?

7. Describe a scenario where database sharding can be used to improve performance. How does it work?

8. What strategies would you employ to optimize the performance of a web application at the code level?

9. Explain various caching strategies and how they impact system performance.

10. Can you describe a situation where you had to scale a system to meet increased demand? What challenges did you face, and how did you overcome them?

Resources:

• Design Data Intensive Applications (Book)

• Scaling Software Systems: 10 Key Factors (Blog Post)

• High Scalability (Blog)

• Reliable Product Launches at Scale (SRE Book Chapter)

• Scalable Systems 101 (Blog Post)

• Perspectives on the CAP Theorem (Paper)

Days 4-5: Performance Tuning Techniques

Day 4: Optimizing Application Performance

• Code Optimization: Tips for writing efficient code to reduce latency and increase throughput.

• Concurrency and Parallelism: Techniques for leveraging concurrency and parallel processing to improve application performance.

Day 5: Infrastructure Optimization

• Load Balancing: Learn how to effectively distribute traffic across servers to maximize resource utilization.

• Auto-scaling: Understand how to automatically scale your infrastructure based on the load to maintain optimal performance.

Sample Interview Questions:

1. Describe how you would approach performance tuning in a system that is experiencing high latency.

2. What tools and techniques do you use for identifying performance bottlenecks in a web application?

3. Explain the concept of load balancing and how it can be implemented to improve application performance and availability.

4. How would you implement auto-scaling for a cloud-based application, and what metrics would you monitor to trigger scaling actions?

5. Discuss the impact of database optimization on application performance. What strategies might you employ to optimize database queries?

6. Can you describe a time when you successfully optimized the performance of a system? What was the issue, and what steps did you take to resolve it?

7. What role does caching play in improving system performance, and how do you decide what to cache?

8. How do you ensure that performance optimizations do not negatively impact the maintainability and readability of the code?

Resources:

• Systems Performance (Book)

• SQL Tuning (book)

• Brendan Gregg's Blog (Blog)

• Linux Systems Performance (Video)

Days 6-7: System Design

In the final days of Week 5, our focus shifts towards system design interviews, emphasizing the creation of large-scale systems such as YouTube, Netflix, Uber, or Messenger from the ground up. Candidates will be challenged to outline their approaches to architecting these platforms, considering scalability, data management, user experience, and performance optimization.

This exercise tests the ability to integrate various components: like databases, caching, load balancing, and microservices into a cohesive, scalable system. It's an opportunity for candidates to demonstrate their understanding of complex system architectures and their capability to design solutions that are not only efficient and reliable but also scalable and performant under high loads. This segment aims to mimic real-world challenges, preparing candidates for the intricacies of system design they'll encounter in their careers.

Sample Interview Questions:

1. How would you design a scalable video streaming service like YouTube?

2. Design a global messaging service like WhatsApp or Messenger.

3. How would you architect a ride-sharing service like Uber or Lyft?

4. Design a content delivery network (CDN) to serve web content globally.

5. How would you create a scalable e-commerce platform like Amazon?

6. Design a real-time sports scoring app that can handle millions of users.

7. How would you architect a cloud-based file storage service like Dropbox?

8. Design a scalable social networking platform like Twitter or Instagram.

Resources:

• Grokking the System Design Interview (Paid Course)

• System Design Primer (Guide)

• System Design (Guide)

• System Design Interview (Guide)

Conclusion:

Through understanding key concepts, mastering tuning techniques, and exploring system design challenges, you're now equipped to tackle the scalability and performance questions that come your way.

Linux CLIs are a part of every software engineer's daily workflow. But I still see many developers rely on legacy tools that have been around for decades. It's time to upgrade your CLI toolbelt and switch to faster, more powerful, and flexible tools.

• grep → ripgrep (rg)

  • Significantly faster, respects .gitignore files, and offers a more user-friendly syntax and output. It’s designed for speed and efficiency in searching or filtering large amounts of text. If you were to try just one tool from this list - try ripgrep.

• find → fzf

  • fzf is a general-purpose command-line fuzzy finder and it would enhance your overall productivity. It can be used to search and select files, command history, processes, hostnames, bookmarks, git commits, and much more. If you try command history search with fzf - you'll never go back to the old ways.

    

• cat → bat

  • Adds syntax highlighting for code snippets and integrates with git for side-by-side diff views.

    

• ls → eza / lsd

  • Both offer color-coded output and icons, so provides visually pleasing and structured display. They are also backwards-compatible with ls, so it would be easy to transition.

• du → ncdu

  • Offers an interactive interface, making it easier to navigate and understand disk usage. It's user-friendly and provides a clearer view of what's consuming disk space. No more du -hs *

• man → tldr

  • Simplifies command documentation, providing concise and practical examples. It’s great for quickly learning or recalling a command without wading through verbose manuals.

    

• diff → delta

  • Provides a side-by-side comparison with syntax highlighting, making it far easier to spot differences. It's an excellent tool for code review and comparing file changes.

• curl → curlie

  • Offers a more user-friendly interface for making web requests, combining the power of curl with the ease of use typically found in HTTP clients with a more modern interface.

• dig → dog

  • A modern alternative to dig, featuring a more user-friendly interface, colorful output, and additional DNS query types. It’s designed for readability and simplicity.

Updating your muscle memory and systems to use these tools may take some time, but the payoff in productivity and capabilities is worth it. Give them a shot!

In the first part of our exploration into the fallacies of distributed computing, we looked into four common misconceptions that can significantly impact the design and functionality of distributed systems: the reliability of the network, the illusion of zero latency, the myth of infinite bandwidth, and the false sense of inherent network security.

Now, in Part 2 of this series, we turn our attention to the remaining four fallacies, each presenting unique challenges and requiring careful consideration:

1. Topology Doesn't Change: The oversight of network dynamics and their impact on system performance.

2. There is One Administrator: The simplification of management responsibilities and control in distributed environments.

3. Transport Cost is Zero: The underestimation of the resources required for data movement across the network.

4. The Network is Homogeneous: The assumption that the network environment is uniform and consistent.

Fallacy 5: Topology Doesn't Change

This one is a misconception in distributed computing that disregards the dynamic nature of network topologies. Network topology, the arrangement of various elements (links, nodes, etc.) in a computer network, is not static. Changes can occur due to network expansion, hardware upgrades, outages, or reconfigurations. Designing systems under the assumption of a static topology can lead to significant issues when inevitable changes occur.

1. System Rigidity: Systems designed for a fixed topology may lack the flexibility to adapt to changes, leading to failures or suboptimal performance.

2. Maintenance Challenges: Regular updates or modifications can become cumbersome and risky in a rigidly designed system.

3. Scalability Issues: A system that doesn't account for changing topology might struggle to scale efficiently as the network grows or evolves.

4. Inefficient Resource Utilization: Fixed-topology assumptions can lead to poor resource allocation, as the system cannot dynamically adjust to the most efficient paths or nodes.

Mitigation Strategies:

1. Dynamic Configuration: Implement mechanisms that allow the system to automatically adapt to changes in network topology.

2. Regular Monitoring and Updates: Continuously monitor network topology and perform regular updates to ensure the system aligns with the current network state.

3. Decentralization: Avoid single points of failure by decentralizing functions and resources where feasible.

4. Redundancy: Incorporate redundancy in network paths and nodes to maintain functionality even if parts of the network change or fail.

5. Flexible Protocols: Use network protocols that can handle changes in topology without significant disruption.

6. Testing for Variability: Regularly test the system under varying topological conditions to ensure robustness against changes.

7. Documentation and Communication: Maintain clear documentation of the network topology and ensure effective communication among team members about changes.

By acknowledging and preparing for the inevitability of changing network topologies, distributed systems can be made more adaptable, resilient, and efficient.

Fallacy 6: There is One Administrator

The belief that 'There is One Administrator' is a fallacy in distributed computing that oversimplifies the management and control of distributed systems. In reality, distributed systems often span multiple administrative domains, each with its own policies, procedures, and management styles. Assuming a single administrative control point can lead to serious misjudgments in system design, particularly in areas related to governance, security, and resource sharing.

Implications:

1. Coordination Challenges: Multiple administrators mean coordination becomes more complex, and unilateral decisions are often impractical.

2. Security Policy Conflicts: Differing security policies and practices across administrative domains can lead to inconsistencies and vulnerabilities.

3. Resource Management Issues: Assumptions about resource control and allocation can be misguided when multiple administrators are involved.

4. Compliance Complications: Adhering to various regulatory and policy requirements can be challenging in a multi-administrator environment.

Mitigation Strategies:

1. Distributed Governance: Establish a governance model that accommodates input and decision-making from all administrative domains.

2. Unified Security Standards: Work towards a common set of security standards and practices that are agreeable and applicable across all administrative areas.

3. Flexible Resource Allocation: Implement resource management systems that are flexible and can adapt to the needs and policies of different administrators.

4. Clear Communication Channels: Ensure clear and effective communication channels among various administrators to facilitate coordination and conflict resolution.

5. Decentralized Control Mechanisms: Use decentralized control mechanisms where possible to allow for autonomy within different administrative domains.

6. Comprehensive Documentation: Maintain detailed documentation of system policies, procedures, and agreements that involve multiple administrative domains.

Recognizing the reality of multiple administrators in distributed systems and adopting these strategies can greatly enhance the management, security, and overall functionality of these complex environments.

Fallacy 7: Transport Cost is Zero

A common fallacy in distributed computing, which overlooks the resources required for data movement across a network. This fallacy ignores the costs associated with bandwidth usage, latency, and the energy required for data transmission. In reality, transporting data, especially large volumes over long distances, incurs significant costs and can impact system performance and efficiency.

Implications:

1. Resource Inefficiency: Ignoring transport costs can lead to inefficient use of network resources, such as bandwidth and energy.

2. Increased Operational Costs: Overlooking the cost of data movement can result in unexpectedly high operational expenses, especially in cloud-based services where data transfer fees are involved.

3. Performance Bottlenecks: Underestimating transport costs can cause bottlenecks in system performance, particularly when large data transfers are frequent.

Mitigation Strategies:

1. Data Localization: Keep data as close as possible to its primary users to minimize unnecessary data movement.

2. Bandwidth Management: Monitor and manage bandwidth usage to optimize data transfer processes and reduce costs.

3. Data Compression: Employ data compression techniques to reduce the size of data being transported.

4. Cost-Aware Architecture Design: Design system architectures with a focus on minimizing and optimizing data transport.

5. Selective Data Movement: Be selective about what data needs to be moved and when, avoiding unnecessary data transfers.

By acknowledging the real costs associated with data transport in distributed systems and implementing these mitigation strategies, it is possible to build more efficient, cost-effective, and environmentally friendly systems.

Fallacy 8: The Network is Homogeneous

The assumption that 'The Network is Homogeneous' is a fallacy in distributed computing that ignores the diversity in network environments. This fallacy leads to the expectation that all parts of a network will behave similarly and support the same protocols, performance levels, and features. In reality, networks are composed of a variety of devices, technologies, and configurations, each with its own characteristics and limitations.

For instance, this image above taken from this post shows the various differences in performance, cost, time, and security between AWS VPN and Direct Connect.

Implications:

1. Compatibility Issues: A homogeneous network assumption can lead to compatibility problems when systems encounter different network technologies.

2. Performance Variability: Disregarding network diversity can result in unpredictable performance, as different network segments may have varying capacities and speeds.

3. Scalability Challenges: Scaling a system becomes more complex when it needs to adapt to diverse network environments.

4. Inadequate Error Handling: Systems might not be equipped to handle the range of errors that can occur in a heterogeneous network.

Mitigation Strategies:

1. Cross-Network Compatibility: Design systems and protocols to be compatible with a range of network technologies and standards.

2. Adaptive Performance Tuning: Implement mechanisms that dynamically adjust performance based on the current network environment.

3. Extensive Testing: Test systems in a variety of network conditions to ensure robustness and adaptability.

4. Flexible Architecture: Build a flexible and modular architecture that can easily adapt to different network settings.

5. Detailed Network Analysis: Regularly analyze the network to understand its composition and tailor the system accordingly.

6. Robust Error Handling: Develop comprehensive error handling that can manage the diverse failures and issues that arise in heterogeneous networks.

7. User-Aware Optimization: Optimize system performance based on the specific network characteristics of different user segments.

8. Continuous Monitoring and Updates: Continuously monitor network performance and update the system to handle evolving network environments.

Recognizing the diversity in network environments and adopting these strategies helps in building distributed systems that are more resilient, adaptable, and capable of operating efficiently across heterogeneous networks.

Distributed systems are tricky - it's easy to make wrong assumptions that lead to problems down the road. Back in the 90s, computer scientist L. Peter Deutsch identified several common misconceptions, or "fallacies," that trip up engineers working on distributed systems. Surprisingly these fallacies are still relevant today:

1. The Network is Reliable: It's risky to assume networks are 100% reliable. Networks can and do fail in various ways.

2. Latency is Zero: While we might wish our networks had no latency, that's simply not physically possible - even light takes time to travel distances. Ignoring the inevitable delay in data transmission can lead to unrealistic expectations of system performance.

3. Bandwidth is Infinite: This overlooks the physical and practical limitations on data transfer rates.

4. The Network is Secure: No wonder Security is a growing industry. Assuming inherent security can lead to vulnerabilities and oversight in protective measures.

5. Topology Doesn't Change: This neglects the dynamic nature of network configurations.

6. There is One Administrator: A simplification that fails to consider the complexity of managing distributed systems.

7. Transport Cost is Zero: Overlooking the resources required for data movement.

8. The Network is Homogeneous: Ignoring the diversity in network systems and standards.

These fallacies, if not recognized and addressed, can lead to design flaws, performance issues, and security vulnerabilities in distributed systems. In the following sections, we will break down each of these misconceptions, exploring their implications and how to mitigate the risks they pose in real-world applications.

Fallacy 1: The Network is Reliable

The belief that 'The Network is Reliable' is one of the most common and risky assumptions in the field of distributed computing. This fallacy leads to an underestimation of the likelihood and impact of network failures. In reality, networks are susceptible to a range of issues, from temporary outages and packet loss to more severe disruptions caused by hardware failures, software bugs, or external factors like natural disasters.

Implications:

1. System Downtime: Relying on a flawless network can lead to significant system downtime when inevitable failures occur.

2. Data Loss or Corruption: Without robust handling of network issues, data may be lost or corrupted during transmission.

3. Security Vulnerabilities: Assuming network reliability might lead to neglecting necessary security protocols, potentially exposing the system to attacks.

4. Poor User Experience: Applications that don't account for network unreliability can frustrate users with inconsistent performance.

Mitigation Strategies:

1. Redundancy: Implement redundant network paths and failover mechanisms to maintain system functionality during network failures.

2. Retries and Timeouts: Incorporate intelligent retry mechanisms with exponential backoff and sensible timeout settings to handle transient network issues.

3. Acknowledgment Mechanisms: Use acknowledgment messages to confirm the successful reception of data.

4. Monitoring and Alerting: Establish comprehensive monitoring and alert systems to detect and respond to network issues promptly.

5. Load Balancing: Utilize load balancers to distribute traffic evenly across the network, preventing overload on any single point.

6. Caching: Employ caching strategies to provide users with immediate access to data during short network outages.

7. Data Validation and Correction: Implement checksums and error correction protocols to ensure data integrity.

8. Graceful Degradation: Design systems to degrade functionality gracefully in the face of network issues, maintaining a level of service where possible.

By acknowledging and preparing for the inherent unreliability of networks, distributed systems can be made more robust, resilient, and user-friendly.

Fallacy 2: Latency is Zero

The assumption that 'Latency is Zero' is a common oversight in distributed systems. This fallacy ignores the time taken for data to travel across the network. In reality, latency is affected by a multitude of factors, including physical distance, network bandwidth, router hops, and the quality of connections. Even in high-speed networks, latency can never be completely eliminated.

Implications:

1. Performance Issues: Applications designed without considering latency can suffer from poor performance, especially in scenarios requiring real-time data processing.

2. User Experience Degradation: Interactive applications, such as online gaming or video conferencing, can become frustratingly slow, impacting user satisfaction.

3. Inaccurate System Synchronization: Time-sensitive operations may fail or produce erroneous results due to unexpected delays.

4. Inefficient Resource Utilization: Overlooking latency can lead to suboptimal resource allocation, as systems might wait unnecessarily for responses.

Mitigation Strategies:

1. Geographic Distribution: Place servers and data centers closer to the end-users to minimize physical distance and, consequently, network latency.

2. Optimization of Protocols: Use efficient communication protocols that minimize overhead and are optimized for the specific use case.

3. Asynchronous Communication: Implement asynchronous operations to prevent systems from stalling while waiting for responses.

4. Caching Strategies: Employ caching closer to the user to reduce the need for frequent long-distance data retrieval.

5. Load Balancing: Utilize smart load balancing that considers geographic location and current network latency.

6. Predictive Fetching: Anticipate user needs and prefetch data to minimize perceived latency.

7. Performance Testing: Regularly test the system under realistic network conditions to understand and optimize for latency impacts.

8. User Interface Design: Design UIs to give immediate feedback to the user, masking underlying network latency.

Acknowledging and accounting for network latency is crucial in designing responsive, efficient, and user-friendly distributed systems. By implementing these strategies, engineers can significantly mitigate the impact of latency on system performance and user experience.

Fallacy 3: Bandwidth is Infinite

This one leads to overlooking the limitations in data transmission capacity of networks. This misconception can result in designing systems that expect high data throughput regardless of the actual network capabilities. In reality, bandwidth is a limited resource, influenced by network infrastructure, traffic congestion, and the capabilities of end-user devices.

Implications:

1. Bottlenecks and Slowdowns: Systems designed without bandwidth constraints in mind can suffer from bottlenecks, leading to significant slowdowns.

2. Inefficient Data Handling: Overestimating bandwidth can result in sending large amounts of data unnecessarily, wasting network resources and impacting other operations.

3. Poor Scalability: Systems that don't account for bandwidth limitations may face challenges in scaling up to accommodate more users or data.

4. User Experience Issues: Users on networks with limited bandwidth might experience delays, interruptions, or even service unavailability.

Mitigation Strategies:

1. Data Optimization: Compress data to reduce size and optimize formats for efficient transmission.

2. Adaptive Techniques: Implement adaptive streaming and data transfer techniques that adjust to available bandwidth.

3. Load Balancing: Use load balancing to distribute network traffic efficiently across available paths.

4. Bandwidth Throttling: Intelligently throttle bandwidth usage to avoid overwhelming network capacity.

5. Prioritization of Traffic: Prioritize critical data traffic to ensure essential services remain unaffected during high load.

6. Monitoring and Analytics: Continuously monitor network performance and use analytics to understand and manage bandwidth usage.

7. Scalable Architecture: Design a system architecture that can gracefully adapt to varying bandwidth conditions.

By considering the finite nature of bandwidth and implementing these strategies, distributed systems can be designed to be more efficient, reliable, and user-friendly, even under varying network conditions.

Fallacy 4: The Network is Secure

This fallacy leads to complacency in security measures, overlooking the myriad of threats present in network environments. Network security is not inherent but requires deliberate and continuous effort. Threats can arise from various sources, including external attacks, internal vulnerabilities, and even inadvertent user actions.

Implications:

1. Vulnerability to Attacks: Unsecured networks are prone to various types of cyberattacks like hacking, phishing, and DDoS attacks.

2. Data Breaches: Insufficient network security can lead to unauthorized access and theft of sensitive data.

3. Compliance Issues: Neglecting network security can result in non-compliance with regulatory standards, leading to legal and financial repercussions.

4. Loss of Trust: Security incidents can damage the reputation of an organization and erode user trust.

Mitigation Strategies:

1. Encryption: Use strong encryption for data in transit and at rest to protect against eavesdropping and data breaches.

2. Authentication and Authorization: Implement robust authentication mechanisms and ensure strict authorization controls to regulate access.

3. Regular Security Audits: Conduct regular security audits and vulnerability assessments to identify and rectify potential weaknesses.

4. Firewalls and Intrusion Detection Systems: Deploy firewalls and intrusion detection systems to monitor and protect network traffic.

5. Security Training: Educate employees and users about security best practices and the importance of safeguarding network resources.

6. Up-to-date Software: Keep all network software, including operating systems and applications, updated to patch known vulnerabilities.

7. Network Segmentation: Segment the network to contain breaches and limit access to sensitive areas.

8. Incident Response Plan: Develop and maintain an incident response plan to quickly and effectively address security breaches.

Acknowledging the inherent insecurity of networks and implementing these strategies is essential for maintaining the confidentiality, integrity, and availability of distributed systems and their data.

In this blog post, we've explored the first four of the eight classic fallacies of distributed computing.

Subscribe to not miss the next part, where we'll continue to explain these concepts and provide insights into building more resilient distributed architectures.

Subscribe now