Homelabs are more than a time sink, they're the palate cleanser every career needs

I paid £42 each for a Dell OptiPlex 7040 Micro about 20 months ago. That box was the first node of what’s now an 8-node bare-metal Kubernetes cluster living in my office. It’s also the reason I’m meaningfully better at my job than I was two years ago.

Are you thinking about building one? Do it. Here’s the story.

I started at the absolute WORST time. My wife had just had our son, we were moving out of London to our new family home. Basically what I am saying is, this wasn’t the time for non-essential purchases or time consuming hobbies.

Like a lot of people, I tend to lean on older hardware for my linux setups. My rotation tends to be that my last gaming PC becomes my next workstation. I’m sure, like many.. the windows AI updates very much put you off. It’s what forced me back to Linux as a daily driver for anything non-gaming (soon to be everything, including gaming - future post coming soon).

I knew that for my homelab that I wanted multiple nodes and that I didn’t want to spend too much to start with.

That kind of set out my first set of constraints though.. the first one being no new hardware. I didn’t however have any old machines lying around to start one. So after a bit of research, I jumped on the TinyMiniMicro homelab bandwagon (albeit a bit late). I had originally looked at Pi’s or other similar boards but they had gotten so expensive!

Catching the bug

I managed to find a vendor that was selling Dell 7040 OptiPlex Micro’s for £42 each, bit of a bargain really. Clearly some kind of office clear out. I’m pretty sure that the seller was using them as a way to drive traffic to their shop because after I bought the first 3, orders were limited to 1 per order. That didn’t stop me from eventually picking up all 8 of my nodes from them.

I also had some old hardware from back when I was a student, specifically a pair of HP ProLiant MicroServers (NL40)

I’ll keep the general specs up to date here: My Hardware.

Now, this is not an impressive set of hardware by any means.. but I knew I wanted a bunch of nodes to manage and I knew that I was going to run k8s.

So, I had my original order of 3 7040’s, an cheap 8 port switch and the NL40.

I had also been given an Ender 3 KE for my birthday, so I immediately combined my hobbies and 3D printed a quick stand for it.

Now, no homelab is complete without a kick-arse Naming Scheme and I have been using mine since 2007. After working on/at AWS for a long time, it was time for some cattle.

At this point, my naming scheme had generally been SciFi pilots and captains for workstations and ships for servers. So we started with the controller defiant, she’s a tough little ship, with serenity and the rocinante as workers. Along with moya for the NFS machine.

I had opted, as part of my learning experiment, to utilise AI to build and configure everything. I would be the driver and I’d use this as an opportunity to do something I know with things that were new to me. I also followed by standard ‘learning philosophy’.

Darius’ Learning Philosophy for new tech: Just use it, ignore everything everyone else is writing/saying/doing and just read the manual and use the tool. Good tools are easy for anyone to use. If a technology is good but the tool is bad, build a new tool.

Now, I’m not sure how many people still RTFM but I generally want to bash my head as much as possible with any new tech because that’s how I learn all the things other people will do. It’s also how I decide what is worth wrapping with helpers etc. Topic for another time.

Within a few hours over a few days, I had k8s setup with traefik and metallb and got gitea, act runner and ansible setup. So pretty quickly I had the machines managed under code and everything going via a pipeline.

I didn’t really fancy drilling holes in my new walls to get an ethernet cable to my router. So it’s a Wifi connected homelab.. not ideal but it works.

Why K8s for a homelab?

Good question. So I had been working with a customer for a while that had a really cool provisioning process for bare metal. It was nicely built, well thought through and it worked.

By this point, I had spun up thousands (if not tens of thousands) of EC2 instances. I had worked with VMWare and Proxmox etc. I hadn’t worked with bare metal since I was in the server management club at university.

I have a simple rule for hobbies, particularly ones that some may percieve as large time sinks - if you’re not learning, you’re wasting your time. The knowledge doesn’t have to be immediately valuable but it’s so easy to stop learning.. it’s a muscle that you need to keep working or you lose it.

Where has that led? Well, every machine in the cluster can be rebuilt via PXE network boot from an arch ISO with a custom build script built and packaged in the lab itself. It self-applies it’s own ansible configuration once the base OS install is done. So now these machines sit somewhere between cattle and pets.

Why Arch? Well, I run Arch everywhere else..

what next?

So, I have the 3 nodes and nfs setup. I an automate the deployment of services and I knew I wanted to host webapps, I needed a domain. So I got dariusaliabadi.tech and setup LetsEncrypt via Route53 for all the internally hosted services. Then I did what most people did, I setup pihole and home assistant and all the various other stuff that you’d want to self-host.

I was getting pretty comfortable with using AI tools on the daily by this point, I wanted to start embedding the usage of them into things like n8n etc. This was when I was starting to think about projects like DATHL Inbox. So I added a node outside the cluster via a gaming laptop that had a 3060TI in it. Enough to run small models. I decided not to join it to the cluster, mostly because I wanted it to have the sole purpose of running ollama.

I also added 1 Optiplex to the cluster (outlawstar) and 1 as a windows VDI because I wanted tools like fusion 360. I soon realised that I wasn’t going to be a very good 3d modeller without a lot of investment of time.. so i switched to tinkercad and the windows VDI became enterprise, the fifth k8s node.

the first physical bottleneck

So by this point, I had maxed out my 8 port switch. 5 nodes, 1 laptop, 1 NAS and ethernet connection directly to my workstation (bypassing the various wifi bridging etc).

So the next upgrade here was to go up to either a 16 or 24 port switch, in both cases.. i felt like quite a jump in price (£50+ for a 16 port switch vs the £15 8 port that I had picked up before). Also these switches were pretty dumb and I was starting to think that I probably would want to wire the house for ethernet in the future. So I wanted something that could potentially support VLANs.

Also, had been a while since I did any network segregating at layer 2. AWS was very much layer 3. Opportunity for further skills development.

So I eventually found that I could get an old Cisco SG220 50 Port switch on ebay for less than the price of a new 16 port tplink switch.

New issue though. It was for a 19” rack. So that started The LackRack project.

The lab has continued to grow since then with different ingress and egress routes, a dedicated firewall on OPNSense and various wired in peripherals like a label printer. I’ll write about that later.

What that £42 actually bought me

Read that back. Bare-metal provisioning I hadn’t touched since university. Layer 2 networking I’d let atrophy after years of AWS’s layer 3 abstractions. Kubernetes from the ground up, not the “here’s a managed control plane, please don’t ask questions” version. PXE boot pipelines. VLANs. Cert-manager and Traefik configs I actually understand because I broke them and fixed them at 11pm.

I remember an SA colleague at AWS asking me about why I ran a homelab and built things outside AWS. I developed a rule while I was there, I always wanted to be able to put my feet in the shoes of a customer. The longer I stayed at AWS, the further away my real experience as them was. So I found ways to create scars that would be at least similar to theirs.

That’s what a homelab is. You don’t have prod, so you make prod. You give yourself something whose failure has actual consequences (my wife can’t find her recipes, the notification pipeline goes down, home assistant stops seeing the lights) and then you have to fix it. Every hour spent inside that loop is experience compounding in a way no cert exam and no LLM can hand you.

Learning Time: Homelab vs Certifications

It’s not fair to ask me this. I’ve been anti-certification for a while, actually. Third week at AWS, I was SKO and got pulled into a call to help someone on a customer account. I don’t remember most of the conversation now, I remember that certs came up and my colleague was very proud of themselves (rightly so) that they’d gotten the Pro Cert (i think it was just out). I also remember that they couldn’t provision a lambda and struggled with cloudformation.

I still think certs are worth it in specific spots. They’re a decent forcing function for learning something you need some structure around, and they open certain doors. But if you asked me what a mid-career engineer should spend their next £500 on, I wouldn’t send them to an exam voucher. I’d send them to eBay for ten mini-PCs and tell them to go and build themselves a production environment, build pipelines and write applications.