Category: Projects

A Rigol DSO804 to be specific. Now I can finally diagnose some of the issues that have been plaguing my Motor Controller.

I verified that the 12 volt rail was not coupling with the 3.3v signal line, but the DRV8825 driver soldered to the board is apparently just a cheap knockoff. The nasty screeching sound caused by~ 12kHz pulsing is partially due to a chip malfunction, although my lack of consideration for EMI didn’t make diagnosing the issue easier.

The third PCB revision is underway having now ironed out most of the most pressing and obvious issues – although the tarrifs are going to make getting the right components a bit harder.

After Broadcom aquired VMware, I was once again introduced to a name that had already been floating around the home-lab circles: Proxmox. At the time I still had no idea what it even was, but as my little experiment began to grow and take a life of its own, I began to find myself searching for a more powerful solution than just containerization – virtual machines. I have previously avoided them due to the need for hardware (that was in short supply until recently, mind you). But now that my need for a more flexible setup emerges as the workload increases, this seems to be about the right time to get started.

Virtual machines are wonderful on a conceptual level, but there are some kinks that need to be addressed in the real world. Where is a virtual machine? How does compute get allocated? What software needs a dedicated VM in the first place? Spinning up Virtual Box for a single machine to test things is great, but what if I want to network with multiple VMs?

It turns out the answer is a hypervisor, a scheduler, and a hundred little goblins in running around with tiny hammers. The hypervisor with a dedicated OS can split its own resources into many different virtual machines that each enjoy its own “isolated” environment for the goblins to run around in. This isn’t a replacement for containers, this is a way to supplement them with software that can’t easily be run on a container. In this case, my realization that I can essentially spin up multiple instances of a server without worrying about the physical allocation of hardware – making it possible to run experiments with K8’s. Or Jupyter Hub, which is the only way to get Jupyter Notebook on a phone, which can’t easily be containerized. Even Home Assistant, which runs on-host, despite being run with Docker.

Setting this up might seem like a lot of work, but I promise you – being able to manage machines from a single computer negates all of it. So thus begins the journey of migrating the system onto a higher plane of abstraction. But before I begin, there is one thing to take care of first – making backups. I’ve been burned twice before; I shall not get burned again. I have two copies of all volumes that I used previously in a separate location in case the machine decides to brick itself.

Setting up Janky Hardware

To start off, this is a refurbished HP Elite G3 800 with an i5-6500, sold by an e-waste company for around 70$. I am essentially building this server out of any components that I have lying around. I had an old NVME drive as the primary boot drive, 4TB HDD and 32GB of 2133MHz RAM. In total, the rough cost of the build comes out to ~300$, partially due to the cheaper flash prices due to the recent NAND overproduction.

Most of my hardware didn’t complain when it was going into the server, save the RAM. The BIOS for HP pre-built machines includes no overclocking or XMP profiles. Additionally, the other sticks were mismatched – which caused the computer to default to the lowest supported frequency, hence 2133MHz. One of the sticks was also causing errors, but cleaning the pins with some isopropyl and 10 seconds with a hair dryer fixed it.

Installing Proxmox and Configuring IPs

The next step was setting up the initial Proxmox install. Simple enough, flash a drive with Rufus… and the installer refuses to recognize my older USB drives, claiming that no disks were inserted, despite initially booting to the installer. After a bit of searching, I found that the installer would only detect the faster 3.0 drives. Strange, but not an issue.

There was also a moment where I noticed that the machine’s IP address was missing from the router; it looks as though the server IP that was not present in the router’s records after installation. I eventually manually assigned the server a static IP via the MAC.

Setting Up Storage and VMs

While the “correct” way of doing this probably involves some version of Ansible, I am only setting up three machines – a storage server, and two VMs for hosting containers.

For the storage server, this is a simple install of TrueNAS Core. I set up the VM with the disks passed though directly, installed TrueNAS, added the disks to a drive pool, created a dataset with the pool, added a user with read/write permissions, shared the dataset, and viola – networked storage, shared and accessible. This configuration might make it a bit of a hassle if I want to add additional storage, but I may as well set up a dedicated rackmount NAS if that were the case.

The other two VMs are running Ubuntu Server. After the initial install, I set up SSH keys and Docker. The only issue was mounting the networked storage – apparently, the proper way to mount a device is by editing the “/etc/fstab” file – the mount command does not persist between reboots.

To Cluster or Not?

All in all, I am happy with the progress that I’ve made with this server. The fact that tinkering this project has taken me so far makes me wonder what this will look like in another five years.  I know clustering seems to be the next step, but I don’t think high-availability is going to be going on my resume soon. I’ll feel the need to tinker eventually, but for now – this is done and done!

I’ve previously been using Ghost as the primary software for my website, although that has turned out to be a questionable decision. Software support was fine and the service was stable enough, but general experience of trying to get it to work nicely with Docker (and security updates) has made it a bit of a hassle to manage, especially if I wanted to do something else with my server.

In addition, trying to make the website a portfolio for what I had already done and accomplished proved harder than I expected. A lot of my academic work is not published in part due to academic-integrity agreements. Most everything that you see here is in fact my own work, done with minimal guidance and on my own time, which brings us to the primary reason I am switching over to a VPS.

Server infrastructure is fun to play around with, less fun to maintain when maintenance becomes complicated enough that hearing about a new vulnerability makes you worried that some state actor is going to breach my home network and read my search history. This new site is contained on a remote site, under how many layers of abstractions – a proxy, a firewall, another firewall, a hypervisor, another firewall, and then Docker, with regular snapshots. Short of the datacenter itself getting hit by a missile, it is slightly more secure. Either way, I’ve decided that it would probably be for the best to migrate to a VPS.

So, I have two tasks ahead of me:
• Migrating the data from Ghost to WordPress.
• Migrating the infra from my home to a remote server.

Migrating to the remote server itself was a painless operation. Provisioning the server and setting up the OS was a breeze – two clicks and done, which is a testament to how nice the datacenter itself was. The dashboard that managed the VPS was sluggish as hell for some reason – though the VPS itself was quick enough that it was faster to manage everything over CLI. I added my SSH keys and updated the server, set up users and permissions and everything was set. Nothing I hadn’t already done and I was in familiar territory.

Migrating Ghost to WordPress also quick. I knew there wasn’t any clean way to preserve the data that I had already wrote aside from simply copying the written text, so I simply dumped the contents of the website into several files and re-posted them onto this new site. As a bonus, Word has a much better inbuilt spellchecker than Firefox or Chrome and fixed several typos left out and about. Modifying the article dates wasn’t possible without at least some hassle, but would be possible to publish the articles into the past by modifying the database directly.

Unfortunately, I consider myself to be “good with technology”. I reasoned that it wouldn’t be that hard to roll my own self-hosted instance of WordPress and set up the required software for it. I’ve done it enough times with other sites that I was feeling confident that nothing would break too much and the site migration would be done by the end of the day. Psyche!

The site itself was being accessed directly from a public IP address and I needed to pass it through a reverse proxy if I ever wanted to run more than a single WordPress site. I set up the reverse proxy, edited the DNS records, and changed the site URL. My site immediately crashed.

As it turns out, there were three problems with the method that I had used so far.

1. I had wrongly assumed that I had access upon catastrophic failure.
2. DNS records rely on a working proxy manager and correct IP addresses.
3. SSL can’t be implemented halfway.

Addressing the first problem was just a matter of patience. Rather that pop a hard drive to look through the data that I could salvage, I could only access bits of the server at a time – and instead searched through the files to download the main “wp_posts.ibd” database. As a coincidence, this is about the time that I managed to edit the publication dates to match my prior articles. I hadn’t made many changes beyond moving some articles, so no loss.

The second problem was a slightly more involved process. I didn’t know what to look for, but I knew that the error originated when I tried to redirect from the proxy manager into the docker container (hint) – and the original site that resolved to the IP address was no longer resolving. The reverse proxy listens on ports 80 and 443, which means that any attempt to revisit the site from the IP: PORT resulted in a default page being shown. Accessing the site via IP meant that the site would redirect to the existing URL and then show the default page. Loading the site directly would show a “catastrophic error”. As it turns out, once the proxy manager took over the redirection, directing the page to the VPS IP was a mistake – it needed to be redirected to the internal Docker network.

The third problem was implementing SSL. WordPress can implement SSL on its own and does so, but there is no way of telling it to do so without editing the PHP files directly. Changing the URL to HTTPS within WordPress meant that the site was receiving an SSL Cert, but didn’t know what to do with it – and the site would not resolve. Instead, I opted to first configure SSL at each step – from user to Cloudflare, from Cloudflare to proxy, and finally proxy to site, testing each step. If you examine the little lock in your browser, you’ll see that everything works now.

Despite this crazy little adventure, I wish I migrated the server sooner. Over the course of the last year, I learned enough to begin to work on websites that are publicly accessible and deal with the ensuing security concerns and issues that arise from publicly facing servers. Sure, there are some kinks to work out, but there are other benefits. When on-prem is my-bedroom-closet, there are some things that you notice after a while.

I don’t need to deal with physical hardware or the process of shuffling around a server to deal with power and cooling. Reinstalling Ubuntu becomes a one click process. I managed to get two 9’s of reliability out of consumer hardware and no UPS – amazing for the equipment but simply not worth the effort, this VPS has three 9’s of reliability in their SLA. Load times decreased from ~2 seconds to ~0.5 seconds. No more server hum. Nice.