Aluminum Owls & Silver Mice

Author: andrewc

  • Hyper Converged Infrastructure with Proxmox

    After Broadcom aquired VMware, I was once again introduced to a name that had already been floating around the home-lab circles: Proxmox. At the time I still had no idea what it even was, but as my little experiment began to grow and take a life of its own, I began to find myself searching for a more powerful solution than just containerization – virtual machines. I have previously avoided them due to the need for hardware (that was in short supply until recently, mind you). But now that my need for a more flexible setup emerges as the workload increases, this seems to be about the right time to get started.

    Virtual machines are wonderful on a conceptual level, but there are some kinks that need to be addressed in the real world. Where is a virtual machine? How does compute get allocated? What software needs a dedicated VM in the first place? Spinning up Virtual Box for a single machine to test things is great, but what if I want to network with multiple VMs?

    It turns out the answer is a hypervisor, a scheduler, and a hundred little goblins in running around with tiny hammers. The hypervisor with a dedicated OS can split its own resources into many different virtual machines that each enjoy its own “isolated” environment for the goblins to run around in. This isn’t a replacement for containers, this is a way to supplement them with software that can’t easily be run on a container. In this case, my realization that I can essentially spin up multiple instances of a server without worrying about the physical allocation of hardware – making it possible to run experiments with K8’s. Or Jupyter Hub, which is the only way to get Jupyter Notebook on a phone, which can’t easily be containerized. Even Home Assistant, which runs on-host, despite being run with Docker.

    Setting this up might seem like a lot of work, but I promise you – being able to manage machines from a single computer negates all of it. So thus begins the journey of migrating the system onto a higher plane of abstraction. But before I begin, there is one thing to take care of first – making backups. I’ve been burned twice before; I shall not get burned again. I have two copies of all volumes that I used previously in a separate location in case the machine decides to brick itself.

    Setting up Janky Hardware

    To start off, this is a refurbished HP Elite G3 800 with an i5-6500, sold by an e-waste company for around 70$. I am essentially building this server out of any components that I have lying around. I had an old NVME drive as the primary boot drive, 4TB HDD and 32GB of 2133MHz RAM. In total, the rough cost of the build comes out to ~300$, partially due to the cheaper flash prices due to the recent NAND overproduction.

    Most of my hardware didn’t complain when it was going into the server, save the RAM. The BIOS for HP pre-built machines includes no overclocking or XMP profiles. Additionally, the other sticks were mismatched – which caused the computer to default to the lowest supported frequency, hence 2133MHz. One of the sticks was also causing errors, but cleaning the pins with some isopropyl and 10 seconds with a hair dryer fixed it.

    Installing Proxmox and Configuring IPs

    The next step was setting up the initial Proxmox install. Simple enough, flash a drive with Rufus… and the installer refuses to recognize my older USB drives, claiming that no disks were inserted, despite initially booting to the installer. After a bit of searching, I found that the installer would only detect the faster 3.0 drives. Strange, but not an issue.

    There was also a moment where I noticed that the machine’s IP address was missing from the router; it looks as though the server IP that was not present in the router’s records after installation. I eventually manually assigned the server a static IP via the MAC.

    Setting Up Storage and VMs

    While the “correct” way of doing this probably involves some version of Ansible, I am only setting up three machines – a storage server, and two VMs for hosting containers.

    For the storage server, this is a simple install of TrueNAS Core. I set up the VM with the disks passed though directly, installed TrueNAS, added the disks to a drive pool, created a dataset with the pool, added a user with read/write permissions, shared the dataset, and viola – networked storage, shared and accessible. This configuration might make it a bit of a hassle if I want to add additional storage, but I may as well set up a dedicated rackmount NAS if that were the case.

    The other two VMs are running Ubuntu Server. After the initial install, I set up SSH keys and Docker. The only issue was mounting the networked storage – apparently, the proper way to mount a device is by editing the “/etc/fstab” file – the mount command does not persist between reboots.

    To Cluster or Not?

    All in all, I am happy with the progress that I’ve made with this server. The fact that tinkering this project has taken me so far makes me wonder what this will look like in another five years.  I know clustering seems to be the next step, but I don’t think high-availability is going to be going on my resume soon. I’ll feel the need to tinker eventually, but for now – this is done and done!

  • Fallout: Nostalgia for the Future

    Fallout is something near and dear to my heart, with the original isometric games playing on a small netbook, where the Atom processor would heat up the table if there were more than 4 enemies on the screen at. It sits right next to the memories of “I Love Lucy” re-runs and that hefty Windows 95 laptop. I watched it change from isometric to 3D, as land-lines became a smart phone. I watched as the world around me changed, yet the rate that people can adopt new technology and make aesthetically identifiable technology seems to have stagnated.

    This is a show about many things, but it is also a show about a society and a culture that has essentially been permanently rooted in the past. Fallout is the future, as imagined when the cold war turned hot, if we never miniaturized the transistor, if everything were wiped clean in the under glow of atomic fury. What will our technology look like if a pandemic were to decimate the world and our own culture were left to marinate in its own juices for 200 years? Are cat memes a currency? I like to imagine you’d have a stash to trade, just in case.

    The show follows in the tradition of finding things and relatives, finding your son, finding your father, finding a water chip, finding your father with the water chip, finding who shot you in the head, finding the chip on the person that shot you in the head, finding someone’s head, finding a chip in someone’s head. I wouldn’t be surprised if Vin Diesel was in the last scene to talk about family, fries (this was lost in translation) and then shoot someone, in the head.

    Each scene feels like a nod to parts of the game; it is fantastic. Someone outright says “It really is the most Fallout: New Vegas”, though I also liked the more subtle use of the general props and weapons that that help build the worlds iconic style. Watching someone get shot by a high velocity trash cannon is peak cinema. The world is harsh and death comes easily to the people on the surface, but through it all is the moments of utter mayhem and absurdity that characterize the franchise. Gore seems excessive but apt, given that the action serves as a reminder of a genuine apocalypse, despite the bright colored clothing and whimsy of junk towns that are slightly too clean.

    Unfortunately, the show also diverges from the traditional game’s story in a rather unsubtle manner. One of the primary gameplay drivers, in tandem with the rest of the story, is aligning with factions to make the wasteland a better place, purifying water, establishing settlements, helping people, working with or against various governments to some end. Fallout is a game series is about rebuilding and watching communities grow, often as an incidental bystander, sometimes as a major force. It stings when one of the only locations that persists throughout the game series and grows to become a burgeoning civilization gets blown up by a new faction. I suppose that this makes sense, the game saying that “war never changes” also requires that the wasteland itself stay a wasteland, frozen in time for 200 years. While it means little as a minor city in the entirety of the United States, this is a location that I remember since the first Fallout game.

    I suppose this is also part of the nature of Fallout. The game series itself can’t exist as a “wasteland frontier” forever no matter how impossible the task of remaking civilization is. The show ends with a McGuffin finally realized – and the possibility of change. Of course, all my critique is wiped clean by the fact that there might be a sequel, so 8/10, nuke Shady Sands again.

  • Hello World #2: Migrating the Website

    I’ve previously been using Ghost as the primary software for my website, although that has turned out to be a questionable decision. Software support was fine and the service was stable enough, but general experience of trying to get it to work nicely with Docker (and security updates) has made it a bit of a hassle to manage, especially if I wanted to do something else with my server.


    In addition, trying to make the website a portfolio for what I had already done and accomplished proved harder than I expected. A lot of my academic work is not published in part due to academic-integrity agreements. Most everything that you see here is in fact my own work, done with minimal guidance and on my own time, which brings us to the primary reason I am switching over to a VPS.


    Server infrastructure is fun to play around with, less fun to maintain when maintenance becomes complicated enough that hearing about a new vulnerability makes you worried that some state actor is going to breach my home network and read my search history. This new site is contained on a remote site, under how many layers of abstractions – a proxy, a firewall, another firewall, a hypervisor, another firewall, and then Docker, with regular snapshots. Short of the datacenter itself getting hit by a missile, it is slightly more secure. Either way, I’ve decided that it would probably be for the best to migrate to a VPS.


    So, I have two tasks ahead of me:
    • Migrating the data from Ghost to WordPress.
    • Migrating the infra from my home to a remote server.


    Migrating to the remote server itself was a painless operation. Provisioning the server and setting up the OS was a breeze – two clicks and done, which is a testament to how nice the datacenter itself was. The dashboard that managed the VPS was sluggish as hell for some reason – though the VPS itself was quick enough that it was faster to manage everything over CLI. I added my SSH keys and updated the server, set up users and permissions and everything was set. Nothing I hadn’t already done and I was in familiar territory.


    Migrating Ghost to WordPress also quick. I knew there wasn’t any clean way to preserve the data that I had already wrote aside from simply copying the written text, so I simply dumped the contents of the website into several files and re-posted them onto this new site. As a bonus, Word has a much better inbuilt spellchecker than Firefox or Chrome and fixed several typos left out and about. Modifying the article dates wasn’t possible without at least some hassle, but would be possible to publish the articles into the past by modifying the database directly.


    Unfortunately, I consider myself to be “good with technology”. I reasoned that it wouldn’t be that hard to roll my own self-hosted instance of WordPress and set up the required software for it. I’ve done it enough times with other sites that I was feeling confident that nothing would break too much and the site migration would be done by the end of the day. Psyche!


    The site itself was being accessed directly from a public IP address and I needed to pass it through a reverse proxy if I ever wanted to run more than a single WordPress site. I set up the reverse proxy, edited the DNS records, and changed the site URL. My site immediately crashed.


    As it turns out, there were three problems with the method that I had used so far.

    1. I had wrongly assumed that I had access upon catastrophic failure.
    2. DNS records rely on a working proxy manager and correct IP addresses.
    3. SSL can’t be implemented halfway.

    Addressing the first problem was just a matter of patience. Rather that pop a hard drive to look through the data that I could salvage, I could only access bits of the server at a time – and instead searched through the files to download the main “wp_posts.ibd” database. As a coincidence, this is about the time that I managed to edit the publication dates to match my prior articles. I hadn’t made many changes beyond moving some articles, so no loss.

    The second problem was a slightly more involved process. I didn’t know what to look for, but I knew that the error originated when I tried to redirect from the proxy manager into the docker container (hint) – and the original site that resolved to the IP address was no longer resolving. The reverse proxy listens on ports 80 and 443, which means that any attempt to revisit the site from the IP: PORT resulted in a default page being shown. Accessing the site via IP meant that the site would redirect to the existing URL and then show the default page. Loading the site directly would show a “catastrophic error”. As it turns out, once the proxy manager took over the redirection, directing the page to the VPS IP was a mistake – it needed to be redirected to the internal Docker network.

    The third problem was implementing SSL. WordPress can implement SSL on its own and does so, but there is no way of telling it to do so without editing the PHP files directly. Changing the URL to HTTPS within WordPress meant that the site was receiving an SSL Cert, but didn’t know what to do with it – and the site would not resolve. Instead, I opted to first configure SSL at each step – from user to Cloudflare, from Cloudflare to proxy, and finally proxy to site, testing each step. If you examine the little lock in your browser, you’ll see that everything works now.

    Despite this crazy little adventure, I wish I migrated the server sooner. Over the course of the last year, I learned enough to begin to work on websites that are publicly accessible and deal with the ensuing security concerns and issues that arise from publicly facing servers. Sure, there are some kinks to work out, but there are other benefits. When on-prem is my-bedroom-closet, there are some things that you notice after a while.

    I don’t need to deal with physical hardware or the process of shuffling around a server to deal with power and cooling. Reinstalling Ubuntu becomes a one click process. I managed to get two 9’s of reliability out of consumer hardware and no UPS – amazing for the equipment but simply not worth the effort, this VPS has three 9’s of reliability in their SLA. Load times decreased from ~2 seconds to ~0.5 seconds. No more server hum. Nice.

  • It’s the New Year!

    I haven’t been keeping up with regular posts recently, but here is a update on what I’ve been doing recently and what is going to happen moving forward.

    A Change in Mentality…

    Last year was a doozy, and after 2020 I’m quite sure everyone has lost the ball when it comes to time. Remote work, upcoming elections, another war, and so many things have happened in the world at large that it becomes hard to focus on the here and now – yet we endeavor to do so anyway.

    2023 has not gone the way that I planned for. I expected that I would be able to finish Advent of Code this year, as well as participate in more hackathons, complete all those projects that I’ve started, and yet in such a hurry to complete so many things, I neglected stop wallowing in the details. I speak of regrets, yet they all pale in comparison to what I have actually learned.

    I started this year without knowing what to do with my time and energy, spending time solving small self-contained problems with well defined, technical, algorithmic solutions. Despite knowing so much, I found it hard to apply my skill set to problems that lay outside of my realm of expertise. Through the work that I have done this year, I find myself applying a new paradigm: Learn as you go. There is a time for learning and preparation, but the tools to learn are always applicable, and you need not know everything before you begin.

    No unseen problem can be sufficiently well contained that you can know everything about it. A younger me might have tried to do so much, just know everything about unknown problems, real world experience has taught me that the endeavor is futile. A unseen problem is a monolithic one, insurmountable and unassailable. It looms in the distance and mocks your ability as a piddling thing. Step up close, and it can only ever be a large problem.

    A Status Update!

    After the massive whirlwind of the December months have passed us by, there are some basic projects that I have been working on that I would like to finalize and release. There are several projects I am currently working on, starting with…

    ESP32-C6 Sensor Kit

    This project is really for myself, albeit for a simple reason. I want the ability to monitor various readings throughout my house, and there exists no all in one IoT solution that suits fit what I wanted – monitor relative humidity, temperature, light, CO2, and then send it back a local dashboard. While I can do this with most small platforms, I settled with the ESP32 series because of the built in connectivity.

    RiverReeds

    This project comes at the behest of my parents, who have wanted audio book read-along technology. To my knowledge, there are several projects that have attempted the same thing, but they are long abandoned, and the few that do exist are not open source. Such technology was present in earlier versions of the Kindle, but they were computer-readers using to Speech to Text rather than a true audio book.

    Firewood

    What do you do with an old Kindle that you don’t use anymore? Turn it into a low power status page. Even if everything is available at your fingertips now, sometimes you don’t want to click through an app just to see the time. Why not with the weather or stock tickers or server uptime? The Kindle has a display that you can turn into a status page for just about anything.

    —

    These are all well and good, and I aim to finalize these before the start of Spring in 2024. With great ambition comes a risk of failure, but even in failure we learn and grow.

    What Comes Next?

    There are some projects that I would like to get started, the first of which is a weather analysis project – specifically, modeling temperature as a function of light and humidity. The second project is a method of generating meshes for moiré patterns.

    Other than that, who knows? We can make predictions… and resolutions!

    1. Get a Six-Pack!
      1. Despite the ambition, I did not set this goal with the intent to ‘fail’. I have prior resolutions with ‘being fit’ but it is too amorphous to be considered a goal. Exercising in the name of vanity is much easier to justify.
    2. Get a Job in Software or Analytics!
      1. Despite my expertise as a dishwasher and line cook, white collar jobs pay significantly more and tend to be less physically demanding.
    3. Get a Calendar (and plan more)!
      1. This is self explanatory – but I’ve found my organization habits to be rather haphazard. “Winging-it” doesn’t work when you need to juggle four or five priorities at the same time.
    4. Everything and the Moon!
      1. Consistent sleep schedule
      2. Migrate critical sites to a VPS
      3. Clean the fridge (consistently)
      4. Build an IKEA cabinet
      5. Post more

    I’m feeling great and looking forward to 2024!