I’ve been doing a bit of housekeeping on the home k8s cluster, and one of the things I’m doing is moving from microk8s to k3s. This isn’t really a post about that, but long story short, it’s because of how microk8s does a non-existant job of updating addons, and you basically have to use the DNS (coreDNS) addon as I could never get that to work as a normal helm chart (even with updating the kubelet config).
Anyways as part of that change, I need to create a new cluster, get longhorn running, and restore the volumes it was running in the old cluster. Thankfully, I had tested most of this prior to becoming reliant on longhorn, so I knew the backup and restore process worked well – just point the backTarget variable for longhorn on the new cluster to the same place as the old cluster and magic happens. Unfortunately, I ran into a snag.
The volume restored properly, and I was able to recreate the PVC with the same name, but the deployment kept complaining about it and my Influx DB wouldn’t mount the drive. It kept throwing the error
Attach failed for volume : CSINode does not contain driver driver.longhorn.io
This was super odd though, I could create a new PVC with the same longhorn StorageClass and it would mount. WTF?!
Well, lo-and-behold it was because when I built the new cluster, I decided to use the newest version of longhorn – 1.4.1 – as you do. However, the old cluster was still on 1.4.0, as were the backups. During any upgrades of longhorn, you must do an engine upgrade to the volume. Needless to say, the backups were on engine 1.4.0 (driver), but I only had 1.4.1 (driver) running as I was never prompted to upgrade the engine on the volume when restoring it. So yes, the error message was factual, if not incredibly frustrating.
So, note to self (and others) – when restoring a longhorn volume from backup, make sure you are running the same version as from when the backup was taken. Once the volume is successfully restored and running, you can then upgrade to the latest version via the update steps, and update the engine on the volume. Sadly, there didn’t appear to be a way to do that after the restore, and tbh I didn’t look to see what version was listed as the Engine Image after the restore. I’m just thankful it’s back up and running!
**Update April-ish 2023** So at some point in the past, the printer began falling off the network again. It lasted many months, but something changed (unifi firmware, printer firmware, who knows). The good news though is that I was able to fix it again for the time being!
I was noticing it was falling off at night, which was odd. The only thing going on at night was…Nightly Channel Optimizations. Well, in the latest Unifi Controller (I’m on 7.3.83), you have the ability to exclude specific APs from doing those optimizations. As I had tied the printer to the single AP, I just added that to the exclusion and…tada it’s been on the network for the last 29 days again. Talk about frustrating – this is totally a Brother Wifi thing as no other device in the house has this problem.
And back to the original article…
Having your printer continual to fall off WiFi is the worst. Whenever you actually want to print something, lo-and-behold, you can’t, and you need to spend 2-20 mins fiddling with it to get it back on the network. While all mine took was a printer restart for it to magically reconnect to wifi, this is always how I felt.
After enough frustration, I finally took some time to sit down and fix the problem. After a bit of searching, I stumbled upon this Brother article (granted I’m printing from a Windows PC and my specific printer is a Brother MFC-L2750DW). That at least gave me some hope, as I was using a single SSID for both 5Ghz and 2.4Ghz – you know, like a sane person.
With the above article in hand, I created a new SSID that was only on the 2.4Ghz with the following settings (Unifi Controller 7.0.23):
Broadcasting APs: I have it set to just the 1 where closest to the printer
WiFi Band: 2.4Ghz
WiFi Type: Standard
Multicast Enhancement: ▢
Multicast and Broadcast Control: ▢
Client Device Isolation: ▢
Proxy ARP: ▢
BSS Transition: ▢
Fast Roaming: ▢
802.11 DTIM Period: Auto
Minimum Data Rate Control: Auto
Security Protocol: WPA2
Group Rekey Interval: ▣ 3600 seconds
Hide WiFi Name: ▣
Security: WPA Personal
WiFi Band: 2.4Ghz
Guest Policy: ▢
Broadcasting APs: I have it set to just the 1 where closest to the printer
Multicast and Broadcast Filtering: ▢
Fast Roaming: ▢
Hide SSID: ▣
Group Rekey Interval: GTK rekeying every 3600 seconds
Multicast Enhancement: ▢
RADIU DAS/DAC (CoA): ▢
Beacon Country: ▢
BSS Transition: ▢
TDLS Prohibit: ▢
Point to Point: ▢
P2P Cross Connect: ▢
Proxy ARP: ▢
L2 Isolation: ▢
Legacy Support: ▢
WPA Mode: WPA2 Only
DTIM Mode: Use Default Values
2G Data Rate Control: ▣
Disable CCK Rates: ▢
Also require clients to use rates at or above the specified value: ▢
Send beacons at 1Mbps: ▢
The printer has been online for over 20 days, whereas before it would fall off the network sometimes before it even fell asleep. 🎉🎉
We may have had an issue with a young “midnight surfer” on the internet one night, and it has since taken me a wild ride of VLANs, schedules, traffic shaping, RADIUS servers and SSIDs. I’ll give a bit of an abbreviated journey so you can relive the fun, but the important takeaway is how to do MAC-based port authentication on the switch while also doing it on the WLAN, and having both have the same fallback VLAN.
TL;DR – Having DEFAULT Accept auth-type that assigns a specific VLAN, works for WLAN clients on Unifi APs but does not work for MAC-based authentication on Unifi Switches. This is regardless of specifying a fallback network in the switch configuration or not. Instead, you should use the fallback network in the switch config and scope the Default user to only authenticate for devices on the APs via a huntgroup.
So, have your last user in the user’s config file (i.e. the fallback) look like the following:
Back to our “midnight surfer” – I woke up one night to some giggling to find my son had decided to use an old phone we have to watch tiktok videos. I knew this day would come but was just surprised it had come so fast/soon. Good thing I have all the technology required to lock this down!
My home networking consists of the following equipment:
Between the Qotom and the switch I have a 4-port link aggregation. Do I need 4Gbps between the router and the switch? Probably not, but I’m not using the ports anyways, and why not?! Additionally all the APs have a wired uplink to the switch.
Iteration 1 of the setup was to create 4 VLANs (Trusted, Guest, IoT, and Kids) and have them map to different SSIDs and manually specify the port VLAN on the switches – using a VLAN trunk for the wired APs and the link aggregation to the router. This setup was quick, easy, and worked! However, maintenance was a pain as I now had 3 new SSIDs that I needed to track the passwords for and getting devices onto the new network(s) – and any future devices – was a pain. Additionally, I use a wired connection for my work machine, but I also plug in my personal laptop to the same hub which connects to the same port. Yeah, I could use one of the USW-Flex-Minis and swap the connection the hub everytime, but let’s be honest – that’s annoying. Instead, I knew there had to be a better way.
Low and behold, there is – using a RADIUS server! Oh, and look at that, the incredibly powerful pfSense has a freeRADIUS package!
The initial configuration was pretty simple for wireless:
Add the network devices (switch & APs) as NAS clients with a shared secret (same for all of them)
Update the freeRADIUS EAP-TTLS and EAP-PEAP configuration to use tunneled reply and do not disable weak EAP types as that will cause the switch port MAC-based authentication to fail
Add a new RADIUS profile into the Unifi Controller that’s enabled for wired and wireless networks and specify the pfSense server as the auth server
Edit the wireless network to use RADIUS MAC Authentication. P.S., I highly recommend using the aa:bb:cc:dd:ee:ff format, because you can easily copy/paste from the device info in the Unifi Controller. Note that in the new UI (as shown) the wireless network will still have a Network defined. However, if you revert to the old UI, it will show “RADIUS assigned VLAN”.
Load up the list of users (i.e. the MAC addresses) in freeRADIUS – putting them on whatever VLAN you want (can also be blank!). Use the MAC address in the format you specified in step 3 as both the username and password are both the MAC.
Unfortunately, there is no fallback network/VLAN that you can define in the Unifi Controller for wireless networks. This is unfortunate and would’ve solved a lot of time later. However, you can define your own.
By default, if the user is not in the list, freeRADIUS will send a REJECT answer. However, we can enable a fallback user by setting the username and password as blank, specifying the fallback VLAN ID, adding “DEFAULT Auth-Type := Accept” to the top of this entry, and ensuring this client always the last user in the list as users are identified top-to-bottom.
After doing all that, I was able to move all my wireless clients back to the original SSID I had just moved them off of the previous weekend, and they still have the proper VLAN segregation. Woohoo!
Now, on to the switch ports – which was a multi-hour frustration, granted, it was late, and there was beer involved.
Assuming that you enabled wired networks on the radius profile, you should be able to visit the switch settings > services and enable 802.1X Control, select the previously created RADIUS profile and the Fallback VLAN (Network). If you’re using a default port profile (All), all ports will use the 802.1X Control of “force authorize” – aka doesn’t really do anything with the auth and so there will be no impact. You’ll want to verify the port settings prior to enabling 802.1X control to ensure you don’t lock yourself out prior to creating all the users in the RADIUS server.
Load up the list of users (i.e. the MAC addresses) in freeRADIUS – putting them on whatever VLAN you want (can also be blank!). The username and password are both the MAC address in the format of AABBCCDDEEFF.
In the old Unifi Controller UI you can override profiles and so you need to change the individual port(s) to use “MAC-based” 802.1X control. Otherwise, you can create a new port profile and assign it to the port(s) in question.
Assuming you’ve added users in the RADIUS server for every MAC address on the network, it’ll all just work! Unfortunately, any MAC addresses that are picked up by the DEFAULT rule added in earlier, will not authenticate on the Unifi switch. The RADIUS server correctly authenticates the unknown MAC address and responds with the correct VLAN (as seen in the freeRADIUS logs), but the response message doesn’t contain all the same info which is probably why the switch doesn’t accept it.
To fix the failback you need to scope the DEFAULT user config to only be for your wireless APs. Once that is done, unknown clients to the RADIUS server from the switch will fail authentication and then the switch will use the Fallback VLAN you configured earlier on the switch config.
If you only have one AP, you can edit your DEFAULT user config directly as seen in the code snipped below by replacing <IPAddress> with the IP address of your AP:
For more than 1 AP, it’s easier to create a huntgroup so you can reference all APs at once.
SSH into your pfSense box
Edit the /usr/local/etc/raddb/huntgroups file and create a new huntgroup as in the example, but with the IP Address(es) of your APs.
# huntgroups This file defines the `huntgroups' that you have. A
# huntgroup is defined by specifying the IP address of
# the NAS and possibly a port.
# Matching is done while RADIUS scans the user file; if it
# includes the selection criteria "Huntgroup-Name == XXX"
# the huntgroup is looked up in this file to see if it
# matches. There can be multiple definitions of the same
# huntgroup; the first one that matches will be used.
# This file can also be used to define restricted access
# to certain huntgroups. The second and following lines
# define the access restrictions (based on username and
# UNIX usergroup) for the huntgroup.
# Our POP in Alphen a/d Rijn has 3 terminal servers. Create a Huntgroup-Name
# called Alphen that matches on all three terminal servers.
#alphen NAS-IP-Address == 192.0.2.5
#alphen NAS-IP-Address == 192.0.2.6
#alphen NAS-IP-Address == 192.0.2.7
# My home configuration
<huntgroupName> NAS-IP-Address == <IPAddress1>
<huntgroupName> NAS-IP-Address == <IPAddress2>
<huntgroupName> NAS-IP-Address == <IPAddress3>
Update the DEFAULT user config directly as seen in the code snipped below by adding in the <huntgroupName> to scope the DEFAULT rule as shown below
And…TADA! Now your wireless and wired devices all get tagged with an appropriate or fallback VLAN!
UPDATE: Grrr, after a freeradius update, it seems to have overwritten the huntgroups file. That made it super fun to have a failback – would really nice if Unifi APs would have a fallback VLAN by default.
I’ve been fighting this for awhile (as have a few others based on some google searches), and now that I have it resolved I figured I’d post it here.
High level, I’ve had a Surface Ergonomic Keyboard for awhile, and absolutely love it. However, recently I upgraded from a Surface Pro 5 to a Surface Pro 7 and the keyboard keeps going to sleep – taking forever to wake back up. I’ve been on calls, just hammering the windows key to get it to wake up. Needless to say it’s been super annoying as waiting for 30 seconds or more for your keyboard to start responding again is not ideal for productivity (or sanity).
I’ve seen a few places that I just need to turn off the “allow the computer to turn off this device to save power”. However, it took me a bit to figure out which one. Turns out it’s not until you select Change settings that you can see the Power Management tab in device hardware. So without further ado…
Open Control Panel
Select View devices and Printers (or if your control panel lists all the icons, select Devices and Printers).
Select properties of the Ergonomic Keyboard and go to the Hardware tab
Select Bluetooth Low Energy GATT compliant HID device and select Properties
Click the Change settings button- tada Power Management tab!
Select the Power Management tab, unselect Allow the computer to turn off this device to save power and click the OK buttons until you are back at the devices and printers screen. Yay, now it doesn’t go to sleep!
If for some reason you still don’t see the Power Management tab, you can do the following actions:
Launch your Registry Editor (Windows button and type “Regedit“)
Navigate to: “Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power“
Select the entry (or Create a DWORD (32-bit) Value) called ‘CsEnabled‘
Change the “Value data” to “0” (Base ‘Hexadecimal‘) and select “OK“
I’ve been using OpenVPN for a few things and I’ve been very interested in setting up WireGuard instead as it has a lot less overhead and is less cumbersome than OpenVPN. Well I finally took the plunge last night and it was surprisingly easy after only a few missteps!
One of my use cases is to tunnel all traffic to the VPN server, so it appears as if my internet traffic originates from the VPN server. Here is how I set it up (with thanks to a few otherarticles).
On the Server (Ubuntu 18.04 LTS)
Install WireGuard on the server. I am running Ubuntu 18.04 and so I had to add the repository.
Move to the /etc/wireguard directory (you may need to sudo su)
Generate the public and private keys by running the following commands. This will create two files (privatekey and publickey) in the /etc/wireguard so you can re-reference them while building out the config.
$ umask 077 # This makes sure credentials don't leak in a race condition. $ wg genkey | tee privatekey | wg pubkey > publickey
Create the server config file (/etc/wireguard/wg0.conf). Things to note:
The IP space used is specifically reserved for a shared address space per RFC6598
I only care about IPv4. It is possible to add IPv6 address and routing capabilities into the configuration
For routing, my server’s local interface name is eth0.
You can choose any port number for ListenPort, but note that it is UDP.
Add as many peer sections as you have clients.
Use the key in the privatekey file in place of <Server Private Key>. Wireguard doesn’t support file references at this time.
We haven’t generated the Client public keys yet, so those will be blank.
Well, I’m doing it (migrating my CrashPlan account – see previous post with updates)! This is primarily because I get the feeling the discount will disappear at the end of the month when they officially stop supporting home. For those that haven’t gone through the steps, just taking screenshots as an FYI. Additionally check out the other post as to how I’m managing non-NAS backups.
You get to pick which devices you want to migrate. It will tell you very plainly how much and when your billing changes. Depending on how many devices you pick, the number changes. As mentioned before, I’m keeping my NAS backups, and that’s it.
You update and add your info.
It re-iterates your price.
You agree to a bunch of stuff that they’ve already called out before.
You enter your CC info and agree to auto-bill
All done! (my client will be updated in the background…and on my device I didn’t migrate it updated as I was writing this)
The UI when you log into your account (same user/pass) is now way different/better than the home one. Plus I get some of my storage back on my NAS due to it deleting computer-to-computer backups.
Boo, just got the email today that CrashPlan is leaving the home market. After I don’t know how many years, it looks like I’ll have to find another provider. It looks like there are a few, but with no computer-to-computer options baked in all will be a step back. *sigh*
I’ve been following a lot of different threads on this. Sadly, there are no direct competitors. Turns out CrashPlan (even with the crappy Java app) was the best for a lot of reasons including the following:
Unlimited – I am not a super heavy user with ~1TB of total storage spanning back for the last 10 years of use/versions, but it’s always nice to know it’s there.
Unlimited versions – This is key and has saved my bacon a few times after a migration (computer/drive/other backup to NAS) and you think you have everything, but turns out you don’t until a year later when you’re looking for it.
Family plan (i.e. more than one computer) – nice as I have 3 machines, plus my NAS that I can
Peer-to-peer – one backup solution to rule them all that works on remote networks. Unfortunately, it uses gross ports so doesn’t work anywhere (like in corporate places) and you can’t shove peer-to-peer backups to the cloud, those peers have to upload it directly.
Ability to not backup on specific networks…like when I’m tethered to my phone.
Total sidebar, but speaking of crappy Java apps, I had just migrated to using a docker image of CrashPlan too due the continued pain of updating it with Patter’s awesome SPK. Yay to running everything in docker now instead of native Synology apps.
My current setup consists of 3 Windows machines and a Synology NAS. I had the CrashPlan family account so each of those machines would sync to the cloud, and all the windows machines would sync to the NAS. Nothing crazy, and yes, I know I was missing a 3rd location for NAS storage for those following the 3-2-1 method.
The other cloud options I’ve looked at so far:
Carbonite – no linux client, so non-starter as that’s where I’d like to centralize my data. I used to use them before CrashPlan and wasn’t a fan. I know things change in 10 years, but…
iDrive – Linux client (!) and multiple hosts, but only allows 32 versions, and dedupe seems to be missing so I’m not sure what that would mean for my ~1TB of data. They have a 2TB plan for super cheap right now ($7 for the first year), which could fill all my needs.
So where does that leave me? I’m hopefully optimistic about companies getting more feature parity, and thankfully my subscription doesn’t expire until July of 2018. Therefore, while I’m doing some work, I’m firmly in the “wait and see” camp at this point. However, if I were to move right now, this is what my setup would look like:
Install Synology Cloud Station Backup and configure the 3 Windows systems to backup to the Synology NAS. Similar to CrashPlan, I can uPNP a port through the Firewall for external connectivity (I can even use 443 if I really want/need to). This is my peer-to-peer backup and is basically like-for-like with Crashplan peer-to-peer. This stores up to 32 versions of files, which while not ideal, is ok considering…
Upgrade to CrashPlan Small Business on the NAS. While I’m not thrilled about the way this was handled, I understand it (especially seeing the “OMG I HAVE 30TB IN PERSONAL CRASHPLAN” redditor posts) and that means I don’t have to reupload anything. Send both the Cloud Station Backups and other NAS data to CrashPlan. This gets me the unlimited versions, plus I have 3-2-1 protections for my laptops/desktops.
Use Synology Cloud Sync (not a backup) or CloudBerry to B2 for anything I deem needs that extra offsite location for the NAS. This would be an improvement to my current setup, and I could be more selective about what goes there to keep costs way down.
Hopefully this helps others, and I’ll keep updating this post based on what I see/move towards. Feel free to add your ideas into the comments too.
Just saw this announcement from MSFT. Could be an interesting archival strategy if tools start to utilize it – https://azure.microsoft.com/en-us/blog/announcing-the-public-preview-of-azure-archive-blob-storage-and-blob-level-tiering/
A quick update on some things that have changed. I’ve moved away from Comcast, and now have Fiber! That means, no more caps (and 1Gbps speeds), so I’m more confident to go with my ideas above. So far this is what I’ve done:
Setup Synology Cloud Backup. To ensure I get the best coverage everywhere, I’ve created a new domain name and have mapped 443 externally to the internal synology software’s port. When setting it up in the client, you need to specify <domain>:443, otherwise it attempts to use the default port (it even works with 2FA). CPU utilization isn’t great on the client software, but that’s primarily because the filtering criteria is great (if you just add your Windows user folder, all the temp internet files and caches constantly get uploaded). It would be nice if you could filter file paths too, similar to how CrashPlan does it – https://support.code42.com/CrashPlan/4/Troubleshooting/What_is_not_backing_up (duplicating below in case that ever goes away). I’ll probably file a ticket about that and increasing the version limit…just because.
I still have CrashPlan Home installed on most of my computers at this point as I migrate, but now that I know Synology backup works, I’ll start decommissioning it (yay to lots of java-stolen memory back!).
I’ve played around with a cloudberry docker, but I’m not impressed. I still want to find something else for my NAS stuff to maintain 3 copies (it would be <50GB of stuff). Any ideas?
CrashPlan’s Windows Exclusions – based on Java Regex
This was from when RemoteApp didn’t support creating an image directly from VM.
A1 Std machine, copying a 127GB VHD to a local drive (not temp D:\) via azcopy took 6.5 hours
A4 Std machine, copying a 127GB VHD to D:\ via azcopy took 5 mins 20 secs
A4 Std machine, copying a 127GB VHD to D:\ via save-azurevhd took 10 mins 39 secs
A4 Std machine, copying a 127GB VHD to a local drive (not Temp) via azcopy took 25 mins 21 seconds
A4 Std machine, copying a 127GB VHD to a local drive (not Temp) via save-azurevhd took 52 mins 11 seconds
Copying files into a VM via the two commands is very CPU intensive due to the threading it uses, so utilize a larger box no matter your method. And the hands down winner is to use Azcopy into the local temp D:\ (avoids an extra storage account hop). However, if you want a status bar, utilize save-azurevhd.
Copying VHDs between Storage Accounts
Due to a storage cluster issue in AU East, it has been advised to create new storage accounts and migrate VHDs to the new storage accounts. MSFT had provided us with a script, but it was taking hours/days to copy (and kept timing out).
Instead, we spun up a D4v2 machine in the AU East region, and I was able to have 6 azcopy sessions happening all at once with the /SyncCopy command. Each was running >100MB/sec whereas other async methods were running at <5MB/sec. You will see a ton of CPU utilzation during this, but the faster the machine, the better. Additionally, azcopy supports resume. To allow multiple instances of azcopy to run on a machine, utilize the /Z:<folderpath> switch for the journal file.
Stop Azure Blob with Copy Pending
Prior to getting all our copies going with the /SyncCopy, we had a few that were running async. Unfortunately, after stopping that with a CTRL-C and having azcopy stop, the blobs still had a copy pending action on them. This resulted in errors when attempting to re-run the copy with /SyncCopy on a separate machine: HTTP error 409, copy pending.
To fix this, you can force stop the copy. As these were new storage accounts with only these VHDs, we were able to run it against the full container. However, MSFT has an article on how you can do it against individual blobs.