We are utilizing SQL Backup to Azure blob and had a meltdown today where the log backups were erroring out leaving us with 1TB files up in Azure that were locked. Needless to say it happened late last night and so there were multiple hourly files in multiple folder structures all over our storage accounts. It took a bit, but the following script clears out all the locks on blobs within a container in all directories. Please use carefully and don’t run it against your “vhds” container!
Also, it requires the Microsoft.WindowsAzure.Storage.dll assembly from the Windows Azure Storage NuGet package. You can grab this by downloading the commandline nuget file and running the below. Note, it will dump the file you need into .\WindowsAzure.Storage.<ver>\lib\net40\
nuget.exe install WindowsAzure.Storage
Break lease Script Below – one line modification from https://msdn.microsoft.com/en-us/library/jj919145.aspx
$storageAssemblyPath = $pwd.Path + "\Microsoft.WindowsAzure.Storage.dll"
# Well known Restore Lease ID
$restoreLeaseId = "BAC2BAC2BAC2BAC2BAC2BAC2BAC2BAC2"
# Load the storage assembly without locking the file for the duration of the PowerShell session
$bytes = [System.IO.File]::ReadAllBytes($storageAssemblyPath)
$cred = New-Object 'Microsoft.WindowsAzure.Storage.Auth.StorageCredentials' $storageAccount, $storageKey
$client = New-Object 'Microsoft.WindowsAzure.Storage.Blob.CloudBlobClient' "https://$storageAccount.blob.core.windows.net", $cred
$container = $client.GetContainerReference($blobContainer)
#list all the blobs in the container including subdirectories
$allBlobs = $container.ListBlobs($null,1)
$lockedBlobs = @()
# filter blobs that are have Lease Status as "locked"
foreach($blob in $allBlobs)
$blobProperties = $blob.Properties
if($blobProperties.LeaseStatus -eq "Locked")
$lockedBlobs += $blob
if ($lockedBlobs.Count -eq 0)
Write-Host " There are no blobs with locked lease status"
if($lockedBlobs.Count -gt 0)
write-host "Breaking leases"
foreach($blob in $lockedBlobs )
$blob.AcquireLease($null, $restoreLeaseId, $null, $null, $null)
Write-Host "The lease on $($blob.Uri) is a restore lease"
if($_.Exception.RequestInformation.HttpStatusCode -eq 409)
Write-Host "The lease on $($blob.Uri) is not a restore lease"
Write-Host "Breaking lease on $($blob.Uri)"
$blob.BreakLease($(New-TimeSpan), $null, $null, $null) | Out-Null
To get the optimal performance out of your Azure VMs running SQL servers, MS recommends to use Storage Spaces and stripe multiple Azure disks. The nice thing about storage pools in Storage Spaces is that it allows you to add disks behind the scenes without impacting the actual volume.
Now lets say you have a SQL AlwaysOn cluster (2+ nodes), and for performance reasons (IOPS) you realize that you need to add more disks. As Storage Spaces shows all disks (physical, virtual, and storagepools) across the whole cluster, it is possible you won’t be able to simply add them due to naming mismatch. Fear not though, it is still possible if you follow the steps below:
- Add the new disks to the VM
- Log into the VM
- Failover SQL to a secondary if the current VM is the primary
- Stop clustering service on the VM
- Run Get-PhysicalDisks to get the disknames
- Run Add-PhysicalDisk -StoragePoolFriendlyName <storagepool> -PhysicalDisks (Get-PhysicalDisk -FriendlyName <disks>)
- Run Update-HostStorageCache (if we don’t do this sometimes the volume resize doesn’t work)
- Run Resize-VirtualDisk -FriendlyName <diskName> -Size <size>
- Run Update-HostStorageCache (if we don’t do this sometimes the disk resize doesn’t work)
- Run Resize-Partition -Size <size> -DriveLetter <letter>
- Start the clustering service on the machine
- Failback SQL to the VM if required
Hopefully this helps someone as we were beating our heads in for quite a few days (along with MS).
We had an issue recently where an application was not properly getting disconnected from SQL during a failover of an AlwaysOn Availability Group (AOAG). Some background: The application was accessing the primary node, and after the failover the application continued to access the same node. Unfortunately, as it was now read-only, the app was not very happy.
Turns out it was due to the Read-Only configuration of the secondary. We had it set to “Yes” which allows any connections to continue to access the secondary with the assumption the application is smart enough to know it can only read. It appears while using this setting, connections aren’t forcefully closed, causing all sorts of issue.
Setting it to either “No” or “Read-Intent Only” properly severed the connections for us. Yay!
For more info.
Looking to update your Azure ILB endpoints, but are struggling with the Set-AzureEndpoint cmdlet? You should be using the Set-AzureLoadBalancedEndpoint cmdlet instead!
One of the bigger things I see as a complaint about the Surface Pro 3 (SP3) is that by default the Function keys are not the primary button press. Instead the defaults are the shortcut keys. Additionally, it’s not really documented anywhere how to switch what is primary.
Well, you can. Just use CAPS + Fn to switch between what you want to be primary.
Unfortunately, I use a lot of the Fn keys, but also Home and End. Oh well, guess I can’t have my cake and eat it too.
Another one of the great things about using a Synology is having access to all the great packages that both Synology and the community create. You find out about so many different apps you may not have been familiar with. Turns out Bliss was one of those (grab the app from Patters).
I have a fairly large music library I’ve composed over the years, and while I was picky about folder structure, all the other stuff was lacking: tags, images, etc. Back in July of 2012 I gave it a first run and liked what it did. However, I didn’t keep it installed (didn’t think I would need it continually).
Fast forward to last week, and I reinstalled it to help clean things up again – I had to restore lots of my music which meant the tags, folder structure, etc. were all messed up again. I’m loving it all over again for it’s ease of use and continual running in the background.
Turns out in the current release they’ve added some new case rules which I love, but was having a few issues with it. Shot of an email with some of the details, and got an immediate response saying it was being fixed in the next release that was in beta. I asked if they wanted any help with testing, and sure enough Dan (creator) gave me links to the beta which I promptly installed (replaced the bliss-bundle folder after you extract the Linux jar file).
Can’t say enough about the customer service, plus it’s fixing all my music issues in the background. Yay!
One of the challenges with running my own server now, is making sure everything is properly patched. With the big stink around the new Heartbleed vulnerability, it made me look into it a bit.
Turns out, in order to fix you can run the following:
sudo apt-get update
sudo apt-get install -y libssl1.0.0 openssl
# Confirm Build Date is at least Aril 7th 2014
openssl version -a
# Restart all services listed by this command:
sudo lsof -n | grep ssl | grep DEL
With the move of my domains to Azure instead of Dreamhost (oh, did I not mention that?), one of the things I needed to do was setup some SPF records to keep email from going into junk folders.
Now, there is a great article on creating SPF records, but those that know me know I’m not a big reader when there is a lot of text. Pictures FTW!
Anyways, as MS is a big proponent of utilizing SPF, they have created an excellent tool to create the record’s text. Yay team!
I’ve finally migrated my email from google apps to outlook.com. The key item that caused some issues was using domains.live.com. I had added the MX record that proved I owned it (and email was flowing correctly).
However, it was always set to “pending” and never switched to “active”. The problem is that the Hotmail.com MX record is the only MX record you can have. I had some of the old gmail records listed still, but at a much lower priority (higher number). Removing them immediately made it switch to active.