Clustering in Windows 2016 for beginners.
Up until Windows 2016 clustering, things were going smoothly on my studying and preparing for this exam, but clustering is by far the most challenging section of the exam so far.
For those Microsoft techs and engineers that have been using the technology on a daily basis, it isn’t hard to comprehend disk witnesses and quorums, but for those admins that only see these terms in Microsoft exams, it takes a while to comprehend.
Luckily my lab has been able to simulate a cluster of computers and disks. Microsoft has created an awesome product that allows for anybody to be able to simulate a full blown cluster with several machines and several disk to fail-over.
Here are some important snippets of Powershell and notes taken away from studying for this portion of the exam (probably repeated hundreds of times already, but that’s the Internet will live in).
Getting started.
My lab was created on a Windows 10 Pro box with 16GBs of memory. The hard drives have a conservative single SATA 250GB drive, maybe consider a faster newer model running 10K RPM?
I downloaded the latest Windows 2016 Server ISO and created 4 machines, each machine is connected to an Internal Switch on the VM (Internet access is not requried) I used the subnet 192.168.10.0/24:
After created the DC, I ran the following from a PSSession on each server. Familiarize or force yourself to use PS for setting everything, because it can be done and because it’s superior.
add-netipaddress 192.168.10.xx -prefixlength 24; set-netipaddress 192.168.10.xx -interfacealias ethernet (ethernet is the default for a new VM); set-dnsclientserveraddress -serveraddresses 192.168.10.10 (DC) ; add-computer -domainname windows.com -credentials windows.com\administrator
VM-Name /CN
2016-lab1/server1 – Windows 2016 Server Core with ADSI (192.168.10.10)
2016-lab2/server2 – Windows 2016 Server Core Node 1 (192.168.10.20)
2016-lab3/server3 – Windows 2016 Server w/ Desktop Experience Node 2 (192.168.10.30)
2016-lab4/server4 – Windows 2016 Server w/ Desktop Experience ISCSI Target Server (192.168.10.40)
2016-lab3 –
install-windowsfeature Failing-Clustering ; install-windowsfeature FS-FileServer
2016-lab2 – Created the cluster from the node:
install-windowsfeature Failing-Clustering ; install-windowsfeature FS-FileServer
new-cluster -name cluster1 -node server2, server3 -staticaddress 192.168.10.100
(shared address to access resources, if you leave out -staticaddress, the command will return an error), another way to create your cluster, and the way I created my cluster without the CNO is:
new-cluster –name cluster1 -node server1,server2 -administrativeaccesspoint dns -nostorage
Creating the cluster of disks using CSVFS from the VHDX that are attached start with this magic:
Enable-ClusterS2D -CacheState Disabled -AutoConfig:0 -SkipEligibilityChecks -Confirm:$false
Making sure to SkipEligibilityChecks is critical.
Once you have the disk you can create the volume from the pool. The volume with add a new directory to the C:\ClusterStoragePool and create a virtual disk at the same time.
New-Volume -FriendlyName "Parity-1-Vol1" -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2DPool -Size 1GB -ResiliencySettingName Parity
(This method is not recommended, but still allows you to administrator the cluster, without some roles and functionality won’t be available)
This configuration will create a new object in ADSI called CNO (Cluster Named Object). Remember the computer creating the cluster needs to have access to create objects in ADSI, not having the permission can cause your cluster to fail to start. These services may not function properly in an Active-Directory-detached cluster, Hyper-V for live migration (relies on Kerberos), BitLocker drive encryption, or Cluster-Aware updating in selef updating mode. SQL server though has its on authentication method that will function just fine in this mode.
The quorum is the essence of a cluster:
The function is to provide each node with a vote and to prevent a cluster from splitting into two function identical servers. Additionally during the wizard under Advanced quorum configuration you add witness to break a tie:
Disk Witness, File Share, or Cloud Witness (hosting on Azure). A witness is automatically created when there are an even number of nodes. Only 1 witness per cluster. A disk witness is recommended when all nodes have access to the same shared storage.
Dynamic quorum management is the default in Windows 2016, this allows the cluster to continue running without configuring a quorum, where in older versions of Windows might crash a cluster. When a node leaves or is unavailable Dynamic quorum management removes its vote. Failing nodes are removed from voting.
Example, a 5 node cluster losses 3 nodes, you still have 2 nodes running which should function, but only 2 votes, which is below 50%, the cluster shutdowns in this situation. With dynamic quorum management, the three nodes fail, their votes and removed, the two nodes online are now 100% of the voters and the cluster continues to function.
Nodes can be part of a cluster and not have a vote, these are called “Non-Voting Nodes.”
Clustered networking can be broken into a few different types of traffic: Client communication – client access to the application running on the cluster, Cluster communication – The heartbeats and other communication between the cluster, iSCSI – and all other storage traffic that should be separated from the normal traffic, Live Migration – for Hyper-V clusters.
Keys to a successful cluster: Use software physical switches, don’t just depend on layer three VLANS, separate network adapters, separate manufacturers, and NIC teaming to provide failover capability.
Defaults generated during new cluster:
get-clusternetwork
(will disable properties of the network portion of the cluster)
iSCSI network traffic is disabled for cluster communication. networks without a default gateway are configured for cluster communication only, and networks with a default gateway address are configured for client and cluster communication.
You can modify these default to one of the following: Allow Cluster Communication On This Network, Allow Clients to Connect Through This Network, and Do Not Allow Cluster Communication on This Network.
Making changes with Powershell use the
(get-clusternetwork "network").settings
= value (unlike most Powershell commands where you use the set-clusternetwork, this uses the get, which was commonly use to modify changes in ADSI before all the new modules for ADSI was released).
Windows Updates to the Nodes:
Cluster Aware Updating Console is a separate interface that runs on Windows that allows configuration of updates for each node. It’s important all nodes have the same version of the OS they are running to accomplish full functionality.
You specify the type of updating: Self-Updating Mode – one of the cluster nodes has the CAU clustered role installed, enabling it to function as the Update Coordinator. This node updates the nodes in the cluster according to the schedule.
Remote Updating Mode: A computer outside the cluster is the Update Coordinator. This computer must have the Failer-Clustering feature installed.
OS Upgrades in the Nodes:
Mixed operating system mode allows for 2012R2 and 2016 servers to run in the same cluster. Microsoft recommends this mode as a temporary solutions and recommends upgrading the 2012R2 machines in less than month.
To upgrade the functionality of the cluster is done with the irreversible command in Powershell:
Update-ClusterFunctionalLeve
Before running the command, the procedure to to take a node form 2012R2 to 2016 is as follows:
1. Pause the node.
2. Drain the node of its workload by migrating it to other nodes.
This will pause and drain the cluster –
Suspend-Cluster -Drain
3. Evict the node from the cluster.
Remove-Cluster
4. Reformat the system drive and perform a clean installation of Windows Server
2016.
5. Configure network and storage connections.
6. Install the Failover Clustering feature.
Install-WindowsFeature Failover-Clustering
7. Add the newly-installed node back into the cluster.
Add-ClusterNode
8. Reimplement the cluster workload.
Cluster Storage: Fiber Channel, SAS, iSCSI (used in my virtual lab).
Created three disks on my ISCSI target (vhdx 1GB), added the server2 and server3 computers to the ISCSI target, and connected each server to the target server4.
After the targets were connected, they were available as storage space in my cluster.
You can leave them as is and create shares,add them to a pool (requires at least 3-4GB disks), or convert them to Cluster Volumes.
Cluster Shared Volumes create directories on each node c:\clusterstorage. This method allows all nodes to access data simultaneously, unlike a shared NTFS cluster volume that only allows one node at a time due to restraints with NTFS metadata. CSV sits on top of NTFS and acts like a pseudo file system.
add-clustersharedvolume -name "cluster disk 5"
One node owns the volume, but you can move it to another node as needed using the GUI. Cache exists by default, but the default cache size is 0, effectively disabling it. Enable it using (get-cluster).blockcachesize=512.
(get-cluternetwork -name "cluster network 3").metric
Will display the current network paths and the lowest metric is what CSV uses to communicate. Modify by adding = value.
File Server Type:
File server for general use (default), supports DeDupe, File Server Resource Manager, DFS Replication, and other File Servers role services.
Scale-Out File Server for application data: Leaves files open for extended periods of time, client connections are distributed across nodes in the cluster for better throughput. Support SMB protocol, but does not support NFS, DFS Replication, or File Server Resource Manager.
Creating Shares from the clustered disks:
New-smbshare -name share1 -path c:\clusteredstorage\volume1 -full access windows.com\cluster1 (if you have a CNO object), windows.com\server2, windows.com\server3 -continuouslyavailable set-smbpathacl -sharename share1
References:
Here are all the commands included in the Powershell module FailOverClusters:
CommandType Name Version Source ----------- ---- ------- ------ Alias Add-VMToCluster 2.0.0.0 FailoverClusters Alias Disable-ClusterS2D 2.0.0.0 FailoverClusters Alias Enable-ClusterS2D 2.0.0.0 FailoverClusters Alias Get-ClusterS2D 2.0.0.0 FailoverClusters Alias Remove-VMFromCluster 2.0.0.0 FailoverClusters Alias Repair-ClusterS2D 2.0.0.0 FailoverClusters Alias Set-ClusterS2D 2.0.0.0 FailoverClusters Alias Set-ClusterS2DDisk 2.0.0.0 FailoverClusters Function Add-ClusterGroupSetDependency 2.0.0.0 FailoverClusters Function Add-ClusterGroupToSet 2.0.0.0 FailoverClusters Function Disable-ClusterStorageSpacesDirect 2.0.0.0 FailoverClusters Function Enable-ClusterStorageSpacesDirect 2.0.0.0 FailoverClusters Function Get-ClusterDiagnosticInfo 2.0.0.0 FailoverClusters Function Get-ClusterFaultDomain 2.0.0.0 FailoverClusters Function Get-ClusterFaultDomainXML 2.0.0.0 FailoverClusters Function Get-ClusterGroupSet 2.0.0.0 FailoverClusters Function Get-ClusterGroupSetDependency 2.0.0.0 FailoverClusters Function Get-ClusterStorageSpacesDirect 2.0.0.0 FailoverClusters Function New-ClusterFaultDomain 2.0.0.0 FailoverClusters Function New-ClusterGroupSet 2.0.0.0 FailoverClusters Function Remove-ClusterFaultDomain 2.0.0.0 FailoverClusters Function Remove-ClusterGroupFromSet 2.0.0.0 FailoverClusters Function Remove-ClusterGroupSet 2.0.0.0 FailoverClusters Function Remove-ClusterGroupSetDependency 2.0.0.0 FailoverClusters Function Repair-ClusterStorageSpacesDirect 2.0.0.0 FailoverClusters Function Set-ClusterFaultDomain 2.0.0.0 FailoverClusters Function Set-ClusterFaultDomainXML 2.0.0.0 FailoverClusters Function Set-ClusterGroupSet 2.0.0.0 FailoverClusters Function Set-ClusterStorageSpacesDirect 2.0.0.0 FailoverClusters Function Set-ClusterStorageSpacesDirectDisk 2.0.0.0 FailoverClusters Cmdlet Add-ClusterCheckpoint 2.0.0.0 FailoverClusters Cmdlet Add-ClusterDisk 2.0.0.0 FailoverClusters Cmdlet Add-ClusterFileServerRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterGenericApplicationRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterGenericScriptRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterGenericServiceRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterGroup 2.0.0.0 FailoverClusters Cmdlet Add-ClusteriSCSITargetServerRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Add-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Add-ClusterResourceDependency 2.0.0.0 FailoverClusters Cmdlet Add-ClusterResourceType 2.0.0.0 FailoverClusters Cmdlet Add-ClusterScaleOutFileServerRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterServerRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterSharedVolume 2.0.0.0 FailoverClusters Cmdlet Add-ClusterVirtualMachineRole 2.0.0.0 FailoverClusters Cmdlet Add-ClusterVMMonitoredItem 2.0.0.0 FailoverClusters Cmdlet Block-ClusterAccess 2.0.0.0 FailoverClusters Cmdlet Clear-ClusterDiskReservation 2.0.0.0 FailoverClusters Cmdlet Clear-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Get-Cluster 2.0.0.0 FailoverClusters Cmdlet Get-ClusterAccess 2.0.0.0 FailoverClusters Cmdlet Get-ClusterAvailableDisk 2.0.0.0 FailoverClusters Cmdlet Get-ClusterCheckpoint 2.0.0.0 FailoverClusters Cmdlet Get-ClusterGroup 2.0.0.0 FailoverClusters Cmdlet Get-ClusterLog 2.0.0.0 FailoverClusters Cmdlet Get-ClusterNetwork 2.0.0.0 FailoverClusters Cmdlet Get-ClusterNetworkInterface 2.0.0.0 FailoverClusters Cmdlet Get-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Get-ClusterOwnerNode 2.0.0.0 FailoverClusters Cmdlet Get-ClusterParameter 2.0.0.0 FailoverClusters Cmdlet Get-ClusterQuorum 2.0.0.0 FailoverClusters Cmdlet Get-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Get-ClusterResourceDependency 2.0.0.0 FailoverClusters Cmdlet Get-ClusterResourceDependencyReport 2.0.0.0 FailoverClusters Cmdlet Get-ClusterResourceType 2.0.0.0 FailoverClusters Cmdlet Get-ClusterSharedVolume 2.0.0.0 FailoverClusters Cmdlet Get-ClusterSharedVolumeState 2.0.0.0 FailoverClusters Cmdlet Get-ClusterVMMonitoredItem 2.0.0.0 FailoverClusters Cmdlet Grant-ClusterAccess 2.0.0.0 FailoverClusters Cmdlet Move-ClusterGroup 2.0.0.0 FailoverClusters Cmdlet Move-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Move-ClusterSharedVolume 2.0.0.0 FailoverClusters Cmdlet Move-ClusterVirtualMachineRole 2.0.0.0 FailoverClusters Cmdlet New-Cluster 2.0.0.0 FailoverClusters Cmdlet New-ClusterNameAccount 2.0.0.0 FailoverClusters Cmdlet Remove-Cluster 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterAccess 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterCheckpoint 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterGroup 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterResourceDependency 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterResourceType 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterSharedVolume 2.0.0.0 FailoverClusters Cmdlet Remove-ClusterVMMonitoredItem 2.0.0.0 FailoverClusters Cmdlet Reset-ClusterVMMonitoredState 2.0.0.0 FailoverClusters Cmdlet Resume-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Resume-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Set-ClusterLog 2.0.0.0 FailoverClusters Cmdlet Set-ClusterOwnerNode 2.0.0.0 FailoverClusters Cmdlet Set-ClusterParameter 2.0.0.0 FailoverClusters Cmdlet Set-ClusterQuorum 2.0.0.0 FailoverClusters Cmdlet Set-ClusterResourceDependency 2.0.0.0 FailoverClusters Cmdlet Start-Cluster 2.0.0.0 FailoverClusters Cmdlet Start-ClusterGroup 2.0.0.0 FailoverClusters Cmdlet Start-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Start-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Stop-Cluster 2.0.0.0 FailoverClusters Cmdlet Stop-ClusterGroup 2.0.0.0 FailoverClusters Cmdlet Stop-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Stop-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Suspend-ClusterNode 2.0.0.0 FailoverClusters Cmdlet Suspend-ClusterResource 2.0.0.0 FailoverClusters Cmdlet Test-Cluster 2.0.0.0 FailoverClusters Cmdlet Test-ClusterResourceFailure 2.0.0.0 FailoverClusters Cmdlet Update-ClusterFunctionalLevel 2.0.0.0 FailoverClusters Cmdlet Update-ClusterIPResource 2.0.0.0 FailoverClusters Cmdlet Update-ClusterNetworkNameResource 2.0.0.0 FailoverClusters Cmdlet Update-ClusterVirtualMachineConfiguration 2.0.0.0 FailoverClusters
Restoring a Cluster Node’s database using wbadmin’s user interface.
The cluster database stores a copy on each node. If a node fail, you can restore the system and a full backup will provide a non-authoritative restore that is written over once the node comes back online. Another situation is when the node running has the latest version of the database. In this case a special authoritative restore is required using wbadmin’s command UI.
wbadmin get versions (this command works for all wbadmin backups.)
wbadmin get items -version: 11/14/2016:05:09 (displays all versions from this backup)
wbadmin start recovery -itemtype:app -items:cluster –
version:01/01/2008-00:00
After the restore is complete, Microsoft provides the instructions required to complete the process.