Some time ago I started to learn PowerShell a little more and when it comes to dealing with copy processes, I stumbled upon Robocopy, again. This time I decided to get familiar with it and use Robocopy together with PowerShell to automate my backups.
I got rid of my high power consuming server as a 24/7 machine and built a power saving server while using the old one for virtualization and backup purposes. So the goal was a two step backup approach: from the clients to the 24/7 machine (where IP Symcon is hosted) on a daily basis and on weekly basis from the 24/7 machine to the archive server which also has two HBAs.
The automation was supposed to include boot and shutdown of the target archive server.
Additionally I wanted to log into IP Symcon in order to trace the backup status (last backup, success timestamp, error message). For this purpose I already created a blog post dealing with JSON RPC from PowerShell to Symcon.
Having the backup information available in IP Symcon does not only provid
e an overview but could also be used to push messages to mobile devices on backup success. I will write another post about this soon…
I tried to illustrate the process in the following figure:
Backup from Client to 24/7 Server
I am using a very simple PowerShell script here and this is rather minimalistic than advanced:
|
1 2 3 4 5 |
$workBackupSource = "F:\work" $workBackupTarget = "\\DEMUC05DjangoMain\DjangoMain\_Backup\Automated\work" $workBackupLog = $workBackupSource + "\backup"+ (Get-Date -Format 'yyyy.MM.dd') +".log" Robocopy.exe $workBackupSource $workBackupTarget /E /DCOPY:DA /COPY:DAT /PURGE /R:30/W:30 /LOG:$workBackupLog /nfl /ndl /njh /njs /ns /nc /np |
Any errors which may occur during the copy process are logged to the .log file but it is not further handled. This could definitely be scripted better though.
I used a computer group policy to start the script on a shutdown/restart as described here: http://stackoverflow.com/a/18747578
- Run gpedit.msc (Local Policies)
- Computer Configuration -> Windows Settings -> Scripts -> Startup or Shutdown -> Properties -> Add
Backup from 24/7 Server to Archive
Here comes the main part where all the pieces are put together.
First we need to wake the archive server if it is not online yet.
I found a very good working PowerShell snippet on Jeff Wouters’s blog here , so all the credit belongs to him.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
function Send-WOL { param ( [parameter( mandatory=$true, position=0, ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] [ValidateLength(17,17)] [ValidatePattern("^([0-9A-F]{2}[:-]){5}([0-9A-F]{2})$")] [array]$MACAddress ) foreach ($MAC in $MACAddress) { try { $MAC = $mac.split(':') | %{ [byte]('0x' + $_) } $UDPclient = new-Object System.Net.Sockets.UdpClient $UDPclient.Connect(([System.Net.IPAddress]::Broadcast),4000) $Packet = [byte[]](,0xFF * 6) $Packet += $MAC * 16 Write-Verbose ([bitconverter]::tostring($Packet)) [void] $UDPclient.Send($Packet, $Packet.Length) Write-Output "WOL command sent to $MAC" } catch [system.exception] { Write-Output "ERROR: Unable to send WOL command to $MAC" } } } |
The script we will need is the one I used in one of my previous blog posts and which can be used to make JSON Remote Procedure Calls. As mentioned before, the approach is based on a script I found here.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 |
#region JSON RPC Functions Function New-JsonPayload() { [CmdletBinding()] Param( [Parameter()] [string]$payloadJsonRPCVersion = '2.0', [Parameter(Mandatory)] [string]$payloadMethod, [Parameter(Mandatory, ValueFromPipeline)] [array]$payloadParams ) Begin { Write-Verbose "Starting to create payload object and convertig to JSON" } Process { try { $jsonPayload = (New-Object -TypeName PSObject | Add-Member -PassThru NoteProperty jsonrpc $payloadJsonRPCVersion | Add-Member -PassThru NoteProperty method $payloadMethod | Add-Member -PassThru NoteProperty params $payloadParams | Add-Member -PassThru NoteProperty id '1') | ConvertTo-Json -Depth 3 # Content } catch { "Error was $_" $line = $_.InvocationInfo.ScriptLineNumber "Error was in Line $line" } } End { Write-Verbose "JSON Payload created" $jsonPayload } } Function New-JsonRpc() { [CmdletBinding()] Param( [Parameter(Mandatory)] [string]$url, [Parameter(Mandatory)] [string]$authUser, [Parameter(Mandatory)] [string]$authPass, [Parameter(Mandatory)] [string]$method, [Parameter(Mandatory)] [array]$params ) Write-Verbose "Setting credentials" $cred = New-Object -TypeName System.Net.NetworkCredential -ArgumentList $authUser, $authPass Write-Verbose "Creating payload" $payload = New-JsonPayload -payloadMethod $method -payloadParams $params $bytes = [System.Text.Encoding]::ascii.GetBytes($payload) $web = [System.Net.HttpWebRequest]::Create($url) [System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true } $web.Method = 'POST' $web.ContentLength = $bytes.Length $web.ContentType = 'application/json' $web.Credentials = $cred Write-Verbose "Getting request stream and sending payload" $stream = $web.GetRequestStream() $stream.Write($bytes,0,$bytes.Length) $stream.close() Write-Verbose "Getting the streamreader object" $reader = New-Object -TypeName System.IO.Streamreader -ArgumentList $web.GetResponse().GetResponseStream() return $reader.ReadToEnd()| ConvertFrom-Json } #endregion JSON RPC Functions |
The two scripts above are the dependencies for the main script which is scheduled through IP Symcon, these are dot-sourced in the BackUps.ps1 . I leave the comments I wrote when dealing with Robocopy as well as the witch case for the error codes. Maybe it is useful for others.
You need to adjust all the parameters in the first two regions, such as the Symcon URL, user name, password, Symcon variable IDs and paths.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 |
. $PSScriptRoot\JSON_RPC.ps1 . $PSScriptRoot\WOL.ps1 #region IP Symcon config $urlJSON = 'http://192.168.0.20:82/api/' $authUser = 'userName' $authPass = '********' $methodSetValue = 'SetValue' $datetimeNow = Get-Date -Format 'dd.MM.yyyy HH:mm:ss' $paramsLastBackup = @(53411, $datetimeNow) $paramsLastBackupSuccess = @(23232, $datetimeNow) $global:paramsLastBackupErrorMessage = @(52427, $errorMessage) #endregion IP Symcon config #region Backup config [string[]] $macAddresses = "AC:22:00:00:00:00" #MacAddress(es) of target Backup Server / machines to wake on lan $backupSource = 'D:\_Backup\Automated\work' $backupTarget = '\\DEMUC02DJANGO\Django-SecureArchive\work' $backupLog = 'D:\_Backup\Automated\backup.log' #endregion Backup config function Update-LastBackup { #Content $requestLastBackup = New-JsonRpc -url $urlJSON -authUser $authUser -authPass $authPass -method $methodSetValue -params $paramsLastBackup if ($requestLastBackup.result -eq $true) { Write-Host -Object 'Successfully executed JSON RPC' } } function Update-LastBackupSuccess { $requestLastBackup = New-JsonRpc -url $urlJSON -authUser $authUser -authPass $authPass -method $methodSetValue -params $paramsLastBackupSuccess if ($requestLastBackup.result -eq $true) { Write-Host -Object 'Successfully executed JSON RPC' } } function Update-LastBackupErrorMessage($errorMessage) { $global:paramsLastBackupErrorMessage[1] = $errorMessage $requestLastBackupErrorMessage = New-JsonRpc -url $urlJSON -authUser $authUser -authPass $authPass -method $methodSetValue -params $global:paramsLastBackupErrorMessage if ($requestLastBackupErrorMessage.result -eq $true) { Write-Host -Object 'Successfully executed JSON RPC' } } # Check if target backup server is online. If not, boot it up before processing backup $backupServerOnline = Test-Connection -ComputerName DEMUC02Django.elami.gee-life.com -Quiet if($backupServerOnline -eq $False) { Write-Host "Starting BackupServer since it seems to be off." Send-WOL($macAddresses) do { Write-Host "Waiting for Server to come up" $online = Test-Connection -ComputerName DEMUC02Django.elami.gee-life.com -Quiet Write-Host "Target server online?" $online Get-Date Start-Sleep -Seconds 10 } until ($online -eq $True) Write-Host "BackupServer is up:" (Get-Date).ToString() Write-Host "Adding sleep delay of 120 seconds before processing further backup actions" Start-Sleep -Seconds 120 } Write-Host "Starting robocopy job..." # Start the backup Robocopy.exe $backupSource $backupTarget /E /DCOPY:DA /COPY:DAT /PURGE /R:30 /W:30 /LOG:$backupLog /nfl /ndl /njh /njs /ns /nc /np <# RoboCopy Params Essential parameters Include subdirectories (excl empty) /S Include subdirectories (incl empty) /E /DCOPY:DA required for legacy Robocopy versions Deletes destination files/directories which no longer exist /PURGE (/e + /purge = /MIR but sec settings are not overwritten) Number of Retries /R:240 Delay before Retry /W:30 Log Paramters No file names logged: /nfl No directory names logged: /ndl No job header: /njh No job summary: /njs No file sizes: /ns No file classes: /nc No progress: /np More parameters Exclude Files /XF *.m3u Exclude Directory /XD "C:\Path..." #> # Update last backup timestamp in IP Symcon Update-LastBackup # Check if backup errors occured and update those in IP Symcon # might be not precise as error messages could be longer $lastErrorMessage = Get-Content $backupLog | Select-Object -Last 3 if(![string]::IsNullOrEmpty($lastErrorMessage)) { $lastErrorMessageString = $lastErrorMessage -join "" Update-LastBackupErrorMessage $lastErrorMessageString } else { Update-LastBackupSuccess Update-LastBackupErrorMessage "No Errors" } Stop-Computer -ComputerName DEMUC02Django.elami.gee-life.com #region Robocopy ExitCodes - exit codes based on http://ss64.com/nt/robocopy-exit.html #used for debugging switch($lastexitcode) { 0 { Write-Host -Object 'Robocopy succeeded' } 1 { Write-Host 'Robocopy succeeded to write all files with exit code:' $lastexitcode } 2 { Write-Host 'Some extra files or directories were detected. No files were copied Examine the output log for details. Exit code:' $lastexitcode } 3 { Write-Host 'Some files were copied. Additional files were present. No failure was encountered. Exit code:' $lastexitcode } 4 { Write-Host 'Some mismatched files or directories were detected. Examine the output log. Housekeeping might be required. Exit code:' $lastexitcode } 5 { Write-Host 'Some files were copied. Some files were mismatched. No failure was encountered. Exit code:' $lastexitcode } 6 { Write-Host 'Additional files and mismatched files exist. No files were copied and no failures were encountered. This means that the files already exist in the destination directory. Exit code:' $lastexitcode } 7 { Write-Host 'Files were copied, a file mismatch was present, and additional files were present. Exit code:' $lastexitcode } 8 { Write-Host 'Some files or directories could not be copied (copy errors occurred and the retry limit was exceeded). Exit code:' $lastexitcode } 16 { Write-Host 'Serious error. Robocopy did not copy any files. Either a usage error or an error due to insufficient access privileges on the source or destination directories. Exit code:' $lastexitcode } default { if(!$lastexitcode) { Write-Host -Object 'No error code present' } } } #endregion #$global:LASTEXITCODE = $null |
Scheduling the Backups with IP Symcon
The Powershell script could be executed with the windows task planner for instance. I used IP Symcon however, to keep things in one place and learn something new.
|
1 2 3 |
<? IPS_ExecuteEx("D:\_SystemShare\BackupAutomation\StartPowerShellBackup.bat", "", false, false, -1); ?> |
As you see, the IPS_ExecuteEx method is starting a .bat process instead of the PowerShell script directly. Unfortunately I couldn’t execute the .ps1 directly in this method.
The first boolean value of the IPS_ExecuteEx method determines whether a window of the program to be started should be shown or not. This is helpful when it comes to debugging/testing.
For the sake of completeness, here is the no brainer line of the .bat:
|
1 |
powershell D:\_SystemShare\BackupAutomation\BackUps.ps1 |
Now you can just use the IP Symcon default methods for scheduling the backups on regular base such as weekly or daily.
This is how it looks like in the WebFront…you could also schedule the backup through the frontend 🙂
P.S. I know there are a lot of different possibilities when it comes to backups but as I wanted to learn more PowerShell I decided to go this way combined with Robocopy (as this provides sync capabilities compared to new-item copy method of PS)
Hope it may be useful for some others.
Cheers!
No Comment
You can post first response comment.