PowerShell from the Trenches

Three Examples of solutions implemented with PowerShell

Presented by:Jorge Besada - SQL Server Database Administrator from Carnival Cruise Lines email: jbesada@carnival.comOther qualifications:See next page for email signature

Jorge Besada - CPE, MPD(Carnival PowerShell Evangelist - Master PowerShell Developer)------------------------------------------------------------------------------From Jeff Snover, PowerShell creator"It's like programming with hand grenades.”

First application

Here at work we make extensive use of SAN storage and many of the server drives are not local drives. They work well, we forget about them until one of them fails. 99% of the time is the problem is not an actual failure, but the connection/mapping of the drive/SAN is lost.

And when the drive that goes down happens to contain the SQL server installation of an instance you get a lot of calls from angry customers and it is difficult to diagnose initially (especially if it has been a long time since the last incident of missing drives) what actually happened. When you miss a drive holding user database files is not that bad, you see the database as offline and eventually you find the cause.The fix is easy: the network person just does a rescan

From command line:echo rescan | diskpart

So to save ourselves DBAs some aggravation a powershell program was created that does the following:It lists all drives from a list of serversIt compares the list of today’s run with the saved list from yesterday’s run, the differences will be the servers with missing drivesIf there is a difference it runs the rescan command on the affected serversAn email is sent reporting the issue

• #GetMissingDrives.ps1• #sends email with servers and missing drives• #rescans drives

• #how to use it:• #Edit the file servers_list.dat to include the list of servers to check

(a server per line)• #(comments can be added starting with #)• #Sample file:

• ##Server list to monitor hard disk space• ##To add comments start the line with #• ##---------SQLSERVERS-----------------• #CCLTSTINFSQL1• #CCLDEVINFSQL1

• $SERVERLIST = 'C:\Users\jorgebe\Documents\powershell\servers_list.dat'• $EMAILTOLIST = 'jbesada@carnival.com'• $FROM = "SQLDBA@noreply.com"• $SUBJECT = "Servers Missing Drives Report"• $LOGFILE = 'C:\Users\jorgebe\Documents\powershell\today.lst'• $REFERENCE = 'C:\Users\jorgebe\Documents\powershell\reference.lst'

• function ListDrives($Servername)• {• ForEach-Object `• {• get-wmiobject -computername $Servername win32_logicaldisk | select-object systemname, deviceID, Size, Freespace,

DriveType • }• }

• #Returns array of servers and drives• <# Sample output• SERVER1|C• SERVER1|D• SERVER2|C• SERVER2|D• SERVER2|E• ...• #>• function ProcessServers($Serverlist)• {

• $outarray = @()• $Computers = Get-Content -Path $Serverlist;• foreach ($z in $Computers) • {• if ($z -gt "" -and $z[0] -ne "#")• {• $erroractionpreference = "SilentlyContinue"• $a = ListDrives($z)• foreach ($k in $a)• {• if ($k.DriveType -eq 3)• {• $p = $k.deviceid -replace(":", "")

$outarray += $z + '|' + $p

• }• }• • }• }

return $outarray

• • }

Send mail goodie• #Usage: SendMail message • # emailist (as array)• # c:\attach1.txt;c:\attach2.txt • # SFASupport@noreply.com • # "This is the subject"• function SendMail ($report,$emailarray,$attacharray,$from,$subject)• {• $smtpServer = "smtphost.carnival.com"• $msg = new-object Net.Mail.MailMessage• $smtp = new-object Net.Mail.SmtpClient($smtpServer)• $msg.From = $from • foreach ($c in $emailarray)• { • $msg.To.Add($c)• } • $attlist = $attacharray.Split(";")• foreach ($c in $attlist)• { • $att = new-object Net.Mail.Attachment($c)• $msg.Attachments.Add($att)• } • $msg.Subject = $subject• $msg.Body = $report• $smtp.Send($msg)• }

The program

• Write-Host "Processing server list " $SERVERLIST• #remember: server list has the simple list of servers, no drives• #SERVER1• #SERVER2

• #this call expands the original server list to include the drives• #SERVER1|C SERVER1|D SERVER2|C SERVER2|D SERVER2|E

• $r = ProcessServers $SERVERLIST

• #And we save the actual list to a file today.lst• Set-Content $r -Path $LOGFILE -Force

• #We read the reference list (no missing drives, file reference.lst into a variable• $z = Get-Content $REFERENCE

• #And this is one of the best things of PowerShell: the comparison comandlet• #Compare-Object -ReferenceObject $z -DifferenceObject $r• $g = Compare-Object $z $r

• #if there is a difference that means we lost a drive• if ($g)• {

$msg = [char]13 + 'Re-scanning disks...' + [char]13Write-Host $msg

• foreach ($x in $g)• {• $c = $x.InputObject.Split('|')[0]• Write-Host 'Computer: ' $c• $msg = $msg + $c + [char]13• #And this is another extremely cool powershell thing: The Invoke-Command• #to use invoke-command you need to be at PowerShell version 2.0• $h = invoke-command -computername $c {"rescan" | diskpart}• foreach ($f in $h)• {• $msg = $msg + $f + [char]13• }• }• SendMail $msg $EMAILTOLIST $LOGFILE $FROM $SUBJECT

• #The previous code fixed the missing drives calling rescan for the involved servers• #now we reset the reference list, to leave it ready for the next run• $r = ProcessServers $SERVERLIST• Set-Content $r -Path $REFERENCE -Force• }

Quiz Now

The program as listed in the previous slide has a bug. It works as expected, but the bug is there … lurking. I discovered it while preparing this presentation and I have not yet fixed it

Can you find it?Going back to the code screen …

The little bug: suppose you missed more than one drive in a server. You only need to rescan once for all the missing drives to be reattached. The original code would have done the rescan once for each missing drive.This is fixed in the following code:

• if ($g)• {• $msg = [char]13 + 'Re-scanning disks...' + [char]13• Write-Host $msg• $c_previous = "" • foreach ($x in $g)• {• $c = $x.InputObject.Split('|')[0]#here we get the computer name• #Here is the fix:• if ($c -ne $c_previous)• {• Write-Host 'Computer: ' $c• $msg = $msg + $c + [char]13• $h = invoke-command -computername $c {"rescan" | diskpart}• foreach ($f in $h)• {• $msg = $msg + $f + [char]13• }• $c_previous = $c• }• }

End of First ApplicationApp number two: The Data Quest

High level manager request data to be pulled from all ships. After a lot of work two queries were created. The task was to run these two queries in every ship and collect the results in csv files- The queries may have to be re-run after

updates- Satellite connections to the ships are slow, and

sometimes not available

First try

A cmd file was created calling sqlcmd for each ship, executing the two queries on each ship sql server and saving the output to a file:

One_ship.cmd:Sqlcmd –S %1 –s, -W –d ssadserver –I”cpg_query1.sql” –o%1”_1.csv”Sqlcmd –S %1 –s, -W –d ssadserver –I”cpg_query2.sql” –o%1”_2.csv”

All_ships.cmd:One_ship.cmd SHIP1One_ship.cmd SHIP2… many ships

This did not work: the results coming across the network (some files were 60Mb and more), the process hung for hours.We needed a program to do this:- Copy the queries to the ship- Execute them in the ship, keep results in ship

- Compress the output file- Copy results back to office

- All these steps could have been done with cmd files, but I have noticed that I save a lot of time using powershell combined with cmd files. I have a simple rule for this:

- can the program be done with a single bat file- If yes, just do it - If not, most of the time you save time using

powershell as a driver of other scripts

• #CPG_Transfers2.ps1 FILE SHARE VERSION, Jorge's workstation version• #Runs sql queries in ships, collects resulting csv files

• #How to use it:• #Edit the file servers_list.dat to include the list of servers to check (a server per line)• #(comments can be added starting with #)• #Sample file:

• ##To add comments start the line with #• ##---------SQLSERVERS-----------------• #SHIPSERVER1• #SHIPSERVER2

• $SERVERLIST = 'C:\Users\jorgebe\Documents\powershell\ships_list.dat'• $EMAILTOLIST = 'jbesada@carnival.com'• $FROM = "SQLDBA@noreply.com"• $SUBJECT = "CPG Transfer Report - Fileshare version"• $SAVE_FOLDER = '\\CCLDEVINFSQL1\TEMP\backup\'

• #Returns ping number or empty string if no response• function CheckPing($server)• {

$v = (ping $server -n 1)

• foreach ($k in $v)• {• if ($k.StartsWith("Reply"))• {• break• }• }• $l = $k.Replace('<', '=')• $lst = $l.split('=')[2]• if ($lst)• {• $r = $lst.Replace('ms TTL', '')• }• else• {• $r = ""• }• return $r• }

• function ProcessServers($Serverlist)• {

• $outdict = @{}• $Computers = Get-Content -Path $Serverlist;• foreach ($z in $Computers) • {• #Sort by ping process• if ($z -gt "" -and $z[0] -ne "#")• {• $ping = CheckPing($z)• $outdict[$z] = [int]$ping• }

• }

• $sorted = $outdict.GetEnumerator() | Sort-Object Value

• foreach ($j in $sorted)• {• $t1 = Get-Date• $dt1 = $t1.tostring()• Write-Host "Started" $dt1

• $ping = $j.value• $z = $j.name• $ping_current = CheckPing($z)• Write-Host "Ping response original run: " $ping" Current: " $ping_current• if ($ping_current -gt "")• {• Write-Host "Copying queries to ship" $z• $dest = '\\' + $z + '\DBAScripts'• copy "cpg_query1.sql" $dest -Force• copy "cpg_query2.sql" $dest -Force

• Write-Host "Executing CPG queries on ship"• (C:\Users\jorgebe\Documents\powershell\cpg_transfers.cmd $z)

• Write-Host "Compressing results"• (C:\Users\jorgebe\Documents\powershell\cpg_transfers_compress.cmd $z)

• Write-Host "Collecting results"• $src1 = $dest + '\cpg_query1.csv'• $src2 = $dest + '\cpg_query2.csv'• copy $src1 ($SAVE_FOLDER + $z + '_1.csv') -Force• copy $src2 ($SAVE_FOLDER + $z + '_2.csv') -Force

• $SUBJECT = $z + " cpg queries process completed"• $msg = "Started at " + $dt1 + [char]13 + "Completed at " + $dt2 + [char]13• $tt = $t2 - $t1• $msg = $msg + "Elapsed time: " + $tt.ToString()+ [char]13• $msg = $msg + "Ping response: " + $ping + [char]13• SendMail $msg $EMAILTOLIST "" $FROM $SUBJECT• }• else• {• Write-Host "-------------------------------------"• Write-Host "Ship " $z " not responding"• Write-Host "-------------------------------------"• }

}

Vodoo cmd file (cpg_transfers.cmd)The double combination of the 2 sqlcmd calls and xp_cmdshell enables the placing of the output file in the desired remote server location

Line 1:

sqlcmd -S %1,9999 -l 10000 -Q“xp_cmdshell ‘sqlcmd -h -1 -l 10000 -S . -d ssadserver -s, -W -I -id:\DBAScripts\cpg_query1.sql -o d:\DBAScripts\cpg_query1.csv‘“

Line 2:

sqlcmd -S %1,9999 -l 10000 -Q“xp_cmdshell ‘sqlcmd -h -1 -l 10000 -S . -d ssadserver -s, -W -I -id:\DBAScripts\cpg_query2.sql -o d:\DBAScripts\cpg_query2.csv‘“

Remote compress cmd file cpg_transfers_compress.cmd:“compact” is a little known compress utility, it is built in latest server versions

sqlcmd -S %1,9999-l 10000-Q"xp_cmdshell 'compact /c d:\DBAScripts\cpg_query1.csv'“

sqlcmd -S %1,9999-l 10000-Q"xp_cmdshell 'compact /c d:\DBAScripts\cpg_query2.csv'"

SAMPLE OUTPUT FOR FOUR SHIPS

Thursday, March 14, 2013 3:04:22 PMProcessing ships list ships_list.datPing response: 695 Ship: LESQL1.LEGEND.CARNIVAL.COMCopying queries to ship3/14/2013 3:04:23 PM3/14/2013 3:05:04 PMExecuting CPG queries on ship3/14/2013 3:07:00 PMCompressing results3/14/2013 3:07:19 PMCollecting results3/14/2013 3:08:13 PMcompleted 3/14/2013 3:08:13 PMPing response: 840 Ship: SPSQL1.SPIRIT.CARNIVAL.COMCopying queries to ship3/14/2013 3:08:15 PM3/14/2013 3:08:38 PMExecuting CPG queries on ship3/14/2013 3:09:42 PMCompressing results3/14/2013 3:09:58 PMCollecting results3/14/2013 3:10:39 PMcompleted 3/14/2013 3:10:39 PMProcess completed Thursday, March 14, 2013 3:10:40 PM

Thursday, March 14, 2013 4:18:01 PMProcessing ships list ships_list.datPing response: 1517 Ship: IMSQL1.IMAGINATION.CARNIVAL.COMCopying queries to ship3/14/2013 4:18:03 PM3/14/2013 4:18:55 PMExecuting CPG queries on ship3/14/2013 4:25:47 PMCompressing results3/14/2013 4:26:03 PMCollecting results3/14/2013 4:43:25 PMcompleted 3/14/2013 4:43:25 PMPing response: 799 Ship: SESQL1.SENSATION.CARNIVAL.COMCopying queries to ship3/14/2013 4:43:27 PM3/14/2013 4:43:55 PMExecuting CPG queries on ship3/14/2013 4:49:34 PMCompressing results3/14/2013 4:49:50 PMCollecting results3/14/2013 5:55:56 PMcompleted 3/14/2013 5:55:56 PMProcess completed Thursday, March 14, 2013 5:55:56 PM

App Number Three:Original Request

• There was an old Vb.net application used to send files from corporate office to the remote locations and collect other files from them. It used MSMQ. At some time the application started to have intermittent failures and after some research it was detected that the files had grown overtime to sizes that exceeded the message queue limits

Original Plan “A”Solution

• Add code to the old VB.net application to incorporate file splitting before the transmission and file joining after reception

• A couple of PowerShell functions were created to do the split/join

SplitDeveloped using advanced techniques (next slide)

function split($inFile, $outfolder, $outPrefix){

$bufSize = (Get-ItemProperty $infile).Length / $CHUNKSWrite-Host 'bufsize = ' $bufSize

$stream = [System.IO.File]::OpenRead($inFile) $chunkNum = 1 [Int64]$curOffset = 0 $barr = New-Object byte[] $bufSize

while( $bytesRead = $stream.Read($barr,0,$bufsize)){ $outFile = $outfolder + '\' + "$outPrefix$chunkNum" $ostream = [System.IO.File]::OpenWrite($outFile) $ostream.Write($barr,0,$bytesRead); $ostream.close(); echo "wrote $outFile" $chunkNum += 1 $curOffset += $bytesRead $stream.seek($curOffset,0); }}

rm ($WORKFOLDER + '\*') -Force split $SOURCEFILE $WORKFOLDER $FILEPREFIX

Advanced Technique

• I learned this technique in TechEd 2012 from the PowerShell gurus

Steal from The BestAndCreate the Rest

Join function join($outfile, $folder, $prefix){$cnt = (Get-ChildItem $folder).Count$ostream = [System.IO.File]::OpenWrite($outFile)$i = 1while ($i -lt $cnt + 1){Write-Host $i$file = $folder + '\' + $prefix + $i$bufSize = (Get-ItemProperty $file).LengthWrite-Host $file $bufSize$stream = [System.IO.File]::OpenRead($file) $barr = New-Object byte[] $bufSize

while( $bytesRead = $stream.Read($barr,0,$bufsize)){ $ostream.Write($barr,0,$bytesRead);}$i++}$ostream.close();}

Intermezzo

• All this is fine and dandy, but …• This thing of taking an old vb.net app, adding a

powershell hack to it looks like a … hack

• Can we have a more elegant approach using PowerShell for everything?

• Let’s see the “Plan B”

FileExchanger.ps1- What it does:

The contents of the source main folder \\MAINSERVER\outbound_miamiare sent to the shared folders in the remote servers, each server receives the corresponding filesbased on the file id in the file name to be sentThe contents of the source main folder \\MAINSERVER\inbound_miamireceives the files from all the remote servers

Remote servers:The contents of shared outbound folder (sample folder \\REMOTESERVER\outbound )are sent to the mainserver collecting folder \\MAINSERVER\inbound_miami

How to test:

First:Place files with the correct file name format in the remote server folderSample values:For remote server xxdevsql1 file id used was 300Sample file name is 000000_300_MOP_15269.zip in \\xxdevsql1\app\outboundFor remote server ccltstsql1 file id is 350

Second:Place files with the correct file name format in the mainserver outbound folderSample values:300_000000_MOF_12010.zip350_000000_MOF_12012.zip375_000000_MOF_11256.zip400_000000_CORP_9897.zip

Third: Execute the powershell program FileExchanger.ps1 from MAINSERVER

Results:The files from remote servers will end up in the mainserver inbound folderThe files from the mainserver outbound folder will end up in the corresponding inbound folder in the remote serversThe extra files in the mainserver outbound folder (for remote servers not in the list) are not processed.

FileExchanger.ps1 – the code - 1

#MAINSERVER SHARES$FiletoSendLoc1 = '\\ccldevsql1\outbound_miami'$PlacetoSave1 = '\\ccldevsql1\inbound_miami‘

$REMOTESERVERS = @{300="Miami|xxdevsql1"325="New York|ccltstsql1"350="Rio de Janeiro|ccltstcorpsql1"400="London|ccldevinfsql1"}

function CheckPing($server){$v = (ping $server -n 1)$l = $v[2].Replace('<', '=')$lst = $l.split('=')[2]$r = $lst.Replace('ms TTL', '')return $r}

function ExtractItems($file_id){$server_code, $server_name = $REMOTESERVERS[$file_id].Split('|')return $server_code, $server_name}

FileExchanger.ps1 – the code - 2

function SendFiles($file_id){ Write-Host 'Sending files to ' $REMOTESERVERS[$file_id] $server_code, $server_name = ExtractItems($file_id) Write-Host 'Server name: ' $server_name ', Server code: ' $server_code $dest = '\\' + $server_name + '\app\inbound' $p = $FiletoSendLoc1 + '\' + $file_id + '_*.zip' Write-Host 'destination folder: ' $dest copy $p $dest -Force -Verbose}

function ReceiveFiles($file_id){Write-Host 'Receive files from ' $REMOTESERVERS[$file_id]$server_code, $server_name = ExtractItems($file_id)Write-Host 'Server name: ' $server_name ',Server code: ' $server_code $src = '\\' + $server_name + '\app\outbound\000000_' + $file_id + '_' + '*.zip'Write-Host 'destination folder: ' $PlacetoSave1copy $src $PlacetoSave1 -Force -Verbose}

function MainLoop{foreach ($file_id in $REMOTESERVERS.get_keys()){Write-Host 'Processing' $file_id$server_code, $server_name = ExtractItems($file_id)$t = CheckPing($server_name)Write-Host 'Checking ping response for the record:' $server_name $t 'ms'if ($t -lt 1000){SendFiles($file_id)ReceiveFiles($file_id)}else{Write-Host 'Ping response too high, skipping ' $server_name ' in this run' }}

}

MainLoop

FileExchanger.ps1 – the output

Final Notes

I was fortunate to attend TechEd 2012 this year in Orlando.

It was a highly technical event, I attended most of the PowerShell conferences:

Final Notes

Ooops … wrong slide here

Jeff Snover – PowerShell Creator

... Snover, (the architect behind the product now responsible for the entire windows server platform):

"It's like programming with hand grenades.”

I know people!

Conclusion

If you want to be a good PowerSheller …

You Better Wear Vibram Shoes