Post on 17-Jan-2018
description
transcript
SharePoint Online Migration API and Improvements
Simon BourdagesMicrosoft PM
© SPintersection. All rights reserved.http://www.SPintersection.com
Introduction
Overview Moving content into O365
Current tech New Capabilities Things to come
The Migration API and PowerShell Using the PowerShell Building Code for the API Tips and Lessons Learned
A dedicated API created by Microsoft in order to provide the 3rd party vendors a dedicated way to bring large volumes of contents into SharePoint Online and OneDrive for Business.
The Migration API is live since May 2015
The Migration API
© SPintersection. All rights reserved.http://www.SPintersection.com
Moving Content in O365Current Tech
The Migration API was built to bring large Amount of content into the O365 Cloud
Requires the Use of Azure Timer Job based Best way to avoid CSOM throttling Scalable and in continuous improvement
© SPintersection. All rights reserved.http://www.SPintersection.com
Moving Content in O365Current Tech
Source Package SharePoint/ODBFinal Destination
Azure Temporary Storage
File Share,SharePoint On-Prem and potentially any other data Source
Create package for the API to be able to accept it
Use the Power of ingestion of Azure to bring the content faster in Microsoft network
Timer Job based import in a scalable way that will not hurt the service using back end resources
© SPintersection. All rights reserved.http://www.SPintersection.com
Moving Content in O365Current Tech
File and Folders Basic Metadata Permissions Versions
© SPintersection. All rights reserved.http://www.SPintersection.com
Moving Content in O365New Capability
File and Folders Basic Metadata Permissions Versions
Encryption at Rest Alerts Taxonomy
© SPintersection. All rights reserved.http://www.SPintersection.com
Moving Content in O365Things to come
File and Folders Basic Metadata Permissions Versions
Encryption at Rest Alerts Taxonomy
Removing the need of a customer Azure Account TAP program available for Developers
More Based on Usage and Requests
© SPintersection. All rights reserved.http://www.SPintersection.com
PowerShell
© SPintersection. All rights reserved.http://www.SPintersection.com
The Migration API and PowerShellWhat is What
The PowerShell Commands PowerShell based Client side tool Performs the end to end content migration
The Migration API CSOM based Call Requires Transformation of data Server side Service Requires the content to already be in Azure container
© SPintersection. All rights reserved.http://www.SPintersection.com
The Migration API and PowerShellTooling responsibility
Source Package SharePoint/ODBFinal Destination
Azure Temporary Storage
File Share,SharePoint On-Prem and potentially any other data Source
Create Package for the API to be able to accept it
Use the Power of ingestion of Azure to bring the content faster in Microsoft network
Timer Job Based import in a scalable way that will not hurt the service using back end resources
Client Tool Azure API SPO Migration API
© SPintersection. All rights reserved.http://www.SPintersection.com
PowerShellThe Commands
New-SPOMigrationPackage
ConvertTo-SPOMigrationTargetedPackage
Set-SPOMigrationPackageAzureSource
Submit-SPOMigrationJob
New
Convert
Set
Submit
© SPintersection. All rights reserved.http://www.SPintersection.com
PowerShellcommands Flow
Convert Set Submit
NewExported Content
Package SharePoint/ODBFinal Destination
Azure Temporary Storage
File Share Server
SharePoint On Premise Export-SPWeb
From On-Premise
© SPintersection. All rights reserved.http://www.SPintersection.com
PowerShellSupported scenarios
File and Folders Basic Metadata Permissions Versions
File Share SP 2010, 2013, 2016
© SPintersection. All rights reserved.http://www.SPintersection.com
The first command will read the list of content targeted by the source path and then will generate the appropriate XML to perform migration. Here are the required parameters to enter (others are optional)
SourcefilesPath : pointing to your content you want to migrate OutputPackagePath : pointing to your Temporary folder
Ex:New-SPOMigrationPackage -SourcePath C:\Users\Pictures\ -OutputPackagePath C:\Users\test\tempPackage
PowerShellNew-SPOMigrationPackage
© SPintersection. All rights reserved.http://www.SPintersection.com
The Second command will simply convert the xml generated in your temporary folder into a format that the destination will understand. This will be the final package. Here are the required parameters to enter (others are optional)
TargetwebURL : point to the Web of the destination SourceFilesPath : pointing to your content you want to migrate SourcePackagePath : pointing to your Temporary package folder OutputPackagePath : pointing to your final package folder targetDocumentLibraryPath : This is the path to your destination library Credentials : SPO credential that has admin rights to the destination site
Ex:ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/ -SourcePackagePath C:\Users\test\tempPackage -OutputPackagePath C:\Users\test\SPOpackage -targetDocumentLibraryPath Documents -sourcefilespath C:\Users\Pictures\ -Credentials (Get-Credential admin@datamigration.onmicrosoft.com)
PowerShellConvertTo-SPOMigrationTargetedPackage
© SPintersection. All rights reserved.http://www.SPintersection.com
The third command will simply convert the XML generated into a format that the destination will understand. Here are the required parameters to enter (others are optional)
SourceFilePath : Pointing to the content you want to migrate SourcePackagePath : Pointing to the final package folder AccountName : The name of the Azure storage account Accountkey : The key to the Azure storage account AzureQueueName : The name of the azure queue blob(optional but recommended)
We also recommend to take the output of this command and store it in a variable for the next command.$temporaryVariable = Set-SPOMigrationPackageAzureSource….
Ex:$azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath C:\Users\Pictures\ -SourcePackagePath C:\Users\test\SPOpackage -AccountName mystoragename -AccountKey 000000000000000000000000000000000000000000000000000000000000000000000000000000== -AzureQueueName migrationqueue
PowerShellSet-SPOMigrationPackageAzureSource
© SPintersection. All rights reserved.http://www.SPintersection.com
Ship drive requirements
Max 6TB drives SATA II or III Format the drive ( not a requirement but still makes things easier) Prepare your container Install the Azure tool :
http://go.microsoft.com/fwlink/?linkid=301900&clcid=0x409
The last command will simply convert the XML generated into a format that the destination will understand. Here are the required parameters to enter (others are optional)
TargetwebURL : Point to the Web of the destination MigrationPackageAzureLocations : The temporary variable from the previous
command Credentials : SPO credential that has admin rights to the destination site
Ex:Submit-SPOMigrationJob -TargetWebUrl https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/ -MigrationPackageAzureLocations $azureSource -Credentials (Get-Credential admin@datamigration.onmicrosoft.com)
PowerShellSubmit-SPOMigrationJob
© SPintersection. All rights reserved.http://www.SPintersection.com
PowerShellRemarks for PowerShell
Export-SPWeb Target a List Uncompressed
New-SPOMigrationPackage Be careful of SPO limitations
ConverTo-SPOMigrationTargetedPackage Using different output folder is recommended
Set-SPOMigrationPackageAzureSource The Azure Containers will be required for the Submit-SPOMigrationJob
Submit-SPOMigrationJob The destination list needs to exist
© SPintersection. All rights reserved.http://www.SPintersection.com
Migration API
© SPintersection. All rights reserved.http://www.SPintersection.com
Building code for the Migration API
Uses PRIME package formatting Recommended to stay with what the PowerShell commands supports SPO Migration TAP program
spomigtap@microsoft.com The Migration API is agnostic of the source
© SPintersection. All rights reserved.http://www.SPintersection.com
The Migration API calls[ClientNS.ClientCallableMethod] public Guid CreateMigrationJobEncrypted( Guid gWebId, string azureContainerSourceUri, string azureContainerManifestUri, string azureQueueReportUri, MigrationOption AES256CBCKey)
[ClientNS.ClientCallableMethod] public Guid CreateMigrationJob ( Guid gWebId, string azureContainerSourceUri, string azureContainerManifestUri, string azureQueueReportUri,)
© SPintersection. All rights reserved.http://www.SPintersection.com
Migration Job Status
[ClientNS.ClientCallableMethod] public Guid SPMigrationJobState (Guid MigrationJobId)
© SPintersection. All rights reserved.http://www.SPintersection.com
The encryption method should follow the AES CBC 256 Standard The IV should be different for every file including the manifests in the
package. should be stored as a metadata on each files.
Name = [IV] Value = [Base64Encoded byte array of the IV]
Extra Requirement for Encryption
© SPintersection. All rights reserved.http://www.SPintersection.com
Migration API EncryptedQueue
Here is the JSON content in the queue message{"Label", "Encrypted"},
{"JobId", "[JobId value]"},{"IV", ”[IV value in base64format]”}, {"Content", “[encrypted message in base64string]”}
Once decrypted the messages will be the same as the API without encryption.
© SPintersection. All rights reserved.http://www.SPintersection.com
Migration API Logs
[Date]Message[Date] [Action] message[Date] [item type] [SPObject name] [SPObject path] [SPObject ID] [Action] message
Note that there may be other structures, and in some cases message will overlap into multiple lines, such as
© SPintersection. All rights reserved.http://www.SPintersection.com
Optimizations
Parallel jobs Find the perfect package size
Number of items Weight of metadata Weight in GB over the wire
Single Report queue for every jobs Keeping every threads running No over provisioning
© SPintersection. All rights reserved.http://www.SPintersection.com
Please use Event Board to fill out a session evaluation.
Questions?
Thank you!
© SPintersection. All rights reserved.http://www.SPintersection.com
Sample Code
$userName = "admin@contoso.onmicrosoft.com"$adminSite = "https://contoso-admin.sharepoint.com" $sourceFilePath = "D:\data\documents\"$packagePath = "d:\data\documentPackage"$targetWebUrl = "https://contoso.sharepoint.com/sites/finance" $spoPackagePath = "d:\data\documentPackageForSPO"$targetLibrary = "Shared Documents" $azureAccountName = "acctName"$azureAccountKey = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000=="$azureQueueName = "contosoMigrationQueue" $cred = Get-Credential $userNameNew-SPOMigrationPackage -SourceFilesPath $sourceFilePath -OutputPackagePath $packagePath ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl $targetWebUrl -SourceFilesPath $sourceFilePath -SourcePackagePath $packagePath -OutputPackagePath $spoPackagePath -TargetDocumentLibraryPath $targetLibrary -Credentials $cred$azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFilePath -SourcePackagePath $spoPackagePath -AccountName $azureAccountName -AccountKey $azureAccountKey -AzureQueueName $azureQueueNameSubmit-SPOMigrationJob -TargetWebUrl $targetWebUrl -MigrationPackageAzureLocations $azureSource -Credentials $cred