Azure Storage – Magic xpi : How to Get the Content list of a Blob Container and Download BlockBlobs locally using REST API

Goal : List a container blob storage and download and save each content (blockblob) in a file locally

Describe the different steps for connecting Magic xpi with Azure Storage using REST API

1/ Check that you can call the Azure Storage REST API using Postman with your credentials using Shared Key

(*) Instead of Shared Key, Authorization can be done with Azure Active Directory(Microsoft Entra ID)

For example, the URL should be in this format : https://{{azure_storage_account}}.blob.core.windows.net/devtest/?restype=container&comp=list

to get the containers list

You need to set {{azure_storage_account}}, {{header_date}} and {{header_authorization}} variables.

(*) The main difficulty is to generate the {{header_authorization}} key that must be in the format : SharedKey<space>{{AccountName}}:{{SignatureEncoded}}

example : SharedKey stmagictrfdevtest:+t5tQFW23cZ63NUZt4gQMuBUSBPSh52qa+Q5NOoFCEI=

In the Postman project, you can use the Pre-request Script to set the 3 parameters below

You can use Microsoft Azure Storage Explorer to retrieve your azure storage account and check that you can list the content of a blob container.

2/ In other to set the {{header_authorization}} with Magic xpi, a solution is to use a powershell script that generates the signature and the header_date for the right operation (GET, PUT, LIST)

Example for downloading a blob :

In this powershell script, we redirect the encoded signature and the header_date in the magic xpi project directory

(*) Be aware that you need to have a different signature for listing (LIST), downloading (GET) or uploading (PUT)

(*) Documentation reference : https://kb.sos-berlin.com/display/PKB/Feature+Proposal+-+Azure+Blob+Storage+-+Shared+Key+Authentication#FeatureProposalAzureBlobStorageSharedKeyAuthentication-Example

3/ Use a FlowData component to update a flow variable for the AccountName

4/ Drag and Drop a FileManagement component and use Command Line method to call a bat file that will call your powershell script

4/ Use a FlowData component to update 3 flow variables (F.signatureEncoded, F.Auth and F.UTCdate

F.Auth will hold the {{header_authorization}}

F.SignatureEncoded with : UTF8FromAnsi (RepStr (RepStr (File2Blb (‘<magic xpi installation directory>\Runtime\projects\AzureStorage2\AzureStorage2\signatureencoded\token.txt’),ASCIIChr (13),”),ASCIIChr (10),”))

F.Auth with : ‘SharedKey ‘&Trim (F.AccountName)&’:’&Trim (F.SignatureEncoded)

F.UtCdate with : File2Blb (‘<magic xpi installation directory>\Runtime\projects\AzureStorage2\AzureStorage2\signatureencoded\date.txt’)

5/ Create a REST Client Magic xpi resource to call Azure Storage (Project=>Settings=>Resources)

Define 4 header parameters (Authorization, x-ms-date, x-ms-version, x-ms-blob-type) and 2 query parameters (restype, comp)

6/ Use a REST Client connector to make the call to azure storage REST API.

7/ Now you need to parse the xml result with the Datamapper component and make a loop to download each blockblob

Drag and Drop a Datamapper component and configure an XML as a source and a CallFlow as destination

8/ In the called Flow, you need to generate a signature for the desired blockblob you want to download

Like in step 2/, you can call a powershell and pass the blockblob name to generate the signature.

in the bat file, you call the powershell script :

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass .\sas2download.ps1 %1

Powershell should be like this to receive the parameter.

9/ After getting the tokendownload.txt and the datedownload.txt, you can proceed like in step 4/ to update the Authorization

10/ Add a path in the REST Client Magic xpi resource to call Azure Storage Download (Project=>Settings=>Resources)

11/ Use the REST Client connector to make the call using Get operation.

12/ Drag & Drop Filemanagement Component to save the content on a file locally with a timestamp.

Assume that the Response of the previous step (REST Client Download) goes into C.UserBlob.

13/ Document reference : https://learn.microsoft.com/en-us/rest/api/storageservices/list-containers2?tabs=shared-key

Documentation reference : https://kb.sos-berlin.com/display/PKB/Feature+Proposal+-+Azure+Blob+Storage+-+Shared+Key+Authentication#FeatureProposalAzureBlobStorageSharedKeyAuthentication-Example

Laisser un commentaire

Votre adresse e-mail ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Retour en haut