AWS S3 and PowerShell Tutorial

Introduction

This is going to be a short tutorial on how to use two tools, PowerShell and Amazon AWS S3, to easily create and update files in the cloud. This can be used to maintain a repository of files that you may need to access from multiple systems, an online archive, or as a cloud backup mechanism which you can run automatically with Task Scheduler.

The advantage of using S3 as opposed to something like Dropbox is that S3 is a cloud storage solution, so you can push files to it without having it sync an entire folder. While it's not free, it's also a lot cheaper than many professional backup solutions out there, and can work fine for small to medium deployments.

Creating the needed AWS accounts

  1. The first thing you need to do is register for an AWS account if you don't have one already at http://aws.amazon.com/. Once created, verified and provisioned with a payment method, this will allow you to use any Amazon cloud services.
  2. Next, you need to create a user account. This is done by clicking the Services button at the top, then IAM, which is their identity service. Creating users is important because you don't want to use your root account for any more than necessary.
  3. Click on Users on the left side, then Create a new user. You can give it any name, but make sure you download the credentials before finishing. This will give you a text file with two important items: the access key and the secret you will need to connect to the API with PowerShell.
  4. Next, you need to assign this user to a group. Click on Groups on the left side and Create a new group. In the Permissions tab, you can pick from one of the many templates that Amazon has to define what the user will be able to do. Pick one that allows it to use the S3 API fully:
  5. Go back to your user, select it, then click on User actions. From there, add the user to the group you just made.

Making your S3 bucket

  1. Go back to the main AWS console by clicking the box on the upper left, and select S3. Any file stored in S3 has to be placed inside of buckets, so we will create one.
  2. Click on Create bucket and enter a name for it. You can optionally check some of the Properties for it such as versioning, encryption, permissions and so on.
  3. The last step before we can leave the web browser is downloading the AWS PowerShell tools. Go to http://aws.amazon.com/powershell/ and click on AWS Tools for Windows.

Using the PowerShell tools

  1. The tools are very simple to install, simply click Next until done. The setup will create the necessary files and add Windows PowerShell for AWS in your Start menu. Alternatively, you can use Import-Module AWSPowerShell in a normal PS window.
  2. Start the PowerShell tools, and the first thing it should ask you is for your access key and secret that we downloaded earlier. This will allow you to use the various commands as this user. Alternatively, you can use the Set-AWSCredentials cmdlet.

From here, you can use the various command line tools to upload files:

<powershell>
Write-S3Object -BucketName mybucket -Key path/to/file.ext -File C:\file.ext

Or even a whole folder:

<powershell>
Write-S3Object -BucketName mybucket -KeyPrefix path/to/ -Folder C:\temp

You can get a list of items stored in your bucket with:

<powershell>
Get-S3Object -BucketName mybucket

Finally, get the file back with:

<powershell>
Read-S3Object -BucketName mybucket -Key path/to/file.ext -File C:\newfile.ext

Getting more advanced

The advantage of using S3 and PowerShell is how you can create scripts to automatically accomplish tasks, such as daily backups. If you want to use these commands as part of a script, you're going to have to manage authentication yourself. You can do it with this command:

<powershell>
Set-AWSCredentials -AccessKey mykey -SecretKey mysecret -StoreAs s3profile

Then, add the -ProfileName s3profile argument to any command we've seen above.

S3 also has advanced features you can use such as encryption and versioning, to keep track of file changes. There are also many tools out there to work with S3 storage, such as S3 Browser for Windows which gives you an Explorer style interface, Linux tools and even mobile apps.

Conclusion

One thing to keep in mind is that S3 isn't free, so make sure you review your bill before uploading your whole library! Also note that the AWS PowerShell tools work for other services like EC2 and SNS as well. Feel free to explore them in the official documentation.



© 2007-2018 Patrick Lambert - All resources on this site are provided under the MIT License - You can contact me at: contact@dendory.net