Skip to content

Configure aws cli

The aws s3api is useful for doing advanced s3 operations, e.g. dealing with object versions. The following explains how to set this up with our s3.cern.ch endpoint.

Setting up aws

All of the information required to set up aws-cli can be found in the existing .s3cfg file used when using s3cmd.

$> yum install awscli 
$> aws configure
AWS Access Key ID [None]: <your access key> 
AWS Secret Access Key [None]: <your secret key>
Default region name [None]:
Default output format [None]:

Listing buckets using aws-cli

$> aws --endpoint-url=https://s3.cern.ch s3api list-buckets
{
  "Buckets": [
     {
         "Name": <bucket1>,
         "CreationDate": <timestamp> 
     },
     {
       ....
     }
   ],
   "Owner": {
        "DisplayName": <owner>,
        "ID": <owner id>
    }

}

Deleting all object versions

We provide here a script to help user make sure all versions of their objects are deleted.

Usage:

$> ./s3-delete-all-object-versions.sh -b <bucket> [-f]
   -b: bucket name to be cleaned up
   -f: if omitted, the script will simply display a summary of actions. Add -f to execute them. 

Copying files to S3 using aws-cli

Single file cp

The aws tool provides a cp command to move files to your s3 bucket:

$> aws --endpoint-url=https://s3.cern.ch/ s3 cp <file> s3://<your-bucket>/
upload: ./<file> to s3://<your-bucket>/<file>

Whole directory

Using the --recursive flag you can transfer a whole directory at a time.

$> aws --endpoint-url=https://s3.cern.ch/ s3 cp <your-directory> s3://<your-bucket>/ --recursive
upload: <your-directory>/<file0> to s3://<your-bucket>/<file0>
upload: <your-directory>/<file1> to s3://<your-bucket>/<file1>
...
upload: <your-directory>/<fileN> to s3://<your-bucket>/<fileN>

You can the use aws ls to check that your files have been properly uploaded:

$> aws --endpoint-url=https://s3.cern.ch/ s3 ls s3://<your bucket>/
2019-10-25 11:31:40          <size> <file0>
2019-10-25 11:31:40          <size> <file1>
...
2019-10-25 11:31:40          <size> <fileN>

Additionally, aws cp provides an --exclude flag to filter files not to be transfered, the syntax is: --exclude "<regex>"

AWS reference manual