Skip to content

Object Storage management through s3cmd

s3cmd is a convenient command-line tool to manipulate S3 buckets. The .s3cfg file requires the following configuration to work with our S3 cluster:

host_base = s3.cern.ch
host_bucket = %(bucket)s.s3.cern.ch
access_key = <your access key goes here>
secret_key = <your secret key goes here>

We also highly recommend: 

use_https = True

Simple script to create credentials for s3cmd and fill automatically the template

Log into lxplus8, and set the environment for the project you want to access. Execute the following snippet to create a new pair of credentials as well as the configuration file to be used by s3cmd. 

S3_HOST=$(openstack catalog show s3 -f json -c endpoints | jq --raw-output '.endpoints[] | select(.interface | contains("public")) | .url' | cut -f3 -d/)
ACCESS_KEY=$(openstack ec2 credentials create -f value -c access)
SECRET_KEY=$(openstack ec2 credentials show $ACCESS_KEY -f value -c secret)

cat << EOF > S3CONFIG.cfg
host_base = ${S3_HOST}
host_bucket = %(bucket)s.${S3_HOST}
access_key = ${ACCESS_KEY}
secret_key = ${SECRET_KEY}
use_https = True
EOF

If you have access to different projects, please rename the file S3CONFIG.cfg accordingly.

In all s3cmd commands you need to point to this configuration file through --config=FILE, if you place it in $HOME/.s3cfg it will be taken as default.

Create a bucket

Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file

In order to create a bucket in the Object Storage service, you can use s3cmd with the following command. 

[lxplus]$ s3cmd mb s3://BUCKET
Bucket 's3://BUCKET/' created
[lxplus]$ s3cmd ls
2018-11-19 13:02  s3://BUCKET

Once created you can start adding files.

Add files to a bucket

Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file

You can add files to the bucket with the following s3cmd commands: 

[lxplus]$ s3cmd put ~/test.json s3://BUCKET
upload: '/home/username/test.json' -> 's3://BUCKET/test.json'  [1 of 1]
 3511 of 3511   100% in    0s     7.68 kB/s  done
[lxplus]$ s3cmd -c FILENAME.cfg ls s3://BUCKET
2018-11-19 13:07      3511   s3://BUCKET/test.json

s3cmd also allows to set additional properties to the objects stored. In the example below, we set the content type with the --mime-type option and the cache-control parameter to 1 hour with --add-header. 

s3cmd put --mime-type='application/json' --add-header='Cache-Control: max-age=3600' ~/test.json s3://BUCKET

Remove files from a bucket

Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file

You can delete files from the bucket with the following s3cmd command: 

[lxplus]$ s3cmd -c FILENAME.cfg del s3://BUCKET/test.json
delete: 's3://BUCKET/test.json'
[lxplus]$ s3cmd -c FILENAME.cfg ls s3://BUCKET
[lxplus]$

Delete the bucket

Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file

Once the bucket is empty you can delete it with the following command: 

[lxplus]$ s3cmd -c FILENAME.cfg rb s3://BUCKET
Bucket 's3://BUCKET/' removed
[lxplus]$ s3cmd -c FILENAME.cfg ls
[lxplus]$

Advanced features

Object Expiration with s3cmd

You can set an object expiration policy on a bucket, so that objects older than a particular age will be deleted automatically. The expiration policy can have a prefix, an effective date, and number of days to expire after.

s3cmd v2.0.0 can be used to set or review the policy:

[lxplus]$ s3cmd expire s3://dvanders-test --expiry-days 2
Bucket 's3://dvanders-test/': expiration configuration is set.

[lxplus]$ s3cmd getlifecycle s3://dvanders-test
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <ID>ir0smpb610i0lthrl31jpxzegwximbcz3rrgb1he2yfxgudm</ID>
        <Prefix/>
        <Status>Enabled</Status>
        <Expiration>
            <Days>2</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

Additional s3cmd expire options include: 

  --expiry-date=EXPIRY_DATE
                        Indicates when the expiration rule takes effect. (only
                        for [expire] command)
  --expiry-days=EXPIRY_DAYS
                        Indicates the number of days after object creation the
                        expiration rule takes effect. (only for [expire]
                        command)
  --expiry-prefix=EXPIRY_PREFIX
                        Identifying one or more objects with the prefix to
                        which the expiration rule applies. (only for [expire]
                        command)
and these commands can set/get/delete a lifecycle policy:

  Upload a lifecycle policy for the bucket
      s3cmd setlifecycle FILE s3://BUCKET
  Get a lifecycle policy for the bucket
      s3cmd getlifecycle s3://BUCKET
  Remove a lifecycle policy for the bucket
      s3cmd dellifecycle s3://BUCKET

Static websites with s3cmd

s3.cern.ch supports S3 Static Websites only via HTTP.

For example:

[lxplus]$ s3cmd mb s3://mywebsite
Bucket 's3://mywebsite/' created
[lxplus]$ s3cmd ws-create s3://mywebsite
Bucket 's3://mywebsite/': website configuration created.
[lxplus]$ s3cmd put -P index.html s3://mywebsite/
upload: 'index.html' -> 's3://mywebsite/index.html'  [1 of 1]
Public URL of the object is: http://mywebsite.s3.cern.ch/index.html

Now browse to http://mywebsite.s3-website.cern.ch/.

Note: DNS is case-insensitive, so please use lower-case bucket names when configuring an S3 Static Website.

Also note that an FQDN static website may be hosted in S3 using the FQDN as the bucket name and by creating a DNS CNAME of that FQDN to <bucketname>.s3-website.cern.ch.

For instance:

  • create a bucket mywebsite.cern.ch
  • create a CNAME record from mywebsite.cern.ch to mywebsite.cern.ch.s3-website.cern.ch.
  • access http://mywebsite.cern.ch once DNS changes have been propagated

Bucket policy

s3.cern.ch supports S3 Bucket policy to restrict or grant capabilities to roles accessing resources.

Bucket policies can get quite large, note that there is a 20 kB size limit per policy.

Grant another S3 user Read Access to a bucket

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["s3:GetObject"],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::mybucket/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam:::user/others3user"
        ]
      }
    }
  ]
}
Apply policy on "mybucket": 
[lxplus]$ s3cmd setpolicy mypolicy.json s3://mybucket

Grant Read-Only Permission to any public anonymous user

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"AddPerm",
      "Effect":"Allow",
      "Principal": "*",
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::mybucket/*"]
    }
  ]
}

More Information