Object Storage management through s3cmd
s3cmd is a convenient command-line tool to manipulate S3 buckets. The .s3cfg file requires the following configuration to work with our S3 cluster:
host_base = s3.cern.ch
host_bucket = %(bucket)s.s3.cern.ch
access_key = <your access key goes here>
secret_key = <your secret key goes here>
We also highly recommend:
use_https = True
Simple script to create credentials for s3cmd and fill automatically the template
Log into lxplus8, and set the environment for the project you want to access. Execute the following snippet to create a new pair of credentials as well as the configuration file to be used by s3cmd.
S3_HOST=$(openstack catalog show s3 -f json -c endpoints | jq --raw-output '.endpoints[] | select(.interface | contains("public")) | .url' | cut -f3 -d/)
ACCESS_KEY=$(openstack ec2 credentials create -f value -c access)
SECRET_KEY=$(openstack ec2 credentials show $ACCESS_KEY -f value -c secret)
cat << EOF > S3CONFIG.cfg
host_base = ${S3_HOST}
host_bucket = %(bucket)s.${S3_HOST}
access_key = ${ACCESS_KEY}
secret_key = ${SECRET_KEY}
use_https = True
EOF
If you have access to different projects, please rename the file S3CONFIG.cfg accordingly.
In all s3cmd commands you need to point to this configuration file through --config=FILE, if you place it in $HOME/.s3cfg it will be taken as default.
Create a bucket
Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file
In order to create a bucket in the Object Storage service, you can use s3cmd with the following command.
[lxplus]$ s3cmd mb s3://BUCKET
Bucket 's3://BUCKET/' created
[lxplus]$ s3cmd ls
2018-11-19 13:02 s3://BUCKET
Once created you can start adding files.
Add files to a bucket
Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file
You can add files to the bucket with the following s3cmd commands:
[lxplus]$ s3cmd put ~/test.json s3://BUCKET
upload: '/home/username/test.json' -> 's3://BUCKET/test.json' [1 of 1]
3511 of 3511 100% in 0s 7.68 kB/s done
[lxplus]$ s3cmd -c FILENAME.cfg ls s3://BUCKET
2018-11-19 13:07 3511 s3://BUCKET/test.json
s3cmd also allows to set additional properties to the objects stored.
In the example below, we set the content type with the --mime-type option
and the cache-control parameter to 1 hour with --add-header.
s3cmd put --mime-type='application/json' --add-header='Cache-Control: max-age=3600' ~/test.json s3://BUCKET
Remove files from a bucket
Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file
You can delete files from the bucket with the following s3cmd command:
[lxplus]$ s3cmd -c FILENAME.cfg del s3://BUCKET/test.json
delete: 's3://BUCKET/test.json'
[lxplus]$ s3cmd -c FILENAME.cfg ls s3://BUCKET
[lxplus]$
Delete the bucket
Note: We are assuming that the configuration file is placed in $HOME/.s3cfg . If it is not the case you need to add the parameter --config=FILENAME with the location of your configuration file
Once the bucket is empty you can delete it with the following command:
[lxplus]$ s3cmd -c FILENAME.cfg rb s3://BUCKET
Bucket 's3://BUCKET/' removed
[lxplus]$ s3cmd -c FILENAME.cfg ls
[lxplus]$
Advanced features
Object Expiration with s3cmd
You can set an object expiration policy on a bucket, so that objects older than a particular age will be deleted automatically. The expiration policy can have a prefix, an effective date, and number of days to expire after.
s3cmd v2.0.0 can be used to set or review the policy:
[lxplus]$ s3cmd expire s3://dvanders-test --expiry-days 2
Bucket 's3://dvanders-test/': expiration configuration is set.
[lxplus]$ s3cmd getlifecycle s3://dvanders-test
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>ir0smpb610i0lthrl31jpxzegwximbcz3rrgb1he2yfxgudm</ID>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>2</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
Additional s3cmd expire options include:
--expiry-date=EXPIRY_DATE
Indicates when the expiration rule takes effect. (only
for [expire] command)
--expiry-days=EXPIRY_DAYS
Indicates the number of days after object creation the
expiration rule takes effect. (only for [expire]
command)
--expiry-prefix=EXPIRY_PREFIX
Identifying one or more objects with the prefix to
which the expiration rule applies. (only for [expire]
command)
and these commands can set/get/delete a lifecycle policy:
Upload a lifecycle policy for the bucket
s3cmd setlifecycle FILE s3://BUCKET
Get a lifecycle policy for the bucket
s3cmd getlifecycle s3://BUCKET
Remove a lifecycle policy for the bucket
s3cmd dellifecycle s3://BUCKET
Static websites with s3cmd
s3.cern.ch supports S3 Static Websites only via HTTP.
For example:
[lxplus]$ s3cmd mb s3://mywebsite
Bucket 's3://mywebsite/' created
[lxplus]$ s3cmd ws-create s3://mywebsite
Bucket 's3://mywebsite/': website configuration created.
[lxplus]$ s3cmd put -P index.html s3://mywebsite/
upload: 'index.html' -> 's3://mywebsite/index.html' [1 of 1]
Public URL of the object is: http://mywebsite.s3.cern.ch/index.html
Now browse to http://mywebsite.s3-website.cern.ch/.
Note: DNS is case-insensitive, so please use lower-case bucket names when configuring an S3 Static Website.
Also note that an FQDN static website may be hosted in S3 using the FQDN as the bucket name and by creating a DNS CNAME of that FQDN to <bucketname>.s3-website.cern.ch.
For instance:
- create a bucket
mywebsite.cern.ch - create a CNAME record from
mywebsite.cern.chtomywebsite.cern.ch.s3-website.cern.ch. - access http://mywebsite.cern.ch once DNS changes have been propagated
Bucket policy
s3.cern.ch supports S3 Bucket policy to restrict or grant capabilities to roles accessing resources.
Bucket policies can get quite large, note that there is a 20 kB size limit per policy.
Grant another S3 user Read Access to a bucket
{
"Version": "2012-10-17",
"Statement": [
{
"Action": ["s3:GetObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3:::mybucket/*",
"Principal": {
"AWS": [
"arn:aws:iam:::user/others3user"
]
}
}
]
}
Apply policy on "mybucket":
[lxplus]$ s3cmd setpolicy mypolicy.json s3://mybucket
Grant Read-Only Permission to any public anonymous user
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"AddPerm",
"Effect":"Allow",
"Principal": "*",
"Action":["s3:GetObject"],
"Resource":["arn:aws:s3:::mybucket/*"]
}
]
}