关键术语

简要概述

主要对对象存储一些基本概念做个介绍。

关键术语

术语中文名示例
endpoint访问域名s3.amazonaws.com、ds.test.opsaid.net
region存储地域cn-guangzhou
request_style请求风格Virtual hosted‐style 或 path‐style(不建议)
object_key对象名称hello/test.txt
bucket存储桶uptime
signature version签名版本支持 v4、v2

签名版本

标准 S3 有两种签名方式,分布为版本2、版本4。

计算 v2

Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature;

Signature = Base64( HMAC-SHA1( UTF-8-Encoding-Of(YourSecretAccessKey), UTF-8-Encoding-Of( StringToSign ) ) );

StringToSign = HTTP-Verb + "\n" +
	Content-MD5 + "\n" +
	Content-Type + "\n" +
	Date + "\n" +
	CanonicalizedAmzHeaders +
	CanonicalizedResource;

CanonicalizedResource = [ "/" + Bucket ] +
	<HTTP-Request-URI, from the protocol name up to the query string> +
	[ subresource, if present. For example "?acl", "?location", or "?logging"];

CanonicalizedAmzHeaders = <described below>

其中 CanonicalizedAmzHeaders 遵守如下计算方式:

步骤
将每个 HTTP 标头名称转换为小写。例如,“X-Amz-Date”改为“x-amz-date”
根据标头名称按字典顺序排列标头集
将相同名称的标头字段合并为一个“header-name:comma-separated-value-list”对,并按照 RFC 2616 中第 4.2 节中的规定,两个值之间不留空格。例如,可以将元数据标头“x-amz-meta-username: fred”和“x-amz-meta-username: barney”合并为单个标头“x-amz-meta-username: fred,barney”
通过将折叠空格(包括新建行)替换为单个空格,“展开”跨多个行的长标头(按照 RFC 2616 中第 4.2 节允许的方式)
删除标头中冒号周围的空格。例如,标头“x-amz-meta-username: fred,barney”改为“x-amz-meta-username:fred,barney”
最后,请向生成的列表中的每个标准化标头附加换行字符 (U+000A)。通过将此列表中所有的标头规范化为单个字符串,构建 CanonicalizedResource 元素

计算 v4

HTTP Authorization 请求头示例如下(增加换行是为了方便阅读,实际为空字符串):

Authorization: AWS4-HMAC-SHA256
Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request, 
SignedHeaders=host;range;x-amz-date,
Signature=fe5f80f77d5fa3beca038a248ff027d0445342fe2855ddc963176630326f1024
字段名称描述
AWS4-HMAC-SHA256使用版本 v4 签名,固定值
Credential用户 AccessKeyId和范围信息,格式:{your-access-key-id}/{date:YYYYMMDD}/{region}/s3/aws4_request
SignedHeaders已签名请求头的列表,只需包含请求头名字,用分号分隔,必须全部小写,并按字符顺序对其进行排序
Signature计算出的256位签名信息,以64个小写十六进制字符串形式表示

TODO;

数据结构

Object

// An object consists of data and its descriptive metadata.
type Object struct {
    _ struct{} `type:"structure"`

    // The algorithm that was used to create a checksum of the object.
    ChecksumAlgorithm []*string `type:"list" flattened:"true" enum:"ChecksumAlgorithm"`

    // The entity tag is a hash of the object. The ETag reflects changes only to
    // the contents of an object, not its metadata. The ETag may or may not be an
    // MD5 digest of the object data. Whether or not it is depends on how the object
    // was created and how it is encrypted as described below:
    //
    //    * Objects created by the PUT Object, POST Object, or Copy operation, or
    //    through the AWS Management Console, and are encrypted by SSE-S3 or plaintext,
    //    have ETags that are an MD5 digest of their object data.
    //
    //    * Objects created by the PUT Object, POST Object, or Copy operation, or
    //    through the AWS Management Console, and are encrypted by SSE-C or SSE-KMS,
    //    have ETags that are not an MD5 digest of their object data.
    //
    //    * If an object is created by either the Multipart Upload or Part Copy
    //    operation, the ETag is not an MD5 digest, regardless of the method of
    //    encryption.
    ETag *string `type:"string"`

    // The name that you assign to an object. You use the object key to retrieve
    // the object.
    Key *string `min:"1" type:"string"`

    // The date the Object was Last Modified
    LastModified *time.Time `type:"timestamp"`

    // The owner of the object
    Owner *Owner `type:"structure"`

    // Size in bytes of the object
    Size *int64 `type:"integer"`

    // The class of storage used to store the object.
    StorageClass *string `type:"string" enum:"ObjectStorageClass"`
}

常用接口

以下 HTTP 接口均以 虚拟主机风格展示。

Bucket 相关

AWS S3 接口HTTP 接口说明
CreateBucketPUT {bucket}.{endpoint}/创建存储桶
HeadBucketHEAD {bucket}.{endpoint}/验证存储桶是否存在或是否有权限
DeleteBucketDELETE {bucket}.{endpoint}/删除存储桶
ListBucketsGET {endpoint}/列出有权限的存储桶
GetBucketLocationGET /?location查询当前 bucket 所在区域

Object 相关

AWS S3 接口HTTP 接口说明
CopyObjectPUT {bucket}.{endpoint}/{object_key}拷贝对象
DeleteObjectDELETE {bucket}.{endpoint}/{object_key}删除对象
DeleteObjectsPOST {bucket}.{endpoint}/?delete批量删除
GetObjectGET {bucket}.{endpoint}/{object_key}获取对象
HeadObjectGET {bucket}.{endpoint}/{object_key}查询对象
ListObjectsGET {bucket}.{endpoint}/?delimiter=列出对象
ListObjectsV2GET {bucket}.{endpoint}/?list-type=2列出对象
PutObjectPUT {bucket}.{endpoint}/{object_key}创建对象

分片相关

AWS S3 接口HTTP 接口说明
CreateMultipartUploadPOST {bucket}.{endpoint}/{object_key}?uploadsxx
UploadPartPUT {bucket}.{endpoint}/{object_key}xx
CompleteMultipartUploadPOST {bucket}.{endpoint}/{object_key}?uploadId=xx
AbortMultipartUploadDELETE {bucket}.{endpoint}/{object_key}?uploadId=xx
UploadPartCopyPUT {bucket}.{endpoint}/{object_key}?partNumber=&uploadId=xx
ListMultipartUploadsGET {bucket}.{endpoint}/?uploadsxx
ListPartsGET {bucket}.{endpoint}/{object_key}?max-parts=&part-number-marker=&uploadId=xx