384 lines
21 KiB
XML
384 lines
21 KiB
XML
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE section [
|
|
<!-- Some useful entities borrowed from HTML -->
|
|
<!ENTITY ndash "–">
|
|
<!ENTITY mdash "—">
|
|
<!ENTITY hellip "…">
|
|
<!ENTITY nbsp " ">
|
|
<!-- Useful for describing APIs -->
|
|
<!ENTITY COPY '<command xmlns="http://docbook.org/ns/docbook">COPY</command>'>
|
|
<!ENTITY GET '<command xmlns="http://docbook.org/ns/docbook">GET</command>'>
|
|
<!ENTITY HEAD '<command xmlns="http://docbook.org/ns/docbook">HEAD</command>'>
|
|
<!ENTITY PUT '<command xmlns="http://docbook.org/ns/docbook">PUT</command>'>
|
|
<!ENTITY POST '<command xmlns="http://docbook.org/ns/docbook">POST</command>'>
|
|
<!ENTITY DELETE '<command xmlns="http://docbook.org/ns/docbook">DELETE</command>'>
|
|
]>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
|
xml:id="large-object-creation">
|
|
<title>Large objects</title>
|
|
<para>By default, the content of an object cannot be greater than
|
|
5 GB. However, you can use a number of smaller objects to
|
|
construct a large object. The large object is comprised of two
|
|
types of objects:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><emphasis role="bold">Segment objects</emphasis>
|
|
store the object content. You can divide your content
|
|
into segments, and upload each segment into its own
|
|
segment object. Segment objects do not have any
|
|
special features. You create, update, download, and
|
|
delete segment objects just as you would normal
|
|
objects.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>A <emphasis role="bold">manifest object</emphasis>
|
|
links the segment objects into one logical large
|
|
object. When you download a manifest object, Object
|
|
Storage concatenates and returns the contents of the
|
|
segment objects in the response body of the request.
|
|
This behavior extends to the response headers returned
|
|
by &GET; and &HEAD; requests. The
|
|
<literal>Content-Length</literal> response header
|
|
value is the total size of all segment objects. Object
|
|
Storage calculates the <literal>ETag</literal>
|
|
response header value by taking the
|
|
<literal>ETag</literal> value of each segment,
|
|
concatenating them together, and returning the MD5
|
|
checksum of the result. The manifest object types
|
|
are:</para>
|
|
<variablelist wordsize="10">
|
|
<varlistentry>
|
|
<term><emphasis role="bold">Static large
|
|
objects</emphasis></term>
|
|
<listitem>
|
|
<para>The manifest object content is an
|
|
ordered list of the names of the segment
|
|
objects in JSON format. See <xref
|
|
linkend="static-large-objects"
|
|
/>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term><emphasis role="bold">Dynamic large
|
|
objects</emphasis></term>
|
|
<listitem>
|
|
<para>The manifest object has no content but
|
|
it has a
|
|
<literal>X-Object-Manifest</literal>
|
|
metadata header. The value of this header
|
|
is
|
|
<literal>{container}/{prefix}</literal>,
|
|
where <literal>{container}</literal> is
|
|
the name of the container where the
|
|
segment objects are stored, and
|
|
<literal>{prefix}</literal> is a
|
|
string that all segment objects have in
|
|
common. See <xref
|
|
linkend="dynamic-large-object-creation"
|
|
/>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<note>
|
|
<para>If you make a © request by using a manifest object
|
|
as the source, the new object is a normal, and not a
|
|
segment, object. If the total size of the source segment
|
|
objects exceeds 5 GB, the © request fails.
|
|
However, you can make a duplicate of the manifest object
|
|
and this new object can be larger than 5 GB.</para>
|
|
</note>
|
|
<section xml:id="static-large-objects">
|
|
<title>Static large objects</title>
|
|
<para>To create a static large object, divide your content
|
|
into pieces and create (upload) a segment object to
|
|
contain each piece.</para>
|
|
<para>You must record the <literal>ETag</literal> response
|
|
header that the &PUT; operation returns. Alternatively,
|
|
you can calculate the MD5 checksum of the segment prior to
|
|
uploading and include this in the <literal>ETag</literal>
|
|
request header. This ensures that the upload cannot
|
|
corrupt your data.</para>
|
|
<para>List the name of each segment object along with its size
|
|
and MD5 checksum in order.</para>
|
|
<para>Create a manifest object. Include the
|
|
<parameter>?multipart-manifest=put</parameter> query
|
|
string at the end of the manifest object name to indicate
|
|
that this is a manifest object.</para>
|
|
<para>The body of the &PUT; request on the manifest object
|
|
comprises a json list, where each element contains the
|
|
following attributes:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><code>path</code>. The container and object name
|
|
in the format:
|
|
<code>{container-name}/{object-name}</code></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>etag</code>. The MD5 checksum of the
|
|
content of the segment object. This value must
|
|
match the <literal>ETag</literal> of that
|
|
object.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>size_bytes</code>. The size of the segment
|
|
object. This value must match the
|
|
<literal>Content-Length</literal> of that
|
|
object.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<example>
|
|
<title>Static large object manifest list</title>
|
|
<para>This example shows three segment objects. You can
|
|
use several containers and the object names do not
|
|
have to conform to a specific pattern, in contrast to
|
|
dynamic large objects.</para>
|
|
<literallayout class="monospaced"><xi:include href="samples/slo-manifest-example.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>The <literal>Content-Length</literal> request header
|
|
must contain the length of the json content—not the
|
|
length of the segment objects. However, after the &PUT;
|
|
operation completes, the <literal>Content-Length</literal>
|
|
metadata is set to the total length of all the object
|
|
segments. A similar situation applies to the
|
|
<literal>ETag</literal>. If used in the &PUT;
|
|
operation, it must contain the MD5 checksum of the json
|
|
content. The <literal>ETag</literal> metadata value is
|
|
then set to be the MD5 checksum of the concatenated
|
|
<literal>ETag</literal> values of the object segments.
|
|
You can also set the <literal>Content-Type</literal>
|
|
request header and custom object metadata.</para>
|
|
<para>When the &PUT; operation sees the
|
|
<parameter>?multipart-manifest=put</parameter> query
|
|
parameter, it reads the request body and verifies that
|
|
each segment object exists and that the sizes and ETags
|
|
match. If there is a mismatch, the &PUT;operation
|
|
fails.</para>
|
|
<para>If everything matches, the manifest object is created.
|
|
The <literal>X-Static-Large-Object</literal> metadata is
|
|
set to <literal>true</literal> indicating that this is a
|
|
static object manifest.</para>
|
|
<para>Normally when you perform a &GET; operation on the
|
|
manifest object, the response body contains the
|
|
concatenated content of the segment objects. To download
|
|
the manifest list, use the
|
|
<parameter>?multipart-manifest=get</parameter> query
|
|
parameter. The resulting list is not formatted the same as
|
|
the manifest you originally used in the &PUT;
|
|
operation.</para>
|
|
<para>If you use the &DELETE; operation on a manifest object,
|
|
the manifest object is deleted. The segment objects are
|
|
not affected. However, if you add the
|
|
<parameter>?multipart-manifest=delete</parameter>
|
|
query parameter, the segment objects are deleted and if
|
|
all are successfully deleted, the manifest object is also
|
|
deleted.</para>
|
|
<para>To change the manifest, use a &PUT; operation with the
|
|
<parameter>?multipart-manifest=put</parameter> query
|
|
parameter. This request creates a manifest object. You can
|
|
also update the object metadata in the usual way.</para>
|
|
</section>
|
|
<section xml:id="dynamic-large-object-creation">
|
|
<title>Dynamic large objects</title>
|
|
<para>You must segment objects that are larger than 5 GB
|
|
before you can upload them. You then upload the segment
|
|
objects like you would any other object and create a
|
|
dynamic large manifest object. The manifest object tells
|
|
Object Storage how to find the segment objects that
|
|
comprise the large object. The segments remain
|
|
individually addressable, but retrieving the manifest
|
|
object streams all the segments concatenated. There is no
|
|
limit to the number of segments that can be a part of a
|
|
single large object.</para>
|
|
<para>To ensure the download works correctly, you must upload
|
|
all the object segments to the same container and ensure
|
|
that each object name is prefixed in such a way that it
|
|
sorts in the order in which it should be concatenated. You
|
|
also create and upload a manifest file. The manifest file
|
|
is a zero-byte file with the extra
|
|
<literal>X-Object-Manifest</literal>
|
|
<code>{container}/{prefix}</code> header, where
|
|
<code>{container}</code> is the container the object
|
|
segments are in and <code>{prefix}</code> is the common
|
|
prefix for all the segments. You must UTF-8-encode and
|
|
then URL-encode the container and common prefix in the
|
|
<literal>X-Object-Manifest</literal> header.</para>
|
|
<para>It is best to upload all the segments first and then
|
|
create or update the manifest. With this method, the full
|
|
object is not available for downloading until the upload
|
|
is complete. Also, you can upload a new set of segments to
|
|
a second location and update the manifest to point to this
|
|
new location. During the upload of the new segments, the
|
|
original manifest is still available to download the first
|
|
set of segments.</para>
|
|
<example>
|
|
<title>Upload segment of large object request:
|
|
HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/large-object-upload-segment-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>No response body is returned. A status code of
|
|
<returnvalue>2<replaceable>nn</replaceable></returnvalue>
|
|
(between 200 and 299, inclusive) indicates a successful
|
|
write; status <errorcode>411</errorcode>
|
|
<errortext>Length Required</errortext> denotes a missing
|
|
<literal>Content-Length</literal> or
|
|
<literal>Content-Type</literal> header in the request.
|
|
If the MD5 checksum of the data written to the storage
|
|
system does NOT match the (optionally) supplied ETag
|
|
value, a <errorcode>422</errorcode>
|
|
<errortext>Unprocessable Entity</errortext> response is
|
|
returned.</para>
|
|
<para>You can continue uploading segments like this example
|
|
shows, prior to uploading the manifest.</para>
|
|
<example>
|
|
<title>Upload next segment of large object request:
|
|
HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/large-object-upload-next-segment-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>Next, upload the manifest you created that indicates the
|
|
container the object segments reside within. Note that
|
|
uploading additional segments after the manifest is
|
|
created causes the concatenated object to be that much
|
|
larger but you do not need to recreate the manifest file
|
|
for subsequent additional segments.</para>
|
|
<example>
|
|
<title>Upload manifest request: HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/upload-manifest-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<example>
|
|
<title>Upload manifest response: HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/upload-manifest-resp.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>The <literal>Content-Type</literal> in the response for
|
|
a &GET; or &HEAD; on the manifest is the same as the
|
|
<literal>Content-Type</literal> set during the &PUT;
|
|
request that created the manifest. You can easily change
|
|
the <literal>Content-Type</literal> by reissuing the &PUT;
|
|
request.</para>
|
|
</section>
|
|
<section xml:id="comparison_dynamic_static_objects">
|
|
<title>Comparison of static and dynamic large objects</title>
|
|
<para>While static and dynamic objects have similar behavior,
|
|
this table describes their differences:</para>
|
|
<table rules="all">
|
|
<caption>Static and dynamic large objects</caption>
|
|
<col width="20%"/>
|
|
<col width="40%"/>
|
|
<col width="40%"/>
|
|
<thead>
|
|
<tr>
|
|
<td/>
|
|
<th>Static large object</th>
|
|
<th>Dynamic large object</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<th>End-to-end integrity</th>
|
|
<td><para>Assured. The list of segments includes
|
|
the MD5 checksum (<literal>ETag</literal>)
|
|
of each segment. You cannot upload the
|
|
manifest object if the
|
|
<literal>ETag</literal> in the list
|
|
differs from the uploaded segment object.
|
|
If a segment is somehow lost, an attempt
|
|
to download the manifest object results in
|
|
an error.</para></td>
|
|
<td><para>Not guaranteed. The eventual consistency
|
|
model means that although you have
|
|
uploaded a segment object, it might not
|
|
appear in the container listing until
|
|
later. If you download the manifest before
|
|
it appears in the container, it does not
|
|
form part of the content returned in
|
|
response to a &GET; request.</para></td>
|
|
</tr>
|
|
<tr>
|
|
<th>Upload order</th>
|
|
<td><para>You must upload the segment objects
|
|
before you upload the manifest
|
|
object.</para></td>
|
|
<td><para>You can upload manifest and segment
|
|
objects in any order. You are recommended
|
|
to upload the manifest object after the
|
|
segments in case a premature download of
|
|
the manifest occurs. However, this is not
|
|
enforced.</para></td>
|
|
</tr>
|
|
<tr>
|
|
<th>Removal or addition of segment objects</th>
|
|
<td><para>You cannot add or remove segment objects
|
|
from the manifest. However, you can create
|
|
a completely new manifest object of the
|
|
same name with a different manifest
|
|
list.</para></td>
|
|
<td><para>You can upload new segment objects or
|
|
remove existing segments. The names must
|
|
simply match the
|
|
<literal>{prefix}</literal> supplied
|
|
in
|
|
<literal>X-Object-Manifest</literal>.</para></td>
|
|
</tr>
|
|
<tr>
|
|
<th>Segment object size and number</th>
|
|
<td><para>Segment objects must be at least
|
|
1 MB in size (by default). The final
|
|
segment object can be any size. At most,
|
|
1000 segments are supported (by
|
|
default).</para></td>
|
|
<td><para>Segment objects can be any
|
|
size.</para></td>
|
|
</tr>
|
|
<tr>
|
|
<th>Segment object container name</th>
|
|
<td><para>The manifest list includes the container
|
|
name of each object. Segment objects can
|
|
be in different containers.</para></td>
|
|
<td><para>All segment objects must be in the same
|
|
container.</para></td>
|
|
</tr>
|
|
<tr>
|
|
<th>Manifest object metadata</th>
|
|
<td><para>The object has
|
|
<literal>X-Static-Large-Object</literal>
|
|
set to <literal>true</literal>. You do not
|
|
set this metadata directly. Instead the
|
|
system sets it when you &PUT; a static
|
|
manifest object.</para></td>
|
|
<td><para>The <literal>X-Object-Manifest</literal>
|
|
value is the
|
|
<literal>{container}/{prefix}</literal>,
|
|
which indicates where the segment objects
|
|
are located. You supply this request
|
|
header in the &PUT; operation.</para></td>
|
|
</tr>
|
|
<tr>
|
|
<th>Copying the manifest object</th>
|
|
<td><para>Include the
|
|
<parameter>?multipart-manifest=get</parameter>
|
|
query string in the © request. The
|
|
new object contains the same manifest as
|
|
the original. The segment objects are not
|
|
copied. Instead, both the original and new
|
|
manifest objects share the same set of
|
|
segment objects.</para>
|
|
</td>
|
|
<td><para>The © operation does not create a
|
|
manifest object. To duplicate a manifest
|
|
object, use the &GET; operation to read
|
|
the value of
|
|
<literal>X-Object-Manifest</literal>
|
|
and use this value in the
|
|
<literal>X-Object-Manifest</literal>
|
|
request header in a &PUT; operation. This
|
|
creates a new manifest object that shares
|
|
the same set of segment objects as the
|
|
original manifest object.</para></td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</section>
|
|
</section>
|