in Break-Fix

Amazon S3 File Size Limitation – Solved

If you attempt to store a file larger than roughly 5GB, the Amazon service will generate the following error and fail to store the file:

<Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed object size</Message><ProposedSize>6091682399</ProposedSize><MaxSizeAllowed>5368709120</MaxSizeAllowed></Error>

One way around this limitation is use the GNU/Linux command split to divide the file into several smaller chunks, which are in turn stored in Amazon’s S3. It’s important to know the order of these chunks as the original file is split. The good news is that the split command preserves the order of the chunks by adding a suffix to each chunk. Here’s an example of the split command in action. In this example, we’re splitting file outfile.tmp, which is of size 2.3GB, into chunks of 1GB each:

# du -sh outfile.tmp
2.3G    outfile.tmp

# split -a 1 -b 1073741824 outfile.tmp outfile.tmp.

# ls -alh outfile.tmp.*
-rw-r–r– 1 root root 1.0G Apr 11 09:51 outfile.tmp.a
-rw-r–r– 1 root root 1.0G Apr 11 09:51 outfile.tmp.b
-rw-r–r– 1 root root 274M Apr 11 09:51 outfile.tmp.c

As you can see, split is appending alphabetical letters to the end of the split file name. This comes handy when reassembling the file. Here’s a quick proof of concept following our earlier example:

# md5sum outfile.tmp
e1f4bbcfc2309b3c7ea48028c3f1c9e9 outfile.tmp
# cat outfile.tmp.a outfile.tmp.b outfile.tmp.c > reassembled.tmp
# md5sum reassembled.tmp
e1f4bbcfc2309b3c7ea48028c3f1c9e9 reassembled.tmp

All as planned! That’s all folks.

Write a Comment

Comment

Webmentions

  • Homepage

    … [Trackback]

    […] Informations on that Topic: blog.unixy.net/2009/04/amazons-s3-file-size-limitation-solved/ […]

  • ????????

    Amazon S3 File Size Limitation – Solved –

    Top baccarat Guide!

  • ????????

    Amazon S3 File Size Limitation – Solved –

    Yes, generally speaking, online casinos are safe to play at. However, like most businesses that are found online, there are good eggs and sadly multiple bad ones.

  • ????????

    Amazon S3 File Size Limitation – Solved –

    There are many statistics that it is possible to select from nowadays to decipher whether or not a league is good or not. Let us now take a better look at the ideal Serie A clubs. That bowl game determines the very best collegiate squad every year. The…

  • ??????

    Ten Factors That Affect Baccarat Site’s Longevity.

    What Everyone Is Saying About baccarat And What You Should Do

  • ??????

    […] There you can find 29281 more Info on that Topic: blog.unixy.net/2009/04/amazons-s3-file-size-limita… […]

  • Homepage

    … [Trackback]

    […] Read More: blog.unixy.net/2009/04/amazons-s3-file-size-limitation-solved/ […]

  • Cpanel File Upload Size – SH Xiang Gelila

    […] Amazon S3 File Size Limitation – Solved | UNIXy – If you attempt to store a file larger than roughly 5GB, the Amazon service will generate the following error and fail to store the […]

  • ??? ???

    ??? ???

    grand seiko

  • payday loans

    payday loans…

    Say “thanks” you to your mothers and fathers that they gave you the planet…

  • My Homepage

    … [Trackback]…

    […] Informations on that Topic: blog.unixy.net/2009/04/amazons-s3-file-size-limitation-solved/ […]…

  • Elektrische Zahnbuerste

    … [Trackback]…

    […] Informations on that Topic: blog.unixy.net/2009/04/amazons-s3-file-size-limitation-solved/ […]…