Support Amazon S3 based storage for LFS

Add a storage implementation storing large objects in Amazon S3.
The AmazonS3Repository pre-signs download and upload requests.

AWS access and secret key are expected to be in the
$HOME/.aws/credentials file in the following format:

[default]
  accessKey = ...
  secretKey = ...

Use AWS version 4 request signing [1] because it is more secure and
supported by all regions. The version 3 signing is not supported in
newer regions.

In follow up changes we should:

- implement getVerifyAction() and do actual verification. Subclasses of
S3Repository can implement caching for object meta data (size) in order
to avoid extra roundtrips to S3. Verification should ensure that meta
data store and content of S3 storage are in sync

- HEAD request used in S3Repository.getSize() seems to always return
Content-length 0 in contrast to the documentation [2]. So getSize() does
detect if the object exists in S3 or not but in case the object exists
it always returns size 0

[1] http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
[2] https://forums.aws.amazon.com/thread.jspa?threadID=223616

Change-Id: Ic47f094928a259e5264c92b3aacf6d90210907a8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>

1 files changed, 19 insertions, 2 deletions

diff --git a/org.eclipse.jgit.pgm/resources/org/eclipse/jgit/pgm/internal/CLIText.properties b/org.eclipse.jgit.pgm/resources/org/eclipse/jgit/pgm/internal/CLIText.properties
index 544e234e3a..33fb3ba497 100644
--- a/org.eclipse.jgit.pgm/resources/org/eclipse/jgit/pgm/internal/CLIText.properties
+++ b/org.eclipse.jgit.pgm/resources/org/eclipse/jgit/pgm/internal/CLIText.properties
@@ -74,6 +74,12 @@ invalidHttpProxyOnlyHttpSupported=Invalid http_proxy: {0}: Only http supported.
 jgitVersion=jgit version {0}
 lineFormat={0}
 listeningOn=Listening on {0}
+lfsNoAccessKey=No accessKey in {0}
+lfsNoSecretKey=No secretKey in {0}
+lfsProtocolUrl=LFS protocol URL: {0}
+lfsStoreDirectory=LFS objects stored in: {0}
+lfsStoreUrl=LFS store URL: {0}
+lfsUnknownStoreType="Unknown LFS store type: {0}"
 mergeConflict=CONFLICT(content): Merge conflict in {0}
 mergeCheckoutConflict=error: Your local changes to the following files would be overwritten by merge:
 mergeFailed=Automatic merge failed; fix conflicts and then commit the result
@@ -107,6 +113,7 @@ metaVar_file=FILE
 metaVar_filepattern=filepattern
 metaVar_gitDir=GIT_DIR
 metaVar_hostName=HOSTNAME
+metaVar_lfsStorage=STORAGE
 metaVar_linesOfContext=lines
 metaVar_message=message
 metaVar_n=n
@@ -123,9 +130,11 @@ metaVar_refs=REFS
 metaVar_refspec=refspec
 metaVar_remoteName=name
 metaVar_revision=REVISION
+metaVar_s3Bucket=BUCKET
+metaVar_s3Region=REGION
+metaVar_s3StorageClass=STORAGE-CLASS
 metaVar_seconds=SECONDS
 metaVar_service=SERVICE
-metaVar_lfsStorage=STORAGE
 metaVar_treeish=tree-ish
 metaVar_uriish=uri-ish
 metaVar_url=URL
@@ -169,6 +178,7 @@ remoteMessage=remote: {0}
 remoteRefObjectChangedIsNotExpectedOne=remote ref object changed - is not expected one {0}
 remoteSideDoesNotSupportDeletingRefs=remote side does not support deleting refs
 repaint=Repaint
+s3InvalidBucket=Invalid S3 bucket ''{0}''
 serviceNotSupported=Service ''{0}'' not supported
 skippingObject=skipping {0} {1}
 statusFileListFormat=\t%1$s
@@ -221,7 +231,7 @@ usage_Glog=View commit history as a graph
 usage_IndexPack=Build pack index file for an existing packed archive
 usage_LFSDirectory=Directory to store large objects
 usage_LFSPort=Server http port
-usage_LFSRunStore=fs: store lfs objects in file system
+usage_LFSRunStore=Store (fs | s3), store lfs objects in file system or Amazon S3
 usage_LFSStoreUrl=URL of the LFS store
 usage_LongFormat=Always output the long format
 usage_LsRemote=List references in a remote repository
@@ -240,6 +250,12 @@ usage_RepositoryToReceiveInto=Repository to receive into
 usage_RevList=List commit objects in reverse chronological order
 usage_RevParse=Pick out and massage parameters
 usage_RevParseAll=Show all refs found in refs/
+usage_S3Bucket=S3 bucket name
+usage_S3Expiration=Authorization validity in seconds, default 60 sec
+usage_S3Region=S3 region (us-east-1 | us-west-1 | us-west-2 | eu-west-1 |\
+ eu-central-1 | ap-southeast-1 | ap- southeast-2 | ap-northeast-1 |\
+ ap-northeast-2 | sa-east-1), see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
+usage_S3StorageClass=S3 storage class (REDUCED_REDUNDANCY | STANDARD), default REDUCED_REDUNDANCY
 usage_ServerSideBackendForJgitFetch=Server side backend for 'jgit fetch'
 usage_ServerSideBackendForJgitPush=Server side backend for 'jgit push'
 usage_ShowCacheTree=Show cache tree
@@ -353,6 +369,7 @@ usage_resetHard=Resets the index and working tree
 usage_resetSoft=Resets without touching the index file nor the working tree
 usage_resetMixed=Resets the index but not the working tree
 usage_runLfsStore=Run LFS Store in a given directory
+usage_S3NoSslVerify=Skip verification of Amazon server certificate and hostname
 usage_setTheGitRepositoryToOperateOn=set the git repository to operate on
 usage_show=display one commit
 usage_showRefNamesMatchingCommits=Show ref names matching commits