LFS: Enable LFS support for the CLI, better error handling
Enable LFS support for the CLI by registering the according filters.
Errors during filter creation must be propagated up the call stack, as a
failure to create a filter should be treated as fatal if the filter is
required.
Change-Id: I3833757839bdda97cd01b6c21c1613d199e2692d
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
LFS: support merge/rebase/cherry-pick/diff/compare with LFS files
Respect merge=lfs and diff=lfs attributes where required to replace (in
memory) the content of LFS pointers with the actual blob content from
the LFS storage (and vice versa when staging/merging).
Does not implement general support for merge/diff attributes for any
other use case apart from LFS.
Change-Id: Ibad8875de1e0bee8fe3a1dffb1add93111534cae
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If JGit built in LFS support is enabled for the current repository (or
user/system), any existing pre-push hook will cause an exception for the
time beeing, as only a single pre-push hook is supported.
Thus either native pre-push hooks OR JGit built-in LFS support may be
enabled currently, but not both.
Change-Id: Ie7d2b90e26e948d9cca3d05a7a19489488c75895
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Transfer data in chunks of 8k Transferring data byte per byte is slow,
running checkout with CleanFilter on a 2.9MB file takes 20 seconds.
Using a buffer of 8k shrinks this time to 70ms.
Also register the filter commands in a way that the native GIT LFS can
be used alongside with JGit.
Implements auto-discovery of LFS server URL when cloning from a Gerrit
LFS server.
Change-Id: I452a5aa177dcb346d92af08b27c2e35200f246fd
Also-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
LfsServerText: Fix formatting of invalidPathInfo message
The message is formatted as:
Invalid pathInfo '/abc' does not match '/{SHA-256}'
but should be:
Invalid pathInfo: '/abc' does not match '/{SHA-256}'
(i.e. including a colon) to be consistent with other messages.
Change-Id: Ic38aa7d33dd02d7954b95c331a73919a90c69991
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
InvalidLongObjectIdException: Fix formatting of message
The message is formatted as:
Invalid id: : abcde...
but should be:
Invalid id: abcde...
Change-Id: Ie15cacdcf2f168edaee262e6cf8061ebfe9d998d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Pretty printing the response is useful for human readers, but most
(if not all) of the time, the response will be read by programs.
Remove it to avoid the additional overhead of the formatting and
extra bytes in the response. Adjust the test accordingly.
Note that LfsProtocolServlet already doesn't use pretty printing,
so this change makes FileLfsServlet's behavior consistent. In fact,
both classes now have duplicate Gson handling; this will be cleaned
up in a separate change.
Change-Id: I113a23403f9222f16e2c0ddf39461398b721d064
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
LfsServerText: Fix formatting of invalidPathInfo message
The message is formatted as:
Invalid pathInfo '/abc' does not match '/{SHA-256}'
but should be:
Invalid pathInfo: '/abc' does not match '/{SHA-256}'
(i.e. including a colon) to be consistent with other messages.
Change-Id: Ic38aa7d33dd02d7954b95c331a73919a90c69991
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
InvalidLongObjectIdException: Fix formatting of message
The message is formatted as:
Invalid id: : abcde...
but should be:
Invalid id: abcde...
Change-Id: Ie15cacdcf2f168edaee262e6cf8061ebfe9d998d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Pretty printing the response is useful for human readers, but most
(if not all) of the time, the response will be read by programs.
Remove it to avoid the additional overhead of the formatting and
extra bytes in the response. Adjust the test accordingly.
Note that LfsProtocolServlet already doesn't use pretty printing,
so this change makes FileLfsServlet's behavior consistent. In fact,
both classes now have duplicate Gson handling; this will be cleaned
up in a separate change.
Change-Id: I113a23403f9222f16e2c0ddf39461398b721d064
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
DownloadTest: Use enums from org.apache.http.HttpStatus
Instead of using hard-coded HTTP status codes, use the enums
which makes it a bit easier to see what's expected.
Change-Id: I2da5d25632f374b8625d64da4df70d1c9c406bb1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
According to the specification [1], the error response status code
should be 422 when there is a validation error with one or more of
the objects in the request
[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md#response-errors
Change-Id: Id03fe00a2109b896d9a154228a14a33bce5accc3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The FileLfsRepository.out member could have been accessed from multiple
threads which would corrupt the content.
Don't store the AtomicObjectOutputStream in the FileLfsRepository.out but
move it to the ObjectUploadListener which is instantiated per-request.
Add a parallel upload test.
Change-Id: I62298630e99c46b500d376843ffcde934436215b
Signed-off-by: Saša Živkov <sasa.zivkov@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Support LFS protocol and a file system based LFS storage
Implement LfsProtocolServlet handling the "Git LFS v1 Batch API"
protocol [1]. Add a simple file system based LFS content store and the
debug-lfs-store command to simplify testing.
Introduce a LargeFileRepository interface to enable additional storage
implementation while reusing the same protocol implementation.
At the client side we have to configure the lfs.url, specify that
we use the batch API and we don't use authentication:
[lfs]
url = http://host:port/lfs
batch = true
[lfs "http://host:port/lfs"]
access = none
the git-lfs client appends the "objects/batch" to the lfs.url.
Hard code an Authorization header in the FileLfsRepository.getAction
because then git-lfs client will skip asking for credentials. It will
just forward the Authorization header from the response to the
download/upload request.
The FileLfsServlet supports file content storage for "Large File
Storage" (LFS) server as defined by the Github LFS API [2].
- upload and download of large files is probably network bound hence use
an asynchronous servlet for good scalability
- simple object storage in file system with 2 level fan-out
- use LockFile to protect writing large objects against multiple
concurrent uploads of the same object
- to prevent corrupt uploads the uploaded file is rejected if its hash
doesn't match id given in URL
The debug-lfs-store command is used to run the LfsProtocolServlet and,
optionally, the FileLfsServlet which makes it easier to setup a
local test server.
[1]
https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md
[2] https://github.com/github/git-lfs/tree/master/docs/api
Bug: 472961
Change-Id: I7378da5575159d2195138d799704880c5c82d5f3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Markus Duft
David Pursehouse
Saša Živkov
Matthias Sohn