Heap always copied whole blocks, which leads to AIOOBEs. LocalFile
didn't overwrite the method and thus caused NPEs.
Change-Id: Ia37d4a875df9f25d4825e6bc95fed7f0dff42afb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
TransportHttp: support preemptive Basic authentication
If the caller knows already HTTP Basic authentication will be needed
and if it also already has the username and password, preemptive
authentication is a little bit more efficient since it avoids the
initial 401 response.
Add a setPreemptiveBasicAuthentication(username, password) method
to TransportHttp. Client code could call this for instance in a
TransportConfigCallback. The method throws an IllegalStateException
if it is called after an HTTP request has already been made.
Additionally, a URI can include userinfo. Although it is not
recommended to put passwords in URIs, JGit's URIish and also the
Java URL and URI classes still allow it. The underlying HTTP
connection may omit these fields though. If present, take these
fields as additional source for preemptive Basic authentication if
setPreemptiveBasicAuthentication() has not been called.
No preemptive authentication will be done if the connection is
redirected to a different host.
Add tests.
Bug: 541327
Change-Id: Id00b975e56a15b532de96f7bbce48106d992a22b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
TransportHttp: shared SSLContext during fetch or push
TransportHttp makes several HTTP requests. The SSLContext and socket
factory must be shared over these requests, otherwise authentication
information may not be propagated correctly from one request to the
next. This is important for authentication mechanisms that rely on
client-side state, like NEGOTIATE (either NTLM, if the underlying HTTP
library supports it, or Kerberos). In particular, SPNEGO cannot
authenticate on a POST request; the authentication must come from the
initial GET request, which implies that the POST request must use the
same SSLContext and socket factory that was used for the GET.
Change the way HTTPS connections are configured. Introduce the concept
of a GitSession, which is a client-side HTTP session over several HTTPS
requests. TransportHttp creates such a session and uses it to configure
all HTTP requests during that session (fetch or push). This gives a way
to abstract away the differences between JDK and Apache HTTP connections
and to configure SSL setup outside.
A GitSession can maintain state and thus give all HTTP requests in a
session the same socket factory.
Introduce an extension interface HttpConnectionFactory2 that adds a
method to obtain a new GitSession. Implement this for both existing
HTTP connection factories. Change TransportHttp to use the new
GitSession to configure HTTP connections.
The old methods for disabling SSL verification still exist to support
possibly external connection and connection factory implementations
that do not make use of the new GitSession yet.
Bug: 535850
Change-Id: Iedf67464e4e353c1883447c13c86b5a838e678f1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
TransportHttp: make the connection factory configurable
Previously, TransportHttp always used the globally set connection
factory. This is problematic if that global factory is changed in
the middle of a fetch or push operation. Initialize the factory to
use in the constructor, then use that factory for all HTTP requests
made through this transport. Provide a setter and a getter for it
so that client code can customize the factory, if needed, in a
TransportConfigCallback.
Once a factory has been used on a TransportHttp instance it cannot
be changed anymore.
Make the global static factory reference volatile.
Change-Id: I7c6ee16680407d3724e901c426db174a3125ba1c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Protocol V2: don't log spurious ACKs in UploadPack
UploadPack may log ACKs in protocol V2 that it doesn't send (if it
got a "done" from the client), or may log ACKs twice. That makes
packet log analysis difficult.
Add a new constructor to PacketLineOut to omit all logging from an
instance, and use it in UploadPack.
Change-Id: Ic29ef5f9a05cbcf5f4858a4e1b206ef0e6421c65
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
This fixes errorprone error [SynchronizeOnNonFinalField]: Synchronizing
on non-final fields is not safe: if the field is ever updated, different
threads may end up locking on different objects.
Change-Id: I42fe5bde825151693e2da2d5b6cd6e1d34038dbc
[spotbugs]: Fix potential NPE in FileSnapshot constructor
File#getParent can return null which caused this spotbugs warning.
FS.FileStoreAttributes#get already gets the parent directory if the
passed File is not a directory and checks for null. Hence there is no
need to get the parent directory in the FileSnapshot constructor.
Change-Id: I77f71503cffb05970ab8d9ba55b69c96c53098b9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
FileSnapshot: don't try to read file attributes twice
If file doesn't exist set state to MISSING_FILE immediately. Doing that
by calling File#lastModified and File#length effectively does the same
since they set the value to 0 if the file doesn't exist.
Log an error if a different exception than NoSuchFileException is
caught.
Change-Id: I0d4396b9f80446692a088d17522d64f735ce6708
Make all transports request protocol V2 when fetching. Depending on
the transport, set the GIT_PROTOCOL environment variable (file and
ssh), pass the Git-Protocol header (http), or set the hidden
"\0version=2\0" (git anon). We'll fall back to V0 if the server
doesn't reply with a version 2 answer.
A user can control which protocol the client requests via the git
config protocol.version; if not set, JGit requests protocol V2 for
fetching. Pushing always uses protocol V0 still.
In the API, there is only a new Transport.openFetch() version that
takes a collection of RefSpecs plus additional patterns to construct
the Ref prefixes for the "ls-refs" command in protocol V2. If none
are given, the server will still advertise all refs, even in protocol
V2.
BasePackConnection.readAdvertisedRefs() handles falling back to
protocol V0. It newly returns true if V0 was used and the advertised
refs were read, and false if V2 is used and an explicit "ls-refs" is
needed. (This can't be done transparently inside readAdvertisedRefs()
because a "stateless RPC" transport like TransportHttp may need to
open a new connection for writing.)
BasePackFetchConnection implements the changes needed for the protocol
V2 "fetch" command (stateless protocol, simplified ACK handling,
delimiters, section headers).
In TransportHttp, change readSmartHeaders() to also recognize the
"version 2" packet line as a valid smart server indication.
Adapt tests, and run all the HTTP tests not only with both HTTP
connection factories (JDK and Apache HttpClient) but also with both
protocol V0 and V2. The SSH tests are much slower and much more
focused on the SSH protocol and SSH key handling. Factor out two
very simple cloning and pulling tests and make those run with
protocol V2.
Bug: 553083
Change-Id: I357c7f5daa7efb2872f1c64ee6f6d54229031ae1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Use Map interface instead of ConcurrentHashMap class
On Android, the co-variant override of ConcurrentHashMap.keySet()
introduced in Java 8 was undone. [1] If compiled Java code calls that
co-variant override directly, one gets a NoSuchMethodError exception
at run-time on Android.
Making the code call that method via Map.keySet() side-steps this
problem.
This is similar to bug 496262, where the same problem cropped up when
compiling with Java 8 against a Java 7 target, but here we cannot use
bootclasspath. We build against Java 8, not against the Android version
of it.
Recent Android versions should have some bytecode "magic" that adds the
co-variant override in bytecode (see the commit referenced in [1]), but
on older Android version this problem may still occur. (Or perhaps the
"magic" is ineffective...) There are two pull requests on Github for
this problem, both from 2020, [2][3] while the Android commit [1] is
from March 2018. Apparently people still occasionally run into this
problem in the wild.
[1] 0e8b937ded/ojluni/src/main/java/java/util/concurrent/ConcurrentHashMap.java (1244)
[2] https://github.com/eclipse/jgit/pull/104
[3] https://github.com/eclipse/jgit/pull/100
Change-Id: I7c07e0cc59871cb7fe60795e22867827fa9c2458
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
If a file exists in head, merge, and the working tree, but not in
the index, and we're doing a force checkout, the checkout must be
an "update", not a "keep".
This is a follow-up on If3a9b9e60064459d187c7db04eb4471a72c6cece.
Bug: 569962
Change-Id: I59a7ac41898ddc1dd90e86b09b621a41fdf45667
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
[spotbugs] Fix incorrect lazy initialization in SystemReader
This fixes two warnings of type LI_LAZY_INIT_STATIC.
Change-Id: I26a7a48aed9d0a0547e908a56b7014a7620fadd8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] Don't use class from java.util.concurrent for locking
Use a dedicated Lock object to lock the scheduler in
RepositoryCache#configureEviction to fix spotbugs warning
JLM_JSR166_UTILCONCURRENT_MONITORENTER.
Change-Id: I003dcf0ed1a0a3f4eea5d8a2f51a07473d28a928
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] Fix potential NPE in FS.FileStoreAttributes#get
Path#getParent can return null, return fallback filestore attributes in
that case.
Change-Id: Ic09484d527bc87b27964b625e07373b82412f2da
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] Fix FileReftableStack#equals to check for null
This fixes spotbugs warning NP_EQUALS_SHOULD_HANDLE_NULL_ARGUMENT.
This implementation violated the contract defined by
java.lang.Object.equals() because it did not check for null being passed
as the argument. All equals() methods should return false if passed a
null value.
Change-Id: I607f6979613d390aae2f3546b587f63133d6d73c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] FileReftableDatabase: extract lock to local variable
This fixes UL_UNRELEASED_LOCK_EXCEPTION_PATH raised by spotbugs in
#compactFully.
Change-Id: I370578ad9a027c5c9709d60a1dfafdac0cfca908
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] DfsReftableDatabase: extract lock to local variable
This fixes UL_UNRELEASED_LOCK_EXCEPTION_PATH raised by spotbugs in
#DfsReftableDatabase and #clearCache.
Change-Id: Ifd3189288d2a8e64139c02cd105eb335fa2f68cf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] Silence NP_BOOLEAN_RETURN_NULL in IgnoreNode#checkIgnored
Also mark the return value @Nullable to enable null analysis in Eclipse.
Change-Id: I5b286d657d432f4b32afd4dd370f76892b115422
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] Fix potential NPE in WorkingTreeIterator#isModified
File#list can return null. Fix the potential NPE by using Files#list
which is also faster since it retrieves directory entries lazily while
File#list retrieves them eagerly.
Change-Id: Idf4bda398861c647587e357326b8bc8b587a2584
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[spotbugs] Fix potential NPE in FileRepository#convertToReftable
File#listFiles can return null. Use Files#list which does not return
null and should be faster since it's returning directory entries lazily
while File#listFiles fetches them eagerly.
Change-Id: I3bfe2a52278244fc469143692c06b05d9af0d0d4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If TransportException is thrown in the finally block of execute()
ensure that the exception handled in the previous catch block isn't
suppressed.
Change-Id: I670acdfb4d36e7a419a9a79ae9faab2e085a43ee
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If TransportException is thrown in the finally block of
downloadPackedObject() ensure that the exception handled in the previous
catch block isn't suppressed.
Change-Id: I23982a2b215e38f681cc1719788985e60232699a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This was experimental code and never used in production.
Change-Id: Ia3da7f2b82d9e365cec2ccf9397cbc47439cd150
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Split out loose object handling from ObjectDirectory
The ObjectDirectory class manages the interactions for the entire object
database, this includes loose objects, packfiles, alternates, and
shallow commits. To help reduce the complexity of this class, abstract
some of the loose object specific details into a class which understands
just this, leaving the ObjectDirectory to focus more on the interactions
between the different mechanisms.
Change-Id: I39f3a74d6308f042a2a2baa57769f4acde5ba5e0
Signed-off-by: Martin Fick <mfick@codeaurora.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The ObjectDirectory class manages the interactions for the entire object
database, this includes loose objects, packfiles, alternates, and
shallow commits. To help reduce the complexity of this class, abstract
some of the packfile specific details into a class which understands
just this, leaving the ObjectDirectory to focus more on the interactions
between the different mechanisms.
Change-Id: I5cc87b964434b0afa860b3fe23867a77b3c3a4f2
Signed-off-by: Martin Fick <mfick@codeaurora.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Some clients may wish to allow NO_CHANGE lightweight tag updates
without setting the force flag. (For instance EGit does so.)
Command-line git does not allow this.
Propagate the RefUpdate result via the RefAlreadyExistsException.
That way a client has the possibility to catch it and check the
failure reason without having to parse the exception message, and
take appropriate action, like ignoring the exception on NO_CHANGE.
Change-Id: I60e7a15a3c309db4106cab87847a19b6d24866f6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
TagCommand: make -f work with lightweight tags for NO_CHANGE
JGit treated a NO_CHANGE RefUpdate as an error in all cases. But when
updating a lightweight tag, this is a successful result if -f was
specified.
Change-Id: Iddfa6d6a6dc8bf8fed81138a008ebc32d5f960bd
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Add the two config constants from C git that can switch on signing
of annotated tags. Add them to the GpgConfig, and implement actually
signing a tag in TagCommand.
The interactions between command line options for "git tag" and config
options is a bit murky in C git. There are two config settings for it:
* tag.gpgSign is the main option, if set to true, it kicks in if
neither -s nor -u are given on the command line.
* tag.forceSignAnnotated signs only tags created via "git tag -m",
but only if command-line option "-a" is not present. It applies
even if tag.gpgSign is set explicitly to false.
Giving -s or -u on the command line also forces an annotated tag
since lightweight tags cannot be signed.
Bug: 386908
Change-Id: Ic8a1a44b5f12f47d5cdf3aae2456c1f6ca9ef057
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Factor out a common ObjectBuilder as super class of CommitBuilder
and TagBuilder, and make the GpgSigner work on ObjectBuilder.
In order not to break API, add the new method for signing an
ObjectBuilder in a new interface GpgObjectSigner.
The signature for a tag is just tacked onto the end of the tag
message. The message of a signed tag must end in LF.
Bug: 386908
Change-Id: I5e021e3c927f4051825cd7355b129113b949455e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Added check for null on DirCacheEntry in checkoutEntry method
Observed the error when trying to force checkout from a branch
that had no changes on it. When the 'keep()' method from 'DirCacheCheckout'
method was called the 'DirCacheEntry e' was null and was passed like
this to the 'checkoutEntry()' method where the 'getObjectId()' is
being called on the 'e' object
Change-Id: If3a9b9e60064459d187c7db04eb4471a72c6cece
Fix PackInvalidException when fetch and repack run concurrently
We are running several servers with jGit. We need to run repack from
time to time to keep the repos performant. I.e. after push we test how
many small packs are in the repo and when a threshold is reached we run
the repack.
After upgrading jGit version we've found that if someone does the clone
at the time repack is running the clone sometimes (not always) fails
because the repack removes .pack file used by the clone. Server
exception and client error attached.
I've tracked down the cause and it seems to be introduced between jGit
5.2 (which we upgraded from) and 5.3 and being caused by this commit:
Move throw of PackInvalidException outside the catch -
afef866a44
The problem is that when the throw was inside of the try block the last
catch block catched the exception and called openFailed(false) method.
It is true that it called it with invalidate = false, which is wrong.
The real problem though is that with the throw outside of the try block
the openFail is not called at all and the fields activeWindows and
activeCopyRawData are not set to 0. Which affects the later called tests
like: if (++activeCopyRawData == 1 && activeWindows == 0).
The fix for this is relatively simple keeping the throw outside of the
try block and still having the invalid field set to true. I did
exhaustive testing of the change running concurrent clones and pushes
indefinitely and with the patch applied it never fails while without the
patch it takes relatively short to get the error.
See: https://www.eclipse.org/lists/jgit-dev/msg04014.html
Bug: 569349
Change-Id: I9dbf8801c8d3131955ad7124f42b62095d96da54
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add constants for parsing git wire protocol version
This would allow other JGit users to access and reuse the constants.
Change-Id: I1608802f45586af5f8582afa592e26679e9cebe3
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add support for reading symrefs from pack capabilities
A SymbolicRef is added to the advertised refs for any symref in
capabilities whose target is an advertised ref; this may replace an
existing entry, such as HEAD.
When cloning, if any advertised HEAD is symbolic then use the target
rather than looking for an advertised ref with a matching objectId.
Add --symref option to LsRemote command.
Bug: 514052
Change-Id: Idfb48e6f6e8dcfe57a6896883fe6d84d533aa9d0
Signed-off-by: Lee Worrall <worrall.la@gmail.com>
GC#deleteOrphans: handle failure to list files in pack directory
- log an error
- either there is no list or it is incomplete hence return immediately
Change-Id: Ieee5378ca06304056b9ccc30c1acd5f52360052d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If pack or index files are guarded by a pack lock (.keep file)
deleteOrphans() should not touch the respective files protected by the
lock file. Otherwise it may interfere with PackInserter concurrently
inserting a new pack file and its index.
The problem was caused by the following race.
All mentioned files are located in "objects/pack/".
File endings relevant in "pack" dir:
.pack
.keep
.idx
.bitmap
When ReceivePack receives a pack file it executes the following steps:
ReceivePack.service():
receivePackAndCheckConnectivity():
receivePack():
receive the pack
parse the pack, returns packLock (.keep file)
PackInserter.flush():
write tmpPck file: "insert_<random>.pack"
write tmpIdx file: "insert_<random>.idx"
real pack name: "pack-<SHA1>.pack"
real index name: "pack-<SHA1>.idx"
atomic rename tmpPack to realPack
atomic rename tmpIdx to tmpIdx
execute commands
unlock pack by removing .keep file
trigger auto gc if enabled
When PackInserter.flush() renames the temporary pack to the final
"pack-xxx.pack" file the temporary pack index file "insert_xxx.idx"
has no matching .pack file with the same base name for a short interval.
If deleteOrphans() ran during that interval it deduced the pack index
file was orphaned. Subsequently the missing pack index caused
MissingObjectExceptions since objects contained in the pack couldn't be
looked up anymore.
Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=13544
Change-Id: I559c81e4b1d7c487f92a751bd78b987d32c98719
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Just allocate the two string objects directly. The previously used
new StringBuilder(0).toString() returns the same object for both END
and DELIM when run on Java 15, which breaks the wire protocol since
then END and DELIM cannot be distinguished anymore.
Bug: 568950
Change-Id: I9d54d7bf484948c24b51a094256bd9d38b085f35
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
(cherry picked from commit 7da0f0a8f3)
Just allocate the two string objects directly. The previously used
new StringBuilder(0).toString() returns the same object for both END
and DELIM when run on Java 15, which breaks the wire protocol since
then END and DELIM cannot be distinguished anymore.
Bug: 568950
Change-Id: I9d54d7bf484948c24b51a094256bd9d38b085f35
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
(cherry picked from commit 7da0f0a8f3)
Just allocate the two string objects directly. The previously used
new StringBuilder(0).toString() returns the same object for both END
and DELIM when run on Java 15, which breaks the wire protocol since
then END and DELIM cannot be distinguished anymore.
Bug: 568950
Change-Id: I9d54d7bf484948c24b51a094256bd9d38b085f35
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Allow to resolve a conflict by checking out a file
DirCacheEditor unconditionally applied a PathEdit to all stages in the
index. This gives wrong results if one wants to check out a file from
some commit to resolve a conflict: JGit would update the working tree
file multiple times (once per stage), and set all stages to point to
the checked-out blob.
C git replaces the stages by the entry for the checked-out file.
To support this, add a DirCacheEntry.setStage() method so that
CheckoutCommand can force the stage to zero. In DirCacheEditor, keep
only the zero stage if the PathEdit re-set the stage.
Bug: 568038
Change-Id: Ic7c635bb5aaa06ffaaeed50bc5e45702c56fc6d1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Document that setLastModified sets time of symlink target
Due to Java bug JDK-8220793 [1] Java cannot set timestamps of a symlink
but only of the symlink target. This bug was fixed in Java 13. Since we
don't have a use case to set the timestamp of the symlink itself simply
document the current behavior of setLastModified methods.
[1] https://bugs.openjdk.java.net/browse/JDK-8220793
Change-Id: Ibc28c1702a1b9845602bd257606fbd44803a43fb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
PerformanceLogContext threw NullPointerException when multiple threads
tried to add an event to the PerformanceLogContext. The cause for this
is that the ThreadLocal was initialized only in the class constructor
for the first thread; for subsequent threads it was null.
To fix this initialize eventRecords for each thread.
Change-Id: I18ef67dff8f0488e3ad28c9bbc18ce73d5168cf9
Signed-off-by: Alexa Panfil <alexapizza@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix IOException occurring when calling
GC on a repository with absent objects/pack folder.
Change-Id: I5be1333a0726f4d7491afd25ddac85451686c30a
Signed-off-by: Nail Samatov <sanail@yandex.ru>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>