Sync with upstream @ https://github.com/mstrap/bugtraq

Create new host keys, one with ECDSA and one with Ed25519 algorithms.
For the Ed25519 currently the EdDSA library from i2p is used. This
requires some quirks, compared to a modern BouncyCastle. But the SSHD
library used cannot use BouncyCastle yet for Ed25519.

No DSA key is generated anymore, but we still support existing ones.

(cherry picked from commit d8fbdda2ab3fa48e92bdf37399d4b75c48409c5c@rpardini:master)

# Conflicts:
#	.classpath
#	build.moxie
#	src/test/java/com/gitblit/tests/SshUnitTest.java

These show more clearly what the code is supposed to do.

The `StoredUserConfig` only escaped the escape character, i.e. backslash.
But it does not escape control characters like tab or newline. This
introduces a vulnerability where an attacker can create new entries
in their user account and create new accounts.
In addition, other characters are also not properly handled. Field values
with a comment character need to be quoted. This only happens for the
`#` character and only when the value starts with it. Also the quote
is note escaped in values.

This change completely rewrites the `escape` method of `StoredUserConfig`.
It takes care of properly escaping characters that need escaping for the
git configuration file format.

This fixes #1410

Te `StoredUserConfig` did not handle sections without a subsection.
When the subsection did not exist, i.e. was `null`, then the subsection
name would be set to the string "null". This is not how the config file
format works. It should create a `[SECTIONNAME]` entry instead.

This fix handles a `null` subsection correctly, by handling it as a
section without a subsection.

Add unit tests for exploiting the email address or display name
in the config user service by using newlines in the values.

As with explicit links, also for reference links in markdown documents
which point to repository-relative files the links are broken. They do
not take the path to the repository into account.
This fix is related to commit b23269 which fixed issue #1358
for explicit links.

The `daysAgo` method seemed to want to normalize on a calendar day? I
can't really tell what it was trying to do, but the problem is that it
does not take into account any time shift due to time zones so it never
really worked outside of GMT.
So instead a new `calendarDaysAgo` method is added (because I am unsure
on what the `daysAgo` method is trying to do. It can probably be removed).
The new method cleanly calculates difference in calendar days because it
normalizes the two given time stamps on the same time zone.

The `timeAgo` method now used the new method. This fixes #1248.

For some reason the `TimeUtilsTest` class is, like almost all tests, in
the `com.gitblit.tests` package. But this way all methods in classes
which we might predominately need for tests have to be public.
So move the unit test class `TimeUtilsTest` to the same package as the
class it is testing, i.e. `com.gitblit.utils.TimeUtils`.
This way we ca set the new added methods which get the current time
passed in to be at least not public.

Add tests for `timeAgo` to analyse issue #1248.
The tests are dependent on when they run as they time functions use the
current date and time. To make them testable in a reproducible way, we
need the ability to pass in what we think is "now". So add overloaded
methods that take a `now` parameter so that we can pass in the current
time.

To prevent that we have a resource file in a resource bundle broken and
not loading undiscovered for years, add a unit test that will load the
resource properties file for each of the languages.
In order to check if the file was loaded and the bundle mechanism
didn't fall back on the default, a new property key is added to each
language file, solely for the purpose to be checked in the unit test.

When a branch has a slash in the name, the raw servlet was not able
to find the path under that branch. This is due to the replacement of
the forward slash character for URLs. It was not taken into account
when comparing the branch name later.

This fixes #1290 and its duplicates #1234 and #813.

Refactor the `getBranch` and `getPath` methods to take a String as
second parameter, which is the already sanitised path info. Don't get
the path info from a passed in request anymore.

The methods are only ever called from within `processRequest`, which
already does some checks on the path info, like removing a leading
slash character. So no need to do that every time again the methods
and passing a request for that.

When creating a link for raw display, a trailing slash is stripped from
the end of the base URL. Also do this for the repository, as well as
stripping leading slashes from the repository and the path values.

Zero out the password to remove it from memory after use.
This is only a first step, implementing it for one method:
`AuthenticationManager.authenticate(String, char[], String)`.

The upgrade of a MD5 stored password hash to a PBKDF password hash
destroys the stored password. The has check zeroes out the password that
is tested, so that the new hash is built over the zeroed out value.

This fix prevents that an also adds a check to the test.

Fixes #1335

With the feature of passwords getting automatically
upgraded to a hashed version, the default `admin`
password in the test-users file will get stored
as MD5 hashed during test execution. Commit this
change, so that the file isn't always showing up
as changed.

Renames `static final` variables according to convention to be in all
upper case. That makes it easier to see that in an `equals` comparison
the final variable should come first as it will not trigger a NPE.

Also strip parameters from the URL when extracting the repository
name from it. Parameters can not be part of a repository name, and
this way an empty repository name can be detected.

Fixes #1092

The SshDaemonTest would fail under Windows. That is because JGit looks
at the `GIT_SSH` environment variable. If it is set, the tool the variable
is pointing to is used for the SSH connection. This is a problem when
it is set to "Plink" under Windows, because Plink will not recognize the
server key and will not find it in the registry, cached as a known host.
Since a test can/should not add the key to the registry but simply wants
to ignore it, but there is no way to tell Plink to do so, the tests would
fail.

This patch filters the `GIT_SSH` environment variable from JGit's
`SystemReader`, so that the internal SSH client is used.

Currently the tests cannot run as no Redis is available. Needs to be
fixed at some time.

Integrate the `PasswordHash` class and subclass in the user
and password editing and authentication. Replaces the old code and
the previous `SecurePasswordHashingUtils` class.

Integrate the work of pingunaut to add support for PBKDF2 password
hashing. A new class `PasswordHashPbkdf2` is added, which builds
on his `SecurePasswordHashUtils` class, but makes it a subclass
of `PasswordHash`. This will replace the original class when
integrating the new PasswordHash way into GitBlit.

Instead of having to deal with the implementation details of hashing
and verifying passwords in multiple places, have a central unit
be responsible for it. Otherwise we need to edit three different places
when adding a new hashing scheme.
With this class adding a new hashing scheme just requires creating a
new subclass of `PasswordHash` and registering its type in the enum
`PasswordHash.Type`.
The rest of the code will use a common interface for all hashing
schemes and doesn't need to be changed when a new one is added.

Fix for #725.
Also covers #527.

This is a squashed commit of the following commits,
merging and closing pull request #1267:

commit 55fee41769
Author: Martin Spielmann <mail@martinspielmann.de>
Date:   Mon Nov 6 17:19:53 2017 +0100

    declared local variable final to fix travis build

commit 131e4d14a4
Author: Martin Spielmann <mail@martinspielmann.de>
Date:   Mon Nov 6 14:11:55 2017 +0100

    fix formatting (use tab for identation)

commit 8da5f6d596
Author: Martin Spielmann <mail@martinspielmann.de>
Date:   Mon Nov 6 13:45:39 2017 +0100

    Add repositoryListType tree. Addresses #725, 527 and includes #1224

commit 6c061651fb
Merge: f365daa3 40ee9653
Author: Martin Spielmann <mail@martinspielmann.de>
Date:   Sat Nov 4 13:19:08 2017 +0100

    Merge remote-tracking branch 'collapsible/ticket/527' into 725_nested_repos

commit f365daa3b1
Author: Martin Spielmann <mail@martinspielmann.de>
Date:   Sat Nov 4 13:10:24 2017 +0100

    first working version of tree model

Create a regression test for issue #999.
Add directories with '[]' ans '()' in the name to the
hello-world repository, so that they can be used in unit
tests for repository paths with special characters.

When symbolic links under the base repository folder point to repositories
outside the base repository folder, the forming of relative repository
names failed and resulted in NullPointerExceptions.

Create the relative path by not following symbolic links, i.e. the link
name is taken as is and not resolved to the external path.

This also changes the whole `exactPath` method to work on Paths, instead
of Files.

Fixes #891 and fixes #837.

This hello-world.git repo is created using the native Git for Windows
software.

Various test classes in the GitBlitSuite test suite require the presence
of the hello-world.git repo in github.com/git/ which has been missing,
hence causing many test failures in the suite. This recreation of the
hello-world.git repo aims to conform to the many test cases'
requirements in the suite, and to be checked in as part of the gitblit
repo, thus eliminates the requirement of a remote hello-world.git repo
during the test run. The repo is now stored is a zip ball in the new
src/test/data folder.

The hello-world repo's various commit IDs were hard-coded in various
test classes. These commit IDs, which must now have new values in the
recreated repo, are now extracted out to the
src/test/data/hello-world.properties file. The gitblit's build.xml is
modified to generate the HelloworldKeys.java file containing the
hello-world.properties file's key strings, in similar fashion as the
existing generation of the com.gitblit.Keys.java file. And these key
strings in HelloworldKeys.java are now used in the various test classes,
thus eliminating the hard-coding of the hello-world repo's commit IDs in
the test code.

During the test run by GitBlitSuite test suite, some repos from GitHub
were cloned and became part of the test data. These repos are now zipped
to be part of gitblit repo itself, thus eliminating the network fetch at
the start of test run which can be slow, especially with the JGit repo
cloning which is huge and time consuming. The cloned JGit repo is now
zipped and checked in to gitblit, along with the other 4 repos
(hello-world, ambition, gitective and ticgit). They will be unzipped
during the test suite run and be available in the local file system,
thus avoiding the need for some network fetch.

Special note on the zipped JGit repo: this repo is big (and growing all
the time on GitHub), and takes up about 32MB of disk space after cloning
from GitHub. I've made it smaller by resetting HEAD back to a commit of
5 years ago (with git reset --hard <commitId> command), to put it back
to roughly where/when the tests were written for it (which is not quite,
because there are tons of commit history since which can't be removed.)
The local JGit repo is then garbage-collected (with git gc --prune
--aggressive) to reduce its size to about 19MB.

Zipped it is still 17MB. This is a lot of MBs for a few tests.
So the JGit repo is not included in this commit.

Fixes #1275

Most of failures were due to temporary test repos, users and/or teams
being left behind after the test run, and these left-over stuff in
$baseFolder/data/git caused assertion errors in many tests in subsequent
test runs. This fix tries to delete those left-over stuff at the end of
each test, mainly in their @Afterclass code blocks.

PushLogTest.java is deleted as it doesn't work, and has been superseded
with better tests in various protocol test suites (GitServletTest,
GitDaemonTest, SshDaemonTest, etc.)

Check if tickets need to be reindexed when the server starts. This is the
case if no ticket index exists. In that case the ticket index is built.

This is done during the start of the `ITicketService`.

For this the interface of `ITicketService` needed to change. The `start`
method was defined abstract and the specific ticket services had to
implement it. None does any real starting stuff in it.
The `start` method is now final. It calls a new abstract method `onStart`
which the specific ticket services need to implement. In the existing
implementations I just changed `start` to `onStart`.

Change from the index version of a repository index being stored in a config
file to also using index directories with the version in the name. For that,
`LuceneRepoIndexStore` is added, which adds the fixed `lucene` part to the path.
It also gives out the location of the `lucene.conf` file, which is now stored in
the index directory. This way it is automatically deleted when the directory is
deleted.

I believe that it should also provide means to store branch aliases and tips,
i.e. hide the config file completely. But this isn't implemented with this
commit, the `LuceneService` is still aware that a config file is used.

In order to be able to update the index definition, the ticket index
is assigned a version number, 2. This way the definiton can be updated
and compatability with existing index files can be checked.
The actual index is stored in a directory of name `indexVersion_codecVersion`.
This wayit is veriy easy to check if an index of a certain version exists on the
filesystem. It allows to have multiple indexes of different versions present,
so that a downgrade of the software is possible without having to reindex
again. Of coure, this is only possible if no new tickets were created since these
would be missing in the old index.

A new class `LuceneIndexStore` is introduced, which abstracts away the versioned
index directory. The idea is, that this provides one place to keep the Lucene
codec version and to allow to code compatibility rules into this class, so that
older indices can still be used if they are compatible.

Addresses #1166

Introduce our own wrapper `SecureRandom` around `java.security.SecureRandom`.
This a) makes sure that the PRNG is seeded on creation and not when
random bytes are retrieved, and
b) uses a static instance in the `UserModel` so that lags do not occur
during operation due to potentially seeding getting blocked on Unix
when reading from the system's entropy pool. To keep the random data
still secure, the static instance will reseed all 24 hours, also a
functionality of the wrapper class.

This fixes #1063 and extends and closes PR #1116

The regular expression used for user mentions used to work
only inside sentences. Also, since it tested for whitespace, the
whitespace would get replaced, too, which would join lines together.

Instead the new regex uses boundary matchers to match against
word boundaires. As these are not capturing only the actual user
mention can be captured and is then replaced. Also, this way the
regex can ignore punctuation like in "@jim, look at this."

Since Gibtlit now requires Java 7 we can use named capture groups.
This makes the use of a centrally defined regular expression much
safer. The (admittedly only) group to capture the user name is named
"user" and can be referenced by this name. By using the name instead
of a group number, the regex could be changed without the code using
it breaking because the group number changed.

A simple test is added for user mentions, which unfortunately
has to deal with the full markdown replacement, too.

Fixes #985

The `SshKeysDispatcher` tests that use the keys list command are failing
on Windows because they assume a Unix line ending after each key. But
the command will use a system line ending. So this fix uses system line
endings in the reference string for the assert, too.

In addition, two `assertTrue(false)´ are replaced with a proper `fail`.