Preserve BOM in web editor (#28935)

The `ToUTF8*` functions were stripping BOM, while BOM is actually valid
in UTF8, so the stripping must be optional depending on use case. This
does:

- Add a options struct to all `ToUTF8*` functions, that by default will
strip BOM to preserve existing behaviour
- Remove `ToUTF8` function, it was dead code
- Rename `ToUTF8WithErr` to `ToUTF8`
- Preserve BOM in Monaco Editor
- Remove a unnecessary newline in the textarea value. Browsers did
ignore it, it seems but it's better not to rely on this behaviour.

Fixes: https://github.com/go-gitea/gitea/issues/28743
Related: https://github.com/go-gitea/gitea/issues/6716 which seems to
have once introduced a mechanism that strips and re-adds the BOM, but
from what I can tell, this mechanism was removed at some point after
that PR.

1 files changed, 1 insertions, 1 deletions

diff --git a/tests/integration/migration-test/migration_test.go b/tests/integration/migration-test/migration_test.go
index 556a54015b..40fcf95705 100644
--- a/tests/integration/migration-test/migration_test.go
+++ b/tests/integration/migration-test/migration_test.go
@@ -141,7 +141,7 @@ func readSQLFromFile(version string) (string, error) {
 	if err != nil {
 		return "", err
 	}
-	return string(charset.RemoveBOMIfPresent(bytes)), nil
+	return string(charset.MaybeRemoveBOM(bytes, charset.ConvertOpts{})), nil
 }
 
 func restoreOldDB(t *testing.T, version string) bool {