Docusaurus-ify (#26051)

This PR cleans up the docs in a way to make them simpler to ingest by
our [docs repo](https://gitea.com/gitea/gitea-docusaurus).

1. It includes all of the sed invocations our ingestion did, removing
the need to do it at build time.
2. It replaces the shortcode variable replacement method with
`@variable@` style, simply for easier sed invocations when required.
3. It removes unused files and moves the docs up a level as cleanup.

---------

Signed-off-by: jolheiser <john.olheiser@gmail.com>

1 files changed, 40 insertions, 0 deletions

diff --git a/docs/content/administration/search-engines-indexation.en-us.md b/docs/content/administration/search-engines-indexation.en-us.md
new file mode 100644
index 0000000000..7898e8146e
--- /dev/null
+++ b/docs/content/administration/search-engines-indexation.en-us.md
@@ -0,0 +1,40 @@
+---
+date: "2019-12-31T13:55:00+05:00"
+title: "Search Engines Indexation"
+slug: "search-engines-indexation"
+sidebar_position: 60
+toc: false
+draft: false
+aliases:
+  - /en-us/search-engines-indexation
+menu:
+  sidebar:
+    parent: "administration"
+    name: "Search Engines Indexation"
+    sidebar_position: 60
+    identifier: "search-engines-indexation"
+---
+
+# Search engines indexation of your Gitea installation
+
+By default your Gitea installation will be indexed by search engines.
+If you don't want your repository to be visible for search engines read further.
+
+## Block search engines indexation using robots.txt
+
+To make Gitea serve a custom `robots.txt` (default: empty 404) for top level installations,
+create a file called `robots.txt` in the [`custom` folder or `CustomPath`](administration/customizing-gitea.md)
+
+Examples on how to configure the `robots.txt` can be found at [https://moz.com/learn/seo/robotstxt](https://moz.com/learn/seo/robotstxt).
+
+```txt
+User-agent: *
+Disallow: /
+```
+
+If you installed Gitea in a subdirectory, you will need to create or edit the `robots.txt` in the top level directory.
+
+```txt
+User-agent: *
+Disallow: /gitea/
+```