<p>In this tutorial we're going to setup the EndGameV3 Anti DDOS / Load Balancer / WAF service popularized by Dread, it was originally built to block off the incessant <ahref="https://blog.torproject.org/tor-network-ddos-attack/">DDOS attacks</a> that onion services were facing. Because of that, EndGame was developed, along with the <ahref="https://blog.torproject.org/introducing-proof-of-work-defense-for-onion-services/">Proof of Work (POW) Defense released by TorProject</a> for more details you can click <ahref="https://community.torproject.org/onion-services/ecosystem/technology/pow/">here</a>. </p>
Endgame should be on a separate server to your backend server. It only proxies content from your backend to the user. You will still need to configure your backend to handle requests from the Endgame Front.
This is the same system that anti-DDOS services like Cloudflare, Indusface, and Imperva use to protect websites from attacks. The difference is this is self-hosted and fully controlled by you for your own needs and made for darknet networks.
</pre></code>
<p>Now we'll first cover how to have a single Endgame V3 front, to redirect to 2 onion backends, but keep in mind that there is very high latency involved here. The ideal setup as we'll see later, is to have local redirection behind the Endgame front. And we'll also make use of the onionbalance technology to setup multiple Endgame fronts for the same Master Onion!</p>
</div>
</div><!-- /row -->
</div><!-- /container -->
</div><!-- /grey -->
<!-- +++++ Second Post +++++ -->
<divid="anon3">
<divclass="container">
<divclass="row">
<divclass="col-lg-8 col-lg-offset-2">
<h2><b>Prerequisites </b></h2>
<p>First of all you need at least 2 backend servers that are reachable via their .onion links. To set them up you can look at my tutorial <ahref="../torwebsite/index.html">here</a>.</p>
<p>In this example we'll use my 2 main websites' onion links: </p>
<p>Obviously these are not the same service, but we'll pretend that they are. Now let's setup Endgame on a blank Debian 12 VM with internet access:</p>
<p>Let's first download the tar.gz file for endgamev3, and verify the sha256sum hash of the tar.gz file according to the original <ahref="http://g66ol3eb5ujdckzqqfmjsbpdjufmjd5nsgdipvxmsh7rckzlhywlzlqd.onion/post/821adc377e813881ac74">post</a>'s recommendations. Here i downloaded it from the git mirror i host, but if you want the original URL you can get it from here: <b>http://g66ol3eb5ujdckzqqfmjsbpdjufmjd5nsgdipvxmsh7rckzlhywlzlqd.onion/EndGameV3.tar.gz</b></p>
<p>From there we can follow the steps as detailed in the <ahref="https://git.nowhere.moe/nihilist/EndGameV3/src/branch/main/README.md">README.md</a></p>
<b>Go to sourcecode/gobalance and build gobalance with go. Read the README.md about how to compile and generate the gobalance configuration. With that configuration you will be able to see your MASTERONION url. The starting before .key is your master onion address. You will use that as your MASTERONION in the EndGame.config ending it with '.onion'.</b>
or simply use your python onionbalance one! Drop in replacement support (no multisite)!
# Running
After you have configured your gobalance, you will need a tor process on your localhost. There is a provided torrc file. Run it with Tor like this:
- `tor -f torrc`
After that run gobalance
- `./gobalance`
If you need to run these in the background (in the event your server connection dies or drops) you can use `nohup` or a detached terminal session.
I, /u/Paris, recommend just running it locally with geo redundancy to not need to worry about server crashes or compromises. Onion key safety is your absolute priority. When it's compromised your operation is done.
</code></pre>
<p>So as advised, let's compile gobalance using go:</p>
-rw-r--r-- 1 nihilist nihilist 76 Mar 6 2023 torrc
drwxr-xr-x 6 root root 4096 Apr 13 14:23 vendor
</code></pre>-->
<p>So in this case our MASTERONION url is <b>uyrpqcefaftpfq755fv4fe3noc6l3be4qehq3twcu3nfzvycdqko2yyd.onion</b> Next step is to edit the endgame.config file:</p>
<pre><codeclass="nim">
root@debian:~/endgame/sourcecode/gobalance# cd ../..
root@debian:~/endgame# vim endgame.config
</code></pre>
<p>To edit the file we follow the setup process from the README:</p>
<pre><codeclass="nim">
Open up and <b>edit the endgame.config</b>, you will need to change your <b>TORAUTHPASSWORD. Change it to a random alphanumeric password of your choice</b>. This is just used for authentication on nginx's layer to send circuit kill commands.
You have two options for how EndGame sends the traffic to your backend. You can <b>have it direct it to an onion address, or you can have it locally proxy to a server on the same network.</b>
Tor Proxy: You will need to set both of the <b>BACKENDONION variables to your main onion service you want protected</b>. This means your origin application server needs to have tor running with its own onion service address. You put that onion address on the BACKENDONION(1/2). If you have multiple backends (highly recommended) you can put different backend addresses to have load balancing and fallover. It's easy to add in even more by customizing endgame for your needs.
Local Proxy: Change LOCALPROXY to true and edit the PROXYPASSURL to the specific IP or hostname of your backend location. It will default to connect on port 80 via http but you can edit line 320 of the site.conf to change that to your specific needs.
Enable I2PSETUP and/or TORSETUP by setting them to true. You can also enable TORINTRODEFENSE and TORPOWDEFENSE to provide more protection against introduction attacks on the Tor network.
Edit KEY and SALT to a secure cookie value. PROTECT THESE VALUES. If they get leaked, an attacker could generate EndGame cookies and hurt your EndGame protection.
<b>KEY: is your encryption key used for encryption. It should be to be between 68 and 128 random alphanumeric characters.
SALT: is your salt for the encryption key. It must be exactly 8 alphanumeric characters.</b>
Branding is important. EndGame makes it easy to use your own branding on it. By default, it will use dread's branding, but you should change it.
<b>HEXCOLOR and HEXCOLORDARK</b> are for the specific colors used on the pages. Set <b>HEXCOLOR to your main site color and HEXCOLORDARK to just a slightly darker version of it.</b>
<b>SITENAME, SITETAGLINE, SITESINCE is all information about your site. Self-explanatory.</b>
<b>FAVICON is used as your site's favicon in base64.</b> This limits the amount of requests a browser may do when first loading the queue page. Make sure this value is set to something. Otherwise people's connections will get cut off from the queue when their browser makes a request to the favicon.ico.
<b>SQUARELOGO</b> is used as the icon for the queue running man and the main splash logo on the captcha page. In base64 format.
<b>NETWORKLOGO</b> is used as a bottom network icon for on the captcha page which allows different sites a part of the same organization to be shown. In base64 format.
After you are done EndGame's configuration, you should archive everything except the sourcecode folder. Transfer the archive to a blank debian 12 system. As root, extract the archive and run setup.sh like './setup.sh'. At the end of the setup, it will export an onion address (and i2p if set but don't add that to gobalance) which you can provide to users or add to your gobalance configuration.
</code></pre>
<p>Let's first generate our KEY (max 128chars) and SALT (8 chars):</p>
#session length is in seconds. Default is 12 hours.
SESSION_LENGTH=43200
#Rate Limits!
#Make sure to set these to reasonable defaults! Having them too low for your site can cause lots of disconnections while having them too high can make endgame ineffective!
#Set the request rate to the max requests on your largest page! (try to keep it below 10! The lower you go the better endgame's protection is!)
#Keep the stream limit 1 to 2 higher than your request limit!
If you are using the TORMINWORK=true setting generally you don't need gobalance at all. You can just have a single front handle the load. Gobalance can't handle POW at this time as there are tor control systems to really handle it at this time. When arti is finished with POW onionbalance will be finished in rust. We are waiting for that. Should be within the year.
If you are running endgame and onionbalance on the same server are you doing it wrong. Onionbalance is designed to combine multiple fronts into a single descriptor for load balancing. One address with multiple fronts holding it up. If you only have a single server just run endgame and pass the ending address to others. No onionbalance needed at all.
If you want to have a custom address created use a tool like mkp224o and generate a custom one. You can then transfer the custom one over to your front (replacing the /etc/tor/hidden_service files with the generated one). Then go into /etc/nginx/sites-enabled/site.conf and change out the onion address to your new one. After that reboot and your new custom address should be live on your front.
</pre></code>
<p>Now from here you can run the setup.sh file. However just in case if you messed up at any point up until now, i recommend you take a VM snapshot to revert to in case if something goes wrong.</p>
<imgsrc="1.png"class="imgRz">
<p>Snapshot taken, now let's run setup.sh:</p>
<pre><codeclass="nim">
root@debian:~/endgame# ./setup.sh
Welcome To The End Game DDOS Prevention Setup...
Proceeding to do the configuration and setup. This will take awhile.
The system will reboot after finishing setup!
Generating Master Key... should only take a second...
Apr 13 15:27:20 debian startup.sh[513]: Apr 13 15:27:20.586 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
Apr 13 15:27:20 debian startup.sh[521]: Apr 13 15:27:20.596 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9060
Apr 13 15:27:20 debian startup.sh[522]: Apr 13 15:27:20.610 [notice] Tor 0.4.8.11 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.0.11, Zlib 1.2.13, Liblzma 5.4.1, Libzstd 1.5.4 and Glibc 2.36 as libc.
Apr 13 15:27:20 debian startup.sh[522]: Apr 13 15:27:20.610 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
<p>And here you can see that the endgame systemd service launched successfully, but as hinted above, we need to edit the <b>gobalance config.yaml</b> file:</p>
<pre><codeclass="nim">
root@debian:~# vim endgame/sourcecode/gobalance/config.yaml
Apr 13 15:32:26 debian startup.sh[1004]: Apr 13 15:32:26.730 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
Apr 13 15:32:26 debian startup.sh[1006]: Apr 13 15:32:26.732 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9060
Apr 13 15:32:26 debian startup.sh[1007]: Apr 13 15:32:26.740 [notice] Tor 0.4.8.11 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.0.11, Zlib 1.2.13, Liblzma 5.4.1, Libzstd 1.5.4 and Glibc 2.36 as libc.
Apr 13 15:32:26 debian startup.sh[1007]: Apr 13 15:32:26.740 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
<p>Now before we continue, keep in mind that we will access the backend servers using the URL <b>http://gllrw5gzdvje5axxexdtncpxzbrgxp2l5hghbiysgtfxpjmloah2qrqd.onion/</b>, so we need to make sure that the nginx server there has that URL aliased:</p>
<p>Then after restarting the endgame service we check if it works as intended by accessing the main onion at the following URL <b>http://gllrw5gzdvje5axxexdtncpxzbrgxp2l5hghbiysgtfxpjmloah2qrqd.onion/</b>:</p>
<imgsrc="3.png"class="imgRz">
<p>Here as you can see we're placed into a queue before being redirected to the captcha:</p>
<imgsrc="2.png"class="imgRz">
<p>Then we fill in the captcha before getting redirected to one of the backend servers:</p>
<imgsrc="4.png"class="imgRz">
<p>And that's it ! We have been redirected to one of the backend servers as intended. We can redo the whole process again by picking a new Identity in the tor browser to see if it redirects us to the other backend server as intended:</p>
<imgsrc="5.png"class="imgRz">
<p>Now you can scale your hidden service operations over multiple backend servers, while still protecting against DDOS attacks.</p>
<p>Now there are 2 issues with that setup. First there are performance issues due to latency as in this current setup we have to go through Tor twice to get to the backend server, that's not ideal at all.</p>
<imgsrc="6.png"class="imgRz">
</br></br><b>Problem 2: Not big enough!</b>
<p>Second issue, why is there <ahref="https://blog.torproject.org/cooking-onions-finding-onionbalance/">onionbalance</a> in the mix and only one endgame frontend ?! As detailed in the article, onionbalance allows us to introduce high availability over a single onion service (like DNS Load balancing): </p>
<pre><codeclass="nim">
Onionbalance is software designed and written by Donncha O'Cearbhaill as part of Tor's Summer of Privacy 2015. It allows onion service operators to achieve the property of high availability by allowing multiple machines to handle requests for a single onion service. You can think of it as the onion service equivalent of load balancing using round-robin DNS.
</pre></code>
<p>The true goal here is to have multiple <b>Endgame fronts</b> spread across multiple servers, <b>under the same Master Onion Link!</b> So for this second part of the tutorial we have this following setup: </p>
<imgsrc="7.png"class="imgRz">
<p>So here we'll have 3 local backend servers for the same service, and we're going to have 2 Endgame servers, which will be under the same master subnet.</p>
<p>Let's first setup our 3 local backend servers, this time they will be 3 instances of the same service:</p>
<pre><codeclass="nim">
root@backend1:~# apt install nginx vim -y
root@backend1:~# vim /var/www/html/index.nginx-debian.html
<p>We clone that VM to the 2 other backend servers:</p>
<imgsrc="8.png"class="imgRz">
<p>We edit the web page of the 2 backend servers to know on which backend server we land, and we have the following: </p>
<pre><codeclass="nim">
[ LAN-Home ] [ /dev/pts/3 ] [~]
→ curl 10.99.99.216
Welcome to our Nihilist's service!
We are currently on backend server 1 !
[ LAN-Home ] [ /dev/pts/3 ] [~]
→ curl 10.99.99.219
Welcome to our Nihilist's service!
We are currently on backend server 2 !
</pre></code>
<!--<img src="10.png" class="imgRz">-->
<p>Our backend servers are now be reachable locally at the local IPs <b>10.99.99.216, and 10.99.99.219 on port 80</b></p>
<imgsrc="11.png"class="imgRz">
<imgsrc="12.png"class="imgRz">
<p>Now let's setup our 2 Endgame Fronts servers on 2 other VMs, this will be a repeat of what we went through above for our previous endgame setup:</p>
<imgsrc="13.png"class="imgRz">
<p>So as advised, here we need to configure endgame once, and then archive everything except the sourcecode</p>
<pre><codeclass="nim">
After you are done EndGame's configuration, you should archive everything except the sourcecode folder. Transfer the archive to a blank debian 12 system. As root, extract the archive and run setup.sh like './setup.sh'. At the end of the setup, it will export an onion address (and i2p if set but don't add that to gobalance) which you can provide to users or add to your gobalance configuration.
</pre></code>
<p> So on our Endgame front 1, we configure gobalance to generate the .key file: (not on Endgame front 2!)</p>
<pre><codeclass="nim">
root@endgame1:~/endgame/sourcecode/gobalance# go get -u ; go mod vendor ; go build -o gobalance main.go ; ./gobalance g
#session length is in seconds. Default is 12 hours.
SESSION_LENGTH=43200
#Rate Limits!
#Make sure to set these to reasonable defaults! Having them too low for your site can cause lots of disconnections while having them too high can make endgame ineffective!
#Set the request rate to the max requests on your largest page! (try to keep it below 10! The lower you go the better endgame's protection is!)
#Keep the stream limit 1 to 2 higher than your request limit!
<p>Here notice that we set both TORPOWDEFENSE and TORMINWORK to false, this is due to not being supported by gobalance as gobalance does not handle POW and won't combine the descriptors correclty. We also left the BACKENDONIONs to empty strings, and instead changed the LOCALPROXY to true, with the PROXYPASSURL set to one of our local backend servers 10.99.99.216. as advised in the README:</p>
<pre><codeclass="nim">
Local Proxy: Change LOCALPROXY to true and edit the PROXYPASSURL to the specific IP or hostname of your backend location. It will default to connect on port 80 via http but you can edit line 320 of the site.conf to change that to your specific needs.
</pre></code>
<p>Then here, we need to archive the entire endgame folder, and extract it to our endgame2 front before running setup.sh:</p>
<pre><codeclass="nim">
root@endgame1:~/endgame# cd ..
root@endgame1:~# tar -czvf EndgameV3-nihilism.tar.gz endgame >/dev/null
root@endgame1:~# ls
endgame EndgameV3-nihilism.tar.gz go
</pre></code>
<p>Now that the archive is created, let's transfer it to endgame2 and extract it: </p>
<pre><codeclass="nim">
[term1]
root@endgame1:~# ip a | grep inet ; python3 -m http.server 9090
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host noprefixroute
inet 10.99.99.221/24 brd 10.99.99.255 scope global dynamic enp1s0
inet6 fe80::5054:ff:fe00:3bfa/64 scope link
Serving HTTP on 0.0.0.0 port 9090 (http://0.0.0.0:9090/) ...
<p>Now back to our endgame front 1: we run setup.sh:</p>
<pre><codeclass="nim">
root@endgame1:~/endgame# ./setup.sh
Welcome To The End Game DDOS Prevention Setup...
Proceeding to do the configuration and setup. This will take awhile.
The system will reboot after finishing setup!
</pre></code>
<p>Let it run and take note of the endgame front instance TOR hostname as displayed at the end, which will be our first endgame instance's .onion link:</p>
#session length is in seconds. Default is 12 hours.
SESSION_LENGTH=43200
#Rate Limits!
#Make sure to set these to reasonable defaults! Having them too low for your site can cause lots of disconnections while having them too high can make endgame ineffective!
#Set the request rate to the max requests on your largest page! (try to keep it below 10! The lower you go the better endgame's protection is!)
#Keep the stream limit 1 to 2 higher than your request limit!
<p>Once that's done, we run setup.sh on the Endgame front 2 VM, here also let it run and take note of the endgame front instance TOR hostname as displayed at the end:</p>
<pre><codeclass="nim">
root@endgame2:~/endgame# ./setup.sh
Welcome To The End Game DDOS Prevention Setup...
Proceeding to do the configuration and setup. This will take awhile.
<p>Then we test if both endgame fronts work starting with front 1 on the URL http://xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd.onion :</p>
<imgsrc="23.png"class="imgRz">
<p>and we test the other one on the URL http://e7nqqkmeripx7hxov4hy32ovg34nxsrooy7ipxaeaw5edkek73dgowad.onion :</p>
<imgsrc="22.png"class="imgRz">
<p>And lastly we enable gobalance to make sure that the masteronion can redirect to either of the 2 after editing the config.yaml file: </p>
INFO[2024-04-14 19:26:24] [ADAPTIVE] Waiting for 2 instance descriptors.
INFO[2024-04-14 19:26:24] [*] FetchInstanceDescriptors() called [*]
INFO[2024-04-14 19:26:24] Initiating fetch of descriptors for all service instances.
INFO[2024-04-14 20:02:22] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $5FAE28CF4D1C520341EE104BF72516F4308B9485~nuker
INFO[2024-04-14 20:02:22] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $778DCB9DB6CDD5FF2F1A85571308B492D6DFF962~TorHet
INFO[2024-04-14 20:02:22] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $B72663DDF48F7047003DE6E3927936994DA44152~Unnamed
INFO[2024-04-14 20:02:22] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $7AC2E7E67ADAC32F6F46AF607896637BBB455D64~mysocratesnote
INFO[2024-04-14 20:02:23] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $484F666C491BCDE22B45E0E19D1CEA5ACC5A9611~WinstonSmith
INFO[2024-04-14 20:02:24] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $3CA0D15567024D2E0B557DC0CF3E962B37999A79~QuintexAirVPN30
INFO[2024-04-14 20:02:24] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $846B3EAAF0C07FF72FC79AEBB11FA3ADC58F240F~dc6jgk5b
INFO[2024-04-14 20:02:24] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $455469D1C610E43498ECF88E83E29C0A694EF73B~whyza1
INFO[2024-04-14 20:02:24] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $65379EED488599B0A1512E80D4743BC9125CB306~BlueMold
INFO[2024-04-14 20:02:25] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $C111AC86B3719F0A2FB6254725CB5A7A62C9B451~LauchYT
INFO[2024-04-14 20:02:25] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $235396838BB8FC7AFA529042B19615DF9E2AF218~soP49mzpYUFEwVdiFN3
INFO[2024-04-14 20:02:26] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $4ADB08AFCF04657E0A0288AA230EEB74A96B1CEE~Lemminkainen
INFO[2024-04-14 20:02:27] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $961B9D86125A08FAA9F7E742B228307EF7E2E082~TorNodeCzech
INFO[2024-04-14 20:02:28] Successfully uploaded descriptor for xkdrgt35dw4rtmqacjdaymocotcvarfqlodtw2dfykqn3rk2eqxmr4yd to $FBDE535D810756541B31C452C6694876C96A7FC1~BSDnodeAMS
</pre></code>
<p>And now we test if the masteronion <b>http://ehfs47i5jjzlyolgd24ogkndodz7n4mlbcmhdakyma4lfh2alossuvad.onion/</b> is reachable:</p>
<imgsrc="18.png"class="imgRz">
<imgsrc="19.png"class="imgRz">
<p>Here we see that we were on the endgame front 1 instance, which redirected us to the backend server 1:</p>
<imgsrc="20.png"class="imgRz">
<p>Then we check if it still works after shutting down the endgame front 1:</p>
<pre><codeclass="nim">
root@endgame1:~# shutdown now
</pre></code>
<p>As you can see it still works:</p>
<imgsrc="18.png"class="imgRz">
<imgsrc="19.png"class="imgRz">
<p>And as you can see here we're being redirected to the backend server 2 as intended:</p>
Until there is Nothing left.</p></br></br><p>Creative Commons Zero: <ahref="../../../../opsec/runtheblog/index.html">No Rights Reserved</a></br><imgsrc="\CC0.png">
