Markdown
Server Management H3
This is a report for the Configuration Management Systems course taught by Tero Karvinen
Date: 15-11-2020
This report is written completely in markdown. As a guide for the syntax i used this cheatsheet. Ill leave a link to the actual .md file over here.
I was reading this book on O'reilly learning and in chapter 6 i learned that salt supports having salt-states in a git repository natively. From what i understand it should be done by changing a few things in the salt-master config. I did look at this documentation for this task as well.
Before i tried configuring salt to use a git repository i did a few other things. I started by creating a git repository on github and cloned it to my master with the following command.
git pull https://github.com/heiskane/salt-states.git
I added a simple state to add a file to the slave and with the following commands added it to my repository.
git add .
git commit -m 'hello world'
git push
Note that i already have set my email and username for git so i didnt need to do it here.
Before moving on i looked at the output of git log --patch (after adding something more and pushing it).
This will show all the changes made to the repository as seen below
To show the usecase for git diff i added some more text in hello.txt and ran the command.
Finally i ran the git blame command on the hello.txt file.
This shows when each line in the file was modified and by whom. I pushed the changes i made before moving on.
To do a stupid change to the repository i ran head /dev/urandom >> hello.txt. To revert changes and go to the previous version i ran git reset --hard.
As seen in the following image the change i did was reverted and i have returned to the working version.
At this point i wanted to configure salt to use my git repository so i edited my configuration file at /etc/salt/master. I added the following lines:
fileserver_backend:
- gitfs
gitfs_remotes:
- https://github.com/heiskane/salt-states.git
After doing these changes i restarted salt master with the following command.
sudo systemctl restart salt-master.service
I tried applying the state but got a bunch of errors.
Atfer a while of googling i remembered that salt uses the master branch as the base environment but because i created a repository on GitHub my master branch is actually called main. I realized this while lookin at this article.
I tried adding the following configuration:
gitfs_remotes:
- https://github.com/heiskane/salt-states.git
- saltenv:
- base:
- ref: main
I restarted salt-master but this gave me a bunch of errors. Atleast now i have something to debug.
I found this post on stackoverflow and decided to try removing the addition i made so the gitfs_remotes looks like this again:
gitfs_remotes:
- https://github.com/heiskane/salt-states.git
I restarted salt-master and tried applying my state with the following command:
sudo salt slave-1 state.apply hello saltenv=main
As seen in the image this was a success so now i want to either add the main branch as the base environment or try to rename the main branch to master like it probably should be to begin with. After failing to find any clear documentation on how to change the default branch i decided to just change the name of my main branch to master. I found this handy guide that showed how to do the opposite of what i was doing but it was easy to go off that anyway. I started by renaming my branch to master and pushing it to github.
git branch -m main master
git status
git push -u origin master
After doing this i had to go to the repository branch settings on github and change the default branch to the master branch i just created.
After changing the default branch i deleted the main branch with the following command.
git push origin --delete main
Now running sudo salt 'slave-1' state.apply hello will look for the hello state in my github repository.
Do note that at this point salt looks for states only in my github repository. This can be changed so that salt uses both the remote repository and the local /srv/salt directory by changing fileserver_backed configuration as follows:
fileserver_backend:
- gitfs
- roots
Now i can run states from /srv/salt and my repository on github. To verify this works i ran a state from /srv/salt.
With all of that out of the way it was time to create a new state and this time i wanted to configure a proxy that would work with proxychains. I had experimented a bit with proxies like squid but this time i wanted to get a SOCKS5 proxy running. From what i have read there are many advantages SOCKS5 has over an http proxy like squid. According to this article the SOCKS is designed to route any type of traffic generated by any protocol or program. To achieve this i decided to try dante and mostly used this guide for it.
I started by actually installing it.
sudo apt install dante-server
Note that when i create the slaves with vagrant they are updated in the process so i dont do that here. As instructed in the guide i made a backup for dantes configuration file with the following command.
sudo mv /etc/danted.conf /etc/danted.conf.bak
Now to create the actual configuration file i opened it with sudoedit /etc/danted.conf and copy-pasted the configuration from the guide into my file. I only changed the interface to eth1 because that is the interface for the NAT network my machines are on. Ill leave my current configuration here.
logoutput: /var/log/socks.log
internal: eth1 port = 1080
external: eth1
clientmethod: none
socksmethod: none
user.privileged: root
user.notprivileged: nobody
client pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: error connect disconnect
}
client block {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect error
}
socks pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: error connect disconnect
}
socks block {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect error
}
After a quick restart with sudo systemctl restart danted i made sure it was working with sudo systemctl status danted and ss -ltn.
As seen in the image the dante service is running and listening on port 1080. To actually use it i wanted to use proxychains4. This can be installed with the following command. I did this part on another VM on the same network.
sudo apt install proxychains4
To configure proxychains i opened the configuration file with sudoedit /etc/proxychains4.conf and added the following configuration.
dynamic_chain
proxy_dns
remote_dns_subnet 224
tcp_read_time_out 15000
tcp_connect_time_out 8000
[ProxyList]
socks5 172.28.128.88 1080
Finally to use the dante proxy with proxychains4 i used the following command.
proxychains curl http://heiskanen.rocks
While the first time i ran the command it took a while to respond but it worked and sped up on the subsequent tries.
As a quick note my website is returning a 301 response because the http requests are redirected to https. With the manual configuration out of the way it was time shut these machines down and spin up some fresh minions. I started working on the sls file in my git repository by creating the init.sls file and added a simple state to install dante.
install_dante:
pkg.installed:
- name: dante-server
service.running:
- name: danted
- enable: True
- require:
- pkg: install_dante
Trying to run it i quickly realized that i would have to push the changes first to apply this state. I didnt want to push what i havent tested yet so i tried to add the following configuration to my salt master.
gitfs_remotes:
- https://github.com/heiskane/salt-states.git
- file:///home/niko/salt-states
I believe this is the way to make it work but for some reason it didnt. So for now ill work on this state locally then add it to the repository once its working. After moving my dante directory to /srv/salt i had to cange the permissions to be able to edit it.
sudo chown -R root:root dante/
I tried breaking the local dante/init.sls to see which one get excecuted first. Applying the state before i moved the file was successful so i know it works. After breaking the local file everything still ran just fine so salt was getting the state from github first. I tried changing the order of the fileserver_backend so they would look like this.
fileserver_backend:
- roots
- gitfs
But this didnt make it prefer the local files either so i decided to just remove the dante directory from my repository for now and work on it locally. Okay so running the state wasnt entirely successful. The installation went just fine but when trying to make sure it is running i got the following error.
ID: install_dante
Function: service.running
Name: danted
Result: False
Comment: Service danted is already enabled, and is dead
Started: 15:19:58.662098
Duration: 144.061 ms
At this point i remembered that when manually testing it wasnt successfully running before i added the configuration file so this behaviour i somewhat expected. I guess i have to add the service.running module later and just add the config file first. To get the config on the target i used a similar state.
add_config:
file.managed:
- name: /etc/danted.conf
- source: salt://dante/danted.conf
- require:
- pkg: install_dante
With this file added i wanted to add the service.running module back but after adding the file this time. I edited my dante.conf just so it would be diffrent and the watch would reload dante service.
reload_dante:
service.running:
- name: danted
- enable: True
- reload: True
- watch:
- file: /etc/danted.conf
- require:
- pkg: install_dante
Trying to run this still gave me an error.
ID: reload_dante
Function: service.running
Name: danted
Result: False
Comment: Failed to reload danted.service: Job type reload is not applicable for unit danted.service.
Started: 15:37:20.447768
Duration: 28.773 ms
Remembering that i used restart in my manual configuration i tried changing the reload to restart in the sls file. I changed the conf file again and applied the state. This time it ran without any issues.
----------
ID: reload_dante
Function: service.running
Name: danted
Result: True
Comment: Service restarted
Started: 15:38:43.306572
Duration: 66.47 ms
Changes:
----------
danted:
True
Summary for slave-1
------------
Succeeded: 3 (changed=2)
Failed: 0
------------
Total states run: 3
Total run time: 117.837 ms
At this point i ran systemctl status danted and ss -ltn on the slave to verify that the danted service was running and listening on the specified port. Lastly the most essential test is to actually use it for what its for. I logged into a fresh machine then installed and configured proxychains4 as i had done previously only changing the proxy ip. Running the same proxychains command shows that the configuration was successful.
With the proxy up and running i want to try and get it working with authentication. I followed the previous guide again and made some changes to the /etc/danted.conf so that it looks like this. I did this part locally on the slave first
# danted.conf
logoutput: /var/log/socks.log
internal: eth1 port = 1080
external: eth1
clientmethod: none
socksmethod: username
user.privileged: root
user.notprivileged: nobody
client pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: error connect disconnect
socksmethod: username
}
client block {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect error
}
socks pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: error connect disconnect
}
socks block {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect error
}
Running the proxychains command again gives denied now as expected.
vagrant@slave-3:~$ proxychains4 curl http://heiskanen.rocks
[proxychains] config file found: /etc/proxychains4.conf
[proxychains] preloading /usr/lib/x86_64-linux-gnu/libproxychains.so.4
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] Dynamic chain ... 172.28.128.89:1080 ... heiskanen.rocks:80 <--denied
curl: (7) Couldn't connect to server
With that tested i added username and password to the /etc/proxycahins4.conf file like this.
[ProxyList]
# add proxy here ...
# meanwile
# defaults set to "tor"
socks5 172.28.128.89 1080 vagrant vagrant
Running the command again gives a successful response from the website.
[proxychains] config file found: /etc/proxychains4.conf
[proxychains] preloading /usr/lib/x86_64-linux-gnu/libproxychains.so.4
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] Dynamic chain ... 172.28.128.89:1080 ... heiskanen.rocks:80 ... OK
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://heiskanen.rocks/">here</a>.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at heiskanen.rocks Port 80</address>
</body></html>
Now that the authentication is working i want the create a separete user for the proxy. I ran the command seen in the guide i mentioned earlier to create a system user without a home directory.
sudo useradd -r heiskane
sudo passwd heiskane
I changed the proxychains config again to use this new user.
socks5 172.28.128.89 1080 heiskane salasana
Somehow this worked without any kind of issues. I guess i can back to automating this with salt. I started by moving this new danted.conf file to my salt master and started working on making the new user. I used the user.present module for which i created a password with openssl passwd -6. For some reason the salt documentation suggests using -1 for an MD5 but that does not sound like a very smart idea so i used -6 for a SHA-512 hash instead. I added this state under the install_dante one.
dante_user:
user.present:
- name: heiskane
- shell: /usr/sbin/nologin
- password: $6$Pojx5jnjRxDO2E96$tzZazL8Pvdv3YaLULra.UukbBPCE/Pn3g6smmbODAI6teQEBA4j4iJzGWrALYcj/z.4BKlEU/d2gLbbEbr99Y1
Applying this state seems to have worked but because the user was already present so i wanted to apply this state to a machine that i havent touched yet. Applying the state worked just fine again so i went ahead and sanity checked it by using proxychains from another machine and again somehow everything worked just fine. Since everything was going so well i wanted to make dante only to allow users that are part of the correct group as well. Before actually trying to do anything i added this working state in my git repository with the name git_dante. I looked at this documentation and decided to just try adding a group to my config (locally on the slave). I added a group to the following part in /etc/danted.conf.
client pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: error connect disconnect
socksmethod: username
group: socksusers
}
Trying to use the proxy now just gives a timeout. I expected it to give access denied but i guess this is somewhat expected behaviour since i havent added anyone to the socksusers group.
First i created the group with sudo groupadd socksusers and added my user to the group with sudo usermod -aG socksusers heiskane. To verify that heiskane was part of the socksusers group i ran id heiskane which gave me the following output.
uid=997(heiskane) gid=997(heiskane) groups=997(heiskane),1001(socksusers)
Trying to use proxychains now still gave me a timeout. I checked danted service and apparently it failed to start after i added the group to it.. Running systemctl status danted gives the following errors.
Nov 13 16:53:40 slave-1 systemd[1]: Starting SOCKS (v4 and v5) proxy daemon (danted)...
Nov 13 16:53:40 slave-1 systemd[1]: Started SOCKS (v4 and v5) proxy daemon (danted).
Nov 13 16:53:40 slave-1 danted[18077]: Nov 13 16:53:40 (1605286420.581519) danted[18077]: warning: openlogfile(): could not open or create logfile "/var/log/socks.log" for writing: Read-only file system
Nov 13 16:53:40 slave-1 danted[18077]: Nov 13 16:53:40 (1605286420.581896) danted[18077]: alert: configparsing(): could not (re)open logfile "/var/log/socks.log": Read-only file system
Nov 13 16:53:40 slave-1 danted[18077]: Nov 13 16:53:40 (1605286420.582492) danted[18077]: error: /etc/danted.conf: problem on line 15 near token "}": the settings in this client-rule requires the client to provide a user/group-name in some way, but this rule specifies, implicitly or explicitly, the clientmethod none, which cannot be depended on to provide this information. Please see the Dante manual for more information
Nov 13 16:53:40 slave-1 danted[18077]: Nov 13 16:53:40 (1605286420.582626) danted[18077]: alert: mother[1/1]: shutting down
Nov 13 16:53:40 slave-1 systemd[1]: danted.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 16:53:40 slave-1 systemd[1]: danted.service: Failed with result 'exit-code'.
This error seems like the issue was with the clientmethod so i looked at the documentation and noticed that rfc931 and pam were the only available options so i chose rfc931. With the clientmethodset to rfc931 i managed to get dante running but trying to use it gave me some errors despite my user being in the socketusers group.
vagrant@slave-3:~$ proxychains4 curl http://heiskanen.rocks
[proxychains] config file found: /etc/proxychains4.conf
[proxychains] preloading /usr/lib/x86_64-linux-gnu/libproxychains.so.4
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] Dynamic chain ... 172.28.128.89:1080 ... heiskanen.rocks:80 <--socket error or timeout!
curl: (7) Couldn't connect to server
After looking into it more it seems like dante wants the user to provide username or group somehow but doesnt accept the way that proxychains is doing it(?). It seems odd that just using the username works just fine but the group requires another level of authetication. Either way im not going to bother with the group option any longer and will stick with the username and password authentication i already have working. while slowly finishing up for the day i did notice one thing that ill have to look into and thats the fact that salt creates the home directory for the user by default it seems. Currently the module is working just fine so ill keep it this way for now. Ill leave the full init.sls below
install_dante:
pkg.installed:
- name: dante-server
dante_user:
user.present:
- name: heiskane
- shell: /usr/sbin/nologin
- password: $6$Pojx5jnjRxDO2E96$tzZazL8Pvdv3YaLULra.UukbBPCE/Pn3g6smmbODAI6teQEBA4j4iJzGWrALYcj/z.4BKlEU/d2gLbbEbr99Y1
add_config:
file.managed:
- name: /etc/danted.conf
- source: salt://dante/danted.conf
- require:
- pkg: install_dante
reload_dante:
service.running:
- name: danted
- enable: True
- restart: True
- watch:
- file: /etc/danted.conf
- require:
- pkg: install_dante
# Enable dante here even if conf file is not changed
# because dante needs a working config file before it can run
danted:
service.running:
- enable: True
- require:
- pkg: install_dante
One thing was still bothering me and i wanted to still figure out. I wanted to be able to work on modules locally before pushing them in my git repository easily. I remembered that i could create environments in the /etc/salt/master config file so i created a dev environment like shown below.
file_roots:
base:
- /srv/salt
dev:
- /srv/salt-dev
I restarted salt-master and put a random working state in the /srv/salt-dev directory then ran it with the following command.
sudo salt slave-1 state.apply local saltenv=dev
Applying this was successful so it got me thinking if i could just use the base environment to run the local sls files. I renamed the git_dante directory to dante in my repository and broke the local dante init.sls file so ill know which file gets executed. Applying the state with sudo salt slave-1 state.apply dante env=base failed as expected but running it with sudo salt slave-1 state.apply dante failed as well so it seems like it prefers the local files after all. Not sure where i got confused about all this but i think that ill have to use the dev environment locally. One idea i had was to use the local directory for the repository for my dev environment so i modified my salt master config as follows.
file_roots:
base:
- /srv/salt
dev:
- /home/niko/salt-states
Now when i run sudo salt -v slave-1 state.apply dante saltenv=dev the local files are used and i dont have to push experimental tweaks to github. I verified that this was working by breaking the init.sls file in the repository but didnt commit any changes. Applying the state with sudo salt slave-1 state.apply dante saltenv=dev failed as expected while running without specifying the saltenv worked just fine. Finally ill leave a link to the github repository below.