Salt intro

This is a report for the Configuration Management Systems course taught by Tero Karvinen

Date: 31-10-2020

To start working with salt i had to install it first. I ran into some issues because salt wasnt in the apt packages for Ubuntu 20.04. This issue was easily solved by looking at the documentation and following the instructions. Documentation: https://repo.saltstack.com/#ubuntu

Note that the second command has to be run as the root user. As seen in the documentation the commands to add salt to apt are:

wget -O - https://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
echo 'deb http://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest focal main' > /etc/apt/sources.list.d/saltstack.list

Now i just had to update my machine and install salt-master.

sudo apt update -y && sudo apt upgrade -y
sudo apt install salt-master

To verify that salt-master is running i ran the following command:

systemctl status salt-master
image

I initially didnt notice i had a firewall enabled so my slave wasnt connecting to the master. After noticing this i added the salt ports (4505 and 4506) to the allowed ports in my firewall.

sudo ufw status
sudo ufw allow 4505
sudo ufw allow 4506
sudo ufw enable
sudo ufw status

To create a minion i downloaded vagrant from over here https://www.vagrantup.com/downloads and with these commands (in PowerShell) i started a new Ubuntu 18.04 machine:

vagrant init bento/ubuntu-18.04
vagrant up

Source: http://terokarvinen.com/2017/04/11/vagrant-revisited-install-boot-new-virtual-machine-in-31-seconds/index.html?fromSearch=

To connect to this new machine i used:

vagrant ssh

On the slave machine i ran updates and installed salt-minion

sudo apt update
sudo apt install salt-minion

Again i verified that the service is running with systemctl

image

Now to configure a master i edited the configuration file in /etc/salt/minion with "sudoedit" to add the master and an id:

sudoedit /etc/salt/minion
image

At this point my slave wasnt connecting to the master so i noticed that vagrant set its network adapter to NAT while my master was on its own NAT Network. I could have probably put the slave on the same network relatively easily but for now i honestly just didnt feel like restarting any machines so i just put my master on the same network as the slave.

At this point i relearned why i was using the NAT Network mode in the first place (so VMs can communicate to each other). For the machines to communicate to each other i thought i have to put them on my NAT Network but after a while of reading some documentation it seems like my only options to vagrant were NAT and Host-Only Network. I looked at some more documentation and noticed that on the Host-Only network the VMs can communicate to each other.

image

To have vagrant create a machine that connects to the Host-Only Network i had to create a Vagrantfile with the following content:

image

To use this file i ran “vagrant up” in the directory where the file was located.

Creating a VM based off this file will make a Ubuntu 18.04 machine and put it on Host-Only and give it a static ip (because i ran into some errors using dhcp). Initially i used 10.0.2.69 as the ip but with that i could ping the master but couldnt connect to the 4505 and 4506 port so i looked at the masters ip (198.168.56.102) and assumed they should be in the same address space so it worked with 192.168.56.199 as the slave ip. The only issue i have at the moment is that my master doesnt have internet connection but i can fix that when i shut it down by adding a network adapter with NAT mode to it. Now i just had to redo everything i had done to the slave and use the new master ip.

To recap i have Ubuntu 20.04 with Host-Only Network adapter as the master and the slave i have created with the vagrant file so that it has Host-Only Network adapter (and NAT) with a static ip within the same space as the masters ip.

Some sources i used to get the master and slave to communicate to each other: https://www.nakivo.com/blog/virtualbox-network-setting-guide/ https://app.vagrantup.com/bento/boxes/ubuntu-18.04 https://www.vagrantup.com/docs/providers/virtualbox/networking https://medium.com/@JohnFoderaro/how-to-set-up-a-local-linux-environment-with-vagrant-163f0ba4da77

After all this the slave finally communicates to the master. Running the command seen in the image below i can see the fingerprint of the slave

image

To verify this fingerprint the following command can be run on the slave:

sudo salt-call --local key.finger
image

I learned this for this book: https://www.packtpub.com/product/learning-saltstack-second-edition/9781785881909

This key can be accepted with the following command:

sudo salt-key -A
image

Note that this command accepts all pending keys. An individual key can be accepted with the following command:

sudo salt-key -a slave_name

I also verified the connection by running a command on the slave

image

Now to create a salt state i followed these instructions: http://terokarvinen.com/2018/salt-states-i-want-my-computers-like-this/

First i created the directory for the sls file with:

sudo mkdir -p /srv/salt/hello

Then i created a salt state and a file to copy to the slave with:

sudoedit /srv/salt/hello/hello.sls
sudoedit /srv/salt/hello/hello.txt

In the sls file i created a state that uses the file.managed module to copy a file to the slaves tmp directory.

image

Initially i tried applying my state like this:

image

But figured out that because i put it in a subdirectory i need to specify it in the command like this:

sudo salt '*' state.apply hello.hello
image

This might look a little weird but the first "hello" is the name of the subdirectory in /srv/salt and the second one is the name of the sls file.

I then verified that this worked by making sure the file exists on the slave.

image

Now to collect some of that sweet data from the slave i ran the following command:

sudo salt '*' grains.items | less

There i looked for some interesting items and used the following command to gather more specific data:

sudo salt --out=json '*' grains.item osfinger kernelrelease ipv4 | jq

As seen in the command i used json as the output format and piped it over to jq to get the nice colors.

image

Source: http://terokarvinen.com/2018/salt-quickstart-salt-stack-master-and-slave-on-ubuntu-linux/

Before i moved on i wanted to create more slaves because using all this just for one slave is a bit overkill. I looked at some documentation and created this vagrant file:

image

Source: https://scotch.io/courses/getting-started-with-vagrant-for-local-development/provisioning-vmprovision-how-to-customize-the-box

This will configure the new vagrant vm automatically so i dont have to touch it but it still creates only one machine. While i was tinkering with this i decided to fix this networking as well. I wanted to make my master server visible from the outside world and i didnt want to deal with the Host-Only adapter on my slaves. I set my (master) Ubuntu VM to use my NAT Network and forwarded the salt ports to my public ip in virtualbox.

image

Now all the slaves can just connect to my public ip to get instructions. I verified that this was working by running the following commands on my webserver (in germany):

nc -vz MY_IP_HERE 4505
nc -vz MY_IP_HERE 4506

To make vagrant startup multiple machines i looked at some more documentation. Not gonna lie this took a while to automate nicely but i eventually succeeded. I stubbornly wanted to do this in a for loop in the vagrant file and had some issues giving each machine a diffrent minion id. I didnt manage to get the variables in the vagrant file to work so decided to just give each machine a random md5 hash (for now) at the end of the name. The last thing that gave me issues was the fact that i was restarting salt-minion in the wrong place. Apparently the commands in the vagrant file are excecuted from the outside first working inwards. I moved my restart command inside the commands in the loop and eventually it all worked out. Note that i changed the master ip to my public ip.

image

Sources:
https://www.vagrantup.com/docs/multi-machine https://www.vagrantup.com/docs/vagrantfile/tips https://www.vagrantup.com/docs/provisioning/shell.html https://groups.google.com/g/vagrant-up/c/EUON-Jh2ASA

I didnt find a built-in way of running a command on each vagrant vm so i decided to just do some PowerShell magic. The following command will work as a for loop that runs a command on each of my (3) vagrant VMs

1..3 | ForEach-Object { vagrant ssh -c 'sudo salt-call --local key.finger' slave_$_ }

Sources:
https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7
https://www.calebwoods.com/2015/05/05/vagrant-guest-commands/

This command came in very handy when troubleshooting the vagrant machines but now i can use it to look at the fingerprint of all the slaves and verify that they match with the ones im seeing on the master then accept them all.

image image

Now i have a working vagrant file that creates any amout of virtual machines that i want and configures them as slaves to my master so i can jump straight to configuring them with salt.
Finally to try some other state functions i created a new subfolder (called heiskane) in /srv/salt and added a file called "testing.sls". In there i added states to make sure curl is installed, add a user and make sure salt-minion is running (which will obviously be always true but im just testing). I applied this state with the following command:

sudo salt '*' state.apply heiskane.testing
image image

Sources:
https://docs.saltstack.com/en/latest/ref/states/all/salt.states.user.html https://docs.saltstack.com/en/latest/ref/states/all/salt.states.service.html

In the end i did also try running 8 machines with vagrant and everything was working smoothly without issues. All i had to do was to set my loop in the vagrant file to go up to 8 and run "vagrant up" in the same directory as the file. A moment later i had 8 slaves connecting to my master.

image