When installing, configuring and fine-tuning the software my server runs, many modifications occur.
Applications may tamper with settings when cron jobs run them.
I may tinker with properties in an ini file to improve performances.
Because of these changes, an application on the server may later stop working.
When this happens, I need to figure out what happened and fix it.
Version control tells me what changes intervened.
It's precious information I use to examine the problem.
What is version control?
I'm using git. It's the version control system Linus Torvalds created and it's widely used.
I put together many files, usually containing only text, that I need to build an application or for any other reason.
I tell git to track them.
Every now and then I commit. It means that I tell git to save the files as they currently are. This results in many versions of them.
When the application stops working, git tells me what I modified since the last working version. I will be able to find the difference that crippled it.
Why to use git to control a server's configuration?
If I control my server's configuration using git, I'll fix problems more easily. It's like using git to track application code. It let me know what changed since the last time everything was running smoothly.
When I install a package, it may overwrite one or more configuration files. The package installer may tell me what it's about to change or it may not. I may let the package do its job and forget to check if it will have side-effects. Even if I do check, there may later be undesired effects.
I may edit system files like hosts or fstab to correct a malfunction, but there may later be problems somewhere else.
I may be fine-tuning my server's performance and want to keep the fixes I make to later compare the results.
Users who were able to log in to the server, may not be able do it any more and I suspect that an intruder meddled with ssh-related files.
I may be configuring a website in Nginx or Apache and testing different configurations to find the best one.
I may assign multiple ip addresses to a single network card. This configuration needs to be tested with similar ones running on other servers, maybe remote ones.
I may reduce the property innodb_buffer_pool_size in the file /etc/mysql/my.cnf because MySQL runs all the same and I need memory for another application. This affects a script that runs at night when another system administrator is in charge. The latter will find git very useful because he will know what changed since the last time the now-broken script run correctly.
Why I need to set a couple of environment variables to use git in this scenario
Usually git finds the files it's supposed to track in a single folder.
For example, the files that compose my website https://leadershipcoachfortech.com/ are in the folder /vol/WORKnARCH/SwProjects/leadershipcoachfortech.
But, in this case, I'm using git to track files system-wide.
In the folder /var/spool/cron/crontabs/ there are the crontab files, in /srv/scripts/ there are the scripts cron runs, and /root/.bashrc and /root/.profile contain interesting environment variables.
Git must work from the only folder that encompasses all these files, the folder '/'.
But I don't want the git repository to be in the root folder, it's a messy arrangement.
This means that I need to have the following two variables in the environment.
The first one tells git where to put the repository, the place where git keeps the history of changes. The second tells git that the files to track are under the folder '/'.
How I set these two environment variables
I run a script before starting to operate on this git repository.
I don't add the variables to files like .bashrc or .profile because I don't want them to affect git operations for the entire system. There may be other git repositories that wouldn't work if these variables are set.
The script is the following one.
[[ == ]] ; GIT_WORK_TREE=/ GIT_DIR=/srv/gitrepo4config
What git is not supposed to track
The environment variable GIT_WORK_TREE I set above tells git that, potentially, it can track the entire file system. I limit the tracking scope using the file .gitignore git refers to for instructions about what not to track.
Initially, I tell git to track nothing and then I incrementally add folders and files to track.
Here it is the .gitignore file.
* !*/ /root/.wp-cli /root/drush-backups /srv/sites/intranet.emanuelesantanche.com /srv/sites/rankit.emanuelesantanche.com /usr/lib/nodejs !/etc/** !/var/spool/cron/crontabs/** !/srv/scripts/** !/root/.ssh/** !/root/.config/rclone/** !/root/.bashrc !/root/.profile /root/.config/rclone/rclone.conf *.rpmnew *.rpmorig *.rpmsave *.old blkid.tab blkid.tab.old nologin ld.so.cache prelink.cache mtab mtab.fuselock .pwd.lock *.LOCK network/run adjtime lvm/cache lvm/archive X11/xdm/authdir/authfiles/* ntp.conf.dhcp .initctl webmin/fsdump/*.status webmin/webmin/oscache apparmor.d/cache/* service/*/supervise/* service/*/log/supervise/* sv/*/supervise/* sv/*/log/supervise/* *.elc *.pyc *.pyo init.d/.depend.* openvpn/openvpn-status.log cups/subscriptions.conf cups/subscriptions.conf.O fake-hwclock.data check_mk/logwatch.state *~ .*.sw? .sw? \ DEADJOE
Let me put git at work
Now I need to tell git to create a repository and copy to it the files I want to track.
I use the following command.
Git creates an empty repository. I need to copy the files to it. But I don't want to risk to copy the entire file system to it if I make mistakes defining the file .gitignore.
I simulate the copy.
Git will list the files it will copy to the repository when I'm ready to do it for real.
When I'm satisfied with the dry run, I start the copy.
I tell git that the first version of the files is complete.
How do I check the changes?
During the day, I will make changes to configuration files. Running processes will cause many more of them.
Giving a git status command, I know which files were modified. The command also tells me if files were created.
The command git diff shows the changes, line by line.
I use this information to solve problems the changes may have caused.
Then I add the changes to the repository.
Automating git status check
I don't want to do this every day manually. I write a script that automates this task away.
GIT_WORK_TREE=/ GIT_DIR=/srv/gitrepo4config git status -s
Then I invoke it from crontab. It will run at 07:40 UTC every day.
40 7 * * * /srv/scripts/check-systemwide-git-repo.sh
If there are modifications, the server will send me an email with a list of them.
I will log in to the server and figure out why the changes occurred. Then I will run a git add -A command and a commit like above.
Automating the commit
I'm happy with having to commit the changes manually.
In the future, I may want to automate this activity.
I'll use an improved version of the script above.
It will be like this.
GIT_WORK_TREE=/ GIT_DIR=/srv/gitrepo4config PATH_GIT_STATUS_OUTPUT=/tmp/`basename `-git-status.out git status -s > [[ -s ]] ; cat git add -A git commit -m
I will still be able to edit the comment if I want it to be more expressive than the generic "Auto commit".
The command git commit --amend will do it.
I need to remember to set the environment variable EDITOR to my favourite editor, nano, or the amend command won't work.