I chose a nice Friday evening and a good Scotch to upgrade an older Ubuntu LTS to the latest and greatest. And all went well, until I wanted to connect one of the clients via VPN. All I saw was this nasty little line in the log files of the server
... VERIFY ERROR: depth=0, error=CRL has expired: ...
Now, that’s not good. After a little bit of digging I found out, that I am not the only one running into that issue when migrating to OpenVPN 2.4 when using CRLs. The good news is, that there is a fix for it. The bad news is, that it is of course not available in Ubuntu right now. But fret not, there is a workaround. It is not a nice one, but you can regenerate the CRL by doing the following.
Modify the OpenSSL configuration, that you use to manage your certificates. If you use Easy RSA, then it is most likely in /etc/openvpn/easy-rsa/ and it is called openssl-1.0.0.cnf. Look for the default expiration for certificates and CRLs. In my case that looked like this:
default_days = 3650 # how long to certify for
default_crl_days= 30 # how long before next CRL
Increase the default to something like this:
default_days = 3650 # how long to certify for
default_crl_days= 3650 # how long before next CRL
And now regenerate the CRL. This is assuming you are using Easy RSA and you are in the folder /etc/openvpn/easy-rsa:
openssl ca -gencrl -keyfile keys/ca.key -cert keys/ca.crt -out keys/crl.pem -config ./openssl-1.0.0.cnf
After a restart of the OpenVPN server, the clients should be able to connect again.
Sending email in Linux is pretty straight forward, once an email server is set up. Just use mutt or mail and all is good. But sometimes you actually want to test if SMTP is working correctly. And not only on your box, but on a remote box. That is of course easy using a MUA like Thunderbird or Sylpheed, but that is not always feasible on a remote server in a remote network.
Luckily there is a solution to it using just the command line. To be precise, there are multiple solutions. Starting with SMTP by hand using telnet, but that is pretty hardcore. So how about mailx or swaks.
mailx is part of multiple packages, like mailutils, but I prefer the heirloom-mailx version. This version allows you specify a lot of SMTP connection details. Just check out the manpage. So on a Debian based distribution quickly install it with
apt-get install heirloom-mailx
An email can be sent with:
echo "Testbody" | mailx -v \
-r "firstname.lastname@example.org" \
-s "Test Subject" \
-S smtp="smtp.example.com:587" \
-S smtp-use-starttls \
-S smtp-auth=login \
-S smtp-auth-user="email@example.com" \
-S smtp-auth-password="changeme" \
-S ssl-verify=ignore \
This would send an email to firstname.lastname@example.org using the SMTP server smtp.example.com with STARTTLS, without verifying the SSL certificate. There are of course tons of other options. Just play around with it.
After moving a server to a new machine with Ubuntu 16.10 I received some strange Postfix SMTP errors. Which turned out to be a connection issue to the MySQL server:
postfix/cleanup: warning: connect to mysql server 127.0.0.1: Too many connections
Oops, did I forgot to up max_connections during the migration:
# grep max_connections /etc/mysql/mysql.conf.d/mysqld.cnf
max_connections = 8000
Nope, I didn’t. Did we all of a sudden have a surge in clients accessing the database. Let me check and ask MySQL, and the process list looked fine. But something was off. So let’s check the value in the SQL server itself:
mysql> show variables like 'max_connections';
| Variable_name | Value |
| max_connections | 214 |
1 row in set (0.01 sec)
Wait, what?! A look into the error log gave the same result:
# grep max_connections /var/log/mysql/error.log
2017-06-14T01:23:29.804684Z 0 [Warning] Changed limits: max_connections: 214 (requested 8000)
Something is off here and ye olde oracle Google has quite some hits on that topic. And the problem lies with the maximum allowed number of open files. You can’t have more connections, than open files. Makes sense. Some people suggest to solve it using /etc/security/limits.conf to fix it. Which is not so simple on Ubuntu anymore, because you have to first enable pam_limits.so. And even then it doesn’t work, because since Ubuntu is using systemd (15.04 if I am not mistaken) this configuration is only valid for user sessions and not services/demons.
So let’s solve it using systemd’s settings to allow for more connections/open files. First you have to copy the configuration file, so that you can make the changes we need:
cp /lib/systemd/system/mysql.service /etc/systemd/system/
Append the following lines to the new file using vi (or whatever editor you want to use):
After restarting MySQL it was finally obeying the setting:
mysql> show variables like 'max_connections';
| Variable_name | Value |
| max_connections | 8000 |
1 row in set (0.01 sec)
I think I am a little bit late, but I finally took the plunge and am now the (hopefully) proud owner of a smartwatch. Not sure how smart this little gizmo is, but at least it looks neat.
But first to the decision itself. It is not, that I don’t have a watch. Actually, I have two: a Seiko and a Bulova. Both nice watches and the Seiko is with me for quite a while. And they are perfect instruments to read, well.. time. What more can a smartwatch offer? Not much it seems. But I am somewhat of a geek, so I have to try it. As a result I started reading reviews and a lot of posts about smartwatches. I am not a big Apple fan (except for their designs) and it seemed, that the Apple Watch has its problems with the haptic and the general usability. In fact it seems to be so bad, that people are getting rid of their watches. Not that the Android world is so much better, but there is more diversity. And it seemed to me, that the Huawei Smartwatch was kinda okay. So I ordered one on ye good olde Amazon. And of course USPS messed up and delivered it to the wrong building, which meant, that I had to chase the package down. But finally I could hold it in my hands.
The watch is in a very stylish and high quality looking black cube with gold lettering.
With shaking hands I took the top part of and opened the lid…
This is a nice looking watch. Like the packaging the watch looks very high quality and has a very nice soft leather wrist band. Yes, I chose the leather version.
The setup was very easy and straight forward after I installed the Android Wear app on my Nexus 6P (also Huawei… what a coincidence). Bluetooth pairing and WiFi set up was all more or less automatically. And after an OS upgrade the watch is up and running. I could even test some of the notification settings. This was easier, than I thought it would be. I still have to explore some of the Android Wear apps, but for now I am good.
I think the biggest problem so far was choosing a clock face. For the moment I am using “Moon Phase”, but that might change. Here is the standard and the standby clock face.
So this is the very first impression of my new little gizmo. Let’s see how this watch works in real life conditions. Is it really a smart watch or is it just an overhyped toy? What apps make sense to use and which don’t? And the battery life is of course a concern. After charging it, the battery drained quickly to about 75%, but that was also after a couple of system updates and other play things. Also, I currently have the display always on, hence the standby clock face.
It was that time of the year, when I had to renew some SSL certificates. Renewing and updating them in the server is a nice and easy process. But checking, whether the server is delivering the correct certificate and, that I updated and popluated the intermediate certificates correctly, is a different story.
For websites it is quite easy. Every browser is pretty verbose about the certificate of an https connection. But mail clients are not so talkative. Luckily openssl can help here.
To get the certificate in PEM form to compare you can simply call this command. Of course you have to replace and with the correct values, like example.com:993 for IMAPS on example.com:
openssl s_client -showcerts -connect :
If you want it a little bit more verbose, then you can pipe it again through openssl to get a more human readable version:
While debugging my check disk problem after the 15.10 upgrade, I saw that I forgot to enable external commands. That is handy, when you want to re-schedule a check to see, if your changes took effect. Again, something that is easily activated. So if you see something like this, then you might want to make some changes:
Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!
The external command file may be missing, Nagios may not be running, and/or Nagios may not be checking external commands. An error occurred while attempting to commit your command for processing.
First stop the Nagios service with systemctl, service or with the init script. Whatever your distribution prefers. Then edit as root the configuration file /etc/nagios3/nagios.cfg and check if the variable check_external_commands is set to 1:
Afterwards update the rights to the external command with the following:
Another day another foo, this time done to the check_disk plugin for Nagios on Ubuntu. After updating to 15.10 my disk space check all of a sudden failed with this one here:
DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied
It seemed a little odd, especially when I could access that file normally before. So something has changed and the workaround is actually fairly easy. As root edit the file /etc/nagios-plugins/config/disk.cfg and change the command for check_all_disks. You need to add -A -i ‘/sys’ to the command line. So your command for check_all_disks will look like this:
Finally the Owncloud people got their updater to work for me, and I went ahead and updated. As a result I had to make some tweaks to CloudPress to keep my users active. Check it out and let me know if it works for you. Find my this new version here.
As a longtime OpenVPN user on Linux I thought it would be an easy task to set up OpenVPN as a service on Windows. Well, I was right… and couldn’t be wronger. Setting up the service is part of the installation notes for OpenVPN. Just search for “Running OpenVPN as a Windows Service” in the notes and you will find a pretty good description that should get you up and running in no time.
But the devil is of course in the detail. This kind of setup works perfect for servers, or in general for machines with good internet connectivity that never go into standby or hibernation. Once you use a laptop, which you usually put into standby on a regular basis, or you simply have a sluggish WiFi connection, then you have a problem. And the problem is, that OpenVPN is unable to re-establish the tunnels and, at least in my case, causes quite some CPU load on the machine. The problem is actually well known to the OpenVPN team (see here and here).
So what are your options to fix it. In OpenVPN’s people suggest to use either OpenVPN Service for Windows or NSSM. I tried my best with the OpenVPN Service for Windows, but I couldn’t get it to work. So sorry guys, I can’t recommend that one. Then I tried NSSM and hit similar hurdles but the documentation was better and I could get it actually to work.
So without further ado, I present to you the setup of OpenVPN as a service in Windows using NSSM.
For starters you have to download and install the latest version of OpenVPN (Download) and NSSM (Download). NSSM does not come with an installer. That means, you have to create a folder for instance in your “Program Files” directory (or whatever directory name %PROGRAMFILES% represents). And then, depending on your operating system, you copy the win32 or win64 version of nssm.exe into that directory. Now open a console with adminstrator rights and navigate to the newly created folder and you can try to execute NSSM to get the command line parameters:
NSSM: The non-sucking service manager
Version 2.24 64-bit, 2014-08-31
To show service installation GUI:
nssm install [<servicename>]
To install a service without confirmation:
nssm install <servicename> <app> [<args> ...]
To show service editing GUI:
nssm edit <servicename>
To retrieve or edit service parameters directly:
nssm get <servicename> <parameter> [<subparameter>]
nssm set <servicename> <parameter> [<subparameter>]
You can control NSSM completely from the command line, but it also has an actually usable GUI. You can start the installation process by doing the following
or if you want to give already a service name (can’t be changed with NSSM once it is installed!). Note that if you have multiple tunnels, then you have to setup multiple services. So give it a meaningful name.
nssm install "My OpenVPN Service"
The second command should give you the following window:
The next step is filling in all the information necessary for NSSM to set up OpenVPN as a service.
Path: This is the path to the OpenVPN binary and should usually be C:\Program Files\OpenVPN\bin\openvpn.exe. Startup directory: This is the path to the directory where you store your OpenVPN configuration files. Usually that is C:\Program Files\OpenVPN\config. But if you want to run the OpenVPN UI with manual started tunnels in parallel, then you should create a separate folder, e.g. C:\Program Files\OpenVPN\config-nssm. Otherwise it is easy to confuse manual tunnels with service tunnels. In my sample I won’t use manual tunnels, so I go with the default. Arguments: This is the configuration file for the tunnel, that should reside in the above defined startup directory.
Display name: This is the name that is basically visible everywhere. Most of the time this is the same as the service name, but this is up to you. Description: As the name says, this is a description, that can be viewed later on in the services area. Startup type: This is the standard service startup type setting for a windows. Most likely you want to choose Automatic here. But you have the choice between Automatic, Automatic (Delayed Start), Manual and Disabled here.
Here you can define as who this service needs to run, but unless you are doing something very special here you can leave it to the default setting (Local System Account).
Running Windows 10, this tab can be more important. Using the system account you will be able to install the service, but when starting the service, you might see an error. In the event log it will show up with the following message:
Program C:\Program Files\OpenVPN\bin\openvpn.exe for service OpenVPN siteopsvpn (NSSM) exited with return code 3221225794.
This basically means, that you use an account that has no rights to execute OpenVPN. I solved it, by running the service as a user, that has administrative rights on that machine. You can even create a special user to do that.
The dependencies tab is important, because here we have to add the services that OpenVPN is depending on (Dhcp and tap0901).
In this tab you can control how the service is handled by the processor. For instance if it should only run on a specific processor or a higher priority. For the normal use case this can be left alone.
Unless you run into strange problems you can leave this one alone.
This is again a tab that you don’t have to touch under normal circumstances.
Now we have to do something again. With the OpenVPN UI you have the ability to take a look at log files. Well, with services you don’t, unless you define them here. You can use the same for all redirections, but I prefer to have a separate log for stdin, stdout and stderr. Log files are usually located in C:\Program Files\OpenVPN\log.
This tab is an extension of the I/O tab, as it configures the log rotation. I set it to rotate and left the rest alone. But you can decide on different rotation strategies, so that the files don’t get too big, too old, or whatever the problem might be.
This last tab can be ignored for the usual use case. But you might have a special case where you have to add or even replace the environment, then this tab is your friend.
Now a last chance to think about the service name… You are good? Okay, then click “Install Service” and NSSM will install OpenVPN as a service that can survive standby and sluggish network connections. You have to start the newly create service with either the net command, the nssm command or via the services in the control panel.
If you have to change anything you can do that by calling NSSM with the edit parameter.
nssm edit "My OpenVPN Service"
And, as mention before, you can do all this on the command line. Here is the sequence.
nssm install "My OpenVPN Service" C:\Program Files\OpenVPN\bin\openvpn.exe
nssm set "My OpenVPN Service" AppDirectory "C:\Program Files\OpenVPN\config"
nssm set "My OpenVPN Service" AppParameters myvpnconfig.ovpn
nssm set "My OpenVPN Service" AppStdin "C:\Program Files\OpenVPN\log\myservice-stdin.log"
nssm set "My OpenVPN Service" AppStdout "C:\Program Files\OpenVPN\log\myservice-stdout.log"
nssm set "My OpenVPN Service" AppStderr "C:\Program Files\OpenVPN\log\myservice-sterr.log"
nssm set "My OpenVPN Service" AppRotateFiles 1
nssm set "My OpenVPN Service" DependOnService Dhcp tap0901