OpenVPN 2.4 CRL Expired Foo

I chose a nice Friday evening and a good Scotch to upgrade an older Ubuntu LTS to the latest and greatest. And all went well, until I wanted to connect one of the clients via VPN. All I saw was this nasty little line in the log files of the server

... VERIFY ERROR: depth=0, error=CRL has expired: ...

Now, that’s not good. After a little bit of digging I found out, that I am not the only one running into that issue when migrating to OpenVPN 2.4 when using CRLs. The good news is, that there is a fix for it. The bad news is, that it is of course not available in Ubuntu right now. But fret not, there is a workaround. It is not a nice one, but you can regenerate the CRL by doing the following.

Modify the OpenSSL configuration, that you use to manage your certificates. If you use Easy RSA, then it is most likely in /etc/openvpn/easy-rsa/ and it is called openssl-1.0.0.cnf. Look for the default expiration for certificates and CRLs. In my case that looked like this:

default_days = 3650 # how long to certify for
default_crl_days= 30 # how long before next CRL

Increase the default to something like this:

default_days = 3650 # how long to certify for
default_crl_days= 3650 # how long before next CRL

And now regenerate the CRL. This is assuming you are using Easy RSA and you are in the folder /etc/openvpn/easy-rsa:

openssl ca -gencrl -keyfile keys/ca.key -cert keys/ca.crt -out keys/crl.pem -config ./openssl-1.0.0.cnf

After a restart of the OpenVPN server, the clients should be able to connect again.

Happy VPN’ing

MySQL max_connections limited to 214 on Ubuntu Foo

After moving a server to a new machine with Ubuntu 16.10 I received some strange Postfix SMTP errors. Which turned out to be a connection issue to the MySQL server:

postfix/cleanup[30475]: warning: connect to mysql server 127.0.0.1: Too many connections

Oops, did I forgot to up max_connections during the migration:

# grep max_connections /etc/mysql/mysql.conf.d/mysqld.cnf
max_connections = 8000

Nope, I didn’t. Did we all of a sudden have a surge in clients accessing the database. Let me check and ask MySQL, and the process list looked fine. But something was off. So let’s check the value in the SQL server itself:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 214 |
+-----------------+-------+
1 row in set (0.01 sec)

Wait, what?! A look into the error log gave the same result:

# grep max_connections /var/log/mysql/error.log
2017-06-14T01:23:29.804684Z 0 [Warning] Changed limits: max_connections: 214 (requested 8000)

Something is off here and ye olde oracle Google has quite some hits on that topic. And the problem lies with the maximum allowed number of open files. You can’t have more connections, than open files. Makes sense. Some people suggest to solve it using /etc/security/limits.conf to fix it. Which is not so simple on Ubuntu anymore, because you have to first enable pam_limits.so. And even then it doesn’t work, because since Ubuntu is using systemd (15.04 if I am not mistaken) this configuration is only valid for user sessions and not services/demons.

So let’s solve it using systemd’s settings to allow for more connections/open files. First you have to copy the configuration file, so that you can make the changes we need:

cp /lib/systemd/system/mysql.service /etc/systemd/system/

Append the following lines to the new file using vi (or whatever editor you want to use):

vi /etc/systemd/system/mysql.service

LimitNOFILE=infinity
LimitMEMLOCK=infinity

Reload systemd:

systemctl daemon-reload

After restarting MySQL it was finally obeying the setting:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 8000 |
+-----------------+-------+
1 row in set (0.01 sec)

The universe is balanced again.

Check SSL Connection Foo

It was that time of the year, when I had to renew some SSL certificates. Renewing and updating them in the server is a nice and easy process. But checking, whether the server is delivering the correct certificate and, that I updated and popluated the intermediate certificates correctly, is a different story.

For websites it is quite easy. Every browser is pretty verbose about the certificate of an https connection. But mail clients are not so talkative. Luckily openssl can help here.

To get the certificate in PEM form to compare you can simply call this command. Of course you have to replace and with the correct values, like example.com:993 for IMAPS on example.com:


openssl s_client -showcerts -connect :

If you want it a little bit more verbose, then you can pipe it again through openssl to get a more human readable version:


openssl s_client -showcerts -connect : | openssl x509 -text

Sometimes the connection itself is not supporting SSL or TLS directly, so you have to give it a hint. For instance for SMTP connection with STARTTLS you can use:


openssl s_client -showcerts -connect :25 -starttls smtp | openssl x509 -text

In my version of s_client only smtp, pop3, imap and ftp were supported protocols. If you are looking for more information about this you will find it in the man pages of openssl and s_client.

Nagios check_disk Foo on Ubuntu 15.10

Another day another foo, this time done to the check_disk plugin for Nagios on Ubuntu. After updating to 15.10 my disk space check all of a sudden failed with this one here:


DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied

It seemed a little odd, especially when I could access that file normally before. So something has changed and the workaround is actually fairly easy. As root edit the file /etc/nagios-plugins/config/disk.cfg and change the command for check_all_disks. You need to add -A -i ‘/sys’ to the command line. So your command for check_all_disks will look like this:


# 'check_all_disks' command definition
define command{
command_name check_all_disks
command_line /usr/lib/nagios/plugins/check_disk -w '$ARG1$' -c '$ARG2$' -e -A -i '/sys'
}

Restart Nagios and all is good. After I fixed it this way I found, that it is actually filed as a bug 1516451 in Ubuntu’s Launchpad here

Happy monitoring.

OpenVPN Windows Service Foo (Updated)

As a longtime OpenVPN user on Linux I thought it would be an easy task to set up OpenVPN as a service on Windows. Well, I was right… and couldn’t be wronger. Setting up the service is part of the installation notes for OpenVPN. Just search for “Running OpenVPN as a Windows Service” in the notes and you will find a pretty good description that should get you up and running in no time.

But the devil is of course in the detail. This kind of setup works perfect for servers, or in general for machines with good internet connectivity that never go into standby or hibernation. Once you use a laptop, which you usually put into standby on a regular basis, or you simply have a sluggish WiFi connection, then you have a problem. And the problem is, that OpenVPN is unable to re-establish the tunnels and, at least in my case, causes quite some CPU load on the machine. The problem is actually well known to the OpenVPN team (see here and here).

So what are your options to fix it. In OpenVPN’s people suggest to use either OpenVPN Service for Windows or NSSM. I tried my best with the OpenVPN Service for Windows, but I couldn’t get it to work. So sorry guys, I can’t recommend that one. Then I tried NSSM and hit similar hurdles but the documentation was better and I could get it actually to work.

So without further ado, I present to you the setup of OpenVPN as a service in Windows using NSSM.

For starters you have to download and install the latest version of OpenVPN (Download) and NSSM (Download). NSSM does not come with an installer. That means, you have to create a folder for instance in your “Program Files” directory (or whatever directory name %PROGRAMFILES% represents). And then, depending on your operating system, you copy the win32 or win64 version of nssm.exe into that directory. Now open a console with adminstrator rights and navigate to the newly created folder and you can try to execute NSSM to get the command line parameters:

C:\Program Files\NSSM>nssm.exe
NSSM: The non-sucking service manager
Version 2.24 64-bit, 2014-08-31
Usage: nssm

To show service installation GUI:

nssm install [<servicename>]

To install a service without confirmation:

nssm install <servicename> <app> [<args> ...]

To show service editing GUI:

nssm edit <servicename>

To retrieve or edit service parameters directly:

nssm get <servicename> <parameter> [<subparameter>]

nssm set <servicename> <parameter> [<subparameter>]

nssm reset <servicename> <parameter> [<subparameter>]

To show service removal GUI:

nssm remove [<servicename>]

To remove a service without confirmation:

nssm remove <servicename> confirm

To manage a service:

nssm start <servicename>

nssm stop <servicename>

nssm restart <servicename>

nssm status <servicename>

nssm rotate <servicename>

C:\Program Files\NSSM>

You can control NSSM completely from the command line, but it also has an actually usable GUI. You can start the installation process by doing the following

nssm install

or if you want to give already a service name (can’t be changed with NSSM once it is installed!). Note that if you have multiple tunnels, then you have to setup multiple services. So give it a meaningful name.

nssm install "My OpenVPN Service"

The second command should give you the following window:
NSSM Installer

The next step is filling in all the information necessary for NSSM to set up OpenVPN as a service.

Application

Path: This is the path to the OpenVPN binary and should usually be C:\Program Files\OpenVPN\bin\openvpn.exe.
Startup directory: This is the path to the directory where you store your OpenVPN configuration files. Usually that is C:\Program Files\OpenVPN\config. But if you want to run the OpenVPN UI with manual started tunnels in parallel, then you should create a separate folder, e.g. C:\Program Files\OpenVPN\config-nssm. Otherwise it is easy to confuse manual tunnels with service tunnels. In my sample I won’t use manual tunnels, so I go with the default.
Arguments: This is the configuration file for the tunnel, that should reside in the above defined startup directory.
NSSM Application Tab

Details

Display name: This is the name that is basically visible everywhere. Most of the time this is the same as the service name, but this is up to you.
Description: As the name says, this is a description, that can be viewed later on in the services area.
Startup type: This is the standard service startup type setting for a windows. Most likely you want to choose Automatic here. But you have the choice between Automatic, Automatic (Delayed Start), Manual and Disabled here.
NSSM Details

Log on

Here you can define as who this service needs to run, but unless you are doing something very special here you can leave it to the default setting (Local System Account).
NSSM Log On

Update

Running Windows 10, this tab can be more important. Using the system account you will be able to install the service, but when starting the service, you might see an error. In the event log it will show up with the following message:

Program C:\Program Files\OpenVPN\bin\openvpn.exe for service OpenVPN siteopsvpn (NSSM) exited with return code 3221225794.

This basically means, that you use an account that has no rights to execute OpenVPN. I solved it, by running the service as a user, that has administrative rights on that machine. You can even create a special user to do that.

Dependencies

The dependencies tab is important, because here we have to add the services that OpenVPN is depending on (Dhcp and tap0901).
NSSM Dependencies

Process

In this tab you can control how the service is handled by the processor. For instance if it should only run on a specific processor or a higher priority. For the normal use case this can be left alone.
NSSM Process

Shutdown

Unless you run into strange problems you can leave this one alone.
NSSM Shutdown

Exit actions

This is again a tab that you don’t have to touch under normal circumstances.
NSSM Exit Actions

I/O

Now we have to do something again. With the OpenVPN UI you have the ability to take a look at log files. Well, with services you don’t, unless you define them here. You can use the same for all redirections, but I prefer to have a separate log for stdin, stdout and stderr. Log files are usually located in C:\Program Files\OpenVPN\log.
NSSM I/O

File rotation

This tab is an extension of the I/O tab, as it configures the log rotation. I set it to rotate and left the rest alone. But you can decide on different rotation strategies, so that the files don’t get too big, too old, or whatever the problem might be.
NSSM File Rotation

Environment

This last tab can be ignored for the usual use case. But you might have a special case where you have to add or even replace the environment, then this tab is your friend.
NSSM Environment

Now a last chance to think about the service name… You are good? Okay, then click “Install Service” and NSSM will install OpenVPN as a service that can survive standby and sluggish network connections. You have to start the newly create service with either the net command, the nssm command or via the services in the control panel.

If you have to change anything you can do that by calling NSSM with the edit parameter.

nssm edit "My OpenVPN Service"

And, as mention before, you can do all this on the command line. Here is the sequence.

nssm install "My OpenVPN Service" C:\Program Files\OpenVPN\bin\openvpn.exe
nssm set "My OpenVPN Service" AppDirectory "C:\Program Files\OpenVPN\config"
nssm set "My OpenVPN Service" AppParameters myvpnconfig.ovpn
nssm set "My OpenVPN Service" AppStdin "C:\Program Files\OpenVPN\log\myservice-stdin.log"
nssm set "My OpenVPN Service" AppStdout "C:\Program Files\OpenVPN\log\myservice-stdout.log"
nssm set "My OpenVPN Service" AppStderr "C:\Program Files\OpenVPN\log\myservice-sterr.log"
nssm set "My OpenVPN Service" AppRotateFiles 1
nssm set "My OpenVPN Service" DependOnService Dhcp tap0901

Happy tunneling.

Lync CTRL-Enter Foo (Update)

It seems that Microsoft’s products creating more and more foos. This time it is Lync, also known as Skype for Business.

It is okay, that Lync uses the Enter key to send a message or the CTRL-Enter combination start a call. But it is not okay that there is no way to change these key combinations. Especially when all other instant messengers that I use, default to CTRL-Enter to send a message.

So now I have to use Lync at work, which starts a call when I want to send a message and no direct way of changing it. The only thing that is possible, is removing the CTRL-Enter key combination so that I don’t accidentally start a call.

A solution is described in a TechNet blog post. So here is what worked for me in Lync 2013 (a.k.a. Office 15.0).

First open the registry editor (regedit.exe) and navigate to the following key. If you have a different Lync/Office version you might just change the version number. Create the missing pieces of the key, if necessary. In my case I only found the Microsoft part and had to create everything else.

HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\15.0\Lync\DisabledShortcutKeysCheckBoxes

Create a new string value inside that key:

Value Name: CtrlEnter
Value Data: 13,8

Restart Lync and the CTRL-Enter foo should be gone.

Update: Fixed the registry key (Lync, not Office). Thank you Karel, for pointing that out to me.

Windows 8.1 Store App Can’t Open Foo

Windows continues to kick my behind. Out of the blue the Windows 8.1 installation on my workstation didn’t want to open the store app. Not that I really need or want it. But the same happened for the settings app, which was a little bit of a problem.

I should have taken a screenshot but the error message was something along the lines of “This app can’t open” and “Refreshing your PC might help fix it”.

It seems that I was not the only one with that problem there is even a Microsoft blog post about it. The reason seems to be that the affected apps are not registered anymore with Windows.

So here is how it can be fixed. In a command prompt that runs as administrator I ran the following commands. When you look at the blog post it seems that you don’t need administrative rights but you never know. Now enter the following commands to re-register the system apps:

Store app:

powershell -ExecutionPolicy Unrestricted Add-AppxPackage -DisableDevelopmentMode -Register $Env:SystemRoot\WinStore\AppxManifest.XML

Camera app:

powershell -ExecutionPolicy Unrestricted Add-AppxPackage -DisableDevelopmentMode -Register $Env:SystemRoot\camera\AppxManifest.xml

File Manager (One Drive) app:

powershell -ExecutionPolicy Unrestricted Add-AppxPackage -DisableDevelopmentMode -Register $Env:SystemRoot\FileManager\AppxManifest.xml

Settings app:

powershell -ExecutionPolicy Unrestricted Add-AppxPackage -DisableDevelopmentMode -Register $Env:SystemRoot\ImmersiveControlPanel\AppxManifest.xml

These execution of these commands, especially for the store app, might take a while.

This would have been easy, but of course a Microsoft products never make things easy on me. So I hit the problem that the a registry key doesn’t have the right owner and I got the error 0x8007064A.

So you have to open the registry editor (Windows + R and then regedit) and navigate to the registry key:

HKEY_CURRENT_USER\Software\Classes\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppModel\Repository\Packages

And then change the ownership of this key to SYSTEM:

  1. On the Edit menu, click Permissions.
  2. Click Advanced, and then click the Owner tab.
  3. Under Change owner to, click the new owner, and then click OK.

avelsieve Updates…

I was using SquirrelMail and the Sieve plugin avelsieve for quite a while. And I once made the commitment on the mailing list to do some further development on avelsieve to make it work with newer PHP versions and fix other problems. The result was my own little update and some not yet released stuff that I only used myself.

But things changed and I moved on to Roundcube. And more importantly the spare time that I could spend on this project got less and less. I am really sorry, but I won’t be doing any further development on avelsieve. For anybody who wants to use my patched version, it is still and will be up there for download.

Outlook Monthly Calendar View Foo

MS Outlook is in itself already a very annoying and not very pleasant piece of software. But sometimes you have to use it, no matter what. I will live.

But running again and again into the problem that Outlook conveniently forgets the correct calendar view really became a problem. Especially when it happens to everybody and they come to me to fix it.

At least it is easy to fix, but let me describe the problem first. It happens quite frequently that Outlook simply forgets the saved calender view. In our case that was usually the monthly view. It happens when you run a filter on your calendar. Outlook assumes the filtered view to be the new default. Well, that is just wrong.

Depending on the version of Outlook you have to perform one of the following steps.

Outlook 2007
Menu bar > View > Arrange By > Current View > Day/Week/Month

Outlook 2010
Menu bar > View > Change View > Manage Views > Select view name "Calendar" in the list Click "Apply View"

Mac OS Re-Install With New Apple ID Foo

Re-installing MacOS should be a piece of cake. Just go into recovery mode by pressing Command+R during boot and start recovering. It should be…

But that is only true for the common case. If you perform the internet recovery and you are using a brand new Apple ID, then you are screwed. In that case you will see a message like this:

This Apple ID has not purchased Lion. You must sign in with an Apple ID that was used to purchase OS X Mavericks.

This happened to me, when I purchased a Mac online with a new Apple ID and tried to activate it. Well, it simply doesn’t work out of the box.

But fear not, there is quite simple fix, although it requires an activated Mac.

On that activated Mac log into the Mac App Store with your fresh Apple ID. Then find the MacOS version that you want to install, in this case Mavericks. Click download and it will give you some hard time, because it is already installed. But simply ignore that. Click continue to download the installer. You will see the download process in purchase area. You can go there and pause the download, because you don’t really want to install it. Now you should log out of the Mac App Store, if this is not your machine. And now, as a last step restart the Mac. Of course the one that you want to re-install. You can now use your fresh Apple ID that just “bought” MacOS.