Send Email from the Command Line using an External SMTP Server

Sending email in Linux is pretty straight forward, once an email server is set up. Just use mutt or mail and all is good. But sometimes you actually want to test if SMTP is working correctly. And not only on your box, but on a remote box. That is of course easy using a MUA like Thunderbird or Sylpheed, but that is not always feasible on a remote server in a remote network.

Luckily there is a solution to it using just the command line. To be precise, there are multiple solutions. Starting with SMTP by hand using telnet, but that is pretty hardcore. So how about mailx or swaks.

mailx

mailx is part of multiple packages, like mailutils, but I prefer the heirloom-mailx version. This version allows you specify a lot of SMTP connection details. Just check out the manpage. So on a Debian based distribution quickly install it with

apt-get install heirloom-mailx

An email can be sent with:

echo "Testbody" | mailx -v \
-r "me@example.com" \
-s "Test Subject" \
-S smtp="smtp.example.com:587" \
-S smtp-use-starttls \
-S smtp-auth=login \
-S smtp-auth-user="me@example.com" \
-S smtp-auth-password="changeme" \
-S ssl-verify=ignore \
memyselfandi@example.com

This would send an email to memyselfandi@example.com using the SMTP server smtp.example.com with STARTTLS, without verifying the SSL certificate. There are of course tons of other options. Just play around with it.

Swaks – Swiss Army Knife for SMTP

Swaks, the swiss army knife for SMTP, is a great little tool on the command line, that also offers an option to test SMTP servers. And it supports of course encryption using TLS.

Just install it with the packet manager of your distribution. Here is the Debian based version:

apt-get install swaks

And you can send an email with:

echo "This is the message body" | swaks \
--to "memyselfandi@example.com" \
--from "me@example.com" \
--server smtp.example.com \
--auth LOGIN \
--auth-user "me@example.com" \
--auth-password "changem" \
-tls

Yes, TLS is activate with a single dash parameter. Swaks, being a Perl script, can also just be downloaded from the Swaks homepage and works nicely in Cygwin.

MySQL max_connections limited to 214 on Ubuntu Foo

After moving a server to a new machine with Ubuntu 16.10 I received some strange Postfix SMTP errors. Which turned out to be a connection issue to the MySQL server:

postfix/cleanup[30475]: warning: connect to mysql server 127.0.0.1: Too many connections

Oops, did I forgot to up max_connections during the migration:

# grep max_connections /etc/mysql/mysql.conf.d/mysqld.cnf
max_connections = 8000

Nope, I didn’t. Did we all of a sudden have a surge in clients accessing the database. Let me check and ask MySQL, and the process list looked fine. But something was off. So let’s check the value in the SQL server itself:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 214 |
+-----------------+-------+
1 row in set (0.01 sec)

Wait, what?! A look into the error log gave the same result:

# grep max_connections /var/log/mysql/error.log
2017-06-14T01:23:29.804684Z 0 [Warning] Changed limits: max_connections: 214 (requested 8000)

Something is off here and ye olde oracle Google has quite some hits on that topic. And the problem lies with the maximum allowed number of open files. You can’t have more connections, than open files. Makes sense. Some people suggest to solve it using /etc/security/limits.conf to fix it. Which is not so simple on Ubuntu anymore, because you have to first enable pam_limits.so. And even then it doesn’t work, because since Ubuntu is using systemd (15.04 if I am not mistaken) this configuration is only valid for user sessions and not services/demons.

So let’s solve it using systemd’s settings to allow for more connections/open files. First you have to copy the configuration file, so that you can make the changes we need:

cp /lib/systemd/system/mysql.service /etc/systemd/system/

Append the following lines to the new file using vi (or whatever editor you want to use):

vi /etc/systemd/system/mysql.service

LimitNOFILE=infinity
LimitMEMLOCK=infinity

Reload systemd:

systemctl daemon-reload

After restarting MySQL it was finally obeying the setting:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 8000 |
+-----------------+-------+
1 row in set (0.01 sec)

The universe is balanced again.

Huawei Smartwatch Experiment

I think I am a little bit late, but I finally took the plunge and am now the (hopefully) proud owner of a smartwatch. Not sure how smart this little gizmo is, but at least it looks neat.

But first to the decision itself. It is not, that I don’t have a watch. Actually, I have two: a Seiko and a Bulova. Both nice watches and the Seiko is with me for quite a while. And they are perfect instruments to read, well.. time. What more can a smartwatch offer? Not much it seems. But I am somewhat of a geek, so I have to try it. As a result I started reading reviews and a lot of posts about smartwatches. I am not a big Apple fan (except for their designs) and it seemed, that the Apple Watch has its problems with the haptic and the general usability. In fact it seems to be so bad, that people are getting rid of their watches. Not that the Android world is so much better, but there is more diversity. And it seemed to me, that the Huawei Smartwatch was kinda okay. So I ordered one on ye good olde Amazon. And of course USPS messed up and delivered it to the wrong building, which meant, that I had to chase the package down. But finally I could hold it in my hands.

Unboxing

The watch is in a very stylish and high quality looking black cube with gold lettering.

2016-09-07-22-51-34

With shaking hands I took the top part of and opened the lid…

2016-09-07-22-51-58

This is a nice looking watch. Like the packaging the watch looks very high quality and has a very nice soft leather wrist band. Yes, I chose the leather version.

2016-09-07-22-52-39

2016-09-07-22-52-53

The setup was very easy and straight forward after I installed the Android Wear app on my Nexus 6P (also Huawei… what a coincidence). Bluetooth pairing and WiFi set up was all more or less automatically. And after an OS upgrade the watch is up and running. I could even test some of the notification settings. This was easier, than I thought it would be. I still have to explore some of the Android Wear apps, but for now I am good.

I think the biggest problem so far was choosing a clock face. For the moment I am using “Moon Phase”, but that might change. Here is the standard and the standby clock face.

2016-09-07-22-53-07

2016-09-07-22-54-39

So this is the very first impression of my new little gizmo. Let’s see how this watch works in real life conditions. Is it really a smart watch or is it just an overhyped toy? What apps make sense to use and which don’t? And the battery life is of course a concern. After charging it, the battery drained quickly to about 75%, but that was also after a couple of system updates and other play things. Also, I currently have the display always on, hence the standby clock face.

To be continued…

Check SSL Connection Foo

It was that time of the year, when I had to renew some SSL certificates. Renewing and updating them in the server is a nice and easy process. But checking, whether the server is delivering the correct certificate and, that I updated and popluated the intermediate certificates correctly, is a different story.

For websites it is quite easy. Every browser is pretty verbose about the certificate of an https connection. But mail clients are not so talkative. Luckily openssl can help here.

To get the certificate in PEM form to compare you can simply call this command. Of course you have to replace and with the correct values, like example.com:993 for IMAPS on example.com:


openssl s_client -showcerts -connect :

If you want it a little bit more verbose, then you can pipe it again through openssl to get a more human readable version:


openssl s_client -showcerts -connect : | openssl x509 -text

Sometimes the connection itself is not supporting SSL or TLS directly, so you have to give it a hint. For instance for SMTP connection with STARTTLS you can use:


openssl s_client -showcerts -connect :25 -starttls smtp | openssl x509 -text

In my version of s_client only smtp, pop3, imap and ftp were supported protocols. If you are looking for more information about this you will find it in the man pages of openssl and s_client.

Nagios Enabling External Command on Debian based Distributions

While debugging my check disk problem after the 15.10 upgrade, I saw that I forgot to enable external commands. That is handy, when you want to re-schedule a check to see, if your changes took effect. Again, something that is easily activated. So if you see something like this, then you might want to make some changes:


Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!

The external command file may be missing, Nagios may not be running, and/or Nagios may not be checking external commands. An error occurred while attempting to commit your command for processing.

First stop the Nagios service with systemctl, service or with the init script. Whatever your distribution prefers. Then edit as root the configuration file /etc/nagios3/nagios.cfg and check if the variable check_external_commands is set to 1:


check_external_commands=1

Afterwards update the rights to the external command with the following:


dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3

And then start Nagios again. Et voila, you can call external commands.

Nagios check_disk Foo on Ubuntu 15.10

Another day another foo, this time done to the check_disk plugin for Nagios on Ubuntu. After updating to 15.10 my disk space check all of a sudden failed with this one here:


DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied

It seemed a little odd, especially when I could access that file normally before. So something has changed and the workaround is actually fairly easy. As root edit the file /etc/nagios-plugins/config/disk.cfg and change the command for check_all_disks. You need to add -A -i ‘/sys’ to the command line. So your command for check_all_disks will look like this:


# 'check_all_disks' command definition
define command{
command_name check_all_disks
command_line /usr/lib/nagios/plugins/check_disk -w '$ARG1$' -c '$ARG2$' -e -A -i '/sys'
}

Restart Nagios and all is good. After I fixed it this way I found, that it is actually filed as a bug 1516451 in Ubuntu’s Launchpad here

Happy monitoring.

OpenVPN Windows Service Foo (Updated)

As a longtime OpenVPN user on Linux I thought it would be an easy task to set up OpenVPN as a service on Windows. Well, I was right… and couldn’t be wronger. Setting up the service is part of the installation notes for OpenVPN. Just search for “Running OpenVPN as a Windows Service” in the notes and you will find a pretty good description that should get you up and running in no time.

But the devil is of course in the detail. This kind of setup works perfect for servers, or in general for machines with good internet connectivity that never go into standby or hibernation. Once you use a laptop, which you usually put into standby on a regular basis, or you simply have a sluggish WiFi connection, then you have a problem. And the problem is, that OpenVPN is unable to re-establish the tunnels and, at least in my case, causes quite some CPU load on the machine. The problem is actually well known to the OpenVPN team (see here and here).

So what are your options to fix it. In OpenVPN’s people suggest to use either OpenVPN Service for Windows or NSSM. I tried my best with the OpenVPN Service for Windows, but I couldn’t get it to work. So sorry guys, I can’t recommend that one. Then I tried NSSM and hit similar hurdles but the documentation was better and I could get it actually to work.

So without further ado, I present to you the setup of OpenVPN as a service in Windows using NSSM.

For starters you have to download and install the latest version of OpenVPN (Download) and NSSM (Download). NSSM does not come with an installer. That means, you have to create a folder for instance in your “Program Files” directory (or whatever directory name %PROGRAMFILES% represents). And then, depending on your operating system, you copy the win32 or win64 version of nssm.exe into that directory. Now open a console with adminstrator rights and navigate to the newly created folder and you can try to execute NSSM to get the command line parameters:

C:\Program Files\NSSM>nssm.exe
NSSM: The non-sucking service manager
Version 2.24 64-bit, 2014-08-31
Usage: nssm

To show service installation GUI:

nssm install [<servicename>]

To install a service without confirmation:

nssm install <servicename> <app> [<args> ...]

To show service editing GUI:

nssm edit <servicename>

To retrieve or edit service parameters directly:

nssm get <servicename> <parameter> [<subparameter>]

nssm set <servicename> <parameter> [<subparameter>]

nssm reset <servicename> <parameter> [<subparameter>]

To show service removal GUI:

nssm remove [<servicename>]

To remove a service without confirmation:

nssm remove <servicename> confirm

To manage a service:

nssm start <servicename>

nssm stop <servicename>

nssm restart <servicename>

nssm status <servicename>

nssm rotate <servicename>

C:\Program Files\NSSM>

You can control NSSM completely from the command line, but it also has an actually usable GUI. You can start the installation process by doing the following

nssm install

or if you want to give already a service name (can’t be changed with NSSM once it is installed!). Note that if you have multiple tunnels, then you have to setup multiple services. So give it a meaningful name.

nssm install "My OpenVPN Service"

The second command should give you the following window:
NSSM Installer

The next step is filling in all the information necessary for NSSM to set up OpenVPN as a service.

Application

Path: This is the path to the OpenVPN binary and should usually be C:\Program Files\OpenVPN\bin\openvpn.exe.
Startup directory: This is the path to the directory where you store your OpenVPN configuration files. Usually that is C:\Program Files\OpenVPN\config. But if you want to run the OpenVPN UI with manual started tunnels in parallel, then you should create a separate folder, e.g. C:\Program Files\OpenVPN\config-nssm. Otherwise it is easy to confuse manual tunnels with service tunnels. In my sample I won’t use manual tunnels, so I go with the default.
Arguments: This is the configuration file for the tunnel, that should reside in the above defined startup directory.
NSSM Application Tab

Details

Display name: This is the name that is basically visible everywhere. Most of the time this is the same as the service name, but this is up to you.
Description: As the name says, this is a description, that can be viewed later on in the services area.
Startup type: This is the standard service startup type setting for a windows. Most likely you want to choose Automatic here. But you have the choice between Automatic, Automatic (Delayed Start), Manual and Disabled here.
NSSM Details

Log on

Here you can define as who this service needs to run, but unless you are doing something very special here you can leave it to the default setting (Local System Account).
NSSM Log On

Update

Running Windows 10, this tab can be more important. Using the system account you will be able to install the service, but when starting the service, you might see an error. In the event log it will show up with the following message:

Program C:\Program Files\OpenVPN\bin\openvpn.exe for service OpenVPN siteopsvpn (NSSM) exited with return code 3221225794.

This basically means, that you use an account that has no rights to execute OpenVPN. I solved it, by running the service as a user, that has administrative rights on that machine. You can even create a special user to do that.

Dependencies

The dependencies tab is important, because here we have to add the services that OpenVPN is depending on (Dhcp and tap0901).
NSSM Dependencies

Process

In this tab you can control how the service is handled by the processor. For instance if it should only run on a specific processor or a higher priority. For the normal use case this can be left alone.
NSSM Process

Shutdown

Unless you run into strange problems you can leave this one alone.
NSSM Shutdown

Exit actions

This is again a tab that you don’t have to touch under normal circumstances.
NSSM Exit Actions

I/O

Now we have to do something again. With the OpenVPN UI you have the ability to take a look at log files. Well, with services you don’t, unless you define them here. You can use the same for all redirections, but I prefer to have a separate log for stdin, stdout and stderr. Log files are usually located in C:\Program Files\OpenVPN\log.
NSSM I/O

File rotation

This tab is an extension of the I/O tab, as it configures the log rotation. I set it to rotate and left the rest alone. But you can decide on different rotation strategies, so that the files don’t get too big, too old, or whatever the problem might be.
NSSM File Rotation

Environment

This last tab can be ignored for the usual use case. But you might have a special case where you have to add or even replace the environment, then this tab is your friend.
NSSM Environment

Now a last chance to think about the service name… You are good? Okay, then click “Install Service” and NSSM will install OpenVPN as a service that can survive standby and sluggish network connections. You have to start the newly create service with either the net command, the nssm command or via the services in the control panel.

If you have to change anything you can do that by calling NSSM with the edit parameter.

nssm edit "My OpenVPN Service"

And, as mention before, you can do all this on the command line. Here is the sequence.

nssm install "My OpenVPN Service" C:\Program Files\OpenVPN\bin\openvpn.exe
nssm set "My OpenVPN Service" AppDirectory "C:\Program Files\OpenVPN\config"
nssm set "My OpenVPN Service" AppParameters myvpnconfig.ovpn
nssm set "My OpenVPN Service" AppStdin "C:\Program Files\OpenVPN\log\myservice-stdin.log"
nssm set "My OpenVPN Service" AppStdout "C:\Program Files\OpenVPN\log\myservice-stdout.log"
nssm set "My OpenVPN Service" AppStderr "C:\Program Files\OpenVPN\log\myservice-sterr.log"
nssm set "My OpenVPN Service" AppRotateFiles 1
nssm set "My OpenVPN Service" DependOnService Dhcp tap0901

Happy tunneling.

Lync CTRL-Enter Foo (Update)

It seems that Microsoft’s products creating more and more foos. This time it is Lync, also known as Skype for Business.

It is okay, that Lync uses the Enter key to send a message or the CTRL-Enter combination start a call. But it is not okay that there is no way to change these key combinations. Especially when all other instant messengers that I use, default to CTRL-Enter to send a message.

So now I have to use Lync at work, which starts a call when I want to send a message and no direct way of changing it. The only thing that is possible, is removing the CTRL-Enter key combination so that I don’t accidentally start a call.

A solution is described in a TechNet blog post. So here is what worked for me in Lync 2013 (a.k.a. Office 15.0).

First open the registry editor (regedit.exe) and navigate to the following key. If you have a different Lync/Office version you might just change the version number. Create the missing pieces of the key, if necessary. In my case I only found the Microsoft part and had to create everything else.

HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\15.0\Lync\DisabledShortcutKeysCheckBoxes

Create a new string value inside that key:

Value Name: CtrlEnter
Value Data: 13,8

Restart Lync and the CTRL-Enter foo should be gone.

Update: Fixed the registry key (Lync, not Office). Thank you Karel, for pointing that out to me.