Cheat sheets FOR software development

Posted: November 9th, 2021 | Author: | Filed under: General | No Comments »
  1. Git Cheat Sheet A list of handy git commands to make your life easier
  2. Git Explorer
  3. JavaScript (JS) Cheat Sheet Online HTML and CSS also
  4. JavaScript Cheatsheet – Comprehensive PDF included – iLoveCoding
  5. GRID: A simple visual cheatsheet for CSS Grid Layout
  6. FLEX: A simple visual cheatsheet for flexbox
  7. React cheat sheet

Best practices in translating your platform

Posted: October 27th, 2021 | Author: | Filed under: Developers | No Comments »

Translating a platform becomes more important as the market shifts toward globalization and the internet became available even for poor countries. But as a developer it is a challenge to organize the translations in such a way that serves all parties involved in the process.

Let's begin with stories:

  1. as a developer I want to be able to use the same translations for common buttons such as edit, delete, add, etc
  2. as a developer I want to have the translations grouped by a model so I can find them easily when working on a particular feature.
  3. as a developer I want to split the translation by modules/packages/bundles so I can stay focused on a single part of the application when working on it
  4. as a project manager I want to have the translations grouped by business logic so I can give them to a translator then to a reviewer

Possible implementations related to the location of the translation files

Note that <somepackage> is the name of a package/module/bundle and <language> is the language code.

  1. All files under the same translation folder:
  2. Translations inside the same folder of different packages
  3. All files under language-related folders:
    – this one opens the possibility of having not only 'yaml translation files' but also entire documents translated and stored here, like emails in both text and html formats.
  4. Nested paths, e.g. the list of translations is made of a sequence of files where the keys in the later files overwrites the first ones.
    – while is neat for the developer, it may be difficult for the project manager to cover all implications

Possible approaches related to the position of the translation keys inside the files

  1. Use a <global> key for global translations and <model> keys for specifics
    edit: Edit
    create_user: Create user
    – the problem here may be that sometimes seems awkward not to place an "invitation" keyword under an "organization" model because the "invitation" is also a model, but in the application it makes sense to relate the invitation to the organization only
  2. Use business logic based on the features implemented (by menus, for example):
    create_user: Create user
    purchased_plan: Purchased plan

Every approach seems to have some advantages and disadvantages, but choosing the proper approach has to be done very early in the project's lifetime and once chosen one stay stick with it in order to avoid frustrations. Which one do you prefer?

How to switch PATH in fish shell – with example script

Posted: October 22nd, 2021 | Author: | Filed under: Server side | No Comments »

fish is a Unix shell with a focus on interactivity and usability. Fish is designed to give the user features by default, rather than by configuration.[4] Fish is considered an exotic shell since it does not rigorously adhere to POSIX shell standards, at the discretion of the maintainers.

Although being "an exotic shell" once installed you'd never want to remove it since it proves so useful in the daily tasks. However, one of the annoying side effects is that it is not easy to remove paths from the PATH variable, which is required when wanting to change the environment. Here is an example:  changing from PHP74 to PHP72 on a MacOS, for both the CLI and FPM configurations.

Scripts to switch PHP versions from PHP72 to PHP74

> cat ~/

#!/usr/bin/env fish

#remove PHP72
for argv in /usr/local/opt/[email protected]/bin /usr/local/opt/[email protected]/sbin;
    while contains $argv $fish_user_paths
        set -l index (contains -i $argv $fish_user_paths)
        set –erase –universal fish_user_paths[$index]

#add PHP74
for argv in /usr/local/opt/[email protected]/bin /usr/local/opt/[email protected]/sbin;
    if contains $argv $fish_user_paths
        echo "Path already contains $argv"
fish_add_path $argv

echo $fish_user_paths | tr " " "\n" | nl

brew services stop [email protected]
brew services start [email protected]

Run it:

> ~/

     1 /usr/local/opt/[email protected]/sbin

     2 /usr/local/opt/[email protected]/bin

Stopping `[email protected]`… (might take a while)

==> Successfully stopped `[email protected]` (label: [email protected])

==> Successfully started `[email protected]` (label: [email protected])

Scripts to switch PHP versions from PHP74 to PHP72

> cat ~/

#!/usr/bin/env fish

#remove PHP74
for argv in /usr/local/opt/[email protected]/bin /usr/local/opt/[email protected]/sbin;
    while contains $argv $fish_user_paths
        set -l index (contains -i $argv $fish_user_paths)
        set –erase –universal fish_user_paths[$index]

#add PHP72
for argv in /usr/local/opt/[email protected]/bin /usr/local/opt/[email protected]/sbin;
    if contains $argv $fish_user_paths
        echo "Path already contains $argv"
        fish_add_path $argv

echo $fish_user_paths | tr " " "\n" | nl

brew services stop [email protected]
brew services start [email protected]


Run it:


     1 /usr/local/opt/[email protected]/sbin

     2 /usr/local/opt/[email protected]/bin

Stopping `[email protected]`… (might take a while)

==> Successfully stopped `[email protected]` (label: [email protected])

==> Successfully started `[email protected]` (label: [email protected])

5 steps to install XHProf + XHGui on MacOS El Captain

Posted: November 10th, 2016 | Author: | Filed under: General | No Comments »

PHP (Symfony) Profiling with XHProf and XHGui







  1. Verify if PECL is installed. If it is not installed, install PEAR then PECL:

#which pecl

#which pear

#port search –name –line –glob 'php*' |grep -i pear
php5-pear    20161027    php www    Obsolete port, replaced by php55-pear
php53-pear    20161027    php www    Optional port adds the PEAR repository to the include path for php53.
php54-pear    20161027    php www    Optional port adds the PEAR repository to the include path for php54.
php55-pear    20161027    php www    Optional port adds the PEAR repository to the include path for php55.
php56-pear    20161027    php www    Optional port adds the PEAR repository to the include path for php56.
php70-pear    20161027    php www    Optional port adds the PEAR repository to the include path for php70.

#sudo port install php55-pear

No trace of PECL on the system yet so try another PEAR installation:

#sudo curl -o /usr/local/bin/pear

#sudo chmod +x /usr/local/bin/pear

#which pecl

It seems that PECL did not install with the methods above so take the third approach:

#sudo php /usr/lib/php/install-pear-nozlib.phar -d /usr/local/lib/php -b /usr/local/bin

#which pecl

2. Install the modules for PHP starting with XHProf:

#php -v
PHP 5.5.38 (cli)

#port search –name –line –glob 'php*' |grep -i xh
php-xhprof    0.9.4    php devel    A Hierarchical Profiler for PHP
php53-xhprof    0.9.4    php devel    A Hierarchical Profiler for PHP
php54-xhprof    0.9.4    php devel    A Hierarchical Profiler for PHP
php55-xhprof    0.9.4    php devel    A Hierarchical Profiler for PHP
php56-xhprof    0.9.4    php devel    A Hierarchical Profiler for PHP
#sudo port install php55-xhprof

3. Before installing the XHGui module for php, install its requirements: MongoDB and its PHP modules:

#port search –name –line –glob 'php*' |grep -i mongo
php-mongo    1.6.14    php databases devel    Mongo Database Driver
php-mongodb    1.1.9    php databases devel    MongoDB Database Driver
php53-mongo    1.6.14    php databases devel    Mongo Database Driver
php53-mongodb    1.0.1    php databases devel    MongoDB Database Driver
php54-mongo    1.6.14    php databases devel    Mongo Database Driver
php54-mongodb    1.1.9    php databases devel    MongoDB Database Driver
php55-mongo    1.6.14    php databases devel    Mongo Database Driver
php55-mongodb    1.1.9    php databases devel    MongoDB Database Driver
php56-mongo    1.6.14    php databases devel    Mongo Database Driver
php56-mongodb    1.1.9    php databases devel    MongoDB Database Driver
php70-mongodb    1.1.9    php databases devel    MongoDB Database Driver

#sudo port install php55-mongodb
#sudo port install php55-mongo
#ls /opt/local/lib/php55/extensions/no-debug-non-zts-20121212/

On my system the modules were placed in another folder than the one where PHP is configured so a copy is needed:
#sudo cp /opt/local/lib/php55/extensions/no-debug-non-zts-20121212/mongo* /usr/lib/php/extensions/no-debug-non-zts-20121212/
#sudo cp /opt/local/lib/php55/extensions/no-debug-non-zts-20121212/ /usr/lib/php/extensions/no-debug-non-zts-20121212/

Continue by enabling the extensions:

#sudo vi /etc/php.ini

This command actually installs the MongoDB database. The ones above installed the php driver only :

#brew install mongodb

#sudo chgrp staff /usr/local/var/log/mongodb /usr/local/var/mongodb

#sudo chmod 0775 /usr/local/var/log/mongodb /usr/local/var/mongodb

Add this to /usr/local/etc/mongod.conf:

fork: true

#mongod –config /usr/local/etc/mongod.conf


Optional, use a tool to install Pref into System Preferences in order to start mongodb

#cd <downloads>
double-click on MongoDB.prefPane to install it and launch it

4. Install XHGui and configure it as a Virtual Server:

#cd <some directory for xhgui>
#git clone
#cd xhgui/
#chmod 0777 cache/
#php install.php

#sudo vi /etc/apache2/extra/httpd-vhosts.conf
<VirtualHost *:80>
ServerAdmin [email protected]
DocumentRoot "/Sites/xhgui/webroot"
ServerName xhgui.lh
ServerAlias www.xhgui.lh
ErrorLog "/private/var/log/apache2/xhgui.lh-error_log"
CustomLog "/private/var/log/apache2/xhgui.lh-access_log" common

#sudo vi /etc/hosts       xhgui.lh www.xhgui.lh

#sudo apachectl restart

Optional, but recommended: Add indexes to MongoDB to improve performance. (

# mongo
use xhprof
db.results.ensureIndex( { 'meta.SERVER.REQUEST_TIME' : -1 } )
db.results.ensureIndex( { 'profile.main().wt' : -1 } )
db.results.ensureIndex( { 'profile.main().mu' : -1 } )
db.results.ensureIndex( { 'profile.main().cpu' : -1 } )
db.results.ensureIndex( { 'meta.url' : 1 } )

5. Add this to VirtualHost of the site needing profiling (/etc/apache2/extra/httpd-vhosts.conf):

php_admin_value auto_prepend_file "/Sites/xhgui/external/header.php"

That's it. See it in browser:


xhgui xhprof

PHP + Apache + mod_fcgid + suexec on Ubuntu and Mac

Posted: August 14th, 2014 | Author: | Filed under: General | No Comments »


On linux machines (Mac too) the default installation of Apache+PHP is sometime not suitable for modern script due to at least two reasons:

      1. speed
      2. user rights

While the first is obvious, the second one needs some explanations and the best way is by giving a real-world example. In the Symfony framework the content is prepared under a "cache" folder for achieving a very good speed in processing requests. The framework also uses some command prompt statements for achieving various tasks such as database structure update, warming up the cache, etc. As one knows, all requests processed by Apache runs by default under its user, be it "apache", "www" or "nobody". On the other hand the command line statements run under the current logged-in user, let's call it "myuser". Both actions above uses the same cache folder and here is where the conflict appears, since one cannot update the other's files. Hence the need to run the script under the same user, both withing the Apache and from the command prompt. Plus that is faster (developers would like this).

Goal & solution

One recommended configuration for achieving the speed is to replace the standard mod_php with mod_fcgid, meaning that the process won't run anymore inside Apache but as an external process called by Apache when needed.

As for running the process as another user than Apache's, there is suexec which consists of a module running under apache and a binary executable acting as a wrapper.

The configuration presented here uses virtual hosts, a know feature of Apache letting one run multiple websites on the same computer, each with its own configuration, folder and user account. The process of configuring the server is split in two parts, first for configuring the mod_fcgid module then adding suexec mechanism.

Configuring mod_fcgid

The configuration starts by disabling the standard php module and installing the mod_fcgid module. The steps differs based on the linux distribution therefore the steps below will be given for Ubuntu and Mac so the reader will have to figure out how it works for other ditributions.


sudo apt-get install libapache2-mod-fcgid php5 php5-cgi

Note that this article assume that apache is already installed. If php5 is also installed is even better, just be sure php5-cgi is also.


sudo port install php5 +fastcgi

sudo port install mod_fcgid

Then disable mod_php end enable mod_fcgid:


sudo a2dismod php5
sudo a2enmod fcgid


sudo vi /private/etc/apache2/httpd.conf

Comment this line: #LoadModule php5_module libexec/apache2/

Add/uncomment this line: LoadModule fcgid_module libexec/apache2/

Add this line to your php.ini (Ubuntu: /etc/php5/cgi/php.ini, Mac: /etc/php.ini)



Tip: the location of php.ini differs and not only that, there may be two different files, one for PHP as module and the other for PHP as CGI. Locate them in the system with:

locate php.ini

A better approach can be to use php itself to tell where is its ini file:

php -i |grep php.ini


Loaded Configuration File => /etc/php.ini

Additional useful information, in case one needs to know what PHP version runs from command line, especially where there are two PHP versions installed:

#which php

#php -v

PHP 5.4.24 (cli) (built: Jan 19 2014 21:32:15)


Now is the time to configure mod_fcgid and this is done in two places: one global place inside apache's own confguration files and the other in each site. The global configuration file looks like this:

Ubuntu (/etc/apache2/mods-enabled/fcgid.conf):

<IfModule mod_fcgid.c>
AddHandler    fcgid-script .fcgi
FcgidConnectTimeout 20
FcgidFixPathinfo 1
FcgidMaxRequestLen 2097152

Mac (/etc/apache2/other/fcgid.conf):

<IfModule mod_fcgid.c>
FcgidIPCDir /usr/lib/apache2/fcgid/sock
FcgidProcessTableFile /usr/lib/apache2/fcgid/fcgid_shm
FcgidConnectTimeout 20
FcgidFixPathinfo 1

# fix for:   mod_fcgid: read data timeout in 40 seconds
FcgidIdleTimeout 3000
FcgidMinProcessesPerClass 100
FcgidConnectTimeout 120
FcgidIOTimeout 400
FcgidMaxProcesses 150
FcgidBusyTimeout 600

# to get around upload errors when uploading images increase the MaxRequestLen size to 15MB
FcgidMaxRequestLen 15728640

FcgidProcessLifeTime 7200

Although the Mac configuration seems more complex, it is only because more configuration directives have been used. Normally the two may look the same, except the paths of FcgidIPCDir and FcgidProcessTableFile

Tip: Be aware that the names of the directives were changed since mod_fcgid 2.3.2  (it is currently at version 2.3.9). The reason you're warned here is because many answers online uses PHP_Fix_Pathinfo_Enable instead of FcgidFixPathinfo. (all changes in the table here). The most important directives to use are: FcgidFixPathinfo, FcgidMaxRequestLen and FcgidBusyTimeout.


The site configuration file should include the followings somewhere inside <VirtualHost> tags:

Ubuntu (/etc/apache2/sites-available/ & Mac (/etc/apache2/extra/httpd-vhosts.conf):

<IfModule mod_fcgid.c>
#SuexecUserGroup myuser mygroup
<Directory /var/www/ >
Options +ExecCGI
AllowOverride All
AddHandler fcgid-script .php
FcgidWrapper /var/www/php-fcgi-scripts/php-fcgi-starter .php
Order allow,deny
Allow from all


Tip: Note that in the above configuration the line with SuexecUserGroup had been commented. This is to avoid seeing errors caused by Suexec while setting up fcgid. Later the line will be un-commented. Or, if you feel lucky, un-comment it now.

Once the above are set, restart apache:

Ubuntu: sudo service apache2 restart

Mac: sudo apachectl restart


Now is is time to talk about the log files because they tell valuable information about the problems which may occur.

Apache's log file to watch is named "error_log" and there you may find issues related to why the server does not start, when some SSL configuration is broken or when suexec has issues.

Suexec has its own log in the same folder as Apache's and there you will find issues related to user rights or problems when running the executable.

Each site is recommended to have its own access and error log files. There you will find errors related to the script or accessing the script, acting as a starting point in solving the issues. In case of errors related to suexec you will find here only some indications, while the details have to be found in the suexec file above

Configuring suexec

This executable binary usually comes with Apache but on a Mac installation this binary was missing. With or without it, sometimes is necessary to adjust its default settings, e.g. those it has been compiled with. Below all these situations will be covered so one can skip some steps if the default installation fits the needs.

Begin with checking the location end existence of suexec:

# which suexec

If the binary is not in the path you won't get any result. Try this:

# locate suexec

The output should be a path like /usr/lib/apache2/suexec on Ubuntu or /usr/bin/suexec on Mac, but the location may vary.

Once you located it, run it to see its current settings. Since it is supposed to be run by root only, you need to use "sudo":


# sudo /usr/lib/apache2/suexec -V
-D AP_DOC_ROOT="/var/www"
-D AP_HTTPD_USER="www-data"
-D AP_LOG_EXEC="/var/log/apache2/suexec.log"
-D AP_SAFE_PATH="/usr/local/bin:/usr/bin:/bin"
-D AP_USERDIR_SUFFIX="public_html"


#sudo /usr/bin/suexec -V
-D AP_DOC_ROOT="/Users"
-D AP_LOG_EXEC="/var/log/apache2/suexec_log"
-D AP_SAFE_PATH="/usr/local/bin:/usr/bin:/bin"

Pay attention to DOC_ROOT : it is the only path allowed to run executable (wrapper) from. The HTTPD_USER is useful in verifying that the wrapper is accessible by this user Finally, UID and GID are constraints for minimum user and group IDs, as they are defined in the /etc/passwd file.

The wrapper file is being launched by Apache for any request whose extension ends in PHP, as defined in the site's configuration file above and suexec must have the rights to access it. Also, the file must be inside DOC_ROOT. The file looks like this:

export PHPRC
exec /usr/lib/cgi-bin/php

Tip: PHPRC is the path up to php.ini. The executable on the last line must be the cgi version of php binary. Sometimes (on Mac) it is called "php-cgi" and is under /opt/local/bin/. Just check yours using "which" or "locate" and run it as "php-cgi -v" to verify that it is ok and has the expected version.

So, create the file under DOC_ROOT, let's say "/var/www/php-fcgi-scripts/php-fcgi-starter" with the above content and make it executable (chmod a+x /var/www/php-fcgi-scripts/php-fcgi-starter). Take care yourself of creating the folder php-fcgi-scripts and assigning it the proper user and group (myuser and mygroup).

Tip: an easy way to verify that a user can access that file is to add "-v" as parameter for php on the last line then call it with:

su myuser -c "/var/www/php-fcgi-scripts/php-fcgi-starter"

In other words: "run this command as user myuser" and it will display the PHP version. Remove the parameter afterward. Repeat: remove the parameter afterward (so it will run whatever PHP file will be send by Apache as parameter)


Back on track now, if everything goes well you should remove the comment to SuexecUserGroup and restart Apache. Prepare a simple PHP file in the site's root folder with the following content:

<?php phpinfo();

If everything was setup successfully then you'll get the nice long listing with PHP's current configuration. Among those lines you will find references to mod_fcgid and all other features configured.


Additional info

Sometime installing on MacOS using brew or port is not enough due to missing packages. In this case compiling from sources is a good alternative and it is not that difficult. Here is a sample of configuring PHP but also making use of "port" for installing supporting libraries

Download PHP from sources first then run this from the folder where the files were unpacked:

$ sudo port install libpng freetype mcrypt

$ sudo './configure' '–prefix=/opt/local' '–mandir=/opt/local/share/man' '–infodir=/opt/local/share/info' '–program-suffix=55' '–includedir=/opt/local/include/php55' '–libdir=/opt/local/lib/php55' '–with-config-file-path=/opt/local/etc/php55' '–with-config-file-scan-dir=/opt/local/var/db/php55' '–disable-all' '–enable-bcmath' '–enable-ctype' '–enable-dom' '–enable-filter' '–enable-hash' '–enable-json' '–enable-libxml' '–enable-pdo' '–enable-session' '–enable-simplexml' '–enable-tokenizer' '–enable-xml' '–enable-xmlreader' '–enable-xmlwriter' '–with-bz2=/opt/local' '–with-mhash=/opt/local' '–with-pcre-regex=/opt/local' '–with-libxml-dir=/opt/local' '–with-zlib=/opt/local' '–without-pear'  '–enable-fileinfo' '–enable-phar' '–disable-fpm' '–enable-cgi'  –with-freetype-dir –enable-mbstring –with-curl –with-mcrypt=/opt/local –with-zlib –with-gd –with-pdo-mysql –with-mysqli –with-openssl –with-gettext=/opt/local –with-xmlrpc –with-xsl –with-png-dir=/opt/local –with-jpeg-dir=/opt/local –with-freetype-dir=/opt/local


Downloading Apache from sources requires this configuration:

./configure –with-layout=Darwin –enable-suexec –with-suexec-caller=_www –with-suexec-uidmin=500 –with-suexec-gidmin=10 –with-suexec-docroot=/Users/ –with-suexec-userdir=Sites \
–prefix=/usr/local/apache2 –enable-so \
–enable-rewrite –enable-ssl –enable-mods-shared='rewrite ssl'


Setup of Mercurial server with cPanel – the easy way

Posted: March 23rd, 2012 | Author: | Filed under: General | No Comments »

Mercurial is a cross-platform, distributed revision control tool for software developers. In this article will be presented a handy way to install and configure Mercurial on a server with cPanel . All operations presented will only need a standard FTP/SSH access to a user account, except the initial installation of the Mercurial RPM which needs root rights.  This installation procedure is very suitable for those creating websites and wanting a quick solution to track the changes in their source code using a repository stored right next to the website.


The information below were compiled from several sources with some modifications to suit the above needs.  The aim is to use only cPanel and SSH with user rights (e.g. no root required !)  and have a repository ready in 5 minutes.  The repository will then be accessed securely as


Intial preparation:   install the Mercurial package as root

Here is a sample installation from a RPM on an OpenSuse linux:

$ sudo rpm -ihv mercurial-1.0-1.2.x86_64.rpm


Step 1.  Create subdomain

Login in cPanel and create  Do not create the directory under /public_html  so the repository won't mix with the files of the website


Step 2. Create the configuration file of the repository

Use a FTP client and navigate into the directory of the subdomain. Create a folder for your repo, i.e. projectrepo . Use some name other than the one you want to access the repo with because later there will be some URL rewiting rules added into a .htaccess file which won't work otherwise.  In this example the repo url will be made as

Continue navigating into cgi-bin where two files has to be be uploaded: hgweb.config containing the repository configuration and hgwebdir.cgi containing the code needed to run the repository.  Create the two files on your computer and upload them into /cgi-bin:


project = ~/hg/projectrepo


#!/usr/bin/env python
from mercurial import demandimport; demandimport.enable()

import cgitb

import os
os.environ["HGENCODING"] = "UTF-8"

from mercurial.hgweb.hgwebdir_mod import hgwebdir
import mercurial.hgweb.wsgicgi as wsgicgi

application = hgwebdir('hgweb.config')

Change the attributes of hgwebdir.cgi to be executable (0755).  This can be made within the FTP client (at least TotalCommander implements it and probably many other such applications have that option. It is under Files->Change attributes).


Step 3. Initialize repository

Using a SSH console go into the ~/hg folder and run these commands:

hg init

find . -type f -exec chmod 0666 "{}" \;

find . -type d -exec chmod 0777 "{}" \;


The last two commands are needed when the web server (Apache) runs the scripts under a different user than the owner of the account.  In order to create files and folder full write rights are needed for user apache or nobody.

A configuration file will have to be created and uploaded into the new repository. For this , create a file named hgrc with the following content:


description = My repository
push_ssl = false
allow_push = john

# enable snapshot downloads
allow_archive = gz zip bz2

baseurl = /

In this configuration the ssl is disabled which may be a security problem. If your host allows https connections then you may set this to true later. However, the repository is protected through HTTP Basic authentication – see below.

Upload the file above under ~/hg/project/.hg/


 Step 4. Secure the repository

Create a .htaccess file with the following content for uploading in the root of your subdomain:


# Taken from
# Used at
Options +ExecCGI
RewriteEngine On
#write base depending on where the base url lives
RewriteBase /cgi-bin
RewriteRule ^$ hgwebdir.cgi  [L]

# Send requests for files that exist to those files.
RewriteCond %{REQUEST_FILENAME} !-f
# Send requests for directories that exist to those directories.
RewriteCond %{REQUEST_FILENAME} !-d
# Send requests to hgwebdir.cgi, appending the rest of url.
RewriteRule (.*) /cgi-bin/hgwebdir.cgi/$1  [QSA,L]

The rules above allows accessing the repository with an url like instead of and is a replacement for the original procedure described in the Mercurial documentation where one has to modify httpd.conf, as root.

Finally, password protect the repository within cpanel. The operations below will update .htaccess, adding the necessary lines for basic authentication:



Access your repository within a browser with an URL like:

http://john:[email protected]/hg/project

Use the URL above in your Mercurial client too.



Postfix with DKIM, Domainkeys, SPf and Sender-ID

Posted: April 3rd, 2010 | Author: | Filed under: Server side | Tags: , , , , | 11 Comments »

The authentication methods are a great tool for fighting against spammers and every server admin should have it implemented on their systems. But since it is not that simple to configure them, they often prove to be a headache with not enough documentation available. I had to pay myself the tribute (in hours spent) for making the system work in the configuration I wanted and for the others having the same problems here is my story.My configuration is : CentOS 5, Postfix 2.3.3

The server is running some websites and I wanted to be capable of sending signed emails within PHP and also using some email client through SMTP. Since there are more than one website I wanted also a simple way to add/modify domains in the filters' configuration.

Whenever I write "domain.tld" you should use your own domain name (,, etc)

The tools I installed are listed here under the Authentication section. They are:

  • dkfilter – a SMTP-proxy signing filter. Quite easy to install but, as the author says, it is good for sending authenticated email served by the port 587 not 25 which is why I give up on it. My server must be able to send emails from PHP and act as a SMTP mail server for another server.
  • dkim-milter – DKIM signer/verifier (
  • dk-milter – Domainkeys sender/verifier. ( Pay attention to these names because it is a lot of confusion because not only DK and DKIM sounds alike but their executable are called "filters": dkim-filter and dk-filter. Couldn't be much more complicated for newbies.
  • sid-milter – Sender ID verifier. Note that it's a verifier only ! (


Installing dkim. For this follow one of these links:

How to Setup DomainKeys Identified Mail (DKIM) with Postfix and Ubuntu Server

Setting up DKIM, SPF, Domainkeys DNS, Regular DNS on CentOS 5.3

I won't go through the installation steps you can read about in tutorials as above. Instead I will focus on what should be checked, customized, verified so that will help in custom installations or troubleshooting.

So, for the installation I will focus on:

subject value description
username/usergroup domainkeys.domainkeys This is the owner of the signer/verifier processes. Doesn't matter the names but once created will be used for all filters
folders for configuration /etc/mail/dkim/
These must be owned by the user above. When adding new domains I have to modify some files here
initialization scripts /etc/init.d/dkim
Essentially is the same start-up script configured for each filter. Good to know where they are when testing the configuration
folders for private keys /var/db/domainkeys/domain.tld/ These must be owned by the user above. Here I'll place the private keys
folders & files for processes' PIDs /var/run/dkim-milter/pid
These also have to be owned by the user above. The filters will create/remove the "pid" files as they need

OK, so here there are some background information. DKIM and Domainkeys uses two DNS records and a private key to sign the emails. The DNS records are

  1. policy record. It is defined as a TXT record :
    _domainkey.domain.tld IN TXT "t=y;o=~"

    t=y means the domain is in test mode. When everything works it can be changed to "n"

    o=~ means that some emails will be signed. "o=-" means that all domains are signed

  2. selector record. Also a TXT record storing the key and other parameters:
    default._domainkey.domain.tld IN TXT  "g=*; k=rsa; p=MIGfMA0...ABC"

    "default" is the selector name and will be the same as the one used for generating the keys – see below

    "_domainkey" is a keyword preceding the domain name

    "k=" key type. Optional. Default is rsa

    "p=" public key data.

    "g=" granularity of the key. Optional, default is "*"

    "v=DKIM1" version. It is recommended but I got some errors when I tried to use the same record for Domainkeys because it is defined for DKIM only so I did not use it.  Here is the yahoo tool I tested with

Note: in the two dns records above I used "domain.tld" only because my external nameserver service required it in its web interface. If you own the namesever you will probably not use it as will be appended automatically by named/bind

Important : the server may sign ALL emails with its own domain name or may sign it with other domains but in this case all other domains must have the two DNS records maintained. I went for the first option.

Generating the keys

there are three ways to generate them:

  • online Domain Key / DKIM Key Generation Wizard
  • basic:
    openssl genrsa -out private.key 1024
    openssl rsa -in private.key -out public.key -pubout -outform PEM
  • script included with the dkim:
    dkim-genkey -r -d domain.tld

The result will always be two files, one private key to copy into /var/db/domainkeys/domain.tld/default and one to use it in the DNS record as value for "p=" parameter. Pay attention that I used "default" as filename for the private key in order to use that file for a selector named "default" and defined as such in the DNS record. It is possible to have different selectors which would handle different emails or domains and so I can have:

/var/db/domainkeys/domain.tld/m1.pem – this is created for the selectod m1 and I added an extension just to remember it is a private key file. the extension, if exists, must be "pem" or "private".

/var/db/domainkeys/otherdomain.tld/default – this one will serve emails for another domain with another selector named "default"

The init script

The init script will start|stop the filter and there is where I put the command to do that. It may have two forms, either with the configuration given as parameters to the executable app or with it uses a single parameter which points to a configuration file containing all settings. I used the 2nd option:

COMMAND="dkim-filter -x /etc/dkim.conf"

Then I put in the dkim.conf file all the configuration parameters I needed. Don't forget to make it readable by the user defined above (domainkeys). Some of the most important parameters I put there are:

KeyList                 /etc/mail/dkim/keylist
MTA                     MSA
Selector                default
UserID                  domainkeys:domainkeys
Socket                  inet:8891@localhost

Don't forget that the port numbers will be used in your /etc/postfix/ file:

smtpd_milters = inet:localhost:9967,inet:localhost:8891,inet:localhost:8892
non_smtpd_milters = inet:localhost:9967,inet:localhost:8891,inet:localhost:8892
milter_protocol = 2

I have here 3 filters, dkim, domainkeys and sender-id, each on its own port. You will find below the entire configuration file I created.

The keylist file

the keylist file contains the relation between the sender-pattern, the signing domain and the path to the private key. I use this keylist file in order to manage easier the domains on my webhost server. For a singe domain one would only use the dkim's configuration file. My keylist file's location is defined in the configuration :

KeyList /etc/mail/dkim/keylist

And contains :


..meaning: all emails (*) will be signed with the domain "domain.tld" using the private key from the file "/var/…/default".

The selector used in  the signature will be the filename portion of  keypath. If the  file referenced by keypath cannot be opened, the  filter will try  again by appending ".pem" and then ".private" before giving up. (man dkim-filter.conf)

Useful commands, part I

  • postsuper -d ALL

    Delete all mail queue. Sometimes due to too many unsent emails the logs are filled in continuously by the emails in queue. For a clean start it's good to have an empty mail queue. Be careful on a production server (you may just ignore it)
  • mail [email protected]

    Send emails from command line. good for a quick test. Verify the results on a yahoo account for both DKIM and Domainkeys
  • telnet localhost 25
    EHLO domain.tld
    MAIL FROM: [email protected]
    RCPT TO: [email protected]
    bla bla

    This is a suite of commands to test sending email through SMTP. Or you can use an email client set for sending through SMTP protocol
  • php -r "mail('[email protected]','test mail','bla bla','From: [email protected]');"

    Let the email be sent within PHP. This will test PHP mail() configuration

    A note here: At a certain point my mail sent from PHP was not signed with Domainkeys, only with DKIM. After several tests I found that the $headers in my php script were using Windows separator for new like (\r\n) instead of Linux (\n). I never thought it matters in headers !

  • Online tools: DomainKey Implementor's Tools and Library for email servers & clients especially "DomainKey Policy Record Tester." and "DomainKey Selector Record Tester."
  • tail -f /var/log/maillog

    this is where I look for errors

2. Domainkeys

As above, start with the download & installation from the given links from sourceforge or with the ones below:

Quick And Easy Setup For DomainKeys Using Ubuntu, Postfix And Dkim-Filter

man dk-filter – DomainKeys filter for sendmail online version of the man page for dk-filter

The DNS records follows the same rules as the DKIM above. In fact both DKIM and Domainkeys will use the same DNS records

The init script

The dk-filter does not support all parameters supported by DKIM and especially the one which would allow me to put all configuration in one file therefore the command from this initialization script has all parameters in one line:

COMMAND="dk-filter -u $USER -p inet:$PORT@localhost -l -P $PIDFILE -d $DOMAIN -S $SELECTOR -c simple -k -s $KEYFILE -i $IFILE -m $SUBMISSION_DAEMON -D"

  • -u userid it runs under
  • -p socket to be used by the filter for allowing connections from postfix. It must use a different port number than dkim (of course)
  • -l log calls to syslog
  • -P pidfile – the file used to write the process ID into
  • -d domain list or path to a file containing the domain list one per line (wildcard * allowed in the domain name )
  • -S selector, the name of the selector to be used
  • -k Causes -s to be interpreted as the location of a key list
  • -s keyfile or keyfilepath. Since -k is used this one will be a path to my keyfile (/etc/mail/domainkeys/keylist)
  • -i a file of internal hosts whose mail should be signed rather than verified (/etc/mail/domainkeys/ilist)
  • -D Sign subdomains too

The content of the keylist (differs from DKIM's !):


The ilist contains some IPs (local network's):

Note that I am using "/etc/mail/dkim/trusted-hosts" as value for the -d (domain) parameter. This is a path pointing to a dkim folder because I want to make as little changes as possible when adding or removing a domain so I use it for both filters.

The keylist file

This is another small difference than the DKIM's keylist file content. It looks like this (/etc/mail/domainkeys/keylist) :


Each line is in the form "sender-pattern:keypath" where sender-pattern is compared against the sender and keypath is the path to the private key file. The filename portion of this keypath is also the selector name

Some problems I run into

At some moment I got in maillog : "can't read SMFIC_EOH reply packet header: Success" which was solved after setting the SUBMISSION_DAEMON above as MSA (instead of smtp as I saw somewhere)

Playing with milter_protocol=4 in /etc/postfix/ I got "can't read SMFIC_DATA reply packet header: Success" which was because I've given a value which was not supported. Also milter_protocol=6 thrown error so I let it set on : milter_protocol = 2

Useful commands, part II

  • postconf -d

    Display the default postfix configuration
  • postconf -n

    Display the non-default parameters of the postfix configuration
  • /usr/sbin/sendmail -t -f [email protected] [email protected]
    From: [email protected]
    Subject: test it
    bla bla

    Test sending email using the sendmail command as defined in the php.ini . Note that php.ini has an additional parameter for sendmail, "-i" which suppress the use of a dot as "end of input" data.
  • ps aux |grep dk

    This should confirm that the filters are running (dk-filter and dkim-filter). It was useful to check this because when I got errors from filters they had just ended silently.

3. Sender-id

This would be the 3rd filter installed and is aimed for hotmail accounts. There is a confusion between Sender-ID and SPF because they both use the same DNS registration record but the implementation of the Sender-ID protocol is similar with DKIM above. Read more about SPF vs Sender ID

Note that in order to sign the emails correctly one would only need the SPF record – see next section. At least this should work for 80% of the emails sent to hotmail/MSN according to the link above.

The init script

I named the init script "/etc/init.d/sender-id" and the command used to start the filter is:

COMMAND="sid-filter -u $USER -t -p inet:$PORT@localhost -l -P $PIDFILE -d $DOMAIN -D -h"

The parameters means:

  • -u username to run under
  • -t test mode (never reject messages)
  • -p socketspecs the socket for postfix to communicate with
  • -l log via syslog (/var/log/maillog)
  • -P the filename which will store the PID of the current process
  • -d a list of domains whose emails should be ignored by this filter.
    Warning : this is a list of the domains to be ignored so if you want to see this filter's signature in the header of your emails then use an empty value for DOMAIN. Somehow strange in my opinion.
  • -D the way to handle the DNS errors (soft failures)
  • -h adds a header indicating the presence of this filter

4. SPF

The SPF authentication is used by google and is quite simple requiring no software installation on the server. In order to generate a SPF record for your domain you will probably have to go to the most-used tool :

Setup wizard to create SPF records for your domain

The record it will create should be placed as a TXT record at your domain and looks like this:

v=spf1 a a:external.domain.tld ~all


  • "v=spf1" is the SPF protocol version
  • "a" mechanism matching current domain (e.g. my own domain.tld)
  • "a:external.domain.tld" mechanism matching an external domain which may use my own domain.tld to send emails
  • "mx" mechanism matches the current domain's MX records. Useful if I want to have a secondary server act only as mail server
  • "all" mechanism matches everything else. It usually stay at the end. The character which prefixes this mechanism means "SoftFail" and it only indicates that all other domains than the one defined won't be recognizable by this SPF record

Definition: Mechanisms can be used to describe the set of hosts which are designated outbound mailers for the domain.

Observation: the SPF standard specify that there should be used the new DNS registration record type SPF instead of TXT but because of compatibility with the current DNS implementations it is recommended to put both TXT and SPF records with the same content.

Useful commands, part III

Here are some of the tools and commands I used to verify that the record is in place and the configurations above are working:

  • dig domain.tld TXT

    This would query the nameserver for the TXT record and should return something like:
    domain.tld.             300     IN      TXT     "v=spf1 a a:external.domain.tld ~all"
  • Send an email with an empty subject to [email protected] . I strongly recommend using an email client because sending from comand prompt resulted sometime in failed Domainkey verification due to probably some extra characters added into the headers. The result will arrive in email and should look like this: Thank you for using the verifier,
    The Port25 Solutions, Inc. team
    Summary of Results
    SPF check: pass
    DomainKeys check: pass
    DKIM check: pass
    Sender-ID check: pass
    SpamAssassin check: ham
  • cd /etc/init.d/; chkconfig --add sender-id

    this adds the script to the list of programs which should run after restart. Do this for each init script

Last thoughts

I invite any reader to help improving this newbie-guide based on their experience and knowledge. There is plenty room for suggestions and additions to this article but this is an area where with all documentation available it seems that it is not enough or not covers all configuration or problems which may occur when setting up such a system. I spent myself several (many) good hours reading and learning, trying and failing or succeeding and finally writing this article hoping that will help others to save some of their time and neurons while trying to follow the latest recommendation of the war against spammers.

Although not required, a donation is always welcome end encouraging 😉


using PayPal



DKIM configuration file /etc/dkim.conf

DKIM startup script /etc/init.d/dkim

Domainkeys startup script /etc/init.d/domainkeys

Sender-id startup script /etc/init.d/sender-id


How to Setup DomainKeys Identified Mail (DKIM) with Postfix and Ubuntu Server

Setting up DKIM, SPF, Domainkeys DNS, Regular DNS on CentOS 5.3 at

Set Up DKIM On Postfix With dkim-milter (CentOS 5.2)

DomainKey Implementor's Tools and Library for email servers & clients

Set Up DKIM For Multiple Domains On Postfix With dkim-milter 2.8.x (CentOS 5.3)

Setting Up SPF, SenderId, Domain Keys, and DKIM

Setting Up SPF, SenderId, Domain Keys, and DKIM

Domain Key / DKIM Key Generation Wizard

Domainkeys and DKIM with Postfix – Gentoo Linux wiki

Postfix Add-on Software

DomainKeys Identified Mail (DKIM) Signatures – RFC

Postfix / DKIM

How To Implement SPF In Postfix

Email authentication guide made by BITS

Return Path’s Best Practice Guide

Last update: May 5th, 2010

RegEx email parser

Posted: April 16th, 2008 | Author: | Filed under: Developers | No Comments »

Split email formats in pieces through preg_match:

$str=' (|(([^a-zA-Z0-9_\-.]|)(.*?)(|[^a-zA-Z0-9_\-.]+?)([a-zA-Z0-9_\-.]+?)(|[^a-zA-Z0-9_\-.]+?)))(|[^a-zA-Z0-9_\-.])([a-zA-Z0-9_\-.]+?)(|[^a-zA-Z0-9_\-.])@(.+?)\.([^>]+)';

Email validity check

Posted: April 16th, 2008 | Author: | Filed under: Developers | No Comments »

PHP code:

<pre lang="php">
function is_email($email) {
  $x = '\d\w!\#\$%&\'*+\-/=?\^_`{|}~';    //just for clarity
  return   count($email = explode('@', $email, 3)) == 2 && 
      strlen($email[0]) < 65 && strlen($email[1]) < 256 && 
      preg_match("#^[$x]+(\.?([$x]+\.)*[$x]+)?$#", $email[0]) && 
      preg_match('#^(([a-z0-9]+-*)?[a-z0-9]+\.)+[a-z]{2,6}.?$#', $email[1]);

Contact us

Posted: March 22nd, 2008 | Author: | Filed under: General | No Comments »