Archive for the ‘Utilities’ Category

Sending Emails With Attachments using Amazon SES

leave a comment »

Amazon Simple Email Service (Amazon SES) is a very reliable email service provided by Amazon for sending emails – transactional, communication to your customers, registration or bulk emails like newsletters etc.
It is easy to set up and also scales well. In addition to all this, it is quite cheap. Gone are the days when one had to pay steep prices to the hosting provider for email campaigns with all kinds of cap/limit set to it.
With SES you can send 1000s of emails in a day and even if your limit exceeds, you can request them for raising the threshold. Along with high deliverability, Amazon SES also provides real-time access to your sending statistics and built-in notifications for bounces and complaints to help you fine-tune your email-sending strategy.

So how does one get started? They have written quite a good documentation here, so I will not dwell into that.

My focus here is to share the way to do it in php. There are many ways to download the required package but I used aws phar. You need pear to download phar and set it up.

Setting up phar

  1. sudo apt-get install pear (If you don’t have pear already)
  2. sudo pear -D auto_discover=1 install pear.amazonwebservices.com/sdk

Now , to send an email with attachment – here is the code:

Include the phar file

require '/some path/php/lib/php/AWSSDKforPHP/aws.phar';
use Aws\Ses\SesClient;
global $client;

Create client object

$client = SesClient::factory(array(
'key' => 'your key',
'secret' => 'your secret',
'region' => 'your region'

The file you want to attach in the email

$myfile = "path to file to be attached";
$file_size = filesize($myfile);
$handle = fopen($myfile, "r");

Read the file content

$content = fread($handle, $file_size);

Format $content using RFC 2045 semantics using chunk_split. It inserts \r\n at the end of every chunk. This is required by your smtp client – else they just reject your attachment.

$content = chunk_split(base64_encode($content));
$header = "";
$message = "some html string";

You need to put a unique boundary to the multipart message

$uid = md5(uniqid(time()));
$header = "From: ".$source_email_id." <".$source_email_id.">\r\n";
$header .= "Reply-To: ".$replyto_email_id."\r\n";
$header .= "To: ".$dest_email_id."\r\n";
$header .= "Bcc: ".$bcc_email_id."\r\n";
$header .= "Subject: ".$subject_of_the_email."\r\n";
$header .= "MIME-Version: 1.0\r\n";
$header .= "Content-Type: multipart/mixed; boundary=\"".$uid."\"\r\n\r\n";
$header .= "This is a multi-part message in MIME format.\r\n";
$header .= "--".$uid."\r\n";
$header .= "Content-type:text/html; charset=iso-8859-1\r\n";
$header .= "Content-Transfer-Encoding: 7bit\r\n\r\n";
$header .= $message."\r\n\r\n";
$header .= "--".$uid."\r\n";
$header .= "Content-Type: text/csv; name=\"".$myfile."\"\r\n";
use different types here

$header .= "Content-Transfer-Encoding: base64\r\n";
$header .= "Content-Disposition: attachment; filename=\"".$myfile."\"\r\n\r\n";
$header .= $content."\r\n\r\n";
$header .= "--".$uid."--";
$msg['RawMessage']['Data'] = base64_encode($header);
$msg['RawMessage']['Source']= $src;
$msg['RawMessage']['Destinations'] = array($dest);

Now send the mail

$result = $client->sendRawEmail($msg);
//save the MessageId which can be used to track the request
$msg_id = $result->get('MessageId');

So as you would have observed, there are quite a few complications that one has to handle while sending an email with attachment like the unique id, putting proper end characters at the end of your message chunks, base 64 encoding etc.

Hope this helps.

Written by rationalspace

June 6, 2014 at 6:52 pm

Posted in Cloud, Utilities

Making FTP work on Amazon EC2

leave a comment »

Though FTP is not a great way to transfer files on a server due to security issues, sometimes, due to reasons like specific requirements of business partners’ etc, you have to set it up. The better way to transfer files however, is SFTP. I already wrote a blog about how to set it up here.

Now about FTP setting up.

  1. Check whether vsftpd is there on your server. If not , install it. sudo apt-get install vsftpd
  2. Check if port 21 is open. telnet IP 21. If not, go to your EC2 dashboard, check the security group to which your server is attached, edit it – Add port 21 there and save.
  3. Edit vsftpd.conf
    1. vi /etc/vsftpd.conf
    2. Disable anonymous login :
    3. Allow local users to be able to login :
    4. Allow users to write in their FTP directory :
    5. Restrict users to their local directory: chroot_local_user=YES
    6. Check your listen address :
    7. Save file and restart ftp :
      /etc/init.d/vsftpd restart
    8. Do a netstat and check output
      netstat -a | grep 21
      It should show
      tcp 0 0* LISTEN
  4. You may need to open passive mode instead of active if your firewalls or client’s firewalls don’t allow communication. To make passive mode work, you need to do some more extra stuff
    1. Make sure ports greater than 1024 are open in your security group. So add a range 1024-1048 and save.
    2. In vsftpd.conf , specify the range of ports that will be used for opening a passive connection.
    3. It turns out that vsftpd advises the incoming PASV command the internal IP of EC2 instance, which FTP clients would not be able to resolve. To solve this problem, we explicitly tell vsftp to use our public IP address instead of asking the server for it. If you don’t have an Elastic IP associated with the instance, you will need to enable pasv_addr_resolveand provide your public DNS
    4. Restart vsftpd.
  5. You may also want to read up on passive vs active modes. I found a nice answer here on stackoverflow.

Written by rationalspace

May 27, 2014 at 5:09 pm

Posted in Cloud, Security, Utilities

Tagged with , ,

Analysing apache logs

leave a comment »

View Apache requests per day

awk '{print $4}' <log file> | cut -d: -f1 | uniq -c

Code breakdown:

awk ‘{print $4}’ example.com Use the awk command to print out the$4th column of data from the Apache access log which is the time stamp.
cut -d: -f1 | uniq -c Use the cut command with the -delimter set to a colon : and grab the -field of data that shows up 1st before the delimiter. Then use the uniq -ccommand to uniquely count up the hits.

You should get back something like this:

6095 [20/Jan/2014
7281 [21/Jan/2014
6517 [22/Jan/2014
5278 [23/Jan/2014

View Apache requests per hour

grep "23/Jan" | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c

Code breakdown:

grep “23/Jan” progolfdeal.com Use the grep command to only show hits from today from the Apache access log.
cut -d[ -f2 | cut -d] -f1 Use the cut command with the -delimter set to an opening bracket [and print out the -field of data that shows up 2nd, then use the cutcommand again with the -delimter set to a closing bracket ] and print out the-field of data that shows up 1st which gives us just the time stamp.
awk -F: ‘{print $2″:00″}’ Use the awk command with the -Field delimiter set to a colon :, then print out the $2nd column of data which is the hour, and append “:00” to the end of it.
sort -n | uniq -c Finally sort the hours numerically, and then uniquely count them up.

You should get back something like this:

200 00:00
417 01:00
244 02:00
242 03:00
344 04:00
402 05:00
522 06:00
456 07:00
490 08:00
438 09:00
430 10:00
357 11:00
284 12:00
391 13:00
163 14:00

View Apache requests per minute

grep "23/Jan/2013:06" | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'


Code breakdown:

grep “23/Jan/2013:06” example.com Use the grep command to only show hits from today during the 06th hour from our Apache access log.
cut -d[ -f2 | cut -d] -f1 Use the cut command with the -delimter set to an opening bracket [and print out the -field of data that shows up 2nd, then use the cutcommand again with the -delimter set to a closing bracket ] and print out the-field of data that shows up 1st which gives us just the time stamp.
awk -F: ‘{print $2″:”$3}’ Use the awk command with the -Field delimiter set to a colon :, then print out the $2nd column which is the hour, follwed by the $3th colum which is the minute.
sort -nk1 -nk2 | uniq -c Sort the hits numerically by the 1st column which is the hour, then by the2nd column which is the minute.
awk ‘{ if ($1 > 10) print $0}’ Finally use the awk command with an ifstatment to only print out data when the $1st colum which is the number of hits in a minute is greater than 10.

You should get back something similar to this:

12 06:10
11 06:11
16 06:12
13 06:20
11 06:21
12 06:28
12 06:30
16 06:31
14 06:39
11 06:40
15 06:52
32 06:53
43 06:54
14 06:55

Written by rationalspace

February 27, 2014 at 3:43 pm

Posted in Apache, Utilities

Powerful tool to fight comment spam

leave a comment »

As soon as you have set up wordpress for your personal / business blog, you will face the issue of being bombarded with spam comments from all over the world! This is not only an issue of junk spoiling the content quality of your page but also bad from SEO perspective.

Thankfully, there is one tool that helps you fight this issue well. All you need to do is set this up – which is again pretty straightforward. It is free for personal blogs and they charge quite minimal for business too. So here is how you can get started –



Written by rationalspace

December 30, 2013 at 1:01 pm

Posted in Utilities

Remove empty lines from a file

leave a comment »

Here is an easy way to do this :

awk -F, 'length>NF+1' file


-F, = Field separator is “,” – assuming a csv
length>NF+1 = length of the line is greater than number of fields + 1

If yes, then this is not an empty line – so redirect the output to another file.


Written by rationalspace

April 23, 2013 at 3:35 pm

Posted in Utilities

Tagged with , ,

Easy way to split Camel Case words

leave a comment »

A number of times we are stuck with words that are camel case : – as in meaningful if you split them by capital letter.

For example : FluidDensity

You might want to create a table where column names are camel case. This may be useful if your display names are not too different from camel case words. You need not store another mapping of  database column names/machine names with display names/UI names.

You can just split the column name and display it.

We can use a simple regex : preg_split('/(?=[A-Z])/',$camelcaseword);
(?= This is a unique way provided by PHP  to start  positive lookahead

[A-Z] Any characters by which you want to split – in this case we need to split by caps

) End of look ahead assertion

Written by rationalspace

April 1, 2013 at 10:54 am

Posted in Utilities

Easy way to trim white-space around images

leave a comment »

Often the images that one gets from different publishers/content owners have huge amount of white-space around them. And simply resizing it makes it look bad cause the object becomes very tiny in the center with lots of whites around it.

This can be easily solved using “imagemagick” package.

  • sudo apt-get install imagemagick
  • convert -trim “<source image>” “<dest image>”

Written by rationalspace

February 14, 2013 at 4:29 pm

%d bloggers like this: