Saturday, April 27, 2013

The Webserver on Raspberry Pi

in this blog I will describe what I have done to implement a webserver on my raspberry pi. I was making thoughts about the user interface I want to have for my control system. I actually started with a command line interface in C++ to access the sensor data and to control the actuators (ventilator). But I realized very soon that I can never remember the syntax of command lines and it almost always leads me to look into the source code to understand how the syntax was. Also, a graphical user interface is much nicer and is something to show off. In order to save time I decided to use the command line commands, and to call them from the graphical interface. Next I had to decide how program the graphical interface. I am a C++ programmer, but I am not familiar with screen programming at all. But it would be really nice, if I could look from my work place into the basement data, or from any other place such with a smart phone. So the next decision came pretty fast: I need a webserver. Here the list of the requirements I had:
  • Make use of the available command line commands
  • Need a webserver
  • Need a website to access the basement data and to control the actuators
 So in the following chapters I will go more into detail:

 The Webserver

I am not programmer who has done with webservers a lot and I have little knowledge about php, python, css and all these other knicks knacks available on the market. But I know perl, and I remember that perl was used (and is still used) for dynamic websites. To save time again, I decided that I want to use perl, because I do not want to learn yet another language, which is doing all the same anyway. And perl has system and open comamnds to call my C++ command line commands and it can also read out the return values easily. So I need a webserver, which can do perl (probably most webserver can do anyway).
Next I had to decide which webserver I use. After a little research I found these:
 I was reading pro and cons, and I got as many opinions about webservers as websites I have visited, such as: lighttpd has memory leaks, nginx performs so well, apache is overloaded, cherokee does not perform, fhem has too small community etc etc. I made my decision upon ease of use or ease of install on the raspberry pi. I found this website, and I was seeing that lighttpd is very easy to install (probably the others too, but first comes first) and it can do perl (probably the others too). Installation of lighttp was no problem at all.
The next decision was, what gateway I use to generate HTML code with perl. Again I made a little research, but much less than before. Pretty fast I decided to use standard CGI, just because I used it in the past once, because lighttpd supports it and because I found the book Web, Graphics & Perl/Tk from O'Reilly in my bookshelf where it is described (I have to admit I did not read much). I do not remember that I had to do something special in /etc/lighttpd/lighttpd.conf, I left the setup as is after installation. I would just recommend to everyone to take a look into this file to see if perl is enabled and to check the documents root. What is important though, is that you set the permission of the document root directly /var/www appropriately and that you set the group and owner to www-data (which is the owner's name of the webserver). I also created a directory cgi-bin in /var/www for the perl programs, see also this website.

Webpage Programming

 Again I had to do some research because during my first trials, I created HTML content directly with perl print statements and the perl code did not really look good (perl code never looks good in my opinion). But I found some good perl tutorials here, which helped me to make a choice on using perl's HTML::Template. Here you can separate your perl program from your HTML code, and you dont have to use perl's print statements to generate HTML.

The Login Page

 First I did not want to spent too much time in all this, but I wanted a minimum in security. So a password entry is needed, before I can use my website. Considering that I do not have the raspberry pi running all the time, I think this is sufficient at the moment. I am not a security specialist, but I know I should not store the password in plain text for lookup on the server. I need to encrypt the typed in password and and I need to encrypt the lookup password and then compare both encrypted password. Another question was, where to store the encrypted password? I can use a file, but what if there are more than one user who want to have access. I thought that storing encrypted passwords in a simple file is not a good idea, so I chose mysql database to store them. The good part about perl is, that it has password encryption support Crypt::PasswdMD5 and mysql support DBI. So I installed the packages with the cpan command. I had to setup mysql first after installation. I basically did these commands on the raspberry pi:

mysql -u root -p #starts my sql with root as user, and new command prompt mysql should show up

mysql> create database yourdatabase;
mysql> grant all privileges on yourdatabase.* to pi@localhost;
mysql> use yourdatabase
mysql> create table users(login varchar(20), password varchar(80));

I found this program to generate a md5 encrypted password from plain text.  Also I created a perl program to add a user into the database with the encrypted password as paramter (actually I pipe it). An extract of my perl program looks like this:

use DBI;
my dbname="yourdatabase"; #see above database creation
my username="pi"

my pw=....; 
my data1 = "DBI:mysql:dbname";
my data2 = DBI->connect(data1, username, undef, { RaiseError => 1 }) || die( "cant open DB!");
my sql="insert users values('ARGV[0]', 'pw');";
my abfr1=data2->prepare(sql);


I apologize for not showing the perl dollar signs. But blogger seems to have a problem with it.

Now I can add users with their encrypted passwords into the database. Back to the login page. Perl has another convenient library is CGI::Session. This is useful to pass on parameters from one page to another. Also it has the feature to expire, which gives another security element into my webpages. Below you find an extract of the perl code to generate the login page from the login HTML template. The template just consists of a form with user entry, password entry and a submit button, so I will not explain any further. If the user is entering the user name and password into the login webpage, then data is sent to the webserver and the program below is called again. This time it will call the function user_authenticated, which is just comparing the entered user name/password with the user name/password in the database. If they match, the webserver redirects to, which is the perl code to generate my website.

use CGI::Session;
use HTML::Template;
use Crypt::PasswdMD5;
use DBI;

my session   = CGI::Session->new or die CGI::Session->errstr;
my template  = HTML::Template->new(
                      filename          => './login.tmpl',
                      associate         => [session],
                      die_on_bad_params => 0,
                      global_vars       => 1,
                      cache             => 0 );
if ( cgi->param('Login') ) {
    my home = './';
    print cgi->redirect(home) if user_authenticated();
print session->header;
print template->output;

I apologize for not showing the perl dollar signs. But blogger seems to have a problem with it.

 The login page looks like this:



The Curve Tab

The control system runs for two years with only few interruption. I was interested how the sensor data has developed during the past two years, so I think this could be best represented as a curve.
Each time the control system gathers data from the sensors, it is stored in the 1-wire controller's eeprom. I implemented a couple of command line commands to query the sensor data from the 1-wire controller into the raspberry pi. The question was now, where to put the data. The answer came fast, since I started with a database while creating the login page. So it came natural to store the sensor data into the same database. So I created a new table with the followin mysql command:

mysql> use yourdatabase;
mysql> create table sensorvalues(id int not null primary key auto_increment, datestamp date, ...  ,mot1 int); 

Then I implemented a perl program which calls one of my command line commands to gather the sensor data from the 1-wire controller eeprom and stores them into the mysql database. Meanwhile the 1-wire controller has gathered more than 2500 entries, while each entry contains data and time stamps, two relative humidity values, two temperature humidity sensor values, two absolute humidity values, and five temperature sensor values. So I grouped the relative humidity data in one graph, the temperature humidity sensor values in one graph, the absolute humidity values in one graph and the five temperature sensor values in one graph. Therefore I will have four graphs with several curves on each. I have to admit that I did not read much in the book Web, Graphics & Perl/Tk from O'Reilly, but one interesting thing I have seen is, that graphs can be easily created with gnuglot. The installation for gnuplot is described here. The good thing about gnuplot is, that you can run it as a command line and it creates a picture file.
So what I need is a mysql command which retrieves all data within a time range (start date and end date) and stores it into a file, then a gnuplot command is executed with the file as parameter and with additional view settings. The perl commands to access the mysql database look like this:

my data2 = DBI->connect(data1, username, undef, { RaiseError => 1 }) || die("cant open DB!");
my @ergf;

startdate = ARGV[0];
enddate = ARGV[1];
sql = "select * from sensorvalues where datestamp between \"startdate\" and \"enddate\";";
my abfr1=data2->prepare(sql);

I apologize for not showing the perl dollar signs. But blogger seems to have a problem with it.

 Next the perl program stores all the data into a file called "data". Then it executes the gnuplot command line which looks like this for relative humidity picture:

open(GP, "| /usr/bin/gnuplot") or die "couldnt open gnuplot: !";
print GP "set terminal png size 1000,250\n";
print GP "set xdata time\n";
print GP "set timefmt \"%Y-%m-%d\"\n";
print GP "set xrange [\"startdate\":\"enddate\"]\n";
print GP "set yrange [0:100]\n";
print GP "set format x \"%Y-%m-%d\"\n";
print GP "set timefmt \"%Y-%m-%d %H:%M:%S\"\n";
print GP "set linestyle  2 lt 2\n";
print GP "set output \"relhum\_startdate\_enddate.png\"\n";
print GP "set xtics tics\n";
print GP "set style line 1 lt 2 lc rgb \"green\" lw 2\n";
print GP "set style line 2 lt 2 lc rgb \"red\" lw 2\n";
print GP "plot \"data\" using 1:3 ls 1 title 'relative humidity 0' with lines, \"data\" using 1:4 ls 2 title 'relative humidity 1' with lines\n";
close GP;

Note the line set xtics tics. I had trouble with the x-axis labels, which did overlap. By setting xtics to an appriopriate number, the overlapping problem disappeared. The code for the other graphs is similar, so I will skip this.
I added in the curve tab HTML template a HTML select forms, where I can setup the start date and the end date. As with the login page, I use HTML::Templates to split the perl program from the HTML code. I figured very fast that handling of dates are not easy. When calling the webpage I wanted that the browser shows the current date as the end date, and one week before that as a start date. That means I need to calculate seven days from todays date. In some cases it is easy to calculate, but in other cases not. Here gets perl convenient again, because there are libraries which are already doing this. I installed Date::Calc with cpan and I included the code below:

(sec,min,hour,mday,mon,year,wday,yday,isdst) = localtime(time - 7 * 86400);

 Below you see a picture of the curve tab webpage:


In the picture above you see the select forms to setup a start date and a end date, and submit button. The default setup is todays date as end date, and 7 days before that as start date. And you see two graphs with relative humidity and temperature humidity sensor curves.

The Sensor Tab

It is good to see the sensor values on curves, but it is not really easy to read out the exact value from them. So I came to the conclusion that I need another tab where I can see the exact values from a specific date and time. 

I created an HTML select form so I can choose the date and the time from when I want to show the data. The time select form offers predefined time values, which are the times when the 1-wire controller stores the sensor values into its eeprom. After submiting with the button the webserver alls the sensor tab perl program, which is accessing the mysql database with the command:

sql = "select * from sensorvalues where timestamp like \"time0[0]:%\" and datestamp = \"sensorvalues{'sample_year'}-mon0-day0\";";

I apologize for not showing the perl dollar signs. But blogger seems to have a problem with it.

Then the perl program is storing the values into HTML::Template parameters and it is generating the HTML code from the template.

The Camera Tab

The camera tab shows the last feature I have implemented, but it is not finalized yet. When choosing the camera tab, the perl program generates HTML with a picture from a webcam, which I have installed in my backyard. I have described the implementation of the webcam in this forum (I used a raspberry pi for this too!).
The problem which I was facing here is how to get the picture from my raspberry pi driving the webcam to my raspberry pi running the webserver. I decided to use scp in combination with secure/private keys. When things go automatically, I cannot use something where I need to type in a password manually. So I have to get around it. Fortunately it can be done without passwords, if I use the scp command with the -i option and with a private key generated by ssh-keygen. I stored the public key on the server side, which is in my case the raspberry pi driving the webcam. I created a perl script which is executing the following command:

my @args = ("ssh", "-i /somedir/privkey", "pi\ ", "\"find /data/webcam/motion/* -type f -name \\*.jpg | sort -n | tail -1;\"");
open(IN, "@args |") or die "Can't open pipe: !";

I apologize for not showing the perl dollar signs. But blogger seems to have a problem with it.

The command above is first accessing the webcam raspberry via ssh. Then it executes the find command to create a list of jpg files (in this case the list has size one) the webcam has created. Then the perl program is executing the following:

my ret = system("scp -q -i /somedir/privkey pi\ /var/www/cgi-b

I apologize for not showing the perl dollar signs. But blogger seems to have a problem with it.

It accesses with scp the raspberry pi driving the webcam and copies the jpg file directly into the cgi-bin. From there the HTML code can reference the picture.
A cron job is executing both commands above and it is starting them once an hour.This means that the picture is only updated once an hours as well.
Below you can see a picture of the camera tab.

Probably I will combine this in future with a HTML select form, where I can choose the picture I want to see.

 Final Words.

 I am pretty happy with the result of my project, though I cannot tell that it is finalized yet. The camera feature needs to be improved. I am still struggling about what to do to make it more usable. I thought about using swf files instead of jpg files, so I can see some movements.
The webserver's performance is ok. It is definitively not fast, but it is sufficient for me. Especially the curve creation tab perl program takes some time (about 5-10 seccond), so I need to be patient there.
Another few words about perl. I used to dislike perl very much before I have done this project. The reason was that I cannot create good looking perl programs. But this completely has changed now after I realized how large the cpan library for perl is. I had the impression that for each problem I ran into, there was a perl library already available, such as Date::Calc, CGI::Session or Crypt::PasswdMD5. Meanwhile I really like perl!

Saturday, April 20, 2013

The Network

To have internet access to the control system, I decided to use the Raspberry Pi with the EW-7811UN wireless USB adapter which connects it to the internet router. My router can setup a dynamic DNS, so I am able to access the raspberry pi, without knowing the IP address. Having domain name is also convenient for the webserver programming, so the Raspberry Pi's webserver does not need to figure out the router's IP address. On the SPI port I added the CAN controller MCP2515, so the Raspberry can communicate with the other devices via CAN protocol, see Picture below:

There are two more controllers attached to the CAN bus: the 1 wire controller and the ventilator controller.

The 1 Wire Controller

The 1 wire controller contains interfaces for two 1 wire buses. I decided to connect the temperature sensors on the first 1 wire bus, and the humidity sensors on the second. The reason why I have a 1 wire controler is that 1 wire sensors are widely available. In theory they can even drive their power from the signal line. But I am not using the 1 wire sensors in that way (I could though). Sensors with CAN interfaces are less available, they are larger, they need power supply, and they are more expensive.

The Ventilator Controller

The ventilator controller uses two S202 S12 solid state relais to apply 220V to the ventilators to turn them on and off.

Final Words

Why did I choose CAN and 1 wire instead of using zigbee, FS20, or something similar with wireless? There are several reasons:
  • Zigbee, FS20 or anything else with wireless were not very popular or available during the time I designed the control system. I am not aware that there are sensors having zigbee (maybe there are, but I did not look), and if yes, I can imagine they are expensive.
  • I am a little suspicious about the reliability of radio controlled systems. E.g. the operating range of these radio controlled systems is rather limited. Probably I would have needed repeaters all over the house. And what if the neighbor uses the same FS20 system. So when he opens his garage door with FS20, he triggers my ventilators as well? 
  • All the devices need power supply anyway, so there must be a cable which supplies power somehow. In my system the devices draw power from the CAN cable.
So I can have radio controlled devices, which need power and probably repeaters (which also need power), or I install in a one time effort (I know it is not a little effort) a cable in the house with bus signals strands and power strands.

Sunday, April 14, 2013

The Humidity Sensor

Commercial humidity sensors usually measure only the relative humidity, because it is easy to realize technically. The relative humidity however does not describe how much water is contained in the air. It only describes the relation between the partial pressure of water vapor and the saturated vapor pressure. Alternatively you can interprete the relative humidity as the relation between absolute humidity and the saturated vapor density. In the NASA note D-8401 , you can find all formulas describing the relationships between absolute humidity and relative humidity, see formula (5). The absolute humidity comes down to a formula having temperature, relative humidity and the saturated vapor pressure as variables, see formula (2) combined with (5). The saturated vapor pressure is approximately a logarithmic function with the temperature as variable. I implemented it in the code as a lookup table. Below you find the saturated vapor pressure curve I created:

The humidity sensor

On the humidity sensor which I designed, I use the sht11 from Sensirion to measure temperature and relative humidity. It contains a proprietary 2 wire interface, so connecting to I2C will not work. The processor on the humidity sensor is a atmega16 from Atmel. The code running on the processor implements Sensirion's 2 wire protocol. Also I connected the addressable switch DS2408 to the processor. This converts bit signals applied from the processor to the switch into 1-wire signals. This is how my control system communicates with the humidity sensors.

Code to calculate absolute humidity from relative humidity and temperature

here I want to show how I have implemented a C function to calculate the absolute humidity from the relative humidity and the temperature. Above I was telling that I implemented the saturated vapor pressure as a lookup table. This is how I have done it:

   static int es[] =

The first entry corresponds to the temperature -25 degree Celsius. The last entry correspons to 45 degree Celsius. So I can just reference the array with a Celsius value (added with 25), and I get the corresponding pressure value. The piece of code below just protects the processor from overrunning memory:

    if(T < -25.0) {
      T = -25.0;
    } else if(T > 45.0) {
      T = 45.0;

Last but not least, the absolute humidity is calculated with the code below:

    H = es[(int)(T+ 25)]; // saturated vapor pressure, need 25 degrees for offset
    H *= U; Multiply relative humidity with saturated vapor pressure
    H /= 461.5; // divide by gas constant
    H /= (T+ 273.0); // divide by temperature in Kelvin

U is the relative humidity, T the temperature. Both values come directly from the humidity sensor sht11. 461,5 is the value of the gas constant.

Final words

There are other examples to calculate the absolute humidity, e.g. here. The method I use is simple. The sensors I have in my basement run constantly with this code for more than 2 years. The circuit I desgined with DS2408, atmega16  and sht11 runs stable and I have not noticed any problems until now.

Wednesday, April 10, 2013

Home Automation with Raspberry Pi

In this blog I want to describe how I reduce humidity in my basement without using an energy consuming air dryer. So the control system I implemented is blowing air into the basement, when the air outside of the house is dryer, then inside the basement.

The control system

I have installed two ventilators, five temperature sensors and one humidity sensor in my basement. I installed one humidity sensor outside of my house, as well. The humidity sensors measure the relative humidity, but the sensors translate the values into absolute humidity by doing some math (see here).  Below a picture of the controllers. The right circuit (red) is a controller for the ventillators. It contains two S202 S12 solid state relais to control both ventillators. The controller has a CAN interface to communicate with the left controller. The left controller (green) gathers the temperatures, humidity values from all sensors installed in the basement. It has a CAN interface and 1 wire bus interfaces. Also the circuit has a 24LC1025 eeprom to store the gathered data persistently. I used DS18B20 temperature sensors and connected them directly to the 1 wire bus coming from the controller. Both controllers use at90can128 mikrocontrollers.

The controllers have USB ports. When I wanted to get all measured values onto my laptop, I had to go down to the basement and dump out the eeprom values from USB. In the picture below you can see the humidity values of both humidity sensors from the eeprom dump. The data was gathered from mid 2011 to beginning 2013. In the middle of the curves you can see two lengthy interruptions. I switched off the control system, because I was renovating the basement.

This data gathering was pretty time consuming, so I prefered to have wireless and get the data from the living room or from the office. I needed something that works with wireless on the one hand, and retrieve the sensor data from the controller's eeprom on the other hand. So I came up with the raspberry pi.

Bringing in the raspberry pi

The raspberry has SPI and I2C ports, but no CAN. So I need something which translates SPI (or I2C) to CAN. There are chips available for doing this. Such as the MCP2515 . The chip is widely available, and runs with 3.3V which can be powered from the raspberry pi. Also I needed another chip which is a CAN transceiver. It connects the MCP2515 chip to the CAN bus. I used SN 65HVD230D, which is also widely available and which operates also on 3.3V.

This is the schematic:

I soldered this all together with silver wire and now it looks like this from top, see picture below. On the left upper corner in the picture below you find the raspberry pi. On the buttom right corner you find the components from the schematic above:

On the raspberry runs the wheezy operating system which can be downloaded here. In its forum I described what I have done to install the necessary software to get the raspberry working with SPI, so see CAN for home automation, in case you are interested. The latest wheezy versions have a straightforward way to get the WLAN USB stick running (WIFI config on the wheezy screen). I used the EW-7811UN from EDIMAX.

I created a couple of C functions to retrieve the eeprom data from the basement's controller via CAN. CAN for home automation forum entry describes what I have used to create the functions. The problem with CAN is, that it is rather slow. I cannot dump each time the eeeprom if I want to see the whole curve. This would take more than 5 minutes. So I installed mysql on raspberry and dump all eeprom values into a database. I implemented another script which queries the controller's eeprom once a day for new data and the data is stored into the database. 

The User Interface

As a last step, I needed a user interface. I decided to use a web server and implement a web page to display the data. After little research I found ligtttp and installed it on the raspberry. The decision came mainly because I found somewhere a website which helped me to get this installed easily. lighttpd has perl support, which is convenient for me, because I did not have to learn yet another language like php or python. Also perl has excellent support concerning date calculation needed to setup a time period. So I configured lighttpd to support CGI and wrote the dynamic webpage in perl. I created a login page and a curve display page. The curves I created with gnuplot. It needs to be installed on the raspberry and it works fine, some more information you can find here. Below you can see a snap shot from the webpage:

In the snap shot above you see menu boxes to setup the time period and three more curves. The upper curve above shows the relative humidity values during the time period mid 2011 to beginning 2013. The middle curve shows the temperatures values from the humidity sensors. And the lower curve shows the calculated absolute humidity from both curves. So each time the absolute humdity values of the one curve is above the other, the ventillators are turned on automatically for a short period of time (around 20 min.).


For now, I am pretty happy with my implementations. I can go into my office and take a look into my basement sensor values, and make my thoughts about them. But I was thinking, it would be good to have rain information added to the curves above. Rain information could give me a indications if the humidity values are correlated with the rainy weather. They probably are, but it would be nice if that graphic shows this as well. So it seems that there is more to come.