[Home] [Downloads] [Search] [Help/forum]

Gammon Forum

See www.mushclient.com/spam for dealing with forum spam. Please read the MUSHclient FAQ!

[Folder]  Entire forum
-> [Folder]  Electronics
. -> [Folder]  Microprocessors
. . -> [Subject]  Tiny web server for Arduino or similar
Home  |  Users  |  Search  |  FAQ
Username:
Register forum user name
Password:
Forgotten password?

Tiny web server for Arduino or similar

Postings by administrators only.

[Refresh] Refresh page


Posted by Nick Gammon   Australia  (21,322 posts)  [Biography] bio   Forum Administrator
Date Tue 21 Jul 2015 03:27 AM (UTC)

Amended on Tue 21 Jul 2015 05:27 AM (UTC) by Nick Gammon

Message
This post describes a "HTTP server" for the Arduino (or similar processors). It is specifically designed to work in very tight memory situations. Features are:


  • Minimal memory (RAM) requirements (about 256 bytes)
  • Small code size (around 2.5 KB)
  • No use of the String class, or dynamic memory allocation (to avoid heap fragmentation)
  • Incoming HTTP (client) requests decoded "on the fly" by a state machine
  • Handles all of:

    • Request type (eg. POST, GET, etc.)
    • Path (eg. /server/foo.htm)
    • GET parameters (eg. /server/foo.htm?device=clock&mode=UTC)
    • Header values (eg. Accept-Language: en, mi)
    • Cookies (as sent by the web browser)
    • POST data (ie. the contents of forms)

  • Doesn't care what Ethernet library you are using - you call your Ethernet library, and send a byte at at time to the HTTP library.
  • Compact and fast


Traditional web servers assume they have a reasonable amount of memory to hand (as they normally would, with modern PCs) and "batch up" headers, cookies, etc. in memory, and then call a client script (eg. a PHP script) once everything has been received. However this can mean large amounts of memory need to be devoted to strings (eg. cookies).

The "state machine" approach, on the other hand, detects things the moment they arrive, and sorts them into key/value pairs. For example:


/server/foo.htm?device=clock&mode=UTC


That URL consists of two GET parameters, namely:


device = clock
mode = UTC


By decoding on the fly, the library can call a user-supplied function (one you provide) so that the function would get the key (eg. "device") and the value (eg. "clock") and act on it immediately, if wanted.

Then that string data can be discarded, keeping memory usage very low.

Structure of HTTP client messages


To understand what the library is doing, you need to understand what the HyperText Transfer Protocol (HTTP) is doing.

Web clients (ie. web browsers such as Firefox, Chrome, Internet Explorer, etc) connect to a web server (such as Apache), usually on port 80, using TCP, and send an HTTP header, followed optionally by other text. It looks something like this:


POST /forum/showpost.php?id=12345&page=2 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:39.0)
Host: www.example.com
Cookie: foo=bar;
Accept-Language: en, mi

action=save&user_id=1234


Breaking down this into its component parts:



The HTTP server library processes each of those parts as it reaches them and then calls a function you supply to handle that part.

To do this you derive a custom class from the library class, overriding the virtual functions in it, so you can write your own handlers, like this:


// derive an instance of the HTTPserver class with custom handlers
class myServerClass : public HTTPserver
  {
  virtual void processPostType        (const char * key, const byte flags);
  virtual void processPathname        (const char * key, const byte flags);
  virtual void processHttpVersion     (const char * key, const byte flags);
  virtual void processGetArgument     (const char * key, const char * value, const byte flags);
  virtual void processHeaderArgument  (const char * key, const char * value, const byte flags);
  virtual void processCookie          (const char * key, const char * value, const byte flags);
  virtual void processPostArgument    (const char * key, const char * value, const byte flags);
  };  // end of myServClass

myServerClass myServer;   // instance of this class


You don't have to provide all those functions. Any you omit will silently ignore that part of the HTTP message. For example, you may not care what the HTTP version is.

As an example, the processPostType function will get the word POST, GET, or similar.

Since this signals the start of the transaction with the client, you could respond with a simple header, if you were planning to send debugging information. For example:


void myServerClass::processPostType (const char * key, const byte flags)
  {
  println(F("HTTP/1.1 200 OK"));
  println(F("Content-Type: text/plain"));
  println();   // end of headers

  print (F("GET/POST type: "));
  println (key);
  } // end of processPostType


Flags


The "flags" parameter is a bit mask which conveys extra information (you can ignore it). Possible bit masks are:

  • FLAG_NONE --> no problems
  • FLAG_KEY_BUFFER_OVERFLOW --> the key was truncated
  • FLAG_VALUE_BUFFER_OVERFLOW --> the value was truncated
  • FLAG_ENCODING_ERROR --> %xx encoding error


For example, if the flag FLAG_VALUE_BUFFER_OVERFLOW is set, then some of the value was truncated, because it would not fit into the holding buffer.

eg.


if (flags & FLAG_VALUE_BUFFER_OVERFLOW)
  {
  // the value was too large to fit
  }


POST type


This would normally be the word GET or POST (in uppercase). It is the start of the transaction and could be used to set up a response. Note that if you are planning to set cookies in response to POST data (which arrives later) you should not send the blank line which indicates the end of the headers.

Pathname


This is the "path" part of the URL, for example the text in bold and underlined:


http://somesite.com/forum/showpost.php?id=12938


In a normal web server this is the resource that we are "getting" - that is, the file that will be displayed, or the script that will be run.

In the example below we just echo back the pathname to the client:


void myServerClass::processPathname (const char * key, const byte flags)
  {
  print (F("Pathname: "));
  println (key);
  }  // end of processPathname


GET arguments


Arguments on the URL line (GET arguments) are provided to your callback function, one at a time, in the order in which they appear.

For example the text in bold and underlined:


http://somesite.com/forum/showpost.php?id=12938&action=delete


In that example we would get two GET arguments:


  • Name: id, Value: 12938
  • Name: action, Value: delete


Example code that just echoes the GET arguments back:


void myServerClass::processGetArgument (const char * key, const char * value, const byte flags)
  {
  print (F("Get argument: "));
  print (key);
  print (F(" = "));
  println (value);
  }  // end of processGetArgument


A more useful application would be to detect things like "light=on" or "fridge=off".

HTTP version


The final word on the first response line is the HTTP version. You can provide a handler if you want to discover what that is (eg. "HTTP/1.1").


void myServerClass::processHttpVersion (const char * key, const byte flags)
  {
  print (F("HTTP version: "));
  println (key);
  }  // end of processHttpVersion


Header lines


Following the first line will be a series of header lines, for example identifying the web browser name.

An example would be:


Host: 10.0.0.241 
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:39.0) Gecko/20100101 Firefox/39.0 
Accept: text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8 
Accept-Language: en-US,en;q=0.5 
Accept-Encoding: gzip, deflate 
DNT: 1 
Cookie: action=timetodance;
Connection: keep-alive 


For each header line (except the Cookie line, discussed below) your processHeaderArgument function will be called with a key/value pair. For example:


  • Key: Connection, Value: keep-alive
  • Key: Accept-Encoding, Value: gzip, deflate


Example code that just echoes the header lines back:


void myServerClass::processHeaderArgument (const char * key, const char * value, const byte flags)
  {
  print (F("Header argument: "));
  print (key);
  print (F(" = "));
  println (value);
  }  // end of processHeaderArgument


Cookies


Cookies are pieces of text stored at the client end (the web browser) and automatically sent to the server when you connect to it. The cookies are sent per URL, so for example my testing with 10.0.0.241 gives me only the cookies in the web browser identified by that URL.

The library identifies the "Cookie:" header, and breaks up the cookie header further into individual cookies (you do not get the entire Cookie header line).

For each cookie your processCookie function is called.


void myServerClass::processCookie (const char * key, const char * value, const byte flags)
  {
  print (F("Cookie: "));
  print (key);
  print (F(" = "));
  println (value);
  }  // end of processCookie


This gives you the opportunity to find information you had previously sent to the client (using setCookie) in order to main a "memory" from one page to the next.

Note that if you are planning to set cookies they have to be set as part of the header returned to the client, that is, before any HTML is sent.

POST arguments


Finally, if this is a POST request (that is, not a GET request) there may be POST arguments. Typically these are because of a form on the web page.

A simple form might look like this:


<form METHOD="post" ACTION="/activate_leds">
<br>LED: 3 <input type=checkbox name="led_3" value=1 >
<br>LED: 4 <input type=checkbox name="led_4" value=1 >
<br>LED: 5 <input type=checkbox name="led_5" value=1 >
<p><input Type=submit Name=Submit Value="Process">
</form>


On the web browser that would look like:



For each form input type (which is not zero) you will get a call to processPostArgument. So in this case if the user checked LED 3 and LED 5 you would get processPostArgument called for those two inputs. The name would be "led_3" (for example) and the value would be "1" in this case.



void myServerClass::processPostArgument (const char * key, const char * value, const byte flags)
  {
  if (memcmp (key, "led_", 4) == 0 && isdigit (key [4]) )
    {
    int which = atoi (&key [4]);
    if (which >= LOW_PIN && which <= HIGH_PIN)
      digitalWrite (which, HIGH);
    }
  }  // end of processPostArgument


Speed and memory usage


One of my test programs (that turns on LEDs based on a form) returns this information:


Binary sketch size: 13,662 bytes (of a 32,256 byte maximum)
...
Free memory = 1413 bytes
Time taken = 198 milliseconds. 


Thus we can see that there is still plenty of program memory (flash memory) over to do other things. There is also plenty of RAM over (1413 / 2048 bytes). Also the page was processed in about 1/5 of a second.

The state machine is fast because it is mainly working with single bytes (there are very few string comparisons) and at a particular moment, only one "state path" is taken, which minimizes CPU usage.

It is also compact because it only ever saves the current key/value pair, before calling the callback routine and then discarding it.


Putting it all together - minimal example


This minimal example calls the HTTP library, without overriding any functions, and thus is a minimal "do nothing" example. However it does return a response to the client.


// Tiny web server demo
// Author: Nick Gammon
// Date:  20 July 2015

#include <SPI.h>
#include <Ethernet.h>
#include <HTTPserver.h>

// Enter a MAC address and IP address for your controller below.
byte mac[] = {  0x90, 0xA2, 0xDA, 0x00, 0x2D, 0xA1 };

// The IP address will be dependent on your local network:
byte ip[] = { 10, 0, 0, 241 };

// the router's gateway address:
byte gateway[] = { 10, 0, 0, 1 };

// the subnet mask
byte subnet[] = { 255, 255, 255, 0 };

// Initialize the Ethernet server library
EthernetServer server(80);


// derive an instance of the HTTPserver class with custom handlers
class myServerClass : public HTTPserver
  {

  };  // end of myServerClass

myServerClass myServer;


void setup ()
  {
  // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip, gateway, subnet);
  server.begin();
  }  // end of setup

void loop ()
  {
  // listen for incoming clients
  EthernetClient client = server.available();
  if (!client)
    {
    // do other processing here
    return;
    }

  myServer.begin (&client);
  while (client.connected() && !myServer.done)
    {
    while (client.available () > 0 && !myServer.done)
      myServer.processIncomingByte (client.read ());

    // do other stuff here

    }  // end of while client connected

  client.println(F("HTTP/1.1 200 OK"));
  client.println(F("Content-Type: text/plain"));
  client.println();

  client.println(F("OK, done."));
  

  // give the web browser time to receive the data
  delay(1);
  // close the connection:
  client.stop();

  }  // end of loop


The important part is these lines which "feed" incoming data into the state machine:


  myServer.begin (&client);
  while (client.connected() && !myServer.done)
    {
    while (client.available () > 0 && !myServer.done)
      myServer.processIncomingByte (client.read ());

    // do other stuff here

    }  // end of while client connected


As the comment indicates, while there is no outstanding data, you can do other things, like take measurements. Once the incoming data is terminated (or the client disconnected) then you leave this processing loop.

The "myServer.begin" line passes to the library the current ethernet client address. This lets the callback functions send responses to the client by doing print or println function calls.

Provide callback functions


The example below overrides all of the callback functions, and then echoes back all the information to the client. If you run this you should see information about your web browser.


// Tiny web server demo
// Author: Nick Gammon
// Date:  20 July 2015

#include <SPI.h>
#include <Ethernet.h>
#include <HTTPserver.h>

// Enter a MAC address and IP address for your controller below.
byte mac[] = {  0x90, 0xA2, 0xDA, 0x00, 0x2D, 0xA1 };

// The IP address will be dependent on your local network:
byte ip[] = { 10, 0, 0, 241 };

// the router's gateway address:
byte gateway[] = { 10, 0, 0, 1 };

// the subnet mask
byte subnet[] = { 255, 255, 255, 0 };

// Initialize the Ethernet server library
EthernetServer server(80);

// derive an instance of the HTTPserver class with custom handlers
class myServerClass : public HTTPserver
  {
  virtual void processPostType        (const char * key, const byte flags);
  virtual void processPathname        (const char * key, const byte flags);
  virtual void processHttpVersion     (const char * key, const byte flags);
  virtual void processGetArgument     (const char * key, const char * value, const byte flags);
  virtual void processHeaderArgument  (const char * key, const char * value, const byte flags);
  virtual void processCookie          (const char * key, const char * value, const byte flags);
  virtual void processPostArgument    (const char * key, const char * value, const byte flags);
  };  // end of myServerClass

myServerClass myServer;

// -----------------------------------------------
//  User handlers
// -----------------------------------------------


void myServerClass::processPostType (const char * key, const byte flags)
  {
  println(F("HTTP/1.1 200 OK"));
  println(F("Content-Type: text/plain"));
  println();

  print (F("GET/POST type: "));
  println (key);
  } // end of processPostType

void myServerClass::processPathname (const char * key, const byte flags)
  {
  print (F("Pathname: "));
  println (key);
  }  // end of processPathname

void myServerClass::processHttpVersion (const char * key, const byte flags)
  {
  print (F("HTTP version: "));
  println (key);
  }  // end of processHttpVersion

void myServerClass::processGetArgument (const char * key, const char * value, const byte flags)
  {
  print (F("Get argument: "));
  print (key);
  print (F(" = "));
  println (value);
  }  // end of processGetArgument

void myServerClass::processHeaderArgument (const char * key, const char * value, const byte flags)
  {
  print (F("Header argument: "));
  print (key);
  print (F(" = "));
  println (value);
  }  // end of processHeaderArgument

void myServerClass::processCookie (const char * key, const char * value, const byte flags)
  {
  print (F("Cookie: "));
  print (key);
  print (F(" = "));
  println (value);
  }  // end of processCookie

void myServerClass::processPostArgument (const char * key, const char * value, const byte flags)
  {
  print (F("Post argument: "));
  print (key);
  print (F(" = "));
  println (value);
  }  // end of processPostArgument

// -----------------------------------------------
//  End of user handlers
// -----------------------------------------------

void setup ()
  {
  // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip, gateway, subnet);
  server.begin();
  }  // end of setup

void loop ()
  {
  // listen for incoming clients
  EthernetClient client = server.available();
  if (!client)
    return;

  myServer.begin (&client);
  while (client.connected() && !myServer.done)
    {
    while (client.available () > 0 && !myServer.done)
      myServer.processIncomingByte (client.read ());

    // do other stuff here

    }  // end of while client connected

  client.println(F("OK, done."));

  // give the web browser time to receive the data
  delay(1);
  // close the connection:
  client.stop();

  }  // end of loop


You should call the begin function each time you get a new incoming connection, as that resets the state machine back to the start, ready for a new HTTP session.

Utility functions


fixHTML

This converts the special HTML characters < > and & into &lt; &gt; and &amp;

This should be used if you are planning to echo back data which might contain those symbols. The function also sends the data to the client, to save having to allocate a temporary buffer to hold the converted string.

urlEncode

This "percent-encodes" a string so that things like spaces and other special characters get turned into hex, for example a space becomes %20. This also sends the converted string to the client.

setCookie

This emits a "Set-Cookie:" header, omitting characters (such as ";") that are not permitted in cookies.

Key/value sizes


There are two constants in the library controlling the maximum sizes of keys and values:


  static const size_t MAX_KEY_LENGTH = 40;     // maximum size for a key
  static const size_t MAX_VALUE_LENGTH = 100;  // maximum size for a value


You can make them larger if you expect larger keys (eg. header names) or larger values (eg. header values). However the larger you make them, the more RAM will be used.

General nature of HTTP response


The first things you send back to the web client are:


  • A response line (eg. "HTTP/1.1 200 OK")
  • One or more header lines, for example describing the nature of the response (text or HTML, etc.)
  • A blank line, which separates the headers from the rest of the response
  • The text or HTML data


For example, to send plain text:


HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Server: HTTPserver/1.0.0 (Arduino)

Some text here


To send HTML:


HTTP/1.1 200 OK
Content-Type: text/html
Connection: close
Server: HTTPserver/1.0.0 (Arduino)

<!DOCTYPE html>
<html>

<head>
<title>Arduino test</title>
</head>

<body>
<h1> My heading </h1>
<p>Some text
</body>
</html>


Use the F() macro


My examples show extensive use of the F() macro, for example:


  client.println(F("OK, done."));


You should use the F() macro for sending string constants, otherwise the string is copied into RAM. You will use up RAM very quickly if you have length amounts of HTML in your code, without using F().

Source code


Download the library from: https://github.com/nickgammon/HTTPserver

Install it into your "libraries" folder underneath your sketchbook folder.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.


6,612 views.

Postings by administrators only.

[Refresh] Refresh page

Go to topic:           Search the forum


[Go to top] top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.

[Home]


Written by Nick Gammon - 5K   profile for Nick Gammon on Stack Exchange, a network of free, community-driven Q&A sites   Marriage equality

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( https://gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Hosted at FutureQuest]