Building an OpenStreetMap tile server on Ubuntu 12.04 LTS

osm1
As a part of Wise Earth Technology programming challenge, I have build a local open street map tile server in my Ubuntu 12.04 LTS x86 machine. OpenStreetMap (OSM) is a collaborative project to create a free editable map of the world. The OSM tile server stack is a collection of programs and libraries that work together to create a tile server. As so often with OpenStreetMap, there are many ways to achieve this goal and nearly all of the components have alternatives that have various specific advantages and disadvantages. This tutorial describes the most standard version that is also used on the main OpenStreetMap.org tile server. After installation, you should have your own working tileserver with the standard OSM Mapnik stylesheet, into which you can import an extract of the OSM data for rendering.

These are the main components of OpenStreetMap:

  1. Mod_tile is a system to serve raster tiles. It provides a dynamic combination of efficient caching and on the fly rendering. Due to its dynamic rendering, only a small fraction of overall tiles need to be kept on disk, reducing the resources required.
  2. Renderd is a daemon process to serve tile  requests in a queue.
  3. Mapnik is a tool kit for rendering maps. It is designed to be fast and is suitable for tile generation on high-end servers.
  4. osm2pgsql is used to convert OpenStreetMap data to postGIS-enabled PostgreSQL databases.

I have made this tutorial simple to reduce hassles for those who want to quickly setup a local tile server (it took me 2 days for the same)

Installation

The following commands need to be entered into the terminal to set things up:

In case you do not have add-apt-repository installed, add it with:

sudo apt-get install python-software-properties

Add the repository containing the packages and update you local package list too:

sudo add-apt-repository ppa:kakrueger/openstreetmap && sudo apt-get update

Install the package libapache2-mod-tile and its dependencies. During configuration, it will ask you few questions. To make sure the automatic setup scripts work, you should keep the defaults. However, in the question about permissions for users of the database, you want to add your own username after the user www-data separated by a space to be able import data under your user.

Note: You are going to use apache webserver to run the OSM. The apache is run as www-data user. Make sure what ever changes you make in db or file make it accessible to www-data user. Otherwise you are going to get a lot of errors while rendering the map using a web browser.

sudo apt-get install libapache2-mod-tile

This Apache module receives map tiles requests and satisfies them from tiles stored on disk, or queries a renderd instance to create them. This package will install mapnik, mod_tile, apache2, osm2psql and renderd packages. Alongside files to generate coastlines and ocean areas for small scale maps will be downloaded since it is faster than reading the entire database for this information. Downloading the coastline data requires about 400Mb of download.

Loading data into our server

Get the latest OSM data

Retrieve a piece of OpenStreetMap data from http://planet.openstreetmap.org/. Since the whole planet is at least 18GB when compressed, there are links to smaller country or state sized extracts on that page. Since it is smaller and faster to interact with, the PBF file format is preferable when available. In this case we’ll download the country extract of India from http://download.geofabrik.de/

wget http://download.geofabrik.de/asia/india-latest.osm.pbf

Import map data

Import the data into your postgresql database with osm2pgsql. There are a number of different parameters one can use with osm2pgsql that depend on your available hardware and the size of the data extract you want to import. The most likely ones you will need to set are “-C”, “–slim” and “–number-processes”. -C specifies the number of Mb osm2pgsql will use to cache data. So this depends on the amount of RAM you have available. –slim keeps the complete osm data on disk, necessary for updates and low memory environments. –number-processes specifies the number of parallel processes that are used for some parts of the import process. The optimal value mostly depends on the speed of your disk system and the available processor cores. Make sure that you run it as a www-data user in-order to make the same user as the owner of the tables which will be created during the map import.
If you have a multicore process (I have a server with 8 core xeon processor and 16 GB RAM):

sudo -u www-data osm2pgsql -C 16000 --slim --number-processes 8 india-latest.osm.pbf

If you don’t have plenty (my desktop has only 2GB) RAM memory and an old CPU, I would suggest you to run this command

sudo -u www-data osm2pgsql -C 2048 --slim --number-processes 4 india-latest.osm.pbf --cache-strategy sparse

Make sure that you don’t get any errors while running the above given commands. (Sample output -> http://paste.pound-python.org/show/34623/)

In-order to check if the above given commands works properly check the owner of the tables of gis database.

sudo -u postgres -i
postgres@username:~$ psql -d gis
gis=# \d

               List of relations
 Schema |        Name        | Type  |  Owner   
--------+--------------------+-------+----------
 public | geography_columns  | view  | postgres
 public | geometry_columns   | table | www-data
 public | planet_osm_line    | table | www-data
 public | planet_osm_nodes   | table | www-data
 public | planet_osm_point   | table | www-data
 public | planet_osm_polygon | table | www-data
 public | planet_osm_rels    | table | www-data
 public | planet_osm_roads   | table | www-data
 public | planet_osm_ways    | table | www-data
 public | spatial_ref_sys    | table | www-data
(10 rows)

If it doesn’t make the owner of these tables as www-data you can manually do it by two ways:

 sudo /usr/bin/install-postgis-osm-user.sh gis www-data

or

sudo -u postgres -i
postgres@username:~$ psql -d gis
gis=# ALTER TABLE table_name OWNER TO www-data;

mod_tile is designed to always serve up-to-date tiles (see updating below). As it is generally not feasible to re-render all changed tiles at the time of change, mod_tile initiates re-rendering of outdated tiles at the time of serving. As such mod_tile needs to know when the planet was imported. This is done by changing the timestamp on the file planet-import-complete

sudo -u www-data touch /var/lib/mod_tile/planet-import-complete

Make sure that the below given directories and files are owned by www-data user:

/var/www/osm/ <- directory and all files in it
/var/run/renderd/renderd.sock
/var/lib/mod_tile <- directory and all files in it

Finally, you need to restart the rendering daemon after which all should be ready.

sudo /etc/init.d/renderd restart

You can check if it is working by going to http://Your_Server_Address/osm/0/0/0.png. If it works, you will get a 256x256px image. If it doesn’t, lets debug this issue.

Troubleshooting

Stop the rendering daemon

sudo /etc/init.d/renderd stop

Change the localhost in the configuration files to your servername
File #1 /var/www/osm/slippymap.html

var newLayer = new OpenLayers.Layer.OSM("Local TileServer", "http://localhost/osm/${z}/${x}/${y}.png", {numZoomLevels: 19});

to

var newLayer = new OpenLayers.Layer.OSM("Local TileServer", "http://Your_Server_address/osm/${z}/${x}/${y}.png", {numZoomLevels: 19});

in my case it was

var newLayer = new OpenLayers.Layer.OSM("Local TileServer", "http://10.30.10.30/osm/${z}/${x}/${y}.png", {numZoomLevels: 19});

In /etc/apache2/httpd.conf add this line:

ServerName Your_Server_Address

In /etc/apache2/sites-enabled/tileserver_site, change this line from

ServerName localhost

to

ServerName Your_Server_Address

In /etc/renderd.conf, change this line from

;HOST localhost

to

HOST Your_Server_Address

Inside /etc/mapnik-osm-data/inc/, we need to care only about two files settings.xml.inc and datasource-settings.xml.inc you must be very carefully when commenting.
In datasource-settings.inc, the lines with host and port should be commented as:

<Parameter name="type">postgis</Parameter>
<Parameter name="password">gis</Parameter>
<!-- Parameter name="host">localhost</Parameter -->
<!-- Parameter name="port">%(port)s</Parameter -->
<Parameter name="user">www-data</Parameter>
<Parameter name="dbname">gis</Parameter>
<!-- this should be 'false' if you are manually providing the 'extent' -->
<Parameter name="estimate_extent">false</Parameter>
<!-- manually provided extent in epsg 900913 for whole globe -->
<!-- providing this speeds up Mapnik database queries -->
<Parameter name="extent">-20037508,-19929239,20037508,19929239</Parameter>

A working settings.xml.inc looks like this:

<!ENTITY symbols "symbols">
<!ENTITY osm2pgsql_projection "&srs900913;">
<!ENTITY dwithin_900913 "0.1">
<!ENTITY dwithin_4326 "0.00001">
<!ENTITY dwithin_node_way "&dwithin_900913;">
<!ENTITY world_boundaries "/usr/share/mapnik-osm-data/world_boundaries">
<!ENTITY prefix "planet_osm">

Lets now start the rendering in debugging mode:

sudo -u www-data renderd -f -c /etc/renderd.conf

If it doesn’t displays any errors, then your OSM will be successfully render in http://Your_Server_Address/osm/slippymap.html

You can use render_list to pre-render tiles in another terminal:

sudo render_list -a -s /var/run/renderd/renderd.sock

When you run the above command you could will get this output in renderd terminal:

renderd[1723]: DEBUG: Got command RenderBulk fd(12) xml(default), z(0), x(0), y(0)
renderd[1723]: DEBUG: DONE TILE default 0 0-0 0-0
renderd[1723]: DEBUG: DONE TILE default 0 0-0 0-0 in 1.904 seconds

Tuning your System

A tile server can put a lot of load on hard- and software. The default settings may therefore not be appropriate and a significant improvement can potentially be achieved through tuning various parameters.

Tuning postgresql

The default configuration for PostgreSQL 9.1 needs to be tuned for the amount of data you are about to add to it. Edit the file /etc/postgresql/9.1/main/postgresql.conf and make the following changes:

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

These changes require a kernel configuration change, which needs to be applied every time that the computer is rebooted. As root, edit /etc/sysctl.conf and add these lines near the top after the other kernel definitions:

# Increase kernel shared memory segments - needed for large databases
kernel.shmmax=268435456

Reboot your computer. Run this:

sudo sysctl kernel.shmmax

and verify that it displays as 268435456.

NOTE: The above changes improved the rendering performance a lot in my machine with quad core processor and 2GB RAM.

After adding few lines  in /var/www/osm/slippymap.html file, I got something like this:

snapshot43

Advertisements

36 comments

  1. Great ! this is very helpful , but does this work as offline map tile server ? I couldn’t get it to work when I am not connected to the internet even in the “local tiles” setup . I appreciate your answer .
    Thanks

    1. Yes, it will work as an offline map. As I have mentioned in this post, you need to download the map file of the region which you want to render. The tiles will be generated only if you sent a request to your tileserver.

  2. in List of relations i am getting this

    Schema | Name | Type | Owner
    ——–+——————–+——-+———-
    public | geography_columns | view | postgres
    public | geometry_columns | table | gis
    public | planet_osm_line | table | www-data
    public | planet_osm_nodes | table | www-data
    public | planet_osm_point | table | www-data
    public | planet_osm_polygon | table | www-data
    public | planet_osm_rels | table | www-data
    public | planet_osm_roads | table | www-data
    public | planet_osm_ways | table | www-data
    public | spatial_ref_sys | table | gis

    is there any problem with it to proceed further?

  3. I’m having a real difficult time setting this up. I initially followed the tutorial on switch2osm.org but then found this tutorial and changed the owner on pretty much everything (including /home/(myuser)/src) to www-data – I’m still not able to get this to work. I changed the owner on all but the geography_columns table to www-data in postgres as well. I was able to get renderd to start under user www-data with no errors and it still just shows blank at localhost/osm_tiles/0/0/0.png (under the previous switch2osm tutorial I set it up to use /osm_tiles/ in /usr/local/etc/rendered.conf). I changed /etc/apache2/sites-enabled/000-default.conf to match that of the switch2osm.org instructions, and put LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so into /etc/apache2/mods-enabled/mod_tile.load. Are there any real good log files I can use to find out what may be causing the lack of transfer? I did a tail -f on the apache2 access & error logs and they are useless here.

    Also, just out of the blue, instead of using a virtual host file for the config for my local webserver, wouldn’t it make more sense to put it under the main apache2.conf file?

  4. i cant “see” tileserver requests from another ip..

    example

    ssh inside 10.10.10.200 > wget 10.10.10.230/osm/10/532/356.png –> OK!

    but using slappymap.html from 10.10.10.200/slappymap.html , i got pink tiles..

    if i right click “view image” i “see” the correct tile..

    1. Hi, so its the slippymap.html which display the map on your favorite browser. You could configure the coordinates of the certain street in this file.
      Yes, your coordinates should be added here:
      maxExtent: new OpenLayers.Bounds(-20037508.34,-20037508.34,20037508.34,20037508.34),
      Hope, it helps!

  5. I followed every step and got the server running. I also have no problem getting pre-rendered tiles from the server, but I just can’t get it to render my tiles on demand. The first time I try to get the tile the apache error.log says:

    [info] tile_storage_hook: handler(tile_server), uri(/osm/14/8590/5338.png)
    [info] Requesting style(default) z(14) x(8590) y(5338) from renderer with priority 5
    [warn] request_tile: Failed to read response from rendering socket No such file or directory

    Each try after that says:

    [info] tile_storage_hook: handler(tile_server), uri(/osm/14/8590/5338.png)
    [warn] socket connect failed for: /var/run/renderd/renderd.sock with reason: Conncetion refused
    [notice] Failed to connect to renderer

    I’m guessing the second one means that renderd was stopped after the first one. But any idea where the first one comes from?

  6. For some reason I do not see the postgresql directories when I try your walkthrough. postgresql startup scripts are there but I do not have the directory, in fact the only dir I see is this: /etc/postgresql-common/pg_upgradecluster.d/ …. Any idea why ?

  7. Seshagiri

    Easy to follow install instructions, thank you.

    I’ve had an issue as well accessing the maps while the server is operating disconnected from the internet. During my initial build I installed only a small North American State and while connected to the internet your build worked very well where only the area that I had installed would be displayed

    When I attempted to run the same build connected to our internal network but disconnected from the internet the maps would no longer load. When I re-established a connection to the internet the maps came on line. It’s as if the maps server needs access to the internet to continue to render data.

    Is there a loop back address or potentially a host file setting that needs to be made?

    Thanks in advance for any assistance you can provide in clearing this issue up

  8. This my table:

    List of relations
    Schema | Name | Type | Owner
    ——–+——————–+——-+———-
    public | geography_columns | view | postgres
    public | geometry_columns | table | gis
    public | planet_osm_line | table | www-data
    public | planet_osm_nodes | table | www-data
    public | planet_osm_point | table | www-data
    public | planet_osm_polygon | table | www-data
    public | planet_osm_rels | table | www-data
    public | planet_osm_roads | table | www-data
    public | planet_osm_ways | table | www-data
    public | spatial_ref_sys | table | gis
    (10 rows)

    When I am trying to change it using the method you mentioned is not working:

    When i use below command:

    sudo /usr/bin/install-postgis-osm-user.sh gis www-data

    output is:

    createuser: creation of new role failed: ERROR: role “www-data” already exists
    Granting rights to user ‘www-data’
    GRANT
    GRANT
    GRANT
    GRANT
    GRANT
    GRANT
    GRANT
    GRANT
    GRANT
    GRANT

    when I use another method then it’s also not working. The table after using second method is same as the above. Please reply me as soon as possible why it’s not working!.

    Thanks in advance!!

  9. Hi

    In the Troubleshooting area you mentioned mapnik-osm-data directory but on the installed system there wasn’t any mapnik-osm-data directory but mapnik-osm-carto-data directory.
    /usr/share/mapnik-osm-carto-data
    /etc/mapnik-osm-carto-data/

    And there wasn’t any inc directory in the /etc/mapnik-osm-carto-data/ as well
    I created inc directory with two files settings.xml.inc and datasource-settings.xml.inc as described
    and restarted the system but without any result

    I renamed mapnik-osm-carto-data to mapnik-osm-data (2 instances) without any result
    The link http://192.168.109.39/osm/0/0/0.png is showing me png world picture
    The link http://192.168.109.39/osm/slippymap.html showing me nothing

    Also I don’t have /etc/apache2/httpd.conf file but /etc/apache2/apache2.conf

    OS Ubuntu 14.04.2 LTS

    Will be grateful for the help

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s