Linux

Building an OpenStreetMap tile server on Ubuntu 12.04 LTS


osm1
As a part of Wise Earth Technology programming challenge, I have build a local open street map tile server in my Ubuntu 12.04 LTS x86 machine. OpenStreetMap (OSM) is a collaborative project to create a free editable map of the world. The OSM tile server stack is a collection of programs and libraries that work together to create a tile server. As so often with OpenStreetMap, there are many ways to achieve this goal and nearly all of the components have alternatives that have various specific advantages and disadvantages. This tutorial describes the most standard version that is also used on the main OpenStreetMap.org tile server. After installation, you should have your own working tileserver with the standard OSM Mapnik stylesheet, into which you can import an extract of the OSM data for rendering.

These are the main components of OpenStreetMap:

  1. Mod_tile is a system to serve raster tiles. It provides a dynamic combination of efficient caching and on the fly rendering. Due to its dynamic rendering, only a small fraction of overall tiles need to be kept on disk, reducing the resources required.
  2. Renderd is a daemon process to serve tile  requests in a queue.
  3. Mapnik is a tool kit for rendering maps. It is designed to be fast and is suitable for tile generation on high-end servers.
  4. osm2pgsql is used to convert OpenStreetMap data to postGIS-enabled PostgreSQL databases.

I have made this tutorial simple to reduce hassles for those who want to quickly setup a local tile server (it took me 2 days for the same)

Installation

The following commands need to be entered into the terminal to set things up:

In case you do not have add-apt-repository installed, add it with:

sudo apt-get install python-software-properties

Add the repository containing the packages and update you local package list too:

sudo add-apt-repository ppa:kakrueger/openstreetmap && sudo apt-get update

Install the package libapache2-mod-tile and its dependencies. During configuration, it will ask you few questions. To make sure the automatic setup scripts work, you should keep the defaults. However, in the question about permissions for users of the database, you want to add your own username after the user www-data separated by a space to be able import data under your user.

Note: You are going to use apache webserver to run the OSM. The apache is run as www-data user. Make sure what ever changes you make in db or file make it accessible to www-data user. Otherwise you are going to get a lot of errors while rendering the map using a web browser.

sudo apt-get install libapache2-mod-tile

This Apache module receives map tiles requests and satisfies them from tiles stored on disk, or queries a renderd instance to create them. This package will install mapnik, mod_tile, apache2, osm2psql and renderd packages. Alongside files to generate coastlines and ocean areas for small scale maps will be downloaded since it is faster than reading the entire database for this information. Downloading the coastline data requires about 400Mb of download.

Loading data into our server

Get the latest OSM data

Retrieve a piece of OpenStreetMap data from http://planet.openstreetmap.org/. Since the whole planet is at least 18GB when compressed, there are links to smaller country or state sized extracts on that page. Since it is smaller and faster to interact with, the PBF file format is preferable when available. In this case we’ll download the country extract of India from http://download.geofabrik.de/

wget http://download.geofabrik.de/asia/india-latest.osm.pbf

Import map data

Import the data into your postgresql database with osm2pgsql. There are a number of different parameters one can use with osm2pgsql that depend on your available hardware and the size of the data extract you want to import. The most likely ones you will need to set are “-C”, “–slim” and “–number-processes”. -C specifies the number of Mb osm2pgsql will use to cache data. So this depends on the amount of RAM you have available. –slim keeps the complete osm data on disk, necessary for updates and low memory environments. –number-processes specifies the number of parallel processes that are used for some parts of the import process. The optimal value mostly depends on the speed of your disk system and the available processor cores. Make sure that you run it as a www-data user in-order to make the same user as the owner of the tables which will be created during the map import.
If you have a multicore process (I have a server with 8 core xeon processor and 16 GB RAM):

sudo -u www-data osm2pgsql -C 16000 --slim --number-processes 8 india-latest.osm.pbf

If you don’t have plenty (my desktop has only 2GB) RAM memory and an old CPU, I would suggest you to run this command

sudo -u www-data osm2pgsql -C 2048 --slim --number-processes 4 india-latest.osm.pbf --cache-strategy sparse

Make sure that you don’t get any errors while running the above given commands. (Sample output -> http://paste.pound-python.org/show/34623/)

In-order to check if the above given commands works properly check the owner of the tables of gis database.

sudo -u postgres -i
postgres@username:~$ psql -d gis
gis=# \d

               List of relations
 Schema |        Name        | Type  |  Owner   
--------+--------------------+-------+----------
 public | geography_columns  | view  | postgres
 public | geometry_columns   | table | www-data
 public | planet_osm_line    | table | www-data
 public | planet_osm_nodes   | table | www-data
 public | planet_osm_point   | table | www-data
 public | planet_osm_polygon | table | www-data
 public | planet_osm_rels    | table | www-data
 public | planet_osm_roads   | table | www-data
 public | planet_osm_ways    | table | www-data
 public | spatial_ref_sys    | table | www-data
(10 rows)

If it doesn’t make the owner of these tables as www-data you can manually do it by two ways:

 sudo /usr/bin/install-postgis-osm-user.sh gis www-data

or

sudo -u postgres -i
postgres@username:~$ psql -d gis
gis=# ALTER TABLE table_name OWNER TO www-data;

mod_tile is designed to always serve up-to-date tiles (see updating below). As it is generally not feasible to re-render all changed tiles at the time of change, mod_tile initiates re-rendering of outdated tiles at the time of serving. As such mod_tile needs to know when the planet was imported. This is done by changing the timestamp on the file planet-import-complete

sudo -u www-data touch /var/lib/mod_tile/planet-import-complete

Make sure that the below given directories and files are owned by www-data user:

/var/www/osm/ <- directory and all files in it
/var/run/renderd/renderd.sock
/var/lib/mod_tile <- directory and all files in it

Finally, you need to restart the rendering daemon after which all should be ready.

sudo /etc/init.d/renderd restart

You can check if it is working by going to http://Your_Server_Address/osm/0/0/0.png. If it works, you will get a 256x256px image. If it doesn’t, lets debug this issue.

Troubleshooting

Stop the rendering daemon

sudo /etc/init.d/renderd stop

Change the localhost in the configuration files to your servername
File #1 /var/www/osm/slippymap.html

var newLayer = new OpenLayers.Layer.OSM("Local TileServer", "http://localhost/osm/${z}/${x}/${y}.png", {numZoomLevels: 19});

to

var newLayer = new OpenLayers.Layer.OSM("Local TileServer", "http://Your_Server_address/osm/${z}/${x}/${y}.png", {numZoomLevels: 19});

in my case it was

var newLayer = new OpenLayers.Layer.OSM("Local TileServer", "http://10.30.10.30/osm/${z}/${x}/${y}.png", {numZoomLevels: 19});

In /etc/apache2/httpd.conf add this line:

ServerName Your_Server_Address

In /etc/apache2/sites-enabled/tileserver_site, change this line from

ServerName localhost

to

ServerName Your_Server_Address

In /etc/renderd.conf, change this line from

;HOST localhost

to

HOST Your_Server_Address

Inside /etc/mapnik-osm-data/inc/, we need to care only about two files settings.xml.inc and datasource-settings.xml.inc you must be very carefully when commenting.
In datasource-settings.inc, the lines with host and port should be commented as:

<Parameter name="type">postgis</Parameter>
<Parameter name="password">gis</Parameter>
<!-- Parameter name="host">localhost</Parameter -->
<!-- Parameter name="port">%(port)s</Parameter -->
<Parameter name="user">www-data</Parameter>
<Parameter name="dbname">gis</Parameter>
<!-- this should be 'false' if you are manually providing the 'extent' -->
<Parameter name="estimate_extent">false</Parameter>
<!-- manually provided extent in epsg 900913 for whole globe -->
<!-- providing this speeds up Mapnik database queries -->
<Parameter name="extent">-20037508,-19929239,20037508,19929239</Parameter>

A working settings.xml.inc looks like this:

<!ENTITY symbols "symbols">
<!ENTITY osm2pgsql_projection "&srs900913;">
<!ENTITY dwithin_900913 "0.1">
<!ENTITY dwithin_4326 "0.00001">
<!ENTITY dwithin_node_way "&dwithin_900913;">
<!ENTITY world_boundaries "/usr/share/mapnik-osm-data/world_boundaries">
<!ENTITY prefix "planet_osm">

Lets now start the rendering in debugging mode:

sudo -u www-data renderd -f -c /etc/renderd.conf

If it doesn’t displays any errors, then your OSM will be successfully render in http://Your_Server_Address/osm/slippymap.html

You can use render_list to pre-render tiles in another terminal:

sudo render_list -a -s /var/run/renderd/renderd.sock

When you run the above command you could will get this output in renderd terminal:

renderd[1723]: DEBUG: Got command RenderBulk fd(12) xml(default), z(0), x(0), y(0)
renderd[1723]: DEBUG: DONE TILE default 0 0-0 0-0
renderd[1723]: DEBUG: DONE TILE default 0 0-0 0-0 in 1.904 seconds

Tuning your System

A tile server can put a lot of load on hard- and software. The default settings may therefore not be appropriate and a significant improvement can potentially be achieved through tuning various parameters.

Tuning postgresql

The default configuration for PostgreSQL 9.1 needs to be tuned for the amount of data you are about to add to it. Edit the file /etc/postgresql/9.1/main/postgresql.conf and make the following changes:

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

These changes require a kernel configuration change, which needs to be applied every time that the computer is rebooted. As root, edit /etc/sysctl.conf and add these lines near the top after the other kernel definitions:

# Increase kernel shared memory segments - needed for large databases
kernel.shmmax=268435456

Reboot your computer. Run this:

sudo sysctl kernel.shmmax

and verify that it displays as 268435456.

NOTE: The above changes improved the rendering performance a lot in my machine with quad core processor and 2GB RAM.

After adding few lines  in /var/www/osm/slippymap.html file, I got something like this:

snapshot43

Setting up your own DNS server on Ubuntu


Setting up a master server

This blog was written in a hurry and I apologize for typos and grammatical errors. Firstly you have to install a package called bind9, which is most widely used DNS software on the internet

$ sudo apt-get install bind9

Configure the DNS to cache requests and forward unknown requests to other DNS servers

$ sudo vim /etc/bind/named.conf.local

Add this line to the same

zone "am" {
     type master;
     file "/etc/bind/db.am";
     notify no;
};

Now use an existing zone file as a template:

sudo cp /etc/bind/db.local /etc/bind/db.am

Make it to look something like this:

;
; BIND data file for local loopback interface
;
$TTL    604800
@       IN      SOA     ns.am. root.am. (
                             10         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL

@       IN      NS      ns.am.
ns      IN      A       10.30.10.90
@       IN      MX      9 mail.am.
;@       IN      A       127.0.0.1
;@       IN      AAAA    ::1
cs      IN      A       10.30.11.55
wna     IN      A       10.30.9.46

After saving the am.db file, restart the bind server

$ sudo /etc/init.d/bind9 restart

Now, lets move to client machine.
Edit /etc/resolv.conf file to change the nameserver IP to that of the master DNS server.

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.30.11.55

Restart the network

$ sudo /etc/init.d/networking restart

That’s it! Now we will be able to ping to cs.am and wna.am servers.

(We used different machines when we did it in lab and when we were writing this blog – hence IP differs!)

assignment

Setting up a secondary server

In case if the primary server fails, we would need a slave server as back up.
Edit /etc/bind/named.conf.local in the primary master as follows:

zone "am" {
             type master;
             file "/etc/bind/db.am";
             allow-transfer { 10.30.10.90; };
        };

We gave cs.am as the secondary server

In the cs.am, we need to install bind9 package and then edit the /etc/bind/named.conf.local file in-order to make it as a secondary server.

zone "am"{
               type slave;  
               file "/var/cache/bind/db.am";  
               masters { 10.30.11.55; }; 
      };

You may need to restart the bind server to take the effect in secondary server. In-order to check the secondary server, you can shut down the bind9 sevrer and ping to cs.am for any echo replies.

In the client machines, we need to make changes in /etc/resolv of the client machines as follows to apply the effects of secondary server that we have just added now:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.30.11.55
nameserver 10.30.10.90

In-order to add an IP address of a stable DNS server to a named.conf.options file. This IP address is used when a local DNS server do not know to answer a name resolution query.
You can add these lines to /etc/bind/named.local.options

forwarders {
              8.8.8.8;
        };

Reference:
[1] https://help.ubuntu.com/community/BIND9ServerHowto

Sample vimrc and gdbinit file


As I don’t keep any backup of vimrc and gdbinit files before the installation of new version of Ubuntu/Kubuntu, I always have to search in the internet for a sample files. So I thought from next time onwards I should make my life much more simpler i.e I should have it in a safe place => my blog! This is the vimrc file which I always use:

" vim configuration file

" the following line sets tabs to 4 spaces.
set ts=4

" the following line adds line number to the text.
set number

" the following line activates the syntax highlighting.
syntax on

" the following line says to vim to highlight the searched words.
set hlsearch

" the following line says to vim to execute incremental search.
set incsearch

" the following line enables the use of the mouse.
set mouse=a

" make searches case insensitive
set ignorecase

" i don't use autoindent, but here's how to configure it:
set autoindent

" To automatically indent braces
set cindent

" show the editing mode on the last line
set showmode

" To indent four spaces (instead of one tab of 8 spaces)
set shiftwidth=4 

This is the gdbinit file

set print demangle on
set demangle-style auto
set verbose on 
set complaints 1000 
set language c++ 
set print vtbl on 
set print object on 
set print sym on 
set print pretty on 
disp /i $rip
define pst 
set $pos=&$arg0 
set $strlen = {byte}$pos 
print {char}&$arg0.st@($strlen+1) 
 end
document pst 
 Print out a Pascal string 
 end

Installing TinyOS on Ubuntu


Distribution used: Precise Pangolin (12.04 x64)

TinyOS doesn’t have repository for precise. Hence we will use Lucid’s repository in-order to download and install TinyOS. Add this line to /etc/apt/sources.lst

deb http://tinyos.stanford.edu/tinyos/dists/ubuntu lucid main

Update the apt-get repository using this command:

$ sudo apt-get update

Install TinyOS through apt-get

$ sudo apt-get install tinyos-2.1.1

In-order to execute the compile the nesc codes you may need to change the ownership of TinyOS root directory to your user.

$ sudo chown vayya:vayya -R /opt/tinyos-2.1.1/

Install python development package (headers) (*optional only for TOSSIM)

$ sudo apt-get install python-dev

Add the following lines to .bashrc in your home directory

export TOSDIR=$TOSROOT/tos
export CLASSPATH=$TOSROOT/support/sdk/java/tinyos.jar:.$CLASSPATH
export MAKERULES=$TOSROOT/support/make/Makerules
export PATH=/opt/msp430/bin:$PATH
source /opt/tinyos-2.1.1/tinyos.sh

Install the Java tools by:

$ sudo tos-install-jni
Installing 64-bit Java JNI code in /usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64 ....
done

Installing Java docs
Go to /opt/tinyos-2.1.1/support/sdk/java

$ make
$ make install
$ make javadoc 

Compiling and installing my first TinyOS program

Go to opt/tinyos-2.1.1/apps/Blink and compile the code using command make.

$ make micaz
mkdir -p build/micaz
compiling BlinkAppC to a micaz binary
ncc -o build/micaz/main.exe -Os -fnesc-separator=__ -Wall -Wshadow -Wnesc-all -target=micaz -fnesc-cfile=build/micaz/app.c -board=micasb -DDEFINED_TOS_AM_GROUP=0x22 --param max-inline-insns-single=100000 -DIDENT_APPNAME=\"BlinkAppC\" -DIDENT_USERNAME=\"seshagiri\" -DIDENT_HOSTNAME=\"ATI\" -DIDENT_USERHASH=0x9ec8e428L -DIDENT_TIMESTAMP=0x50b486edL -DIDENT_UIDHASH=0xef33ececL -fnesc-dump=wiring -fnesc-dump='interfaces(!abstract())' -fnesc-dump='referenced(interfacedefs, components)' -fnesc-dumpfile=build/micaz/wiring-check.xml BlinkAppC.nc -lm
compiled BlinkAppC to build/micaz/main.exe
2048 bytes in ROM
51 bytes in RAM
avr-objcopy --output-target=srec build/micaz/main.exe build/micaz/main.srec
avr-objcopy --output-target=ihex build/micaz/main.exe build/micaz/main.ihex
writing TOS image

Where Micaz is a widely used wireless mote (see the image below)

Done! Let the coding begins now 🙂

Installing MongoDB in Ubuntu 12.04 and Securing


Installing MongoDB

What is MongoDB?

For the n00bs, you can get an introduction about MongoDB from this link.

Configure Package Management System (APT)

The Ubuntu package management tool (i.e. dpkg and apt) ensure package consistency and authenticity by requiring that distributors sign packages with GPG keys. Issue the following command to import the 10gen public GPG Key:

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

Create a /etc/apt/sources.list.d/10gen.list file

$ sudo touch /etc/apt/sources.list.d/10gen.list

and include the following line for the 10gen repository.

echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" >> /etc/apt/sources.list.d/10gen.list

Now reload your repository:

$ sudo apt-get update

Install Packages

$ sudo apt-get install mongodb-10gen

Configure MongoDB

Few very initial things to remember are:

  • Remove IP Binding from all to just the IP (private or localhost), you expect to get Connection Request
    # Only accept local connections
    bind_ip = 127.0.0.1
  • Change the default Port Bindings
    # Connecting port of mongodb
    port = 7171
  • Authentication for all users
    Creating MongoDB Users
    To adding a new user is fairly straight forward …
    Read/Write User

    $ mongo
    > use admin
    > db.addUser("admin", "@mm@123@m@+@rAn!")

    This creates a read/write user for the database admin (you can choose any username you wish.)
    Restart the mongo with --auth enabled.

    sudo mongod --auth --dbpath /var/lib/mongodb/ 

    User Authentication in MongoDB

    $ mongo localhost:7171/admin --username "admin" -password "@mm@123@m@+@rAn!"
  • Give only required permissions (like no update/delete permissions to select query users)

    Read Only User

    > db.addUser("monkey", "tinkerbelle", true)

    The “true” parameter there makes the user read only (great for parts of the application code that you want to make sure never accidentally preform a write operation.)

  • Removing all the default databases
    > show dbs
    admin 0.203125GB
    local (empty)
    test (empty)

    Databases test and local are defualt one’s. So we can drop all default databases.

    > use test;
    switched to db test
    > db.dropDatabase();
    { "dropped" : "test", "ok" : 1 }
    > use local;
    switched to db local
    > db.dropDatabase()
    { "dropped" : "local", "ok" : 1 }
  • Setup ssh keys for required master-slave connection, removing involvement of passwords
    In-order to tunnel a connection to a mongodb over ssh like so.

    $ ssh -L LOCAL_PORT_NO_TO_LISTEN:host_with_mongo_connection:MONGODB_REMOTE_PORT_NO user@IP_ADDRESS_OF_THE_BOX_WITH_MONGODB

    Thus, if the mongod is running on the server you have ssh access to, then the command would look like this:

    $ ssh -L LOCAL_PORT_NO_TO_LISTEN :localhost:MONGODB_REMOTE_PORT_NO user@ssh_and_mongo_box

     

  • You can even setup an encrypted tunnel (vpn) for connection between your application and mongodb

    Traffic to and from mongod Instances

    This pattern is applicable to all mongod instances running as standalone instances or as part of a replica set.

    The goal of this pattern is to explicitly allow traffic to the mongod instance from the application server. In the following examples, replace <ip-address> with the IP address of the application server:

    $ iptables -A INPUT -s <ip-address> -p tcp --destination-port MONGODB_PORT_NO -m state --state NEW,ESTABLISHED -j ACCEPT
    $ iptables -A OUTPUT -d <ip-address> -p tcp --source-port MONGODB_PORT_NO -m state --state ESTABLISHED -j ACCEPT

    The first rule allows all incoming traffic from <ip-address> on port 27017, which allows the application server to connect to the mongod instance. The second rule, allows outgoing traffic from the mongod to reach the application server.

    Optional: If you have only one application server, you can replace <ip-address> with either the IP address itself, such as: 10.23.33.3. You can also express this using CIDR notation as 10.23.33.3/32. If you want to permit a larger block of possible IP addresses (i.e while playing attack/defense type CTF as a team in a VPN) you can allow traffic from a /24 using one of the following specifications for the <ip-address>, as follows:

    10.23.33.3/24
    10.23.33.3/255.255.255.0

Sample secure config file of mongodb (/etc/mongodb.conf)

# mongodb.conf
# Where to store the data.
# Note: if you run mongodb as a non-root user (recommended) you may
# need to create and set permissions for this directory manually,
# e.g., if the parent directory isn't mutable by the mongodb user.
dbpath=/var/lib/mongodb
# Only accept local connections
bind_ip = 127.0.0.1
#where to log
logpath=/var/log/mongodb/mongodb.log
logappend=true
# Connecting port of mongodb
port = 7171
# Disables write-ahead journaling
# nojournal = true
# Enables periodic logging of CPU utilization and I/O wait
#cpu = true
# Turn on/off security. Off is currently the default
#noauth = true
auth = true
# Verbose logging output.
#verbose = true
# Inspect all client data for validity on receipt (useful for
# developing drivers)
#objcheck = true
# Enable db quota management
#quota = true
# Set oplogging level where n is
# 0=off (default)
# 1=W
# 2=R
# 3=both
# 7=W+some reads
#diaglog = 0
# Ignore query hints
#nohints = true
# Disable the HTTP interface (Defaults to localhost:28017).
nohttpinterface = true
# Turns off server-side scripting. This will result in greatly limited
# functionality
noscripting = true
# Turns off table scans. Any query that would do a table scan fails.
#notablescan = true
# Disable data file preallocation.
#noprealloc = true
# Specify .ns file size for new databases.
# nssize = <size>
# Accout token for Mongo monitoring server.
#mms-token = <token>
# Server name for Mongo monitoring server.
#mms-name = <server-name>
# Ping interval for Mongo monitoring server.
#mms-interval = <seconds>
# Replication Options
# in master/slave replicated mongo databases, specify here whether
# this is a slave or master
#slave = true
#source = master.example.com
# Slave only: specify a single database to replicate
#only = master.example.com
# or
#master = true
#source = slave.example.com
# in replica set configuration, specify the name of the replica set
# replSet = setname

Shrink VirtualBox image and exporting to OVA/OVF file format


Recently, I have done an OS security assignment on hardening a Linux desktop edition machine for hosting a web server. After completing this task, I thought to export this image in-order to use it in the future for any CTF’s.  When I just exported the image into a OVA its size came around 1.2 GB. Then I Googled and found an interesting command call dd (used to convert and copy files) which will zero out the unused space in the virtual machine. Later, after doing the below given steps, I was able to reduce the size of OVA file to 70% of its orginal. It became 830 MB in the end! Hence I thought to blog about it.

$ dd if=/dev/zero of=/bigemptyfile
$ sudo rm -rf /bigemptyfile

If the free space on the guest OS is not much left, you can use the following alternative:

cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f zero.fill

Shutdown the Virtual machine. If you want to shrink the Virtual Disk Image (VDI), then you can run this command to reduce its size:

$ VBoxManage modifyhd --compact Ubuntu.vdi

You can also export the virtual machine into standard Open Virtualization Formats

  • If you choose an ova extension, then all the files will be combined into one single Open Virtualization Format Archive.
  • If you use an ovf extension, several files will be written separately. (I always prefer ova file)

Image

Hardening Ubuntu 12.04 Desktop edition for hosting a web server


As a part of my Operating System security course, I was asked to harden a Ubuntu Desktop edition for hosting a web server. I used Ubuntu 12.04 as a guest machine in Virtualbox. Its better to install packages like openssh-server and vim before you start executing the below given commands.

Removing all the unwanted applications

Remove ubuntu-desktop and all the gnome packages. *This is not required if you are using a server edition*

$ sudo apt-get remove gnome-* 
$ sudo apt-get remove alacarte app-install-data-commercial apport-gtk apturl at-spi binfmt-support bluez-gnome brasero brltty-x11 capplets-data cli-common compiz compiz-core compiz-fusion-plugins-extra compiz-fusion-plugins-main compiz-gnome compiz-plugins compizconfig-backend-gconf contact-lookup-applet dcraw deskbar-applet desktop-file-utils dmz-cursor-theme doc-base docbook-xml ekiga eog espeak espeak-data evince evolution evolution-common evolution-data-server evolution-data-server-common evolution-exchange evolution-plugins evolution-webcal example-content f-spot fast-user-switch-applet file-roller firefox firefox-3.0 firefox-3.0-branding firefox-3.0-gnome-support firefox-gnome-support gamin gcalctool gconf-editor gconf2 gconf2-common gdebi gdm gdm-guest-session gedit gedit-common ggzcore-bin gimp gimp-data gksu gstreamer0.10-alsa gstreamer0.10-gnomevfs gstreamer0.10-plugins-base gstreamer0.10-plugins-base-apps gstreamer0.10-plugins-good gstreamer0.10-pulseaudio gstreamer0.10-schroedinger gstreamer0.10-tools gstreamer0.10-x gtk2-engines gtk2-engines-murrine gtk2-engines-pixbuf gucharmap guile-1.8-libs gvfs gvfs-backends gvfs-bin gvfs-fuse human-icon-theme human-theme hwtest hwtest-gtk jockey-gtk language-selector libart2.0-cil libasound2-plugins libatspi1.0-0 libavahi-glib1 libavahi-gobject0 libavahi-ui0 libavc1394-0 libbabl-0.0-0 libbeagle1 libbonobo2-0 libbonobo2-common libbonoboui2-0 libbonoboui2-common libcairo-perl libcairomm-1.0-1 libcamel1.2-14 libcanberra-gnome libcanberra-gtk-module libcanberra-gtk0 libcanberra0 libcdio-cdda0 libcdio-paranoia0 libcdio7 libcompizconfig0 libcryptui0 libdecoration0 libdeskbar-tracker libdmx1 libdv4 libebackend1.2-0 libebook1.2-9 libecal1.2-7 libedata-book1.2-2 libedata-cal1.2-6 libedataserver1.2-11 libedataserverui1.2-8 libeel2-2 libeel2-data libegroupwise1.2-13 libespeak1 libexchange-storage1.2-3 libexempi3 libflickrnet2.1.5-cil libfreezethaw-perl libgadu3 libgail-gnome-module libgamin0 libgconf2-4 libgconf2.0-cil libgdata-google1.2-1 libgdata1.2-1 libgdiplus libgegl-0.0-0 libggz2 libggzcore9 libggzmod4 libgimp2.0 libgksu2-0 libglade2-0 libglade2.0-cil libglew1.5 libglib-perl libglib2.0-cil libglibmm-2.4-1c2a libglitz-glx1 libglitz1 libgmime-2.0-2a libgmime2.2-cil libgnome-desktop-2-7 libgnome-keyring0 libgnome-keyring1.0-cil libgnome-mag2 libgnome-media0 libgnome-menu2 libgnome-pilot2 libgnome-speech7 libgnome-vfs2.0-cil libgnome-window-settings1 libgnome2-0 libgnome2-canvas-perl libgnome2-common libgnome2-perl libgnome2-vfs-perl libgnome2.0-cil libgnomecanvas2-0 libgnomecanvas2-common libgnomecups1.0-1 libgnomekbd-common libgnomekbd3 libgnomekbdui3 libgnomeprint2.2-0 libgnomeprint2.2-data libgnomeprintui2.2-0 libgnomeprintui2.2-common libgnomeui-0 libgnomeui-common libgnomevfs2-0 libgnomevfs2-bin libgnomevfs2-common libgnomevfs2-extra libgp11-0 libgpod3 libgtk-vnc-1.0-0 libgtk2-perl libgtk2.0-cil libgtkglext1 libgtkhtml-editor-common libgtkhtml-editor0 libgtkhtml2-0 libgtkhtml3.14-19 libgtkmm-2.4-1c2a libgtksourceview-common libgtksourceview1.0-0 libgtksourceview2.0-0 libgtksourceview2.0-common libgtkspell0 libgtop2-7 libgtop2-common libgucharmap7 libgvfscommon0 libgweather-common libgweather1 libhesiod0 libidl0 libiec61883-0 libjpeg-progs libkpathsea4 liblaunchpad-integration1 liblircclient0 liblpint-bonobo0 libmbca0 libmeanwhile1 libmetacity0 libmldbm-perl libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-cairo1.0-cil libmono-cairo2.0-cil libmono-corlib1.0-cil libmono-corlib2.0-cil libmono-data-tds1.0-cil libmono-data-tds2.0-cil libmono-i18n1.0-cil libmono-i18n2.0-cil libmono-security1.0-cil libmono-security2.0-cil libmono-sharpzip0.84-cil libmono-sharpzip2.84-cil libmono-sqlite2.0-cil libmono-system-data1.0-cil libmono-system-data2.0-cil libmono-system-web1.0-cil libmono-system-web2.0-cil libmono-system1.0-cil libmono-system2.0-cil libmono0 libmono1.0-cil libmono2.0-cil libnautilus-burn4 libnautilus-extension1 libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libnet-dbus-perl libnotify1 liboil0.3 liboobs-1-4 libopal-2.2 libopenobex1 liborbit2 libpam-gnome-keyring libpanel-applet2-0 libpangomm-1.4-1 libpisock9 libpisync1 libpolkit-gnome0 libpoppler-glib3 libportaudio0 libpt-1.10.10 libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libpt-1.10.10-plugins-v4l2 libpulse-browse0 libpulsecore5 libpurple-bin libpurple0 librarian0 librsvg2-common libschroedinger-1.0-0 libscim8c2a libsexy2 libshout3 libsilc-1.1-2 libsndfile1 libsoup2.4-1 libspeexdsp1 libsqlite0 libstartup-notification0 libtie-ixhash-perl libtotem-plparser12 libtracker-gtk0 libtrackerclient0 libuuid-perl libv4l-0 libvte-common libvte9 libwnck-common libwnck22 libwv-1.2-3 libx11-xcb1 libxevie1 libxml-twig-perl libxml-xpath-perl libxml2-utils libxres1 libzephyr3 metacity metacity-common mobile-broadband-provider-info mono-common mono-gac mono-jit mono-runtime mousetweaks mtools nautilus nautilus-cd-burner nautilus-data nautilus-sendto nautilus-share network-manager-gnome notification-daemon obex-data-server onboard openoffice.org-gnome openoffice.org-gtk pidgin pidgin-data pidgin-otr pkg-config policykit-gnome pulseaudio pulseaudio-esound-compat pulseaudio-module-gconf pulseaudio-module-hal pulseaudio-module-x11 pulseaudio-utils python-beagle python-brlapi python-cairo python-gconf python-gdata python-glade2 python-gmenu python-gnome2 python-gnome2-desktop python-gnomecanvas python-gst0.10 python-gtk2 python-gtkhtml2 python-gtksourceview2 python-launchpad-integration python-notify python-numeric python-pkg-resources python-pyatspi python-pyorbit python-rdflib python-sexy python-virtkey python-vte rarian-compat rhythmbox rss-glx scim scim-bridge-agent scim-bridge-client-gtk scim-gtk2-immodule scim-modules-socket screen-resolution-extra screensaver-default-images seahorse seahorse-plugins sgml-data software-properties-gtk sqlite sqlite3 ssh-askpass-gnome synaptic syslinux system-config-printer-gnome system-tools-backends tangerine-icon-theme tomboy totem totem-common totem-gstreamer totem-mozilla totem-plugins tracker tracker-search-tool tracker-utils transmission-common transmission-gtk tsclient ubufox ubuntu-artwork ubuntu-desktop ubuntu-docs ubuntu-gdm-themes ubuntu-sounds ubuntu-system-service ubuntu-wallpapers untex update-manager update-notifier usb-creator vinagre vino whois wv xbase-clients xbitmaps xdg-user-dirs-gtk xsane xsane-common xscreensaver xscreensaver-data xscreensaver-gl xsltproc xterm xulrunner-1.9 xulrunner-1.9-gnome-support yelp zenity
$ sudo apt-get remove ubuntu-desktop

Removing the GUI and switching to console mode

Now we have to modify grub config. Open /etc/default/grub with vim (install vim if you haven’t yet -> sudo apt-get install vim)

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

to

GRUB_CMDLINE_LINUX_DEFAULT="text"

Also comment GRUB_HIDDEN_TIMEOUT=0 This line is for un-hiding the GRUB menu
Now we will upgrade GRUB configuration

$ sudo update-grub

Ubuntu 12.04 Desktop edition use lightdm for GUI. We need to disable the same

$ sudo update-rc.d -f lightdm remove

Now restart your machine.

Lets start hardening Ubuntu

Install Bastille Linux. The Bastille Linux project aims to provide an interactive tool for the purpose of performing additional security hardening measures to increase the over-all security, and decrease the susceptibility of compromise for your Ubuntu system. This guide is designed to assist in the installation, and execution of the Bastille Linux tool for the purpose of hardening the security of your Ubuntu system.

$ sudo apt-get install bastille

Then run

$ bastille -c

Bastille will do major hardening. Still there are some steps we need to follow in-order to harden it more.

Step 1:  A regular user should not have write access to system files. Worst case you can run a

find -user OTHERSUSERNAME -writable

to search the system for any potentially writable files, but that shouldn’t be an issue.
Step 2: Removing read access from /proc and also remove execution permission from various suid set system binaries like ping, passwd, ifconfig, nmap, scp, who, finger, ssh etc if you want people to remain semi-anonymous. Its best to do a scan of SUID files across the whole system and disable as many as possible, or remove execute/read permission for non root users.

$ find / -perm -4000 -o -perm -2000 -print

Step 3: chmod these files to to 0700 if you are allowing ssh access to several users

/bin/ping
/usr/bin/who
/usr/bin/w
/usr/bin/locate
/usr/bin/whereis
/sbin/ifconfig
/bin/nano
/usr/bin/vi
/usr/bin/which
/usr/bin/gcc
/usr/bin/make
/usr/bin/apt-get

Step 4: Enable sticky bit (chmod +t) for preventing directory listing in /home/var/www/public_html and all other highly sensible data directories.

/root 0700 
/boot 0700
/home 1711
/etc 0711
/tmp 1777 or 1733
/var/www/public_html 1755
/var/www/public_html/files 0644
/media and /mnt 0700

Step 5: Disable Compilers in-order not to compile Kernel or other exploits on the server

chmod 750 /usr/bin/gcc
chmod 750 /usr/bin/*++*
chmod 750 /usr/bin/*cc*
chmod 750 /usr/bin/as
chmod 750 /usr/bin/mysqlaccess

Securing SSH

Open /etc/ssh/sshd_config

  • Disable ‘password based login‘ and only permit ‘public key authentication‘.
  • Disable Root login ‘PermitRootLogin yes‘ -> ‘PermitRootLogin no
  • Reduce the ‘LoginGraceTime‘ to 120
  • Disable ‘X11Forwarding‘, ‘X11Forwarding yes‘ -> ‘X11Forwarding no

Root Notification

Edit .bashrc under /root to get notified by email when someone logs in as root and add the following:

$ echo 'ALERT - Root Shell Access (Server Name) on:' `date` `who` | mail-s "Alert: Root Access from `who | cut -d"(" -f2 | cut -d")" -f1`" YOUR_EMAIL_ID

Choosing a secure password

Ubuntu uses per default sha512 hashed passwords. sha512 is a 128bit hash and nowadays not too secure anymore. There is support for an alternate hash, blowfish. This has proven quite secure in openbsd.
To enable it, you have to install libpam-unix2.

$ sudo apt-get install libpam-unix2

Next replace all references of pam_unix.so with pam_unix2.so in all files under /etc/pam.d
Note: The number of references in the files depends on installed pam modules.
In common-password add the following:

password required      pam_unix2.so nullok use_authok obscure min=9 max=72     blowfish

To enable password checks when changing passwords, install libpam-passwdqc and add the following to your /etc/pam.d/common-password:

$ sudo apt-get install libpam-passwdqc

edit /etc/pam.d/common-password and add

password required      pam_passwdqc.so max=72 similar=deny enforce=everyone retry=2 ask_oldauthtok=update check_oldauthtok

Note: To finalize the changes, all passwords have to be reentered!!
If you restart at this point, you will be unable to log in again!
Set a new password with the passwd command ( can be the same as before, but it has to be reset to use the new hash!)

Problems: For those of you in trouble after a reboot:
you can boot to your system without a password, if you append a init=/bin/bash to your grub kernel parameter.
To enter that, type “e” at the grup prompt and you can edit the entry.

IMPORTANT: This does only secure Ubuntu passwords! You still can log in with things like knoppix or other methods like described above!
For a fully secured system you have to encrypt hard drives!

There is a quite simple way to test or to change to the blowfish hashing without changing all old passwords. Simply add above the old lines the following lines

in /etc/pam.d/common-account

account sufficient pam_unix2.so
account required pam_unix.so

/etc/pam.d/common-auth

auth sufficient pam_unix2.so nullok
auth required pam_unix.so nullok_secure

/etc/pam.d/common-session

session sufficient pam_unix2.so
session required pam_unix.so

As the authentication with the pam_unix2.so modules ist sufficient the blowfish hashed passwords work, but if something fail, the old (pam_unix.so) module will do their job.

Checking for Rootkits

Install chkrootkits

$ sudo apt-get install chkrootkits

You can run it with the following command: ./chkrootkit
Add it to crontab to schedule daily automatic scans in the system:
vim /etc/cron.daily/chkrootkit.sh

#!/bin/bash
./chkrootkit | mail -s "Daily chkrootkit from Server Name" admin@myhost.com
chmod 755 /etc/cron.daily/chkrootkit.sh

Hardening Kernel (/etc/sysctl.conf)

sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can configure various Linux networking and system settings such as:

  • Limit network-transmitted configuration for IPv4
  • Limit network-transmitted configuration for IPv6
  • Turn on execshield protection
  • Prevent against the common ‘syn flood attack’
  • Turn on source IP address verification
  • Prevents a cracker from using a spoofing attack against the IP address of the server.
  • Logs several types of suspicious packets, such as spoofed packets, source-routed packets, and redirects.

Sample /etc/sysctl.conf

Edit /etc/sysctl.conf and update it as follows. The file is documented with comments. However, I recommend reading the official Linux kernel sysctl tuning help file (see below):

# The following is suitable for dedicated web server, mail, ftp server etc. 
# ---------------------------------------
# BOOLEAN Values:
# a) 0 (zero) - disabled / no / false
# b) Non zero - enabled / yes / true
# --------------------------------------
# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2

########## IPv4 networking start ##############
# Send redirects, if router, but this is just server
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Accept packets with SRR option? No
net.ipv4.conf.all.accept_source_route = 0

# Accept Redirects? No, this is not router
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0

# Log packets with impossible addresses to kernel log? yes
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

# Ignore all ICMP ECHO and TIMESTAMP requests sent to it via broadcast/multicast
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Prevent against the common 'syn flood attack'
net.ipv4.tcp_syncookies = 1

# Enable source validation by reversed path, as specified in RFC1812
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

#Enable ExecShield protection
kernel.exec-shield = 1
kernel.randomize_va_space = 1

# TCP and memory optimization 
# increase TCP max buffer size setable using setsockopt()
#net.ipv4.tcp_rmem = 4096 87380 8388608
#net.ipv4.tcp_wmem = 4096 87380 8388608

# increase Linux auto tuning TCP buffer limits
#net.core.rmem_max = 8388608
#net.core.wmem_max = 8388608
#net.core.netdev_max_backlog = 5000
#net.ipv4.tcp_window_scaling = 1

# increase system file descriptor limit    
fs.file-max = 65535

#Allow for more PIDs 
kernel.pid_max = 65536

#Increase system IP port limits
net.ipv4.ip_local_port_range = 2000 65000

Apache Hardening

Edit /etc/apache2/apache.conf
Turn off these variables:

TraceEnable off

Edit /etc/php5/fpm/php.ini

change

disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,

to

disable_functions = exec, passthru, shell_exec, system, proc_open, popen, curl_exec, curl_multi_exec, parse_ini_file, show_source

The Apache web server is a crucial part of the website infrastructure. It has a number of built in features that can improve your website resistance to attacks

Do not allow browsing outside the document root

Allowing browsing outside the document root is inviting trouble. Unless you have a specific need to allow it, disable this feature. First, you’ll need to edit the document root Directory entry like so (/etc/apache2/http.conf)

ServerName localhost
ServerSignature Off
ServerTokens Prod
<Directory />
Order Deny, Allow
Deny from all
Options None
AllowOverride None
</Directory>

Now, if you need to add options to any directory within the document root, you will have to add a new Directory entry for each one.

Disable Apache Signature and/or Apache Banner

Apache Signature or Apache Banner is basically the same thing. It is an application name together with version name that is printed when performing a web request. Nobody actually needs this information at all, but it is enabled by default. You need to alter the Apache configuration file to disable it.
Add these lines to this file /etc/apache2/apache2.conf

ServerSignature Off
ServerTokens ProductOnly

The Trace HTTP Request

HTTP TRACE request is used to echo back all received information. It can be tricked to print HTTP cookies and as a result steal HTTP session. Basically this request can be used as part of the Cross Site Scripting attack, or XSS. It is recommended to disable it as a security precaution.

Add the following to the web-server’s configuration file /etc/apache2/apache2.conf

TraceEnable off

Disable directory indexing

You can fix this problem by disabling the Apache autoindex module. In some Apache installations it is called mod_autoindex.so. In Ubuntu, you just need to remove the following files:

/etc/apache2/mods-enabled/autoindex.load
/etc/apache2/mods-enabled/autoindex.conf

So you can do it running the following commands:

rm -f /etc/apache2/mods-enabled/autoindex.load
rm -f /etc/apache2/mods-enabled/autoindex.conf

Disable WebDAV

WebDAV is a file access protocol created over HTTP protocol. It allows you to upload and download files, and change file contents from the website. This service is required only in very rare cases.
You can fix this problem by disabling Apache dav, dav_fs and dav_lock modules. In Ubuntu you just need to remove the following files:

/etc/apache2/mods-enabled/dav.load
/etc/apache2/mods-enabled/dav_fs.conf
/etc/apache2/mods-enabled/dav_fs.load
/etc/apache2/mods-enabled/dav_lock.load

So you can do it running the following commands:

rm -f /etc/apache2/mods-enabled/dav.load
rm -f /etc/apache2/mods-enabled/dav_fs.conf
rm -f /etc/apache2/mods-enabled/dav_fs.load
rm -f /etc/apache2/mods-enabled/dav_lock.load

Create a chroot’ed Apache environment

Chroot is a kind of virtual environment supported operating systems such as Linux and FreeBSD. When an application is executed in chrooted environment it has no access to the parent disk and to other resources. You can follow this link to create a chroot’ed Apache environment

Additional Steps

If your webserver runs together with MySQL server it brings additional potential security problem. MySQL can read any files located on you server including the one located in different chrooted environments. It happens because of the FILE permission. By default only MySQL root has it.

MySQL

Resetting the Root Password

We’ve to killall the demon process of mysql i.e mysqld

$ sudo /etc/init.d/mysql stop

Restart mysqld with --skip-grant-tables and --skip-network, to connect without a password and with all privileges and also to prevent remote clients from connecting.

$ sudo mysqld --skip-grant-tables --skip-networking &

Now connect to MySQL client

$ mysql

Reset the root account password using this command:

mysql> UPDATE mysql.user SET Password=PASSWORD('YOUR_NEW_ROOT_PASSWORD') WHERE User='root';
FLUSH PRIVILEGES;

exit mysql and restart the mysql service

$ sudo service mysql restart

Now you can login as a root user in MySQL client.

Securing MySQL

mysql> USE mysql;

Get rid of the test database, which is accessible to anyone from anywhere

msyql> select * from db where Db like 'test%';
msyql> drop database test;
mysql> delete from db where Db like 'test%';

Remove the anonymous users

msyql> show grants for ''@'localhost';
msyql> revoke on all *.* from ''@'localhost';
mysql> delete from user where User='' and Host='localhost'; 
mysql> flush privileges;

Lower system privileges
To protect your database, make sure that the file directory in which the MySQL database is actually stored is owned by the user “mysql” and the group “mysql”.

ls -l /var/lib/mysql

In addition, ensure that only the user “mysql” and “root” have access to the directory /var/lib/mysql.
The mysql binaries, which reside under the /usr/bin/ directory, should be owned by “root” or the specific system “mysql” user. Other users should not have write access to these files.

ls -l /usr/bin/my*

Remove Anonymous and obsolete accounts
To check whether this is the case,

mysql> select * from mysql.user where user="";

If it returns any user(s), immediately drop that user.

mysql> drop user 'WHAT_EVER_USERNAMES_RETURNED_BY_THE_PREVIOUS_COMMAND';

Disable or restrict remote access
Edit /etc/mysql/my.cnf file and add these lines to disable remote MySQL login and listen only to the localhost

skip-networking
bind-address=127.0.0.1

Lower DB privileges
To disable the usage of the “SHOW DATABASES” command, the following parameter should be added in the [mysqld] section of the /etc/mysql/my.cnf

skip-show-database

Enable Logging
It is recommended that you enable transaction logging, by adding the following line to [mysqld] section of the /etc/msyql/my.cnf file

log =/var/log/mylogfile

Remove History
We should remove the content of the MySQL history file (~/.mysql_history), where all executed SQL commands are stored (especially passwords, which are stored as plain text):

$ cat /dev/null > ~/.mysql_history

Automatically updating the machine

$ sudo apt-get install unattended-upgrades
$ sudo dpkg-reconfigure unattended-upgrades

NOTE: This may not contain all the security measures. I would like to request the reader to add comments about the security measures which are not mentioned in this post. Your suggestions and feedbacks are always welcomed. Thanks in advance 🙂

References:
[1]: http://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-selinux.html
[2]: http://www.greensql.com/articles/mysql-security-best-practices
[3]: https://help.ubuntu.com/community/MysqlPasswordReset
[4]: https://help.ubuntu.com/community/ApacheMySQLPHP
[5]: http://www.kyplex.com/docs/apache-security.html
[6]: http://ubuntuforums.org/showthread.php?t=300208
[7]: http://www.cyberciti.biz/faq/linux-kernel-etcsysctl-conf-security-hardening/
[8]: https://help.ubuntu.com/community/BastilleLinux
[9]: http://askubuntu.com/questions/70188/how-do-i-boot-into-console-mode
[10]: http://askubuntu.com/questions/164941/why-removing-gnome-core-does-not-remove-all-of-its-dependencies