Host Your Own EteSync Server

last updated 2020-11-28 08:52:28 by Simon Vandevelde

This guide is outdated, and only here for legacy reasons. If you want the up to date version, you can find it on the EteSync wiki

Hi! Welcome to my guide on self hosting an EteSync server on the Raspberry Pi Zero.

Requirements:

Computer running a Linux flavour;
A Raspberry Pi Zero;
A micro SD card;
A working internet connection;
A domain name, unless you want to read it locally;
The ability to port-forward your router;
The ability to read.

Setup the Pi.

In the first step, we'll be setting up the Raspberry Pi zero. If your Pi already runs an operating system, you can skip to the next step.

In this guide, we will be using the `Raspbian `_ operating system. Raspbian is based on Debian, which is an OS known for its stability. This means that we can setup up the server and have minimal maintenance afterwards.

Download the image

Go to the download url and download the Lite version of Raspbian. The lite version ships without desktop, and is thus only usable through the terminal. If you would like a desktop, download the Raspbian version with desktop.

Flash the image

Once the image is downloaded, we can flash it to our SD card. Open up a terminal, and execute lsblk. Find the name of your micro SD. If you're not sure which device name belongs to the SD card, unplug it, run lsblk, plug it in and rerun lsblk. The new device that shows up is your SD card. It is extremely important that you choose the right name. You could seriously mess up your filesystem! Next we're going to write the image to the SD. To do this, navigate to where the Raspbian image is located and execute the following command. Fill in the name of the image and the name of the /dev/ device.

    
    $ dd bs=4M if=20xxI-xx-xx-raspbian-xx.img of=/dev/sdX conv=fsync progress=status

Setup SSH and WiFi

Now that the image is written, we can almost start the Raspberry Pi. However, it is important that we are able to connect to the Pi afterwards. See this tutorial page to do a headless setup of the WiFi and to enable SSH.

Prepare the Pi

Now that we have the image ready, go ahead and stick it in the Pi. It should connect to your network and allow SSH connections. Next we need the Pi's IP address to connect to it. To find out the IP address of your Pi, there are multiple ways. An easy one is to check your router's connected devices. Another one is using the command arp -a to list connected devices. After finding out the IP, SSH to it by using the following command:


    $ ssh pi@192.168.x.x

The first thing we need to do is change the password (and preferably the user too!). To change the password, type passwd. To change the user, create a new user like so:

    $ sudo useradd -m 
    $ sudo groupadd sudo  # Add user to sudo group
    $ sudo passwd

Now close the ssh connection, and reconnect to the Pi using the new user. After having created a new user, you might want to consider deleting the default Pi user: sudo userdel pi.

Download install dependencies

Now that we have our own user, we need to update and upgrade the Pi. Do this using the following command.

    $ sudo apt get update && sudo apt get upgrade

Note: this might take a very long time.

Next up, we need to install three packages at the start.

    $ sudo apt get install python3-pip git
    $ pip3 install virtualenv

Setting up the basic server

Clone the server code

First, we need to clone the server code from the GitHub repository. To download the server in your home directory, execute the following commands:


    $ cd ~
    $ git clone blablabla.git
    $ cd server-skeleton

Create a virtual environment

Before installing our dependencies, we should set up a Python virtual environment. This virtual environment keeps the Python packages we install for the server seperated from our users' packages. More info on this can be sound here. This is technically optional, but I highly recommend it. Setting up a virtual environment is simple. In ~/server-skeleton, simply do the following.


    $ virtualenv -p python3 venv
    $ source venv/bin/activate

By sourcing venv/bin/activate, we 'activate' the virtual environment. Any dependencies now installed, will be installed to the virtual environment instead of our user's environment.

Install dependencies

The server GitHub repo comes with a file called requirements.txt file. This file contains all the necessary Python requirements for the servercode to run. Installing them is very simple. .. code-block:: console


    $ pip3 install -r requirements.txt

Create an ini file

The server requires a basic .ini file to run. This file contains some settings for the server. Luckily for us, the the GitHub repo already contains a basic ini called etesync-server.ini.example!

All we need to do is copy the file to etesync-server.ini and adjust one line. To do this, run the cp command.


    $ cp etesync-server.ini.example etesync-server.ini

Afterwards, open the new ini file in your favorite terminal text editor (for beginners I recommend Nano) and set allowed hosts to "*".

Test the basic server!

Now we need to initialise the server and then we can run it.


    $ ./manage.py migrate
    $ ./manage.py runserver 0.0.0.0:8000

Now try surfing to the Pi's IP address followed by the portnumber (192.168.x.x:8000) in a web browser and check the page. You should see a webpage saying "It works!". If not, try to retrace your steps to find out where it went wrong.

3. uWSGI

Context

We can't just run our basic server and expose it to the outside world. Intead, we build a server stack: different layers on top of each other, with each layer talking to a top one and a bottom one. The uWSGI docs say the following: A web server faces the outside world. It can serve files (HTML, images, CSS, etc) directly from the file system. However, it can’t talk directly to Django applications; it needs something that will run the application, feed it requests from web clients (such as browsers) and return responses.

A Web Server Gateway Interface - WSGI - does this job. WSGI is a Python standard.

uWSGI is a WSGI implementation. In this tutorial we will set up uWSGI so that it creates a Unix socket, and serves responses to the web server via the uwsgi protocol. At the end, our complete stack of components will look like this:

    
    the web client <-> the web server <-> the socket <-> uwsgi <-> Django

In this tutorial, we use uWSGI as WSGI, and Nginx as web server.

Installing uWSGI

Installation of uWSGI is done via Pip. We invoke the installation command as sudo, because the root user needs access to uWSGI later.


    $ sudo pip3 install uwsgi

Testing uWSGI

We can already test whether or not our uWSGI installation works, by using the following command.


    $ uwsgi --http :8000 --module etesync_server.wsgi --virtualenv venv

This command will have uWSGI start an EteSync server instance and expose it on port 8000. Check in your webbrowser if you can see the "It works!" again. If not, you should see an error pop up in the terminal to help debugging.

4. Nginx

The final layer of our server stack is Nginx. Nginx will form our proper web server.

Installing Nginx

Nginx is installable via the apt.


    $ sudo apt-get install nginx

Setup Nginx

Next, we have some setup to do. First of all, we're going to create Django's static files, so that Nginx can access these.


    $ ./manage.py collectstatic

After creating the static files, we need to create a configuration file for Nginx. Create a new file called etesync_nginx.conf and paste in the following configuration. Don't forget to change the server_name and fill in the correct path to your static folder!

Next, we need to move this configuration file to /etc/nginx/sites-available/ and then symlink it to /etc/nginx/sites-enabled to enable it. Nginx will start to use the config file after a restart. If we then run the uWSGI server like before but on port 8001 instead of 8000, you should be able to surf to the same url as before and see "It Works!".


    $ sudo cp etesync_nginx.conf /etc/nginx/sites-available/
    $ sudo ln -s /etc/nginx/sites-available/etesync_nginx.conf /etc/nginx/sites-enables/etesync_nginx.conf
    $ systemctl restart nginx
    $ uwsgi --socket :8001 --module etesync_server.wsgi

If you encounter any errors, check the Nginx log which is located in /var/log/nginx/error.log.

Finalize Nginx setup

Now that we now Nginx works, we want to make a last small adjustment. Currently, our uWSGI setup exposes the Django server on port 8001. However, a more elegant solution would be to expose the Django server using a file socket. Change the /etc/nginx/sites-available/etesync_nginx.conf file to accept sockets by commenting in # server unix:///tmp/etesync_server.sock; and commenting out server 127.0.0.1:8001;.

The file should now look like this.

Finalize uWSGI setup

Of course, we still need to setup our uWSGI to also use a websocket. For this, copy the following code into a file called uwsgi.ini.

This .ini file specifies some parameters to our uWSGI, like for example to location of where it should expose the socket and which users should be able to use it. Since Nginx runs under the user www-data, we use the .ini to make sure Nginx is able to open socket.

If we now want to run uWSGI, we do so by using the following command.


    $ uwsgi --ini uwsgi.ini

5. Run server at boot

Now that we have a fully functional webserver, we want to make sure that it is able to launch at startup so that we don't have to start it manually very time. Luckily, Nginx does this already. (Recall how we used a systemd command earlier to restart it).

We can do the exact same thing for uWSGI, after creating a systemd unit file. Create a new file and paste in the following (don't forget to change ):

This unit executes exactlythe same command as we had to do earlier. By saving this file as /etc/systemd/system/etesync_server.service, we can use it via systemd.

To start and enable the uWSGI unit, do the following.


    $ systemd start etesync_server
    $ sytemd enable etesync_server #this makes it launch at startup

6. Open up the server to the world!

It would surely stink if the EteSync server could only be used on a local network. Luckily for us, it is very easy to expose it to the outside world. Unluckily for us, this is also very dangerous. We could run the EteSync server over HTTP, but obviously HTTPS is more safe.

First, we need to download certbot. To setup SSL for HTTPS, we need to first make sure our server is reachable on HTTP from the outside. To do this, you should port-forward your router to point port 80 and 443 to your pi's local IP address. Next, have your domain name resolve to your external IP address (if you don't know this, check here). Now you should be able to reach your EteSync server using your domain name, followed by :8000.

To setup HTTPS, we're going to need a certificate. Thanks to certbot, this is a breeze. All we need to do is install it, and have it run. It should even automatically change our etesync_nginx.conf file to support HTTPS.


    $ sudo apt-get install certbot python-certbot-nginx
    $ sudo certbot --nginx

And that's it! Our server is fully set up. All there's left to do, is create an admin super user, login to our EteSync server, create a non-super user and you're good to go.


    $ ./manage.py createsuperuser