1. Introduction

Welcome to the installation instructions for LSDTopoTools, which is a collection of programs for analysing topography and modelling hydrology, landscape evolution and soil development.

1.1. Components of LSDTopoTools

LSDTopoTools has several components. These can be used in isolation or together.

  • LSDTopoTools command line tools: these run the core computational components of LSDTopoTools. These do most of the heavy lifting of LSDTopoTools.

  • lsdviztools a series of python scripts for visualising LSDTopoTools outputs. You can run these interactively but they also come with command line scripts.

  • lsdtopytools a python wrapper for LSDTopoTools so you get much of the functionality of the LSDTopoTools command line tools within the python environment.

1.2. Basic installation

If you are already familiar with Linux and/or Conda this should relatively painless.

  1. If you are in a Linux or MacOS all you really need is Miniconda. You should use the python 3.8 version.

  2. In windows don’t install conda for windows! Create an Ubuntu subsystem using these instructions and install conda there.

  3. Once you have conda you can install the command line tools like this:

    $ conda -c conda-forge install lsdtopotools
  4. Now you can use the command line tools!

1.3. lsdtopotools Python tools

  1. You install lsdtopytools using conda

    $ conda -c conda-forge install lsdtopytools
  2. You install lsdviztools using pip:

    $ pip install lsdviztools
  3. These both have a bunch of dependencies so it is easier to use our docker container that has everything pre-installed. You can read about that below.

1.4. Menu of more detailed instructions

  • Installation using Docker. This works for MacOS and Linux well. It works for Windows 10 Enterprise edition. If you don’t have those operating systems you will need to use VirtualBox + Vagrant (see below).

  • Native installation on a Linux system. If your native operating system is Linux you can just install LSDTopoTools after getting all the libraries you need. The documents are made under the assumption that you use Ubuntu, but if you are a Linux user we are confident you’ll be able to port the instructions to your native system (usually this involves replacing sudo apt-get with sudo yum).

  • University of Edinburgh installation. This software is developed at the University of Edinburgh and is used in teaching. We include instructions specific to students and staff at the University of Edinburgh for use on the School of GeoSciences servers.

What installation method should I use?

2. Installing LSDTopoTools using Docker

As of October 2018, this is the preferred method to install LSDTopoTools. We have various containers on the docker hub which can be run with simple pull commands. The only software you need to run these is Docker.

Update June 2019

Docker is very easy to install on Linux and MacOS. On Windows it is a bit complicated (and possibly needs the intervention of your IT team). This will MASSIVELY CHANGE quite soon: Windows is currently pre-releasing a fully functioning Linux subsystem that will allow Docker to be run with the same speed and efficiency as though you were on a native Linux operating system. You can read all about it on this blog post. Hopefully this will be fully deployed to all Windows 10 users before 2020.

Docker and LSDTopoTools quickstart
  1. Download and install Docker. We will assume this actually is working.

  2. Create an LSDTopoTools directory on your host operating system that you will share with the LSDTopoTools docker containers.

    1. We will assume this is in C:\LSDTopoTools on Windows

    2. Or \LSDTopoTools on MacOS and Linux.

  3. Pull the full LSDTopoTools container and run it with a linked volume:

    1. For windows:

      $ docker run --rm -it -v C:/LSDTopoTools:/LSDTopoTools lsdtopotools/lsdtt_pcl_docker
    2. For MacOS or Linux:

      $ docker run --rm -it -v /LSDTopoTools:/LSDTopoTools lsdtopotools/lsdtt_pcl_docker
    3. Or if you have a different directory to LSDTopoTools data on your host machine:

      $ docker run --rm -it -v /PATH/TO/YOUR/DATA:/LSDTopoTools lsdtopotools/lsdtt_pcl_docker
  4. Once you run this, you will need to run the script:

    # Start_LSDTT.sh
  5. When you are finished with your docker session type ctrl+d.

2.1. Should I use Docker or Vagrant?

Short answer: If you have MacOS, Linux or Windows 10 Enterprise, use Docker. Otherwise use Vagrant.

Long answer: There are many, many, many articles comparing vagrant to docker. The docker setup is probably a bit simpler than the vagrant setup but (and this is a big but) it won’t work on a Windows machine unless you have Windows 10 Enterprise. It does work on any recent version of Linux and MacOS. If you have MacOS, native Linux or Windows 10 Enterprise you should consider using Docker since it involves fewer steps than our VirtualBox+Vagrant setup.

2.2. Installing Docker

Docker is software for creating and managing containers. These containers are not complete operating systems, like ones created using VirtualBox and Vagrant, so the boot time is quicker (usually almost straight away). Docker is perhaps simpler to use but does not work on every operating system.

To set up LSDTopoTools using Docker:

  1. Download and install Docker Community Edition. Instructions on different operating systems are in the Docker documentation.

    1. In Windows follow the instructions in the Docker documentation.

      Firstly, this only works on Windows 10. And you need control of your computer, and you need to know what you are doing. If you have competent IT, ask them to do it. However, this will become MUCH EASIER in a few months when Windows releases its linux subsystem 2, which should allow native docker usage. See this blog post from Microsoft.
    2. On MacOS we recommend installing docker using brew: brew cask install docker

    3. In Ubunutu you can use $ sudo apt-get install docker-ce docker-ce-cli containerd.io

    4. In Debian use $ sudo apt-get install docker-ce docker-ce-cli containerd.io

    5. See the docker website for instructions in your flavour of Linux

  2. We will henceforth assume that you actually have a functioning version of Docker on your host machine. If you want more details about how to use docker, or getting it set up (particularly in Windows, in Linux and MacOS this is more straightforward), see our section on docker notes.

2.3. The four LSDTopoTools images

LSDTopoTools has four different containers. If you are an LSDTopoToools developer you will probably use all four. Most users will only use two of them. The four containers are:

  1. lsdtt-alpine: This is the stripped-down, lightweight LSDTopoTools container. It uses a linux distrubution called the alpine linux distribution. For many users, this is the only container you will need. It is an 88 Mb download and when built, it is a bit under 300 Mb.

  2. lsdtt-pcl: A few of our routines (notably the terrace extraction routines and any routines using swath profiling) require something called the point cloud library (PCL). This library is ENORMOUS and a massive pain to install. It does everything the lsdtt-alpine container does, and more. It is big: the download is 644 Mb and built it is over 2 Gb. However, if you need this one you can use it instead of the lsdtt-alpine container.

  3. lsdtt-viz: This container installs all the python routines you need if you want to use our visualisation tools. Our tools require multiple python packages, meaning that this container is also quite large. If you use a GIS to look at our data, you won’t need this. However we do all our visualisation in python and have written lots of automation scripts, so if you think you will used LSDTopoTools routinely you might want to grab this dockerfile.

  4. lsdtt-docs: Our documentation is all written in asciidoc format, and processed with the extremely awesome asciidoctor. It runs in the Ruby programming language, and requires some stuff to be installed. We have a container for building the documentation so you don’t need to worry about installing the correct stuff. It is very unlikely you will need this unless you are an LSDTopoTools developer.

Which container do I use?
  • Pick either lsdtt-alpine or lsdtt-pcl for the main programs. The former is a small container but can’t do the terrace algorithms. If you anticipate extracting terraces or floodpains use the latter. It is much bigger: lsdtt-alpine is an 88 Mb download and ~240Mb unpacked. lsdtt-pcl is a 664 Mb download and ~2 Gb unpacked.

  • lsdtt-viz is for our python plotting routines. If you just intend to use a GIS to look at LSDTopoTools, you don’t need this. But if you want to reproduce the figures we have in our papers you will need this container. LSDMappingTools needs a lot of python packages so the container is large, I’m afraid.

  • lsdtt-docs: contains the software needed to compile our documentation. The documentation is available on a website (you are reading it) so you only need this container if you are actually going to be writing LSDTopoTools documentation.

How do I get the containers

There are two options for getting the containers!

  • Option 1: Download the images from Docker hub: .

  • Option 2: Build the containers from the raw dockerfiles.

Option 1 is better. Option 2 is really only for LSDTopoTools developers. We went to the trouble of building hte containers so you don’t have to.

2.4. Before you do anything else: Set up a directory to share with docker containers.

When you run LSDTopoTools, or our visualisation routines, you will want to see the resulting files. Docker containers have self contained filesystems that cannot be read on your host operating system unless you tell docker to "share" the directories. These are called "volumes" in docker speak.

  1. First, make a directory in your host operating system where you will put all the files associated with LSDTopoTools. I put mine in C:\LSDTopoTools\

  2. To share a directory in docker you use the -v flag.

    1. Following the -v flag you need to tell it which directories to share.

    2. The format is: /Host/directory:/Container/directory

  3. When you run docker (see below) you will need to ensure that the host directory name is the same as the directory you just created for LSDTopoTools.

2.5. First option: pull the docker images

  1. Okay, this should be fairly easy.

  2. If you want to run the full docker installation from the get go, simply run:

    $ docker run --rm -it -v C:/LSDTopoTools:/LSDTopoTools lsdtopotools/lsdtt_pcl_docker

    This will pull the repo and run it at the same time.

    This container is 663 Mb so if you just want to do some initial testing use the lightweight alpine version!
  3. If you want the lightweight version (but it can’t do floodplains and terraces) use:

    $ docker run --rm -it -v C:/LSDTopoTools:/LSDTopoTools lsdtopotools/lsdtt_alpine_docker
  4. Either one of these commands will result in your command line terminating in the # symbol. This indicates that you are insida a docker container.

  5. Once you are in one of the two docker containers, run the install script:

    # Start_LSDTT.sh

    This script grabs the latest version of the code and builds the programs if you don’t have one that is up to date. It also ensure that the LSDTopoTools programs are in your path.

    You will need to run Start_LSDTT.sh every time you start a new Docker session.
  6. If you want to follow our examples later in this documentation, you should also grab the example data:

    # Get_LSDTT_example_data.sh
    You only need to perform this step the first time you start the LSDTopoTools container.
  7. You can now navigate to your data and start using LSDTopoTools. See our section on first analyses in LSDTopoTools.

    1. If you ran Get_LSDTT_example_data.sh you will now have a data directory /LSDTopoTools/data.

2.6. Second option: build the docker containers

Unless you are a docker purist, please ignore this section and use option 1 instead.
  1. Create a folder where you are going to put files associated with docker. In the examples I will assume these are in C:\LSDTopoTools\ (this is a windows path, since the majority of organisations use the Windows operating system).

    1. If you have git on your computer, you can clone our dockerfiles into this directory.

      $ git clone https://github.com/LSDtopotools/lsdtt_alpine_docker.git
      $ git clone https://github.com/LSDtopotools/lsdtt_pcl_docker.git
    2. If you don’t have git, download and unzip the contents of the repository.

  2. If you are only running LSDTopoTools for routine analysis and looking at the results in GIS software, you will only need one docker container.

  3. We are only going to work with the lsdtt-alpine container. But the steps are basically the same for all the other containers.

  4. Start a terminal (MacOS or Linux) or powershell (Windows) with docker enabled.

  5. Go into the directory inside you docker directory (in my case C:\Docker\LSDTopoTools\) and go into the lsdtt_alpine directory. In windows, if your directory is called the same thing as my directory, you can get there with:

    PS> cd C:\Docker\LSDTopoTools\lsdtt_alpine_docker
  6. Build the docker container:

    PS> docker build -t lsdtt-alpine .
  7. The -t flag in docker stands for "tag". It is the name of the container. You could call it anything you want but all documentation henceforth assumes you have called it lsdtt-alpine.

    1. If you are building a different container, use a different tag, e.g., lsdtt-pcl for the lsdtt-pcl container.

  8. Now you need to run this container:

    $ docker run -it -v C:/LSDTopoTools:/LSDTopoTools lsdtt_alpine_docker
  9. After that, run the install script:

    # Start_LSDTT.sh

    This script grabs the latest version of the code and builds the programs if you don’t have one that is up to date. It also ensure that the LSDTopoTools programs are in your path.

    You will need to run Start_LSDTT.sh every time you start a new Docker session.
  10. If you want to follow our examples later in this documentation, you should also grab the example data:

    # Get_LSDTT_example_data.sh
    You only need to perform this step the first time you start the LSDTopoTools container.
  11. You can now navigate to your data and start using LSDTopoTools. See our section on first analyses in LSDTopoTools.

    1. If you ran Get_LSDTT_example_data.sh you will now have a data directory /LSDTopoTools/data.

2.7. Some notes on Docker

You might want to read the docker documentation, but docker documentation is similar to documentation for the turbo encabulator. Below are some very simple notes to get you started.

Docker quick reference

Here are some shortcuts if you just need a reminder of how docker works.

List all containers

$ docker ps -a

List containsers with size

$ docker ps -as

Remove all unused conainers

$ docker system prune

Complete cleanup: stop everything, remove containers, then remove images.

$ docker stop $(docker ps -aq)
$ docker rm $(docker ps -aq)
$ docker rmi $(docker images -q)

2.7.1. Docker on Linux

After you install docker on Linux, you will need to add users to the docker permissions:

$ sudo usermod -a -G docker $USER

Once you have done this you will need to log out and log back in again.

2.7.2. Docker for Windows

I have not made any scientific study of this but most LSDTopoTools users are on Windows operating systems.

Firstly, you need to have Windows 10 Enterprise. It will not work otherwise. If you don’t have Windows 10 Enterprise but are on Windows you probably should use the Vagrant instructions. If you do have Windows 10 enterprise then you can download and install Docker for Windows CE. After you install this you will need to restart your computer not once but twice: once after intall and a second time to activate the hyper-V feature, which allows you to have 64 bit guest operating systems.

Second, if you have that and have it installed, you also need to add yourself to the docker-users group. To do that, do this (instructions from here: https://github.com/docker/for-win/issues/868):

  1. Logon to Windows as Administrator

  2. Go to Windows Administrator Tools

  3. Look for Windows Computer Management and click on it.

  4. Or you can skip steps 1, right mouse clicking Computer Management, go to more, and select run as administrator and provide Administrator password.

  5. Double click docker-users group and add your account as member.

  6. Also add your account to Hyper-V Administrator. This was added when you installed docker for Windows.

  7. Log off from Windows and log back on.

  8. Click on Windows icon on bottom left and start Docker for Windows. This will start docker windows service.

  9. Start Windows Powershell and type docker --version. It will show Docker version 17.09.1-ce, build 19e2cf6. This is the latest version.

2.7.3. Building a base image

By the time these notes go online, there should be a functioning docker image that you can download from the docker hub. The instructions will be clear and users can just start running LSDTopoTools from within docker. These notes exists as something of a history or cheat sheet of how we made the base image.

Before we do anything, you should look at some notes about running a functioning image of ubuntu: http://phusion.github.io/baseimage-docker/

I have no idea if that stuff is important! But we will try to test images that both include and don’t incude a fully formed startup environment.

The other big issue with docker is the volume syncing. This is more difficult to understand syncing in vagrant.

A simple docker image

So basically for LSDTopoTools functionality we might want a few different base images:

  • An image just for the c++ code.

  • An image just for the python scripts.

  • An image just for building documentation.

  • An image with everything installed.

The reason why we might separate the first three images is because the python image will be HUGE whereas the LSDTopoTools source code is small. So people might want to be offered a slimm version.

Lets start with a very simple python image. The docker documentation for getting you started up is not so useful because they are written to get you running an app in the docker box. For our purposes we want:

  1. To log into to the docker container via ssh or access it as though you have a little linux box running

  2. To be able to access you file system (we don’t want to be having to ftp files in an out of the container each time or worse yet losing all our work when the docker session ends).

For python, we could go with a full python build but that results in very large containers (over a GB). We will follow advice for a minimal python installation. For this we will use the Alpine base box.

  1. First, create a directory that will contain the files you need. I’ll call it python_test.

  2. Go into this directory. You will keep something called a dockerfile in this directory. It has instructions for what stuff is included in the container. We will start with an incredibly simple one. Create the follwoing textfile, called dockerfile:

    FROM python:3.6-alpine
    
    COPY . /app
    WORKDIR /app
  3. Before we do anything we need to build the container. You build it with some name, like:

    $ docker build -t chumbucket .

    In this command, the -t tells docker to tag the container, the chumbucket gives the container its tag, and the . means that you want to call the Dockerfile from this directory.

  4. This downloads some stuff and then hopefully tells you the container is sucessfully built. You can look at it with docker image ls.

  5. You need to actually run the image if you want to do anything with it:

    $ docker run -it chumbucket sh
  6. The -it means interactive and tag and you give it the tag. You then tell it to run the shell (sh). To get out of the shell simply type ctrl-d.

  7. This gets you into a python container, and in addition it has a (very old version of) pip installed. It breaks if you try to install even simple things like pandas. This makes it totally unusable so we are going to have to add some stuff to the docker file. Unfortunately the alpine distro has all sorts of fundamental problems, for example you can’t install scipy.

A slightly more complicated python image

Okay, that container only has a very simple version of python on it. You can then install pip and try to get some packages working, but this is not always straightforward. Your friendly LSDTopoTools developers prefer the miniconda environment, so we are going to build a container on top of the miniconda3 image.

  1. The dockerfile to start looks like this:

    FROM conda/miniconda3
    
    COPY . /app
    WORKDIR /app
  2. Again, we need to build this container in order to use it:

    $ docker build -t chumbucket2 .
  3. We can see what images are available with the call

    $ docker images -a
  4. You will notice the conda image is 228 MB vs the 78.7 MB: the conda version has more bells and whistles involved in getting miniconda to work. But the end result is that you can install things using conda. Let’s update the image with a conda install. Here is the dockerfile:

    # Use the miniconda distribution.
    FROM conda/miniconda3
    
    # Add the app directory and set app to the working directory
    COPY . /app
    WORKDIR /app
    
    # Now add some conda packages
    RUN conda install -y numpy
  5. And we can build this once more (with the same build call so we don’t create a totally new container):

    $ docker build -t chumbucket2 .
  6. Now we can open this container in the interactive mode:

    $ docker run -it chumbucket2 sh
  7. If you run python inside this container and then import numpy as np you can see that it has been installed.

  8. From here you should be able to see that we can install all sorts of stuff from the dockerfile using the RUN conda install command.

    Some Gotchas in simple containers

    So far it seems like smooth sailing? Well there are a few gotchas.

    Firstly, in the command docker run -it chumbucket2 sh you can see we are using sh: this is in most of the "getting started" docker documentation. However, using sh really sucks since it does not feature any of the Linux command line shortcuts you are used to. Instead, you should us bash.

    Secondly even through you have copied the ./app directory into the container, it does not communicate with your storage. So if you update anything in that directory on your home computer, you need to completely rebuild the container!

Mounting volumes in Docker
  1. What we really want is exchange of files between your host operating system and your docker container.

  2. You might start by reading about docker volumes. But like most Docker documenattion that is a bit impenetrable so you might move on to some plain english summaries: summary 1 and summary 2.

  3. The upshot of this is that you don’t really want a docker volume container, but rather you want to link your docker container to your host file system.

  4. On MacOS and Linux this works automatically, but on Windows 10 you need to activate it. Right click on the docker icon in the system tray, go to Settings and then to Shared Drives and activate drive sharing.

  5. Once you have done this you can activate the drive sharing with the -v flag in a docker run command (see below).

An LSDMappingTools container
  1. Okay, lets make a container for LSDMappingTools.

  2. The dockerfile looks like this:

    # This is the LSDMappingTools dockerfile. It is currently set up to install the
    # latest versions of the various required python packages
    
    # To run this you will need to connect it to your host directories if you want to be able to retrieve
    # your figures and get data in and out of the container.
    # On windows you need to allow docker access to the host drive
    # Right click on the docker icon in your system tray
    # Then select "settings"
    # Then "Shared Drives" and activate the drive you want
    # In MacOS and Linux this is automatically set up.
    # After you build the container
    # e.g.
    # docker build -t lsdmt .
    #
    # Then you need to run it with the volume attached
    # e.g.
    # docker run C:/Docker_containers/python_containers/LSDMappingTools:/LSDTopoTools/LSDMappingTools lsdmt bash
    
    # Use the miniconda distribution.
    FROM conda/miniconda3
    
    LABEL MAINTAINERS = "Simon Marius Mudd <simon.m.mudd@ed.ac.uk>, Fiona J Clubb <clubb@uni-potsdam.de>"
    
    # Set LSDTopoTools to be the working directory
    # This will create a directory in the root of the container.
    # When you start the container you will be placed in this directory.
    WORKDIR /LSDTopoTools
    
    # Now add some conda packages
    RUN conda config --add channels conda-forge
    RUN conda install -y numpy matplotlib pandas scipy
    RUN conda install -y gdal shapely fiona rasterio pyproj
    RUN conda install -c conda-forge descartes
    This takes a very long time to set up!
  3. This doesn’t include the actual python scripts! You need to copy them into a directory where docker can access them (on my machine this means withing the C: drive) and then add them using a volume command:

  4. First build the container:

    $ docker build -t lsdmt .
  5. Then run the container in interactive mode with the volume attached:

    $ docker run -it -v C:/Docker_containers/python_containers/python_test1/LSDMappingTools:/LSDTopoTools/LSDMappingTools lsdmt bash
  6. Note that this container has many python packages so it is rather larege: nearly 4GB (to see, type docker ps -as).

An LSD Documentation container
  1. We use asciidoctor for our documentation. I am pleased to say that there is a dockerfile for asciidoctor.

  2. Simply copy the asciidoctor dockerfile and add two lines:

        awesome_print \
        bundler \

    these come after the line tile \ on line 55.

  3. Now build the container

    $ docker built -t asc .
  4. Then open that container up linking to the proper directory. I use:

    $ docker run -it -v C:/Docker_containers/ruby_containers/LSDTT_documentation/master:/documents/LSDTT_documents asc bash
  5. You can find this dockerfile within the LSDTT Documentation repository.

  6. When that is running, go into the documentation directory and run our python script:

    # cd LSDTT_documentation
    # python compile_LSDTTDocs.py
  7. The resulting website will be in the directory html_build.

The LSDTT core code container
  1. We actually have two containers for this since a subset of our code uses PCL which is massive.

  2. The simpler version, used in most of our code, is the following:

    # Pull base image.
    FROM ubuntu:18.10
    
    RUN apt-get update
    RUN apt-get install -y build-essential git gdal-bin python-gdal libfftw3-dev cmake
    
    WORKDIR /LSDTopoTools
  3. We can also use an Alpine linux version, which is much smaller:

    FROM alpine
    MAINTAINER Simon Mudd (simon.m.mudd@ed.ac.uk) and Fiona Clubb (clubb@uni-potsdam.de)
    
    # install essential packages
    RUN apk upgrade -U && \
        apk update && \
        apk add --virtual build-dependencies build-base gcc wget git && \
        apk add --repository http://dl-cdn.alpinelinux.org/alpine/edge/main libressl2.7-libcrypto && \
        apk add gdal --update-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing && \
        apk add fftw-dev cmake && \
        rm -rf /var/cache/apk/* && \
        rm -rf /tmp/*
    
    # update to avoid weird apk error
    RUN apk update
    
    # clone LSDTT repo
    WORKDIR /LSDTopoTools/
  4. After we build this container we can run it with:

    $ docker run -it -v C:/Docker/cpp_containers/LSDTopoTools:/LSDTopoTools/ lsdtt-alpine
The LSDTT PCL container

Some of our routines need the [Point Cloud Library]. This is a HUGE library that needs millions of additional bits of software. The resulting container is enormous, and takes an age to build so you should only use this if you need the PCL versions of LSDTopoTools.

One quite annoying thing is that for anything later than Ubuntu 14 you need to build PCL from source. This takes a long time. The whole installation process takes an age. Sorry.

This is the dockerfile:

# Pull base image.
FROM ubuntu:16.04

# These are the basic build tools
RUN apt-get update
RUN apt-get install -y build-essential git gdal-bin python-gdal libfftw3-dev cmake

# And this is the point cloud library

# install pcl
RUN apt-get install -y libpcl-dev

# We need this as well
RUN apt-get install -y libproj-dev

WORKDIR /LSDTopoTools

3. Installing natively on a Linux system

ONLY DO THIS IF YOU ARE A DEVELOPER!! If you just want to use the code follow the basic instructions.

The compute power of LSDTopoTools is all found within the LSDTopoTools command line tools, which you can install using miniconda:

$ conda -c conda-forge lsdtopotools

The below instructions are for developers who work with the source code. Unless your name is Simon, Fiona, Stuart, Boris, David, Marina, or Martin and are now or formerly associated with the University of Edinburgh’s Land Surface Dynamics group it is highly unlikely you will need any of this.

3.1. Get the associated software you need

  1. Start a terminal session.

  2. Install some essential softare (we will use Ubuntu installation, you need to modify these if you are on another version of Linux). This assumes you have the GNU c++ tools:

    $ sudo apt-get install -y git
    $ sudo apt-get install -y gdal-bin
    $ sudo apt-get install -y python-gdal
    $ sudo apt-get install -y libfftw3-dev
    $ sudo apt-get install -y cmake

3.2. Option 1: Use the script

We made a script for Linux users. It does the following:

  • Makes an LSDTopoTools directory

  • Clones and compiles the code

  • Downloads the example data

  • Spawns a bash shell that knows where the command line tools are.

To use it:

  1. Grab the script

    $ wget https://raw.githubusercontent.com/LSDtopotools/LSDTT_Edinburgh_scripts/master/LSDTT_native_linux_setup.sh
  2. Run the script

    $ sh LSDTT_native_linux_setup.sh
  3. You will need to tell this script where you want to install LSDTopoTools.

  4. The first time you run this it will take a while to compile and get all the files. But you can run it thereafter for any terminal session and it will add the actually LSDTopoTools programs to your $PATH for that session.

  5. If you don’t want to bother with the script each time, we suggest adding the location of the LSDTopoTools prgrams to your $PATH in your .bashrc file. If you don’t know what that previous sentence means, just use the script.

3.3. Option 2: Do it by hand

3.3.1. Get the LSDTopoTools2 code and install

  1. You probably should make a directory where you keep everything associated with LSDTopoTools. A suggested name for this directory is LSDTopoTools: it says what it does on the tin. Go into that directory once you have created it.

  2. Next, clone the LSDTopoTools2 repository here:

    $ git clone https://github.com/LSDtopotools/LSDTopoTools2.git
  3. Go into that new directory:

    $ cd LSDTopoTools2
  4. Run the setup tool

    $ sh lsdtt2_setup.sh
  5. You can now navigate to your data and start using LSDTopoTools. See our section on first analyses in LSDTopoTools.

3.3.2. Starting an LSDTopoTools2 session after your first installation

  • Whenever you return to run an LSDTopoTools2 session, you will need to tell the computer where your program files are. You cave two options:

    • Run sh lsdtt2_setup.sh in the LSDTopoTools2 directory. This gets the latest version of the software, recompiles, and points your session to the program files.

    • Run sh lsdtt2_terminal.sh in the LSDTopoTools2 directory. This just points your session to the program files.

  • You must do one of those two things otherwise your computer will not know where the program files are.

  • As an alternative, you can go into yor .bashrc file and add the /path/to/LSDTopoTools2/bin directory to your $PATH. You will need to change the full path to reflect your own directory system. If you don’t know what a .bashrc file is you are probably better off running the lsdtt2_terminal.sh script.

3.3.3. Getting the example data

  1. Create a data directory. The documentation assumes this directory structure:
    | Your home directory (native linx) or the root directory (docker)
    --|LSDTopoTools
    ----| LSDTopoTools2
    ------| subdirectories with source code and programs
    ----| data
    ------| subdirectories with different projects
    ----| Other LSDTopoTools directories (e.g., LSDMappingTools, LSDTT_documentation)

  2. Go into the data directory and grab the example data from github, then unzip and clean the zip file:

    $ cd data
    $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/archive/master.zip
    $ unzip master.zip
    $ mv ./ExampleTopoDatasets-master ./ExampleTopoDatasets
    $ rm master.zip
  3. You should now have the example data for further examples.

3.4. Installing LSDTopoTools using Windows Subsystem for Linux

Some users have had difficulties getting Docker to install on Windows as it can require changing a setting in the BIOS. If this is the case, you can try to install the code natively using Windows Subsystem for Linux. This is a piece of software from Microsoft that allows you to run a small Linux environment (in our case Ubuntu) from your Windows operating system. We will download this software from the Microsoft Store, which should be preinstalled on any Windows 10 system.

  1. Firstly, download the Ubuntu app from Microsoft Store. If you can’t find it, then you can also get it from this link. You might need to restart your computer as prompted following this step.

The download here is around ~200 Mb, so make sure you have enough space on your system for this. You don’t need to register to the store in order to download the app.
  1. Launch the Ubuntu app by clicking on it from the Start menu. You will then be prompted to create a new user account and password - you can choose whatever you want for this, but make sure you remember it! The password is important for installing packages in the next steps.

  2. The first time you launch, make sure you update and upgrade your standard packages by typing:

    $ sudo apt update && sudo apt upgrade
  3. Now install the dependencies for LSDTopoTools:

    $ sudo apt install -y build-essential git gdal-bin python-gdal libfftw3-dev cmake
  4. Then make a new directory for the LSDTopoTools code and download it from GitHub:

    $ mkdir LSDTopoTools && cd LSDTopoTools
    $ git clone https://github.com/LSDtopotools/LSDTopoTools2.git
  5. This will create a directory called ~/LSDTopoTools/LSDTopoTools2 in your home. Navigate to this directory and run the start up script:

    $ cd LSDTopoTools2
    $ bash lsdtt2_setup.sh
  6. This script will set up the code: we then need one more command to add the binary to your local path. First run:

    $ cd bin
    $ pwd
  7. This will print out a directory path. Copy this directory path, let’s call it /path/to/your/bin for the sake of this example, and run:

    $ export PATH=/path/to/your/bin:$PATH
  8. This will link the bin folder to the system path and make the code executable from anywhere in the Linux subsystem.

3.4.1. Accessing data from Windows

Once installed in the linux subsystem, Microsoft strongly warns against accessing to Ubuntu file system from windows as it can damaged it quite easily. However the opposite is fairly easy: the easiest way to do that is to store data in your C drive, accessible from the linux subsystem with:

$ cd /mnt/c/

4. Installing LSDTopoTools at the University of Edinburgh.

These instructions are for students and staff within the School of GeoSciences at the University of Edinburgh.

Quick Setup for Edinburgh Geoscience users

If it is the first time you are using LSDTopoTools

  1. First, map your network drive. Instructions are below.

  2. In the windows search bar, type "powershell" and start a windows powershell.

  3. Type ssh UUN@ssh.geos.ed.ac.uk.

  4. It will ask you a yes/no question about adding a key. Type yes.

  5. Enter your EASE password. and enter your password. UUN is your university user name.

  6. This puts you into the university ssh server, which is slow. Move into a better one by typing ssh burn.

  7. Download the setup script:

    $ wget https://raw.githubusercontent.com/LSDtopotools/LSDTT_Edinburgh_scripts/master/LSDTT_UEdin_setup.sh
  8. Now run the script:

    $ sh LSDTT_UEdin_setup.sh
  9. The script will ask you for your Edinburgh university username (UUN) and then if you want to install LSDTopoTools in your home directory.

    1. If you are an undergratudate or taught master’s student, answer yes to put LSDTopoTools in your home directory.

    2. For staff and research students, we recommend saying no since this puts the code in your datashare directory, which has more space (I’m afraid taught students don’t have access to this resource).

If you have already followed the instructions above once

  1. Go into the School of GeoScience servers using instructions above.

  2. Run the script:

    $ sh LSDTT_UEdin_setup.sh
  3. You will need to enter your UUN and answer the yes/no question as above, but everything will go very quickly since you’ve already downloaded everything.

4.1. Mapping your network drive

You will need access to your data on the network drive to use LSDTopoTools. You only need to do this one in any computer lab (and possibly only once ever):

  1. Open a file explorer window and under computer select the "University drive":

    The U drive
    Figure 1. The university drive
  2. Then go into the Science and Engineering folder ("SCE"):

    SCE
    Figure 2. The science and engineering folder
  3. Then into the GeoSciences folder ("GEOS"):

    GEOS
    Figure 3. The geosciences folder
  4. Then into the Drive Mapping folder ("drive_mapping"):

    Drive mapping
    Figure 4. The drive mapping folder
  5. And finally double click the shortcut "Access Netdata":

    Access netdata
    Figure 5. The access netdata shortcut

4.2. Getting on to our servers with with ssh in a powershell window

  1. In the windows search bar, type "powershell" and start a windows powershell.

    Starting powershell
    Figure 6. Start powershell
  2. Type ssh UUN@ssh.geos.ed.ac.uk.

    ssh onto server. Use your own UUN
    Figure 7. Go onto a server using the ssh command. You must use your own university username (UUN)!
  3. It will ask you a yes/no question about adding a key. Type yes.

  4. Enter your EASE password. and enter your password. UUN is your university user name.

  5. This puts you into the university ssh server, which is slow. Move into a better one by typing ssh burn.

    You MUST move servers since the GeoSciences ssh server is very old and cannot run the software.
    ssh onto a better server
    Figure 8. Go onto burn. You could also try achray

    Alternatives to this method are programs called MobaXterm, putty, or secure shell client. But basically in all these you need to connect to the server UUN@ssh.geos.ed.ac.uk and then enter your password.

4.3. Downloading and running the setup script

  1. Navigate to your home directory

    $ cd ~
  2. Get the setup script from the internet

    $ wget https://raw.githubusercontent.com/LSDtopotools/LSDTT_Edinburgh_scripts/master/LSDTT_UEdin_setup.sh
    Grab the startup script from the internet
    Figure 9. Grab the startup script from the internet using wget. Brief note: for downloading large files, wget is much faster than downloading files through a browser.
  3. Now run the script:

    $ sh LSDTT_UEdin_setup.sh
    Run the startup script
    Figure 10. Run the script.
  4. Follow the prompts.

Every time you start a new session, you will need to run this setup script. The second time you run it, it will not set up LSDTopoTools, but it will check the software for updates and it will also tell your computer where the actual programs are.

4.3.1. Excruciatingly detailed information about what the script does (only read this if you are an enthusiast that must know how everything works)

This is what the setup script does. You don’t need to do this if you have run the script! It remains here for reference.
  1. For staff and research students, their largest data storage is in their datastore. Undergraduates and taught master’s stuents don’t have access to this, unfortunately. If you tell the script you want things in your home directory, the script will skip this.

  2. Go into your T: drive by entering: cd /exports/csce/datastore/geos/users/UUN

  3. Imagine having to type this every time you go into the servers. It is very annoying. You can cut that step out by making a symbolic link.

  4. In linux you can always navigate to your home directory using:

    $ cd ~
  5. You make a link using the ln command:

    $ ln -s /exports/csce/datastore/geos/users/UUN MyDataStore
    You need to replace UUN above with your university user name.
    You don’t have to call the link MyDataStore, you can call it whatever you like. But try to remember what you called it.
  6. Once you do that, you can navigate into your network drive in Linux with the commands:

    $ cd ~
    $ cd MyDataStore
Don’t be messy
  1. You can organise your directories however you want, but I find it is easier to keep everything contained in a single LSDTopoTools directory.

  2. Go into your network drive (see above).

  3. I suggest making a directory called LSDTopoTools (you can use mkdir LSDTopoTools).

  4. If you want to be clever, you can make an additional link to this new directory:

    $ cd ~
    $ ln -s /exports/csce/datastore/geos/users/UUN/LSDTopoTools LSDTT_Directory
    $ cd LSDTT_Directory
    You need to replace UUN above with your university user name.
  5. If you do that, whenever you want to go into your *LSDTopoTools directory, you just enter this:

    $ cd ~
    $ cd LSDTT_Directory
  6. Do that in order to follow the next steps.

  7. Inside this directory you can put the source code, any data or projects, and the python visualisation code. But in the next step we will just get the source code in set up the main LSDTopoTools programs.

Cloning the LSDTopoTools2 repository

LSDTopoTools2, which has the main analysis programs, lives in a github repository and you use a program names git to get it. Don’t worry, this is easy.

  1. Type (or copy and paste): git clone https://github.com/LSDtopotools/LSDTopoTools2.git

  2. It will download some stuff. Now you need to build the software.

Building the software
  1. Type: sh lsdtt2_setup.sh

  2. Wait a bit until it finishes. It should take a minute or two.

  3. You can now navigate to your data and start using LSDTopoTools. See our section on first analyses in LSDTopoTools.

Starting an LSDTopoTools2 session after your first installation
  • Whenever you return to run an LSDTopoTools2 session, you will need to tell the computer where your program files are. You have two options:

    • Run sh lsdtt2_setup.sh in the LSDTopoTools2 directory. This gets the latest version of the software, recompiles, and points your session to the program files.

    • Run sh lsdtt2_terminal.sh in the LSDTopoTools2 directory. This just points your session to the program files.

  • You must do one of those two things otherwise your computer will not know where the program files are.

  • As an alternative, you can go into yor .bashrc file and add the /path/to/LSDTopoTools2/bin directory to your $PATH. You will need to change the full path to reflect your own directory system. If you don’t know what a .bashrc file is you are probably better off running the lsdtt2_terminal.sh script.

Getting the example data
  1. Create a data directory. The documentation assumes this directory structure:
    | Your home directory (native linx) or the root directory (docker)
    --|LSDTopoTools
    ----| LSDTopoTools2
    ------| subdirectories with source code and programs
    ----| data
    ------| subdirectories with different projects
    ----| Other LSDTopoTools directories (e.g., LSDMappingTools, LSDTT_documentation)

  2. So, if you don’t have a data directory, make one. Again, we suggest putting this in your LSDTopoTools directory. Go into your LSDTopoTools directory (see instructions above):

    $ mkdir data
  3. Go into the data directory and grab the example data from github, then unzip and clean the zip file:

    1. If you are an undergraduate, the script will grab the small samples:

      $ cd data
      $ wget https://github.com/LSDtopotools/SmallExampleTopoDatasets/archive/master.zip
      $ unzip master.zip
      $ mv ./SmallExampleTopoDatasets-master ./ExampleTopoDatasets
      $ rm master.zip
    2. If you have a datastore directory (e.g., are a PGR student or staff) it will get the large example data (since you have more space):

      $ cd data
      $ wget https://github.com/LSDtopotools/ExampleTopoDatasets/archive/master.zip
      $ unzip master.zip
      $ mv ./ExampleTopoDatasets-master ./ExampleTopoDatasets
      $ rm master.zip
  4. You should now have the example data that is used in the rest of this documentation.

5. Extra dependencies for swaths and point clouds

Do I need to read this section?

If you have followed our docker installation you don’t need to read any of this: all the tools mentioned below are included in the LSDTopoTools PCL container.

Some of our tools require the point cloud library. To get this working you need a few different utilities and libraries (take a deep breath):

  1. The cmake utility. This is like make but is required for our tools that examine point clouds, since it is required by something called the point cloud library.

  2. pcl: The Point Cloud Library.

  3. libLAS: a library for working with LAS format data.

Our Vagrant installation installs these automatically. The docker installation (since it is 3rd party) does not. However you can install on an Ubuntu system (and the Docker installation is Ubuntu) using the following instructions.

5.1. Install the Point Cloud Library

If you are using an older version of ubuntu there is a package. Newer versions of Ubuntu don’t have it, so you will unfortunately need to install by hand.

5.1.1. Ubuntu 14 and 12

This is relatively painless (we hope):

$ sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
$ sudo apt-get update
$ sudo apt-get install libpcl-all

5.1.2. Ubuntu 16.04 LTS

First, try this. It is possible it will only work on 64 bit systems:

$ sudo apt-get update
$ sudo apt-get install libpcl-dev

If that doesn’t work you need to install from source. I am very sorry about this.

To install the pcl library from source you use the following steps:

This will take at least an hour!! It will also download an insane amount of stuff onto your machine. You must ensure that you follow the steps in the correct order.

Install oracle-java8-jdk:

$ sudo add-apt-repository -y ppa:webupd8team/java && sudo apt update && sudo apt -y install oracle-java8-installer

Install various dependencies and pre-requisites:

$ sudo apt -y install g++ cmake cmake-gui doxygen mpi-default-dev openmpi-bin openmpi-common libusb-1.0-0-dev libqhull* libusb-dev libgtest-dev
$ sudo apt -y install git-core freeglut3-dev pkg-config build-essential libxmu-dev libxi-dev libphonon-dev libphonon-dev phonon-backend-gstreamer
$ sudo apt -y install phonon-backend-vlc graphviz mono-complete qt-sdk libflann-dev libflann1.8 libboost1.58-all-dev

Install the eigen linear algebra package:

$ cd ~/Downloads
$ wget http://launchpadlibrarian.net/209530212/libeigen3-dev_3.2.5-4_all.deb
$ sudo dpkg -i libeigen3-dev_3.2.5-4_all.deb
$ sudo apt-mark hold libeigen3-dev

Install VTK (this has some plotting functionality, this takes AGES.):

$ wget http://www.vtk.org/files/release/7.1/VTK-7.1.0.tar.gz
$ tar -xf VTK-7.1.0.tar.gz
$ cd VTK-7.1.0 && mkdir build && cd build
$ cmake ..
$ make
$ sudo make install

Now install the Point Cloud Library v1.8:

$ cd ~/Downloads
$ wget https://github.com/PointCloudLibrary/pcl/archive/pcl-1.8.0.tar.gz
$ tar -xf pcl-1.8.0.tar.gz
$ cd pcl-pcl-1.8.0 && mkdir build && cd build
$ cmake ..
$ make
$ sudo make install

Do some clean up:

$ cd ~/Downloads
$ rm libeigen3-dev_3.2.5-4_all.deb VTK-7.1.0.tar.gz pcl-1.8.0.tar.gz
$ sudo rm -r VTK-7.1.0 pcl-pcl-1.8.0

You should now have pcl installed on your system! You can do a small test of the installation if you want to:

$ cd ~
$ mkdir pcl-test && cd pcl-test

Create a CMakeLists.txt file and save this in the directory pcl-test:

cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
project(pcl-test)
find_package(PCL 1.2 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

add_executable(pcl-test main.cpp)
target_link_libraries(pcl-test ${PCL_LIBRARIES})

SET(COMPILE_FLAGS "-std=c++11")
add_definitions(${COMPILE_FLAGS})

Create a file and call it main.cpp:

#include <iostream>

int main() {
    std::cout << "hello, world!" << std::endl;
    return (0);
}

Compile the cpp file and then test it :

$ mkdir build && cd build
$ cmake ..
$ make
$ ./pcl-test

Output should be hello, world!

5.2. Extra libraries for the floodplain and terrace extraction

As of January 2018 liblas does not compile on our vagrant server since liblas3 is not available for Ubuntu 12. We have updated the vagrantfiles to Ubuntu 14 so you should use the new vagrantfiles (this will require downloading a new vagrant box if you were using the old vagrantfile). We don’t use Ubuntu 16 since it is such an incredible pain to install the point cloud library on it.

You might find you need to install a couple of other libraries for the floodplain and terrace drivers to compile properly. If you are using Ubuntu, you can do this using the following commands:

$ sudo apt-get update
$ sudo apt-get install libgoetiff-dev
$ sudo apt-get install liblas-dev

6. Installing the LSDTopoTools python toolchain

LSDTopoTools is written in C++ but also includes a number of automation and visualisation rountines written in python.

Because LSDTopoTools handles primarily spatial data, a number of python packages for handling spatial data are needed in the LSDTopoTools python toolchain. We have attempted to make setting up this toolchain as painless as possible using miniconda and conda environments.

Please see the chapter on LSDTopoTools visualisation for instructions on how to install the LSDTopoTools python toolchain.

The preferred way to install LSDTopoTools is via a Docker container (see previous sections), but because our python tools use a large number of python packages we have a separate container for these tools.

7. Get a text editor!

A text editor isn’t part of LSDTopoTools but you need one to do anything. You will need to modify parameter files and you might even get to the stage of opening Pass:[c++] or Python scripts and editing them so you will need an editor.

If you are Linux native you probably chortle superiorly if you learn someone doesn’t use vi to modify files, but most students have usually only interacted with text files via a word processor.

A word processor (e.g., Word, or whatever fasionable software is on your fruit-based computer) will not work to modify code: these convert your text into binary files and add a bunch of formatting.

You need to work with raw ascii files, and for that you need a text editor. We like these:

  • Brackets is a multi platform text editor (it works on all operating systems) and is free. It has nice file navigation. This documentation was written using Brackets.

  • Atom is another multi platform editor. Similar to Brackets. Also free.

  • PSPad is a freeware text editor (it will ask you occasionally for a donation). I quite like it for Pass:[c++] code and other code since it has a quite nice built-in function navigator. It only works on Windows.