1. Introduction
LSDTopoTools is research software, and unfortunately it can misbehave. In this document we list common problems associated with running of the code.
If you have a problem that is not listed here, please contact the lead developer of LSDTopoTools, and the team will try to find a solution and post it in this appendix.
2. Computational issues
We will list common computational problems here.
2.1. The code doesn’t compile
If the code doesn’t compile, make sure
-
You have installed a Linux operating system.
-
You are actually in that operating system and not in your host operating system (i.e. windows or MacOS).
If you are at Edinburgh you might also get compile problems because you are on an old server. See the Troubleshooting problems on Edinburgh servers section.
2.2. Segmentation faults and core dumps
Segmentation faults are caused when LSDTopoTools attempts to access some data in your computer’s memory (think of it as the computer’s brain) that is not there. Sometimes when this happens you might also get a core dump. These sorts of faults are vexing because the program will not tell you where the problem is when it crashes.
Most of these faults are due to the following three reasons:
-
LSDTopoTools is looking for a file that is not there. You solve this by checking filenames.
-
There is a bug in the LSDTopoTools source code that wasn’t uncovered in testing.
-
LSDTopoTools runs out of memory.
| The most persistent and common problem for our users is when LSDTopoTools runs out of memory. There is not much we can do about this without totally rewriting the code (which is several hundred thousand lines long at this point). If you get a segmentation fault or a core dump there is a high probability that you have run out of memory. See details below, but to fix this you either need to use a smaller DEM or give more memory to your system. |
2.2.1. The missing file problem
In recent years we have added error checking capabilities so that if LSDTopoTools will tell you if it can’t find a file. However we have not done this with every single component of the code so occasionally LSDTopoTools will crash because it can’t find a file. To fix this you need to check all your file names and directory paths.
2.2.2. The naughty developer problem
Sometimes the LSDTopoTools developers release code that has a bug. Bad developers, bad!! To fix this you need to tell us about it and we will try to find the problem. Unless you are a proficient C++ hacker there is not much you can do about this other than to wait for us to address the problem. However, in most cases the crashing software is caused by the memory problem rather than a bug.
2.2.3. The LSDTopoTools memory problem
Let me start by saying that the developers of LSDTopoTools are geoscientists and not professional software developers. When we built LSDTopoTools we were aiming for speed and we don’t know how to do data streaming, so the way LSDTopoTools works is to dump everything into your computer’s memory and work with the data there. This was not a problem for us because we use Linux servers that have may hundreds of gigabytes of memory.
However, topographic datasets are getting bigger and bigger, and we have also tried to distribute LSDTopoTools to people who don’t have a fancy and fast rack workstation, so memory has become an issue.
For most of our applications the following things happen:
-
The DEM is loaded.
-
A filled raster is created.
-
An object containing all the information about the flow network (how all the pixels are connected) is created. This is called the Flow Info object.
-
An object containing how all the river networks and junctions are connected is created.
In addition to these quite standard operations, any additional computation is done in yet more data structures that are in your computer’s memory. The Flow Info object alone is very memory intensive. It takes up 10-20 times the memory of your DEM. What does this mean? Everything together means that you will need something like 30 times the amount of memory as the size of your DEM to do operations involving any sort of flow routing or channel networks.
The memory problem and vagrant machines
For people without a Linux workstation, you need to install with either a container or with a LSDTT_installation.html#_installing_lsdtopotools_using_virtualbox_and_vagrant[virtual machine]. Both of these options give you limited memory. The problem is most severe in the Vagrant setup.
The default vagrantfile setting is to give your server 3GB of memory. This is the line of the vagrant file where it does that:
# Customize the amount of memory on the VM:
# You should adjust this based on your computer's memory
# 32 bit machines generally cannot exceed 3GB of memory
vb.memory = "3000"With this amount of memory you are limited to operations that involve channel networks on DEM of ~100MB. We have not done extensive testing of this but our anecdotal evidence suggests this is the limit with 3GB of memory. If you want to process a 1 GB DEM then you will probably need around 30GB of memory.
If you want to process bigger DEMs you need to give your machine more memory by changing the vb.memory setting in your vagrantfile.
The memory problem and 32-bit vagrant machines
There is one more wrinkle to this. Most Windows machines have a default setting that they cannot have 64-bit guest operating systems. If you try to start up a 64-bit vagrant machine on a windows computer it usually simply will not work. Our testing on MacOS suggests that most of these systems will allow a 64-bit operating system.
Docker allows 64 bit operating systems so this is really a problem limited to Vagrant implementations on Windows machines. If you have Windows 10 Enterprise you might consider using Docker.
There is a limit to the amount of memory a 32-bit system can have; it is around 3GB. So unless you are willing to change your Windows computer’s default settings, you are limited to "small" DEMs. Sorry about that.
3. Troubleshooting vagrant problems
3.1. Vagrant up doesn’t work
Sometimes, on slower machines, vagrant up times out. Just run vagrant up again.
3.2. Vagrant cant see my files
Make sure your the directory structure on the host machine is correct:
| Vagrantboxes
| Ubuntu_vagrant
| vagrantfile
| LSDTopoToolsIf the LSDTopoTools directory is in the wrong place or misspelled (it is case sensitive) then vagrant will not sync your files.
3.3. Vagrant is completely messed up
To start again from scratch, go to the directory with the vagrantfile and run:
$ vagrant destroy| This will erase your Ubuntu box and start again. Meaning you will have to download the 700Mb operating system again. |
4. Troubleshooting problems on Edinburgh servers
4.1. NX doesn’t work
Sometimes, NX decides it won’t work for ambigous reasons. You can always log on to our servers via ssh. An SSH program is avaialble in all computing labs. The address of the geoscience servers is ssh.geos.ed.ac.uk.
4.2. I can’t see my files
-
Undergraduates: Your normal file storage area at Edinburgh is generally refered to as the M drive. You have a second drive, usually called the T drive, that you acces via the netdata path. To activate that path follow these instructions.
-
Taught MSc students: We think this works the same as undergraduates.
-
MSc by research: We think this works the same as PhD students.
-
PhD students/Staff: When you ssh/NX into the servers you will be in your M drive. You will need to create a shortcut to your T drive or and shared drives
5. Troubleshooting Docker
At the moment our Docker installation is provided by a 3rd party. We have not extensively tested this because it does not work on our machines (you need either MacOS or Windows 10 Enterprise). The LSDTopoTools developer who works on a fruit-based operating system, Stuart Grieve, is trying to get an in-house implementation running but for most Docker questions I am afraid there won’t be that much support.
5.1. Setup script won’t work
To fix this problem you need to change permissions on the install script:
chmod 0744 /bin/install.sh