Navigate back to main LSDTopoTools landing page

1. Introduction

Your phone and your fancy computer with a fruit-based operating system have lots of nice icons that you tap, click or swipe to get apps and programs. These have been designed by someone with a degree in design from the Glasgow School of Art and a PhD in Computer Science from Stanford. That person usually has a cool Gaelic name like Mhairi but grew up in Hong Kong, Phuket and Hartford, Connecticut and in her spare time has a closed loop organic farm that produces the best Tilapia this side of Manaus.

Unfortunately for you, you need to use some scientific software partially programmed by someone named Mudd who grew up in Riverside. There is no fancy user interface. There are no icons at all! The software runs in Linux. Why do scientists make everything so difficult!!!

Well, it turns out that it is much, much, much easier to install the common tools scientists use within Linux (MacOS is built on top of Linux). Also, although you will not believe it at first, doing things in Linux via the command line is faster than clicking on buttons. There is a reason most programmers still use this interface.

Modern Linux operating systems like Ubuntu or Debian have windowing systems like, uh, Windows and MacOS but in this section we will use something called the shell, which is a command line tool (you type commands onto a line).

If you really want to learn about the shell you should go to the software carpentry website. What you will find here are the commands that are essential for working with our software, LSDTopoTools, which are a subset of what you will find in the software carpentry tutorials.

2. Getting into a Linux shell

Getting a shell window will vary depending on your system.

  • If you are on a native linux system you should already know how to get a window into the shell (it is usually called a terminal window). If you don’t know how to do that google is your friend ("ubuntu terminal").

  • If you are using a virtual machine managed by Vagrant, one you either ssh or use putty.exe to access your virtual machine, you will be in a terminal window. You will find instructions on how to do this along with the Vagrant documentation.

  • If you are using Docker, you will run docker from command line (the Docker instructions will tell you how to do that) and you will end up in a shell within your Docker container.

  • If you are using mobaXterm, like we do at the link::SDTT_installation##_getting_on_to_our_servers_with_mobaxterm[University of Edinburgh], then just start a new local session and then ssh into a server (e.g., ssh burn).

3. I am in a terminal window. Now what?

The first thing you need to know about is figuring out where you are, and how to move around.

If you want to know all about this, go to software carpentry. Here we will give you just the basics.

Note that if you are in a Linux shell you will usually have the dollar symbol ($) as a command prompt. So if you see this:

Shell prompt in native Linux
$

it means you will be entering a shell command.

On Docker, the shell is denoted by a hash symbol

Shell prompt in Docker
#

3.1. Files and directories

Everything in a computer is organised into files. These are collections of numbers and/or text that contain information and sometimes programs. These files need to be put somewhere so they are organised into directories which can be nested: i.e., a directory can be inside of another directory. Linux users are born with this knowledge, but users of more intuitive devices and operating systems are not.

  • If you want to know where you are type pwd

    $ pwd
/usr

    In this case when you type pwd it tells you that you are in the /usr directory. The slash symbol: / separates directories. So if you have /usr/Data/Rabbits it means that the Rabbits directory is in the Data directory, which is in the usr directory.

  • To see what is there in the directory type ls:

    $ ls
a_directory another_directory a.file Data

    Both files and directories are listed when you type ls. If you type ls -l you get loads more information. See software carpentry for more information.

  • To move to a new directory, type cd:

    $ cd another_directory
$ pwd
/usr/another_directory

  • To go up a directory type cd ..

    $ cd ..
$ pwd
/usr

  • You can also jump straight to the directory you want: just type cd and then the full path:

    $ pwd
/I/am/lost/in/a/vortex/of/directories
$ cd /usr/Data/Rabbits
$ pwd
/usr/Data/Rabbits

  • Going to your home directory: type cd ~. Every Linux user has a home directory:

    $ cd ~
$ pwd
/my/home/directory

  • If you want to make a new directory type mkdir and then the directory name.

    $ mkdir a_new_directory

  • To move a file type mv and then the name of the file and the name of the destination:

    $ mv a.data b.data

  • To copy a file type cp and then the name of the file and new file. This produces a duplicate with a new name and/or location.

  • To delete a file type rm and the filename. This is irreversible so be careful!!

  • To delete a directory type rmdir and the directory name. The directory must be empty.

  • To delete a non-empty directory, i.e. all the files and sub-directories within it, type rm -r and the directory name. Be VERY careful where you type this command as it will delete everything in a directory. IF YOU GET THIS WRONG YOU CAN DELETE YOUR WHOLE FILE SYSTEM.

3.2. Useful basic tasks

There are a few tasks that are rather useful. I’ll list two of them here:

  • Downloading files from the internet: type wget and then the web address (with the http:// and all that stuff). This seems to be MUCH faster that downloading stuff via a web browser. It is really useful for big files.

  • Unzipping stuff: type tar -xzf if you have a file with the extension .tar.gz. Again, this is much faster than doing it in a windows file explorer. If you have a .zip file, you can use unzip file.name.

    *

3.3. Keyboard shortcuts

One of the reasons programmers love the shell is that there are loads of shortcuts. This speeds things up when you are working. The most useful ones seem to be:

  • Up arrow lets you scroll through all your past commands.

  • Tab will auto-complete file and directory names. So you only need to type the first few letters of a directory and then tab and it will give you the entire file or directory name. This is SUPER CONVINIENT. This feature almost makes learning linux worthwhile on its own.

  • ctrl-a or the Home key moves the cursor to the beginning of the line.

  • ctrl-e or the End key moves the cursor to the end of the line.

4. Setting up your file system and keeping it clean ESSENTIAL READING IF YOU ARE NEW TO LINUX

The past decade has seen rapid advances in user interfaces, the result of which is that you don’t really have to know anything about how computers work to use modern computing devices. I am afraid nobody associated with this software project is employed to write a slick user interface. In addition, we don’t want a slick user interface since sliding and tapping on a touchscreen to get what you want does not enhance repoducible data analysis. So I am very sorry to say that to use this software you will need to know something about computers.

If you are not familiar with Linux or directory structures this is essential reading.

4.1. Files and directories

Everything in a computer is organised into files. These are collections of numbers and/or text that contain information and sometimes programs. These files need to be put somewhere so they are organised into directories which can be nested: i.e., a directory can be inside of another directory. Linux users are born with this knowledge, but users of more intuitive devices and operating systems are not.

Our software is distributed as source code. These are files with all the instructions for the programs. They are not the programs themselves! Most of the instructions live in objects that have both a cpp and hpp file. They are stored in a directory together. Contained alongside these files are a few other directories. There is always a TNT folder for storing some files for matrix and vector operations. There will always be another directory with some variant of the word driver in it. Then there might be some other directories. Do not move these files around! Their location relative to each other is important!!.

Inside the folder that has driver in the name there will be yet more cpp files. There will also be files with the extension make. These make files are instructions to change the source code into a program that you can run. The make file assumes that files are in specific directories relative to each other so it is important not to move the relative locations of the cpp files and the various directories such as the TNT directory. If you move these files the you will not be able to make the programs.

We tend to keep data separate from the programs. So you should make a different set of folders for your raw data and for the outputs of our programs.

DO NOT put any spaces in the names of your files or directories. Frequently LSDTopoTools asks for paths and filenames, and tells them apart with spaces. If your filename or directory has a space in it, LSDTopoTools will think the name has ended in the middle and you will get an error.

4.2. Where to put your files

The instructions are slightly different for Docker installation and native Linux installation of LSDTopoTools/

  • If you are using Docker, there will be a file system on your host operating sytem, and another inside the Docker continer.

    • Inside the Docker container, the files are in the directory /LSDTopoTools

      • Inside this directory you will have a directory for the source code: /LSDTopoTools/LSDTopoTools2

      • You should also have a directory called data: /LSDTopoTools/data

      • And if you use our python mapping scripts another one with *LSDMappingTools: /LSDTopoTools/LSDMappingTools

    • In your host operating system, you should also have a directory called LSDTopoTools but it is up to you where to put it.

  • If you are doing a native Linux installation (and this can work in MacOS, which is built on top of a Linux system, or in Windows using the Linux)

    • You will need a directory somewhere called LSDTopoTools.

      • Inside this directory you will have a directory for the source code: /LSDTopoTools/LSDTopoTools2. If you know how to modify your .bashrc file, you will want to add to your path the directory /my/path/to/LSDTopoTools/LSDTopoTools2/bin. You need to modify /my/path/to to match your actual directory structure.

      • You should also have a directory called data: /LSDTopoTools/data

      • And if you use our python mapping scripts another one with *LSDMappingTools: /LSDTopoTools/LSDMappingTools

I would HIGHLY recommend a nested structure in your data directory. Make subdirectories organised by location so you are not mixing topographic data.

4.3. OBSOLETE Directories in Vagrant

This is a historical section only for people who have an old Windows (pre-Windows 10) operating system
Only use these instructions if you have an old Windows operating system

If you use our vagrantfiles then you will need some specific directories. Our instructions assume this directory structure so if you have your own Linux box and you want to copy and paste things from our instructions then you should replicate this directory structure.

If you use vagrant there will be directories in your host machine (i.e., your computer) and on the client machine (i.e., the Linux virtual machine that lives inside of you host machine). The file systems of these two computers are synced so if you change a file in one you will change the file in the other!

4.3.1. Vagrant directories in windows

In windows you should have a directory called something like VagrantBoxes. Inside this directory you need directories to store the vagrantfiles, and a directory called LSDTopoTools. The directories need to look like this

|-VagrantFiles
  |-LSDTopoTools
  |-Ubuntu32_FFTW
    | vagrantfile
  |-Ubuntu32
    | vagrantfile
The names VagrantBoxes and Ubuntu32, Ubuntu32_FFTW don’t really matter; you just need to have a place to contain all your files associated with vagrant and within this subdirectories for vagrantfiles. If your vagrantfile is for and Ubuntu 32 bit system that includes FFTW then you might call the folder Ubuntu32_FFTW.
There MUST be an LSDTopoTools directory one level above your vagrantfile. This directory name is case sensitive.

When you make the LSDTopoTools directory it will initially be empty. When you run vagrant up it will fill up with stuff (see below).

4.3.2. Vagrant directories in the client Linux machine

If you use our vagrantfiles to set up LSDTopoTools, they will construct a file system. It will have a directory LSDTopoTools in the root directory and within that directory will be two directories called Git_projects and Topographic_projects. The file system looks like this:

Ubuntu file system
|-LSDTopoTools
  |-Git_projects
    |-LSDTopoTools_AnalysisDriver
      |- lots and lots of files and some directories
    |-LSDTopoTools_ChannelExtraction
      |- lots and lots of files and some directories
    |-More directories that hold source code
  |-Topographic_projects
    |-Test_data
      |- Some topographic datasets

These files are in the root directory so you can get to the by just using cd \ and then the name of the folder:

$ cd /LSDTopoTools/Git_projects

Here is the clever bit: after you have run vagrant up these folders will also appear in your host system. But they will be in your VagrantBoxes folder. So in Linux the path to the git projects folder is /LSDTopoTools/Git_projects and in Windows the path will be something like C:\VagrantBoxes\LSDTopoTools\Git_projects.

4.4. Know where your data is

When you run a makefile it will create a program sitting in some directory. Your data, if you are being clean and organised, will sit somewhere else.

You need to tell the programs where the data is! People raised on smartphones and tablets seem to struggle with this. In many laboratory sessions I have the computational equivalent of this converation: Student "I can’t get into my new apartment. Can you help?" Me: "Where did you put your keys?" Student: "I don’t know." Please don’t be that student.

Most of our programs need to look for another file, sometimes called a driver file and sometimes called a parameter file. We probably should use a consistent naming convention for these files but I’m afraid you will need to live with our sloppiness. You get what you pay for, after all.

The programs will be in your source code folders, so for example, you might have a program called get_chi_profiles.exe in the /LSDTopoTools/Git_projects/LSDTopoTools_ChiMudd2014/driver_functions_MuddChi2014 directory. You then have to tell this program where the driver or parameter file is:

A typical call to an LSDTopoTools program
$ pwd
/LSDTopoTools/Git_projects/LSDTopoTools_ChiMudd2014/driver_functions_MuddChi2014
$ ./git_get_profiles.exe /LSDTopoTools/Topographic_projects/Test_data/ Example.driver

In the above examples, ./git_get_profiles.exe is calling the program.

/LSDTopoTools/Topographic_projects/Test_data/ is the folder where the driver/parameter file is. We tend to keep the topographic data and parameter files together. The final / is important: some of our programs will check for it but others won’t (sorry) and they will not run properly without it.

Example.driver is the filename of the driver/parameter file.

In the above example it means that the parameter file will be in the folder /LSDTopoTools/Topographic_projects/Test_data/ even though your program is in a different folder (/LSDTopoTools/Git_projects/LSDTopoTools_ChiMudd2014/driver_functions_MuddChi2014/).

5. Shell cheat sheet

Here are some resources and commands that we hope you find useful.

5.1. External resources

You can start by watching the videos listed below (in the [Videos] section), but if you want more information there are numerous websites that list common shell commands:

If you want a gentle introduction to the shell, Software Carpentry is an excellent resource.

Software carpentry’s tutorials on the shell are here: http://software-carpentry.org/v5/novice/shell/index.html

5.2. Commands that will save you vast amounts of time

The Linux and GNU developers have spent many hours writing help pages called man pages (short for "manual", as in, "read the manual"). For lots of detail on how to run some of these commands, you can usually type man followed by the name of the command, e.g. man cp. For more concise info, you can usually type COMMAND --help, e.g. cp --help.

Here is a brief reference:

Table 1. Essential commands
Shortcut What it does

cd /directory1

change directory to /directory1. See notes here.

cd ..

Go up a directory level

ls

Lists all the files within a directory

ls -l

Lists all the files with details

ls *pattern

Lists all files ending with pattern (so you could look for python files with ls *.py)

ls pattern*

Lists all files starting with pattern

ls *pattern*

Lists all files with pattern in them.

mv file1 file2

Moves a file. The files can have a directory path

cp file1 file2

Copies file1 to file2. Can include a directory path

mkdir directoryname

Makes a directory with the name directoryname

rm file1

Removes the file with filename file1. The pattern rules using * apply here (see ls commands).

rmdir directoryname

Removes the directory directoryname. It has to be empty. If you want to remove a directory that contains files you need to use rmdir -r directoryname. Be warned: mess this up and you can delete lots of files!! There is no undelete!
Table 2. Commands that will make things go MUCH faster
Shortcut What it does

tab

Autocompletes file and directory names.

ctrl-e

Moves cursor to end of command line

ctrl-a

Moves cursor to beginning of command line

Up and down arrows

Scrolls through history of commands