Category: Guide

This tutorial provides step by step instructions on how to install Ubuntu 20.04 with a GNOME environment.

As announced on the Ubuntu official website, Canonical has just released a brand new version of Ubuntu on the 23th of April 2020.

Named Focal Fossa (or 20.04), this version of Ubuntu is a long term support version offering the following new features :

• The GNOME (v3.36) environment is available by default when installing Ubuntu 20.04;
• Ubuntu 20.04 uses the v5.4 of the Linux kernel : as a reminder, it adds some interesting memory management improvements as well as compatibility with many different hardware devices;
• Ubuntu toolchain has been upgraded : it now uses Python 3.8.2, OpenJDK 11 (or JavaSE 11), GCC 9.3 and recent versions of NGINX and Apache;
• Ubuntu desktop environment has seen some improvements and now looks more recent and modern;
• Improvements have been made on the ZFS filesystem such as native encryption, improved device removal and pool TRIM.

Now that you have a complete idea of the new features of Ubuntu 20.04, let’s see how you can install Ubuntu easily on your computer or server.

Here are the steps to install Ubuntu 20.04.

Create a Bootable Ubuntu 20.04 on Linux

Unless you are working on a virtualized environment (such as VirtualBox), you will need to create a bootable USB stick in order to start the installation.

As a quick reminder, if you are working on Windows, you will have to use Rufus but the steps are essentially the same.

The Ubuntu 20.04 ISO file is available on the official Ubuntu website.

• How to Install Nvidia Drivers on Ubuntu 20.04
• How to Install Python Pip on Ubuntu 20.04
• How to Install Eclipse IDE on Ubuntu 18.04

When visiting the website, you essentially have two options :

• Download an ISO file containing a desktop environment: this is suited for personal computers as well as system administrators not very familiar with the shell environment.
• Download an ISO file for server administration: you may need to choose this version if you don’t need the desktop environment and rather have features such as an SSH server installed by default.

For this tutorial, we are going to use the ISO file containing a desktop environment.

First of all, make sure to plug the USB stick where you want the ISO file to be stored : the ISO file is about 2.5 GBs in size, so make to choose a large enough USB stick.

Plug the USB stick in the USB port

When inserting the USB stick, your distribution should automatically detect the new device.

You can also check that your device was correctly mounted by running the following command

$ df

In this case, the USB stick is linked to the “sdc” filesystem and mounted on the “/home/devconnected/usb” mountpoint.

If you are not very familiar with filesystems and partitions, make sure to read our guides on mounting filesystems.

If you have any doubts, you can also use the “lsblk” command in order to list all block devices available on your computer.

$ lsblk

Download Ubuntu 20.04 ISO file

In order to download the ISO file, you have two options : you can either download it directly from the website provided before or use the wget command in order to download the file.

As a reminder, we are going to install the image containing the desktop environment.

In order to download it, execute the “wget” command followed by the link to the ISO file.

$ wget https://releases.ubuntu.com/20.04/ubuntu-20.04-desktop-amd64.iso

After a while, the ISO file should be downloaded on your machine.

Copy the image to your USB drive

Now that the ISO is downloaded, it is time to copy it to your USB drive.

$ sudo dd bs=4M if=ubuntu-20.04-desktop-amd64.iso of=/dev/sdc && sync

Boot on the USB drive

Now that your USB stick is ready, you are ready to boot on it.

Depending on the BIOS you are using, you may need to press one of the following keys to escape the normal boot : ESC, F1, F2 or F8.

Install Ubuntu 20.04 from your USB drive

Now that you booted directly from your USB drive, you should see the following screen.

Ubuntu will simply start to check files and their integrity to make sure that you have the correct ISO version.

When this step is done, you will be prompted with this window : you can either try Ubuntu or install Ubuntu.

In our case, we are going to install Ubuntu.

Follow Ubuntu installation steps

On the next screen, you are immediately asked to choose a keyboard layout for your computer.

For the example, we are going to choose “English (US) > English(US)”, however feel free to choose the layout that you feel comfortable with.

When clicking on “Continue“, you are asked the type of installation that you want.

As we are targeting a rather minimal installation of Ubuntu, we are not going to install the third-party software suggested. We can install them later on when needed.

Stick with the defaults and click on “Continue“.

Next, you are asked for the partitioning that you want to perform on your instance.

Usually, you can stay with an automatic partitioning (which is essentially “Erase disk and install Ubuntu“).

If you are an advanced system administrator, and if you want to have your own partitioning, you can go for the other option.

However, even if you are going for an automatic partitioning, make sure to have LVM setup from the beginning.

Also, if you are not very comfortable with LVM and dynamic volume management, make sure to check our advanced guide on LVM.

Disk encryption will not be used, however if you need this security, make sure to check the encryption box.

Now that everything is ready, you can click on “Install Now” at the bottom of the previous window.

A confirmation window will open asking if you are sure with all the changes that are going to be performed. Read carefully this window as you won’t be able to go back after clicking on “Continue“.

Essentially, the installation wizard will create one partition (named sda) on your computer and it is going to create two volume groups : one for your root filesystem and one for the swap partition.

Click on “Continue” when you have carefully read the modifications.

On the next screen, choose your location on the map by clicking directly on it and click on “Continue“.

Finally, you are going to create a user for your instance, make sure to choose a name, a computer name (in order to identify it over the network) as well as a secure password.

If you are working on a local network, make sure to choose a computer instance name that is unique, otherwise you might create conflicts related to DNS.

From there the installation will start.

Reboot your system on installation complete

When the installation is done, you will be prompted with a window asking you to restart now.

Click on “Restart now” and wait for the system to start booting up again.

Boot in your new Ubuntu image

Now that the system is started, the user you created in the installation process is automatically selected.

Use the credentials specified before and log in into your newly created account.

Once you are logged in, pretty much like in the CentOS installation, you will be asked to connect your online accounts (meaning email and social accounts).

For this tutorial, we are not going to connect any accounts, but feel free to do it if you want to.

Livepatch setup

Next, you will be asked if you want to install Livepatch. In short, Livepatch is used to install critical features that would usually require a system reboot.

In production environments, this is a crucial feature as your server might be the host for websites or applications that cannot be easily interrupted.

To install Livepatch, simply click on “Set up Livepatch” in the initialization wizard.

You will be asked to provide your credentials in order to complete this step.

Also, you will be asked to have an Ubuntu account in order to setup Livepatch. If this is not the case, make sure to create a Canonical account if you don’t have one already.

In this case, we already have a Ubuntu account, so we are simply going to connect to it.

When this is done, Livepatch will be enabled and active on your system.

Awesome, you can click on “Next“.

Complete final installation steps

You are almost at the end of the process.

Then, you are asked if you want to send statistics and computer information directly to Canonical.

For this step, you can choose if you want to send information or not, but for the tutorial, we are going to choose not to do it.

Click on “Next“.

Again, you are asked if you want applications to have your geolocation. I would recommend to have this option desactivated for now.

If an application needs your geolocation, you will be prompted with it anyway. As stated, “privacy controls can be changed at any time from the “Settings” application.”

Ready to go!

You have successfully completed the Ubuntu 20.04 installation, and you will be able to have all those amazing tools via the “Software” application.

Going Further with Ubuntu 20.04

Now that your server is completely installed, you are ready to create your users and install software that will be needed throughout your Ubuntu journey.

Update your Ubuntu server

First of all, after installing your server (especially if you installed Livepatch in previous steps) you will be prompted with some basic software update.

Simply click on “Install” and wait for the updates to be completely installed.

At the end of the installation, you will be required to restart your server.

If you don’t have this option available, note that you can install update manually by using the APT package manager.

# Updates cache with newly available software
$ sudo apt-get update

# Applies updates downloaded with the previous command
$ sudo apt-get upgrade

Adding your first user

In order to add your first user to your Ubuntu 20.04 server, use the following command

$ sudo adduser <user>
$ sudo passwd <user>

Adding a user to administrators

In order to add a new user to administrators, use the following command

$ sudo usermod -aG sudo <user>

Conclusion

In this tutorial, you learnt how you can easily install the most recent Ubuntu version : Focal Fossa or Ubuntu 20.04.

This tutorial covers the very first steps of server creation, however you might want to set up a SSH server or install reverse proxies on your server. Those aspects will be covered in our upcoming tutorials.

Until then, we recommend that you familiarize yourself with your new distribution, more specifically with the software installer that contains all the useful software.

Monitoring disk I/O on a Linux system is crucial for every system administrator.

When performing basic system troubleshooting, you want to have a complete overview of every single metric on your system : CPU, memory but more importantly a great view over the disk I/O usage.

In our previous tutorial, we built a complete Grafana dashboard in order to monitor CPU and memory usages.

In this tutorial, we are going to build another dashboard that monitors the disk I/O usage on our Linux system, as well as filesystems and even inodes usage.

We are going to use Prometheus to track those metrics, but we will see that it is not the only way to do it on a Linux system.

This tutorial is split into three parts, each providing a step towards a complete understanding of our subject.

• First, we will see how disk I/O monitoring can be done on a Linux system, from the filesystem itself (yes metrics are natively on your machine!) or from external tools such as iotop or iostat;

• Then, we will see how Prometheus can help us monitoring our disk usage with the Node exporter. We are going to set up the tools, set them as services and run them;

• Finally, we are going to setup a quick Grafana dashboard in order to monitor the metrics we gathered before.

Ready?

Lesson 1 – Disk I/O Basics

(If you came only for Prometheus & the Node Exporter, head over to the next section!)

On Linux systems, disk I/O metrics can be monitored from reading a few files on your filesystem.

Remember the old adage : “On Linux, everything is a file“?

Well it could not be more true!

If your disks or processes are files, there are files that store the metrics associated to it at a given point in time.

• Windows Server Monitoring using Prometheus and WMI Exporter | How to Install WMI Exporter in Windows?
• Complete Node Exporter Mastery with Prometheus | Monitoring Linux Host Metrics WITH THE NODE EXPORTER
• Monitoring Linux Processes using Prometheus and Grafana | Prometheus & Grafana Linux Monitoring

A complete procfs tour

As you already know it, Linux filesystems are organized from a root point (also called “root”), each spawning multiple directories, serving a very different purpose for a system.

One of them is /proc, also called procfs. It is a virtual filesystem, created on the fly by your system, that stores files related all the processes that are running on your instance.

The procfs can provide overall CPU, memory and disk information via various files located directly on /proc :

• cpuinfo: provides overall CPU information such as the technical characteristics of your current CPU hardware;
• meminfo: provides real time information about the current memory utilization on your system;
• stat: gathers real time metrics on CPU usage, which is an extension of what cpuinfo may provide already.
• /{pid}/io: aggregates real time IO usage for a given process on your system. This is very powerful when you want to monitor certain processes on your system and how they are behaving over time.

> sudo cat /proc/cpuinfo
dev@dev-ubuntu:/proc$ sudo cat cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz

As you guessed it, Linux already exposes a set of built-in metrics for you to have an idea of what’s happening on your system.

But inspecting files directly isn’t very practical. That’s why we will have a tour of the different interactive tools that every sysadmin can use in order to monitor performances quickly.

5 Interactive Shell Utilities for Disk I/O

Practically, a sysadmin rarely inspects files on the proc filesystem, it uses a set of shell utilities that were designed for this purpose.

Here a list of the most popular tools used to this day :

iotop

iotop is an interactive command line utility that provides real time feedback on the overall disk usage of your system. It is not included by default on Linux systems, but there are easy-to-follow resources in order to get it for your operating system : https://lintut.com/install-iotop-on-linux/

To test it and see it live, you can execute the following command:

# > sudo iotop -Po

# > sudo iotop -Po --iter 4 >> /var/log/iotop <

You should now see an interactive view of all the processes (-P) that are consuming I/O resources on your system (-o).

iotop command

Results are pretty self explanatory : they provide the disk read usage, the disk write usage, the swap memory used as well as the current I/O used.

Handy!

The Other Contenders : iostat, glances, netdata, pt-diskstats

iotop is not the only tool to provide real time metrics for your system.

Furthermore, iotop requires sudo rights in order to be executed.

This list would probably deserve an entire article on its own, but here are alternatives that you can use to monitor your disk usage :

• iostat: features the disk usage for your devices, especially your block devices;
• glances: which is a cross platform system monitoring that showcases real time metrics for your entire system over an accessible Web UI;
• netdata: that was already introduced in our best dashboard monitoring solutions and that provides a plugin to access disk usage;
• pt-diskstats: built by Percona, pt-diskstats is a utility that retrieves disk usage and formats it in a way that can be easily exported and used by other external tools for analysis.

Glances command
Now that you know a bit more about how you can natively monitor your disks on Linux systems, let’s build a complete monitoring pipeline with Prometheus and the Node Exporter.

Lesson 2 – Node Exporter Mastery

Besides the tools presented above, Prometheus can be one of the ways to actively monitor your disks on a Linux system.

If you not familiar with Prometheus, do not hesitate to check the definitive guide that we wrote available here.

As a quick reminder, Prometheus exposes a set of exporters that can be easily set up in order to monitor a wide variety of tools : internal tools (disks, processes), databases (MongoDB, MySQL) or tools such as Kafka or ElasticSearch.

In this case, we are going to use the Node Exporter (the github documentation is available here).

The Node Exporter is an exporter designed to monitor every single metric that you could think of on a Linux system : CPU, memory, disks, filesystems and even network tracking, that is very similar to netstat.

In our case, we are going to focus on everything that is related to disks : filesystem and global disk monitoring, represented by the filesystem and diskstats collectors.

a – Installing the Node Exporter

Prometheus installation was already explained in our previous guide, head over to this link to see how this is done.

Once Prometheus is completely set up and configured, we are going to install Node Exporter as a service on our instance.

For this installation, I am using an Ubuntu 16.04 instance with systemd.

From root, head over to the /etc/systemd/system folder and create a new node exporter service.

> cd /etc/systemd/system/
> touch node-exporter.service

Before configuring our service, let’s create a user account (prometheus) for the node exporter.

> sudo useradd -rs /bin/false prometheus

Make sure that your user was correctly created. The following command should return a result.

> sudo cat /etc/passwd | grep prometheus

When created, edit your node service as follows.

[Unit]
Description=Node Exporter
After=network.target
 
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=[path to node exporter executable]
 
[Install]
WantedBy=multi-user.target

Make sure that you correctly reload your system daemon and start your new service.

> sudo systemctl daemon-reload
> sudo systemctl start node-exporter

# Optional : make sure that your service is up and running.
> sudo systemctl status node-exporter

Congratulations! Your service should now be up and running.

Note : Node Exporter exposes its metrics on port 9100.

b – Set up Node Exporter as a Prometheus Target

Now that the node exporter is running on your instance, you need to configure it as a Prometheus target so that it starts scrapping it.

Head over to the location of your Prometheus configuration file and start editing it.

On the “scrape_configs” section of your configuration file, under “static_configs”, add a new target that points to the node exporter metrics endpoint (:9100 as a reminder from previous section)

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
            - targets: ['localhost:9100']

Make sure to restart your Prometheus server for the changes to be taken into account.

# Service
> sudo systemctl restart prometheus

# Simple background job
> ps aux | grep prometheus
> kill -9 [pidof prometheus]
> ./prometheus &

Head over to Prometheus Web UI, and make sure that your Prometheus server is correctly scrapping your node exporter.

c – Start Exploring Your Data

Now that your Prometheus server is connected to the node exporter, you will be able to explore data directly from Prometheus Web UI.

We are going to focus here on metrics related to disk usage, just to make sure that everything is set up correctly.

In the expression field, type the following PromQL query :

> irate(node_disk_read_bytes_total{device="vda"}[5s]) / 1024 / 1024

As a quick explanation, this query provides a rate of the disk read operations over a period of 5 seconds, for my vda disk in megabytes per second.

If you are able to see data in the graph, it means that everything is correctly set up.

Congratulations!

Lesson 3 – Building A Complete Disk I/O dashboard

Now that our Prometheus is storing data related to our system, it is time to build a complete monitoring dashboard for disk usage.

As a quick reminder, here’s the final look of our dashboard:

Our final dashboard is made of four different components:

• Global tracking of filesystems: given all the filesystems mounted on a given partition, we will have a complete tracking of all the space available on our filesystems;
• Write and read latency of our disks: as we have access to read and write times and the overall number of reads and write operations actually completed, we can compute latencies for both metrics;
• Check of the number of inodes available on our system;
• The overall I/O load rate in real time

As always, the different sections have been split so make sure to go directly to the section you are interested in.

a – Filesystem tracking

Our first panel will be monitoring filesystems and more precisely the overall space remaining on various filesystems.

The node exporter exports two metrics for us to retrieve such statistics : node_filesystem_avail_bytes and node_filesystem_size_bytes.

One divided by the other will give us the overall filesystem usage by device or by mountpoint, as you prefer.

As always, here’s the cheatsheet:

b – Read & Write Latencies

Another great metric for us to monitor is the read and write latencies on our disks.

The node exporter exports multiple metrics for us to compute it.

On the read side, we have:

• node_disk_read_time_seconds_total
• node_disk_reads_completed_total

On the write side, we have:

• node_disk_write_time_seconds_total
• node_disk_writes_completed_total

If we compute rates for the two metrics, and divide one by the other, we are able to compute the latency or the time that your disk takes in order to complete such operations.

Let’s see the cheatsheet.

c – Number of inodes on our system

Now that we know the read and write latencies on system, you might want to know the number of inodes still available on your system.

Fortunately, this is also something exposed by the node exporter:

• node_filesystem_files_free
• node_filesystem_files

Following the same logic, one divided by the other gives us the number of inodes available on our instance.

Here’s the Grafana cheatsheet:

d – Overall I/O load on your instance

This is an easy one.

As a global healthcheck for our system, we want to be able to know the overall I/O load on your system. The node exporter exposes the following metric :

• node_disk_io_now

Computing the rate for this metric will give the overall I/O load.

The final cheatsheet is as follows:

Congratulations!

You built an entire dashboard for disk I/O monitoring (with a futuristic theme!)

Bonus Lesson : custom alerts for disk I/O

Visualizations are awesome, but sometimes you want to be alerted when something happens on your disks. If you followed the definitive guide on Prometheus, you know that Prometheus works with the Alert Manager to raise custom alerts.

Samuel Berthe (@samber on Github), and creator of awesome-prometheus-alerts made a very complete list of alerts that you can implement in order to monitor your systems.

Here are the rules related to disk usage, all credits goes to @samber for those.

- alert: UnusualDiskReadRate
  expr: sum by (instance) (irate(node_disk_read_bytes_total[2m])) / 1024 / 1024 > 50
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Unusual disk read rate (instance {{ $labels.instance }})"
    description: "Disk is probably reading too much data (> 50 MB/s)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

- alert: UnusualDiskWriteRate
  expr: sum by (instance) (irate(node_disk_written_bytes_total[2m])) / 1024 / 1024 > 50
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Unusual disk write rate (instance {{ $labels.instance }})"
    description: "Disk is probably writing too much data (> 50 MB/s)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

- alert: OutOfDiskSpace
  expr: node_filesystem_free_bytes{mountpoint ="/rootfs"} / node_filesystem_size_bytes{mountpoint ="/rootfs"} * 100 < 10
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Out of disk space (instance {{ $labels.instance }})"
    description: "Disk is almost full (< 10% left)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

- alert: UnusualDiskReadLatency
  expr: rate(node_disk_read_time_seconds_total[1m]) / rate(node_disk_reads_completed_total[1m]) > 100
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Unusual disk read latency (instance {{ $labels.instance }})"
    description: "Disk latency is growing (read operations > 100ms)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

A Word To Conclude

Towards this tutorial, you learned that you can easily monitor disk I/O on your instances with Prometheus and Grafana.

Monitoring such metrics is essential for every sysadmin that wants concrete clues of server bottlenecks.

Now you are able to tell if they are coming from read latencies, write latencies or a filesystem that is running out of space.

With Disk I/O, we just scratched the surface of what the node exporter can do, there are actually many more options and you should explore them.

If you created a very cool dashboard with the tools shown above, make sure to show it to us, it is always appreciated to see how creative you can be!

I hope that you learned something new today.

Until then, have fun, as always.

Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY

This tutorial focuses on how to create swap space on Ubuntu 20.04 using a swap partition or a swap file.

For system administrators, it is quite common for servers to run out of RAM.

In some cases, you may have launched too many programs or some of the programs are already using too many resources.

In this case, you don’t have many options.

You can either stop some processes in order to gain some RAM, but you may not be able to do that.

Fortunately, you have another solution : adding some swap space.

In this tutorial, you will learn how you can add swap space on Ubuntu 20.04 : via a swap partition or a swap file.

Prerequisites

Before starting, make sure that you have sudo privileges on Ubuntu 20.04.

$ groups

If you see the “sudo” groups in the group list, you should be able to create a swap partition.

Check Existing Swap Partitions

In order to check existing swap partitions on your system, you have to use the “swapon” command.

$ sudo swapon --show

By default, during your Ubuntu installation, it is very common for a swap partition to be created already.

As a consequence, you should see the following output on your console.

As you can see, I own a swap partition named “dm-1” of size 980 Mb.

However, my current RAM is sufficient for my server, so it is not used at all.

• Fdisk Command in Linux (Create Disk Partitions)
• How To Add Swap Space on Debian 10 Buster
• How To Install and Configure Debian 10 Buster with GNOME

If no swap spaces are configured on your server, you should see the following output.

Add Swap Space on Ubuntu 20.04 using a swap partition

The first way to add swap space on Ubuntu is to add a swap partition.

In order to add this partition, we are going to use :

• the “lvcreate” command if you are on a LVM device (meaning a disk partition formatted as LVM)
• the “fdisk” command if you are on a “regular” disk.

a – Create Swap Partition on a LVM device

First of all, you need to inspect the existing partitions on your host.

$ sudo fdisk -l

As you can see here, I already own three partitions :

• /dev/sda1 : my bootable partition as I am dealing with a MBR disk here;
• /dev/sda2 : an extended partition of size 14.5 GBs;
• /dev/sda5 : my primary LVM partition.

Knowing that, you can either create a swap partition on the extended partition or on your primary LVM partition.

In this case, we choose to create this partition on the LVM one.

As a consequence, we will have to check that we have some space for it.

As a reminder, /dev/sda5 is a disk partition, also called a physical volume, linked to a volume group.

In order to check if we have some remaining space, we need to use the “vgs” command.

If you are not familiar with LVM, I recommend that you read first our dedicated article on the subject.

$ sudo vgs

In this case, we have about 980 Mb of remaining space which should be enough for our swap partition.

In order to create a new logical volume on this volume group, we are going to use the “lvcreate” command.

$ sudo lvcreate -L 500M -n lvswap vgubuntu

Note : you may be prompted to delete a swap signature on a given logical volume. You can press “y”, you are asked this information because there was an existing partition with the same name before.

You can verify that your logical volume was correctly created using the “lvs” command.

$ sudo lvs

b – Create Swap Partition on a regular partition

You can skip this section and go to the next one if you already created your partition on a LVM device.

If you are dealing with a “regular” Linux partition, you will have to use the fdisk command.

In our case, our “regular” (Linux type) partition is named “/dev/sda2“.

$ sudo fdisk /dev/sda

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): n

To create a new partition, you have to use the “n” option.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p) :

You can choose to create a “primary” partition. If you have more than four primary partitions, then you can create a logical one.

Select (default p) : p
Partition number (2-4, default 2):

You can leave the defaults in this case.

On the next prompt, you are asked to specify the first sector, you can also leave it to the defaults.

All space for primary partitions is in use
Adding logical partition 5
First sector (48291898-65062911, default 48291840):

Finally, you are asked to specify the total size of the partition to be created. In our case, we are going to choose a 1GB swap space (or “+1G“).

Last sector, +/-sectors or +/-size{K,M,G,T,P} : +1G

Created a new partition 5 of type 'Linux' and of size 1 GiB.

By default, your partition will be created with a “Linux” type by default.

You want to change that to a swap partition type.

To achieve that, type “t” for “type” and change it to “82” (for swap)

Command (m for help): t
Partition number (1,2,5, default 5): 1
Hex code (type L to list all codes): 82

Hit Enter, and make sure that your partition type was correctly changed.

Changed type of partition 'Linux' to 'Linux swap / Solaris'

Don’t forget to write your changes to the disk using the “w” command.

Command (m for help): w

The partition table has been altered
Syncing disks.

c – Enabling your swap partition

First, make sure to run the mkswap in order for the swap headers to be correctly set for the partition.

# For a regular partition
$ sudo mkswap /dev/sda5

# For a LVM partition
$ sudo mkswap /dev/vgubuntu/lvswap

Now that headers are set, it is as simple as activating your swap space using “swapon“.

# For a regular partition
$ sudo swapon /dev/sda5

# For a LVM partition
$ sudo swapon /dev/vgubuntu/lvswap

Now that your swap is on, make sure to list your swap partitions again with the “–show” option.

$ sudo swapon --show

Awesome, you successfully created a swap partition on Linux!

d – Make your swap space permanent

As any other partitions, you need to make your partitions permanent.

If you were to restart your server just now, changes would not be persisted.

First of all, you need to get your UUID for the newly created partition.

$ sudo blkid | grep "swap"

Copy the UUID value and edit the fstab file to append the following changes.

$ sudo nano /etc/fstab

UUID=<copied value>   none   swap  defaults   0   0

Save your file, and restart your system to make sure that your changes are permanent.

$ sudo reboot
$ sudo swapon --show

Awesome, your changes are now permanent!

Add Swap Space on Ubuntu 20.04 using a swap file

Another common way of adding space is to create a file dedicated to it.

Many tutorials are not specifying this detail, but swap files cannot contain any holes at all.

It means that you should not use the cp command on your host to create a swap file.

For this, you are going to use the “dd” command that makes sure that you don’t have any holes in your file.

a – Create a swapfile using dd

To add swap space, run the following command

$ sudo dd if=/dev/zero of=swapfile bs=1MB count=$((1*2014))

In this case, you are going to create a 2Gb swap space.

Make sure that your swap file was created by issuing the following command.

$ ls -l swapfile

b – Secure your swapfile with permissions

Swap files are meant to be used for memory optimization purposes.

As a consequence, they should not be edited or modified in any way.

As a consequence, we are going to change its permissions.

$ sudo chmod 600 swapfile

In this case, the file is read-write only for the root account.

c – Enable your swapfile

In order to enable your swap file, you can use the “mkswap” command followed by the “swapon” one.

$ sudo mkswap swapfile
$ sudo swapon swapfile

Awesome, you successfully created your swap file!

The last thing that you have to do is to make it permanent using fstab.

d – Make your swap file permanent

To make your swap space permanent, edit the /etc/fstab file and paste the following changes.

$ sudo nano fstab

<path_to_file>/swapfile none swap defaults 0 0

Save your file and restart to make sure that the swap space is still mounted.

$ sudo reboot
$ sudo swapon --show

You have successfully added some swap space on your Ubuntu 20.04!

Troubleshooting

When adding swap space on Ubuntu 20.04, you may run into the following error.

• swapon: /swapfile: read swap header failed.

This error is happening when you don’t run the mkswap command before running the swapon command.

As a reminder, mkswap sets the header for the file or the partition to be used as swap space.

If you forget to run the mkswap command, Linux won’t be able to assign it as swap space on your host.

• Failed to find logical volume “vg/lv”

This error happens when you deleted a logical volume that was defined to be loaded in your initramfs configuration.

In order to fix that, you have two choices :

• Run “update-initramfs -u“

• Make sure that no logical volumes were originally mentionned in the “resume” file of your initramfs.

Icons made by Freepik from FlatIcon.

This tutorial focuses on setting up and configuring an SSH server on a Ubuntu 20.04 desktop environment.

As a system administrator, you are probably working with SSH on a regular basis.

Short for Secure Shell, SSH is a network protocol used in order to operate remote logins and commands on machines over local or remote networks.

SSH comes as an evolution to the Telnet protocol: as its name describes it, SSH is secure and encrypts data that is transmitted over the network.

As a power user, you may want to onboard new machines with SSH servers in order to connect to them later on.

In this tutorial, we are going to see how you can install and enable SSH on Ubuntu 20.04 distributions.

We are also going to see how you can install OpenSSH on your fresh Ubuntu distribution.

• How To Generate Git SSH Keys | Process of Git Generate SSH Key on Windows, Linux, Mac
• How To Setup SSH Keys on GitHub | How to Generate SSH Keys Windows & Linux?
• How to Install Nvidia Drivers on Ubuntu 20.04

Prerequisites

In order to install a SSH server on Ubuntu 20.04, you need to have sudo privileges on your server.

Note : there are no practical differences between adding a user to sudoers on Ubuntu and Debian.

To check whether you have sudo privileges or not, you can launch the following command.

sudo -l

If you see the following lines on your terminal, it means that you currently belongs to the sudo group.

User user may run the following commands on server-ubuntu:
    (ALL : ALL) ALL

Alternatively, you can run the “groups” command and verify that “sudo” is one of the entries.

groups

By default, SSH should already be installed on your host, even for minimal configurations.

To check that this is actually the case, you can run the “ssh” command with the “-V” option.

ssh -V

As you can see, I am currently running OpenSSH 8.2 on Ubuntu with the OpenSSL 1.1.1 version (dated from the 31th of March 2020).

Be careful : this information does not mean that you have a SSH server running on your server, it only means that you are currently able to connect as a client to SSH servers.

Now that all prerequisites are met, let’s see how you can install an OpenSSH server on your host.

Installing OpenSSH Server on Ubuntu 20.04

First of all, as always, make sure that your current packages are up to date for security purposes.

sudo apt-get update

Now that all packages are up-to-date, run the “apt-get install” command in order to install OpenSSH.

sudo apt-get install openssh-server

This command should run a complete installation of an OpenSSH server.

From steps displayed on your console, you should see the following details :

• A configuration file is created in the /etc/ssh folder named sshd_config;
• Symbolic links are created : one named sshd.service (your systemd service) and one in the multi-user target (to boot SSH when you log in).

As stated earlier, a SSH service was created and you can check that it is actually up and running.

sudo systemctl status sshd

By default, your SSH server is listening on port 22 (which is the default SSH port).

If you want to go into further details, you can actually check that the SSH server is listening on port 22 with the netstat command.

netstat -tulpn | grep 22^

(if you don't have netstat by default)
sudo apt-get install net-tools

Awesome!

Your SSH server is now up and running on your Ubuntu 20.04 host.

Enabling SSH traffic on your firewall settings

If you are using UFW as a default firewall on your Ubuntu 20.04 host, it is likely that you need to allow SSH connections on your host.

To enable SSH connections on your host, run the following command

sudo ufw allow ssh

If you are not sure if you are actively using the UFW firewall, you can run the “ufw status” command.

sudo ufw status

Enable SSH server on system boot

As you probably saw, your SSH server is now running as a service on your host.

It is also very likely that it is instructed to start at boot time.

To check whether your service is enable or not, you can run the following command

sudo systemctl list-unit-files | grep enabled | grep ssh

If you have no results on your terminal, you should “enable” the service in order for it to be launched at boot time.

sudo systemctl enable ssh

Important note : this is not a typo, we are actually enabling the SSH service even if we are referring to the sshd service.

If you look for the sshd.service file, you will actually realize that this is a linked unit file.

As a consequence, you can’t directly enable the sshd service, but you have to enable the ssh one.

Configuring your SSH server on Ubuntu 20.04

Before giving any access to your users, it is important for your SSH server to be correctly configured.

If it is done badly, you are at risk when it comes to SSH attackes and your entire infrastructure can be compromised easily.

By default, SSH configuration files are located in the /etc/ssh folder.

In this directory, you are going to find many different files and folders, but the most important ones are :

• ssh_config : is used in order to configure SSH clients. It means that it defines rules that are applied everytime you use SSH to connect to a remote host or to transfer files between hosts;
• sshd_config : is used in order to configure your SSH server. It is used for example to define the reachable SSH port or to deny specific users from communicating with your server.

In this tutorial, we are going to focus on the server part of the configuration, but you can tell a look at the other files if you are curious about them.

Changing SSH default port

The first step towards running a secure SSH server is to change the default assigned by the OpenSSH server.

Edit your sshd_config configuration file and look for the following line.

#Port 22

Make sure to change your port to one that is not reserved for other protocols. I will choose 2222 in this case.

Be careful when you change your default SSH port, you will have to specify it when connecting to it.

Disabling Root Login on your SSH server

By default, on recent distributions, root login is set to “prohibit-password”.

This option means that all interactive authentication methods are banned, allowing only public keys to be used.

In short, you need to setup SSH keys and to use them in order to connect as root.

However, even if we connect without a password, root login is not recommended : if keys are compromised, your entire host is compromised.

As a consequence, you can set this option to “no” in order to restrict it completely.

#PermitRootLogin

PermitRootLogin no

Restarting your SSH server to apply changes

In order for the changes to be applied, you need to restart your SSH server.

sudo systemctl restart sshd

sudo systemctl status sshd

When restarting it, make sure that the server is correctly listening on the custom port your specified earlier.

This information is available on the last lines of the systemd status command.

You can also use the “netstat” command as we already did in the previous sections.

netstat -tulpn | grep 2222

Connecting to your SSH server

In order to connect to your SSH server, you are going to use the ssh command with the following syntax

$ ssh -p <port> <username>@<ip_address>

If you are connecting over a LAN network, make sure to get the local IP address of your machine with the following command

$ sudo ifconfig

For example, in order to connect to my own instance located at 127.0.0.1, I would run the following command

$ ssh -p 2222 <user>@127.0.0.1

You will be asked to provide your password and to certify that the authenticity of the server is correct.

Exiting your SSH server

In order to exit from your SSH server on Ubuntu 20.04, you can hit Ctrl + D or type ‘logout’ and your connection will be terminated.

Disabling your SSH server

In order to disable your SSH server on Ubuntu 20.04, run the following command

sudo systemctl stop sshd

sudo systemctl status sshd

From there, your SSH server won’t be accessible anymore.

Conclusion

In this tutorial, you learnt how you can install, enable, configure and restart your SSH server on Ubuntu 20.04

With this tutorial, you also learnt how you can configure your SSH server in order for it to be robust enough for basic attacks.

If you are interested in Linux system administration, we encourage you to have a look at our other tutorials on the subject.

This article focuses on installing the latest version of Grafana on an Ubuntu 20.04 server.

Recently, Grafana Labs released a brand new version of Grafana : v7.0

This new version featured a whole set of different features : namely a new panel editor, a new explore function as well as new plugins and tutorials for beginners.

As Grafana evolves a lot since our last tutorial, it is time for us to update the Grafana installation guide for Ubuntu 20.04.

In this tutorial, you will learn how you can install and configure a Grafana instance for your Ubuntu server.

Looking to install Grafana on CentOS/RHEL, we have tutorials for other distributions.

Prerequisites

In order to install Grafana, you will need to have sudo privileges on your instance.

To verify that this is the case, you can run the “groups” command and verify that “sudo” is part of the secondary groups.

$ groups

If you are not sure about the method to get sudo rights on Ubuntu, you can check our dedicated guide on the subject.

Now that you have the correct rights, it is time to link your instance to the Grafana repositories.

• How To Setup Telegraf InfluxDB and Grafana on Linux | TIG Stack Setup on Linux
• Complete Node Exporter Mastery with Prometheus | Monitoring Linux Host Metrics WITH THE NODE EXPORTER
• How To Check SSL Certificate Expiration with Grafana | Grafana SSL Certificate Expiry Check

1. Add the Grafana repositories to your server

First of all, you need to add the Grafana APT repositories in order to be able to install packages from them.

If you already have Grafana repositories, you can skip this section and go to the next one.

First, install packages that are needed for Grafana to run properly : apt-transport and software-properties-common.

$ sudo apt-get install -y apt-transport-https
$ sudo apt-get install -y software-properties-common wget

Add the GPG key to the trusted keys

In order to retrieve Grafana packages in a secure way, you need to add the GPG key to your trusted set of keys.

To achieve that, you need to use the “apt-key“ command.

$ wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

OK

A simple “OK” confirmation should be displayed right after the command.

Add the trusted Grafana repositories

Now that everything is configured, you can add the Grafana repositories to the custom APT repositories of your server.

$ echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

deb https://packages.grafana.com/oss/deb stable main

Awesome, now you can update your distribution packages and verify that it links to the Grafana repositories.

To install Grafana, use the “apt-get install” command.

$ sudo apt-get install grafana

Congratulations, you successfully installed the new Grafana v7.0 on your Ubuntu machine!

However, by default, your Grafana server is not started. You will have to configure it and start it manually, which is the purpose of the following sections.

2. Inspect your grafana-server systemd service

If you are using systemd, Grafana created for you a service called “grafana-server“.

To make sure of it, you can run the following command.

$ sudo ls /usr/lib/systemd/system/grafana-server.service

$ cat /usr/lib/systemd/system/grafana-server.service

[Unit]                                                                                          
Description=Grafana instance                                                                    
Documentation=http://docs.grafana.org                                                           
Wants=network-online.target                                                                     
After=network-online.target                                                                     
After=postgresql.service mariadb.service mysql.service                                          
                                                                                                
[Service]                                                                                       
EnvironmentFile=/etc/default/grafana-server                                                     
User=grafana                                                                                    
Group=grafana                                                                                   
Type=simple                                                                                     
Restart=on-failure                                                                              
WorkingDirectory=/usr/share/grafana                                                             
RuntimeDirectory=grafana                                                                        
RuntimeDirectoryMode=0750                                                                       
ExecStart=/usr/sbin/grafana-server                                                  \           
                            --config=${CONF_FILE}                                   \           
                            --pidfile=${PID_FILE_DIR}/grafana-server.pid            \           
                            --packaging=deb                                         \           
                            cfg:default.paths.logs=${LOG_DIR}                       \           
                            cfg:default.paths.data=${DATA_DIR}                      \           
                            cfg:default.paths.plugins=${PLUGINS_DIR}                \           
                            cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}              
                                                                                                
                                                                                                
LimitNOFILE=10000                                                                               
TimeoutStopSec=20                                                                               
UMask=0027   
[Install]
WantedBy=multi-user.target

It is quite important for you to inspect this file, as it provides many information about the server that you just installed.

From the file, you understand that :

• The Grafana server binary is located at /usr/sbin/grafana-server.
• The file that defines all the environment variables is located at /etc/default/grafana-server
• The configuration file is given via the CONF_FILE environment variable.
• The PID of the file is also determined by the PID_FILE_DIR environment variable.
• Logging, data, plugins and provisioning paths are given by environment variables.

The content of the environment file is the following one :

GRAFANA_USER=grafana

GRAFANA_GROUP=grafana

GRAFANA_HOME=/usr/share/grafana

LOG_DIR=/var/log/grafana

DATA_DIR=/var/lib/grafana

MAX_OPEN_FILES=10000

CONF_DIR=/etc/grafana

CONF_FILE=/etc/grafana/grafana.ini

RESTART_ON_UPGRADE=true

PLUGINS_DIR=/var/lib/grafana/plugins

PROVISIONING_CFG_DIR=/etc/grafana/provisioning

# Only used on systemd systems
PID_FILE_DIR=/var/run/grafana

3. Start your grafana-server service

Now that you have learnt about the Grafana configuration variables and how you can arrange your Grafana server, it is time to launch your service.

In order to start your service, you need to execute the “systemctl start” command on the “grafana-server” service.

$ sudo systemctl start grafana-server
$ sudo systemctl status grafana-server

If your service is set as “Active” and “Loaded” (meaning that it will be started on launch), you should be good to go!

Now that everything is configured, we are going to head over to the Web UI in order to create your first dashboard.

4. Launch Grafana v7 Web UI

To open Grafana, you need to open a web browser and navigate to http://localhost:3000.

As a reminder, Grafana works on port 3000 by default, but it may be different if you are already using this port.

You will be presented with this screen when launching the application for the first time.

By default, the login for Grafana is “admin” and the default password is also “admin“.

You will be asked to change your password via a custom chosen password or a strong generated one.

Choose a password and click on “Submit“.

You should now be presented with the default screen for Grafana v7.0, well different from the v6.2 one!

If you are new to Grafana, you should probably follow the tutorial showcased on the “Welcome page“.

Disable new user registrations

The account that you just created will be used as an administrator account for your server.

However, in some cases, you want to be the only one responsible for new user registrations on your server.

That’s why you should think about disabling new user registrations.

By default, user registration is available at http://localhost:3000/signup.

To disable user registration, head back to the configuration file and disable the following section.

$ sudo vi /etc/grafana/grafana.ini

### Content of grafana configuration file

[users]
# disable user signup / registration
;allow_sign_up = true

# Allow non admin users to create organizations
;allow_org_create = true

# Set to true to automatically assign new users to the default organization (id 1)
;auto_assign_org = true

Change the allow_sign_up setting to false, and restart your Grafana server.

;allow_sign_up = false

$ sudo systemctl restart grafana-server

Now you can test that user registration is correctly disabled on your server.

Note that the page will still be accessible but you will be prompted with an error message when you try to create a new user.

Awesome, you successfully disabled user registration on your server!

Start monitoring targets with Grafana

Using Grafana alone is not very useful but you can connect it to many different data sources.

In Grafana, data sources are defined as plugins that one can install in order to connect to it.

On modern architectures, one can connect to cloud datasources like Cloudwatch or Amazon Timestream.

If you are using InfluxDB or Prometheus, note that we wrote tutorials about it on how to setup Grafana with Telegraf and InfluxDB.

The possibilities with those tools are infinite : you can setup a Windows server monitoring or a realtime process monitoring, it is up to you.

Create Your First Dashboard

In order to create all those monitoring dashboards, you will need to create a simple dashboard first.

If you are new to Grafana dashboards, we wrote an extensive guide on how to create Grafana dashboard, so make sure to check it out!

On Linux, the root account is probably one of the most powerful accounts that there is.

Considered the most privileged account on a Unix system, root can perform any tasks needed for system administration.

Navigating a specific folder, killing any process or deleting a directory, root is so powerful that it has to be managed properly.

In this tutorial, you will learn about the different facets of the root account on Ubuntu 20.04.

You will learn to lock and unlock it, to change its password as well as disabling it when trying to remotely access your machine.

Finally, you will know the difference between the root account and the sudo command that is used quite often.

Prerequisites

For most of the commands used in this tutorial, you will need sudo privileges.

If the sudo group is part of your current groups, it means that you should be able to execute the commands listed below.

If not, make sure to check our guide on how to get sudo rights on Ubuntu 20.04.

Check Lock Status of Root Account

Given your distribution, the root account may or may not be locked by default.

By default, when installing Ubuntu 20.04, you created a user account that got the sudo privileges.

As you can see, by default, the “devconnected” user is in the “sudo” group, which allows it to have temporary root rights if needed.

But what about the actual root account?

To know if your root account is locked or not, you can either check the “/etc/shadow” file or use the passwd command with the “-S” option.

• How To Add A User to Sudoers On CentOS 8 | Centos 8 Add User to Group
• How To Install Docker on Ubuntu 18.04 & Debian 10
• How To Install Logstash on Ubuntu 18.04 and Debian 9 | Tutorial on Logstash Configuration

Inspecting the shadow file

On Linux, the shadow file is a very sensitive file : it contains the encrypted passwords for all the users available on your machine.

As a consequence, its content should never be seen or modified by a regular user.

In our case, we are only going to pay attention to the information related to the root account.

In order to know if the root account is locked or not, look for an exclamation mark in the field that should contain the encrypted password. If there is one, that means that the account is locked.

$ sudo getent shadow root

$ sudo cat /etc/shadow | grep root

If you are curious, this point is actually specified in the documentation when reading the page dedicated to “shadow“.

$ man shadow

Using the passwd command

Usually, the passwd command is used in order to change a user’s password on Linux.

However, the “-S” option can be used in order to display the account “status” information.

$ sudo passwd -S root

When using the “-S” option, you want to pay attention to the second column : it actually displays the status of the account (L for “locked” and P for “usable password“).

In this case, the root account is locked while the regular user account has a password.

Locking & Unlocking Root Account

By default, it is recommended to lock the root account and to use dedicated privileged accounts in order to perform critical operations.

In order to lock the root account, you have to use the “usermod” command with the “-L” option for “lock” and specify the root account.

$ sudo usermod -L root

Make sure to verify that the account is correctly locked by using one of the commands we described in the previous section.

In order to unlock the root account, you have to use the “usermod” command with the “-U” and specify the root account.

$ sudo usermod -U root

Changing the root password

In order to change the root password, you have to use the “passwd” and specify the root account.

$ sudo passwd root

After changing your password, the account will be automatically unlocked.

In order to switch to the root account, you can use the well-known “su” command without any arguments (the default account is root).

$ su -

Disabling Root Login over SSH

In some cases, you want to keep the local root account accessible for administration but disabled for remote access.

If you are accessing your machine over SSH, you should disable root login whenever your server is active.

By default, on recent distributions, root login is set to “prohibit-password”, which means that you can still connect to it using SSH key authentication.

In order to disable it completely, head over to your “/etc/ssh/sshd_config” file and identify the line with “PermitRootLogin”.

#PermitRootLogin

PermitRootLogin no

Of course, make sure to restart your SSH server for the modifications to be taken into account.

$ sudo systemctl restart sshd

Conclusion

In this tutorial, you learnt how you can manage the root account on Linux easily.

You learnt that there are many different ways of checking for the lock status of the root account, using the shadow file or the passwd command for example.

If you are interested in Linux System Administration, we have a complete section dedicated to it on the website, so make sure to check it out!

In today’s tutorial, we are going to take a look at one of the most popular monitoring stacks : Grafana and Prometheus.

Prometheus is a time series database, created in 2012 and part of the Cloud Native Computing Foundation, that exposes dozens of exporters for you to monitor anything.

On the other hand, Grafana is probably one of the most popular monitoring tools.

In Grafana, you create dashboards that bind to datasources (such as Prometheus) in order to visualize your metrics in near real-time.

Grafana & Prometheus natively bind together, so today we are going to see how you can setup Prometheus and Grafana on your Linux system.

This tutorial works for the latest versions of Ubuntu (19.04) and Debian (10 Buster).

Ready?

I – What You Will Learn

By following this tutorial, you are going to learn about the following concepts:

• Installing a Prometheus server on your Linux server.
• How to configure authentication and encryption on Prometheus with a reverse proxy
• How to install and configure Grafana for Prometheus
• How to install your first exporter in Prometheus to monitor Linux system metrics.
• How to import your first monitoring dashboard on Grafana

II – Installing Prometheus on Ubuntu and Debian

First, we are going to see how you can install the latest version of Prometheus and configure it for your Linux server.

a – Download Prometheus

First, head over to the Prometheus downloads page.

Make sure to filter for your operating system and your CPU architecture (in my case Linux and amd64)

Make sure to select the latest stable version, and not the “rc” one, as it is not considered stable enough for now.

Download the archive on your system by clicking on the archive, or by running a wget command if you are using the Terminal.

$ wget https://github.com/prometheus/prometheus/releases/download/v2.11.2/prometheus-2.11.2.linux-amd64.tar.gz

You should now have the tar.gz file on your system.

Untar it to extract the files in the archive.

$ tar xvzf prometheus-2.11.2.linux-amd64.tar.gz

The archive contains many important files, but here is the main ones you need to know.

• prometheus.yml: the configuration file for Prometheus. This is the file that you are going to modify in order to tweak your Prometheus server, for example to change the scraping interval or to configure custom alerts;
• prometheus: the binary for your Prometheus server. This is the command that you are going to execute to launch a Prometheus instance on your Linux box;
• promtool: this is a command that you can run to verify your Prometheus configuration.

We are not going to execute directly the Prometheus, instead we are going to configure it as a service.

It will bring more robustness and reliability in case our Prometheus server were to stop suddenly.

• Complete Node Exporter Mastery with Prometheus | Monitoring Linux Host Metrics WITH THE NODE EXPORTER
• MongoDB Monitoring with Grafana & Prometheus | Mongodb Prometheus Grafana Dashboard
• How To Install Prometheus with Docker on Ubuntu 18.04

b – Configuring Prometheus as a service

First of all, for security purposes, you are going to create a Prometheus user with a Prometheus group.

$ sudo useradd -rs /bin/false prometheus

Make sure to move the binaries to your local bin directory.

I stored my binaries in a Prometheus folder, located on my home directory.

Here’s the command to move them to the bin directory.

$ cd Prometheus/prometheus-2.11.2.linux-amd64/ 
$ sudo cp prometheus promtool /usr/local/bin

Give permissions to the Prometheus user for the prometheus binary.

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus

Create a folder in the /etc folder for Prometheus and move the console files, console libraries and the prometheus configuration file to this newly created folder.

$ sudo mkdir /etc/prometheus
$ sudo cp -R consoles/ console_libraries/ prometheus.yml /etc/prometheus

Create a data folder at the root directory, with a prometheus folder inside.

$ sudo mkdir -p data/prometheus

Give the correct permissions to those folders recursively.

$ sudo chown -R prometheus:prometheus data/prometheus /etc/prometheus/*

Great!

You are now set to create a Prometheus service.

Head over to the /lib/systemd/system folder and create a new file named prometheus.service

$ cd /lib/systemd/system
$ sudo touch prometheus.service

In order to see the different launch options for Prometheus, you can run the prometheus command with a h flag.

$ prometheus -h
usage: prometheus [<flags>]

The Prometheus monitoring server

Flags:
  -h, --help                     Show context-sensitive help (also try --help-long and
                                 --help-man).
      --version                  Show application version.
      --config.file="prometheus.yml"
                                 Prometheus configuration file path.
      --web.listen-address="0.0.0.0:9090"

Edit the newly created file, and paste the following content inside.

$ sudo nano prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path="/data/prometheus" \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-admin-api

Restart=always

[Install]
WantedBy=multi-user.target

Save your file, enable your service at startup, and start your service.

$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus<

Now that we have our Prometheus server running, let’s connect to the Web UI to make sure that everything is okay.

Unless you modified it in the service file, your Prometheus server should be running at http://localhost:9090.

By default, this is what you should see on your screen.

By default, Prometheus should start monitoring itself.

Click on Status, then Targets in top bar menu, and verify that you have one target : the Prometheus server itself.

Great! The state is “UP” for the Prometheus target. Everything is running smoothly.

Now that our Prometheus server is running, it is time to configure a reverse proxy for authentication and encryption.

III – Setting up a reverse proxy for Prometheus

Prometheus does not natively support authentication or TLS encryption.

If you are running Prometheus locally, it doesn’t not matter that much.

But if you were to expose Prometheus endpoints to the outside world, you would want to setup a strong authentication for users to check Prometheus.

For this tutorial, I will be using NGINX as a reverse proxy, but there are many other proxies out there that you can choose from.

a – Install NGINX on Debian 10

The steps are pretty much the same for Ubuntu 18.04/19.04.

Simply install nginx by downloading it from the apt sources.

$ sudo apt update
$ sudo apt-get install nginx

This should create a nginx service on your server, and it should be started by default.

$ sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-07-05 06:01:12 UTC; 1 months 10 days ago
     Docs: man:nginx(8)
 Main PID: 15792 (nginx)
    Tasks: 3 (limit: 4704)
   CGroup: /system.slice/nginx.service
           ├─15792 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;

By default, NGINX will start listening for HTTP requests in the default http port, which is 80.

If you are not sure about the port used by NGINX, run the following command.

The last column will tell you the port used by NGINX.

$ sudo lsof | grep LISTEN | grep nginx
nginx     15792     root    8u     IPv4         1140223421       0t0        TCP *:http (LISTEN)

Great! Let’s do a quick browser check to see the default HTTP page distributed by NGINX.

b – Configuring NGINX as a reverse proxy

In this case, we are not interested in using NGINX as a standard web server.

Instead, we want to use NGINX as a reverse proxy.

As a reminder, a reverse proxy is an entity that proxies requests to other entities given a set of internal rules defined in the proxy configuration.

By default, the NGINX configuration file is located at /etc/nginx/nginx.conf.

In this file, you have to make sure that it includes virtual hosts included in the conf.d directory.

$ cat /etc/nginx/nginx.conf

##
# Virtual Host Configs
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

The conf.d directory is where we are going to create our reverse proxy configuration file for Prometheus.

Create a new file in this directory called prometheus.conf, and start editing the file.

$ cd conf.d/
$ sudo touch prometheus.conf

Paste the following configuration in your configuration file.

server {
    listen 1234;

    location / {
      proxy_pass           http://localhost:9090/;
    }
}

Restart your NGINX server, and browse to http://localhost:1234.

$ sudo systemctl restart nginx
(if you have any errors)
$ sudo journalctl -f -u nginx.service

Great!
Prometheus is now sitting behind a reverse proxy.

c – Changing Prometheus configuration for reverse proxying

Now that Prometheus is accessed by a reverse proxy, we need to change the configuration of Prometheus for it to bind to the reverse proxy. Head over to /lib/systemd/system and change your service configuration file.

$ cd /lib/systemd/system 
$ sudo nano prometheus.service

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path="/data/prometheus" \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-admin-api \
  --web.external-url=https://localhost:1234

Restart your service, and make sure that your server is up and running correctly.

$ sudo systemctl daemon-reload
$ sudo systemctl restart prometheus
$ journalctl -f -u prometheus.service

Awesome!

Make sure that Prometheus is still reachable at http://localhost:1234.

d – Enable reverse proxy authentication

Before moving on with the reverse proxy, let’s have a quick look at how you can set up credentials for your Prometheus instance.

Prometheus does not handle built-in authentication, so we are going to rely on reverse proxy authentication for this.

Create a password file in the Prometheus configuration folder.

If you don’t have the htpasswd executable on your system, make sure to install it first.

$ sudo apt-get install apache2-utils
$ cd /etc/prometheus
$ sudo htpasswd -c .credentials admin

Choose a strong password, and make sure that the pass file was correctly created.

Now, it is time to import it into your NGINX configuration file.

Head over to /etc/nginx/conf.d and change your server configuration by adding your credentials.

$ cd /etc/nginx/conf.d

server {
    listen 1234;

    location / {
      auth_basic           "Prometheus";
      auth_basic_user_file /etc/prometheus/.credentials;
      proxy_pass           http://localhost:9090/;
    }
}

Save your file, and restart NGINX

$ sudo systemctl restart nginx
$ journalctl -f -u nginx.service

If everything is okay, go to http://localhost:1234 and you should be prompted with user credentials.

Provide the correct credentials, and you should be good to go!

e – Enabling HTTPS on the reverse proxy<

For the last chapter, we are going to enable the TLS protocol on our NGINX instance.

Using HTTPS will encrypt requests done to the server, but most importantly it will encrypt the server responses containing all the metrics.

Creating keys for Prometheus

For this chapter, we are going to use self-signed certificates.

However, if you have a certificate authority, you can issue a certificate request to them.

You will need the certtool in order to create certificates on your Linux instance.

Install the gnutls related packages (utils for Ubuntu and bin for Debian based distributions)

(Ubuntu)
$ sudo apt-get install gnutls-utils

(Debian)
$ sudo apt-get install gnutls-bin

Create a new directory in /etc/ssl in order to store your Prometheus keys.

$ cd /etc/ssl
$ sudo mkdir prometheus

Go into the prometheus directory, and start by generating a private key for Prometheus.

$ sudo certtool --generate-privkey --outfile prometheus-private-key.pem

Now that you have a private key, let’s generate a certificate (or a public key) for Prometheus.

$ sudo certtool --generate-self-signed --load-privkey prometheus-private-key.pem --outfile prometheus-cert.pem
  # The certificate will expire in (days): 3650
  # Does the certificate belong to an authority? (Y/N): y
  # Will the certificate be used to sign other certificates? (Y/N): y
  # Will the certificate be used to sign CRLs? (y/N): y

Great!

You now have two keys sitting in your /etc/ssl/prometheus.

Configuring HTTPS on NGINX

Now that our keys are created, it is time to import them into our NGINX configuration.

Go back to your /etc/nginx/conf.d directory, and edit your prometheus.conf file.

$ cd /etc/nginx/conf.d
$ sudo nano prometheus.conf

server {
    listen 1234 ssl;
    ssl_certificate /etc/ssl/prometheus/prometheus-cert.pem;
    ssl_certificate_key /etc/ssl/prometheus/prometheus-private-key.pem;

    location / {
      auth_basic           "Prometheus";
      auth_basic_user_file /etc/prometheus/.credentials;
      proxy_pass           http://localhost:9090/;
    }
}

Again, restart your NGINX server, and your virtual host should now be switched to HTTPS.

$ sudo systemctl restart nginx
$ sudo journalctl -f -u nginx.service

Verify that Prometheus is now delivered via HTTPS

We are going to use cURL in order to verify that Prometheus is now served via HTTPS.

If you don’t own cURL on your instance, make sure to install it first.

$ sudo apt-get update
$ sudo apt-get install curl

$ curl -u admin -k https://localhost:1234/metrics
Enter host password for user admin:

From there, this is what you should see on your screen.

Great! We are now ready to install Grafana to visualize Prometheus metrics.

IV – Installing Grafana

The Grafana installation was already covered extensively in our of our previous articles, so make sure to read it to setup your Grafana server.

a – Configure Prometheus as a Grafana datasource

First, head to the datasources panel by clicking on Configuration > Data sources via the left menu.

Click on “Add a datasource”

Select a Prometheus data source on the next window.

Here is the configuration for Prometheus. Make sure to skip the TLS verification as you are using a self-signed certificate.

Click on “Save and Test” at the bottom of your configuration window, and make sure that your data source is working properly.

V – Installing the Node Exporter to monitor Linux metrics

Now that your Prometheus data source is working, it is time to install your first exporter.

As a reminder, exporters are standalone entities that regularly aggregate metrics for a wide variety of targets : operating systems, databases, websites.

If you feel like you are missing some of the concepts about Prometheus, here’s a guide to understand them.

The Node Exporter is an exporter that gathers metrics abour your Linux system : the CPU usage, the memory usage as well as various statistics on your filesystems.

a – Downloading the Node Exporter

First of all, we are going to download the Node exporter on our system.

Head over to https://prometheus.io/download/ and select Linux operating systems for amd64 CPU architectures.

Scroll down, and find the section dedicated to the node exporter.

Simply click on it, or copy the link and run a wget command.

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

Extract the archive to access the files inside it.

$ tar xvzf node_exporter-0.18.1.linux-amd64.tar.gz

Inside your newly created directory, you should now see the node_exporter binary ready for use.

Again, move this executable to your /usr/local/bin folder.

$ cd node_exporter-0.18.1.linux-amd64/
$ sudo cp node_exporter /usr/local/bin

Create a new user for the node exporter, and change the permissions for the node exporter binary.

$ sudo useradd -rs /bin/false node_exporter
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

b – Create a Node Exporter service

Again, we are going to run the node exporter as a service.

Go to the /lib/systemd/system folder and create a new file named node_exporter.service

$ cd /lib/systemd/system
$ sudo touch node_exporter.service

Paste the following content in your node exporter service.

By default, a lot of modules (cpu, mem, disk) are already enabled by default, so we only need to enable a few more.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
    --collector.mountstats \
    --collector.logind \
    --collector.processes \
    --collector.ntp \
    --collector.systemd \
    --collector.tcpstat \
    --collector.wifi

Restart=always

[Install]
WantedBy=multi-user.target

When you are done, enable your service, and start it.

$ sudo systemctl daemon-reload
$ sudo systemctl enable node_exporter
$ sudo systemctl start node_exporter

(Make sure that everything is running smoothly)
$ sudo journalctl -f -u node_exporter.service

c – Configure the Node Exporter as a Prometheus target

Now that the node exporter is up and running, we need to tell Prometheus to scrape it periodically.

To do that, go to your Prometheus configuration file located at /etc/prometheus/prometheus.yml.

$ cd /etc/prometheus
$ sudo nano prometheus.yml

In the static_configs part of your configuration file, add a new entry for the node exporter.

static_configs:
            - targets: ['localhost:9090', 'localhost:9100']

Restart Prometheus for your changes to be applied.

$ sudo systemctl restart prometheus

Head over to https://localhost:1234/targets and make sure that your target is correctly scraped.

Great! Prometheus is now scraping metrics from the Node Exporter.

VI – Building a Grafana dashboard to monitor Linux metrics

The last step will be to import a Grafana dashboard that reflects the metrics that we are gathering with Prometheus.

The “Node Exporter Full” dashboard created by idealista seems to be a great candidate.

Looking to create your own Grafana dashboard using the UI or the API? Follow the guide!

a – Importing a Prometheus dashboard into Grafana

We don’t have to design the entire dashboard by ourselves, we can import idealista’s dashboard right into our Grafana server.

To do so, click on “Import” by hovering the “Plus” icon, and clicking on “Import.

On the next window, simply enter the dashboard ID in the corresponding field (1860 in our case)

The dashboard should be automatically detected by Grafana.

On the next window, select your Prometheus data source, and click on Import.

You are done! Your dashboard is now up and running.

(If you are not seeing metrics right away, make sure to tweak the time parameter located at the top right corner of the dashboard)

So, where should you go from there?

The possibilities are endless.

You can start by adding more exporters to your Prometheus instance.

The full list of Prometheus exporters is available here.

Also, if you are looking for examples for MongoDB and MySQL, I wrote some tutorials about those specific exporters.

If you are looking to continue with the Node exporter, there is a complete guide about it in the monitoring section.

I hope that you learned something new today.

Until then, have fun, as always.