Unix Administration (Winter 2019) Winter 2018

Contact

Dates

Description

Prerequisites

Rules

  1. Refresh your knowledge about Before Class topic before coming to the class. Before Class is volatile and is changed frequently. It becomes stable approximately 12 hours before class. Check it regularly and try to prepare incrementally when something new comes out.
  2. Be curious. If you are not familiar with some command, tool, technology or principle try to learn it on your own. Very good knowledge base is Arch Linux Wiki. There is also tons of tutorials on the Internet and of course manpages. You can also ask me for a detailed explanation.
  3. This course is finished with graded credit (Klasifikovaný zápočet). Your final grade will be based on 1) theory points which measures your activity during the class (questions, quiz, etc.) and 2) the quality of your infrastructure, the ability to fix it when something happens and mandatory homeworks (practical points).
  4. You need at least 7 theory points to be graded. It is possible to get at least 15 theory points during the semester.
  5. Grade 1 is for ≥ 8.5 practical points, 2 ≥ 7.0, 3 ≥ 5.5, otherwise fail. It is possible to get at least 10 practical points during the semester.
  6. Mandatory homeworks have to be done 24 hours before the upcoming class.
  7. You can get extra theory/practice points when you discover any vulnerability, wrong configuration or misbehaving of your colleague student. Also successful attacks count. Contrary if your infrastructure contains obvious shortcomings points will be deducted.
  8. All your machines have to be running all the time and accessible with user teacher, which can run sudo without a password. The access is performed via ssh and this public key.

Notes

Agenda

Date Schedule
2.10.
3.10.

Class Canceled; The lecture starts on the second week of semester. Thus the labs are postponed. Use this time for Before Class preparation of the next class. It is much denser than usual, don’t be scarred.

9.10.
10.10.

Before Class

  • Recall Introduction to Unix class by reading and understanding the Agenda of my Introduction to Unix Labs.
  • What Operating System (OS) do you use? Why? Think about advantages and disadvantages of all OS you know.
  • What devices is Linux suitable/not suitable for? How old is Linux?
  • Refresh your knowledge about TCP/IP networking and be comfortable in shell (history searching, completion, fast command line editing, …).
  • Find a favourite text editor (VIM, nano, emacs, joe, mcedit, …) under Linux and be sure you can comfortably and quickly edit text files.
  • How to copy/paste in Linux? How many clipboards does Linux (X11) have?
  • What is a virtualized OS? Any idea how it works? Read about QEMU virtualization including the Networking part.
  • What is a virtualized network? Any idea how it works?
  • Read ArchLinux Installation guide and identify parts you don’t understand. Try to understand those parts and identify parts you really can’t understand.

On Class

  • Labs organization.
  • Introduction to virtualization infrastructure.
  • Refreshing knowledge about unix, networking and remote working with shell.
  • Redirecting VNC output from the QEMU with ssh forwarding (ssh -L).

After Class (Mandatory homework)

  • Read about OpenSSH.
  • Make the login possible without password, i.e. deploy ssh keys.
  • Create an ssh host alias for an ssh connection to the server. E.g. you will be able to type just ssh c or ssh d and you will be connected to the desired server.
  • Try to install any operating system (Arch Linux would be nice) inside QEMU on the server {c,d,e,f}.
16.10.
17.10.

Before Class

  • What is a virtualized OS? Any idea how it works? Read about QEMU virtualization including the Networking part. Focus on vde.
  • What is the difference between software and hardware virtualization? Can you virtualize different CPU architecture with software virtualization? Can you do it with hardware virtualization (KVM)?
  • What are the parameters for executing qemu with 1 network card connected to the vde-backbone switch on the hypervisor? How to specify amount of RAM, installation ISO image and MAC address in qemu?
  • What is a virtualized network? Any idea how it works?
  • What is a subnet mask, default gateway, IP address and MAC address? What is the difference between IP and MAC address? When do you use IP and when MAC address?
  • Imagine a computer with IP address 1.255.255.0/23. How many IP addresses can be in the same network with this computer?
  • Read ArchLinux Installation guide and identify parts you don’t understand. Try to understand those parts and identify parts you really can’t understand.
  • What is a fork bomb? How does it work? How to prevent your system from users using fork bombs?
  • Imagine you want to clone an existing VM. This means just copy one drive image and you are done. Copy is done with cp command. Read man cp and explain why would cp --reflink=always be helpful in this situation.

On Class

  • First virtual machine (VM) installation.
  • Connecting to the virtual network.
  • Cloning into multiple VMs.
  • Creating your own virtual networks.

After Class (Mandatory homework)

  • Finish the first VM installation. It is going to be a router for your subnetworks.
    • Connect it to the vde_backbone switch. This switch will connect routers of all students together. Qemu option is following: -nic vde,sock=/tmp/vde-backbone.sock,mac=[MAC ADDRESS]
    • Network configuration for your router NIC connected to the vde_backbone:
      • MAC: 52:54:00:36:X:01
      • IP: 10.0.0.X/24
      • GW: 10.0.0.1
      • DNS: 10.0.0.1
      • X = Y + last two UID digits, Y = 10 for c, 30 for d, 50 for e, 70 for f
      • Sum numbers in decimal and use X as fixed string, i.e. do not convert it to HEX for MAC.
    • Network configuration for your router NIC connected to the private switch:
      • IP: 10.0.X.1/24
    • Make the network configuration persistent with systemd-networkd.
  • Create your private virtual switch with vde_switch. man vde_switch is your friend. Save the vde_switch socket to ~/vde-$login-1.sock.
  • Add another network interface card to your router and connect it also to the private switch as described above.
  • Create (clone, just copy the drive image) another VM which will be connected just to the private switch.
    • Think about drive cloning and check man cp. Why cp --reflink=always and cp --sparse=always would be helpful?
    • Network configuration for the second VM:
      • IP: 10.0.X.2/24
      • GW: [YOUR ROUTER]
      • DNS: 10.0.0.1 or [YOUR DNS]
    • Make the network configuration persistent with systemd-networkd.
  • It is mandatory to be able to ping between your router and the second VM, i.e. ping 10.0.X.2 from the router has to work and ping 10.0.X.1 from the second VM has to work.
23.10.
24.10.

Before Class

  • Be familiar how routing works, how to configure network persistently and firewalls. Arch Wiki systemd-networkd, Router, Network configuration, Internet sharing, iptables are good sources for the beginning.
  • Refresh your knowledge about systemd.
  • What is a BGP in the routing context? Try to think, how we can use it in our big network. Take a look at bird, the routing daemon we are going to use.
  • Refresh your knowledge about DNS (types of records, types of servers, how it works…). Check DNS resolving, Unbound and NSD. Unbound and NSD are going to be DNS servers of our choice. Be familiar how they differ.

On Class

  • Solving low entropy state. haveged
  • Installing bird on routers and enabling IP forwarding.
  • Running bird with systemctl.
  • Recommendation about installing dhcp on your “private” interface.
  • Useful tools. ssh -J, tmux/screen, scp, rsync, pssh, terminator, bash/zsh tricks, tcpdump, ngrep, htop, top, mc, ranger, iftop, iotop, systemctl, journalctl

After Class (Mandatory homework)

  • Install recursive DNS server, e.g. unbound on machine 10.0.X.2 and use it as a DNS server for all your machines. No other student should have access to it.
  • If you want to, install dhcp in your “private” network. No dhcp on vde_backbone!
  • Prepare a script or set of scripts which will power on your complete infrastructure after power-failure (everything is down). I.e. the script will create a new switch and execute all the VMs. It should behave intelligently, e.g. when switch is already running, it will kill it or skip its creation.
30.10.
31.10.

Before Class

  • Refresh your knowledge about systemd. Be able to write the most basic systemd .service, .unit, .timer out of your head and be familiar where to place these files and how to start/enable them.
  • Understand how DNSSEC works.
  • Install and try DNS debugging tools drill and dig.
  • Read about SELinux. Understand just basic concepts.

On Class

  • (30.10.) Server d outage, oops. Homework from last week will be checked during next labs.
  • New points (7. and 8.) in the Rules section.
  • systemd service files
  • Authoritative DNS
  • tcpdump

After Class (Mandatory homework)

  • Add user teacher as described in Rule 8.
  • Create another network 10.0.100+X.0/24, i.e. also new switch ~/vde-$login-2.sock.
  • Configure authoritative nameservers (NSD) for $login.una.
    • rt.$login.una. has to have A record 10.0.0.X
    • gw.$login.una. has to have CNAME record rt.$login.una.
    • ns1.$login.una. has to have A record 10.0.X.10
    • ns2.$login.una. has to have A record 10.0.100+X.10
    • Create A records for the rest of your machines.
    • Create also PTR records for all your machines.
    • Make ns1 primary DNS and ns2 secondary DNS and configure AXFR zone transfer between ns1 and ns2.
  • Check if you can resolve your records directly from both authoritative nameservers, i.e. ask your authoritative nameserver directly for some record.
  • Check if your recursive server responds for queries about your domain/other student’s domains.
  • Check if recursive server 10.0.0.1 responds for queries about your domain/other student’s domains.
6.11.
7.11.

Before Class

  • Understand how DNSSEC works.
  • What is iptables and nftables? What is fail2ban? Is the network traffic filtering done in user-space by some program or in kernel-space by the kernel?

On Class

  • DNSSEC

After Class (Mandatory homework)

  • Sign your zones with ldns-keygen and ldns-signzone from ldns package. Read the manpages first.
  • When signing, be aware of expiration date.
  • .una and .0.10.in-addr.arpa are now signed. trusted-una.key
  • Put files with DS records for $login.una, X.0.10.in-addr.arpa and 100+X.0.10.in-addr.arpa to gw.$login.una:/home/teacher/{una.ds,ptr.ds,ptr2.ds}. One zone in one file with one line. DS records from your files are updated in the una. zone every 5 minutes. If there is no such file, DS record is erased.
  • There is very nice attempt (actually 2 different attempts of 2 students) on forging SSH, DNSSEC keys by your colleague. Try to identify what is going on and you can get extra point when you send me e-mail with detailed description of the attack and possible outcome for the attacker. Also describe possible ways of defence. First 3 students will get the point. (It is no longer possible to get the point.)
13.11.
14.11.

Before Class

  • Understand how e-mail works. What is MUA, MDA and MTA?
  • Try to think about attacks mentioned in the previous After Class section.

On Class

  • There is Before lecture for lectures. English and Czech lectures are now merged and starts according to the official schedule, i.e. Monday 17:20, S4. Please attend the lectures, all the concepts will be explained there instead of at the labs. Labs will be more practical now.
  • Attacks (BGP stealing, SSH MITM), e-mail, e-mail etiquette, smtp, imap.
  • Mailing and Posting Etiquette

After Class (Mandatory homework)

  • Create 2 new machines (mail servers)
    • mx1.$login.una. IN A 10.0.X.20
    • mx2.$login.una. IN A 10.0.100+X.20 (UNUSED SO FAR)
  • Install and run postfix on mx1 and mx2.
  • Be able to delivery email (just local delivery) to $login@mx1.$login.una. Test it via telnet mx1.$login.una 25.
  • Be able to read e-mails by mail command on mx1.
  • Just follow Postfix Local mail setup. Basically just small changes are needed in the configuration file.
  • There will be follow-up at the lecture.
18.11.
20.11.
21.11.

Before Lecture

On Lecture

  • E-mail cont’d. Postfix virtual setup. Dovecot. SpamAssassin. DKIM. SPF. PGP (GnuPG).

On Labs

  • Labs Canceled; There is an MFF Open Day on 21.11. hence the labs are canceled. The cancellation applies for both classes because of the synchronization.

After Labs (Mandatory homework)

  • I do care about my teaching performance and your profit from great course. Because if you do not enjoy the course, the learning is shallow, therefore useless in a long-term. Please invest few minutes into providing me a feedback about this course and send me an e-mail with the evaluation of this course (teaching style, usefulness of the knowledge, difficulty, do you enjoy it, what do you hate, etc.). The more information you provide, the better for the course. Hence better for you. You can create a one-time e-mail account to stay anonymous. P.S.: I know that the course is difficult, time consuming and full of self-learning but I believe in this way of teaching.
  • [Deadline in 2 + 1 weeks] Configure virtual mail setup on mx1, i.e. postfix + dovecot and be able to deliver to users which are not present on your system from the whole Internet (actually una. network). Create user $login@$login.una.
    • Be able to receive emails via SMTP (port 25).
    • Be able to download emails via IMAP (port 993).
    • Be able to send emails via submission (port 587).
    • Configure also mx2 as a backup mail server, i.e. just postfix with proper relay_domains option.
    • Configure some MUA (e.g. thunderbird) to send/receive emails. Use it for email communication with your colleagues. Hint: Use port forwarding to use MUA on your laptop.
    • offlineimap and msmtp can help with debugging.
25.11.
27.11.
28.11.

Before Lecture

  • Don’t forget to provide the feedback specified in the previous sections.
  • Refresh your knowledge about drive partitioning (MBR, GPT), system booting (BIOS, EFI), boot loader concept (eg. grub), traditional filesystems (fat, ext), RAID and utilities for manipulating the partition table (fdisk, etc.).

On Lecture

  • Storage (Classical Stack). Drive partitioning, LVM, LUKS, MD-RAID, traditional (no CoW) filesystems.

On Labs

  • Storage (Classical Stack).

After Labs (Mandatory homework)

  • Create 10 drives, 1GB each.
  • Deploy MD-RAID on all drives (RAID5 or RAID6).
  • Encrypt the whole RAID block device with LUKS.
  • On the encrypted block device create LVM.
  • Create 3 logical volumes on top of LVM.
  • Format each logical volume with a filesystem (ext4, xfs are recommended).
  • Fill the array with data, simulate drive failure and drive replacement.
2.12.
4.12.
5.12.

Before Lecture

On Lecture

  • Storage (Modern Stack). CoW Filesystems. Btrfs and ZFS.
  • Automation (Deployment, Configuration Management). Ansible.

On Labs

  • Storage (Modern Stack). Btrfs.
  • Ansible.

After Labs (Mandatory homework)

  • Mailing homework postponed (+1 week).
  • Start thinking about topics you would like to learn in last 2 weeks.
  • Create 10 drives, some with different sizes. Format them with btrfs (data raid5, metadata raid1). Enable compression and autodefrag. Create several subvolumes.
  • Fill the array with data, simulate drive failure and drive replacement. Don’t forget to balance data.
  • Install snapper or btrbk and set up snapshoting every hour.
  • Write a script for sending 10 last snapshots to a different machine.
  • While mounted and running change the raid level to raid10 for data and dup for metadata.
  • Create ansible playbook for setting up gw, ns1, ns2, mx from scratch. NixOS is a valid replacement for ansible task.
9.12.
11.12.
12.12.

Before Lecture

  • Identify differences between full virtualization (QEMU/KVM) and containers (Docker, LXC). Focus on performance, HW accessibility, security and platform independence. Find scenarios where one is clearly beneficial than the other.

On Lecture

  • Containers Theory. Cgroups. Docker. LXC. Monitoring.

On Labs

  • Docker. Collectd. Prometheus.

After Labs (Mandatory homework)

  • Finish thinking about topics for last 2 weeks and let me know.
  • Install collectd (as slave) on all your machines.
  • Install collectd (as master) to collect data from all machines.
  • Install prometheus as a Docker image.
  • Configure prometheus to monitor availability of your machines.
  • Configure some front-end for ploting data from collectd.
16.12.
18.12.
19.12.

Before Lecture

  • How would you build a system for storing 1PB (petabyte) of data? And what about 1EB (exabyte)?

On Lecture

  • Distributed FS (GlusterFS).

On Labs

  • Distributed FS (GlusterFS, Ceph, DRBD)

After Labs (Mandatory homework)

  • Choose 3 machines and use them as nodes for GlusterFS Volume.
  • Each node provides one brick.
  • Create a volume with 3 replicas.
  • On machine 1 create and start an arbitrary VM/Container called HA1.
  • Simulate failure of machine 1 and try to start HA1 on machine 2 or 3.
  • While successfully pinging HA1 I would like to see shutting down HA1 on machine 1 and booting it up on machine 2. The ping will continue to respond when HA1 boots up on machine 2.
  • Do the same with DRBD instead of GlusterFS. Be carefull, you will be working on different layer.
  • Compare and explain differences between both technologies.
6.1.
8.1.
9.1.

Before Lecture

  • Learn the differences between IPv4 and IPv6. (It is not just a bigger address.)
  • Read paper about WireGuard, the next generation kernel network tunnel and forget about IPSec.

On Lecture

  • Optional quiz on lecture. Something from WireGuard paper.
  • IPv6, VPN

On Labs

  • IPv6, Network Redundancy, Advanced Firewall Concepts, VPN, Mikrotik Router OS

After Labs (Mandatory homework)

  • Create a VPN (preferably Wireguard) between 2 of your machines from different networks.
  • Make your infrastructure reachable via IPv6.
  • Your IPv6 for backbone: 2001:718:1e03:822::X/80
  • Your IPv6 subnets: 2001:718:1e03:822:X::/80 and 2001:718:1e03:822:1X::/80
  • It is not necessary to create all DNS records for IPv6 infrastructure, try just few AAAA records.
9.1.
11:00
  • Extra quiz on systemd at my office. This is the last quiz. As a preparation, explore all systemd features.
  • Please fill in the student’s survey in SIS! It is very important for evaluation of teacher’s performance. Thank you!