Unix Administration Labs (2018/2019)

Contact

Dates

Description

Prerequisites

Rules

  1. Refresh your knowledge about "Before Class" topic before coming to the class.
  2. Be curious. If you are not familiar with some command, tool, technology or principle try to learn it on your own. Very good knowledge base is Arch Linux Wiki. There is also tons of tutorials on the Internet and of course manpages. You can also ask me for a detailed explanation.
  3. To obtain credit you need to:
    1. Completely configure infrastructure, i.e. finish all tasks from classes.
    2. Obtain at least 10 points for in class activity. Best strategy is to aim for 1 point per class.
    3. During the semester your infrastructure will be 3 times harmfully changed. You have to fix your infrastructure in at least 2 cases. This has to be done during the class where the infrastructure was broken.

Notes

Agenda

Date Before Class On Class
3.10.
5.10.
  • Refresh your knowledge about TCP/IP networking and basic shell commands you learned in Introduction to UNIX.
  • Find your favourite console text editor (VIM, nano, ed…) and be comfortable in shell (history searching, completion…).
  • Labs organization.
  • Introduction to virtualization infrastructure.
  • Refreshing knowledge about unix, networking and remote working with shell.
10.10.
12.10.
  • Be strong in practical networking and fast in working with shell. If you have problems with working in shell, read [Unix Tutorial].
  • Understand Arch Linux installation guide. [0]
  • Take a quick look into virtualization with QEMU. [2]
  • Take a quick look into network virtualization with VDE. [3]
  • Step-by-step installation of virtual GNU/Linux without installer in QEMU.
  • Virtual networking configuration.
  • Network configuration. MAC:52:54:00:36:X:01, IP: 10.0.0.X/24, GW: 10.0.0.1, DNS: 10.0.0.1, X = Y + ID, Y = 10 for c, 20 for d, 30 for e, 40 for f, ID is last digit from UID (from /etc/passwd)
  • X from previous computation is also used as an increment for VNC port, i.e. your VNC port is 5900+X.
17.10.
19.10.
  • Have fully installed/configured one virtual Arch Linux machine (without networking). All of your machines have to be running all the time!
  • Study man ip and be able to configure MAC, IP addresses and routing.
  • Know how to configure network persistently. [1]
  • Take a deep look into network virtualization with VDE. [3]
  • Take a deep look into routing daemon BIRD (Version 2) and be ready to configure it for BGP protocol. [4]
  • Cloning virtual machines.
  • Virtual networking configuration from previous class. Connecting to vde switch (socket at /tmp/vde-backbone.sock).
  • Router (BIRD) configuration. bird.conf
  • For persistent network configuration use systemd-networkd.
  • Installing and running SSH on your router.
  • Add user teacher which can run sudo without password and assign him this ssh key. This user has to be present on all your machines in the future. For adding user teacher use useradd with -m parameter (it will create also home directory for the user).
  • Network debugging.

24.10.
26.10.
  • Don't forget to add user teacher who can run sudo without password from previous lab. I have to be able to connect to your machine via SSH regarding to performing harmful changes. Your machine has to be running.
  • Be comfortable with everything we did so far, first harmful change is going to probe your knowledge.
  • Refresh your knowledge about DNS.
  • First harmfull change to your infrastructure.
  • Setting hostname of your first machine (gateway) to gw.$login.unixadm.
  • Creating (cloning) second and third machine with hostnames ns{1,2}.$login.unixadm
  • ns1: 10.0.X.2, ns2: 10.0.100+X.2, gw: 10.0.0.X, 10.0.X.1, 10.0.100+X.1
  • Finishing router (BIRD) configuration.
  • Private networks creation. Your first subnet is 10.0.X.0/24 and the second is 10.0.100+X.0/24.
  • Path of your vde switches' sockets: ~/vde-$login-1.sock and ~/vde-$login-2.sock
  • Useful tools for speeding up your work (ssh -J, scp, rsync, pssh, terminator, tmux, screen, bash/zsh tricks, vi keybindings, tiling wm,...).
  • Useful utilities for administration (systemctl, journalctl, htop, top, mc, ranger, iftop, iotop).

31.10.
2.11.
  • Refresh your knowledge about DNS, DNSSEC and firewalls.
  • Feedback.
  • Note about entropy in virtual machines. Haveged installation.
  • Top-level domainname changed from unixadm to una.
  • DNS configuration.
  • Configure recursive DNS server unbound on gw. It has to properly resolve gw.una. Hint:
    forward-zone:
     name: "."
     forward-addr: 10.0.0.1
  • Configure authoritative DNS server nsd on ns{1,2} for your domains, i.e. $login.una, X.0.10.in-addr.arpa and 100+X.0.10.in-addr.arpa. It has to properly resolve for example ns1.$login.una and 2.X.0.10.in-addr.arpa. Add A and PTR records for all you machines (gw, ns1, ns2).
  • ns1 is going to be your master namerserver and ns2 is going to be slave.
  • Configure AXFR zone transfer (synchronization) between master and slave.
7.11.
9.11.
  • Forwarding of port 53 from 10.0.0.0/8 to the Internet is now denied. As a DNS server you should use your own servers with forwarding to 10.0.0.1.
  • Have properly configured BGP server from previous classes.
  • Have properly configured DNS servers for your domains from previous class.
  • Verify your zone $login.una is working: On server {c,d,e,f} do
    drill ns1.$login.una
    and you should get the answer.
  • Refresh your knowledge about DNSSEC and firewalls.
  • Think how to enable DNSSEC on .una domain.
  • Attacks on DNS.
  • DNSSEC configuration.
  • Sign your zones. Be aware of expiration date.
  • Put files with DS records for $login.una, X.0.10.in-addr.arpa and 100+X.0.10.in-addr.arpa to gw.$login.una:/home/teacher/{una.ds,ptr.ds,ptr2.ds}. One zone in one file. DS records from your files are updated in the una. zone every 1 minute. If there is no such file, DS record is erased.
  • Trusted root keys with una zone included here.
  • Enable DNSSEC in your recursive DNS.
  • DHCP server (dhcpd) configuration in your private networks. 10.0.{X,100+X}.3, dhcp{1,2}.$login.una
  • Create new machine and try to assign IP addresses from DHCP.
  • Packet filters (iptables, nftables, tc, shorewall). IP Forwarding. Helpers (fail2ban).
  • Add iptables or nftables to your gateway and add some nice rules to make it secure.
14.11.
16.11.
  • Know how email works. Refresh knowledge about smtp, imap, pop3 protocols and what MUA and MTA are.
  • Working DNS servers with DNSSEC. E-mail without DNS does not work.
  • Know how DKIM and SFP work.
  • Read about following applications: postfix, msmtp, dovecot, spamassassin, rspamd.
  • DNS + DNSSEC wrap-up.
  • E-mail servers configuration.
  • mx1.$login.una IN A 10.0.X.4
  • mx2.$login.una IN A 10.0.100+X.4
  • Postfix configuration for local delivery.
  • Be able to delivery to $login@mx1.$login.una.
  • Be able to read your email by running mail.
21.11.
23.11.
  • Prepared all machines/technologies from previous classes 24 hours before. User teacher has to be everywhere.
  • Be comfortable with everything we did so far, second harmful change is going to probe your knowledge.
  • Second harmful change to your infrastructure. Assignment will be delivered via smtp to mx1.$login.una for $login@mx1.$login.una. Preparation hints:
    • It will require a lot of debugging with drill and thorough understanding of DNS + DNSSEC setup.
    • Read drill's manpage completely.
    • Try to ask for various types of records (A, NS, SOA, PTR, DS, DNSKEY).
    • Try to play with DNSSEC, change DNSKEY, DS records and be able to debug it with drill.
28.11.
30.11.
  • Know how email works. Refresh knowledge about smtp, imap, pop3 protocols and what MUA and MTA are.
  • Working DNS servers with DNSSEC. E-mail without DNS does not work.
  • Know how DKIM and SFP work.
  • Read about following applications: postfix, msmtp, dovecot, spamassassin, rspamd.
  • E-mail servers configuration cont'd. (dovecot, spamassassin, virtual maildirs, dkim, spf, greylisting)
  • Configure postfix + dovecot
  • Be able to deliver to $login@$login.una from anywhere inside una lab.
  • MX records for mx1 and mx2.
  • Establish port forwarding and be able to use your MUA for sending/receiving mails.
  • imap client for debugging offlineimap
  • smtp client for debugging msmtp
  • MTA for reports from local systems (msmtp).
5.12.
7.12.
  • Read ArchWiki Filesystems page and related articles.
  • Clone a new machine called storage and connect 16 drives with some small capacity, e.g. 512MB.
  • Create a machine called storage with 16 drives for data.
  • Don't partition these drives.
  • Create MDRAID on 8 drives and create md0 which will be RAID10. On top of md0 create LUKS. On top of LUKS create ext3 filesystem. Follow [1, 2].
  • Mount it to /mnt/mdraid.
  • On another 8 drives create btrfs filesystem with raid10 for data and metadata. Implement snapshoting solution called snapper(Do snapshots every day). Follow [3, 4].
  • Mount the whole btrfs device to /mnt/btrfs.
  • Create a btrfs subvolume called nfs. Mount the subvolume to /mnt/btrfs-nfs.
  • Use /mnt/btrfs-nfs as an export for NFS server. On ns1 mount that folder to /mnt/nfs. Arch Wiki NFS page.
  • Create machine called backups with arbitrary number of drives.
  • On backups create btrfs filesystem across all drives with arbitrary btrfs raid.
  • Write a script for sending last 5 btrfs snapshots from storage to backups.
  • Optionally: Play with glusterfs.
12.12.
14.12.
  • Review knowledge about OS and network virtualization.
  • Clone new machine called cont.
  • On cont create a container for each of the following technologies: chroot, lxc, systemd-nspawn, docker.
  • Install nextcloud container on cont via docker. Redirect port 80, hence you are able to access nextcloud web interface from your computer/laptop.
  • Create Ansible configuration for automatic configuration of new cloned machine (set networking, add users, install some packages, set hostname, ...).
  • Instead of Ansible configuration from previous task you can try some declarative configuration with NixOS.
  • On all of your machines install collectd.
  • Clone new machine called monitor and receive monitoring information (load, free ram, ...) from all machines running collectd. Store collected information to file or visualise it.
  • Optionally: Clone another 5 machines called cont[1-5]. Implement some nice scenario with Kubernetes and Docker (and glusterfs for example).
  • Optionally: Create Ansible configuration for all your servers (DNS, MX, GW,...).
  • Optionally: Advanced configuration of NixOS.
(Plan)
19.12.
21.12.
  • Prepared all machines/technologies from previous classes 24 hours before. User teacher has to be everywhere.
  • Be comfortable with everything we did so far, third harmful change is going to probe your knowledge.
  • Read about IPv6 and understand it.
  • Read systemd.network manpage and be able to configure IPv6 for client and router.
  • Be able to use knowledge about IPv6 to configure your machine for dual-stack operation (IPv4 + IPv6).
  • Third harmful change to your infrastructure. Assignment will be send to $login@$login.una.
  • Migration of all services to IPv6.
  • Redundancy (routers, DNS, DHCP, MX, bridges, NIC, drives, PSUs, containers, hypervisors).
2.1.
4.1.
  • Class canceled.
  • Class canceled.
(Plan)
9.1.
11.1.
  • Completely configured infrastructure, i.e. have properly configured all tasks from all classes.
  • Be able to reconfigure/add functionality to your infrastructure.
  • Mandatory class: Checking your infrastructure for completeness and asking questions/giving extra tasks to verify your understanding.
  • Credits (Zápočty).

Resources

University of Surrey — Unix Tutorial
Charles University — Introduction to Unix and Introduction to Networking Courses
Manpages
Arch Linux Wiki