Introduction to Unix Labs (Summer 2019)

Contact

Dates

Description

Rules

  1. Broaden your knowledge about Before Class topic before coming to the class. Most of the activity points are awarded in the discussion related to this topic.
  2. Be curious. If you are not familiar with some command, tool, technology or principle try to learn and understand it on your own. There are tons of tutorials on the Internet and most of the information is in man pages. After that you can validate your knowledge in discussion with me. This is the most efficient way of learning.
  3. To obtain a credit you need to:
    1. Obtain at least 10 points for in class activity. Best strategy is to aim for 1-2 points per class.
    2. Obtain at least 75% of all points from scripting tasks. It means 66 points this year.
    3. Propose and implement a useful script which solves some real problem. Optimally write a script for your own usage. Create public github repository for your project and develop it there. Save repo URL to project_url.txt in your intro19 repository and send me email, when the project is considered done. Do not forget to create a github title page README.md with description of your project, help and examples of usage. Try to develop it as a real project.

Notes

Follow-up

Homework Submission (Scripting Tasks)

Agenda

Date Schedule
20.2.
22.2.

Before Class

  • Your motivation for UNIX. Who uses UNIX? What are advantages and disadvantages of UNIX? What devices are suitable/not suitable for UNIX? How old is UNIX? What other operating systems do you know? What is github and what is good for?

On Class

  • Labs organization. Motivation. Motivation. Motivation. First touches with UNIX.
  • Login to KDE session. Running firefox and konsole via Alt-F2.
  • Creating a github account.
  • Running file manager mc in konsole. Browsing the filesystem. Check wikipedia for Filesystem Hierarchy.
  • Running text editor nano in konsole. Editing file. Alternative editor can be mcedit which is part of mc.
  • There exist putty program for Windows. You can use it to connect to the unix lab computers. Follow this guide.

Scripting Tasks

  • [2 points] Create github repository named intro19 and send me email with it’s link for cloning. Make the repository private and add github user asch to the collaborators. You can set up email notifications to be notified when I submit points for the task. This task is done from web browser. No command line commands are needed.
  • [01/test2.txt] [2 points] Create folder 01 and in that folder create file test2.txt which contains text “I Love GIT.”. This can be also done from web browser. No command line commands needed.
27.2.
1.3.

Before Class

  • Last time we used KDE Desktop Environment with konsole terminal emulator. What other Desktop Environments you know and what terminal emulators they have? Is it possible to use konsole in different Desktop Environment than KDE? Why/Why not?
  • What other web browsers are installed on UNIX lab computers?
  • How to set shortcuts in KDE? Set shortcut such that Alt+k executes konsole and Alt+f executes firefox. (You can use different shortcuts.)
  • How to copy/paste in Linux?

On Class

  • Trying different Desktop Environments (xfce, mate, gnome, …).
  • Environment configuration (shell control/completion/shortcuts, PRIMARY and CLIPBOARD selections). Bash Keyboard Shortcuts.
  • Basic shell commands (man, w, who, whoami, id, echo, apropos). Shell commands for working with file system hierarchy (ls, cd, cat, tac, mkdir, rmdir, rm, cp, mv, touch, pwd, tree, chmod, umask). Shell commands making output readable (head, tail, more, less, grep). Counting lines, words and bytes (wc).
  • Chaining commands together with pipes (|). Redirections in all variants (>, >>, <, 2>, &>, tee).
  • Useful special files (., .., .abc, /dev/null). Symbol for home directory (~).
  • Globs and wildcards (*, ?, [a-z], [^a-z]). Nice article about Globs.

Scripting Tasks

  • [02/setup.txt] [1 point] Learn more about environment configuration and configure your system for your best needs. I.e. tweak your UNIX computer the way you like it. Write down a text file (80 characters per line, couple of paragraphs) what your setup is (desktop environment, text editor, file manager, internet browser and other favourite applications, tips and tricks). Write it in nano or another text editor in terminal. Use web browser only for upload.
  • All of the following tasks have to use only commands we used on class.
  • [02/print.sh] [1 point] Write a command for listing files in the current directory sorted by date of modification.
  • [02/last-mod.sh] [1 point] Write a command for listing 22 most recently modified files.
  • [02/smallest.sh] [1 point] Write a command for listing 20 smallest files from /bin which contain letter a.
  • [02/two-chars.sh] [1 point] Write a command for listing all hidden files in your home directory that contains at least 3 letters in their name. This has to work from any location.
  • [02/last-line.sh] [1 point] Write a command that prints last 3 lines from every file ending with .h in /usr/include.
  • [02/incl.sh] [1 point] Write a command which copies all files which do not have any number in their name from /usr/include to ~/my_includes.
6.3.
8.3.

Before Class

  • Last time you did much more than expected. Awesome job guys!
  • What is POSIX? What is bash/zsh/ksh/dash?
  • Is it possible to chain commands together? Is it possible to write your own program for chaining programs together?
  • What is standard input/output? What is standard error output? Write a program in your favorite language which reads some text from standard input (stdin) prints “Hello world!” to standard output (stdout) and “Hello error!” to standard error output (stderr).
  • What does '>>' do?
  • With pipes we can redirect standard output to standard input of the subsequent program. Is it possible to redirect standard error output with a pipe (|)?
  • What is a wildcard? What is a shebang? What is the difference between compiled language and scripting language?
  • What will 'ls ?' output? What about 'ls ??'. And 'ls ???', 'ls ????' and 'ls ?????'? What about 'ls [a-z]' or 'ls [a-z][a-z][a-z][a-z]'? Why?
  • What will 'ls *' do in empty directory? And in directory only with hidden files? Why?
  • Everyone will prepare 5 or more scripting tasks using all the stuff we learned last time. E.g. print first 10 lines containing character ‘a’ from /etc/passwd and save the output to the file. Be original, tasks have to be unique and complex, i.e. no activity points for duplicate and trivial tasks. You can save it to your git repository.

On Class

  • Comment on github notifications and homework submissions on time. Use github for saving Before Class preparation and also for notes. Test your homeworks on different inputs. Try to invent corner cases.
  • Solving scripting tasks you invented.
  • Different shells and different standards (bash, zsh, ksh, dash, sh). POSIX. Comparison of shells.
  • Tools for management (ssh, scp, screen, tmux, top, htop, df, free, ps, pgrep, pkill, kill, killall).
  • Tools for working with text files (diff, vimdiff, patch, comm, cut, paste, split, tr, sort, uniq, join). Calculator (bc).

Scripting Tasks

  • [03/group-sum.sh] [1 point] /etc/group contains names of system groups. Write a command to print only column with group names. Then write a command, which prints one number - all group IDs summed together.

  • [03/unused-groups.sh] [1 point] /etc/passwd contains users present on system with their groups. Print system groups which are not present in /etc/passwd.

  • [03/patch.sh] [1 point] Take some program you wrote and create a patch which replaces every digit with character ‘X’ in its source code. (Hint: Use tr for digit replacement).

  • [03/copy.sh] [1 point] All of the files above copy to folder /tmp on different computer.

  • [03/mem.sh] [2 points] df displays free disc space. Print total number of available space in Megabytes, i.e. sum together available space from all partitions.

  • [03/cpu.sh] [4 points] /proc/cpuinfo contains information about CPU. There is a row called flags which contains capabilities of the CPU. Create a command which will output name of the CPU on the first line and then all flags of the CPU sorted alphabetically. One flag per line indented with 2 spaces. Do this for 2 different CPUs and create an output with three columns. First column for the first CPU, second for the second CPU and third containing flags common for both CPUs: Hint: New line is \n character. The output doesn’t need to be properly aligned but strive for the best result you can achieve! Example:

    Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz Common
      3dnowprefetch                           acpi                                   [many more flags]
      abm                                     aperfmperf                             xtpr
      [many more flags]                       [many more flags]
      xtpr                                    xtpr
13.3.
15.3.

Before Class

  • What does program join do? Is it useful for 03/unused-groups.sh?
  • Again: Write a program which reads from stdin and writes to stdout and stderr. Use it in shell with redirections <, >, 2>.
  • What does ** (NOT POSIX) glob mean? Try it on ls /**/ etc.
  • What is a shell variable? What $PATH variable means (Hint: When you write cat and execute what is going on?)?
  • What actually cat is? Is it command of the shell? What is the difference between cat, /bin/cat, ./cat?
  • Is writing a shell script the same as writing a program?
  • What is a pull request? What is it good for?
  • Prepare one complex scripting task named $sur.sh where $sur is 3 letter prefix of your surname without diacritics. Create a pull request to the asch/intro19. The file $sur.sh has to contain only assignment, not the solution.

On Class

  • Git is not for communication. Use e-mail.
  • Variables ($var), environment (env, $PATH) and exporting variables (export).
  • Finding what binary is executed by written command (whereis, which).
  • Control-flow in shell scripts (;, if, while, for, &&, ||, test, [], true, false).
  • Shell functions (func() { }).
  • Shell arguments ($1, $2, …).
  • ~/.bashrc, ~/.profile for setting up your shell.
  • Aliases (alias l='ls -l').
  • Writing a shell script (#!/bin/sh), checking what shell we have (ls -l $(which sh), making it safe (set -euxo pipefail) and lint it (shellcheck). Line wrap (\). Line limit 80 chars.
  • See man bash for complete reference.
  • See man dash for much more simple reference without bash specific constructs. (Recommended!)

Scripting Tasks

  • [04/mega.sh] [5 points] Implement an executable script which implements all scripting tasks from second class. The script accepts one argument which is name of the scripting task and performs the operation required by the scripting task. For example mega.sh print.sh will do the same action as scripting task [02/print.sh] and mega.sh smallest.sh will do the same action as [02/smallest.sh from the second class and so on. This has to be just one script which means all code has to be inside this script.
  • [04/$surname.sh] [1 point for every hard task] [deadline +1week] Pick 5 hardest tasks from my GitHub asch/intro19 repository and solve it. Name it equally with script on the github. I will not give points for easy tasks.
20.3.
22.3.

Before Class

  • Again: Write a program which reads from stdin and writes to stdout and stderr. Use it in shell with redirections <, >, 2>.
  • What is POSIX? What is it good for? What shell is POSIX compliant? What is a “bashism”? Give an example of “bashism”. Is it a good or bad thing?
  • Since now all pull requests with complex scripting tasks will be honored with 1-2 extra points (If I like it.). The limit is 1 pull request in a week. Name the script properly, it has to contain first three letters from your name and some fitting name, e.g. asc-newest_files.sh.
  • How a shell function “returns” a value? What if I want to save result of some function to variable.
  • What other stuff are you missing for comfortable writing of shell scripts?

On Class

  • Quoting.
    • "" prevents from glob expansion. Try echo "*" vs echo *.
    • '' prevents from variable expanion. Try echo '$var' vs echo $var.
    • Expansion rules are much more complicated. See man dash for more details.
  • Function “return” value (echo). Capturing output of function/command to a variable ($(expr)).
  • Exiting from the shell (exit).
  • Command construction from arguments (eval).
  • Run on background (&).
  • Signals for job control (SIGINT (Ctrl-C), SIGTSTP (Ctrl-Z), SIGCONT (fg/bg)).
  • fg/bg can be used with IDs from jobs output as follows: fg %2 to move process 2 to foreground.
  • List jobs in current session (jobs).
  • Reading from stdin (read).
  • Here-doc redirection (<<).
  • Special Parameters ($*, $@, $#, $?, $$, $!, $0).
  • Parameter Expansion (${parameter:-word}, …).
  • Arithmetics $((expr)).
  • Reading arguments comfortably with shift and getopts.
  • Grouping Commands Together ((list), { list; }).
  • Introducing dash: The shell you want to use for shell scripts!
  • Reading man dash as a source of all knowledge needed for scripting.
  • Tools for http interaction (curl, wget, httpie (binary is called http)).
  • Tool for checking “bashism” (checkbashisms).
  • List of bashisms.
  • Script constructed during the class can be downloaded here.

Scripting Tasks

  • [05/bc.sh] [2 points] create a calculator similar to bc (Hint: eval). Your program must not die on wrong input, e.g. on “asdf”. Some basic error checking has to be present.

  • [05/getmedia.sh] [3-5 points] [deadline +2weeks] Write a script which will extract links to specified media from given web page. The media can be audio (.wav, .mp3, .ogg, …), picture (.jpg, .png, .svg, …), document (.doc, .xls, .pdf, …) or archive (.zip, .rar, tar, tar.gz, .tgz, …). Without any options or with -h and --help program prints usage. The type of output can be specified with options -a or --audio for audio, -p of --picture for pictures etc. You can get extra 2 points (5 total) if you add option -d N, --depth=N which will traverse also all referenced web pages on the current webpage up to depth N (i.e. recursion). Example of output:

    ./getmedia.sh --picture asch.cz/unix
    https://imgs.xkcd.com/comics/command_line_fu.png

    Notes:

    1. Some webservers send compressed response. You have to use gunzip command or curl --compressed option.

    2. Parameters check of your script has to be always present. The script must exit on unknown parameters and print usage.

    3. Deadline extended. [deadline +2weeks].

27.3.
29.3.

Before Class

  • What is IFS variable? What is the difference between ls -l vs echo $(ls -l). Why?
  • What is the difference between echo ls, ls, $(echo ls), echo $(ls), $(ls), (echo ls) and (ls)? Why?
  • Start thinking about your (bigger) script needed for credit. Try something useful for you.

On Class

  • Note about nice tutorials on AbcLinuxu. See references.
  • Finding stuff on filesystem (find).
  • Running a command with arguments from standard input (xargs).
  • Project building tool (make).
  • Filesystem utilities (ln, file, atool (apack, aunpack), tar, dd, du).
  • Information about networking (ip, ifconfig).
  • Textual web browsers (w3m, links, lynx).
  • Text substitution with sed 's///'.
  • Regular Expressions. Can be used in some utilities. E.g. grep, sed, awk, find.

Scripting Tasks

  • [Feedback score:12 WED|FRI 10] Please provide a feedback via e-mail (create a new account if you want be anonymous) about my labs (style of teaching, examples, difficulty, is it good for your knowledge, …). Also tell me what you want to do next time (console oriented desktop environment, review stuff we did so far, something totally different, …).

  • [06/intro19.sh] [6 points] [deadline +3weeks] Create a tool for obtaining information about my labs.

    intro19.sh [option] command
    
    OPTIONS
    -h, --help    Prints help.
    -g, --git     Path to your git repository. Default hardcode path on your filesystem.
    -u, --url     URL to the labs. Default http://asch.cz/intro
    -v, --verbose Print debug outputs. Default off.
    -s, --short   Use short version of outputs.
    
    COMMANDS
    ls     Print all scripting tasks in the following format:
           [Date] [Due Date] [Your Points/Total Points]
           [Complete Tasks Description (Only names of scripts when used with --short)]
           [newline]
           [Repeat until not done for all tasks]
    
    status Prints your score from scripting tasks and activity points and check
           if you are passing. With --short print only summary. The format is following:
           [Task Lab] [Task Names] [Your Points/Max points]
           [Repeat until not done for all task labs]
           [newline]
           Scripting Summary: Actual 74%, Needed 75%, Diff -1% (-2 pts).
           Activity Summary:  Actual 8 pts, Needed 10pts, Diff -2 pts.
    
    next   Prints information about next class.
           [Date] [How many hours left]
           [Before Class Information (With -s prints only number of bullets)]
           [On Class Topics (With -s prints only number of bullets)]
           [Scripting Tasks (With -s prints only number of bullets)]
    
    Examples of usage:    intro19.sh --git mff/unix_introduction/intro19 --url asch.cz/intro ls
                          intro19.sh -s ls
                          intro19.sh --short status
                          intro19.sh status
                          intro19.sh next

    Note: Feel free to extend the output and/or functionality. The output format is not fixed.

3.4.
5.4.

Before Class

  • Execute vimtutor command and take a look at Pavel Satrapa — Vim. Try to learn only basic movements and understand the motivation.
  • Read about typing with all 10 fingers here.
  • Special class — Do not forget sending feedback and topics you are interested in before class.

On Class

  • Off-record class. Completely unrelated to the lecture and exam.
  • Feedback on feedback (activity points, coding style).
  • —Temporary information—
  • cd && curl -O asch.cz/intro/.zshrc
  • cd && curl -O asch.cz/intro/.zshrc.local
  • cd ~/.config && curl asch.cz/intro/herbst.tar.gz | tar xz
  • —Temporary information—
  • vi, vim, nvim
  • Tiling Window Managers (herbstluftwm, i3, sway, awesomewm, xmonad).
  • Various applications with vi-inspired keybindings (alot, mutt, newsboat, tig, ranger, vifm, qutebrowser, chromium + vimium, firefox + vimium-FF).
  • git from command line. Basic git commands: clone, add, commit, pull, push.
  • Usefull tools for time/performance measurement (parallel, time, hyperfine, perf).
  • GNU/Linux (Fedora, Ubuntu, Debian, ArchLinux, Gentoo), NetBSD, OpenBSD, FreeBSD, DragonflyBSD, MacOS, Solaris.
  • Package managers (dnf (Fedora), apt (Debian), pacman (ArchLinux), emerge (Gentoo)).
  • Is it actually useful? How would you measure {time, cache misses, branch misses, page-faults, …} of your program on Windows?
  • Programmer’s environment and work style.

Scripting Tasks

  • Spend 10 minutes thinking about the freedom UNIX-like systems give you VS the others (Windows, MacOS).
  • Answer those 2 questions for yourself: “Do you like freedom or not? What operating system do you use?”
  • If you want to, install Linux (Ubuntu or Fedora are user friendly) on some old computer you don’t use, to Virtual Box, or even (the best option) on your primary laptop/PC.
10.4.
12.4.

Before Class

  • Come prepared to class, i.e. repeat everything we did so far, especially all the basic utilities. You will be coding all 90 minutes.
  • Be familiar with Bootpd exam assignment.

On Class

  • More very useful nvim keybindings:
    • D delete from current position to end of line
    • C delete from current position to end of line and switch to insert mode
    • cw change word, i.e. delete current word and switch to insert mode
    • r replace character under cursor
    • :10<ENTER> jump to line number 10
  • Solving Bootpd exam script.

Scripting Tasks

  • [08/bootpd.sh] [2 points] Submit working and completed Bootpd script. This scripting task will enrich you with both homework and activity points.
17.4.
26.4.

Before Class

  • Read and try examples from Sed tutorial.
  • Come prepared to class, i.e. repeat everything we did so far, especially all the basic utilities. You will be coding all 90 minutes.
  • Be familiar with ANSI to K&R C Converter exam assignment.

On Class

  • Sed and regular expressions revision.
  • Solving ANSI to K&R C Converter exam script.

Scripting Tasks

  • [09/ansi-to-kr.sh] [2 points] Submit working and completed ANSI to K&R C Converter script. This scripting task will enrich you with both homework and activity points.
24.4.
3.5.

Before Class

  • Read and try examples from Awk tutorial (Until chapter about Associative Arrays (included)).
  • Come prepared to class, i.e. repeat everything we did so far, especially all the basic utilities. You will be coding all 90 minutes.
  • Be familiar with Fakulty exam assignment.

On Class

  • Awk!
  • Solving Fakulty exam script.

Scripting Tasks

  • [10/fakulty.sh] [2 points] Submit working and completed Fakulty script. This scripting task will enrich you with both homework and activity points.
8.5.
10.5.
  • Class cancelled for synchronization reason.
15.5.
17.5.

Before Class

  • Find some nice place, where you can solve Sort exam script on your own.

On Class

  • Solve Sort exam script on your own.

Scripting Tasks

  • [11/sort.sh] [20 points] Submit working and complete Sort script. Feel free to submit the script earlier regarding to obtaining the credit sooner.
22.5.
24.5.

Before Class

  • Find some nice place, where you can solve rsync/baf exam script on your own.

On Class

  • Solve rsync/baf exam script on your own.

Scripting Tasks

  • [12/rsync.sh] [20 points] Submit working and complete rsync/baf script. Feel free to submit the script earlier regarding to obtaining the credit sooner.

Git Repositories with Semestral Projects

Resources