I recently completed the Linux Upskill Challenge, a month-long, self-paced, self-led course curated on Reddit. It’s hands-on and is best run with a publically accessible remote server, so that you’re mimicing a production server that your organization might host online. (There’s less pressure in administering a server that’s only locally available!)

These are my thoughts and notes about each day’s content - not formal study notes, but informal jot notes on the things I found interesting and important.

/r/linuxupskillschallenge

Day 0 - Getting Started
Created new Azure Ubuntu VM. Used SSH key for authentication.
Azure returned a .PEM file. Kitty won’t take this - I had to use puttygen.exe to load the PEM and output a PPK. I then created a Kitty profile using this PPK (Connection -> SSH -> Auth) and username azureuser (Connection -> Data -> Autologin username)
https://docs.bitnami.com/azure/faq/get-started/connect-ssh/
linux is case-sensitive

Day 1 - Getting Connected
Log in with Windows Terminal (Powershell): ssh -i .\Downloads\Ubuntu_SSH.pem azureuser@23.96.4.131
uname -a shows some server/OS information
uptime shows the uptime
free shows memory information
df -h shows disk space information

Day 2 - Navigation
cd and cd ~ do the same thing - return you to your home directory
man brings up the manual/help for commands
pushd is a way to create a trail of where you’ve been. dirs -v will list the trail easily. popd will move you around the stack of directories

Day 3 - Working as Administrator
/etc/shadow is where the hashed passwords are kept
sudo will let you run a command as root
sudo -i will pop an interactive sudo session if you’ve a number of commands to run. logoff or exit will bring you back. The command prompt will change from $ to # to show you’re running as sudo.
All sudo uses are logged in /var/log/auth.log
sudo hostnamectl set-hostname XXXX will change the hostname of the server
timedatectl will adjust the time/date settings. TAB will autocomplete and/or show other subsequent commands

Day 4 - Package Management & File Structure
apt is the Ubuntu package manager for working with applications
apt search "search term" will find matching packages
apt list --installed will list all installed packages
apt install "packagename" will install the package
Linux uses a different tree structure than Windows.
/ is the mountpoint - this is eqivalent to C:\
/root is the home directory for the root user
/usr/ is where user directories are kept
/var is where variable files are stored. /var/log is primarily where logs are stored.
/bin is where binary files (non-system level) are stored
/sbin is where system-level binaries are stored
/etc is where most configuration files are stored
/etc/passwd is the file containing userids, SSIDs and passwords

Day 5 - Less
less is similar to more, except it doesn’t read the whole file at load. In less:

G skips to the end
g skipts to the beginning
q quits

The bash shell stores your command history, writing it all to /usr/azureuser/.bash_history (files beginning with a . are hidden). history will list the contents of the file and allow you to manipulate it. Pressing up or down will navigate the history from the prompt.

Day 6 - vi and vim
vi is the classic unix editor. On most modern *nix systems, vi is an alias for vim - vi improved.
h, j, k, l act as the arrow keys within vim, although modern vim instances also have support for the arrow keys
vi has two modes: normal mode and insert mode. insert mode actually allows editing of what’s on the screen, while normal mode is used for inputing commands. esc esc will always return to normal mode.
Many vi commands are made from an operator and a motion. For example, dw will delete (d) a word (w). Quick list of motions:

w will jump to the next word
e will jump to the end of the current word
$ will jump to the end of the line

Typing a number before a motion will repeat it that many times
u will undo actions
x will delete the character your cursor is on
i (insert), a (append), c (change), r (replace)
ce will delete the remaining letters in the current word, after the cursor
c$ will delete the remaining text in the line
33dd will cut the line 33 times (equivalent to deleting 33 rows). They can be pasted wherever your cursor is with p
gg will jump to the start of the file, while G will jump to the end.

:s/old/new will substitute ’new’ for the first instance of ‘old’ in the current line
:s/old/new/g will substitute ’new’ for all instances of ‘old’ in the current line
#,#s/old/new/g will substitute ’new’ for all instances of ‘old’ between the two line numbers provides
%s/old/new/g will substitute ’new’ for all instances of ‘old’ in the file

/SEARCHTERM will search for the first instance of the searchterm in the file. n will jump to the next instance
% while on any bracket/parenthesis will jump to its match
v starts a Visual selection. Then use an operator (ie. :w to write the selected text to a file)
:r FILENAME will merge the contents of FILENAME into the open file.
:r !ls will merge the output of ls into the current file.
:!COMMAND will execute an external program in vi. For example :!ls will print the directory listing.
:q! will quit without saving
:w will write/save the file with the current file name. :w FILENAME will save it with the specified filename.
:wq will save and quit

Day 7 - Server & Services
install apache2 (httpd)
stop the apache service: sudo systemctl stop apache2
start the apache service: sudo systemctl start apache2
check the status: sudo systemctl status apache2. This pulls information from the logs.
Application configuration is controlled by files under the /etc directory for most Linux distros. The Apache config is in /etc/apache2/apache2.conf
In /etc/apache2/apache2.conf there’s the line with the text: IncludeOptional conf-enabled/*.conf This tells Apache that the *.conf files in the subdirectory conf-enabled should be merged in with those from /etc/apache2/apache2.conf at load.
The location of the default webpage is defined by the DocumentRoot parameter in the file /etc/apache2/sites-enabled/000-default.conf … in this case, it’s /var/www/html, so /var/www/html/index.html
sudo apt update, then sudo apt upgrade will install all upgraded packages for any software that’s been installed with the apt package manager.

Day 8 - grep and others

Day 9 - networking & ports
ss is the replacement for the netstat comment. ss -ltpn will print the open ports. (use sudo for it to list the Processes that control these listeners)
iptables, nftables and ufw are all firewall utilities for Linux.
sudo iptables -L will list firewall rules in place.
Set UFW rules to allow SSH but deny HTTP: sudo ufw allow ssh; sudo ufw deny http
Enable UFW: sudo ufw enable

Day 10 - scheduled tasks
see your scheduled tasks with crontab -l, or root with sudo crontab -l
/etc/cron lists the timing of daily, weekly & monthly jobs that run from /etc/cron.daily, etc/cron.weekly, etc.
systemd starts and stops services, but can also be used to run tasks via. timers: systemctl list-timers

Day 11 - finding things
locate will search an index for "*something*" by default. The index is built by a nightly cron job but can be updated with sudo updatedb
find will try to match a file(s) based on criteria: find /var -name access.log or find /home -mtime -3 (find any file under /home modified in the last 3 days)
grep will search within plaintext files for specific text. grep -R -i "PermitRootLogin" /etc/* will search recursively through /etc/ for files containing the case-insensitive string ‘permitrootlogin’
which will list the patch from the PATH statement that a utility runs from: which nano

Day 12 - transferring things
winscp> open sftp://azureuser@23.96.4.131 -privatekey="c:\users\gbeifuss\downloads\ubuntu_ssh.ppk"

Day 13 - permissions
ls -l will list permissions: owner(user), group, others
chmod u-w filename.txt will remove the W permission for the user
chmod g-w filename.txt will remove the W permission for the group
chmod o-w filename.txt will remove the W permission for others
chmod u+w filename.txt will add the W permission for the user
groups will list the groups a user belongs to
groupadd will add a new group (sudo groupadd newgroup)
usermod -a -G group username will add username to group

Day 14 - users & groups
Add a new user with sudo adduser helen. (If not prompted by a password, set one for helen: sudo passwd helen)
Login as helen: sudo su helen
Permissions for sudo are controlled by the visudo utility

Day 15 - repositories…
Linux versions control the versions of applications that can be installed. Ubuntu 18.04 ships with Apache 2.4.29 - even if you install apache with apt 5 years later, you’ll get 2.4.29 installed. (Security patches are made to repositories, but by backporting fixes (from fixed versions) into older versions).
/etc/apt/sources.list is the default apt repository source. This can be edited to enable multiverse/universe repositories, which “*may contain software which has been classified as non-free … and may not include security updates”.
Ubuntu 20.04 ships with 99589 packages (apt-cache dump | grep "Package:" | wc -l)

Day 16 - archiving & compressing
Unlike Windows, Linux gathers content in one step, and compresses in another.
tar stands for ‘Tape ARchive’
tar -cvf myinits.tar /etc/init.d takes a snapshof of the files currently in /etc/init.d
This file can then be compressed: gzip myinits.tar, which will create myinits.tar.gz, a tarball. Tarballs often use the extension .tgz
.bz2 files use a different compression than gzip - it uses higher compression, but takes longer. A .bz2 file can only contain a single file, so it’s usually used to futher compress a tarball or other archive. In practice, these two steps can be condensed into on: tar -cvzf myinits.tgz /etc/init.d

c = create
v = verbose
z = zip/compress
f = specify the output file
x = extract (expand)
j = uncompress a .bz2 file first

Day 17 - from the source
build-essential is a standard bundle of complilers and similar tools: sudo apt install build-essential
Download the latest nmap: wget -v https:/nmap.org/dist/nmap-7.92.tar.bz2
Extract the files: tar -jxvf nmap-7.92.tarb.bz2
./configure is a script which checks the server
make complies the software, typically via the GNU complier gcc
sudo make install will take the compiled files, install them plus documentation, and in some cases setup services and scheduled tasks.
Any software installed outside of apt won’t be updated by apt update, so new releases and security fixes need to be manually tracked.

Day 18 - log rotation
logrotate is used by cron to rotate logs (/etc/cron.daily/logrotate), (/etc/logrotate.conf + /etc/logrotate.d)
For example, edit /etc/logrotate.d/apache2 to adjust the configuration for all apache logs.

Day 19 - inodes, symlinks and other shortcuts
Linux disks use ext3, ext4, zfs, or perhapsntrfs. Above that sits the VFS - Linux Virtual FileSystem
Each filename points to an inode (a numerical value), which is seen most easily in two places: ls -i and stat:

ls -li /etc/hosts
 35356766 -rw------- 1 root root 260 Nov 25 04:59 /etc/hosts
 
stat /etc/hosts
 File: `/etc/hosts'
 Size: 260           Blocks: 8           IO Block: 4096   regular file
 Device: 2ch/44d     Inode: 35356766     Links: 1
 Access: (0600/-rw-------)  Uid: (  0/   root)   Gid: (	0/	root)
 Access: 2012-11-28 13:09:10.000000000 +0400
 Modify: 2012-11-25 04:59:55.000000000 +0400
 Change: 2012-11-25 04:59:55.000000000 +0400

Several filenames could point to the same inode, and have the same contents/permissions/ownerships.dates. These attributes are stored at the inode level, hard links. ln is used to create hard and symbolic links (symlinks):

ln /etc/passwd link1
ln -s /etc/passwd link2

Hard links:

point to files, not directories
can’t reference a file on another disk/volume
link references work even if it is moved
refer to physical locations/inodes on disk

Symbolic links (symlinks):

can link to directories
can reference files/directories on another disk/volume
remain if the original file is deleted
will NOT reference the file anymore if it’s moved
reference abstract names and not physical locations
have their own inode

Day 20 - scripts
Scripts are regular files with the X permission chmod +x filename
Scripts typically start with #!/bin/bash