Linux Essentials FG Table of Contents Terminal Learning Objectives 1. Explain the Purpose of Understanding the Linux Environment 2. Identify Commands to Enumerate Processes 3. Identify Methods of Automation and Logic 4. Identify Critical Locations in the Linux File System 5. Discuss String Manipulation Techniques to Identify Key Information 6. Complete Linux Essentials CTFd Challenges PE 1. Commands, Arguments, and Help 1.1 Commands 1.1.1 Situational Awareness 1.2 Arguments 1.3 Help 1.4 Variables and Command substitution 1.5 Redirection 1.6 Piping 2. Automation and Logic 2.1 For Loops 2.2 If Statements 2.3 While Loops 3. Linux Filesystems 3.1 Linux Filesystem Hierarchy 3.2 Files and Folders 3.3 Linux Users 3.4 Linux Groups 3.5 Permissions 3.5.1 Special Permissions : Sticky Bit 3.5.2 Special Permissions : SUID and SGID 4. String Manipulation 4.1 Grep 4.2 Awk 4.3 Sed 4.4 Regular Expressions 5. Resources 5.1 Linux Resources 5.2 Regex Resources Terminal Learning Objectives 1. Explain the Purpose of Understanding the Linux Environment 2. Identify Commands to Enumerate Processes 3. Identify Methods of Automation and Logic 4. Identify Critical Locations in the Linux File System 5. Discuss String Manipulation Techniques to Identify Key Information 6. Complete Linux Essentials CTFd Challenges PE BASH (Bourne Again Shell) is a CLI (Command Line Interface) that processes commands from a user using a specific syntax. Prior to GUI (Graphical User Interface) programs, the command line was the only option to interact with a computer and manage all the running programs. Today, users typically utilize a GUI desktop environment such as GNOME. The ease of use led to a wider adoption of computing in the general public, which coincided with the decline of CLI usage. However, CLI programs are still used extensively in many different technical occupations. They are less user friendly, require knowledge of CLI specific syntax, and can’t do certain typical user functions like browsing the web. The CLI provides three significant advantages over GUI programs - speed, scaling, and automation. CLI programs consume fewer resources then their graphical counterparts, they accept scripts to automate mundane or repetitive task, and given enough expertise multiple commands can be executed at the same time. It’s imperative for anyone operating in today’s cyberspace to be fluent in the functionality and power of the modern command line. 1. Commands, Arguments, and Help BASH, much like PowerShell, does 4 things: Execute Commands with or without arguments Redirect output from commands to a variety of locations (covered later) Parse text(string) input and output into lines formatted as the user finds useful Accept scripts to automate execution of all the above tasks CLI will be synonymous with the terminal or command line from here on out. 1.1 Commands As previously stated, terminals are interfaces used to run commands on the system. It’s common to find not only all the features of a GUI program in it’s CLI counterpart, but most of the time you actually gain additional functionality as well. Running an executable in a bash terminal student:~$ pwd (1) /home/student (2) 1 pwd (print working directory) command. 2 the output of the command, returned on a new line. "Newline" is a character that is commonly used to represent the beginning of a new line, or end of an old one. While common, not all commands return their results with a newline character added to the end (\n). When using methodologies such as hashing, these invisible blank newline characters should be taken into account. 1.1.1 Situational Awareness After first obtaining access to a system an operator must gather as much information about their environment as possible, this is referred to as situational awareness. pwd is just one command of many on Linux which can provide us some insight. Other commands to help gain situational awareness: hostname or uname -a displays the name of the host you are currently on. whoami shows the user you are currently logged in as (useful after gaining access through service exploitation). w or who shows who else is logged in. ip addr or ifconfig displays network interfaces and configured IP addresses. ip neigh or arp displays MAC addresses of devices observed on the network. ip route or route shows where packets will be routed for a particular destination address. ss or netstat will show network connections, with the appropriate flags will show listening ports nft list tables or iptables -L to view firewall rules. sudo -l displays commands the user may run with elevated permissions. Depending on the age and distribution of Linux some commands may not be installed or require privilege escalation through sudo to run. 1.2 Arguments Almost all commands you ever come across will take additional parameters or arguments that modify the functionality of the base command. In some cases, arguments are even required and the command will not function without it. Running a Command with and without an Argument student:~$ cd / (1) student:~$ ls (2) bin dev home initrd.img.old lib64 media opt root sbin srv tmp var vmlinuz.old boot etc initrd.img lib lost+found mnt proc run snap sys usr vmlinuz student:~$ ls -l (3) drwxr-xr-x 2 root root 4096 Feb 4 2020 bin drwxr-xr-x 3 root root 4096 Feb 4 2020 boot drwxr-xr-x 19 root root 3840 Jan 23 12:29 dev drwxr-xr-x 117 root root 4096 Feb 12 16:49 etc drwxr-xr-x 4 root root 4096 Jan 23 12:25 home _truncated_ 1 cd / required for demo. 2 ls with no arguments. 3 ls -l changes the output to long list format. 1.3 Help Terminals were created in the age before the internet, so when they are installed, the documentation is included. Running a command with an argument to get help student:~$ ls --help (1) Usage: ls [OPTION]... [FILE]... List information about the FILEs (the current directory by default). Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. Mandatory arguments to long options are mandatory for short options too. -a, --all do not ignore entries starting with . -A, --almost-all do not list implied . and .. --author with -l, print the author of each file -b, --escape print C-style escapes for nongraphic characters --block-size=SIZE with -l, scale sizes by SIZE when printing them; e.g., '--block-size=M'; see SIZE format below -B, --ignore-backups do not list implied entries ending with ~ -c with -lt: sort by, and show, ctime (time of last modification of file status information); with -l: show ctime and sort by name; otherwise: sort by ctime, newest first -C list entries by columns _truncated_ 1 --help is an generic argument that should give a simple help document for a command. Linux also provides access to man (manual) pages for the majority of its commands. Man pages at a minimum do the following: Describe the command List all of its arguments The man pages for most commands are available online too. Man page for the LS Command Running the man command to view the man page for ls student:~$ man ls (1) LS(1) User Commands LS(1) NAME ls - list directory contents SYNOPSIS ls [OPTION]... [FILE]... DESCRIPTION List information about the FILEs (the current directory by default). Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. Mandatory arguments to long options are mandatory for short options too. -a, --all do not ignore entries starting with . _truncated_ 1 The man command with ls as an argument. 1.4 Variables and Command substitution Variables are a string of characters with an assigned value. They are used when automating tasks to reduce the amount of time needed to do something. Variables as a concept are easy to explain, but their application is dependent on that task that needs to be done. BASH Variables are always accessed with $ then the variable name, but they are created without a $ character. Assigning a Single Value to a Variable student:~$ echo $a (1) (2) student:~$ a="100" (3) student:~$ echo $a (4) 100 (5) 1 echo the value of the variable a. Notice that it has no value. 2 Nothing is returned. 3 Setting the value of a equals to 100. 4 echo the value of a. 5 Notice that the new value is 100. Variables can also be assigned the output of a command using a technique called Command substitution. Command substitution is done with $(command) instead of the traditional $. Command Substitution in Bash student:~$ directories=$(ls /) (1) student:~$ echo $directories (2) bin dev home initrd.img.old lib64 media opt root sbin srv tmp var vmlinuz.old boot etc initrd.img lib lost+found mnt proc run snap sys usr vmlinuz 1 Assign the variable directories to the output of the ls / command. Note the $(). 2 Execute the echo command the contents of the variable directories. 1.5 Redirection Success and Failure Output from commands by default is sent to the terminal, but it can be redirected to other locations as well. Output from commands are assigned to a stream. There are three types of streams by default: standard input 0 ←--- the default for a command arguments standard output 1 ←--- the default for successful command output standard error 2 ←--- the default for failed commands or errors Redirection is done with the > character and a file name or a numbered stream. Using a 1> redirect output to a file in bash student:~$ directories=$(ls /) (1) workstation21:$ echo $directories 1> thisisanewfile (2) workstation21:$ cat thisisanewfile (3) 1 Creates the directories variable if you haven’t done so already. 2 Send the standard output from the $directories variable into a file named thisisanewfile. 3 Read the contents of thisisanewfile. Using a 2> redirect errors to a location in bash student:~$ ls bacon (1) ls: cannot access 'bacon': No such file or directory workstation21:$ ls bacon 2> errorfile (2) workstation21:$ cat errorfile (3) ls: cannot access 'bacon': No such file or directory 1 Try to list the contents of a directory that does not exist. 2 Send the standard error from the command to errorfile. 3 cat errorfile. Append to a File using the Redirection Operator ( >> ) Redirection allows you to capture the output from a command and send it as input to another command or file. The >> redirection operator appends the output to a given file. 1.6 Piping Piping redirects standardoutput or 1 to standardinput or 0 to be processed as an argument to another command using the | character. Using piping filter command output using grep student:~$ ls -Rlisa /etc | grep syslog (1) 1378 4 -rw-r--r-- 1 root root 1358 Jan 30 2018 rsyslog.conf 1379 4 drwxr-xr-x 2 root root 4096 Feb 2 14:28 rsyslog.d 566 4 -rw-r--r-- 1 root root 1550 Apr 24 2018 usr.sbin.rsyslogd 527 0 lrwxrwxrwx 1 root root 33 Jan 31 2020 usr.sbin.rsyslogd -> /etc/apparmor.d/usr.sbin.rsyslogd 535 0 -rw-r--r-- 1 root root 0 Jan 31 2020 usr.sbin.rsyslogd 768 4 -rw-r--r-- 1 root root 124 Jan 14 2018 rsyslog 958 4 -rwxr-xr-x 1 root root 2864 Jan 14 2018 rsyslog 1043 4 -rw-r--r-- 1 root root 804 Jan 14 2018 rsyslog 1054 4 -rw-r--r-- 1 root root 501 Jan 14 2018 rsyslog ls: cannot open directory '/etc/polkit-1/localauthority': Permission denied (2) 1 Execute ls with arguments, then send the output to grep command using a pipe | to filter for the string syslog. 2 Standard error saying Permissions denied. To remove all errors from your output, consider utilizing standard error redirection. ls -Rlisa /etc 2> /dev/null | grep syslog 2. Automation and Logic The primary benefit of terminals is the automation of repetitive tasks and processing logical statements. These statements and loops execute automatically using pre-programmed conditions that control how and even if they execute. They are invaluable to understand and their usefulness is limited to ones imagination and patience. If Statements, For Loops, and While Loops don’t have good man page entries because they are built into the bash man page. Highly recommend using online resources. 2.1 For Loops For Loops go by many names such as Counting Loops and Interactive Loops, but they all do the same thing - execute a command or commands multiple times with a changing variable as an argument. A complete for loop will have the following: a collection of objects assigned to a variable a variable that represents the value in the collection correctly being worked on a command or commands that will execute with each value in the collection of variables Making a Collection of objects student:~$ objects=$(ls -d /etc/*) (1) student:~$ echo $objects (2) /etc/NetworkManager /etc/PackageKit /etc/UPower /etc/X11 /etc/acpi /etc/adduser.conf /etc/alternatives /etc/anacrontab /etc/apg.conf /etc/apm /etc/apparmor /etc/apparmor.d /etc/apport /etc/apt /etc/at.deny /etc/bash.bashrc /etc/bash_completion /etc/bash_completion.d /etc/bindresvport.blacklist /etc/binfmt.d /etc/byobu /etc/ca-certificates /etc/ _truncated_ 1 Using command substitution to make a variable containing all the files and folders in /etc. 2 Reading the content of the variable $objects. Warning it is ugly. Making a For Loop to iterate on Objects in the collection of Objects student:~$ for item in $objects; do echo $item; done (1) /etc/NetworkManager /etc/PackageKit /etc/UPower /etc/X11 /etc/acpi /etc/adduser.conf /etc/alternatives /etc/anacrontab /etc/apg.conf /etc/apm _truncated_ 1 For each item in located in the objects variable, echo the value of item as the loop executes. 2 The $item variable is will contain each entry in $objects delimited by a space as the loop executes. For Loops TLDP 2.2 If Statements If statements are logical expressions that compare objects against various tests to see if they evaluate as true or false. They are understood in a sentence form like this: If this comparison is true, then do this or Else If this comparison is true, then do this or If nothing is true, do this Making an If Statement to evaluate a series of objects Copy each line one at a time student:~$ for object in $objects; \ (1) do if [ -d $object ]; then echo "$object is a directory"; \ (2) else echo "$object is file" ; \ (3) fi ; \ (4) done (5) /etc/X11 is a directory /etc/acpi is a directory /etc/adduser.conf is a file /etc/alternatives is a directory /etc/anacrontab is a file /etc/apg.conf is a file /etc/apm is a directory /etc/apparmor is a directory student:~$ for object in $objects; do if [ -d $object ]; then echo "$object is a directory"; else echo "$object is a file" ; fi ; done (6) 1 The beginning of the for loop like in section 2.1. 2 if $object is a directory AND it exists, then run echo "$object is a directory". 3 else echo "$object is a file". 4 ends the if statements. 5 ends the for loop started in 1. 6 One liner version of the if statement. One Line For Loop and If Statement for the student’s notes for object in $objects; do if [ -d $object ]; then echo "$object is a directory"; else echo "$object is a file" ; fi ; done If Statements TLDP 2.3 While Loops While statements execute a command or series of commands while a condition is true. Unlike for loops which will eventually run out of objects, While Loops will run forever if their condition never evaluates as false. While loops are great for making things run for a specific amount of time instead of a exact amount of iterations. They are understood in sentence form as follows: While this condition is true, do this thing or series of things, then re-evaluate the condition to see if it is false. Repeat until condition is false. Making an Basic While Loop while [ 1 -eq 1 ]; do echo "To Infinity and Beyond!"; done (1) 1 While 1 equals 1, run the command echo ""To Infinity and Beyond!". 2 Yes, this script will run forever. This While loop will run forever, unless the condition is re-evaluated as false. Practical While Loop Example curtime=$(date +"%s") (1) echo $curtime exittime=$(expr $curtime + 3) (2) echo $exittime while [ $exittime -ge $curtime ]; do echo "To Infinity and Beyond?" ; curtime=$(date +"%s") ; done (3) To Infinity and Beyond? To Infinity and Beyond? To Infinity and Beyond? To Infinity and Beyond? _Truncated_ #It goes for three seconds 1 Use command substitution to set the value of curtime equal to the current time in Epoch time. "%s" = in seconds 2 Use command substitution to set the value of exittime equal to 3 seconds in the future Epoch Time. 3 While exittime is greater than curtime, do echo "To Infinity and Beyond?", then update the curtime variable and check if exittime is still greater or equal to curtime. While Loops TLDP 3. Linux Filesystems A file system is how a computer stores, categorizes, and retrieves data from physical media devices for use in various applications on the system. There are multiple types of file systems, but they all follow a common layout described below: Physical Media contains A Partition that is a formatted section of memory, which contains A File System mounted on a drive, which contains A Hierarchical Format of Objects and their supporting Data and Metadata File systems are a broad topic that range from understand hexadecimal layouts of hard drives to Forensics Techniques to reassemble deleted files. However, the important take away for this course is the understanding of what is located where and what a user can do with it on a Linux Filesystem 3.1 Linux Filesystem Hierarchy Every *Nix system from Ubuntu to Debian has a defined file system layout which is known as the Linux FSH (File System Hierarchy). It is a standard which defines the directory structure on all Linux distributions. What does that mean? Well, by default it defines: The root directory of the file system / Everything starts from this directory. Think of it as the doorway to the Linux Filesystem Essential user commands in /bin Contains commands like ls and echo which every user can use. User Directories in /home Contains directories for every non-root user on the system (with a home directory and login shell) Host specific system configurations in /etc Stands for everything configurable Contains network configurations, system services(daemons), firewall configurations, etc. Variable data files in /var Contains all of the system logs by default Linux Filesystem Hierarchy Standard Wiki Linux Filesystem Hierarchy Standard Detailed 3.2 Files and Folders In a file system, there are two types of objects - files and folders. Folders are a container for files, whilst files are containers for data. Everything without exception falls into one of those two categories. Showing / directories student@linux-opstation-kspt:~$ cd / (1) student@linux-opstation-kspt:/$ (2) student@linux-opstation-kspt:/$ls -l $PWD/* (3) drwxr-xr-x 2 root root 4096 Feb 4 2020 /bin drwxr-xr-x 3 root root 4096 Feb 4 2020 /boot drwxr-xr-x 19 root root 3840 Jan 23 12:29 /dev drwxr-xr-x 117 root root 4096 Feb 12 16:49 /etc drwxr-xr-x 4 root root 4096 Jan 23 12:25 /home 1 execute the command cd into the root directory of /. 2 The directory changed to /. 3 Execute ls in long list format with absolute path. Showing files in /bin student@linux-opstation-kspt:/$ cd /bin (1) student@linux-opstation-kspt:/bin$ ls -ld $PWD/* (2) -rwxr-xr-x 1 root root 1113504 Jun 6 2019 /bin/bash -rwxr-xr-x 1 root root 716464 Mar 12 2018 /bin/btrfs student@linux-opstation-kspt:/bin$ ls -l (3) -rwxr-xr-x 1 root root 1113504 Jun 6 2019 bash -rwxr-xr-x 1 root root 716464 Mar 12 2018 btrfs -rwxr-xr-x 1 root root 375952 Mar 12 2018 btrfs-debug-tree 1 Change directory to /bin. 2 Execute ls in long list format with absolute paths. 3 Execute ls in long list format with relative paths. Remember that there are only two types of objects in Linux - files and folders. Folders can’t be read, but files can. Granted, not every file is human readable. In most computers today, files have subtypes defined by their file signature typically located in the first few bytes of a file. The file signature defines how the operating system will attempt to use the file. Reading a "file" in /bin student@linux-opstation-kspt:/bin$ cat ls | head -n 1 (1) ELF>PX@▒@8 @@@▒888▒ _truncated_ (2) student@linux-opstation-kspt:/bin$ xxd ls | head -n 2 00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000 .ELF............ (3) 00000010: 0300 3e00 0100 0000 5058 0000 0000 0000 ..>.....PX...... 1 Execute cat on ls, then send its standard output to the head command and trim output to display a single row. 2 Aside from the first few characters, that isn’t readable. 3 Execute xxd on ls, then send its standard output to the head command and trim output to display to the first two rows. 4 Look at that file signature! This file signature of 7f45 4c46 stands for Linux Executable Linked file format unique to Linux. File Signature Wikipedia 3.3 Linux Users Users in Linux systems are defined by an ascending numerical value called a UID. The uid value uniquely identifies a user and is contained in the /etc/passwd file along with an associated username. Every user on a Linux system and has an associated value in /etc/passwd. Identify what Username you are whoami student@linux-opstation-kspt:/bin$ whoami (1) student (2) 1 Execute the whoami command. 2 The answer to who I am. Identify what your uid value is with id student@linux-opstation-kspt:/bin$ id (1) uid=1001(student) gid=1001(student) groups=1001(student),27(sudo) (2) 1 Execute the id command. 2 All of my associated ids? 1001 is the UID of student, but its also the GID because it is assigned to the student group. Looking at who I am in the /etc/passwd file with cat student@linux-opstation-kspt:/bin$ cat /etc/passwd | grep student (1) student:x:1001:1001::/home/student:/bin/bash (2) (1) (2) (3) (4) (5) (6) (7) 1 cmd line: Execute cat /etc/passwd and pipe it to grep to filter on student. 2 cmd output: Student entry in the /etc/passwd file. Sections of output lines 1 Username 2 Password. An x character indicates that an encrypted password is stored in /etc/shadow file. 3 UID Value 4 GUID Value 5 User ID Info (GECOS). The comment field 6 Homeome Directory. 7 Command/Shell /bin/bash 3.4 Linux Groups Groups in Linux define a collection of Users and are defined by an ascending GID value. The gid value uniquely identifies a group and is contained in the /etc/group and its associated group name. Looking at who I am in the /etc/group file with cat student@linux-opstation-kspt:/bin$ cat /etc/group | grep student (1) sudo:x:27:ubuntu,student (2) student:x:1001: (3) 1 Execute cat /etc/group and pipe it to grep to filter on student. 2 Shows the sudo group, its gid value, its two members student and ubuntu. 3 Shows the student group, its gid value, and no additional members. By default every user is a member of their own group; therefore, it will not show in /etc/groups. /etc/groups/ shows what is called supplementary group membership 3.5 Permissions Access to objects in Linux by is controlled via strict file permissions metadata. It is formatted as: U The object’s User/Owner G The object’s owning Group O Any subject that is not the owning user or group, AKA: "Others" Each U.G.O permission group has three corresponding permissions of Read, Write, and Execute. Any combination of permissions can be applied to any permissions group. These permissions also have numeric representations of 4, 2, and 1. Permissions when represented by letters, as in rwx, are referred to as Relative, and permissions when represented by numbers, as in 421 are referred to as Octal. It’s also important to understand that file permissions do not overrule directory permissions. If a user does not have read rights to a directory, it also cannot read any of its files even if the file’s permissions allow it Table 1. Linux Permissions broken out Perm Relative Octal On a File On a Directory read r 4 Read the contents of the file List the contents of the directory write w 2 Write content into a file Create/delete in the directory exe x 1 Run the file as an executable Move into the directory Showing Linux Permissions with ls -lisa student@linux-opstation-kspt:/bin$ ls -lisa /bin/dd (1) student@linux-opstation-kspt:/bin$ 130341 76 -rwx r-x r-x 1 root root 76000 Jan 18 2018 /bin/dd (2) (3) (4) (5) (6) 1 Showing permissions. 2 The Owner has Read, Write, and Execute permissions. 3 The Group has Read and Execute permissions. 4 Anyone who is not the User/Owner or belonging to the Group has Read and Execute permissions. 5 The file’s Owner. 6 The files' Group. File and Folder Permissions Demo student@linux-opstation-kspt:/home/student$ sudo su root@linux-opstation-kspt:/home/student$ mkdir testdir root@linux-opstation-kspt:/home/student$ chmod 750 testdir root@linux-opstation-kspt:/home/student$ echo "Can you read me?" ? testdir/file root@linux-opstation-kspt:/home/student$ ls -lisa testdir/ 1020551 4 drwxr-x--- 2 root root 4096 Feb 17 19:00 . 1016881 4 drwxr-xr-x 24 student student 4096 Feb 17 18:59 .. 1022450 4 -rw-r--r-- 1 root root 16 Feb 17 19:00 canttouchthis root@linux-opstation-kspt:/home/student$ exit student@linux-opstation-kspt:/home/student$ cat testdir/canttouchthis cat: testtest/canyouread: Permission denied 1 Change to the root user. 2 make a directory named testdir. 3 change the permissions on the directory to 750 or RWX,R-X,---. 4 Echo some text into a file in the created directory. 5 Show the permissions of the Directory and the file. . represents the directories permissions. 6 Exit root. 7 Try to cat the file as student and get Permission denied. 3.5.1 Special Permissions : Sticky Bit If a user has write access to a directory, they can delete any file from it. That may cause problems though in some directories like /var/tmp. To address this Linux has what is known as the sticky bit. The sticky bit removes the ability to delete files unless the user attempting is the owner of the file. The root account can still delete the file if it wants to. 3.5.2 Special Permissions : SUID and SGID When an executable is ran in Linux, it runs with the permissions of the user who started it. However, SUID and SGID change that to force the executable to run as the owning user or group. These permissions are represented as s in the User or Group field of ls- l. SUID and SGID Demo student@linux-opstation-kspt:~$ ls -l /bin/ping (1) -rwsr-xr-x 1 root root 64424 Jun 28 2019 /bin/ping (2) 1 Execute ls -l on /bin/ping. 2 Notice the s in the users field? What permissions does this executable effectively have? TLDP File Security Learn Linux, 101: Manage file permissions and ownership 4. String Manipulation In Linux all output is some kind of text, regardless of whether it was meant to be read by humans or not. You could scroll and use your finger to find data, or you can use one of pattern matching or text manipulation tools to make life easier. 4.1 Grep Grep is a program that searches data given to it from standard input for patterns of strings specified with regular expressions. Grep is invaluable in Linux because most of the time it is to only way to quickly filter output to find exactly what is needed. grep uses a generic wildcard regular expression match by default. It accepts custom regular expressions too with arguments. Use grep to filter standard output from another command student@linux-opstation-kspt:~$ ls -Rlisa /etc | grep password (1) 1137 4 -rw-r--r-- 1 root root 1440 Jan 31 2020 common-password 1156 4 -rw-r--r-- 1 root root 1160 Oct 9 2018 gdm-password ls: cannot open directory '/etc/polkit-1/localauthority': Permission denied (2) ls: cannot open directory '/etc/ssl/private': Permission denied ls: cannot open directory '/etc/sudoers.d': Permission denied 1 Execute ls -Rlisa then send its standard out to grep to filter for the string password. Use grep to search through a directory for text student@linux-opstation-kspt:~$ grep -R 'network' /etc/ (1) 1 Execute grep -R 'network' /etc/ then send it’s standard out to grep to filter for the string network. 2 The -R is recursive. Grep is a command that looks for a string of characters then presents the results to your screen. Grep is commonly used in conjunction with piping outputs from previous commands. But can also be used as a standalone command, with it’s own options/switches. 4.2 Awk awk is yet another important string manipulation tool. Unlike grep which searches for strings of text, awk allows you to reformat or select sections of text based on delimiters on the fly. Awk is commonly used to create tabular data sets from command output in Bash. However, it is a very flexible tool and its functionality does not end there. Reformat output from a command to create a comma delimited file with awk student@linux-opstation-kspt:~$ ls -l /etc (1) drwxr-xr-x 7 root root 4096 Feb 4 2020 NetworkManager drwxr-xr-x 2 root root 4096 Feb 4 2020 PackageKit drwxr-xr-x 2 root root 4096 Feb 4 2020 UPower _truncated_ student@linux-opstation-kspt:~$ ls -l /etc | awk -F " " '{print$3","$4","$9}' > files.csv (2) student@linux-opstation-kspt:~$ cat files.csv root,root,NetworkManager root,root,PackageKit root,root,UPower _truncated_ 1 The output from ls -l is verbose, maybe all of that information isn’t needed? 2 Lets send the output from ls -l into awk, then set the delimiter to blank space, then tell it to print fields $3,$4,$9, finally send them to a csv file. Crate a variable of all the news articles on https://dailymail.co.uk` student@linux-opstation-kspt:~$ articles=$(curl -L https://www.dailymail.co.uk/ushome/index.html --output - | grep itemprop | grep href | awk -F "\"" '{print$4}'| awk -F "/" '{print$4}') student@linux-opstation-kspt:~$ for article in $articles; do echo $article; done Rush-Limbaugh-dies-aged-70-lung-cancer-battle.html Facebook-BANS-Australians-sharing-news-war-publishers.html Congress-holds-hearing-reparations-slavery-time-BLM-protests-rocked-nation.html Kendall-Jenner-accused-cultural-appropriation-launching-tequila-brand.html MGM-Resorts-resume-24-7-operations-Mandalay-Bay-Park-MGM-Mirage-resorts-Las-Vegas.html _truncated_ 1 Perform a http GET request and filter all the HTML to get to the specific articles by grepping on itemprop, then grep on href, next use awk to cut output into fields separated by \ characters and select column 4, finally cut output into fields separated by / characters and select column 4. 2 Read the variable articles with a for loop. What you can do with these commands is limited by your imagination and knowledge. Go out and learn new ways to use them. Have fun! 4.3 Sed Sed is yet another string manipulation tool, but it edits text instead of filtering or formatting it like the other two. Sed is special because it edits text as it is sent to standard output. It is known as a stream editor. Text edited from sed can also be saved assuming the user executing it has the right permissions. Use sed to change standard out from the cat student@linux-opstation-kspt:~$ cat /etc/passwd | grep root (1) root:x:0:0:root:/root:/bin/bash student@linux-opstation-kspt:~$ cat /etc/passwd | grep root | sed s/root/bacon/g (2) bacon:x:0:0:bacon:/bacon:/bin/bash 1 Execute cat on /etc/passwd then filter the output with grep to filter for root. 2 Using sed to change standard any standard input that matches root to bacon, then send the modified output to the screen. Using sed to clean up the output from the Dailymail Variable in section 4.2 student@linux-opstation-kspt:~$ for article in $articles; do echo $article; done (1) Rush-Limbaugh-dies-aged-70-lung-cancer-battle.html Facebook-BANS-Australians-sharing-news-war-publishers.html Congress-holds-hearing-reparations-slavery-time-BLM-protests-rocked-nation.html for article in $articles; do echo $article |sed -e s/\.html//g -e s/\-/" "/g ; done (2) (3) Rush Limbaugh dies aged 70 lung cancer battle Facebook BANS Australians sharing news war publishers Police 7 shot near transit station north Philadelphia 1 The original output described in 4.2. It is messy and could be cleaned up a bit. 2 Using sed to replace standard input that matches .html with nothing. 3 Using sed to replace standard input from sed that matches the - (dash) character with a space. These are very limited examples, designed to show you the basics. With either Sed or Awk, the only limit is your imagination. 4.4 Regular Expressions Regular expressions, or regex, are a pattern matching language developed in the 1980s with the first use of the Unix operating system. Regex filters on patterns strings that may match multiple permutations. Most internet search engines, online shopping, and really any place there is a search button uses them too. Showing how regular expressions can match on multiple permutations of strings student@linux-opstation-kspt:~$ echo -e "Handel\nHändel\nHaendel" > regexfile (1) student@linux-opstation-kspt:~$ grep -P "H(ä|ae?)ndel" regexfile > (2) (3) Handel Händel Haendel 1 Create a file with 3 similar names in it and save it as regexfile in the current directory. 2 Use grep with -P to specify Perl regular expressions and look for : 3 H,ä or a, e is optional, ndel. Handel Example Reference Showing how regular expressions can match on multiple permutations of strings in files that are too large to search manually student@linux-opstation-kspt:~$ cat results.txt (1) 111-715-255643 (9279815)92-3599127 466-33836614-273 _truncated_ student@linux-opstation-kspt:~$ grep -P '\b\d{3}-\d{2}-\d{4}\b' results.txt 629-75-1985 386-67-7872 478-71-4964 student@linux-opstation-kspt:~$ grep -P '\(\d{3}\)\d{3}-\d{4}\b' results.txt (267)874-4532 (446)146-8923 (548)985-5415 (199)363-3617 1 Attempting to cat the a large file isn’t very helpful. 2 Execute grep to search for the following pattern : nnn-nn-nnnn. 3 Execute grep to search for the following pattern : (nnn)nnn-nnnn. 5. Resources 5.1 Linux Resources Command Cheat Sheet Reference guide containing syntax and examples for the most prevalent computing commands 5.2 Regex Resources Rexegg Regexone Regexr Regex Crosswords