Amigapallo A web development blog by Anssi Kinnunen

Collection of useful linux commands

Here’s a list of linux commands that I’ve gathered during the years. It seems that every time I want extract a tar.gz or chain couple of commands, I’ve totally forgotten how to do it. I don’t like reading man pages either. A good alternative to this is tldr but I’ve yet to install it. The plan is to keep updating this page whenever I have to Google how to use a certain command. I’m hoping this helps other people too!

Common bash commands

Redirect command stdout and stderr into /dev/null:

cat /non/existing/file &> /dev/null
non-existing-command > /dev/null 2>&1

Find all shell script files from a certain directory and make them executable:

find /etc -name "*.sh" -exec chmod +x {} \;

Repeat a command indefinitely between intervals:

while x=0; do \
    free -m | grep -e '^-' \
        | awk '{ print "Free memory: "$4 }'; \
    sleep 1; \
done

Echo multiple lines into a file:

cat > /tmp/lines.log << EOL
line 1
line 2
EOL

Show file chmod in numerical format:

stat --format '%a' /etc/sysctl.conf

Create and extract a tar.gz file:

# Create
tar -zcvf dump.sql.tar.gz dump.sql

# Extract
tar -zxvf dump.sql.tar.gz

Match with multiple words using awk, replace the line contents with sed using regex and run a command for each resulting lines:

cat /var/log/syslog \
    | awk '/error/ && /TLS/ {print $0}' \
    | sed -r 's/^.*number\s([0-9]+)\:$/\1/' \
    | while read line; do \
        echo "Error number: $line"; \
    done

# awk can be used to mach multiple columns also:
route -n | awk '{ if ($8 == "eth0" && $2 != "0.0.0.0") print $2; }'

Replace text in a file with sed:

sed -i s/#retry_files_enabled/retry_files_enabled/ \
    /etc/ansible/ansible.cfg

Set root password without prompt using expect:

#!/usr/bin/expect
spawn sudo -u root passwd
expect "Enter new UNIX password:"
send "root\r"
expect "Retype new UNIX password:"
send "root\r"
interact

Bash scripts

Get the directory of the executed script:

CWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

Promt user to confirm before continuing:

read -p "confirm? " -n 1 -r

if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo "Confirmed!"
fi

Conditional variable assignment and a default value:

env="development"
domain="blog.amigapallo.dev" \
    && [[ $env == "production" ]] && \
    domain="blog.amigapallo.org"

Validate script parameters and print usage instructions if validation fails:

[[ "$1" =~ [production|development] ]] \
    || { echo "Usage: build.sh development|production" >&2; exit 1; }

Do something if a file does not exist:

if [ ! -f /usr/bin/ansible ]; then
    echo "Does not exist!"
fi

Do something if the previous command exited with a failure status code:

if [ $? -ne 0 ]; then
    echo "Failure!"
else
    echo "Success!"
fi

Curl, wget and scp

Set the host header, user agent and basic auth for a curl request. Follow all redirects and ignore non valid (self signed) certificates:

curl --verbose --header 'Host: blog.amigapallo.org' \
    --header "User-Agent: Go-http-client" \
    -u username:password \
    -L \
    --insecure \
    'https://88.198.199.58:80'

Set headers and user agent for a wget request and discard the return data:

wget -O /dev/null http://google.com --server-response --user-agent="Go-http-client" --header="X-Forwarded-For: 172.217.1.206"

Make a SOAP request with curl:

echo "<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns2:getStuff xmlns:ns2="http://ejb.domain.com"><arg0>01110279</arg0><arg1>en</arg1></ns2:getStuff></S:Body></S:Envelope>" > soap.txt

curl --header "Content-Type: text/xml;charset=UTF-8" --data @soap.txt http://domain.com/get

Transfer a file to a remote location using scp:

scp dump.sql.tar.gz user@1.2.3.4:/home/user/dump.sql.tar.gz

Mail servers, ssl certificates, etc.

Show the certificate information of a mail server:

openssl s_client -starttls smtp -crlf -connect smtp.gmail.com:587

Telnet to a mail server and send an email:

# Copy paste the following commands line by line:
telnet 1.2.3.4 25
EHLO domain.com
MAIL FROM: sender@domain.com
RCPT TO: receiver@domain.com
DATA
Subject: Subject!
Body of the message!
.
quit

System management, statistics and monitoring

Get the group and user id from a directory:

GID=$(stat -c "%g" /home/amigapallo)
UID=$(stat -c "%u" /home/amigapallo)

Change user and group ids. The command above and this one may become handy when dealing with mounted volumes:

usermod -u $UID ftp
groupmod -g $GID ftp

# Note that when changing the user and group ids, the existing
# file and directory UIDs and GIDs should be changed also:

find / -gid $OLD_GID ! -type l -not -path '/proc/*' \
    -exec chgrp -h $GID {} \;

find / -user $OLD_UID -not -path '/proc/*' \
    -exec chown -h $UID {} \;

Show the number of connections to a specified port, group them by IP and sort by count:

netstat -plan | grep :80 | grep -v "::" | awk {'print $5'} \
    | cut -d: -f 1 | sort |uniq -c | sort -nk 1

Monitor incoming packets on a certain port and match against their data payload with a keyword:

ngrep -d any port 9052 | grep -i "GET"

Show the total memory and cpu usage for a specified user:

# rss = resident set size, vsz = virtual memory size
echo "rss(KiB) vmem(KiB)"; ps -U mysql --no-headers -o rss,vsz \
    | awk '{rss+=$1; vmem+=$2} END{print rss" "vmem}' | sort -k3

echo "%CPU"; ps -U mysql --no-headers -o %cpu \
    | awk '{cpu+=$1} END{print cpu}' | sort -k2

Monitor a log file for 10 seconds and count the lines that match to a certain keyword:

tail -n0 -f /opt/webserver/logs/access.log | grep "GET" > lines.log \
    & sleep 10; kill $! ; wc -l lines.log; rm lines.log

Kill processes matching to a keyword:

kill -9 `ps -ef | grep docker | grep -v grep | awk '{print $2}'`

Run command as another user using bash as the shell:

su -s /bin/bash mysql -c "env"

Switch to another user and use bash as the shell:

su -s /bin/bash jenkins

Get user and group name by id:

getent group 0 | cut -d: -f1
getent passwd 0 | cut -d: -f1

Managing rpm packages:

# List all installed packages matching to a keyword
rpm -qa | grep epel

# Install rpm package
rpm -ivh epel-release-6-7.noarch.rpm

# Remove (erase) rpm package
rpm -e epel-release-6-7.noarch