In this article we collect all the FAQs sorted and grouped by category.
bash
Rename a group of files based on a regex
file1_tmp.sh
file2_tmp.sh
...
file99_tmp.sh
and suppose you want to remove the ‘_tmp’ middle part in all the files. We can use the command “rename” or “prename”:
rename 's/_tmp//' *.sh
Redirect the standard error to the standard output
To redirect the standard error of a shell command, just use the following syntax:
ls file-that-does-not-exist 2>&1 | grep file
Or even better not to have on the terminal the standard error just throw it in /dev/null
ls file-that-does-not-exist 2>/dev/null
The standard open or standard streams that are automatically opened are:
- 0 standard input
- 1 standard output
- 2 standard error
Using simple arrays in bash
With Bash to explicitly declare an array you have to use
declare -A mioarray
To have the value of element 5 just write
echo $[mioarray[5]
While to change the value you have to write
mioarray[5]='new value'
${mioarray[*]} o ${mioarray[@]} is expanded by all array values while $[#mionome[@]} is expanded with the length of the array
What can we do with the arrays? For example, store the output of a command whose line we would like to go to a different item.
IFS=$'\n' mioarray=($(ls -alh))
At this point we can print output as
IFS=$'\n' echo "${mioarray[*]}"
Attention: Double quotes are important for not having the output on a single line.
Doing so we could do several tests on the same captured output once.
Using associative arrays in bash
To declare an associative array, you must do so
declare -A mioarray
and as an index you can use a string, this means that
mioarray[test]='aa'
echo ${mioarray[test]} # returns 'aa'
Convert a decimal number to binary from Bash
Suppose we are working on a bash script and that for some need, not too difficult from reality, it is essential to have to convert a number or a decimal variable into a binary number.
There are of course several solutions but the one I propose does not make use of external programs but it is bash in all respects.
First we need to know the maximum value of the binary number to be obtained in Bash or in how many bits it can be included.
We consider that we are talking about a byte and therefore 8 bits and that its binary range will go from 00000000 to 11111111 (from 0 to 255) then we can write:
bin=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1})
echo ${bin[*]}

The more is done, in fact, to have the result for example of the number 125 or 78 in binary is enough
echo ${bin[125]}
01111101
echo ${bin[78]}
01001110
If the range of the binary number changes, for example if we double or quadruple it is enough that we define the bin variable with one or two more blocks of the type {0..1}.
So summing up, the bin variable will have to contain as many blocks {0..1} based on the maximum binary number to be obtained, that is, if the maximum number is 2n -1 then it will take n groups {0..1}.
As an example if n=10 and therefore 210 -1=1023 the blocks will be 10
bin=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1})
echo ${bin[1023]}
1111111111
Convert a decimal number to hexadecimal from Bash
As for the faq with the conversion to binary, also in this we make use of the fantastic bash braces extensions to convert to hexadecimal.
So, always considering the bash faq on the binary we start initially by converting a decimal number between 0 and 255 which corresponds to a hexadecimal digit between 0 and FF:
hex=({{0..9},{a..f}}{{0..9},{a..f}})
echo ${hex[*]}

At this point to get the result of the number 200 or 21 is enough
echo ${hex[200]}
c8
echo ${hex[21]}
15
If the range of the hexadecimal number changes, for example if we double or quadruple it is enough that we define the hex variable with one or two more blocks of the type {0..9}, {a..f}.
So summing up, the hex variable will have to contain many blocks {0..9}, {a..f} based on the maximum binary number to be obtained, that is, if the maximum number is 16n -1 then it will take n groups {0. .9}, {a..f}.
As an example if n=4 and therefore 164 -1=65535 the blocks will be 4
hex=({{0..9},{a..f}}{{0..9},{a..f}}{{0..9},{a..f}}{{0..9},{a..f}})
echo ${hex[60000]}
ea60
echo ${hex[37]}
0025
How to convert an IPV4 netmask to CIDR
To switch from an IPV4 netmask to the equivalent CIDR it might occur to do the simplest thing, that is to use a nested “case” block or “if” to obtain the equivalent CIDR so much so that the available netmasks or CIDR are only 32 and that is
Netmask | Address/CIDR | Adresses |
255.255.255.255 | a.b.c.d/32 | 20 |
255.255.255.254 | a.b.c.d/31 | 20 |
255.255.255.252 | a.b.c.d/30 | 21 |
255.255.255.248 | a.b.c.d/29 | 22 |
255.255.255.240 | a.b.c.d/28 | 23 |
255.255.255.224 | a.b.c.d/27 | 24 |
255.255.255.192 | a.b.c.d/26 | 25 |
255.255.255.128 | a.b.c.d/25 | 26 |
255.255.255.0 | a.b.c.0/24 | 27 |
255.255.254.0 | a.b.c.0/23 | 28 |
255.255.252.0 | a.b.c.0/22 | 29 |
255.255.248.0 | a.b.c.0/21 | 210 |
255.255.240.0 | a.b.c.0/20 | 212 |
255.255.224.0 | a.b.c.0/19 | 213 |
255.255.192.0 | a.b.c.0/18 | 214 |
255.255.128.0 | a.b.c.0/17 | 215 |
255.255.0.0 | a.b.0.0/16 | 216 |
255.254.0.0 | a.b.0.0/15 | 217 |
255.252.0.0 | a.b.0.0/14 | 218 |
255.248.0.0 | a.b.0.0/13 | 219 |
255.240.0.0 | a.b.0.0/12 | 220 |
255.224.0.0 | a.b.0.0/11 | 221 |
255.192.0.0 | a.b.0.0/10 | 222 |
255.128.0.0 | a.b.0.0/9 | 223 |
255.0.0.0 | a.0.0.0/8 | 224 |
254.0.0.0 | a.0.0.0/7 | 225 |
252.0.0.0 | a.0.0.0/6 | 226 |
248.0.0.0 | a.0.0.0/5 | 227 |
240.0.0.0 | a.0.0.0/4 | 228 |
224.0.0.0 | a.0.0.0/3 | 229 |
192.0.0.0 | a.0.0.0/2 | 230 |
128.0.0.0 | a.0.0.0/1 | 231 |
0.0.0.0 | 0.0.0.0/0 | 232 |
If we want instead create a function that converts the netmask to CIDR we need to know how to convert a decimal number to binary first.
fromNetmaskToCidr() {
local netmask=$1
local bin=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1})
local b=''
local n
local cidr
for n in ${netmask//./ }; do
if [ $n -gt 255 ]; then
echo -e "\t[ERROR] netmask $netmask format error in '.$n'. I will use .255 insted of .$n"
n=255
fi
if [ $n -ne 0 -a $n -ne 128 -a $n -ne 192 -a $n -ne 224 -a $n -ne 240 -a $n -ne 248 -a $n -ne 252 -a $n -ne 254 -a $n -ne 255 ]; then
echo -e "\t[ERROR] netmask $netmask format error in '.$n' (it must be 0,128,192,224,240,248,252,254,255). I will use .255 insted of .$n"
n=255
fi
# $b is the binary of $netmask
b=${b}${bin[$n]}
done
# remove right "0" bits from $b
b=${b/%0*}
cidr=${#b}
echo $cidr
}
What we do is convert the 4 octets of the netmask into binary and put them in series in the only variable “b” then remove the 0 bits present on the right of the overall binary number.
Finally, just count the number of bits left to get the CIDR which will then be returned on exit.
References
lvm
Adding a disk in LVM
Suppose you want to add a disk or partition in the Linux LVM handler and create a volume group named “quollvg” with a logical volume called “quolllv” of size 50GB.
Check the list of disks with the command:
fidisk -l
Suppose that /dev/sdb is the disk you want to add
We launch the command (lvm)
pvcreate /dev/sdb
If we are going to use a partition, such as disk 1, after creating and associating type 83 (LVM) we use the command:
pvcreate /dev/sdb1
At this point we create the volume group and the logical volume, format and mount the new disk:
vgcreate quollvg /dev/sdb
lvcreate -L 50G -n quolllv quollvg
mkdir /quolllv
mkfs.ext4dev -m 1 /dev/quolvg/quollv
mount /dev/mapper/quollvg-quolllv /quolllv
To see the status of logical volume and volume group:
lvdisplay
vgdisplay
Aggiungere un disco ad un volume group in LVM
Supponiamo di aver un volume group LVM di nome “quollvg” pieno o quasi pieno e di voler aggiungere dell’altro spazio avendo a disposizione un altro disco o un’altra partizione libera.
Supponiamo che il disco libero sia /dev/sdc (o /dev/sdc1) allora quello che dobbiamo fare è semplicemente:
pvcreate /dev/sdc
vgextend quollvg /dev/sdc
Aumentare la dimensione di una partizione LVM
Supponiamo di voler aumentare di 10G la dimensione della partizione ext4 del volume logico “quolllv” appartenente al volume group “quollvg”.
Per prima cosa controlliamo che il volume group “quollvg” abbia almeno 10GB liberi (sarebbe bene per un system administrator lasciare sempre libero in ogni volume group almeno dal 3% al 5% dello spazio nel caso in cui serva fare una snap per il backup o per qualche altro motivo)
vgdisplay quollvg
e controllare la sezione “Free PE / Size” per verificare che ci sia spazio disponibile.
Usare il comando per allargare il logical volume “quolllv“:
lvresize -L +10G /dev/quollvg/quolllv
A questo punto il logical volume “quolllv” è più grande di 10GB; bisogna quindi farlo sapere anche al file system ext4:
resize2fs /dev/mapper/quollvg-quolllv
Il resize2fs viene fatto anche caldo (a disco montato) quindi non preoccupatevi se è occupato da altro e non può esser smontato
Come creare una snapshot di un logical volume
Supponiamo di voler creare una snapshot del logical volume “quolllv” già esistente appartenente al volume group “quollvg“.
Controlliamo prima che ci sia spazio a sufficienza nel volume group:
vgdisplay quollvg
Controlliamo che “Free PE / Size” non sia tutto già occupato e supponiamo di avere 5GB di spazio da poter utilizzare per la snapshot:
lvcreate -L +5G -s -n "quolllv_snap" /dev/quollvg/quolllv
A questo punto abbiamo a disposizione la snapshot come verificabile digitando il comando:
lvdisplay
Possiamo quindi montarla ed utilizzarla:
mkdir /snap
mount /dev/mapper/quollvg-quolllv_snap /snap
Dopo averci lavorato (ad esempio dopo il backup) possiamo tranquillamente eliminare la snapshot:
umount /snap
lvremove -f /dev/quollvg/quolllv_snap
rmdir /snap
openssl
Extracting the contents of a digitally signed file (p7m)
Let’s consider the case where the signed document is a pdf (document.pdf.p7m).
To extract the content file we use the openssl command with the smime parameter for signature verification (-verify)
openssl smime -verify -noverify -in document.pdf.p7m -inform DER -out document.pdf
For more information, see the articol
Extraction of the certificate of a digitally signed files (p7m)
Let’s consider the case where the signed document is a pdf (document.pdf.p7m).
To extract the certificate used to sign the p7m we use the openssl command with the pkcs parameter
openssl pkcs7 -inform DER -in document.pdf.p7m -print_certs -out cert.pem
The certificate will be extracted in the cert.pem em> file. If you want to display textually enough the certificate command x509
openssl x509 -in cert.pem -text -noout
For more information, see the articol
Checking the signature of a p7m document
Consider the case where the signed document is a pdf (document.pdf.p7m).
To verify the signature and extract the content file, we use the openssl command with the smime parameter for signature verification (-verify)
openssl smime -in document.pdf.p7m -inform DER -verify -CAfile CA.pem -out document.pdf
where CA.pem contains the CA certificate (Certificate Authority) that issued the certificate in the p7m.
For more information, see the articol
Trust certificate authority per firma digitale
If you try to verify the signature of a p7m file without the Certification Authority (la CA), the verification fails (“unable to load certificate“) because we do not have CA trust certificates (so-called Trusts).
These Certificate Authorities have been defined by Italian law and are registered on the CNIPA, which since December 2009 has become DigitPA, as a certificate of XML certificates and found them on the same site at https://applicazioni.cnipa.gov.it/TSL/_IT_TSL_signed.xml.
To have openssl manage them you have to put them in its format so:
wget -O - https://applicazioni.cnipa.gov.it/TSL/_IT_TSL_signed.xml | perl -ne 'if (/<X509Certificate>/) {
s/^\s+//; s/\s+$//;
s/<\/*X509Certificate>//g;
print "-----BEGIN CERTIFICATE-----\n";
while (length($_)>64) {
print substr($_,0,64)."\n";
$_=substr($_,64);
}
print $_."\n";
print "-----END CERTIFICATE-----\n";
}' >CA.pem
This way we have all the certificates in a single CA.pem file, unfortunately even those that may have expired. Even if it’s over, we may need to check out an old file so we can also serve the expiration.
For more information, see the article
For script updates, please visit https://github.com/eniocarboni/getTrustCAP7m
perl
Extract all public GPG keys from a file
Suppose you have a file named “file-with-keys-gpg” and want to extract it by deleting any other line in the file.
perl -ne 'print if /-----BEGIN PGP PUBLIC KEY BLOCK-----/../-----END PGP PUBLIC KEY BLOCK-----/' file-with-keys-gpg
Of course by extrapolating the logic we might want to print all the lines between two regexp
perl -ne 'print if /regex1/../regex2/' file.txt
Add 1 to all numbers on a string
$str =~ s/(\d+)/$1+1/ge
Number a text file
Suppose we have a text file (prova.txt) that we want to number. We can use:
perl -pe '$_ = "$. $_"' prova.txt
If you want to simulate the output of cat -n prova.txt just use:
perl -ne 'printf "% 5d %s", $., $_' prova.tx
To remove the numbering we can use:
perl -pe 's/^\s*\d+\s*//' prova.txt
Encode/decode a file or string in Mime Base64
Suppose you have a “test encode” string and a “prova.txt” file you want to encode in Mime base64. To solve the problem we can simply launch the command line:
perl -MMIME::Base64 -e 'print encode_base64("test encode")'
perl -MMIME::Base64 -0777 -ne 'print encode_base64($_)' prova.txt
For the reverse operation just do:
perl -MMIME::Base64 -le 'print decode_base64("string in base64")'
perl -MMIME::Base64 -ne 'print decode_base64($_)' prova.base64
Just print line 25 of a file
Suppose you only want to print a single line of a file, which for convenience is indicated by line 25
perl -ne '$. == 25 && print && exit' file.txt
Of course if we also want to print lines 31 and 57
perl -ne '($. == 25 || $. == 31 || $. == 57) && print' file.txt
While we wanted all the lines between 25 and 57 included
perl -ne 'print if $. >= 25 && $. <= 57' file.txt
Rename a group of files based on a regex
Suppose you want to rename the name of a group of files that have the following template:
file1_tmp.sh
file2_tmp.sh
...
file99_tmp.sh
and suppose you want to remove the ‘_tmp’ middle part in all the files. We can use the command “rename” or “prename”:
rename 's/_tmp//' *.sh
How to count the number of times a single character is present in each line of a file
Let’s suppose we have a text file named “file.csv” and want to count for each line the number of times a character is displayed, such as the character “,“.
Suppose the result is the output of the same file “file.csv” preceded in each line by the count itself.
Just simply the following single line perl:
perl -ne '@c=$_=~/;/g; print scalar(@c)." ". $_;' file.csv
ruby
Random password generation with Ruby
Suppose we want to generate a random, robust and fairly secure password using Ruby then we create a “random_password” function that returns a password with 12 characters or if we want one with more or less we have to pass this value as a parameter.
First we define the range of characters allowed for the password
$CHARS = ('0'..'9').to_a + ('A'..'Z').to_a + ('a'..'z').to_a + ('#'..'&').to_a + (':'..'?').to_a
then the function
def random_password(length=12)
p=''
(0..length).each do
p+=$CHARS[rand($CHARS.size)]
end
return p
end
Finally, just use the newly created function to have the random password:
puts random_password
puts random_password(15)
test_pwd=random_password
puts test_pwd
ssh
ssh: come importare la chiave pubblica su un host
Supponiamo di aver già a disposizione le nostre chiavi pubbliche e private ssh o creiamole per la prima volta con il comando
ssh-keygen
Le chiavi create sono sotto $HOME/.ssh/id* e le pubbliche finiscono con .pub
Abilitare l’ssh-agent ed aggiungere le chiavi con il comando
ssh-add
Per importare la chiave sull’host “mioserver” di cui abbiamo accesso tramite l’utente “mioutente” basta il comando
ssh-copy-id mioutente@mioserver
In tal caso tutte le chiavi pubbliche vengono copiate nel server “mioserver“.
Se si vuol copiare solo la chiave “rsa” basta lanciare il comando
ssh-copy-id -i $HOME/.ssh/id_rsa mioutente@mioserver
Come cambiare la password ssh
ssh-keygen -p
Come lanciare un comando su un host remoto in ssh
Per lanciare un comando remoto basta passarlo come ultimo parametro al comando ssh.
Supponiamo ad esempio di volere la lista dei file della home page di “mioutente” su “mioserver”
ssh mioutente@mioserver ls -alh
Per fare, invece, il backup di tutta la home con il comando tar e salvare tutto in locale tramite comando ssh, lanciamo
ssh mioutente@mioserver tar czvf - . >miobackup.tgz
Utilizzare ssh come server/proxy SOCKS5
Supponiamo di voler utilizzare “mioserver” come server/proxy SOCKS5 e configurarlo sul browser locale facciamo
ssh mioutente@mioserver
mioutente@mioserver:~$ ssh -N -D 8080 localhost
A questo punto basta configurare nel browser l’utilizzo del server “mioserver” su porta “8080” come SOCKS5 e navigare.
Aggiungere l’opzione “-f” di ssh per lasciare il comando in background.
Come redirigere una connessione remota in locale tramite ssh
Supponiamo che l’host “mioserver” sia autorizzato ad accedere su una DMZ al server mysql della server farm, per poter accedere direttamente anche con il computer locale al server mysql possiamo lanciare il comando locale
ssh -L 3333:mysql_server_name_o_ip:3306 mioutente@mioserver
Lasciando il comando attivo ora abbiamo un canale/tunnel criptato tra il computer locale e “mioserver” sulla porta locale 3333 che si connette direttamente al server mysql in DMZ (di nome mysql_server_name_o_ip).
Possiamo quindi connetterci direttamente al server mysql con il comando
mysql -P 3333 -h 127.0.0.1 -u mioutentemysql -p miodatabasemysql
Oppure se si preferisce utilizzare phpMyAdmin basta installarlo in locale e configurarlo sulla porta locale 3333.
Per togliere il canale criptato basta chiudere la connessione ssh.
Redirigere un servizio locale in remoto con ssh
Supponiamo di volerci connetterci da casa al pc dell’ ufficio (che può accede al server “mioserver” in server farm) non raggiungibile nemmeno in VPN; basta che dal pc di ufficio lanciamo il comando ssh:
ssh -N -f -R 2222:localhost:22 mioutente@mioserver
lasciandolo attivo (senza spegnere il computer).
Da casa basta abilitare la VPN per poter raggiungere “mioserver” e connettersi al server
ssh mioutente@mioserver
mioutente@mioserver:~$ ssh -P 2222 utente_pc_lavoro@localhost
utente_pc_lavoro@localhost:~$
In tal modo si riesce ad entrare nel pc dell’ufficio da casa.