quoll faq

Faq

Posted on Posted in Faq, System administrator
Vedi L'articolo in Italiano

In this article we collect all the FAQs sorted and grouped by category.

bash

Rename a group of files based on a regex

Suppose you want to rename the name of a group of files that have the following template:
file1_tmp.sh
file2_tmp.sh
...
file99_tmp.sh
and suppose you want to remove the ‘_tmp’ middle part in all the files. We can use the command “rename” or “prename”:
 rename 's/_tmp//' *.sh
 

Redirect the standard error to the standard output

To redirect the standard error of a shell command, just use the following syntax:

ls file-that-does-not-exist 2>&1 | grep file

Or even better not to have on the terminal the standard error just throw it in /dev/null

ls file-that-does-not-exist 2>/dev/null

The standard open or standard streams that are automatically opened are:

  • 0 standard input
  • 1 standard output
  • 2 standard error

 

Using simple arrays in bash

With Bash to explicitly declare an array you have to use

declare -A mioarray

To have the value of element 5 just write

echo $[mioarray[5]

While to change the value you have to write

mioarray[5]='new value'

${mioarray[*]} o ${mioarray[@]} is expanded by all array values while $[#mionome[@]} is expanded with the length of the array

What can we do with the arrays? For example, store the output of a command whose line we would like to go to a different item.

IFS=$'\n' mioarray=($(ls -alh))

At this point we can print output as

 IFS=$'\n' echo "${mioarray[*]}"

Attention: Double quotes are important for not having the output on a single line.

Doing so we could do several tests on the same captured output once.

 

 

Using associative arrays in bash

To declare an associative array, you must do so

declare -A mioarray

and as an index you can use a string, this means that

mioarray[test]='aa'
echo ${mioarray[test]} # returns 'aa'

 

Convert a decimal number to binary from Bash

Suppose we are working on a bash script and that for some need, not too difficult from reality, it is essential to have to convert a number or a decimal variable into a binary number.

There are of course several solutions but the one I propose does not make use of external programs but it is bash in all respects.

First we need to know the maximum value of the binary number to be obtained in Bash or in how many bits it can be included.

We consider that we are talking about a byte and therefore 8 bits and that its binary range will go from 00000000 to 11111111 (from 0 to 255) then we can write:

bin=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1})
echo ${bin[*]}
Convert a decimal number to binary from Bash
bash – binary number array

The more is done, in fact, to have the result for example of the number 125 or 78 in binary is enough

echo ${bin[125]}
01111101
echo ${bin[78]}
01001110

If the range of the binary number changes, for example if we double or quadruple it is enough that we define the bin variable with one or two more blocks of the type {0..1}.

So summing up, the bin variable will have to contain as many blocks {0..1} based on the maximum binary number to be obtained, that is, if the maximum number is 2n -1 then it will take n groups {0..1}.

As an example if n=10 and therefore 210 -1=1023 the blocks will be 10

bin=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1})
echo ${bin[1023]}
1111111111

Convert a decimal number to hexadecimal from Bash

As for the faq with the conversion to binary, also in this we make use of the fantastic bash braces extensions to convert to hexadecimal.

So, always considering the bash faq on the binary we start initially by converting a decimal number between 0 and 255 which corresponds to a hexadecimal digit between 0 and FF:

hex=({{0..9},{a..f}}{{0..9},{a..f}})
echo ${hex[*]}
how to convert a decimal number to hexadecimal with bash
bash- hexadecimal number of array

At this point to get the result of the number 200 or 21 is enough

echo ${hex[200]}
c8
echo ${hex[21]}
15

If the range of the hexadecimal number changes, for example if we double or quadruple it is enough that we define the hex variable with one or two more blocks of the type {0..9}, {a..f}.

So summing up, the hex variable will have to contain many blocks {0..9}, {a..f} based on the maximum binary number to be obtained, that is, if the maximum number is 16n -1 then it will take n groups {0. .9}, {a..f}.

As an example if n=4 and therefore 164 -1=65535 the blocks will be 4

hex=({{0..9},{a..f}}{{0..9},{a..f}}{{0..9},{a..f}}{{0..9},{a..f}})
echo ${hex[60000]}
ea60
echo ${hex[37]}
0025

How to convert an IPV4 netmask to CIDR

To switch from an IPV4 netmask to the equivalent CIDR it might occur to do the simplest thing, that is to use a nested “case” block or “if” to obtain the equivalent CIDR so much so that the available netmasks or CIDR are only 32 and that is

NetmaskAddress/CIDRAdresses
255.255.255.255a.b.c.d/3220
255.255.255.254a.b.c.d/3120
255.255.255.252a.b.c.d/3021
255.255.255.248a.b.c.d/2922
255.255.255.240a.b.c.d/2823
255.255.255.224a.b.c.d/2724
255.255.255.192a.b.c.d/2625
255.255.255.128a.b.c.d/2526
255.255.255.0a.b.c.0/2427
255.255.254.0a.b.c.0/2328
255.255.252.0a.b.c.0/2229
255.255.248.0a.b.c.0/21210
255.255.240.0a.b.c.0/20212
255.255.224.0a.b.c.0/19213
255.255.192.0a.b.c.0/18214
255.255.128.0a.b.c.0/17215
255.255.0.0a.b.0.0/16216
255.254.0.0a.b.0.0/15217
255.252.0.0a.b.0.0/14218
255.248.0.0a.b.0.0/13219
255.240.0.0a.b.0.0/12220
255.224.0.0a.b.0.0/11221
255.192.0.0a.b.0.0/10222
255.128.0.0a.b.0.0/9223
255.0.0.0a.0.0.0/8224
254.0.0.0a.0.0.0/7225
252.0.0.0a.0.0.0/6226
248.0.0.0a.0.0.0/5227
240.0.0.0a.0.0.0/4228
224.0.0.0a.0.0.0/3229
192.0.0.0a.0.0.0/2230
128.0.0.0a.0.0.0/1231
0.0.0.00.0.0.0/0232
Netmask to CIDR

If we want instead create a function that converts the netmask to CIDR we need to know how to convert a decimal number to binary first.

fromNetmaskToCidr() {
  local netmask=$1
  local bin=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1})
  local b=''
  local n
  local cidr
  for n in ${netmask//./ }; do
    if [ $n -gt 255 ]; then
      echo -e "\t[ERROR] netmask $netmask format error in '.$n'. I will use .255 insted of .$n"
      n=255
    fi
    if [ $n -ne 0 -a $n -ne 128 -a $n -ne 192 -a $n -ne 224 -a $n -ne 240 -a $n -ne 248 -a $n -ne 252 -a $n -ne 254 -a $n -ne 255 ]; then
      echo -e "\t[ERROR] netmask $netmask format error in '.$n' (it must be 0,128,192,224,240,248,252,254,255). I will use .255 insted of .$n"
      n=255
    fi
    # $b is the binary of $netmask
    b=${b}${bin[$n]}
  done
  # remove right "0" bits from $b
  b=${b/%0*}
  cidr=${#b}
  echo $cidr
}

What we do is convert the 4 octets of the netmask into binary and put them in series in the only variable “b” then remove the 0 bits present on the right of the overall binary number.
Finally, just count the number of bits left to get the CIDR which will then be returned on exit.

References

lvm

Adding a disk in LVM

Suppose you want to add a disk or partition in the Linux LVM handler and create a volume group named “quollvg” with a logical volume called “quolllv” of size 50GB.

Check the list of disks with the command:

fidisk -l

Suppose that /dev/sdb is the disk you want to add

We launch the command (lvm)

pvcreate /dev/sdb

If we are going to use a partition, such as disk 1, after creating and associating type 83 (LVM) we use the command:

pvcreate /dev/sdb1

At this point we create the volume group and the logical volume, format and mount the new disk:

vgcreate quollvg /dev/sdb
lvcreate -L 50G -n quolllv quollvg
mkdir /quolllv
mkfs.ext4dev -m 1 /dev/quolvg/quollv
mount /dev/mapper/quollvg-quolllv /quolllv

To see the status of logical volume and volume group:

lvdisplay
vgdisplay

 

 

Aggiungere un disco ad un volume group in LVM

Supponiamo di aver un volume group LVM di nome “quollvg” pieno o quasi pieno e di voler aggiungere dell’altro spazio avendo a disposizione un altro disco o un’altra partizione libera.

Supponiamo che il disco libero sia /dev/sdc (o /dev/sdc1) allora quello che dobbiamo fare è semplicemente:

pvcreate /dev/sdc
vgextend quollvg /dev/sdc

 

Aumentare la dimensione di una partizione LVM

Supponiamo di voler aumentare di 10G la dimensione della partizione ext4 del volume logico “quolllv” appartenente al volume group “quollvg”.

Per prima cosa controlliamo che il volume group “quollvg” abbia almeno 10GB liberi (sarebbe bene per un system administrator lasciare sempre libero in ogni volume group almeno dal 3% al 5% dello spazio nel caso in cui serva fare una snap per il backup o per qualche altro motivo)

vgdisplay quollvg

e controllare la sezione “Free PE / Size” per verificare che ci sia spazio disponibile.

Usare il comando per allargare il logical volume “quolllv“:

lvresize -L +10G /dev/quollvg/quolllv

A questo punto il logical volume “quolllv” è più grande di 10GB; bisogna quindi farlo sapere anche al file system ext4:

resize2fs /dev/mapper/quollvg-quolllv

Il resize2fs viene fatto anche caldo (a disco montato) quindi non preoccupatevi se è occupato da altro e non può esser smontato

Come creare una snapshot di un logical volume

Supponiamo di voler creare una snapshot del logical volume “quolllv” già esistente appartenente al volume group “quollvg“.

Controlliamo prima che ci sia spazio a sufficienza nel volume group:

vgdisplay quollvg

Controlliamo che “Free PE / Size” non sia tutto già occupato e supponiamo di avere 5GB di spazio da poter utilizzare per la snapshot:

lvcreate -L +5G -s -n "quolllv_snap" /dev/quollvg/quolllv

A questo punto abbiamo a disposizione la snapshot come verificabile digitando il comando:

lvdisplay

Possiamo quindi montarla ed utilizzarla:

mkdir /snap
mount /dev/mapper/quollvg-quolllv_snap /snap

Dopo averci lavorato (ad esempio dopo il backup) possiamo tranquillamente eliminare la snapshot:

umount /snap
lvremove -f /dev/quollvg/quolllv_snap
rmdir /snap

 

openssl

Extracting the contents of a digitally signed file (p7m)

Let’s consider the case where the signed document is a pdf (document.pdf.p7m).

To extract the content file we use the openssl command with the smime parameter for signature verification (-verify)

openssl smime -verify -noverify -in document.pdf.p7m -inform DER -out document.pdf

For more information, see the articol

Extraction of the certificate of a digitally signed files (p7m)

Let’s consider the case where the signed document is a pdf (document.pdf.p7m).

To extract the certificate used to sign the p7m we use the openssl command with the pkcs parameter

openssl pkcs7 -inform DER -in document.pdf.p7m -print_certs -out cert.pem

The certificate will be extracted in the cert.pem file. If you want to display textually enough the certificate command x509

openssl x509 -in cert.pem -text -noout

For more information, see the articol

Checking the signature of a p7m document

Consider the case where the signed document is a pdf (document.pdf.p7m).

To verify the signature and extract the content file, we use the openssl command with the smime parameter for signature verification (-verify)

openssl smime -in document.pdf.p7m -inform DER -verify -CAfile CA.pem -out document.pdf

where CA.pem contains the CA certificate (Certificate Authority) that issued the certificate in the p7m.

For more information, see the articol

Trust certificate authority per firma digitale

If you try to verify the signature of a p7m file without the Certification Authority (la CA), the verification fails (“unable to load certificate“) because we do not have CA trust certificates (so-called Trusts).
These Certificate Authorities have been defined by Italian law and are registered on the CNIPA, which since December 2009 has become DigitPA, as a certificate of XML certificates and found them on the same site at https://applicazioni.cnipa.gov.it/TSL/_IT_TSL_signed.xml.
To have openssl manage them you have to put them in its format so:

wget -O - https://applicazioni.cnipa.gov.it/TSL/_IT_TSL_signed.xml | perl -ne 'if (/<X509Certificate>/) {
s/^\s+//; s/\s+$//;
s/<\/*X509Certificate>//g;
print "-----BEGIN CERTIFICATE-----\n";
while (length($_)>64) {
print substr($_,0,64)."\n";
$_=substr($_,64);
}
print $_."\n";
print "-----END CERTIFICATE-----\n";
}' >CA.pem

This way we have all the certificates in a single CA.pem file, unfortunately even those that may have expired. Even if it’s over, we may need to check out an old file so we can also serve the expiration.
For more information, see the article

For script updates, please visit https://github.com/eniocarboni/getTrustCAP7m

perl

Extract all public GPG keys from a file

Suppose you have a file named “file-with-keys-gpg” and want to extract it by deleting any other line in the file.

perl -ne 'print if /-----BEGIN PGP PUBLIC KEY BLOCK-----/../-----END PGP PUBLIC KEY BLOCK-----/' file-with-keys-gpg

Of course by extrapolating the logic we might want to print all the lines between two regexp

perl -ne 'print if /regex1/../regex2/' file.txt

Add 1 to all numbers on a string

$str =~ s/(\d+)/$1+1/ge

Number a text file

Suppose we have a text file (prova.txt) that we want to number. We can use:

perl -pe '$_ = "$. $_"' prova.txt

If you want to simulate the output of cat -n prova.txt just use:

perl -ne 'printf "% 5d %s", $., $_' prova.tx

To remove the numbering we can use:

perl -pe 's/^\s*\d+\s*//' prova.txt

Encode/decode a file or string in Mime Base64

Suppose you have a “test encode” string and a “prova.txt” file you want to encode in Mime base64. To solve the problem we can simply launch the command line:

perl -MMIME::Base64 -e 'print encode_base64("test encode")'
perl -MMIME::Base64 -0777 -ne 'print encode_base64($_)' prova.txt

For the reverse operation just do:

perl -MMIME::Base64 -le 'print decode_base64("string in base64")'
perl -MMIME::Base64 -ne 'print decode_base64($_)' prova.base64

 

Just print line 25 of a file

Suppose you only want to print a single line of a file, which for convenience is indicated by line 25

perl -ne '$. == 25 && print && exit' file.txt

Of course if we also want to print lines 31 and 57

perl -ne '($. == 25 || $. == 31 || $. == 57) && print' file.txt

While we wanted all the lines between 25 and 57 included

perl -ne 'print if $. >= 25 && $. <= 57' file.txt

Rename a group of files based on a regex

Suppose you want to rename the name of a group of files that have the following template:

file1_tmp.sh
file2_tmp.sh
...
file99_tmp.sh

and suppose you want to remove the ‘_tmp’ middle part in all the files. We can use the command “rename” or “prename”:

 rename 's/_tmp//' *.sh

 

How to count the number of times a single character is present in each line of a file

Let’s suppose we have a text file named “file.csv” and want to count for each line the number of times a character is displayed, such as the character “,“.
Suppose the result is the output of the same file “file.csv” preceded in each line by the count itself.

Just simply the following single line perl:

perl -ne '@c=$_=~/;/g; print scalar(@c)." ". $_;' file.csv

 

 

ruby

Random password generation with Ruby

Suppose we want to generate a random, robust and fairly secure password using Ruby then we create a “random_password” function that returns a password with 12 characters or if we want one with more or less we have to pass this value as a parameter.

First we define the range of characters allowed for the password

$CHARS = ('0'..'9').to_a + ('A'..'Z').to_a + ('a'..'z').to_a + ('#'..'&').to_a + (':'..'?').to_a

then the function

def random_password(length=12)
  p=''
  (0..length).each do
    p+=$CHARS[rand($CHARS.size)]
  end
  return p
end

Finally, just use the newly created function to have the random password:

puts random_password
puts random_password(15)
test_pwd=random_password
puts test_pwd

ssh

ssh: come importare la chiave pubblica su un host

Supponiamo di aver già a disposizione le nostre chiavi pubbliche e private ssh o creiamole per la prima volta con il comando

ssh-keygen

Le chiavi create sono sotto $HOME/.ssh/id* e le pubbliche finiscono con .pub

Abilitare l’ssh-agent ed aggiungere le chiavi con il comando

ssh-add

Per importare la chiave sull’host “mioserver” di cui abbiamo accesso tramite l’utente “mioutente” basta il comando

ssh-copy-id mioutente@mioserver

In tal caso tutte le chiavi pubbliche vengono copiate nel server “mioserver“.
Se si vuol copiare solo la chiave “rsa” basta lanciare il comando

ssh-copy-id -i $HOME/.ssh/id_rsa mioutente@mioserver

 

Come cambiare la password ssh

ssh-keygen -p

Come lanciare un comando su un host remoto in ssh

Per lanciare un comando remoto basta passarlo come ultimo parametro al comando ssh.

Supponiamo ad esempio di volere la lista dei file della home page di “mioutente” su “mioserver

ssh mioutente@mioserver ls -alh

Per fare, invece,  il backup di tutta la home con il comando tar e salvare tutto in locale tramite comando ssh, lanciamo

ssh mioutente@mioserver tar czvf - . >miobackup.tgz

Utilizzare ssh come server/proxy SOCKS5

Supponiamo di voler utilizzare “mioserver” come server/proxy SOCKS5 e configurarlo sul browser locale facciamo

ssh mioutente@mioserver
mioutente@mioserver:~$ ssh -N -D 8080 localhost

A questo punto basta configurare nel browser l’utilizzo del server “mioserver” su porta “8080” come SOCKS5 e navigare.
Aggiungere l’opzione “-f” di ssh per lasciare il comando in background.

Come redirigere una connessione remota in locale tramite ssh

Supponiamo che l’host “mioserver” sia autorizzato ad accedere su una DMZ al server mysql della server farm, per poter accedere direttamente anche con il computer locale al server mysql possiamo lanciare il comando locale

ssh -L 3333:mysql_server_name_o_ip:3306 mioutente@mioserver

Lasciando il comando attivo ora abbiamo un canale/tunnel criptato tra il computer locale e “mioserver” sulla porta locale 3333 che si connette direttamente al server mysql in DMZ (di nome mysql_server_name_o_ip).
Possiamo quindi connetterci direttamente al server mysql con il comando

mysql -P 3333 -h 127.0.0.1 -u mioutentemysql -p miodatabasemysql

Oppure se si preferisce utilizzare phpMyAdmin basta installarlo in locale e configurarlo sulla porta locale 3333.

Per togliere il canale criptato basta chiudere la connessione ssh.

Redirigere un servizio locale in remoto con ssh

Supponiamo di volerci connetterci da casa al pc dell’ ufficio (che può accede al server “mioserver” in server farm)  non raggiungibile nemmeno in VPN; basta che dal pc di ufficio lanciamo il comando ssh:

ssh -N -f -R 2222:localhost:22 mioutente@mioserver

lasciandolo attivo (senza spegnere il computer).
Da casa basta abilitare la VPN per poter raggiungere “mioserver” e connettersi al server

ssh mioutente@mioserver
mioutente@mioserver:~$ ssh -P 2222 utente_pc_lavoro@localhost
utente_pc_lavoro@localhost:~$

In tal modo si riesce ad entrare nel pc dell’ufficio da casa.

 

Leave a Reply

Your email address will not be published. Required fields are marked *