Scripts --- homemade helpers !!

[Abarbarian]

Personalising or customising how your pc looks or operates is one of the attractions of being a penguinista. Taking control and making your pc perform as you want can be accomplished with simple scripts.
Learning how to use bash or python etc etc is a daunting task for most folk. Luckily there are plenty of script surfers out there on the net eager to share their creations. The trouble is finding them and quite often understanding how they are made and how you can alter them to suit you.
So in this thread I will try to post scripts that are hopefully useful and are either pretty self explanatory or contain clear and understandable instructions.

Here we go with the first one.

Build a custom system tray indicator using Python

"System Tray icons are still considered to be an amazing functionality today. By just right-clicking on the icon, and then selecting which actions you would like to take, you may ease your life a lot and save many unnecessary clicks on daily basis.

When talking about useful system tray icons, examples like Skype, Dropbox and VLC do come to mind:

However, system tray icons can actually be quite a lot more useful; By simply building one yourself for your own needs. In this tutorial, we’ll explain how to do that for you in very simple steps."

 

[Ian]

I've got a few Python scripts on mini servers (to do things like reconnect wifi if it drops, or run patch updates etc...) but they're all very basic things so far. However, I've been using PowerShell in Windows quite a lot recently and I'm pleasantly surprised at how powerful it is. It took me a while to get my head around using it, but I'm a total convert now - it's so much more powerful than the old batch file scripting that I used so much.

On Linux, are scripts like this fairly release agnostic? i.e. would it run on Ubuntu and Fedora with the same code? The example on that link does (with a slightly different install) - is that generally the case for any Python script that interfaces with the GUI?

 

[Abarbarian]

I've got a few Python scripts on mini servers (to do things like reconnect wifi if it drops, or run patch updates etc...) but they're all very basic things so far. However, I've been using PowerShell in Windows quite a lot recently and I'm pleasantly surprised at how powerful it is. It took me a while to get my head around using it, but I'm a total convert now - it's so much more powerful than the old batch file scripting that I used so much.

On Linux, are scripts like this fairly release agnostic? i.e. would it run on Ubuntu and Fedora with the same code? The example on that link does (with a slightly different install) - is that generally the case for any Python script that interfaces with the GUI?

As long as you have the correct version of Python plus all the dependencies needed by a program then it should run on almost any linux as far as I know. Same goes for Bash scripts which also may run if you are running zsh or fish, I think.
If you are running Python 2 and you find a Python 3 script you may be able to get it to run by simply replacing 2 with 3 etc etc .
I am not very up on scripts but I have made a couple to help out with maintenence on my set up.
It is great when you find or create a small script that does in a click or two what used to take half a dozen clicks.

 

[Abarbarian]

So you made a script to automate a task, and set it to run. How do you know that it will, keep on running or exit cleanly ? Simple , you TRAP the little blighter right form the start.
Using Trap to Exit Bash Scripts Cleanly

We have all been there, you write a bash script that creates some temporary files and/or directories for processing some information. You complete your script with some basic testing and all is well. You set your cron job to run the script every day at noon. Six days later you realize that the script has been exiting prematurely and leaving a bunch of trash files in the file system. Or worse, potentially sensitive data is left unprotected. Enter the trap command.

The above scenario is a bit dramatic, but not completely ridiculous. If you are automating tasks with bash scripts you will eventually run into a premature exit. Either as the result of an error, a change in the environment, or an unanticipated user action. Creating scripts that deal with this scenario is imperative to keeping a clean and secure system.

So remember when bashing a penguin remember to trap it first.

 

[Abarbarian]

Tired of killing the same old enemies and aliens in the same old ways. Take a break and create a useful script to take the drudgery out of your computer life.

Example of rsync for simple backups with alias's

I have recently been looking into various ways to make backups, both data and full system backups and there are some excellent methods out there. I tried out a few different offerings and found them a tad too un KISS for my simple needs. Besides I like to tinker.

I have my Arch set up with,

Boot in a EFI partition

Root on a partition

Home on a partition

I wanted to make backups of data in my /home and as I am using Arch more for gaming I wanted to make backups of my root partition as linux game data is kept there aswell as in the /home partition. I am not sure what game files are kept where but there seem to be an awful lot of them spread around all over the place> I can not be bothered tracking down what is where and does it need saving so decided to just make backups of whole partitions with some exclusions.

I have an external usb3 dock where I can slot in two sata drives if needed. This is very handy as I can utilise all my older hdd's and newer ssd's. The dock is only turned on when I need to use it. This means that a lot of the backup solutions that offer scheduled automatic backups or cron jobs are not much use to me.

So I settled on using rsync. I made three rsync commands and made alias's for them and they created folders on my backup drive called HISTORY and saved the necessary data to them. The first run of the commands take some time to run, half an hour to three quarters of an hour, as my home is about 80 GB and root is about 38 GB, boot is a mere 250 MB. On subsequent runs only a couple of minutes is needed, time depends mainly on how many different browsers I have used and how many updates pacman has done.

These are the alias commands,

alias    bback="sudo rsync -vaAXHShix --delete /boot/ /run/media/bloodaxe/HISTORY/BOOT"

alias    sback="sudo rsync -vaAXHShix --delete --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/home/","/var/lib/dhcpcd/*","/lost+found"} / /run/media/bloodaxe/HISTORY/SYSTEM"

alias    hback="rsync -vaHAXShix --delete --exclude=slots --exclude=tempfile --exclude=.local/share/Trash --exclude=projects --exclude=.local/share/gvfs-metadata --exclude=/home/*/.gvfs /home/bloodaxe/  /run/media/bloodaxe/HISTORY/HOME"

here is a sample output after they have run,truncated as the output can be quite long.

16:54:14-->Wed May 27-->~
-->bback
sending incremental file list
*deleting   EFI/BOOT/icons-backup/os_trusty.png
*deleting   EFI/BOOT/icons-backup/os_mac.png
*deleting   EFI/BOOT/icons-backup/mouse.png
.d..t...... EFI/BOOT/icons-backup/
sent 7.62K bytes  received 151 bytes  15.55K bytes/sec
total size is 82.04M  speedup is 10,555.11



16:54:24-->Wed May 27-->~
-->sback
sending incremental file list
.d..t...... dev/
.d..t...... etc/
>f..t...... etc/ld.so.cache
.d..t...... root/
>f.st...... root/.bash_history
.d..t...... tmp/
.d..t...... usr/
.d..t...... usr/bin/
>f..t...... usr/bin/ffmpeg
>f..t...... usr/bin/ffplay
>f..t...... usr/include/libavcodec/avcodec.h
*deleting   var/lib/pacman/local/ffmpeg-1:4.2.3-1/mtree
*deleting   var/lib/pacman/local/ffmpeg-1:4.2.3-1/files
*deleting   var/lib/pacman/local/ffmpeg-1:4.2.3-1/desc
*deleting   var/lib/pacman/local/ffmpeg-1:4.2.3-1/
.d..t...... var/lib/pacman/
.d..t...... var/lib/pacman/local/
cd+++++++++ var/lib/pacman/local/ffmpeg-1:4.2.3-2/
>f+++++++++ var/lib/pacman/local/ffmpeg-1:4.2.3-2/desc
sent 310.49M bytes  received 39.60K bytes  7.48M bytes/sec
total size is 39.58G  speedup is 127.45



16:55:14-->Wed May 27-->~
-->hback
sending incremental file list
.d..t...... ./
>f.st...... .bash_history
>f.st...... client_state.xml
>f.st...... client_state_prev.xml
>f..t...... daily_xfer_history.xml
>f.st...... job_log_www.worldcommunitygrid.org.txt
*deleting   .cache/mozilla/firefox/2wtpmdoa.default/cache2/ce_YSwJaHR0cHM6Ly93d3cuYW1hem9uLmNvLnVr
.d..t...... notices/
>f..t...... notices/archive_www.worldcommunitygrid.org_viewNoticesRSSFeed.action.xml
>f..t...... notices/www.worldcommunitygrid.org_viewNoticesRSSFeed.action.xml
sent 43.75M bytes  received 27.91K bytes  1.79M bytes/sec
total size is 78.03G  speedup is 1,782.43
16:55:47-->Wed May 27-->~
-->

I may have gone overboard with the rsync switches but I wanted to cover all the bases. I included the " i " option as it gives a nice output of what is going on with each file. You get a "f" for a file "d" for directory,the "s" and"t" tell you size and time has altered and "deleting" is obvious.

I still have some fine tuning to do as I run BOINC and there are a lot of small files relating to that in /home that I do not need to backup.Adding them to the rsync command would make for a very long command so I will use the exclude from text option at some time later on.

Whilst searching for information I came across information for using bash to run several commands one after the other so I created a alias to run all three backup commands like so,

alias    allback="bback && sback && hback"

I have to have a separate command for boot as it is on its own partition and I have included the "x" option which stops rsync crossing filesystem boundaries when recursing. Good job I checked the contents of the system backup whilst doing a trial.

Also if playing around with rsync it is best to use the " n " dry run option first before doing a real run through.

When I have time I am going to see if I can successfully transfer the three backups to another ssd and get a functional system running. According to the Arch wiki that should be possible. I need bit of a rest for now so will try at some future date.

 

[Abarbarian]

Still bored I decided to refine my backup strategy,

Example of rsync as script with functions for simple backups

Arch has a folder " ~/bin " for placing user made scripts. Scripts placed there can be called up and run by simply using the scripts name. I can not remember if Arch automatically makes this "~/bin" folder on a new install. If it does then it will be included in your users "$PATH". If you have the folder you can check to see if it included in your user $PATH by running "set",

 Put brain in gear befor pressing enter14:27:35-->Sun May 31-->~
-->set
BASH=/bin/bash
OSTYPE=linux-gnu
PATH=/usr/lib/ccache/bin/:/usr/lib/ccache/bin/:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/bloodaxe/bin:/usr/share/applications:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/bloodaxe/bin:/usr/share/applications
Put brain in gear befor pressing enter14:27:35-->Sun May 31-->~
-->

You can see that I have the "~/bin" (/home/bloodaxe/bin)folder in my $PATH.

If you do not have the "~/bin" folder open a terminal create it, make sure that your "pwd" is "~/" when doing this,

Put brain in gear befor pressing enter14:44:29-->Sun May 31-->~
-->mkdir bin

Then add this to your "~/.baschrc",

export PATH=$PATH:$HOME/bin

and do

 Put brain in gear befor pressing enter14:58:19-->Sun May 31-->~
-->source ~/.bashrc

Now you can make a script. Go to "~/bin" and make a new file give it the name you wish to call your script. Mine is called "dback" shorthand for dailybackup.In the script I have made three functions to back up data from three different partitions, "/boot", my root "/", and "/home" with a fourth function that runs the three scripts in order.The destination for the backups are folders on an external usb drive,BOOT,SYSTEM and HOME. The drive is mounted as "/run/media/bloodaxe/HISTORY".

Now to use the script I run "dback" in a terminal.

Put brain in gear befor pressing enter14:58:14-->Sun May 31-->~
-->dback

However my backup destination is a external drive and if it is not switched on or the drive is not mounted then this happens if I run "dback",

 Put brain in gear befor pressing enter14:58:14-->Sun May 31-->~
-->dback
sending incremental file list
rsync: mkdir "/run/media/bloodaxe/HISTORY/BOOT" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(664) [Receiver=3.1.3]
sending incremental file list
rsync: mkdir "/run/media/bloodaxe/HISTORY/SYSTEM" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(664) [Receiver=3.1.3]
sending incremental file list
rsync: mkdir "/run/media/bloodaxe/HISTORY/HOME" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(664) [Receiver=3.1.3]
Put brain in gear befor pressing enter14:58:19-->Sun May 31-->~
-->

With the drive mounted running "dback" gives me the expected result as detailed in the previous post.

Here is the script,

#!/bin/bash
#backups for /boot , / , and /home. Scripts runs all three functions.

# /boot backup
bback () {
    sudo rsync -vaAXHShix --delete \
         /boot/ \
         /run/media/bloodaxe/HISTORY/BOOT
}

# root backup
sback () {
     sudo rsync -vaAXHShix --delete \
     --exclude=/dev/* \
     --exclude=/proc/* \
     --exclude=/sys/* \
     --exclude=/tmp/* \
     --exclude=/run/* \
     --exclude=/mnt/* \
     --exclude=/media/* \
     --exclude=/home/* \
     --exclude=/var/lib/dhcpcd/* \
     --exclude=/lost+found \
     / \
     /run/media/bloodaxe/HISTORY/SYSTEM
}

# /home backup
hback () {
    rsync -vaHAXShix --delete \
        --exclude=slots \
        --exclude=tempfile \
        --exclude=.local/share/Trash \
        --exclude=projects \
        --exclude=.local/share/gvfs-metadata \
        --exclude=/home/*/.gvfs \
        /home/bloodaxe/ \
        /run/media/bloodaxe/HISTORY/HOME
}

# Establish run order
main ()  {
      bback
      sback
      hback
}

main

I still have some work to do on this script, it does work as is. Though I would like to exclude some more BOINC folders and files to speed up the backup and save some space. To do this I will create a "exclude text file" and add it to the script.I'll post the new script later on once I have worked out exactly how to do it and tested it.

 

[Abarbarian]

I forgot to include this rather important piece of information from the last post and as the site will not let me edit the post I am posting it here.

After you have created your script.

The next thing we have to do is give the shell permission to execute your script. This is done with the chmod command as follows

Put brain in gear befor pressing enter12:25:02-->Mon Jun 01-->~
-->chmod 700 dback

Using "755" will give you read, write, and execute permission. Everybody else will get only read and execute permission. If you want your script to be private (i.e., only you can read and execute), use "700" instead.