Table of Contents

Unixy Stuff

I run Linux at home, and use it constantly at work. On the those few occasions then I have to work on a Windows box I use Cygwin (a unix-like shell for Windows(TM)).

Note: This are also includes web-related things, such as Apache, which arn't really specific to unix & linux systems

Backups

Use https://pypi.org/project/duplicity/

Cygwin

I use Cygwin so much, that I need to keep some tips on it here.

Finding directories NOT containing a specific file

(find -mindepth 2 -maxdepth 2 | grep UNWANTED_FILE_NAME | cut -d/ -f2; command ls) | sort | uniq -u

This creates to output streams, one from the find command and one from ls, that are combined, sorted and finally filtered to remove any entries that occurs more than one.

The streams are combined by the parenthesis which create a sub-shell.

Finding directories, without examining .snapshots

find . -type d \( -name .snapshot -prune -o -name .git \)

Looking in the current directory, ., find entries of type d (directories) for which: if the name is snapshot remove it from consideration, if the name is .git carry on (to implicitly -print the entries to STDOUT).

Renaming

There is actually a rename command, which can be used to change extensions thus: 

rename .rpmnew '' *.rpmnew

This would perform the following: 

main.cf.rpmnew -> main.cf
master.cf.rpmnew -> master.cf

History Substitution

(Explained to me by The Definitive Guide to Bash Command Line History)

If you want to reuse the arguments from your last command, you can do so like so: 

svn status file1 file2 file3
svn diff !!:2  # Equivalent to "svn diff file1 file2 file3"

So that's an “event designator” of !! to select the previous command, coupled with a “word designator” of 2 to select arguments 2 onwards (zero indexed, of course). These are separated by a colon.

Make STDERR red

Add this function to your BASH environment (e.g. copy'n'paste it to your ~/.bash_profile), and you will be able to make STDERR output from your commands appear red, making them easier to see. 

# Red STDERR
# rse <command string>
function rse()
{
	# We need to wrap each phrase of the command in quotes to preserve arguments that contain whitespace
	# Execute the command, swap STDOUT and STDERR, colour STDOUT, swap back
	((eval $(for phrase in "$@"; do echo -n "'$phrase' "; done)) 3>&1 1>&2 2>&3 | sed -e "s/^\(.*\)$/$(echo -en \\033)[31;1m\1$(echo -en \\033)[0m/") 3>&1 1>&2 2>&3
}

Usage

Typical usage is as simple as placing the characters “rse ” infront of the command you want to affect the output of.

Avoid colouring the output of highly interactive programs, such as ncurses-based program (aptitude, cscope etc) or bash. You will get unexpected side-affects.

For instance, try rse bash and you will notice that it “works” in the strict sense, but isn't useful: You will not see a prompt, nor will tab-completion work or other shell-shortcuts (such as UP/DOWN to scroll history), however you will be able to run commands in the usual way and see their output coloured.

If someone knows a work-around for this, please let me know!

Suppose you have a script called rse_test.sh: 

echo STDOUT
echo STDERR >&2

you would colour its output like so: 

rse ./rse_test.sh

and see 

STDOUT
STDERR

However, the following does not colour the output: 

rse echo ERROR >&2

“ERROR” will be printed in the usual colour, because the redirection directive is intepreted by the shell, and not by the “rse” function. Adding parenthesis would group the commands like so: 

( rse ( echo ERROR ) ) >&2

How it works

rse wraps all lines send to STDERR with control characters which mean “set foreground colour to red” and “restore default foreground colour”.

Colouring

The control characters are ANSI escape code sequences which most1) modern terminals understand.

The escape sequences in the code are: 

^[[31;1m -- Set foreground to red (32), set bold (1)
^[[0m    -- Restore default style
I have written “^[” to mean the ESC character, but I have done this by typing two seperate characters - therefore copy'n'pasting this text will have no effect.

To type an ESC character at your shell, press CTRL+V and then hit ESC. You will notice that if you hit backspace both the “^” and “[” characters are removed as once.

In my ~/.bash_profile I have raw ESC characters, but these cannot be (reliably)2) displayed in browsers. So, as a work-around, I have used command-substitution in the code above.

The following text is replaced with an ESC character when it is evalutated by BASH: 

$(echo -en \\033)

The “-e” passed to echo tells it to evaluate the text it will be echo'ing, in this case meaning that “\\033” becomes a single character with octal value 33; the ESC char. The “-n” stops echo printing a newline.

Wrapping lines is achieved by using the sed command: 

sed -e "s/^\(.*\)$/$(echo -en \\033)[31;1m\1$(echo -en \\033)[0m/"

We provide a sed-script on the command line and ask sed to evaluate it with the “-e” option. The script is a single substitute command, of the form: “s/before/after/”.

Our “before” patterns is a match of the entire line: “^.*$” where we capture the line using escaped-parenthesis: “^\(.*\)$”.

Our “after” pattern is essentially: “{RED}\1{NORMAL}” where “\1” is the text that was captured. RED and NORMAL are as described above when explaining ANSI escape code sequences.

Applying the colouring to STDERR

Every process has three file-handles by default:

Number Name Purpose
0 STDIN This is how keystrokes are communicated to the program
1 STDOUT This is where “normal” output is sent
2 STDERR This is where exceptional output, such as errors, are sent

If you are running only a single program, then the distinction between STDOUT and STDERR is arbitrary - they will both appear on your screen. They become significant when you have a pipe-line of programs.

For example, suppose you want to know how many files are in the directory tree under your current directory. You can list all the files using the “find” command, and you will output something like this: 

find: ./ssl/private: Permission denied
find: ./cups/ssl: Permission denied
find: ./vpnc: Permission denied
.
./acpi
./acpi/ac.d
./acpi/ac.d/85-anacron.sh
./acpi/ac.d/cpufreq.sh
./acpi/ac.d/hal-disable-polling.sh
./acpi/ac.d/mount-noatime.sh
./acpi/ac.d/vm-dirty-writeback.sh
./acpi/ac.d/wired-enable-auto-negoatiation.sh
./acpi/ac.d/wireless_set_power.sh
  ...

And you can count the number of lines printed with the word-count utility by specifying the “-l” option for lines: “wc -l”.

Putting these together we get “find | wc -l”, where the output from “find” will be sent as input to “wc -l”. When I run this I get: 

find: ./ssl/private: Permission denied
find: ./cups/ssl: Permission denied
find: ./vpnc: Permission denied
2636

Note that the errors from “find” are still displayed, but so is a line-count from “wc”. This is because the errors were printed on STDERR, which is not sent to “wc” as input.

When two programs are piped together, STDOUT is connected to STDIN. STDERR, on the other hand, remains connected to the parent process, which is our shell in this case. Therefore error messages are not processed by the pipe-line. In our example this was useful, we ended up counting the number of files which we have access to. If error messages were printed to STDOUT, then our count-number would be inaccurate.

How is this relevant? Well, it is not possible to manipulate STDERR in a pipeline, so an alternative approach was required.

Although you cannot manipulate STDERR, you can ask your shell to redirect it. The most common use of this is to suppress errors by redirecting them to /dev/null. However, you can do more than redirect them to files.

rse works by swapping STDOUT and STDERR, modifying the new STDOUT, and then swapping them back. This is achieved by nesting subshells, and having each subshell instruct its parent to swap the streams: 

( ( {USER-CMD} ) 3>&1 1>&2 2>&3 | {MODIFY-NEW-STDOUT} ) 3>&1 1>&2 2>&3

Where “3>&1 1>&2 2>&3” instructs the parent to swap STDOUT/STDERR. This is parsed from right-to-left, and uses the numbers in the table above. This works by shifting both STDOUT/STDERR up by one number and then moving the top-most one to the bottom; effectively a swap. Therefore it reads: “Renumber STDERR as #3, Renumber STDOUT as STDERR, Renumber #3 as STDOUT”.

Note that each redirection directive appears after a closing parenthesis. Commands listed within parenthesis are executed in a new sub-shell, and therefore a parent-child relationship is created. This allows us to instruct the parent to manipulate the child's output streams.

Running the user's command

In order for all this magic to work, the user's command must be run in the deepest sub-shell. This is achieved by re-assembling the arguments passed to the rse() function into a command-line and then executing them. 

eval $(for phrase in "$@"; do echo -n "'$phrase' "; done)

“eval” is a BASH built-in which executes a (single) string passed to in the same way which command you enter at the prompt are.

“$@” is the original list of arguments passed to the function. The code iterates over it and stitched them togther so that eval has a single command strint to execute.

You may see scripts that use $* where I have used “$@”. These differ in how they treat white-space. Consider the command
echo "Hello" "World" "How are you?"

“$@” would tell you the arguments are:

  • Hello
  • World
  • How are you?

while $* would tell you they are:

  • Hello
  • World
  • How
  • are
  • you

Noting that there are quotation marks in the resulting lists.

Ctrl + S in PuTTY

If you ever accidently press ^S when using PuTTY, you'll know that this causes it to stop responding, which I need not mention is highly annoying, but I will because it's that annoying :-).

To get your session to respond again hit Ctrl + Q.

Quoted from http://www.plug.org/pipermail/plug/2004-April/005872.html

Every time I accidentally press Ctrl+s in Putty when connected to any
Linux box via ssh, my session stops responding. How do I restore it?


This is software flow control - left over from the days of serial
connections. Control+S sends XOFF and Control+Q sends XON and together they
make XON/XOFF, or “software,” flow control. Hardware flow control is known
as RTS/CTS and involves switching the signal on a couple of the pins
connected to the serial port.

To disable flow control add this to your shell's init script (e.g. .bashrc): 

# Disable software flow control (XON / XOFF or ^S / ^Q)
stty -ixon

Disabling CapsLock

Caps Lock is utterly useless. It is extremely seldom that I want to type a lot of stuff in upper-case, and when I do I'll just let my text editor upper-case it after I've written it.

More to the point, I use glorious vim, and the difference between pressing “ZZ” and “zz” is closing the editor without saving or scrolling my window so the cursor is in the middle.

Adapted from Handy Hack: Disable Caps Lock under GNOME.

Create/modify ~/.Xmodmap so it contains the following: 

remove lock = Caps_Lock

That's it. Next time X is started, CapsLock will be disabled. If you want the change to take affect immediately (i.e. without logging out), use the following command: 

xmodmap ~/.Xmodmap

SSH port-forwarding

http://www.stearns.org/doc/ssh-techniques-two.current.html

X11: "Client is not authorized to connect to Server"

From: https://wiki.tcl-lang.org/page/Client+is+not+authorized+to+connect+to+Server

Copy xauth token from somewhere with access: 

xauth extract - $DISPLAY

and import it to the place that's having trouble:

xauth merge -

For example, to allow a Docker container access to the X11 display: 

xauth extract - $DISPLAY | base64

and copy the result to the clipboard, and then in the container shell: 

base64 -d <<<'BASE64_GOES_HERE' xauth merge -
1)
Microsoft's cmd.exe being a notable exception, but then I suppose it's a stretch to call it “modern” anyway.
2)
Firefox 2.x could /almost/ do it, it printed a little “left-arrow” icon which looked about right, but then it would truncate the page it was displaying…