Nowadays, we do the majority of our work in fancy, powerful graphical user interface (GUI) tools, and rightfully so. A good piece of software should reduce the work you have to do to an absolute minimum, for example, a good integrated development environment (IDE) will help you format code, handle your imports/includes and visualise meaningful warnings, so you don’t need to waste your time handling all of that manually. Oftentimes, though, several advanced features are buried within countless menus, settings and so on.
We often forget about the terminal. After all, it is a relic of the past, is it not? I beg to differ! There’s often a faster, easier way to accomplish fairly complex tasks by using a just a couple of commands.
I’m sure commands like echo and cat ring a bell. In themselves, they don’t do much, but combine them with others, such as grep, sed, awk and wc and they become far more useful.
Let’s say we have a file in which we have multiple columns of data, and we’re interested only in the contents of the second column. We have a file called ‘test’. We can print the second column simply by typing cat test | awk ‘{print $2}’ to achieve the desired result. Notice the vertical line between the commands? It’s called a pipe. Essentially, it takes the output of the left side and sets it as the input of the right. The cat command will print the contents of the file passed as the argument. The right side, awk executes the print of second column. Awk will execute commands in the awk language upon the input of the command itself.
How about generating random data? In all *nix systems you can use either /dev/random or /dev/urandom as the input. Trying to execute cat /dev/urandom will print rubbish, ad infinitum. You might even hear a couple of beeps, which is just the hexadecimal character, 0x07, printed, also known as the “bell” character. So how can we use /dev/urandom? Let’s say we need to generate 32 bytes of random data. We also want to see what’s generated. How do we do this? Simply:
We’ve used dd to copy contents from file. The “if” parameter here stands for “input file”, which we’ve set to /dev/urandom. We can use any file to do that, it’s especially useful in checking if a binary file has correct content. The other parameters, “bs” and “count”, tell us how much data to copy. “Count” should be self-explanatory, namely the number of blocks to copy from the file. “Bs” might not be so obvious, it stands for block size. We can specify 1M, 1G or similar for 1 megabyte and 1 gigabyte, respectively. If using large block sizes, it might be a good idea to reduce them and increase block count instead, for better performance. You can also specify “of”, for the output file into which to write the bytes. XXD here is the hex viewer, which displays the last two lines in hex, then on the right in ASCII form, as you should be familiar with.
What if, instead, we want to copy the data somewhere else, say in base64 form?
Simple as that! But we can go even further… We can copy that straight to the clipboard:
This will place the string in the clipboard, ready to be pasted anywhere. In a similar way, we can process data from the clipboard. Let’s say we want to decode a base64 string and view the hex value of it:
We can also save the outputs to a file, or receive the input from one. For that we can use the less than (<) or more than (>) operators for input and output, respectively. For example, base64 -d < test > test2 will base64-decode contents of file ‘test’ and write the result into ‘test2’.
Now, when using some of the previous commands you’ve probably noticed there was some extra output. That’s because there are two different streams for output: stdout and stderr. The former stands for standard output, while the latter stands for standard error. The additional data from dd was printed to stderr, so we can get rid of that; simply add 2>/dev/null to the command!
2 stands for stderr (stdout is represented by 1, stdin by 0). If the redirect operator is preceded by one of those numbers, only that stream will be redirected. Here we redirected it to /dev/null, discarding the output.
What if we want to change our string from base64 to base64url encoding? This might sound like a pain, something that will require a script. Not with terminal! We’ve got sed! Assuming the string is in the clipboard, we can do the following:
The option “e” stands for expression. The first regex group stands for replace, replacing + for -, the second one replaces / for _ and the last one removes any = characters.
There are many more things that can quickly be done in the terminal, like resizing an image, converting it into a different format, and so on, all in a single command! I will not describe those here, but save them for another time. I hope any of this helps to speed up your work as much as it helped speed up mine!
