How To Remove the Byte Order Mark (BOM) from UTF-8 Encoded Text Files

The easiest way that I have seen so far for doing so is to use tail and simply read everything except the first three bytes (start reading at the 4th byte), as follows:

tail --bytes=+4 text_file.txt text_file-wo-bom.txt
Continue reading “How To Remove the Byte Order Mark (BOM) from UTF-8 Encoded Text Files”

Perl One-Liner for Replacing Multiple Lines of a Text file With Multiple Lines of Text

When executing ‘search-and-replace’ commands on ASCII under Linux, Unix (or *nix) operating systems, sed works or most cases and makes for reasonably straightforward reading of the script.

If  you want to replace multiple lines of text with multiple lines of text, following is a perl one-liner that does the trick and is much easier to wrangle than trying to do it in sed.

perl -i -pe "BEGIN{undef $/;} s:${EXISTING_LINES}:${REPLACEMENT_TEXT}:smg" file.txt
Continue reading “Perl One-Liner for Replacing Multiple Lines of a Text file With Multiple Lines of Text”

Adding the Contents of a Source File to the Beginning of a Target File

Following is *nix a command that you can use to add the contents of a source text file to the start of another text file (the source file).

echo -e '0r <source_file_name\nw' | ed -s <target_file_name
Continue reading “Adding the Contents of a Source File to the Beginning of a Target File”

Creating a Samba Share and Configuring an Access Control List for a Shared Directory Under Linux

Often administrators would like to configure a Samba share that enables users to have the same access to any files within the share.  Without some additional configuration, directories and files created by one user will not have the r/w permissions for other users that have access to that same share.

The end goal is to have a Samba share in which any new files are created with r/w permissions for a specific group to which all of the members of → Continue reading “Creating a Samba Share and Configuring an Access Control List for a Shared Directory Under Linux”

Use awk to Print from nth element to the End of the Line

If you want to extract from the nth token to the end of the line, following is how you can do that with awk:

Given a source file with the following:

line1 -- 01   0011 1
line2 -- 01   0011 2
line3 -- 01   0011 3
line4 -- 01   0011 4
line5 -- 01   0011 5
line6 -- 01   0011 6
line7 -- 01   0011 7
line8 -- 01   0011 8
line9 -- 01   0011 9
line10 -- 01   0011 
Continue reading “Use awk to Print from nth element to the End of the Line”

Firewall for Ubuntu 14.04 LTS

For whatever reason, Ubuntu 14.04 does not seem to come with a firewall.

There are however two packages which provide, both a firewall and a handy GUI front-end for it.  UncomplicatedFireWall is the main package (ufw) and the GUI is gufw.

To install:

apt-get install gufw

This will install the front-end and the dependent packages

To turn it on:

ufw enable

The default is to block all incoming traffic.

To update and add your own firewall rules and allow incoming → Continue reading “Firewall for Ubuntu 14.04 LTS”

Generate a Random String of a Specified Size with a Shell Script

The following is a one-liner for generating a random string of a fixed size in bash, where the possible characters to use in the string are any digit, letter, and a newline.

By adding the newline, you are fairly sure to prevent getting one long line of text.

< /dev/urandom tr -dc "[:digit:][:alpha:][\n]" | head -c1000 file.out
Continue reading “Generate a Random String of a Specified Size with a Shell Script”

One-Liner for Converting CRLF to LF in Text Files

If you have text files created under DOS/Windows and need to convert the CRLF (carriage return and line feed) characters to LF (line feed) character, here is a quick one-liner.

cat file.txt | perl -ne 's/\x0D\x0A/\x0A/g; print' file.txt.mod

You can also use dos2unix, however, especially under Cygwin I have seen dos2unix fail without giving any meaningful information about why it was unable to complete the task.  In that case, you can just do it by hand. → Continue reading “One-Liner for Converting CRLF to LF in Text Files”

Parsing Command Line Arguments with getopt in Bash

When writing utility scripts in Bash it is tempting to simply pass positional arguments, use $1, $2, etc. and be done with it.  However if you want to either share this utility with other members of your team and/or incorporate it into your system, it makes sense to implement your command line argument parsing in a more flexible and maintainable manner.

Using getopt you can very easily pass a variety of command line options and arguments.

Following is a link → Continue reading “Parsing Command Line Arguments with getopt in Bash”

Vim Search and Replacing with Backreferences

It is often helpful to write search and replace commands that save segments of the matched text to use in the replacement string.

Source text:

This is some text (we want to change) with a phone number (301) 555-1234.
We want to remove the parenthesis, but only from the phone number string.

In this example we have a text file that has a phone number in it and we want to remove the parenthesis that surround ONLY the area code → Continue reading “Vim Search and Replacing with Backreferences”