All Entries Tagged With: "command line"

Replacing Strings in Multiple Files

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal Starts Off:

Wow, our last several Episodes have been really long! So I thought I’d give everybody a break and just show you a cool little sed idiom that I use all the time:

# sed -i.bak 's/foo/bar/g' *

Here we’re telling sed to replace the all instances string “foo” with the string “bar” in all files in the current directory. The useful trick is the “-i.bak” option which causes sed to make an automatic backup copy of each file as .bak before doing the global search and replace.

By the way, you can even do this across an entire directory structure, with a little help from the find and xargs commands:

# find . -type f | xargs sed -i.bak 's/foo/bar/g'

Of course, you could use other search criteria than just “-type f” if you wanted to be more selective about which files you ran sed against.

Oh dear, I hope this isn’t one of those “easy for Unix, hard for Windows” things again. Ed gets so grumpy when I do that.

Ed jumps in:

You nailed it, Hal, with that characterization. Unfortunately, cmd.exe doesn’t include the ability to do find and replace of strings within lines of a file using a built-in command. We can search for strings using the find command, and even process regex with findstr. But, the replacement part just doesn’t exist there.

Thus, most reasonable people will either rely on a separately installed tool to do this, or use Powershell.

For a separately installed tool, my first approach would be use Cygwin, the free Linux-like environment for Windows, and then just run the sed command Hal uses above. Nice, easy, and sensical.

Alternatively, you could download and install a tool called replace.exe.

Or, there’s another one called Find And Replace Text, which, as you might guess, is called FART for short.

To do this in Powershell efficiently, I asked Tim Medin, our go-to guy for Powershell, to comment.

Tim (our Powershell Go-To Guy) says:

This morning when Ed asked me to do a “quick” write up for Powershell, I thought to myself, “This won’t be too bad…” I was wrong.

By default there are aliases for many of the command in Powershell, so I’ll show both the long and short version of the commands (yes, even the short command is long relative to sed).

The Long Way

PS C:\> Get-ChildItem -exclude *.bak | Where-Object {$_.Attributes -ne "Directory"} | ForEach-Object { Copy-Item $_ "$($_).bak"; (Get-Content $_) -replace "foo","bar" | Set-Content -path $_ }

The Short Way (using built in aliases)

PS C:\> gci -ex *.bak | ? {$_.Attributes -ne "Directory"} | % { cp $_ "$($_).bak"; (gc $_) -replace "foo","bar" | sc -path $_ }

This command is rather long, so let’s go through it piece by piece.

gci -ex *.bak | ? {$_.Attributes -ne "Directory"}

The first portion gets all files that don’t end in .bak. Without this exclusion, it will process file1.txt and the new file1.txt.bak. Processing file1.txt.bak results in file1.txt.bak.bak, but it doesn’t do this endlessly, just twice.

The Where-Object (with an alias of ?) ensures that we only work with files and not directories because Get-Content on a directory throws an error.

ForEach-Object { Copy-Item $_ "$($_).bak"; (Get-Content $_) -replace "foo","bar" | Set-Content -path $_ }
Once we get the files, not directories, we want, we then act on each file with the ForEach-Object (alias %). For those of you haven’t yet fallen asleep, I’ll further break down the inner portion of the ForEach-Object:

Copy-Item $_ "$($_).bak"

First, we copy the file to our backup .bak file. We have to use the $() in order to use our variable in a string so we can append .bak.

Finally, we get to the search and replace (and it’s about time, too!).

(Get-Content $_) -replace "foo","bar" | Set-Content -path $_

Get-Content (gc) gets the contents of the file. We wrap it in parentheses so we can act on its output in order to do our replace. The output is then piped to Set-Content (sc) and written back to our file.

We could make this work a little better if we used variables, but then we are more in script-land instead of shell-land which probably violates the almighty laws of this blog. The use of variables turn this more into a scripting exercise instead of shell (OK, we may already be there). For kicks, I’ll show you how we can use variables show you so you can add it to your big bloated belt of windows-fu.

$a = (gci | ? {$_.Attributes -ne "Directory"}); $a | % { cp $_ "$($_).bak"; (gc $_) -replace "foo","bar" | sc -path $_ }
The difference between our original command and this command is that the $a variable grabs a snapshot of the directory before we copy files, so we won’t operate on the new .bak files.

After all this work we have done the same thing as the mighty sed. Sadly even the power of Powershell is no match for efficiency of sed.

Ed closes it out:

Thanks for that, Tim. Nice stuff!

admin | May 22, 2009 | Comments 0

The Browser Count Torture Test

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal just can’t resist:

One of my customers was interested in some stats on which browsers were hitting their web site most often. My first thought was to use a Perl script to parse the user agent strings out of the Apache access_log files. But, hey, I write for Command Line Kung Fu, so how far can I get just using standard Unix tools? Besides, trying to do the same thing in the Windows shell will be a pain in the… humiliating defeat… interesting learning experience for Ed, and I’m committed to the growth of my friends.

First let me show you what I came up with, then I’ll explain it in detail:

# grep 'POST /login/form' ssl_access_log* | \ sed -r 's/.*(MSIE [0-9]\.[0-9]|Firefox\/[0-9]+|Safari|-).*/\1/' | \ sort | uniq -c | \ awk '{ t = t + $1; print} END { print t " TOTAL" }' | \ sort -nr | \ awk '/TOTAL/ { t = $1; next }; { $1 = sprintf("%.1f%%\t", $1*100/t); print}' 46.4% Firefox/3 27.0% MSIE 7.0 14.3% Safari 5.3% MSIE 6.0 3.0% Firefox/2 2.4% - 1.2% MSIE 8.0 0.3% Firefox/1

Here’s the line-by-line interpretation:

1. I didn’t want to count every single page access, but was instead more interested in counting browsers by “user session”. Since the site requires a user login for access, I used posting the secure login form as a proxy for recognizing individual sessions. Close enough for jazz.

2. Now, to pull out the browser name/version from the user agent string. The data here is annoyingly irregular, so it looks like a good task for sed. Notice that I’m using the “-r” option in GNU sed to use extended regular expression syntax: not only does this allow me to use “|” in the regexp, but it also means I don’t need to backwhack my parens to create sub-expressions.

The regular expression itself is interesting. I’m creating a sub-expression match on either “MSIE <vers>.<sub>”, “Firefox/<vers>”, or “Safari” (I don’t find tracking Firefox sub-versions or Safari version numbers that interesting, but as always “your mileage may vary”). Anything that doesn’t match one of these browser patterns ends up matching a hyphen (“-”) character, which are plentiful in Apache access_log entries.

I place “.*” before and after the sub-expression, which matches the rest of the line before and after the browser string. However, since that text is not included in the sub-expression, when I replace the matching line with the sub-expression then the rest of the text is dropped. That leaves us with an output stream of just the browser info, or “-” for lines that don’t match one of the major browsers we’re tracking.

3. Now that we’ve got a data stream with the browser info, it’s time to count it. “… | sort | uniq -c” is the common idiom for this, and we end up with output like:

290 - 34 Firefox/1 363 Firefox/2 5534 Firefox/3 632 MSIE 6.0 3207 MSIE 7.0 139 MSIE 8.0 1708 Safari

4. The next line is a common awk idiom for totalling a column of numbers. We print out each line as it’s processed, but also keep a running total in the variable “t”. After all the input has been processed, we use an “END” block to output the total. Now our output looks like:

290 - 34 Firefox/1 363 Firefox/2 5534 Firefox/3 632 MSIE 6.0 3207 MSIE 7.0 139 MSIE 8.0 1708 Safari 11907 TOTAL

5. The next “sort -nr” not only puts our data into numerically sorted order, but also has the side-effect of moving the “TOTAL” column up to the first line of output. We’re going to make use of this in the awk expression on the next line.

6. The last awk expression is a little psychotic, so let’s take it piece by piece. The first section, “/TOTAL/ { t = $1; next }”, matches our initial “TOTAL” line and puts the total number of entries into the variable “t”. The “next” causes awk to skip on to the next line without printing the current line (“TOTAL”).

The other portion of the awk code will handle all of the other lines in the output. What we’re doing here is replacing the raw count number in the first column with a percentage. The “sprintf(…)” format string looks a little weird, but it means a floating point value with one decimal place (“%.1f”), followed by a literal percent character (“%%”), followed by a tab (“\t”). The numeric value we plug in is the raw count from column 1 of the output, times 100, divided by the “TOTAL” value we extracted from the first line of output.

And there you have it. The agonized squealing you’re hearing is Ed wondering how he’s going to even get close to this in the Windows shell. I can’t wait to see what he comes up with.

Ed responds:

Wow! That’s some serious fu there, Hal. And, I mean both serious and fu.

Thanks for the interesting learning opportunity, kind sir. How delightful!

As you know, we’re kinda hampered with cmd.exe in that we get regex support from findstr, which cannot do extended regular expressions like sed -r. Therefore, we cannot do the funky “|” in the regex. Our resulting command will have to include more piece-parts for each browser.

And, as we discussed in Episode # 25: My Shell Does Math, we have access to simple integer math in cmd.exe via “set /a”, but floating point and division cause problems.

Still, we can get some useful output that tells us the number of each kind of browser and a handy total like this:

C:\> echo MSIE > browser.txt & echo Firefox >> browser.txt & echo Safari >> browser.txt & echo - >> browser.txt & (for /f %i in (browser.txt) do @echo %i & type ssl_access_log | find "POST /login/form" | find /c "%i" & echo.) & del browser.txt MSIE 873

Firefox 1103

Safari 342

- 2327

In this command, I’m first building a little file called browser.txt containing the different browsers strings that I’d like to count. I’ll then iterate over that file using a FOR /F loop. I’d much rather do this by iterating over a string containing “MSIE FIREFOX SAFARI -”, but unfortunately, FOR /F parses strings into a series of variables all in one FOR /F iteration, making it useful for parsing a string into different variables (like %i %j %k, etc.). But, FOR /F used with a string does not pull apart a string into pieces that vary at each iteration through the loop. Boo, FOR /F! So, we compensate by building a little file with one browser per line, and then we iterate over that.

For each browser in browser.txt, we display the browser name (echo %i), and scrape through our ssl_access_log using the plain old find command to look for lines with “POST /login/form”. I then take the output of that, pipe it through find with a /c option to count the number of occurrences of the %i iterator, which is the name of each browser. Note that the – will total all browsers, since their log entries have a dash in them. After my little looping escapade, I delete the temporary browser.txt file that I created at the beginning.

The output, while not as beautiful as Hal’s, still is useful — you see the number of POST login actions per browser, and the total. Why, you could even add a little “& calc.exe” at the end to pop up a calculator to do your percentages. :)

admin | May 20, 2009 | Comments 0

Show Account Security Settings

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed engages:

Yesterday, I was doing a presentation for a bunch of auditors, and a nice question came up from the attendees: “How can I quickly see the local account security settings on a Windows box from the command line?” When I gave the answer, I saw a lot of people’s eyes light up. Of course, whenever an auditor’s eyes start to flicker, we should all watch out. :)

Seriously, though… the vast majority of the people in the room quickly wrote down a note with my answer, so I figured it would make a good episode here.

On Windows, you can see overall security settings for all accounts on the box using the command:

C:\> net accounts Force user logoff how long after time expires?: Never Minimum password age (days): 0 Maximum password age (days): 42 Minimum password length: 0 Length of password history maintained: None Lockout threshold: Never Lockout duration (minutes): 30 Lockout observation window (minutes): 30 Computer role: WORKSTATION The command completed successfully.
A simple little command like that shows really useful information, for auditors, pen testers, general security personnel… great stuff. We’ve got password aging information, minimum password length, password history (so users can’t just reset their password to an older one they used to have), the threshold of bad logon attempts for account lockout, the time duration of account lockout, and the amount of time before a locked out account is re-activated.

The output I show above is the default settings for most versions of Windows, including Win2K, WinXP, and Vista (Yup… minimum password lenght of 0 by default!). On Win2k3, the only difference is that the “Computer role:” says SERVER.

Another nifty related command is:

C:\> net accounts /domain

You can run this on any system that is a member of the domain, and it’ll show you the domain-wide settings for accounts.

Pretty cool, and all in one place.

So, what’ve you got for us on Linux, big guy?

Hal reports in:

I’m sure you all are getting fairly tired of this, but I have to give my usual disclaimers:

1) Different Unix systems handle password security settings in different ways, so we’re just going to focus on Linux

2) The answer is different if you’re working with a network-based authentication database like LDAP or Kerberos, but for purposes of this article we’re just going to stick to local password files

With those disclaimers in mind, the basic answer is simple:

# chage -l hal Last password change : Jul 14, 2007 Password expires : never Password inactive : never Account expires : never Minimum number of days between password change : 0 Maximum number of days between password change : 99999 Number of days of warning before password expires : 7

The “chage” command can be used to get (and set) basic password security parameters for accounts on your Linux system (other Unix variants often use the “passwd” command for this). This is actual output from one of my test systems and shows you the standard Linux defaults for these parameters, which are obviously not terribly secure. You may change the defaults by modifying the /etc/login.defs file, but be aware that the defaults you set in login.defs will only apply to new accounts that you create with the built-in “useradd” program that comes with Linux. If you use some other scheme for creating accounts, then you’ll have to use the “chage” command to manually set these values after you create each account.

If you compare the “chage” output with the output of Ed’s “net accounts” command, you’ll notice that “chage” doesn’t have anything to say about password history settings or “lockout on failure” parameters. That’s because this level of password security is a property of the lower-level PAM configuration on most Unix systems. On Linux, the pam_cracklib and pam_unix modules take care of password history and strong password enforcement, while pam_tally is responsible for “lockout on failure”. Unfortunately there’s no way to audit the settings for these modules other than to look at the actual PAM configuration files, usually found in /etc/pam.d.

admin | May 18, 2009 | Comments 0

File Linking

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Paul pops off:

Creating links between files is a handy feature in UNIX/Linux systems. There are many instances where you need to have a copy of the file (or dirctory) in a particular location, but only want to maintain one original. For example, I was running a program to check the security of my Apache configuration file. It expected the file to exist in “/usr/local/apache2/conf/httpd.conf”, but the original file was located at “/etc/httpd/conf/httpd.conf”. To solve this problem I created a “soft” link as follows:

$ ln -s /etc/httpd/conf/httpd.conf /usr/local/apache2/conf/httpd.conf

The above “ln” command takes the “-s” flag to indicate a soft link, which creates a pointer to the original file. Next you specify the original file, followed by the file that will point to the original. Many will forget which one comes first (the original or the pointer), so don’t forget that the original file always comes first :) Oh, and you can view the links by using the ls -l command:

$ ls -l /usr/local/apache2/conf/httpd.conf lrwxrwxrwx 1 root root 26 Apr 21 13:57 /usr/local/apache2/conf/httpd.conf -> /etc/httpd/conf/httpd.conf

Hal chimes in:

Let me show you one more useful trick with the “ln” command. You can actually create symlinks to an entire directory of files with a single “ln” command:

# cd /usr/local/bin # ln -s ../depot/clamav/current/bin/* .

First we “cd” to /usr/local/bin. The “ln” command creates a link to every object under /usr/local/depot/clamav/current/bin. The names of the links in /usr/local/bin will have the same name as the files under …/clamav/current/bin.

This is how I manage software that I’ve built from source on my systems. In fact, …/clamav/current is itself a symlink to a directory like …/clamav/0.95.1. Whenever I build the latest version of ClamAV, I install it in its own …/clamav/ directory and just change the …/clamav/current symlink to point to the latest and greatest version. Since all the symlinks under /usr/local/{bin,sbin,etc,lib,include} are expressed using the …/clamav/current link, every other link in the hierarchy automatically starts pointing at the right version as soon as I change the …/clamav/current link. And it’s easy to revert too, just in case the new version isn’t working for some reason. Slick.

Ed responds:

Sadly, Microsoft never got around to implementing a pure-play shortcut-creating feature inside of cmd.exe. Because of that, several folks have released third-party tools that do so. Some nice ones include the NT resource kit tool simply called shortcut.exe, Pixelab’s xxcopy, and NirSoft’s NirCmd.

But, downloading a third-party tool isn’t our way at this here blog. So, we must explore other options.

While cmd.exe itself doesn’t have a feature for creating shortcuts, wscript, which is built in, does. There are many examples out on the Internet for creating shortcuts with wscript, but I’ve boiled them down to their bare minimum:

set WshShell = WScript.CreateObject("WScript.Shell" ) set oShellLink = WshShell.CreateShortcut( Wscript.Arguments.Named("shortcut") & ".lnk" ) oShellLink.TargetPath = Wscript.Arguments.Named("target") oShellLink.Save

The above script takes two arguments: the name of the target you want to create a shortcut to (/target:) and the shortcut name itself (/shortcut:). Note that the target could be a file or a directory. To create a shortcut using this script, we could dump all of that stuff above into a file called shortcutter.vbs, and then run it with the wscript interpreter.

“Ah… but that would be a scripting solution and not a command line,” you might say. “You need to create a single command line that addresses the challenge.”

Thanks for the delightful reminder. What, are you on Hal’s payroll? Don’t you have anything better to do with your time than taunt me? ;)

OK… I’ll take your input and respond with this for a command line:

C:\> echo set WshShell = WScript.CreateObject("WScript.Shell" ) > shortcutter.vbs & echo set oShellLink = WshShell.CreateShortcut( Wscript.Arguments.Named("shortcut") ^& ".lnk" ) >> shortcutter.vbs & echo oShellLink.TargetPath = Wscript.Arguments.Named("target") >> shortcutter.vbs & echo oShellLink.Save >> shortcutter.vbs & wscript shortcutter.vbs /target:[source] /shortcut:[shortcut]
It pretty much types itself, doesn’t it? Easy!

Uh…. or not.

I’m simply creating the vbs script, which I’m naming shortcutter.vbs, and then invoking it to create the shortcut. I don’t delete it at the end, because I want to keep it around for future uses. These things come in handy, you know.

admin | May 15, 2009 | Comments 0

Combining multiple PDF’s into one

I found this one on SHELL FU – I haven’t tried it out yet but SHELL FU don’t post commands that don’t work. One comment I saw on Twitter was: Good + Evil = New. Anyway, here it is.

To combine multiple pdfs into one, for printing purposes or distribution:

$ gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=combine.pdf -dBATCH 1.pdf 2.pdf 3.pdf

admin | May 08, 2009 | Comments 0

Recognizing Sub-Directories

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal takes requests:

Loyal reader Lloyd Alvarez writes in with a little problem. He’s writing Javascript that needs to get a directory listing and be able to easily discriminate sub-directories from other objects in the directory. The trick is that he needs code for both Unix/Linux and Windows. Well where else would you come for that kind of service than Command Line Kung Fu blog?

Turns out that Lloyd had doped out a Unix solution on his own:

$ ls -l | gawk '{printf("%s,",$1);for(i=9;i<=NF;i++) printf("%s ",$i);printf(":")}' total,:drwxr-xr-x,dir1 :drwxr-xr-x,dir2 :-rw-r--r--,file1 :-rw-r--r--,file2 ...

Yow! That’s some pretty ugly awk and even uglier output, but Lloyd was able to parse the resulting stream in his Javascript and easily pick out the directories.

But let me make that even easier for you, Lloyd old buddy:

# ls -F dir1/ dir2/ file1 file2 otherdir1/ otherdir2/ otherfile1 otherfile2

Yep, the “-F” option causes ls to append a special character to each object in the directory to let you know what that object is. As you can see, directories have a “/” appended– should be easy to pick that out of the output! Other suffix characters you might see include “@” for symbolic links, “*” for executables, and so on. Regular files get no special suffix (see the above output).

Maybe Lloyd would prefer to just get a list of the sub-directories without any other directory entries:

$ find * -maxdepth 0 -type d dir1 dir2 otherdir1 otherdir2

Or if you’re not into the find command:

$ for i in *; do [ -d $i ] && echo $i; done dir1 dir2 otherdir1 otherdir2

Too many choices, but then that’s Unix for you! I’ll step out of the way now and let Ed shock Lloyd with the Windows madness.

Ed jumps in:

Good stuff, Lloyd! Thanks for writing in.

In Windows, the easiest way to do this is to use the dir command, with the /d option. That option is supposed to simply list things in columns, but it adds a nice little touch — directory names now have square brackets [ ] on either side of them. Check it out:

C:\> dir /d [.] [dir1] file1 [otherdir1] otherfile1 [..] [dir2] file2 [otherdir2] otherfile2

So, just find and parse out those brackets, Lloyd, and you should be good to go. Oh, and remove the . and .. if you don’t want then. Unfortunately, we cannot eliminate them with a /b (for bare), because that removes the brackets.

For Hal’s additional fu for looking for just directories, we can rely on the fact that Windows considers “directoriness” (is that a word?) as an attribute. So, we can list only the directories using:

C:\> dir /ad /b dir1 dir2 otherdir1 otherdir2

Or, if you want only files (i.e., NOT directories):

C:\> dir /a-d /b file1 file2 otherfile1 otherfile2

You could do this kinda stuff with FOR /D loops as well, which give you even more flexibility. For example, if you just want directory names with a slash after them, to give you similar output to Hal’s “ls -F”, you could run:

C:\> FOR /D %i in (*) do @echo %i/ dir1/ dir2/ otherdir1/ otherdir2/

Or, if you really like the colons that your current scripts parse, you could do:

C:\> FOR /D %i in (*) do @echo :%i

By altering that echo statement, you can roll the output however you’d like.

Fun, fun, fun!

admin | May 08, 2009 | Comments 0

Wiping Securely

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed gets his groove on:

The Encrypted File System on Windows stinks. It’s something of an embarrassment for me that it is called EFS, because those are my initials too. My biggest beef with EFS is that it leaves cleartext copies of files around in unallocated space if you simply drag and drop a file into an EFS-protected directory. No, seriously… isn’t that awful? Doh!

But, included with all the stinkatude of EFS is one gem: the cipher command, when used with the /w: option. I don’t use EFS to encrypt, but often rely on cipher to wipe. I used it just last week to help a buddy who was trying to recondition his computer so he could give it to his son for school. He had some work files he needed to clear out, and was planning on simply deleting them through the recycle bin. Ouch! That won’t work very well, as those files would still be recoverable. I broke into a mini-lesson about file systems and secure deletion options.

For wiping files, there are many good options available for download, such as sdelete from Microsoft SysInternals or DBAN to completely blow away a file system. But, what if you are stranded on a desert island and need to securely delete something using only built-in tools? Windows 2000 and better (not XP Home… Microsoft purposely cripples that version) have the cipher command, which includes the /w: option to wipe the unallocated space on a volume that you specify like this:

C:\> cipher /w:c:\folder

This command will cause Windows to overwrite all the unallocated space on the volume with c:\folder three times. First, it overwrites with zeros, then with ones, and then random numbers. Unfortunately, there’s no option to specify overwriting any number of times other than three. Well, unless you want to… uh… do the obvious:

C:\> for /L %i in (1,1,9) do @cipher /w:c:\folder
This command will overwrite your unallocated space 27 times. Oh, and it’ll take a long time on any reasonably sized partition with a lot of open space.

Whenever using cipher to wipe files, there are some hugely important notes to keep in mind.

First off, you have to include the colon between the /w and the folder name. Do people at Microsoft stay up late at night thinking of ways to make their syntax more horrible and inconsistent than ever, or does it just happen?

Second, and this one is huge…. note that cipher won’t actually delete any current files in c:\folder or that folder itself! A lot of people think cipher will securely delete the folder (c:\folder) you specify, and that’s not right. It securely deletes all unallocated (already deleted) files and folders on the entire partition that c:\folder inhabits. That’s a much more thorough (and likely time consuming) process, but realize that it will leave behind c:\folder and its contents. If you want to get rid of them, delete them, and then do a cipher /w:c:\ to wipe the whole partition.

Now, there are major debates as to whether overwriting three times is enough for a good wipe. I’ve read the debates, and am comfortable that, for modern media, three times overwriting is good enough for most uses. If I need stronger destruction of data, it’s best to simply take a hammer to the hard drive.

Hal wipes out:

Most Linux distros these days ship with the “shred” command, which overwrites files and then optionally deletes them:

# shred -n 3 -z -u myfile

Here “-n 3″ specifies three overwrite passes, “-z” means to do a final overwrite with zeroes (nulls) to make it less obvious you’ve been shredding your files, and “-u” means to remove the file once the overwrites are performed.

But you should be aware that using “shred” on an individual file like we do in the above example may still leave traces of the file on disk. That’s because most Linux systems these days use the ext3 file system, which has a file system transaction journal. Even after using “shred” on the file, the contents of the file may be recoverable from the journal using a tool like “ext3grep”.

So the most secure option is to “shred” the entire disk (which overwrites the journal as well):

# shred -n 3 -z /dev/sdb

In these cases, you don’t want to remove the disk device file itself once you’re done with the overwriting so we leave off the “-u” option. This is also why “-u” is a separate option that must be explicitly set– overwriting entire disks is the more common use case.

What if you’re on a non-Linux system and don’t have “shred” installed? Well, you could certainly download the “shred” source code (or “srm”, another popular file deletion tool). But don’t forget that you also have “dd”, which I often use to wipe disks:

# dd if=/dev/urandom of=/dev/sdb bs=4096 count=1000000000 # dd if=/dev/null of=/dev/sdb bs=4096 count=1000000000
The first command overwrites /dev/sdb with up to 4TB of random data (use a bigger “count=” argument for larger disks). The second overwrites your disk with nulls (zeroes). Run the commands multiple times depending on the number of overwrites you’re most comfortable with.

admin | May 06, 2009 | Comments 0

Remote Command Execution

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed starts out:

One of the most frequent questions I get regarding the Windows command line involves how to run commands on a remote Windows machine and get access to the standard output of the command. Sure, Microsoft SysInternals psexec rocks, but it’s not built in. On Linux and Unix, ssh offers some great possibilities here, but neither ssh nor sshd are built-in to Windows (and what’s with that? I mean… we need that. Call Microsoft right now and demand that they build in an ssh and sshd into Windows. Installing a third-party version is certainly doable, but we need it built in… starting about 5 years ago, thank you very much.)

Anyway, while there are many options for running a command on a remote Windows machine using built in tools (such as using at, schtasks, or sc), one of my faves is good old WMIC:

C:\> wmic /node:[targetIPaddr] /user:[admin] process call create "cmd.exe /c [command]"

That’ll run [command] on the target, after prompting you for the given admin’s password.

You won’t see the standard output, though.

To get that, change it to:

C:\> wmic /node:[targetIPaddr] /user:[admin] process call create "cmd.exe /c [command] >> \\[YourIPaddr]\[YourShare]\results.txt"

Make sure you have [YourShare] open on your box so the target machine and [admin] user can write to your share. The results.txt file will have your standard output of the command once it is finished.

Oh, and to execute a command en mass on a bunch of targets, you could use /node:@[filename.txt], in which the filename has one line per machine name or IP address on which you want to run the given command.

Not nearly as elegant as what I’m sure my sparring partners will come up with for Linux, but it is workable.

Hal Replies:

Thanks for throwing us a bone here, Ed. With SSH built into every modern Unix-like operating system, remote commands are straightforward:

$ ssh remotehost df -h

Sometimes, however, you need to SSH as a different user– maybe you’re root on the local machine, but the remote system doesn’t allow you to SSH directly as root, so you have to use your normal user account. There’s always the “-l” option:

$ ssh -l pomeranz remotehost df -h

But what if you want to scp files as an alternate user? The scp command doesn’t have a command line option like “-l” to specify an alternate user.

One little-known trick is that both ssh and scp support the old “user@host” syntax that’s been around since the rlogin days. So these commands are equivalent:

$ ssh -l pomeranz remotehost df -h $ ssh pomeranz@remotehost df -h

Personally, I never use “-l”– I find “user@host” more natural to type and it works consistently across a large number of SSH-based utilities, including rsync.

Unlike wmic, SSH does not have built-in support for running the same command on several targets. The “Unix design religion” is that you’re supposed to do this with other shell primatives:

$ for h in $(< targets); do echo ===== $h; ssh $h df -h; done

By the way, note the “$(< targets)” syntax in the above loop, which is just a convenient alternate form of “`cat targets`”.

Unfortunately, the above loop is kind of slow if you have a lot of targets, because the commands are run in serial fashion. You could add some shell fu to background each ssh command so that they run in parallel:

$ for h in $(< targets); do (echo ===== $h; ssh $h df -h) & done

Unfortunately, this causes the output to be all garbled because different commands return at different speeds.

Frankly, you’re better off using any of the many available Open Source utilities for parallelizing SSH commands. Some examples include sshmux, clusterssh, and fanout (which was written by our friend and fellow SANS Instructor, Bill Stearns). Please bear in mind, however, that while remote SSH commands allow you to easily shoot yourself in the foot, these parallelized SSH tools allow you to simultaneously shoot yourself in both feet, both hands, the head, and every major internal organ all at the same time. Take care when doing these sorts of things as root.

admin | May 04, 2009 | Comments 0

Twiddling with the Firewall

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed kicks it off:

One of the real gems of the Windows command line is netsh. I use it all the time. In Episode #2, about a thousand years ago (gosh… can it be only 2 months?), we talked about using netsh (yeah, and iptables) to display firewall config information. But, you know, there are some commands I run even more often that alter the firewall.

In particular, if I’m in my lab doing an analysis that requires me to shut off the built-in Windows firewall, or when I’m conducting a pen test and want to hack naked without a firewall, I drop the firewall with:

C:\> netsh firewall set opmode disable

That’s soooo much easier than digging through the blasted GUI to find where Microsoft buried the firewall configuration in the given version of Windows.

Simple command, but I use it all the time.

To turn it back on, you’d run:

C:\> netsh firewall set opmode enable

If you want to poke holes through the firewall based on port, you could run:

C:\> netsh firewall add portopening protocol = [TCP|UDP] port = [portnum] name = [NameOfRule] mode = enable scope = custom addresses = [allowedIPaddr]
This syntax would configure the firewall to allow traffic in for the port you specify only if it comes from allowedIPaddr, to make your rule a little safer.

And, to remove the rule, just change “add” to “del”, and only type in the command up to the port number.

Finally, as we mentioned in Episode #2, to show the overall config of the firewall, you can run:

C:\> netsh firewall show config

Hal chimes in:

Firewall software is another one of those areas where there are a number of competing options on different Unix/Linux platforms. For example you’ve got ipfilter on Solaris and {Free,Net}BSD, pf on OpenBSD, and IP Tables on Linux. I’m going to stick with IP Tables for purposes of this discussion, since it’s probably what most of you all deal with the most.

Unfortunately, IP Tables is rather cranky to work with from the command line if you stick with the basic “iptables” command. The developers tried to smash all of the possible functionality configuration options into a single command, and IP Tables is capable of some complicated stuff. The result, however, is that there’s a pretty steep learning curve for doing even basic operations. This is why simplified firewall configuration GUIs are provided by the major Linux distributions.

But let’s try and cover some of the same command-line territory that Ed does in his Windows examples. First, as far as starting and stopping the firewall goes, your best bet is to just run “/etc/init.d/iptables [start|stop]” as root. The init script hides a lot of complexity around loading and unloading kernel modules, firewall rules, and default packet handling policies. For example, here are the manual command equivalents for “/etc/init.d/iptables stop”:

# iptables -P INPUT ACCEPT # iptables -P OUTPUT ACCEPT # iptables -P FORWARD ACCEPT # iptables -F

Yep, four commands– and that’s without even showing you the commands that some init scripts use to unload the IP Tables kernel modules. The first three commands above set the default permit (“ACCEPT”) policy (“-P”) for inbound (“INPUT”) and outbound (“OUTPUT”) packets, as well as for packets being passed through the system (“FORWARD”). It’s possible that your particular firewall configuration doesn’t change the default policies to block packets (“DROP”), but it’s best to be sure. The final “iptables -F” command flushes all filtering rules, which means all packets will now simply be handled by the default “ACCEPT” policies.

The simplest possible example of adding a rule to allowing traffic into your system on a particular port would be something like:

# iptables -A INPUT -p tcp --dport 80 -j ACCEPT

This allows (“-j ACCEPT”) inbound traffic (“-A INPUT”) on 80/tcp (“-p tcp –dport 80″). Deleting the rule is as simple as running the same command but with a “-D INPUT” option (delete from the “INPUT” chain) instead of “-A INPUT” (add to the “INPUT” chain).

However, depending on how your particular Linux vendor sets up their firewall, adding rules directly to the “INPUT” chain may not be the right thing to do. Many vendors set up their own rule chains that pre-empt the default chains. Also, you may have to add rules to the “OUTPUT” chain (or vendor-specific equivalent) to allow the return packets to escape your host, unless you have a “default permit” configuration in the outgoing direction. For these reasons, it’s best to inspect your current rule sets (“iptables -L -v”) before making changes.

I should mention that a simplified command-line interface is now becoming available in some Linux distributions, notably Ubuntu. If all you’re interested in is a host-based firewall, the “ufw” command makes configuration and maintenance much easier. Here are some sample “ufw” commands:

# ufw enable # enable filtering # ufw disable # turn off firewall # ufw status # show current rules # ufw allow 80/tcp # allow 80/tcp to any IP on this host # ufw delete allow 80/tcp # delete above rule

You can also do more complex rules like:

# ufw allow proto tcp from 1.2.3.4 to any port 25

If you’re more used to Cisco extended ACL syntax, or use ipfilter on other Unix systems, the “ufw” command-line syntax will be pretty natural for you.

admin | May 01, 2009 | Comments 0

Environment-A-List

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed starts out:

At a cmd.exe prompt, you can list all environment variables using the “set” command by itself:

C:\> set

Pretty simple. By default, there are a typically a lot of variables set, upwards of two or three dozen.

Another property of set involves the listing of groups of environment variables that start with a substring. For example, to see all of your environment variables that start with the letters pr, you could type:

C:\> set pr

It’s kinda weird, but environment variables in Windows cmd.exe are case insensitive.

We can refer to any one of these variables in a command by using the %[var]%. Thus, we can see the value of the username variable by running:

C:\> echo %username%

Or, we can see the value using the set command with:

C:\> set username

Prior to Vista, Windows didn’t include a built-in whoami command. The closest thing we have is this “set username”. However, be aware that, depending on how your shell is instantiated, you may or may not have this variable set. For example, this variable is typically not set when Metasploit launches a shell on a compromised box. Windows Vista does include a whoami command, but that’s a story for another article. We must keep focus when performing command line kung fu.

Other environment variables besides username that are worthwhile include our path variable:

C:\> set path Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH

The output here will show you both your path, plus the pathext variable (remember, set [string] shows all variables that start with string). The pathext variable is particularly interesting, because it tells us the priority of execution on the machine for various file name suffixes. By default, if I type in “command”, but don’t specify a suffix, command.com will be run first. If that doesn’t exist, the system tries command.exe. Then, it moves onto command.bat and so forth. That behavior can be used for some evil stuff by creating a backdoor called cmd.com or ipconfig.com and inserting it somewhere in the user’s path.

Windows, by the way, includes an implicit “.” at the very beginning of every user’s path. That’s a huge bummer from a security perspective, because users can be tricked into running a backdoor out of their current directory when they try to invoke some other command. Also, note that Windows doesn’t seem to be too proud that “.” is in the path, because it doesn’t show it… but, it behaves as though it’s there allright.

Other useful environment variables include %systemroot%, which tells you which partition and folder your primary Windows software is located in:

C:\> echo %systemroot%
Often when I’m handling an incident, I see attackers who assume the system root is on C:\. But, sometimes, it’s not, and attackers get wrapped around the axel trying to figure out why their tweaks to the OS aren’t having an impact. Whenever I’m doing a penetration test and I compromise a Windows box, I quickly display my systermroot and username variables, because that helps to orient me to the target machine.

Also, the prompt environment variable is called, simply enough, prompt. It’s default value is $P$G, which indicates your current working directory ($P) followed by a greater than sign ($G). You could really freak yourself out by running:

C:\> set prompt=$$ $ ls

You now have a Linux wanna-be for your shell. I tell ya, when I see that $ prompt, and I just want to type “ls”. Alas, no dice. :)

Hal responds:

It’s interesting to me how even something as simple as environmental variables points out the differences between the Windows and Unix command-line philosophies. For example, instead of packing all kinds of functionality into a single command like Windows does with “set”, Unix handles this by providing a toolkit of different programs that all interact with each other. For example, you use the “env” command to dump all of your current environment variable settings. If you want a specific subset, you pipe the output of “env” into “grep”:

$ env | grep PATH MANPATH=/usr/man:/usr/share/man:/usr/openwin/man:/usr/dt/man:/usr/local/man CDPATH=.:/home/hal LD_LIBRARY_PATH=/usr/lib:/usr/openwin/lib:/usr/dt/lib:/usr/local/lib PATH=/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:...

This seems cumbersome– you have to use two commands and a pipe instead of using just one “set” command with different parameters. But the power of this idiom is that you use the same knowledge of “grep” to search any kind of textual data in Unix, whereas under Windows you have to learn the particular idiosyncracies of a whole host of separate command line interfaces that each only deal with very specific types of data.

Anyway, returning to Ed’s examples, you can display the value of a particular variable using “echo”, and we typically set new values using “export” so that the variable setting will be passed along to any subshells we spawn:

$ echo $PATH PATH=/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:... $ export PATH=.:$PATH $ echo $PATH PATH=.:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:... $ export PATH=${PATH/\.:/} $ echo $PATH PATH=/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/sbin:/usr/sbin:...

Here I’m using the current value of $PATH in the “export” expression so that I can prepend “.” (the current working directory) at the front of my search path. Obviously, as Ed points out above, this is a terrible idea from a security perspective, so I then use the same variable substitution operator we used back in Episode 26 to remove the leading dot. Again, I’m leveraging prior knowledge of basic shell building blocks to solve a new problem.

As Ed points out, Unix does have the “whoami” command, plus a number of other mechanisms for figuring out what user you are that provide different types of information:

$ whoami hal $ who am i hal pts/2 2009-04-26 14:44 (:0.0) $ id uid=1000(hal) gid=1000(hal) groups=4(adm),20(dialout),24(cdrom),...

Also, the your login program will generally set the $LOGNAME and $HOME environment variables, and your shell may also set the $USER and/or $USERNAME variables as well.

You can change your prompt by setting the $PS1 environment variable. For example, the default on my Ubunty machine is:

$ export PS1='\u@\h:\w\$ ' hal@elk:~$
There are a whole host of different escape sequences you can use here: “\u” is your user ID, “\h” is the unqualified host name, and “\w” is the current working directory. So if we wanted to emulate the standard Windows command prompt in our Unix shell we could use:

hal@elk:~$ export PS1='\w> ' ~> cd /tmp /tmp>
Of course, it doesn’t look exactly like the Windows prompt, because when you’re in your home directory, the directory is listed as “~”, not $HOME. However, you can use the magic $PWD variable in your prompt instead of “\w” to get a more Windows-like experience:

/tmp> export PS1='$PWD> ' /tmp> cd /home/hal>

The emulation isn’t perfect because Unix uses forward slashes in directory paths, not backslashes. I present the following for the truly psychotic:

/home/hal> export PS1='C:${PWD//\//\\\\}> ' C:\home\hal> cd /tmp C:\tmp>

You think having a “$” prompt will confuse the Windows folks? Try putting the above recipe into your /etc/profile file next April Fool’s Day and watch your Unix users freak out.

By the way, you’ll notice that the variable is $PS1. There’s also $PS2:

C:\home\hal> export PS1='$ ' $ export PS2='more> ' $ for i in * more> do more> ...

Believe it or not, there are even $PS3 (the prompt used by the “select” command) and $PS4 (usually set to “+”, this is the value printed before each command in an execution trace as with “set -x”).

Paul Throws His Hat In:

One of the environment variables that I’ve encountered in the past that has really helped me is “LD_LIBRARY_PATH”. The variable stores the path that will be used by programs to find software libraries. For example, if you’ve got a bunch of custom libraries in /opt/local/lib you could run the following command:

$ export LD_LIBRARY_PATH="/opt/local/lib:/usr/local/lib:/usr/lib:/lib"

This comes in handy when you are using OS X and something such as MacPorts is installed in /opt and has the libraries you are looking for. Yes, these are in fact the libraries you are looking for…

admin | Apr 27, 2009 | Comments 0

Download a specific file from the web

From: http://www.commandlinefu.com

To download a specific file from the web:

curl -f -O http://pcbsd.fastbull.org/7.0.2/i386/PCBSD7.0.2-x86-DVD.iso

or, if you have wget configured:

wget -c http://pcbsd.fastbull.org/7.0.2/i386/PCBSD7.0.2-x86-DVD.iso

-C option will re download from an existing & interrupted download.

admin | Apr 26, 2009 | Comments 0

Create a Directory from the Current Date

Create an alias to create a directory from the current date. This isn’t news, but is quite useful.

alias mkdd='mkdir $(date +%Y%m%d)'

Obviously, if you just want to create the director you could just use the command:

mkdir $(date +%Y%m%d

I’m not teaching granny to suck eggs here – I just want to make sure it’s clear.

admin | Apr 25, 2009 | Comments 0

You’ll Be History

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed starts with:

In Episode #14, I touched briefly on how a cmd.exe user could hit the F7 key to view shell history, and we received a lot of positive feedback on that from folks. But, there’s some more aspects of cmd.exe history that I use all the time that I think are worth bringing up here.

In particular, if you want to get access to your shell history at the command line itself so you can dump it to a file or manipulate it (rather than just displaying it on the screen so you can scroll through it with F7), you could run the following command:

C:\> doskey /history

Dude! The 80’s just called, and they want their doskey command back. :)

Seriously, though, it remains a very useful command. Running doskey this way will show all of the commands typed into the current cmd.exe session. Unfortunately, you won’t see history for other cmd.exe’s on the screen or those invoked in the past. Still, when handling an incident, running this command with the output redirected to a file can be very useful evidence. If the bad guy was kind enough to leave a cmd.exe on the console screen of the machine he controlled the GUI on, you can quickly dump the history to an evidence file. Alternatively, if you, the incident handler or forensics analyst, are typing commands at a cmd.exe and want a record of what you’ve typed, invoking:

C:\> doskey /history > \\evidence_server\share\mycommands.txt

…can be very helpful to shoot your results to an evidence server somewhere.

Oh, and note that cmd.exe “telescopes” commands down, so that if you run the same command multiple times in a row, the history will only show the command once. For example:

C:\> dir > nul

C:\> ipconfig > nul

C:\> hostname > nul

C:\> doskey /history dir ipconfig hostname doskey /history

As we can see, the ipconfig command only appears in history once, even though we ran it three times in a row, due to this telescoping down effect.

By default, the command history stores 50 commands, but is configurable on a user-by-user basis. To change it at the command line for the current user, you could run:

C:\> reg add hkcu\console\^%systemroot^%_system32_cmd.exe /v HistoryBufferSize /t reg_dword /d 0x400

That last number there is a hexadecimal value for the number of commands you want it to remember. 0×400 will make it remember the last 1024 commands. This new history depth will only take effect when you launch new cmd.exe’s going forward. But, you don’t have to logoff or reboot for it to take effect. Just launch another cmd.exe, and you should have your new history depth (which can be viewed in the GUI by right clicking the title bar of a cmd.exe window, selecting “Properties”, and going to the Options tab to view Command History Buffer Size).

Oh, and another thing about cmd.exe history… I often see bashketeers get a little freaked out by how cmd.exe handles shell history when you scroll through it with the up-arrow and down-arrow keys. As we all know, you can hit the up-arrow key in cmd.exe to scroll back to a previous command in modern Windows machines. When you find the command you want, you can hit Enter to re-run that command. No problem there. The issue that puzzles some people is what happens after you’ve run that command from your history, and you then hit the up arrow key again. Let me illustrate it with this screen shot.

Here, I’ve just hit 1 followed by Enter, to seed my history with some bogus commands so we can track things easily. Because there is no program called 1.exe, my shell tells me so. I do this for 2, 3, 4, and 5 as well.

Then, I hit the up arrow key to scroll back to my 3 command. I hit Enter on that. If I hit the up arrow again, of course I get the 3 again.

Now for the interesting part. What happens when I hit the up arrow again? If you were a bash guy, you might think that the system would display the number 5, because before you did 3, you did 5. And, you’d be wrong in a cmd.exe shell, which would display the number 2. What? Why 2?

Well, cmd.exe maintains a shell history pointer, and when you scroll back into your history and then run a command from it, this pointer jumps back to your earlier command. All up or down arrow movements from that point are relative to that older command, not your most recent commands.

This used to flummox me, with my ancient Unix background. But, you know what? I’ll share a dirty little secret with you if you don’t tell anyone. I actually have grown to like (and… gulp… prefer) this behavior. You see, there is a certain locality of command execution from history, and, if I execute a command that was 20 commands ago, there is a reasonably good probability that I’ll want to run another command from 19 commands ago after it.

So, cmd.exe history scrolling isn’t broken… it’s just different. And, it’s different for a reason. You might not agree with the reason, but at least there is one. So many things in Windows are different for no reason whatsoever, you know… but don’t get me started about tracert. ;)

Hal Responds:

I was really happy when Ed brought us back around to command-line history again, because there was a bunch of stuff that I wanted to put into Episode #14, but decided not to in the interests of space. But there’s no stopping me now…

While “doskey /history” is an obvious and intuitive way to access your command history, those wacky Unix folks have decided to force you to use the obscure “history” command instead (that was sarcasm for those of you who missed it because you were trying to remember the silly cmd.exe syntax):

# history [...] 1001 /etc/init.d/httpd restart 1002 vi /etc/httpd/conf/httpd.conf 1003 vi /etc/httpd/conf.d/ssl.conf 1004 /etc/init.d/httpd restart 1005 ps -ef | grep http 1006 ps -ef | grep http 1007 history

Oh, and notice that the bash history saves all of the commands you’ve typed, even the repeats. I think that’s more useful– especially from a forensic perspective– than removing repeated commands, but I’m sure lord high doskey knows better than me.

See the numbers next to the commands in the history output above? You can use “!” to execute the previous command with number . For example, “!1001″ would execute “/etc/init.d.http restart” again (as would “!1004″). I often use this when I’m debugging a problem with a server process like Apache– try to start the server and fail, enter a bunch of commands to fix the problem, “!” to try and start the server, lather, rinse, repeat. In other words, “!” is useful when you are constantly re-running the same command with an arbitrary number of other commands in between attempts.

By default, the command shell keeps your last 500 commands, but you can change this by setting the HISTSIZE variable in your .bashrc or .profile:

export HISTSIZE=1024
Wow, you can even set it in decimal! Crazy! What will those wacky Unix folks think of next?

admin | Apr 24, 2009 | Comments 0

Some Commands From Around the Web

Here’s a few commands that I either use, or have found around the web. Remember to test these out before you use them live. I have for my environment, but your environment might be totally different. Sorry to state the obvious! If you have more send them on!

Recursively change file permissions without affecting directory permissions

$find ./ -type f -exec chmod 633 {} ;

Find more than one file type in the current directory

$find . -maxdepth 1 -type f -name '*.sh' -o -name '*.txt'

Find all specific file types in a directory structure. This command is looking for music files.

List all files opened by a command

$ lsof -c dhcp

List all files opened by a PID

$ lsof -c 15253

admin | Apr 22, 2009 | Comments 0

Renaming Files With Regular Expressions

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal Says:

I admit it, I’m a fan of the CommandLineFu site (hey, it’s a community, not a competition), and like trolling through it occasionally for interesting ideas. This post by vgagliardi shows a cool trick for renaming a bunch of files using regular expressions and the substitution operator in bash. For example, suppose I wanted to convert spaces to underscores in all file names in the directory:

$ for f in *; do mv -- "$f" "${f// /_}"; done

I realize that syntax looks a little crazy. The general form is “${variable/pattern/substitution}”, but in this case we have an extra “/” at the front of “pattern”, which means “replace all instances” rather than only replacing the first instance.

By the way, you can use the standard Unix regular expression syntax for your substitution pattern. For example, here’s a loop to remove all characters from file names except for alphanumeric characters, dot, hypen, and underscore:

$ for f in *; do mv -- "$f" "${f//[^-_.A-Za-z0-9]/}"; done

In this case “[^...]” means match any characters not in the specified set, and we’re performing a null substitution.

Did you notice that we’re also using the “–” argument to the “mv” command, just in case one of our file names happens to start with a “-”? These files are typically a huge pain in Unix. What if we wanted to replace all the “-” characters at the beginning of a file name with underscores?

$ for f in *; do mv -- "$f" "${f/#-/_}"; done

As you can see, starting the pattern with “#” means “match at the front of the string. Or we can match at the end of the string with “%”:

$ find docroot -type f -name \*.htm | \ while read f; do mv -- "$f" "${f/%.htm/.html}"; done

Here we’re using “find” to locate all of the *.htm files in our web docroot and then piping the output into a while loop that renames all these files to be *.html files instead.

There are a couple of problems with the method that I’m using here: (1) if multiple files map to the same name, you’ll end up clobbering all but the last instance of that file, and (2) you get errors if your substitution doesn’t actually modify the file name because the “mv” command refuses to rename a file to itself. We can fix both of these problems with a little extra logic in the loop. Let’s return to our first example of converting spaces to underscores:

$ for f in *; do n="${f// /_}"; [ -f "$n" ] || mv -- "$f" "$n"; done

First we assign the new file name to the variable $n. Then we check to see if a file named “$n” exists– the “mv” command after the “||” is only executed if there is no “$n” file.

I admit that I usually use the Perl rename program for renaming large numbers of files, because (a) the syntax is much more terse, and (b) I love Perl. But this program isn’t always available on all the different flavors of Unix that I end up having to work on. So having this functionality built into the shell is a huge win.

Ed Responds:

When I quickly glanced at Hal’s challenge initially, I thought… “Yeah, that’s pretty easy… findstr supports regex, and I’ll use the ren command to rename the files… No prob.”

And then, I started to write the command, and it got horribly ugly really quickly. Hal squealed with delight when I told him how ugly it was… and believe me… you ain’t seen nothing until you’ve seen Hal squeal with delight.

Anyway, to keep this article from getting unreasonably long, I’m going to address Hal’s original command, which replaced the spaces in file names with underscores. Unfortunately, you see, the parsing, iteration, and recursion capabilities within a single command in cmd.exe are really limiting. For parsing strings, we’ve got FOR /F and a handful of substring operations I covered in Episode 12. For running a command multiple times, we’ve got for /L, as I mentioned in Episode 3. For recursion, well, that’s just plain bad news in a single command unless we bend our rules to create a bat file that calls itself.

To start to address Hal’s original challenge, we can use the following command to determine if there are any files that have at least one space in their names in our current directory:

C:\> dir /b "* *"

That’s pretty straightforward to start, with the /b making dir show only the bare form of output, omitting cruft about volume names and sizes. Note that it will only show files that do not have the hidden attribute set. If you want, you can invoke dir with /a to make it show files regardless of their attributes, hidden or otherwise. Now, let’s see what we can do with this building block.

Plan A: Every File Should Have Four Spaces in Its Name, Right?
My original plan was to wrap that command inside a FOR /F loop, iterating over each file using FOR /F functionality to parse it into its constituent elements. I was thinking something like this:

C:\> for /F "tokens=1-4" %i in ('dir /b "* *"') do ren "%i %j %k %l" "%i_%j_%k_%l"

Well, that’s all very nice, but we’ve got a problem… let me show an example of what this beast creates when I run it in a directory with a file named “file 1.txt” and “file 2 is here.txt”:

C:\> dir /b file_1.txt__ file_2_is_here.txt

Ooops… the file1.txt name has two underscores after it. This option only works if files have exactly four spaces in their names. That’s no good.

Plan B: Let’s Just Change the First Space into an Underscore
Well, how about this… We could write a one-liner that assumes a file will have only one space in its name, and convert that one space into an underscore. That’s not too bad:

C:\> for /f "tokens=1,*" %i in ('dir /b "* *"') do ren "%i %j" "%i_%j"

I’m parsing the output of the dir /b command using parsing logic of “tokens=1,*”, which means use your default delimiters of space and break each line of the output of the dir command into the entity before the first space into %i, and everything afterward into the next iterator variable, %j.

Let’s run that with our same file names as before, yielding:

C:\> dir /b file_1.txt file_2 is here.txt

Well, we got it right for file_1.txt, because there is only one space. But, we only fixed the first space in file 2 is here.txt. Hmmmm… How could we move closer to our result?

Hit the up arrow a couple times to go back to our Plan B FOR /F loop, and hit enter again. Now, running our dir, we get:

C:\> dir /b file_1.txt file_2_is here.txt

Ahh… we nailed our second space. Hit the up arrow again and re-run… and… well, you get the picture. We can take care of one space at a time. Not so bad.

But, who wants to hit the up arrow again and again and again until we get rid of all the spaces? You’d have to re-run my Plan B command N times, where N is the maximum number of spaces inside a file name in the current directory.

Plan C: Make the Shell Re-Run the Command Instead of Doing it Manually
Well, instead of re-running a command a bunch of times, let’s make the shell do our work for us. We’ll just wrap the whole thing in a FOR /L loop to count through integers 1 through 10 (1,1,10) and invoke the FOR /F loop at each iteration through our FOR /L loop:

C:\> for /L %a in (1,1,10) do @for /f "tokens=1,*" %i in ('dir /b "* *"') do ren "%i %j" "%i_%j"

That works, provided that none of the files have more than ten spaces in their name. Ummm… but what if they do? We could raise the number 10 to 20… but that’s kind of a cheap hack, no?

Plan D: Violate the Rules — Make a 3-Line Script
OK… if we had a while construct in cmd.exe, we could simply run my FOR /F loop of Plan B while the dir /b “* *” still returned valid output. But, we don’t have a while command in cmd.exe. If we want to check a condition like that, we only have IF statements. And, if we want to jump around based on the results of IF statements, we need to use GOTOs. And, if we want to use IFs and GOTOs, we can’t dump everything on a single one-line command, but will instead have to create a little bat file.

So, I’m going to have to bend our ground rules for this blog, which require a single command, and instead use a three-line bat file. Here’s a bat file I wrote that converts all of the names of files in the current directory with spaces in them into underscores:

:begin for /F "tokens=1,*" %%i in ('dir /b "* *"') do ren "%%i %%j" "%%i_%%j" if exist "* *" goto begin

There you have it…. kind of an ugly little hack, but it works. Note that I had to change my iterator variables in my FOR loop from %i and %j into %%i and %%j. You have to do that to convert command-lines into bat files in Windows. Also, I’m using an IF statement to test for the existence of “* *”, which would match any file with a space in its name.

A small script in cmd.exe can satisfy Hal’s original challenge. To start addressing his other feats to convert other characters in file names, we could specify options for the FOR /F loop of everything we want to parse out with the syntax “tokens=1,* delims=~!@#$%^&*()+=” and whatever else you wanna take out.

I could drone on and on endlessly here, but I think you get the idea. It ain’t pretty, but it is doable…. Now that should be the cmd.exe mantra.

PS: I too am a fan of the CommandLineFu site. It rocks.

admin | Apr 22, 2009 | Comments 0

My Shell Does Math

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed says:

On Linux and Unix, I frequently rely on bc to do math at the command line. But, bc isn’t built-in to Windows. A lot of people don’t realize that the cmd.exe shell has some built-in capabilities for doing math… really lame math. But still, if you are caught in a pinch, don’t wanna invoke the calc.exe GUI, and want to focus on integer math, cmd.exe can do your bidding. It’s all invoked with the “set /a” command, as in:

C:\> set /a 2+2 4

Sexy, huh? Well, keep in mind that it is integer math only, as illustrated by:

C:\> set /a 3/2 1

That leads to some weirdness, as in:

C:\> set /a 3/2*2 2

Also, we’re working only with signed 32-bit numbers here, so make sure your results stay under approximately two billion. Bigger than that will either push you into negative territory, or outright give you an error message:

C:\>set /a 1000000000*2 2000000000

c:\test>set /a 2000000000*2 -294967296

c:\test>set /a 3000000000*2 Invalid number. Numbers are limited to 32-bits of precision.

We’ve also got some options for using hex (specified with 0x as a prefix):

C:\> set /a 0xA + 0x2 12

Or, you can do octal, by prefacing your numbers with a zero:

C:\> set /a 010 + 01 12

You can even mix and match:

C:\> set /a 0xA + 01 + 1 13

It gets exciting to think that you can do hex or octal math at the cmd.exe command line, until you realize that all of your outputs are in, well, decimal. What genius thought up adding hex and octal as input, without providing options for hex and octal in the output?

Also, we’ve got bitwise AND with &, bitwise OR with |, and XOR with ^. But, again, the output is in… ugh… decimal.

Who uses decimal anymore? I mean, when you are driving, those speed limit signs are all in hex, right? Cruising around at 0×65 mph or 0×100 km/hr can be very exciting. When pulled over, tell the officer that you thought the speed limit sign was hexadecimal. Of course, you’ll have to say that you assumed your speedometer is in decimal… kinda like set /a. Inputs in hex, outputs in decimal. See if that’ll get you out of a ticket. Good luck.

Anyway, for really simple integer math of smallish numbers, set /a can come in handy. I do sometimes use it if I’ve gotta do a quick integer calculation and I don’t wanna take focus off of my cmd.exe on the screen.

Honestly, given the quite limited abilities for cmd.exe to do math this way, I’m kind of expecting my command line kung fu sparring partners working in bash to knock me out with this one. But, in leading with my jaw, I’m hoping in the process that we’ll all learn some completely magical fu from them when doing math in bash. So, lay it on us, guys. What fu have you?

Mr. Bucket read the draft of this post, and kicked in some interesting fu from the dawn of time for doing some hex match using built-in Windows command line tools. I remembered this one, but barely. It was buried in the dark recesses of my mind… I haven’t used this since I bought a really good hex calculator a long long time ago.

Mr. Bucket suggested running the built-in Windows debugger tool:

C:\> debug

Then, at the goofy little “-” prompt, you could type:

- H 4a44 90

The debugger will then print out the sum and difference of the two numbers you provided:

4AD4 49B4

So there…. next time you are stranded on a desert island with only a Windows box and need to do hex addition or subtraction to hack your way off, you’ll be all set. Thanks for the interesting point, Mr. Bucket!

Paul Says:

On UNIX/Linux systems I’ve always found that the bc command was available to me if I need to perform simple math (or even that “new” math people talk about):

It can even do fancy things like square root:

Of course Bash itself can do math as well. This is handy in shell scripts. Below is a command line “one-liner” example:

$ let RESULT=2+2 ; echo $RESULT 4

As long as “2+2″ continues to equal 4 the universe stays in balance and life is good :) Of course, if you leave out the “let” directive you get very different results:

$ RESULT=2+2 ; echo $RESULT 2+2

Here “2+2″ is being treated as a string, and therefore no math will be performed. So remember, when using Bash always “Let there be math”.

Hal Says:

Ummmm, Paul? How about just:

$ echo $((2+2)) 4

Using “$((…))” to put arithmetic expressions in-line is a heck of a lot easier than that tedious “let …” nonsense.

By the way, for those of you that aren’t using bash or a more modern shell with built-in arithmetic, there’s also the really old-school expr command in addition to bc:

$ expr 2 + 2 4

admin | Apr 20, 2009 | Comments 0

Copying and Synchronizing (Remote) Directories

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal Says:

Copying files and directories around is a pretty common task, and there are all sorts of ways to accomplish this in Unix. But I have a strong preference for using the rsync command because (a) it allows me to copy files either within a single machine or between systems over the network, (b) it can keep directories in sync, preserve timestamps, ownerships, etc, and (c) it’s smart enough to only copy files that have changed, which is a big savings on time and bandwidth.

Basic rsync usage is straightforward:

$ rsync -aH dir1/ dir2/ #copy dir1 to dir2 on same machine $ rsync -aH remotehost:/path/to/dir/ localdir/ #copy remote dir to local machine $ rsync -aH dir/ remotehost:/path/to/dir/ #copy local dir to remote machine $ rsync -aH dir/ someuser@remotehost:/path/to/dir/ #copy dir to remote host as someuser
The “-a” option combines the most common rsync options for copying and preserving directories: “-r” for recursive, “-ptog” to preserve permissions/timestamps/owner/group owner, “-l” to copy symlinks as symlinks, and “-D” to preserve device and special files. The only useful option not included in “-a” is the “-H” option to preserve hard links. Even though “-H” adds extra processing time, I tend to always use it, just to be on the safe side.

By the way, the trailing “/” characters on the source directory paths are significant. I could explain it in words, but it’s easier to just show you an example:

$ ls source bar.c baz.c foo.c $ rsync -aH source dir1 $ ls dir1 source $ rsync -aH source/ dir2 $ ls dir2 bar.c baz.c foo.c
Without the trailing “/”, rsync copies the entire directory, including the top directory itself. With the trailing “/”, the contents of the source directory are put into the top of the destination directory.

If you want to keep two directories in sync, you probably want to include the “–delete” option, which deletes files in the target directory that aren’t present in the source:

$ rsync -aH source/ target $ rm source/foo.c $ rsync -aH source/ target $ ls target bar.c baz.c foo.c $ rsync -aH --delete source/ target $ ls target bar.c baz.c

Sometimes there are certain files that you don’t want to copy:

$ ls source bar.c bar.o baz.c baz.o foo.c foo.o $ rsync -aH --delete --exclude=\*.o source/ target $ ls target bar.c baz.c foo.c

Here I’m using “–exclude” to not copy the object files, just the source files. You can put multiple “–exclude” options on a single command line, but after a while it gets annoying to keep typing the same set of excludes over and over again. So there’s also an “–exclude-from=” option to read in a list of exclude patterns from the file “”.

rsync also has a “-n” option like the “make” command, which shows you what would happen without actually copying any files. You usually want to combine “-n” with “-v” (verbose) so you can actually see what rsync would be doing:

$ rsync -aHnv --delete --exclude=\*.o source/ newdir sending incremental file list created directory newdir ./ bar.c baz.c foo.c

sent 90 bytes received 24 bytes 228.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)

Ed Joyfully Responds:

You thought you had me with your fancy shmancy rsync, didn’t ya, Hal? Well… ha! A couple of years ago, you’d have been right, because there was no good built-in Windows tool for synchronizing directories. The xcopy command is a decent file copy tool, but it doesn’t really police file updates to maintain synchronicity (and yes, that oblique reference to “The Police” was intentional). A couple of years ago, you’d have to install a separate tool for synchronizing. A really nice tool for doing that is robocopy, available in various resource kits, such as the Windows 2003 kit here.

But, here’s the really good news: robocopy is built-in to Vista and Windows 2008 server! It’s a really nice synchronization tool, and its name is far cooler than the rather pedestrian “rsync”. “Robocopy” sounds like a cool mechanized buddy, or perhaps even a robotized superhero law enforcement officer.

So, how can we use robocopy to achieve what Hal does with rsync above? Well, to mirror from one local directory to another local directory, you could run:

C:\> robocopy dir1 dir2 /s
The /s makes robocopy recurse subdirectories. You could put the /s up front, but, generally speaking, it’s best to put the source and destination directories first with robocopy, especially when you start to define file and directory exclusions. Note that robocopy cannot copy hard links, so we lose them (i.e., there is no rsync -H equivalent). Note also that robocopy works the same way whether you specifiy dir1 or dir1/, unlike rsync. That’s ok with me even though it is slightly less flexible, as there is less of a chance that I’ll screw something up.

To use robocopy to replicate something to or from a remote directory, just refer to the directory as \\[machine]\[share]\[dir], as you might expect, as in:

C:\> robocopy plans_for_world_domination \\backup\rainbowsANDunicorns /s /z

Another nice feature associated with using robocopy across a network involves what happens when network connectivity is lost. If you invoke it the right way, robocopy maintains status so that it can pick up where it left off when doing a copy. When you invoke robocopy, use the /z option to run it in restartable mode. The /z makes it maintain the status information necessary to restart a copy that is interrupted.

If you want to keep directories in sync (removing files from the destination that have been deleted from the source), you can use the /MIR option (/MIR actually means /E plus /PURGE). As with rsync, robocopy will copy all files by default. To omit certain files, we have a huge number of exclusion options, such as /XF to exclude files and /XD to exclude directories, both of which support wildcards with *.

Thus, to mimic Hal’s fu above, we could run:

C:\> robocopy source target /S /MIR /XF *.o

Oh, and to do a dry run, just printing out information about what would be copied, instead of actually doing the copies, we can invoke robocopy with the /L option, as in:

C:\> robocopy source target /L /S /MIR /XF *.o

If you’d like to copy all file attributes, ownership information, and file permissions, invoke robocopy with /COPYALL.

The output of robocopy is quite nice as well, showing detailed statistics about what was copied, what was skipped (because it was already there, or was excluded), detailed times, and a timestamp of invocation and completion:

c:\> robocopy source target /L /S /MIR /XF *.o
------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows

-------------------------------------------------------------------------------
Started : Sun Apr 12 07:54:51 2009

Source : c:\test\source\ Dest : c:\test\target\

Files : *.*

Exc Files : *.o

Options : *.* /L /S /E /COPY:DAT /PURGE /MIR /R:1000000 /W:30
------------------------------------------------------------------------------

4 c:\test\source\ New File 4 bar.c New File 4 baz.c New File 4 foo.c 1 c:\test\source\hello\

------------------------------------------------------------------------------

Total Copied Skipped Mismatch FAILED Extras Dirs : 2 0 2 0 0 0 Files : 5 3 2 0 0 0 Bytes : 24 12 12 0 0 0 Times : 0:00:00 0:00:00 0:00:00 0:00:00

Ended : Sun Apr 12 07:54:51 2009

Yeah, robocopy!

admin | Apr 17, 2009 | Comments 0

Hal Says:

Paul’s last post left me hankering to talk a little bit more about job control in the Unix shell. Besides, it’ll make Ed feel really bad about the Windows command shell, so what’s not to like?

The simplest form of job control is to hit “^Z” to suspend a currently running process. Once the process is suspended, you get your command prompt back and can issue other commands. You can also type “bg” to force the suspended process into the background, where it will continue to run:

# tail -f /var/log/httpd/error_log [... output not shown ...] ^Z [1]+ Stopped tail -f /var/log/httpd/error_log # bg [1]+ tail -f /var/log/httpd/error_log &

Now you can fiddle with your Apache config file, restart the web server, etc. The “tail -f” process is still running in the background and displaying errors as they appear in the log file. I find this very useful if I’m working remotely to resolve problems on my web server. Actually, I could have simply started that “tail” command in the background by putting a “&” at the end of the command line:

# tail -f /var/log/httpd/error_log & [1] 14050

I often use job control to flip in and out of a superuser shell:

$ /bin/su Password: # [... do some stuff as root ...] # suspend

[1]+ Stopped /bin/su $ [... do some stuff as a normal user ...] $ fg /bin/su #

Notice that you use “fg” to start a suspended process running again. This sure beats having to keep using “su” and re-enter the root password all the time. Of course, you probably should be using “sudo”, but that’s a topic for another day.

By the way, you can also use job control to blip in and out of remote SSH sessions too:

[hal@localhost ~]$ ssh remotehost hal@remotehost's password: Last login: Sun Apr 5 10:06:26 2009 from ... [hal@remotehost ~]$ [hal@remotehost ~]$ ~^Z [suspend ssh]

[1]+ Stopped ssh remotehost [hal@localhost ~]$ fg ssh remotehost

[hal@remotehost ~]$

Note that the key sequence to suspend an SSH session is “\n~^Z”– you have to enter the tilde character at the beginning of a line, not while you’re in the middle of a command-line. By the way, if you’re logged into multiple machines in series, you can use multiple tildes to pick which of the chained SSH sessions you actually want to suspend.

Sometimes you might have multiple jobs suspended at the same time. You can use the “jobs” command to display them, and use the job number displayed in the output to interact with the different jobs:

# jobs [1] Stopped vi /etc/httpd/conf/httpd.conf [2]- Stopped vi /etc/httpd/conf.d/ssl.conf [3]+ Stopped tail -f /var/log/httpd/error_log # bg %3 [3]+ tail -f /var/log/httpd/error_log & # fg %1 vi /etc/httpd/conf/httpd.conf ^Z [1]+ Stopped vi /etc/httpd/conf/httpd.conf # %2 vi /etc/httpd/conf.d/ssl.conf ^Z [2]+ Stopped vi /etc/httpd/conf.d/ssl.conf # fg vi /etc/httpd/conf.d/ssl.conf

In general you do “[fg|bg] %n” to foreground or background job number “n”. However, you can leave off the “fg” if you want because foregrounding a job is the default. If you don’t specify a job number after the “fg” or “bg”, then the most recently manipulated job number is assumed– you can see this job with a “+” next to it in the output of the “jobs” command.

Ed Wimpers:

Job control?!? What are you, Hal… just plain cruel? I long for true job control in my cmd.exe. If I want true job control, I just install Cygwin.

But, there are some job control-like things I can do in a pinch on a box without Cygwin.

For example, if I want to start a process in a separate cmd.exe window, I can use the start command. Here, I’ll use it to start a separate shell window to search for wmic.exe on the C:\ partition, using the file search capabilities we described in Episode #21:

C:\> start dir /s /b c:\wmic.exe

That pops up a separate Window that will display my results. The window will linger displaying my results after it’s done. If I want to start it up minimized, I can run:

C:\> start /MIN dir /s /b c:\wmic.exe

If I want it to start a process in a separate window and then have it disappear when it is done running, I use:

C:\> start cmd.exe /c dir /s /b c:\wmic.exe

Sadly, there’s no easy way to get access to standard out of these separately invoked shells windows while they are running, other than to have them dump their output into a file, with “> filename” at the command line invocation.

But now for the tricky part… how can you mimic the & feature of most Linux shells to run something in the background of the current cmd.exe while still dumping their standard output into the current shell’s display? We can use the /b option of start to put something in the background, as in:

C:\> start /b dir /s /b c:\wmic.exe

So, we’ve got &-like behavior! However, it’s not really full job control, because I can’t pull that job into the foreground, or select one job from a group of backgrounded tasks. But, I can kill my background task, by hitting CTRL-break in the cmd.exe window that spawned it. Also, each time you run “start /b”, it will leave an extra cmd.exe process running until you either exit that process (by typing… you guessed it… “exit”) or killing it.

To kill “jobs” (if I can call them that) more thoroughly, I usually rely on wmic or taskkill, as defined in Episode #22.

So, there you have it… not quite job control, but an ability to kick things into the background so you can keep your current shell active and get more work done. Oh, and install Cygwin. You’ll be glad you did. :)

Paul Says:

I will never forget when I was first learning Linux/UNIX (I actually got my feet wet with Linux/UNIX by using Linux at home, and AIX at work). I was configuring a proxy server, carefully constructing each command line switch and running the command. I’d hit ^Z to put the command in the background and wonder why my proxy server wasn’t listening. I had forgotten to put the & after the command, so it truly did “suspend” the command. I never made that mistake again :)

I would also like to quickly highlight the nohup command. I often use this when I run a command that will take a long time and want to go back and check its output:

$ nohup nmap -sS -T4 -iL edswindowsnetwork -oA myscanresults &

Sometimes processes you start while logged into a remote host will SIGHUP when you logoff, and nohup can prevent that.

admin | Apr 15, 2009 | Comments 0

Information Technology and Security

Information technology and information security are my fields of expertise, and I have the pleasure of working within those fields as a career. The abstract thought process and mix of technical knowledge make it almost like play time. Thinking outside the box is outmoded – you have to think even more abstract since you are trying to see all points of view – from CEO to hacker – from tactical to strategic, and even political.

Information Technology and Security

I’ve posted a lot of information technology and information security related post to this website. I learn from the information I gleen from around the web and I wanted a place where I could refer back, since some of the command line and shortcut stuff is priceless. It doesn’t matter what status I hold at work, I’m always interested in cleaning up my skills, and learning new ones. There should never be a point, even in the executive layer, that we should let go of those skills.

Information Security is an area I have a lot of passion in. I am the Director of Information Technology and Information Security Officer for the company that I work for and, as such, have to keep my finger on the pulse. I have done hacking course and am technically proficient, but I would not say that I am anything other than someone who sees how it can be done, and wants to prevent it happening to the company I work for.

Here’s a Standard Penetration Testing Checklist. See how involved it is, and that is just the entry point. I didn’t write this by the way – why re-invent the wheel, but it’s a great reminder and backbone for penetration testing. All that I am trying to illustrate is the complexity of information security, and that it is all too often overlooked by executives for no other reason than not arming them with enough information. Yes – we should take the blame for some of that. When I presented the base level of hacking techniques to our executive staff I immediately got budget money. I meant to scare them, and boy did I.

You’ll notice in the tech section there is a ton of useful information. This is just a piece of what I find useful – I don’t have time to post it all so I try to post the most interesting – well, to me anyway. More as it comes to me.

admin | Apr 11, 2009 | Comments 0

Ping beep of death

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed says:

Paul threw down a challenge to Hal and me in e-mail just a few minutes ago:

“I want a string of commands that will ping a host and for each time a packet is missing (i.e. timeout) send a beep to the console.”

The solution to this one in Windows includes some useful constructs, so let’s have at it with this fu:

C:\> for /L %i in (1,0,2) do @(ping -n 1 HostIPaddr || echo ^G) & ping -n 2 127.0.0.1

This command starts out with a FOR /L loop, set with an iterator variable of %i that I won’t use, counting from 1 to 2 in steps of 0. In other words, it’ll run forever, or until someone kills it. At each iteration through the loop, I turn off command echo (@), and ping the target host one time (ping -n 1). Pretty pedestrian so far.

Now we get to some interesting stuff. If the ping command fails to get a response, I’ll have it run the echo command to make a beep. As we’ve seen in earlier episodes, we can use cmd1 && cmd2 to make the shell run cmd2 only if cmd1 succeeded. Likewise, we can use cmd1 || cmd2 to make the system run cmd2 only if cmd1 fails. This is more efficient than checking the %errorlevel% environment variable with an IF statement, but I digress. Of course, as always cmd1 & cmd2 means run cmd2 regardless of the success or failure of cmd1.

If the ping fails, we have “echo ^G”. Note that it looks like I typed Shift-6 G, but I didn’t. That little ^G is created by holding down the CTRL key and hitting G. It’s just displayed like ^G. It’s how we can make the echo command ring the system beep. After this, I simply introduce a 1-second delay by pinging localhost twice (first ping instantly, second ping one second later). So, there you have it.

BTW, if you want prettier output from this, you can dump various Standard Output to nul as follows:

C:\> for /L %i in (1,0,2) do @(ping -n 1 HostIPaddr > nul || echo ^G) & ping -n 2 127.0.0.1 > nul

Hal Chimes In:

We can easily do the Unix equivalent of what Ed’s doing in the Windows command shell:

$ while :; do ping -c 1 -w 1 HostIPaddr >/dev/null || echo -e \\a; sleep 2; done

Note the “echo -e \\a” syntax for emitting an “alert” (^G or BEL).

However, in its default mode the standard Unix ping command will run continuously. So it seems like you should be able to get some joy just by piping the output of ping into another command, without having to resort to a loop. Mr. Bucket had the clever idea of filtering the output of ping through awk and then sending the resulting misses to the “say” command on his OS X machine (because talking is much cooler than beeping):

$ ping x.x.x.x 2>&1 | awk -F: '/sendto:/ {print $3}' | say

The above command seems like it should work, but it wasn’t bringing the noise. A confused Mr. Bucket emailed me for clarification.

What’s happening here is that Mr. Bucket’s clever idea is getting kneecapped by the standard buffering behavior of shell pipes. If you run that first ping command by itself and have the output go to your terminal, then you see the results of each ICMP packet, one line at a time. However, the minute you pipe the output of the ping command into another command, the shell switches to page buffering the output between the two commands. In other words, the awk command gets the output of the ping command in 4096 byte chunks, and the output of awk has to fill up 4096 bytes before the “say” command gets anything. If Mr. Bucket had let the command run long enough, eventually he would have gotten a burst of talking from the “say” command, but only long after the actual failed ping events.

Unfortunately, there is no magic shell variable you can set to fix this problem globally. However, applications in the Unix environment can force their output to be unbuffered if the application developer chooses to add this functionality to their program. Sometimes the software does this automatically– the tee program in most Unix environments unbuffers its output by default (using setvbuf(3)). Sometimes you need to add an extra command-line option, like with the awk command on my Ubuntu system (which is really mawk):

$ ping x.x.x.x 2>&1 | awk -W interactive -F: '/sendto:/ {print $3}' | ...

That’ll get you line-buffered output that you can pipe into another program. The GNU grep utility has the “–line-buffered” option, which is similar.

Paul Chimes In:

I have to confess, I really thought this would be something pretty easy. The page buffering thing threw me through a loop too (pun intended). I also discovered a much better way to do this in OS X:

$ ping -A 192.168.1.1

This command will beep (ASCII 0×07) “when no packet is received before the next packet is transmitted.” so says the man pages. This is exactly what I was looking for to monitor hosts while I scan them with various security tools (like Nessus) and alert me if they crash or become unresponsive.

admin | Apr 08, 2009 | Comments 0