All Entries Tagged With: "tricks"

Fixing the Filenames

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal Helps Out

A friend of mine contacted me the other day with an interesting problem. She was trying to recover some files from the backup of an old BBS. In particular, she was trying to get at the attachments for various postings.

The attachment files were in a big directory, but the file names unhelpfully used an internal attachment ID number from the BBS. So we had file names like “attachment.43567″. Now my friend also had a text file she extracted from the BBS that mapped attachment IDs to the real file names:

43567 sekrit plans.doc 44211 pizza-costs.xls ...

So the task was to take the file of “attachment ID to file name mappings” and use that to rename the files in the attachments directory to their correct file names.

I thought about it for a minute, and realized the solution was actually pretty straightforward:

$ while read id file; do mv attachment.$id "$file"; done <id-to-filename.txt

The trickiest part of the exercise was dealing with the file names that had spaces in them, like “sekrit plans.doc”. Luckily the format of the input file was “ID filename”, which meant that I could treat everything after the first whitespace as the file name. And this is exactly what the builtin “read” command will do for you: in this case it puts the first whitespace delimited token into the $id variable and then jams whatever is left over into the last “$file” variable. Once I got the right information into $file, it was simply a matter of making sure to quote this variable appropriately in the “mv” command inside the loop.

So there you go– a quick one-liner for me, but a real time-saver for my friend. And possibly a real time-sink for Tim and Ed as they try and figure out how to do this in their shells. Let’s see, shall we?

Ed Frustrates Hal

Sorry, Hal, but this one just isn’t crushing for me. I know that disappoints you, but sometimes (on fairly rare occasions) we don’t have to work too hard to coax little cmd.exe to do what we want. It does take two little tricks, though, but nothing too freakish.

Here’s the fu:

C:\> for /f "tokens=1,*" %i in (id_to_filename.txt) do @copy attachment.%i "%j"

I’m using a FOR /F loop to read the contents of id_to_filename.txt, one line at a time. Default delimiters of FOR /F parsing are spaces and tabs, which will work just fine for us here, so there’s no need to mess with custom delims. I’ve specified custom parsing of “tokens=1,*”, which will make it assign the first column of the file (the integer in Hal’s example) to my first iterator variable (which is %i). Then, the ,* stuff means to assign all of the rest of the line to my second iterator variable, which will be auto-allocated as %j. The ,* stuff is the first trick, which really comes in handy.

Then, in the body of my loop, I turn off display of commands (@) and invoke the copy command to take the contents of attachment.%i and place it into “%j”. The second trick, those quotes around %j, are important in allowing us to handle any spaces in the file name. Note that I’m using copy instead of move here, because I don’t wanna play Ed-Zilla stomping over the city just in case something goes awry (who’s to say that our id_to_filename.txt file will always look like we expect it to?). I guess you could call it the Hipposhellic oath: First do no harm. After we verify that our copy worked like we wanted with a quick dir command, we can always run “del attachment.*”

Whatcha got, Tim?

Tim frustrates most people

Sorry Hal, this isn’t too bad in PowerShell either. There are a few ways we can accomplish this task, but I elected to pick the shortest version, which also happens to be the one that brings up something we haven’t covered before. Here are the long version and short version of the fu. (The short version is identical but uses built in aliases)

PS C:\> Get-Content id-to-filename.txt | ForEach-Object { $id,$file = $_.Split(" ",2); Rename-Item -Path attachment.$id -NewName $file }

PS C:\> gc id-to-filename.txt | % { $id,$file = $_.Split(" ",2); ren attachment.$id $file }

The Get-Content cmdlet is used to read the contents of the file, and it is piped into Foreach-Object. Inside the Foreach-Object script block is where the line is split. The first parameter used in the Split method defines the delimiter and the second defines how many items it should be split into.

The only problem, the Split method’s output is multi-line. Here is an illustration:

PS C:\> gc id-to-filename.txt -TotalCount 1 | % { $_.Split(" ",2); } 43567 sekrit plans.doc

We need both portions of the split to do the rename, so here is where we bring up a new little trick. We can assign the output of split into variables. Each line is assigned to a variable, the first variable ($id) is assigned the first line and the second variable ($file) receives the remainder. After we have the Id and the Filename we can easily rename the files.

If we wanted to be a little safer then we could use Copy-Item (alias cp or cpi) instead of Rename-Item (alias ren or rni). Once we confirmed the copy was successful we can delete all the attachment files by using “Remove-Item attachment.*” (alias del, erase, ri, or rm).

admin | Mar 02, 2010 | Comments 0

Faster. Higher. Stronger – speeding up tasks

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Tim goes for Gold:

The Olympics are in full swing. The world’s finest athletes are doing everything they can to shave fractions of a second; and not get caught doping. This episode we try to shave off precious seconds from our tasks, and not get caught with our (Irish) coffee at work (again).

We all have those commands that we use regularly, and accessing those commands more quickly would save some time. With PoweShell we can do just that by using Set-Alias.

PS C:\> Set-Alias -Name ss -Value Select-String

The Name and Value paramters are positional, so we don’t have to use the parameter names. This command does the same thing:

PS C:\> Set-Alias ss Select-String

We have created our alias, now how can we use it? If we wanted to search all the files in a directory for the word “test” we had to use this command:

PS C:\> gci | Select-String test

…but now we can use this shorter command:

PS C:\> gci | ss test

The second command is half the length of the first command command. That is a nice efficiency gain.

What if we regularly checked the event log to see the five latest items? Obviously a short command would save us some time, but see what happens when we try to create an alias.

PS C:\> Set-Alias g5 Get-WinEvent -MaxEvents 5 Set-Alias : A parameter cannot be found that matches parameter name 'MaxEvents'.

That didn’t work, but we can do it, we just need to use a function.

PS C:\> function Get-Last5Events { Get-WinEvent -MaxEvents 5 } PS C:\> Get-Last5Events

TimeCreated ProviderName Id Message ----------- ------------ -- ------- 2/23/2010 8:12:1... Service Control ... 7000 The Diagnostic S... 2/23/2010 8:12:1... Microsoft-Window... 135 The Diagnostic P... 2/23/2010 8:12:1... Service Control ... 7000 The Diagnostic S... 2/23/2010 8:12:1... Microsoft-Window... 135 The Diagnostic P... 2/23/2010 8:11:4... Service Control ... 7036 The Computer Bro...

The name we picked for our function is a bit long, so let’s use Set-Alias to create an alias for the function.

PS C:\> Set-Alias g5 Get-Last5Events

So we’ve shaved a few seconds off of our commands, now on to the doping.

In PowerShell we can use snap-ins and modules to extend the shell. There are modules and snap-ins for managing Active Directory, Group Policy, Diagnostics, Exchange 2007 and 2010, SharePoint 2010, IIS 7, VMware, and many more servers and services.

Snap-ins load sets of cmdlets and providers. Modules, which are only available in v2, can include cmdlets, providers, functions, variables, aliases, and much more. Modules are easier to create than snap-ins and they appear destined to replace snapins as the main way to extend PowerShell. You programmers can think of modules as a “class” while a snap-in is just a collection of functions.

Here is how a snap-in is loaded:

PS C:\> Add-PSSnapin VMware.VimAutomation.Core

Wildard characters can be used in the module name. I use it since I can’t ever remeber to type VMware.VimAutomation.Core, so I just type *vmware*. Let’s see what cmdlets have been added.

PS C:\> Get-Command -Module *vmware*

CommandType Name Definition ----------- ---- ---------- Cmdlet Add-VMHost Add-VMHost [-Name] <String> ... Cmdlet Add-VMHostNtpServer Add-VMHostNtpServer [-NtpSer... Cmdlet Apply-VMHostProfile Apply-VMHostProfile [-Entity... Cmdlet Connect-VIServer Connect-VIServer [-Server] <... ...

Now, let’s load a module and then see what new cmdlets are available:

PS C:\> Import-Module GroupPolicy PS C:\> Get-Command -Module GroupPolicy
CommandType Name Definition ----------- ---- ---------- Cmdlet Backup-GPO Backup-GPO -Guid <Guid> -Pat... Cmdlet Copy-GPO Copy-GPO -SourceGuid <Guid> ... Cmdlet Get-GPInheritance Get-GPInheritance [-Target] ... Cmdlet Get-GPO Get-GPO [-Guid] <Guid> [[-Do... ...

We have these new aliases, functions and cmdlets, but we don’t want to go through the same setup everytime. We can automatically load these by editing our profile, but where is it?

There are four profiles, but we typically only work with the one stored in $profile. This file may not exist, so you may need to create it.

PS C:\> $profile C:\Users\tim\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1

We can edit the file with good ol’ notepad.

PS C:\> notepad $profile

Once we have it open we can add all of our aliases, functions, snap-ins and modules.

Set-Alias -Name ss -Value Select-String function Get-Last5Events { Get-WinEvent -MaxEvents 5 } Add-PSSnapin VMware.VimAutomation.Core Get-Command -Module GroupPolicy Clear-Host Write-Host "Welcome to Tim's shell..."

This is a good spot to add those commands and functions that aren’t built into PowerShell such as Test-Hash and Get-Netstat.

Let’s see how the other shells compete.

Hal’s Been There, Done That

“See how other shells compete”? Kid, Unix shells were medalling in the Olympic Aliasing event before Powershell was even a gleam in some demented Microsoft developer’s eye.

Setting up aliases in bash is straightforward. Here’s how I make it easy to access the history command in my shell:

$ alias h=history $ h ... 501 alias h=history 502 h $ type h h is aliased to `history'

Notice that the type command will tell you when a given command is an alias (or a shell built-in, or a regular command, or…), and thus is preferrable to commands like which.

Aside from using aliases to shorten commands that I use frequently, I’ll often use aliases as a way of making sure I get the program I want, even when I type the wrong thing:

alias more=less alias mail=mutt

And, yes, I still regularly use mutt to read my email. You got a problem with that?

Aliases can always have multiple arguments, you just need to be careful with your quoting:

alias clean='rm -f *~ .*~' alias l='ls -AF' alias tless='less +G'

Aliases can do pretty much any shell code you can imagine, but they do have one shortcoming: any variables you put into an alias are expanded when the alias is read during the start-up of your shell. You can’t generally have variables that get interpolated when the alias is executed. For example, I wanted to create an alias that did a “cd” to whatever directory the last file I accessed lives in. But that requires run-time interpolation in order to know the last file accessed, so I can’t do it in an alias.

However, you can use shell functions for this:

$ function f { cd `dirname $_`; } $ cp ~/.bashrc ~/stuff/testing/src/clkf/aliases/bashrc.example $ f $ pwd /home/hal/stuff/testing/src/clkf/aliases

I called the function “f” for “follow”– as in, “copy that file to a directory and then follow it over there”. It’s hugely useful.

You’ll notice that the function uses the magical bash builtin variable “$_”, which is always set to the last argument of the previous command. So my function only works if the last argument of the last command is a file name, but that’s good enough for my purposes. If I had specified a directory instead of a file name in the cp command, then I could have just used “cd !$” instead of “f”:

$ cp ~/.bashrc ~/stuff/testing/src/clkf/aliases $ cd !$ cd ~/stuff/testing/src/clkf/aliases

If you’re wondering what the heck that “!$” is all about, you need to go back and check out Episode #14.

The best place to put your aliases and shell functions is in your ~/.bashrc file. That way, they’ll get read every time you fire off a shell.

Hmmm, I hope this isn’t another one of those “easy for Unix (and Powershell), hard for CMD.EXE” kinds of things. Ed always gets so cranky about those…

Ed’s Not Overly Cranky Today

Cranky? Moi? Nevah. Well, mostly never. Ok… sometimes.

Despite my non-cranky demeanor, support for aliases in the cmd.exe shell isn’t particularly strong. In this corner of the shell-o-sphere, we typically apply two common approaches to mimicking aliases: doskey macros and small batch files. Let’s look at each.

Didja shudder just a little bit when I mentioned doskey? Yeah, it’s very old skool, but it can help us get some work done. To define a macro, you could simply run:

C:\> doskey <MacroName>=<Macro>

For example, if you want to display your running processes by simply running “ps”, you could run:

C:\> doskey ps=wmic process list brief C:\> ps

HandleCount Name Priority ProcessId ThreadCount WorkingSetSize 0 System Idle Process 0 0 1 24576 426 System 8 4 97 581632 ---SNIP---

That simple macro is nice, but it’s got a big limitation we’ve got to overcome by expanding our macro knowledge. To see this limitation, let’s try running our new macro and searching its output for cmd.exe:

C:\> ps | find "cmd.exe" HandleCount Name Priority ProcessId ThreadCount WorkingSetSize 0 System Idle Process 0 0 1 24576 426 System 8 4 97 581632 ---SNIP---

Doh! We only got the full output of our ps macro, without our search taking effect. What happened? Well, the macro substitution by default just ignores anything that follows it on the command line, unless we define the macro to end with $*, which holds the remainder of the command line typed in after the macro. By putting it at the end of our macro, everything typed after the macro will be executed after macro expansion. So, a better ps would be:

C:\> doskey ps=wmic process list brief $* C:\> ps | find "cmd.exe" 23 cmd.exe 8 676 1 1462272 28 cmd.exe 8 4008 1 2785280
That’s what Momma likes.

Next, to more closely mimic what Hal does above with shell history by simply running the “history” command or the “h” command, you could use:

C:\> doskey history=doskey /history $* C:\> doskey h=doskey /history $*

C:\> h

Note that I did use $* here, in case someone wants to pipe our output to another command or to redirect it into a file. If you just run the macro with nothing after it, $* contains nothing, so it executes as you’d expect… no harm, no foul. You can even split up the command line options following your macro, referring to each component as $1 up to $9, using each independently. And, if you want to have multiple commands in a macro, enter them as command1$Tcommand2, rather than using & between them. But, if you really want to use &, modern versions of Windows allow you to define the macro as command1 ^& command2. Likewise, if you want to do any piping in doskey macros, make sure you use a ^|.

If you ever want to list all of the macros you’ve created, run:

C:\> doskey /macros

It should be noted that these macros take precedent over any normal commands you type at cmd.exe, so you could really make your shell useless with them. For example, if you run:

c:\> doskey dir=echo No Way Dude!

c:\> dir No Way Dude!

You’ve lost access to the dir command. To undefine a macro, simply use doskey to redefine a macro name with nothing after the equals sign:

C:\> doskey dir= C:\> dir Volume in drive C has no label. Volume Serial Number is 442A-03DE ---SNIP---

Now, these little toy macros we’ve defined above are cute, but let’s get into some macros that really save us some serious time. Faithful readers of this blog (Hi Mom!) know that many of my episodes include certain constructs, like making a command run forever or invoking delayed variable expansion. Let’s look at some macros for those:

C:\> doskey forever=for /L %z in (1,0,2) do @$*

C:\> forever ipconfig /displaydns & ping -n 2 127.0.0.1 > nul

Here, I’ve created a macro called “forever” that invokes a FOR /L loop, set to count from 1 to 2 in steps of zero. That way, it’ll run forever. In the DO clause of my loop, I turn off the display of commands (@) and run whatever follows the invocation of forever. Note that I have to define a variable for my FOR loop, and I’ve chosen %z. That’s because I want to minimize the chance I’ll have a collision with any variables I might choose for my command to forever-ize. If I had used a %i in my macro, I’d really be thrown for a loop (no pun intended) if I used %i in my follow-up command. With FOR /F loops that have multiple tokens, variables are dynamically allocated alphabetically, so I hang out at the end of the alphabet here to lower the chance of collision.

You could put your ping delay inside the forever macro, but I find it best to keep it out, giving me more flexibility as I define my commands. Note that the command I’m running after forever here will run ipconfig to dump the DNS cache, followed by a one-second delay introduced by pinging myself twice (first ping happens immediately, followed by another ping one second later).

And, to perform delayed variable expansion (described in Episode #12), I could define a macro of:

C:\> doskey dve=cmd.exe /v:on /c "$*"

Note that you can put stuff in a macro _after_ the $* remainder of your command-line invocation, as I’m putting in a close quotation mark here to make sure that my full command gets executed in the context of delayed variable expansion. Now, I can do crazy stuff like this from Episode 58, filling the screen with random numbers to look a little like the Matrix:

C:\> dve for /L %i in (1,0,2) do @set /a !random!

It’s important to note that any environment variables whose value changes in your command must be referred to using !variable_name!, instead of the more traditional %variable_name%.

Ahhh… “But wait,” you say. After running dve in this example, I manually typed out a FOR /L loop that was simply my “forever” macro from before. Couldn’t I just do a dve followed by a forever? Let’s try it:

C:\> dve forever set /a !random! 'forever' is not recognized as an internal or external command, operable program or batch file.

No sir. You can’t nest these suckers. Also, they have a problem if they aren’t the first command included on your command line. Consider:

C:\> echo hello & ps hello 'ps' is not recognized as an internal or external command, operable program or batch file.

So, we’ve got some pretty serious limitations here*, but, as Tim points out, they can shave off a precious few seconds, especially for things we often start our command-lines with, such as the forever FOR loop or delayed variable expansion.

Wanna see a really whacked out macro? This one was provided by Brian Dyson, a cmd.exe warrior like no other (and a long-lost twin of my friend Mike Poor, but that is a story for another day). Brian showed me a macro in which he emulates the !command feature of bash, which Hal alludes to above. Check out this amazing action, which I quote from Brian:

For the hard-core DOSKEY macro lover, a ‘!’ DOSKEY macro that acts (somewhat like) Bash, running the last command:

c:\> doskey !=doskey /history ^| findstr /r /c:"^$*" ^| findstr /rv /c:"^!" ^> "%TEMP%\history.txt" ^&^& ((for /f "tokens=*" %a in (%TEMP%\history.txt) do @set _=%a) ^&^& call echo.%_^% ^& call call ^%_^%) ^& if exist "%TEMP%\history.txt" del "%TEMP%\history.txt"

Here we get the history and pipe it into a `findstr` command searching
for commands that begin with the arguments to `!`. We remove any
previous `!` command and redirect everything into a temporary file. (I
couldn’t find a work-around for the temporary file). If the final
findstr was successful, parse through the temporary file and set ‘_’ to
the last command (like Bash). If this was successful, then echo out the
command and call it via double `call`. Finally, clean up the temporary
history file.

Dude! Nice. Now, to invoke Brian’s macro, you have to run ! followed by a space, followed by a command or letters and it will invoke the last instance of whatever previous command started with the letters you type. Check it out:

C:\> ! di dir Volume in drive C has no label. Volume Serial Number is 442A-03DE

Directory of c:\ ---SNIP---

That space between the ! macro and the first part of the command we want to run is very important. Without it, cmd.exe tries to find a command called !di and bombs. With it, Brian’s macro kicks in and expands it to the most recent command that starts with those letters.

Note that a given set of macros only applies to the shell in which it is created. Your macros aren’t carried to other currently running shells, nor do they even apply to child shells that you spawn. If you exit your shell, they are gone. If you want to make your macros permanent, you should first define them, as we show above and put all those definitions in a file. Then, you can export them into a file, by running:

C:\> doskey /macros > macros.txt

You can call the file anything you want, but I like calling it macros.txt because it’s easy to remember. Then, at any time, you can import these macros by running:

C:\> doskey /macrofile=macros.txt

If you want to apply your macros to every cmd.exe you launch going forward, you can place your macros.txt in a convenient place on your system (such as your common user home directory or even in system32). Then, put a command like the following into any of your autostart locations:

%systemroot%\system32\doskey.exe /macrofile=%systemroot%\system32\macros.txt

For macros, I typically put them in the autostart entry associated with the command shell itself, namely the Autorun Registry key at HKCU\Software\Microsoft\Command Processor. You can define this key by running:

C:\> reg add "hkcu\software\microsoft\command processor" /v Autorun /t reg_sz /d "%systemroot%\system32\doskey.exe /macrofile=%systemroot%\system32\macros.txt"

Be careful! If you already have a command set to autorun via this key, you may want to append the command to your already-existing one by simply inserting an & between the two commands. The reg command will prompt you to confirm or reject the overwrite if you already have something there.

*To get around some (but not all) of the limitations of doskey macros, you could alternatively use small bat files placed in %systemroot%\system32 to kick off familiar commands. For example, you could run:

C:\> echo @wmic process list brief > %systemroot%\system32\ps.bat C:\> ps HandleCount Name Priority ProcessId ThreadCount WorkingSetSize 0 System Idle Process 0 0 1 24576 404 System 8 4 97 774144 ---SNIP---

The advantages of doing your command-line shrinkage with bat files is that you can now run them as little commands themselves, anywhere in your command invocation:

C:\> echo hello & ps hello HandleCount Name Priority ProcessId ThreadCount WorkingSetSize 0 System Idle Process 0 0 1 24576 404 System 8 4 97 745472 ---SNIP---

The downside of this approach is that your bat file must contain whole commands, not just the start of a command, like we can do with macros. That’s why my forever and dve examples above work so well as macros and not as bat files. They are the starter of something else, but not whole commands to themselves, as is the ps and history examples we’ve touched on here.

So, back to Tim’s juicing metaphor. You get to pick your poison with cmd.exe and alias-like behavior. While both methods have some limitations, shell jocks can use macros or bat files to shave off a few precious seconds and boost your performance.

admin | Feb 23, 2010 | Comments 0

Using SED, and such

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed’s Got Sed (well, a little bit of it at least):

In a celebratory mood, I belt out:

C:\> cmd.exe /v:on /c "for /f "delims=" %i in ('echo Hippy barfday spew do you!') do @set stuff=%i& echo !stuff! & set stuff=!stuff:i=a! & echo !stuff! & set stuff=!stuff:arf=irth! & echo !stuff! & set stuff=!stuff:spew=to! & echo !stuff! & set stuff=!stuff: do=! & echo !stuff! & set stuff=!stuff:yo=f! & echo. & echo !stuff!" Hippy barfday spew do you! Happy barfday spew do you! Happy birthday spew do you! Happy birthday to do you! Happy birthday to you!
Happy birthday to fu!

If you couldn’t tell, we’re celebrating the FIRST BIRTHDAY of this blog! Yes, we’ve already made it through one orbit of that big ball of gas at the center of our Solar System, and we’re looking forward to even more. It was one year ago today that three merry command line bandits decided to take a fun little brawl of command-line one-upmanship from Twitter to blog format, so we could get into deeper fu. As a birthday present, my command above shows how we can perform string substitutions at the Windows cmd.exe command line.

I’ve never hid the fact that I’ve often longed for the sed command, which allows for nifty stream editing. Although it’s got a ton of flexible features, one of sed’s most common uses is replacing a string with another string in a stream of data, such as Standard Output. What’s not to like? Well, the fact that we don’t have a built-in equivalent in cmd.exe is one thing that’s a bummer.

So, I got to thinking about this problem the other day, when I realized that I could do the substitution and replacement thing using string altering options in cmd.exe. I could take the data I want to alter, put it in a variable, and then change all occurrences of given strings to other strings using the notation:

set string=%string:original=replacement%

Or, if we use delayed environment variable expansion, we rely on:

set string=!string:original=replacement!

I’ve done just that above, starting by turning on delayed environment variable expansion (cmd.exe /v:on) to execute the command (/c) of FOR /F. My FOR /F command is designed to take the output of my echo command and put it in the variable %i. Alternatively, if I wanted to change text inside of a file, I could have used the “type filename” command instead of echo, iterating through each line of the file making substitutions. I turn off default parsing on spaces and tabs using “delims=”, so that my whole line of data gets shoved into %i. This is all very routine stuff for life inside of cmd.exe, even on your barf^h^h^h^hbirthday.

Then, I move my iterator value into a variable that I can take action on (set stuff=%i). I now can use my string replacement technique to start altering that variable, in a shallow and pale (but useful) mimicking of but one of the features of sed.

I can change individual characters into other characters, such as “i” to “a”:

set stuff=!stuff:i=a!

I can change multi-character substrings like barf into birth:

set !stuff:arf=irth!

I can replace whole words, changing “spew” into “to”:

set stuff=!stuff:spew=to!

I can delete whole words, like ” do”:

set stuff=!stuff: do=!

This one might be worth a note. Here, I’m replacing something with nothing by placing nothing after the equals sign. The item I’m replacing is ” do”, with a space in it. Otherwise, I’d have a double space left behind. And, yes, I could have alternatively replaced “do ” (with a space after it) with nothing. Or, I could have replaced ” do ” with ” ” using !stuff: do = !. There are many options.

And I can even take bigger strings and replace them with smaller strings:

stuff=!stuff:yo=f!

This is really cool and useful, but it does have some limitations. Note that every instance of the substring I specify is replaced, and there really is no means for just changing, for example, the first occurrence of the substring. Also, this doesn’t work for non-printable ASCII characters. You have to be able to type it to get it into that syntax. I’ve also gotta shove everything into a string to make this work, but that’s not so bad.

So, there you have it… a highly obfuscated command for wishing ourselves Happy Birthday.

Whatcha got Hal & Tim?

Tim blows out the candles:

Tim let’s it rip:

PS C:\> "Hippy barfday spew do you!" | Tee-Object -Variable stuff; $stuff -replace "i","a" | Tee-Object -Variable stuff; $stuff -replace "arf","irth" | tee -var stuff; $stuff -replace "spew","to" | tee -va stuff; $stuff -replace " do","" | tee -va stuff; Write-Object; $stuff -replace "yo","f"
Hippy barfday spew do you! Happy barfday spew do you! Happy birthday spew do you! Happy birthday to do you! Happy birthday to you!
Happy birthday to fu!

One orbit around that big ball of gas huh? I’m sure there is joke related to Ed and his love of beans, but we are here to celebrate not disgust.

For those of you who followed the blog since the beginning, the original third bandit was Paul Asadoorian, not me. For those The Three Stooges fans out there, I guess you call me Curly, and Paul would be Shemp. Although, I can’t decide if that would make Hal Moe or Larry. Let’s get back to business and this week’s PowerShell nyuk, nyuk, nyuk.

It is rather fitting that PowerShell is second this week. Cmd’s string replacement is pretty weak and the syntax is terrible, while Linux is the opposite. PowerShell is somewhere in between, but much closer to the linux side of things.

Before we dig into the entire command above, we’ll first do the string substituion without all the extra output.

PS C:\> "Hippy barfday spew do you!" -replace "i","a" -replace "arf","irth" -replace "spew","to" -replace " do","" -replace "yo","f"

Happy birthday to fu!

The Replace operator is used to replace strings, duh! By default, the Replace operator is case insensitive, but to be explicitly case insensitive use the IReplace operator. For a case senstitive replace use the CReplace operator.

Now, let’s do all of Ed’s tricks:

Change individual characters into other characters, such as “i” to “a”:

... -replace "i","a"

Change multi-character substrings like barf into birth:

... -replace "arf","irth"

Replace whole words, changing “spew” into “to”:

... -replace "spew","to"

Delete whole words, like ” do”:

... -replace " do",""

We can even to tricks that Ed can’t, by using regular expressions:
Replace “y” with “f” but only if it is the first character in a word:

PS C:\> "Happy birthday to do yu!" -replace "\sy"," f" Happy birthday to do fu!

Swap the first two words in a line:

PS C:\> "birthday Happy to fu!" -replace "^(\w+)\s(\w+)","`$2 `$1" Happy birthday to fu!

The last command uses regular expression groups. We won’t go into the depths of regex, but in short, “\w+” will grab a word and “\s” will grab a space. The caret (^) is used to anchor the search to the beginning of the string, and the parentheses are used to define the groups. In the replacment portion we use `$1 and `$2 to represent (respectively) the first and second groups (words) found. Since we want to output them in reverse order we use “`$2 `$1″ to put the second word before the first word.

Back to the original command:

We want to display each change as it happens. To pull this off we will have to use the Tee-Object cmdlet. Similar to the linux’s tee command, Tee-Object takes the command output and saves in in a file or variable, as well as sending it down the pipeline or to the console.

If we break it down, this command has three parts that are repeated.

<input object> | Tee-Object -Variable stuff | $stuff -replace <original> <replacment>

We start with the input object “Hippy barfday spew do you!” and pipe it into Tee-Object (alias tee). The only reason we use Tee-Object is so we can display the output and work with it further down the pipeline. After tee, we do the replace. The output of the previous portion becomes the input for the next. Rinse and repeat.

Towards the end of command we throw in the Write-Object cmdlet (alias write, echo) to add the extra line break.

One quick thing to note, when using the Tee-Object cmdlet’s Variable parameter, do not use a $. The parameter accepts a string, which is the name of the variable.

So that a more lucid version of Ed’s highly obfuscated command, and now it is time for Hal to hand out the Birthday spankings.

MoeHal sedsSays

Huh, I was sure Ed was Curly. At Ed’s current rate of hair loss, he’s going to resemble Curly before too much longer.

Hmmm, my Windows colleagues are desperately trying to achieve some of the functionality of sed in their silly little command shells. Here’s a hint guys: Cygwin is your friend. Then you could do things like this:

$ echo 'Hippy Barfday Spew Do You!' | sed 's/$H$i$ppy B$arf$day $Spew D$o $Yo$u!$/\1a\2irth\3T\4F\5/' Happy Birthday To Fu!

The whole trick here is leveraging sub-expressions– the text enclosed in “$…$”– in the first part of the sed substitution expression. Essentially I’m using the sub-expressions to “save” the bits of the line that I want to keep. You’ll notice the bits of our input string that I don’t want are carefully kept outside the “$…$” boundaries.

You can refer to the contents of the sub-expressions on the righthand side of the substitution using the \1, \2, … variables. Sub-expressions are numbered left to right by opening parenthesis– this is an important distinction when you start doing crazy stuff like nested sub-expressions. In this case, however, all I have to do is output the contents of my sub-expressions in order with appropriate text in between them to form our final message.

So really I’m just using sed sub-expressions like a cookie cutter here to chop out the bits of the line I want. This functionality makes sed very useful as a surgical tool for reformatting text into a regular format. Another example of this comes from one of our earliest Episodes where I showed ShempPaul how to bring the sed madness to parse the output of the host command.

Now the problem is that these sed expressions always end up looking awkward because of all of the backwhacks floating around. If you have GNU sed handy, you can use the “-r” (extended regex) option. This allows you do create sub-expressions with “(…)”, saving yourself a lot of backwhack abuse:

$ echo 'Hippy Barfday Spew Do You!' | sed -r 's/(H)i(ppy B)arf(day )Spew D(o )Yo(u!)/\1a\2irth\3T\4F\5/' Happy Birthday To Fu!
Still ugly, but definitely more readable.

Thanks everybody for taking time out of your busy lives to keep up with our humble little blog in the past year. We’ll save you a bit of the birthday cake!

admin | Feb 16, 2010 | Comments 0

Searching Text Strings

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal checks out the mail

We love getting email from readers of the blog. And we love getting cool shell hacks from readers even more. Recently, loyal reader Rahul Sen sent along this tasty little bit of shell fu:

How to search for certain text string in a directory and all its subdirectories, but only in files of type text, ascii, script etc:

$ grep 9898 `find /usr/local/tripwire -type f -print | xargs file | egrep -i 'script|ascii|text' | awk -F":" '{print $1}'` /usr/local/tripwire/te/agent/data/config/agent.properties:tw.local.port=9898 /usr/local/tripwire/te/agent/data/config/agent.properties:tw.server.port=9898

That’s totally cool, Rahul!

Honestly, when I first looked at this I thought, “There’s got to be a shorter way to do this.” But the tricky part is the “only in files of type text, ascii, script, etc” requirement. This basically forces you to do a pass through the entire directory first in order to locate the relevant file types. Thus the complicated pipeline to pass everything through the file command and egrep.

A few minor improvements I might suggest:

1. I’m worried that for a large directory you’ll end up returning enough file names that you exceed the built-in argument list limits in the shell. So it might be better to use xargs again rather than backticks.

2. I probably would have chosen to use sed at the end of the pipeline rather than awk, just to be more terse.

3. You don’t actually need “-print” on modern versions of find– it’s now the default. Only old-timers like me and Rahul end up doing “-print” all the time because we were trained to do so by old versions of find.

So my revised version would look like:

Mmmm-hmmm! That’s some tasty fu! Let’s see what my Windows brethren have cooking…

Ed Unfurls:

This is a helpful little technique. Now, unfortunately at the Windows command line (man, if I only had a dime ever time I said that phrase), we do not have the “file” command to discern the type of file. But, fear not! We do have a couple of alternative methods.

For a first option, we could use a nifty feature of the findstr command to ignore files that have non-printable characters. When run with the /p option, findstr will ignore any files that contain high-end ASCII sequences, letting us skip over EXEs, DLLs, and other stuff. It’s not as fine a grained scalpel as scraping the output of the the Linux file command for script, ascii, and text, but it’ll serve us well as follows:

C:\> findstr /s /p 9898 *

Here, I’m using findstr to recurse the file system (/s) from wherever my current working directory is, skipping files with non-printable characters (/p), looking for the string 9898 in any file (*). If you want to get even closer to the original, we can specify a directory where we want to start the search using the /d: option as follows:

C:\> findstr /d:C:\windows /s /p 9898 *

Now, for our second option, there is another way to refine our search besides the /p option of findstr, getting us a little closer to the file types Rahul specified in Linux using the find command. It turns out that Microsoft actually put an indication of each file’s type in the name of the file itself. You see, by convention, Windows file names have a dot followed by three letters that indicate the file type. Who knew?!?! :)

To map the desired functionality to Windows, we’ll rely on file name suffixes to look inside of .bat, .cmd, .vbs, and .ps1 files (various scripts), .ini files (which often contain config info), and .txt files (which, uh… you know). What’s more, many commands associated with searching files allow us to specify multiple file names, with wild cards, such as the dir command in this example:

C:\> dir *.bat *.cmd *.vbs *.ps1 *.ini *.txt

And, happy to say, dir isn’t the only one that lets us look for multiple file names with wildcards. For my second solution to this challenge, I’m going to use a FOR /R loop. These loops recurse through a directory structure (/R, doncha know) setting an iterator variable to the name of each file that is encountered. Thus, we can use the following command as a rough equivalent to Rahul’s Linux fu:

C:\> FOR /R C:\ %i in (*.bat *.cmd *.vbs *.ps1 *.ini *.txt) do @findstr 9898 "%i" && echo %i

Here, I’m running through all files found under C:\ and its subdirectories, looking inside of any file that has a suffix of .bat, .cmd, etc, running findstr on each file (whose name is stored in %i, which has to be surrounded in quotes for those cases when the value has one or more spaces in the file name) looking for 9898. And, if I successfully find a match, I echo out the file’s name. Now, this output looks a little weird, because the file’s name comes after each line that contains the string. But, that is a more efficient way to do the search. Otherwise, I’d have to introduce unnecessary complexity by using a variable and parsing to store the line of the file and print its name first, then print the contents. I’d certainly do that for prettiness in a script. But, at the command line by itself, I’d eschew the complexity and just go with what I’ve shown above to get the job done.

Now, there’s a gazillion other ways to do this as well. For a third possibility, we could take the first option above (findstr) and use the multiple file suffix specification of option 2 (*.bat *.cmd *.vbs *.ps1 *.ini *.txt) to come up with:

C:\> findstr /d:C:\windows /s 9898 *.bat *.cmd *.vbs *.ps1 *.ini *.txt

I actually like this third approach best, because it’s relatively easy to type, makes a bunch of sense, has nicer-looking output than the FOR /R option, and has better performance.

Fun, fun, fun! Thanks for the great suggestion, Rahul.

Whatcha got for us, Tim?

Tim delivers:

Sadly, PowerShell is missing the same “file” command as the standard Windows command line. Also, there isn’t a PowerShell cmdlet similar to “findstr /p” either, but of course we could use FindStr since all the Windows commands are available in PowerShell. Ed already covered FindStr so we will just use just PowerShell cmdlets.

If we know that the files in question are in a specific directory, not subdirectories, there is a pretty simple command to find the files using the Select-String cmdlet.

PS C:\> Select-String 9898 -Path *.bat,*.cmd,*.vbs,*.ps1,*.ini,*.txt -List | Select Path

Path ---- C:\temp\a.txt

According to the documentation “the Select-String cmdlet searches for text and text patterns in input strings and files. You can use it like Grep in UNIX and Findstr in Windows.” Whoah, big difference there! Grep has much more robust regular expression support when compared to FindStr, and yes, PowerShell does give us rich regular expressions.

Back to the task at hand. We only care if the file contains the text in question, not how many times the text is found in the file. The List parameter is used as a time saver, since it will stop searching after it finds the first match in a file.

For each match, the default console output displays the file name, line number, and all text in the line containing the match. Of course the output is an object and we just want the file’s path, so the results are piped into Select-Object (alias select) in order to get the full file path.

But we want to search though subdirectories too. To do that we have to use Get-ChildItem (alias dir, gci, ls).

PS C:\> Get-ChildItem -Include *.bat,*.cmd,*.vbs,*.ps1,*.ini,*.txt -Recurse | Select-String 9898 -List | Select-Object path

Path ---- C:\temp\subfolder\b.txt C:\temp\a.txt

The Recurse parameter specifies that the search should recursively search through subdirectories. The Include parameter retrieves only the files that match our filter. We could use the Exclude parameter if we wanted to search all files that aren’t exe’s or dll’s.

PS C:\> Get-ChildItem -Exclude *.exe,*.dll -Recurse | Select-String 9898 -List | Select-Object path

The command can be shortened even further since the full parameter name doesn’t have to be used. As long the shortened parameter name isn’t ambiguous a short version of the parameter name can be used. Since there is no other parameters that start with R we or I this command will work as well:

PS C:\> ls -i *.bat,*.cmd,*.vbs,*.ps1,*.ini,*.txt -r | select 9898 -L | select path

There is a catch if you are using version 1 of PowerShell, the Select-String cmdlet natively doesn’t take the input from Get-ChildItem and use it to specify the path. We have to use a For-EachObject loop (alias %) in order to accomplish the same task.

PS C:\> Get-ChildItem -Include *.bat,*.cmd,*.vbs,*.ps1,*.ini,*.txt -Recurse | % { Select-String 9898 -List -Path $_.FullName } | Select-Object path

A side note:
As you probably already know, Windows XP, Vista, and 2003 don’t come with Powershell and require a separate install, but for the love of Pete, install it. Version 2 has been available for quite a while and there are many enhancements over v1 (and even more when compared to cmd). Windows 2008 R2 and Windows 7 come with v2 by default.

PowerShell v1 in Windows 2008 (R1) is an optional feature that needs to be enabled. It can be installed using the Windows command line by running this command:

C:\> ServerManagerCmd.exe -install PowerShell

This is probably the most useful Windows command available (sorry Ed).

admin | Feb 09, 2010 | Comments 0

Time Bandits – Stomping, Stomping, Stomping …

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

I have always wanted to time travel. Since it isn’t possible to go back and kill Hitler I thought, maybe we can go back in time and change some files. Obviously, we don’t have the technology to actually go back in time and make changes. However, what if we could make it appear that we went back in time by altering timestamps.

First, let’s create a few files for our time warp. By the way, this next set of commands are all functionally equivalent.

PS C:\> Write-Output aaaa | Out-File a.txt PS C:\> Write bbbb | Out-File b.txt PS C:\> echo cccc | Out-File c.txt PS C:\> echo dddd > d.txt

Now to see the time related properties available to us for the file object.

PS C:\> Get-ChildItem | Get-Member -MemberType Property | Where-Object { $_.Name -like "*time*" }

TypeName: System.IO.FileInfo

Name MemberType Definition ---- ---------- ---------- CreationTime Property System.DateTime CreationTime {get;set;} CreationTimeUtc Property System.DateTime CreationTimeUtc {get;set;} LastAccessTime Property System.DateTime LastAccessTime {get;set;} LastAccessTimeUtc Property System.DateTime LastAccessTimeUtc {get;set;} LastWriteTime Property System.DateTime LastWriteTime {get;set;} LastWriteTimeUtc Property System.DateTime LastWriteTimeUtc {get;set;}

The Get-Member cmdlet (alias gm) is used to get the properties and methods available for an object that has been sent down the pipeline. In our case, the object sent down the pipeline is the file object. We just want to look at the properties (not methods, scriptproperties, etc) of the object, so we use the MemberType parameter for filtering. Then the Where-Object cmdlet (alias ?) is used to filter for properties with “time” in the name. The properties above are read/write, as shown by {get;set;}. Well, lookey there, we can set the timestamps!

An important side note: The Get-Member cmdlet is an extremely useful command. I can’t begin to say how often I’ve used this command to find properties and methods available for an object. There are other MemberTypes, but we will have to cover those at a later time. For a full description check out the MemberType parameter on this Microsoft help page.

The xxxxxTime and xxxxxTimeUtc properties are actually the same property, the only difference is how they display the date in regards to the UTC (coordinated universal time) offset. In my case, the difference is 6 hours. For the sake of brevity the UTC times will be ignored since they are essentially redundant.

Let’s take a look at our files.

PS C:\> gci | select Name, LastWriteTime, CreationTime, LastAccessTime

Name LastWriteTime CreationTime LastAccessTime ---- ------------- ------------ -------------- a.txt 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM b.txt 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM c.txt 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM d.txt 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM 1/23/2010 11:16:24 AM

Now let’s go back in time.

(gci a.txt).LastAccessTime = Get-Date ("1/1/2010") (gci b.txt).CreationTime = Get-Date ("1/1/2010") (gci c.txt).LastWriteTime = Get-Date ("1/1/2010")

Since there isn’t a cmdlet for setting the time properties, we need to access properties in a different manner. To do this, we get the object and use the dot notation to access the property. The Get-Date cmdlet creates a valid date/time object for our new timestamp. Let’s see how that worked.

PS C:\> gci | select Name, LastWriteTime, CreationTime, LastAccessTime

Name LastWriteTime CreationTime LastAccessTime ---- ------------- ------------ -------------- a.txt 1/23/2010 11:16:24 AM 1/23/2010 12:07:05 PM 1/1/2010 12:00:00 AM b.txt 1/23/2010 11:16:24 AM 1/1/2010 12:00:00 AM 1/23/2010 12:07:05 PM c.txt 1/1/2010 12:00:00 AM 1/23/2010 12:07:05 PM 1/23/2010 12:07:05 PM d.txt 1/23/2010 11:16:24 AM 1/23/2010 12:07:05 PM 1/23/2010 12:07:05 PM

Interesting, the times have been changed, but can we forensically find a difference? After taking an image of the drive and using the istat tool from the SluethKit.org guys it is very obvious that something weird has happened. Let’s take a look at an istat output snippet to see where the problem lies.

$STANDARD_INFORMATION Attribute Values: Flags: Archive Owner ID: 0 Security ID: 408 () Created: Fri Jan 01 00:00:00 2010 File Modified: Fri Jan 23 11:16:24 2010 MFT Modified: Fri Jan 23 11:20:43 2010 Accessed: Fri Jan 23 11:16:24 2010

$FILE_NAME Attribute Values: Flags: Archive Name: a.txt Parent MFT Entry: 9946 Sequence: 10 Allocated Size: 0 Actual Size: 0 Created: Fri Jan 23 11:16:24 2010 File Modified: Fri Jan 23 11:16:24 2010 MFT Modified: Fri Jan 23 11:16:24 2010 Accessed: Fri Jan 23 11:16:24 2010

The PowerShell commands modify the STANDARD_INFO’s created, file modified and accessed times. As you can see, there is a discrepancy in the Creation times between the FILE_NAME and STANDARD_INFO attribute values. Also, if you look at the STANDARD_INFO’s MFT Modified date you can take a good guess as to when this change was made. The MFT Modified stamp is updated to the current system time whenever any of our PowerShell commands make a change to the file. The MFT Modified timestamp only marks the last change, so we could change all the dates to make it more confusing as to what change happened at that time.

While making the changes to the timestamps in PowerShell is effective when looking at the file system via the GUI or command line, not everything is hidden when looking at it with forensic tools.

While we can’t go back in time and bump off Hitler, we did go back in time and take this functionality out of the standard Windows command line. OK, that’s not really true. But what is true is that there is no way in cmd.exe to manipulate time stamps. So Ed won’t be joining us with any cmd fu this week.

Hal’s palms are getting hairy…:

Changing atimes and mtimes on Unix files is easy because we have the touch command. If you simply “touch somefile”, then the atime and the mtime on that file will be updated to the current date and time (assuming you’re the file owner or root).

But if you’re root, you can also specify an arbitrary time stamp using the “-t” flag:

# touch -t 200901010000 /tmp/test # alltimes /tmp/test /tmp/test atime: Thu Jan 1 00:00:00 2009 mtime: Thu Jan 1 00:00:00 2009 ctime: Sun Jan 24 05:33:56 2010

The ctime value is always set to the current date and time, because the touch command is updating the atime and mtime meta-data in the inode and ctime tracks the last meta-data update. By the way, don’t bother looking for the alltimes command in your Unix OS. It’s a little Perl script I wrote just for this Episode (download the script here).

The touch command also has “-a” and “-m” flags that allow you to selectively update only the atime or the mtime:

# touch -a -t 200909090909 /tmp/test # touch -m -t 201010101010 /tmp/test # alltimes /tmp/test /tmp/test atime: Wed Sep 9 09:09:00 2009 mtime: Sun Oct 10 10:10:00 2010 ctime: Sun Jan 24 05:49:29 2010

As you can see from the above example, touch is perfectly willing to set timestamps into the future as well as the past.

OK, so what about tweaking the ctime value? In general, setting the ctime on a file to an arbitrary value requires specialized, file system dependent tools. The good news(?) is that for Linux EXT file systems, the debugfs command will let us muck with inode meta-data. If you’re dealing with other file system types or other operating systems, however, all I can say is good luck with your Google searching.

debugfs has a huge number of options that we don’t have time to get into here. I’m just going to show you how to use set_inode_field to update the ctime value:

# debugfs -w -R 'set_inode_field /tmp/test ctime 200901010101' /dev/mapper/elk-root debugfs 1.41.9 (22-Aug-2009)

The “-w” option specifies that the file system should be opened read-write so that we can actually make changes– by default debugfs will open the file system in read-only mode for safety. We also need to specify the file system we want to open as the last argument. Sometimes this will be a disk partition device name like “/dev/sda1″, but in my case I’m using LVM, so my disk devices have the “/dev/mapper/” prefix. If you’re not sure what device name to use you can always run a command like “df -h /tmp/test” and look for the device name in the first column.

The “-R” option can be used to specify a single debugfs command to run in non-interactive mode. Note that there’s also a “-f” option that allows you to specify a file of commands you want to run. If you leave off both “-R” and “-f” you’ll end up in an interactive mode where you can run different commands at will.

In this case, however, we’re going to use “-R” and run set_inode_field to set the ctime on /tmp/test. As you can see, you use a time stamp specification that’s very similar to the one used by the touch command. And speaking of touch, we could use debugfs to “set_inode_field … atime …” or “set_inode_field … mtime …” instead of touch if we wanted to. This would allow us to update the atime/mtime values for a file without updating the ctime like touch does.

Anyway, now our ctime value should be updated, right? Let’s check:

# alltimes /tmp/test /tmp/test atime: Wed Sep 9 09:09:00 2009 mtime: Sun Oct 10 10:10:00 2010 ctime: Sun Jan 24 05:49:29 2010

That doesn’t look right! What’s going on?

What’s happening is that we’ve run afoul of the Linux disk cache. We actually have updated the information in the inode, but we’ve done it in such a way as to do an end-run around the normal Linux disk access routines, so our changes are not reflected in the in-memory file system cache. The good news is that (at least as of Linux kernel 2.6.16) there’s a simple way to flush the inode cache:

# echo 2 > /proc/sys/vm/drop_caches # alltimes /tmp/test /tmp/test atime: Wed Sep 9 09:09:00 2009 mtime: Sun Oct 10 10:10:00 2010 ctime: Thu Jan 1 01:01:00 2009

That looks better!

By the way, if you “echo 1 > /proc/sys/vm/drop_caches”, that flushes the page cache. If you “echo 3 > /proc/sys/vm/drop_caches” it flushes both the page cache and the inode/dentry cache.

admin | Feb 02, 2010 | Comments 0

Your 5-Step Malware-Analysis Toolkit

From http://www.campustechnology.com By Lenny Zeltser

A LARGE NUMBER of computer intrusions involve some form of malicious software (malware), which finds its way to the victim’s workstation or to a server. When investigating the incident, the IT responder typically seeks to answer questions such as: What actions can the malware specimen perform on the system? How does it spread? How, if at all, does it maintain contact with the attacker? These questions can all be answered by analyzing the offending malware in a controlled environment.

A simple analysis toolkit, built from free and readily available software, can help you and your IT team develop the skills critical to responding to today’s security incidents. The steps below will help get you started. We’ll focus on malware analysis in a Windows environment, since that platform is particularly popular among malware authors.

Step 1: Allocate physical or virtual systems for the analysis lab

A common approach to examining malicious software involves infecting a system with the malware specimen and then using the appropriate monitoring tools to observe how it behaves. This requires a laboratory system you can infect without affecting your production environment.

The most popular and flexible way to set up such a lab system involves virtualization software, which allows you to use a single physical computer for hosting multiple virtual systems, each running a potentially different operating system. Free virtualization software options include:

Running multiple virtual systems simultaneously on a single physical computer is useful for analyzing malware that seeks to interact with other systems, perhaps for leaking data, obtaining instructions from the attacker, or upgrading itself. Virtualization makes it easy to set up and use such systems without procuring numerous physical boxes.

Another useful feature of many virtualization tools is the ability to take instantaneous snapshots of the laboratory system. This way, you can record the state of the system before you infect it, and revert to the pristine environment with a click of a button at the end of your analysis.

If using virtualization software, install as much RAM into the physical system as you can, as the availability of memory is arguably the most important performance factor for virtualization tools. In addition, having a large hard drive will allow you to host many virtual machines, whose virtual file systems typically are stored as files on the physical system’s hard drive.

Take precautions to isolate the malware-analysis lab from the production network, to mitigate the risk that a malicious program will escape.

Because malware may detect that it’s running in a virtualized environment, some analysts prefer to rely on physical, rather than virtual, machines for implementing laboratory systems. Your old and unused PCs or servers can make excellent systems for your malware-analysis lab, which usually doesn’t need high-performing CPUs or highly redundant hardware components.

To allow malware to reach its full potential in the lab, laboratory systems typically are networked with each other. This helps you observe the malicious program’s network interactions. If using physical systems, you can connect them with each other using an inexpensive hub or a switch.

Step 2: Isolate laboratory systems from the production environment

You must take precautions to isolate the malware-analysis lab from the production network, to mitigate the risk that a malicious program will escape. You can separate the laboratory network from production using a firewall. Better yet, don’t connect laboratory and production networks at all, to avoid firewall configuration issues that might allow malware to bypass filtering restrictions.

If your laboratory network is strongly isolated, you can use removable media to bring tools and malware into the lab. It’s best to use write-once media, such as CDs, to prevent malicious software from escaping the lab’s confines by writing itself to a USB key. If using a USB key, which is more convenient than a CD, get a model that includes a physical write-protect switch.

Some malware-analysis scenarios benefit from the lab being connected to the internet. Avoid using the production network for such connectivity. If possible, provision a separate, and usually inexpensive, internet connection, perhaps by dedicating a DSL line to this purpose. Avoid keeping the lab connected to the internet all the time to minimize the chance of malware in your lab attacking someone else’s system on the internet.

If virtualizing your lab, be sure to keep up with security patches released by the virtualization-software vendor. Such software may have vulnerabilities that could allow malware to escape from the virtual system you infected and onto the physical host. Furthermore, don’t use the physical machine that’s hosting your virtualized lab for any other purpose.

Step 3: Install behavioral analysis tools

Before you’re ready to infect your laboratory system with the malware specimen, you need to install and activate the appropriate monitoring tools. Free utilities that will let you observe how Windows malware interacts with its environment include:

File system and registry monitoring: Process Monitor and Capture BAT offer a powerful way to observe in
real time how local processes read, write, or delete
registry entries and files. These tools can help you
understand how malware attempts to embed into the
system upon infection.
Process monitoring: Process Explorer and Process Hacker replace the built-in Windows Task Manager, helping
you observe malicious processes, including local network
ports they may attempt to open.
Network monitoring:Wireshark and SmartSniff are
network sniffers, which can observe laboratory network
traffic for malicious communication attempts, such as
DNS resolution requests, bot traffic, or downloads.
Change detection: Regshot is a lightweight tool for comparing the system’s
state before and after the infection, to highlight
the key changes malware made to the file system and
the registry.

Behavioral monitoring tools can give you a sense for the key capabilities of malicious software. For further details about its characteristics, you may need to roll up your sleeves and perform some code analysis.

Step 4: Install code-analysis tools

Examining the code that comprises the specimen helps uncover characteristics that may be difficult to obtain through behavioral analysis. In the case of a malicious executable, you rarely will have the luxury of access to the source code from which it was created. Fortunately, the following free tools can help you reverse compiled Windows executables:

Disassembler and debugger: OllyDbg and IDA Pro Freeware can parse compiled Windows
executables and, acting as disassemblers, display their
code as Intel x86 assembly instructions. These tools
also have debugging capabilities, which allow you to
execute the most interesting parts of the malicious program
slowly and under highly controlled conditions, so
you can better understand the purpose of the code.
Memory dumper: LordPE and OllyDump help obtain protected code located in the
lab system’s memory and dump it to a file. This technique
is particularly useful when analyzing packed executables,
which are difficult to disassemble because
they encode or encrypt their instructions, extracting
them into RAM only during run-time.

Step 5: Utilize online analysis tools

To round off your malware-analysis toolkit, add to it some freely available online tools that may assist with the reverse engineering process. One category of such tools performs automated behavioral analysis of the executables you supply. These applications look similar at first glance, but use different technologies on the back end. Consider submitting your malware specimen to several of these sites; depending on the specimen, some sites will be more effective than others. Such tools include:

Another set of potentially useful online tools provides details about websites that are suspected of hosting malicious code. Some of these tools examine the sites you specify in real time; others provide historical information. Consider submitting a suspicious URL to several of these sites, because each may offer a slightly different perspective on the website in question:

Real-time threat assessment: Finjan URL Analysis, McAfee Site
Advisor, and Wepawet
Historical reputation data: Norton Safe Web
and WOT (Web of Trust)

Next Steps

With your initial toolkit assembled, start experimenting in the lab with malware you come across on the web, in your e-mail box, on your systems, and so on. You may find this one-page cheat sheet convenient.

Begin analysis with the tools and approaches most familiar to you. Then, as you become more familiar with the inner workings of the malware specimen, venture out of your comfort zone to try other tools and techniques. The tools I’ve listed within each step operate virtually identically. Since they’re all free, you should feel free to try them all. You’ll find that one tool will work better than another, depending on the situation. And with time, patience, and practice, you will learn to turn malware inside out.

admin | Jan 19, 2010 | Comments 0

Advanced Process Whack-a-Mole

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed Prepares to Open Up a Can of Process Whoop-Ass:

I’ve never considered myself a particularly violent man. But, I have to admit it: Sometimes it just feels good to kill processes. I’ve even been heard to mutter a deadpan “Dodge This” in my lab late at night as I obliterate errant or evil processes just begging to meet their maker. Then, to make sure such a process doesn’t pop back up to start bothering me again, I sow the ground with command-line kung fu salt to strangle any other similar process that might pop up in its place.

This technique, which we’ve taken to calling “Process Whack-a-Mole”, can be helpful to people in all walks of life. I’m sure it’s happened to pretty much everyone at some point. You find yourself playing defense in a Capture the Flag tournament against an elite team of ninjas from a three-letter government agency who want to completely control your boxen. They repeatedly gain access, and you have to shew them out before they score and you lose points. To deal with such situations, we can run a command to continuously look for processes with certain characteristics of our adversaries, and then kill them when they appear. We touched upon the idea of killing a process that starts to listen on a given TCP port in Episode #76. But, let’s go further this time, discussing how you can make much more flexible whack-a-mole commands to deal with various process characteristics.

These techniques are useful even outside of Capture the Flag games. I often use them in malware analysis and even system administration when I want to suppress some activity temporarily while I’m analyzing or configuring something else.

The basic structure I use for process whack-a-mole consists of the following three parts:

<Continuous Loopinator> <Process Selector> <Process Terminator>

Quite often the Process Selector and Process Terminator are combined together in a single command, because we can select or filter for the process we want in the same command we use to whack it. However, to filter for certain specific process characteristics, we’ll have to split out these two entities. I’ll show you what I mean in a bit.

We start out with our Continuous Loopinator:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul

This is a simple FOR /L loop that starts counting at 1, goes up to 2, in steps of 0. In other words, it’s the cmd.exe equivalent of while (true), used to keep something running continuously. At the start of the loop, we introduce a 1-second delay by pinging ourselves twice (-n 2) and throwing the standard output away so as not to clutter our output (>nul). That way, we’ll run our Process Selector and Process Terminator approximately every 1 second, helping to minimize our impact on performance. If you want a faster Process Selector, simply omit that ping, and your system will run our whack-a-mole command as fast as it can, but performance may drag.

We then follow with our Process Selector. If you keep in the ping delay, put in an & followed by the Process Selector, which lets us make one command run after another. Otherwise, just put the Process Selector after the @ (which turns off command echo, by the way… no sense having our output clogged up with commands).

The two most common Process Selectors I use are wmic and taskkill, which have the nice property of also including the ability to act as Process Terminators in the same command. Let’s look at wmic first.

The wmic command can be used to select given processes based on our constructing a where clause, using the following syntax:

C:\> wmic process <where clause> <verb clause>

In the where clause, we can specify any attribute or group of attributes of processes that can be listed via wmic process. To get a list of these attributes, you could run:

C:\> wmic process get /?

So, for example, if you want to select a processID of 4242, you could write your wmic command as:

C:\> wmic process where processid=4242

Or, we could look for processes that have a given Parent Process ID:

C:\> wmic process where parentprocessid=3788

Or, we could look for processes with a given name:

C:\> wmic process where name="cmd.exe"

These where clauses also support AND and OR, but you’ve got to make sure you put the guts of your where clause inside of parentheses. The where clauses also support not equals (!=). Check this out:

C:\> wmic process where (name="cmd.exe" and processid != 676)

Now, we haven’t supplied a verb clause here, so all our wmic commands are simply displaying raw, unformatted process information on Standard Output.

Let’s start doing our whack-a-mole by specifying a Process Terminator by using the verb clause of wmic with the simple verb “delete”. That’ll kill a process.

Putting these pieces together, suppose you want to kill all cmd.exe processes other than the current cmd.exe you, the administrator, are running. Let’s assume that your own cmd.exe has a process ID of 676. You could run:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & wmic process where (name= "cmd.exe" and processid!=676) delete

Now, let’s see ‘em try to run a cmd.exe. Every time someone tries to launch one, your loop will kill it.

Next, suppose you want to prevent a given process with processid 4008 from spawning child processes. Maybe process ID 4008 is a cmd.exe shell, and you want to prevent the person who is using it from being able to run any commands that aren’t built-into the shell itself. Or, better yet, maybe process ID 4008 is Tim Medin’s PowerShell process, and you wanted to pee in his Corn Flakes, depriving him of the ability to run any separate EXE commands, forcing him to rely solely on built-in capabilities of Powershell itself. We can do this one without our ping-induced delay to really confound him:

C:\> for /L %i in (1,0,2) do @wmic process where parentprocessid=4008 delete

These wmic where clauses also support substring matches, with the use of “like” and %. For example, suppose you want to continuously kill every process that is running from an EXE with a path in a given user’s directory. You could run:

c:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & wmic process where (executablepath like "c:\\users\\tim\\%") delete

Note that in a “where” clause with the “like” syntax, you have to surround the elements with parens. Also, note that if you have a \ in your where clause, you have to specify it as \\, the first \ indicating an escape, and the second indicating your backslash.

You can combine these where clause elements (=, !=, AND, OR, LIKE, and %) in all kinds of ways to mix and match against various process attributes for whack-a-mole. I’m sure our readers can dream up all kinds of interesting and useful combinations.

But, there is a missing attribute from “wmic process get /?” — it’s the user name that process is running under. To play whack-a-mole based on user names, we can turn to another Process Selector and Terminator: taskkill. I wrote about taskkill filters back in Episode 22, showing how we can use it to kill a process based on its owner username. Here, we’ll wrap that in our whack-a-mole construct:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & taskkill /F /FI "username eq tim"

Sorry, Tim. That’s what you get for using a shell on the same system I’m on. Hal’s over on some lonely Linux that no one ever uses, so I leave him alone. :)

Anyway, where was I? Ah, yes, we were discussing attributes of processes that wmic doesn’t include, but which may be handy in whack-a-mole games. How about DLLs? Suppose a bad guy is hacking your system and keeps trying to inject a DLL into some process, and you want to kill that process. Maybe Evil Badguy (yes, that’s his full name) has injected metsrv.dll, the Metasploit Meterpreter, into a running process, and uses process migration to jump from process to process. Sorry, but getting the system to unload that DLL using only built-in tools at the command line is very difficult, but killing that process is totally doable:

C:\> for /L %i in (1,0,2) do @ping -n 2 127.0.0.1 >nul & taskkill /F /FI "modules eq metsrv.dll"

Now, I mentioned above that the Process Selector and Process Terminator components are typically combined in a single command, as we’ve seen so far with wmic and tasklist. When would you have two different commands for these pieces? Well, one example is with netstat, which can show TCP and UDP port usage and the processes associated with each port. That’s exactly what we used in Episode #76, where our Process Selector was netstat (whose output I parsed with a FOR /F loop to pull out the ProcessID number), and the Process Terminator was taskkill:

C:\> for /L %i in (1,0,2) do @(ping -n 2 127.0.0.1>nul & for /f "tokens=5" %j in ('netstat -nao ^| find ^":47145^"') do @taskkill /f /PID %j)

So, keeping in mind those three components of process whack-a-mole, you can use almost any command that lists processes in pretty much any arbitrary way to build a whack-a-mole command for fun and profit.

And now, for a routine disclaimer: Be careful with any of these commands. If you kill a vital system process, such as lsass.exe, you could bring your whole box down. You have been warned. So, now that you are armed and dangerous, go have fun!

Tim prepares for war:

It seems that Ed has a bit of shell envy. So let’s kick that inferior shell off “our” machine and keep it (and him) off our machine.

As Ed described, the structure for whack-an-Ed whack-a-mole has three parts, and that basic structure will be very similar in PowerShell.

<Continuous Loopinator> { <Process Selector> | <Process Terminator> }

The Continuous Loopinator repeatedly calls the Process Selector whose results are piped into the Process Terminator. Let’s see how each piece works.

Continuous Loopinator:

There are many ways to do a continuous loop, but the easiest and quickest method is to use the While loop.

PS C:\> while (1) { <code to run ad infinitum>; Start-Sleep 1 }

This loop is pretty self explanatory. It is a simple While loop that runs while 1 is true, which it always will be. The Start-Sleep cmdlet (alias sleep) will suspend activity for the specified amount of time. If we wanted a shorter nap we could use the -milliseconds parameter. Since Ed’s command runs every second, we should run ours a bit faster just because we can. How about 5 times a second?

PS C:\> while (1) { <code to run ad infinitum>; Start-Sleep -milliseconds 200 }

Process Terminator:

I’m covering this a bit out of order because the Terminator is so simple, so indulge me for a bit. The cmdlet used for killing is Stop-Process (alias spps or kill). It can even be used for some rudimentary process selection before the assassination. We can kill based on the Process Id:

PS C:\> Stop-Process 1337

…or the process name.

PS C:\> Stop-Process -Name cmd

In the second example every process with the name “cmd” would be stopped, but what if we wanted to be a little more high tech in making Ed’s processes “sleep with the fishes?”

As described earlier, the results of the Process Selector can be piped into our Process Terminator. We can pick any method to retrieve the process(es) to be killed, but more on that later. Here is what it would look like:

<Get Processes> | Stop-Process

By default, Stop-Process will ask for confirmation prior to terminating any process not owned by the current user. To get around that safety mechanism we can use the Force parameter.

<Get Processes> | Stop-Process -Force

Short version:

<Get Processes> | kill -f

We could just kill the processes with Stop-Process by giving it a Process Id or process name, but we want more options. Now let’s see how we can find more processes to kill.

Process Selector:

To get a process or a number of processes we use Get-Process (aliases ps and gps). This is our Process Selector. We have covered this before, but we have a number of ways to get a process or a list of processes. To get help on the command you can run:

PS C:\> Get-Help Get-Process

…or for those who have seen the light and come from the linux side but have a bad memory, this works too:

PS C:\> man ps

To see the examples use the Examples parameter, or use the Full parameter to see everything. From the help we can see how to get a process with a given Process ID. We will be looking for PID 4242:

PS C:\> Get-Process -Id 4242 PS C:\> ps -Id 4242

To get all the cmd.exe processes:

PS C:\> Get-Process cmd PS C:\> ps cmd

Note that the process name does NOT include .exe.

We can also use filters in order to get more granular. We already had our loop to kill all cmd processes, but what if Ed wants to use PowerShell? We need to make sure that we are King of the PowerShell Hill, and any other PowerShell usurper is destroyed. This will find any PowerShell processes that aren’t ours.

PS C:\> Get-Process powershell | ? { $_.ID -ne 21876 }

The weaponized version of the command could look like this:

PS C:\> While (1) { ps powershell | ? { $_.ID -ne 21876 } | kill -f; sleep 1 }

The next thing Ed did, after tee-tee’ing in my Kelloggs, was to prevent me from kicking off any processes from a cmd or PowerShell process. So let’s do the same to him. Unfortunately, the objects returned by Get-Process do not have a Parent Process Id property, so we will have to use WMI to find processes with a given parent.

PS C:\> Get-WmiObject win32_process -Filter "ParentProcessId=5552" | % { Get-Process -Id $_.ProcessID }

Get-WmiObject (alias gwmi) is used to access WMI in order to get all processes with a Parent Process Id of 5552. The results are piped into a ForEach-Object (alias %) loop. In the loop we use Get-Process and the Process Id retrieved from WMI in order to get the process object. We can then pipe that into our kill(er). Similar to what Ed did, we want to run this continuously so he doesn’t have a chance. Here is what our command looks like:

PS C:\> While (1) { gwmi win32_process -Filter "ParentProcessId=5552" | % { ps -Id $_.ProcessID } | kill -f }

We also want to make sure that Ed isn’t able to run anything from his user directory (which includes his desktop).

PS C:\> While (1) { ps | ? { $_.Path -like "c:\users\ed\*" } | kill -f }

We use the Where-Object (alias ?) to filter our list of processes based on the path. The Like operator is used with our wildcard search string in order to find any of Ed’s processes. Again, we pipe the results into Stop-Process in order to kill it.

Just to make sure that Ed doesn’t run anything, we will kill any process where he is the owner. Again, we will have to use WMI in order to find the owner of a process.

PS C:\> While (1) { Get-WmiObject Win32_Process | ? { $_.GetOwner().User -eq "ed" } | % { Get-Process -Id $_.ProcessId } | Stop-Process -Force }

This command is a little complicated, so let’s break it down piece by piece. The While loop portion should be obvious so we’ll skip that bit of explanation. The first chunk…

Get-WmiObject Win32_Process | ? { $_.GetOwner().User -eq "ed" }

We start off by querying WMI and retrieving WMI objects representing each process running on the current machine. The results are then piped into our filter, Where-Object (alias ?). The “current pipeline object”, represented by the variable $_, allows us to access the properties of each object passed down the pipeline. For all intents and purposes, the $_ variable is used to iterate through each object passed down the pipeline. It takes the first object, in our case the first process, and accesses the GetOwner method’s User property. We then check to see if the value is equal (-eq) to “ed”. If it is equal, then our WMI object passes our filter and is sent further down the pipeline. Remember, the objects are WMI Process Objects, not PowerShell Process Objects, and they will need to be converted to the PowerShell version so we can natively deal with them in PowerShell. On to the conversion.

... | % { Get-Process -Id $_.ProcessId } ...

The objects that passed through the filter are now sent into our ForEach-Object (alias %) loop. This loop is used to iterate through each object and execute some fu on each of the WMI objects. Again, $_ represents the current object. To retrieve the PowerShell version of the process object we use Get-Process. We need to use Id parameter with the Process Id property of the current object ($_.ProcessId). Now we have PowerShell Process Objects. YAY!

... | Stop-Process -Force

Finally, the processes are piped into Stop-Process to be destroyed. The Force option is used since we don’t want a confirmation to kill each process.

Next, let’s look for the processes with the injected Meterpreter dll. How do we find this dll? We need to look at the modules a process has loaded. Here is what the Modules property looks like for the PowerShell process.

PS C:\> Get-Process powershell | select modules

Modules : {System.Diagnostics.ProcessModule (powershell.exe), System.Diagnostic s.ProcessModule (ntdll.dll), System.Diagnostics.ProcessModule (kernel 32.dll), System.Diagnostics.ProcessModule (KERNELBASE.dll)...}

As you can see the dll name is wrapped in parenthesis. So here is how we kill it.

PS C:\> Get-Process | ? { $_.Modules -like "*(metsrv.dll)*" } | Stop-Process

Actually, the modules property is a collection of module objects. So we can use a nested Where-Object to filter.

PS C:\> ps | ? { $_.Modules | ? {$_.ModuleName -eq "metsrv.dll" }} | kill

In this command we retrieve all the processes. We then filter the Modules, where the ModuleName is metsrv.dll. The results are piped into Stop-Process.

We can also parse netstat in order to kill a process similar to Episode #76. Let’s take that command and wrap it in our infinite loop.

PS C:\> While (1) { netstat -ano | ? {$_ -like "*:47145 *"} | % { $_ -match "\d+$"; stop-process $matches[0] } }

And as Ed said, be careful not to kill the wrong process or the whole box could go down. Of course, when it is down it is pretty dang hard to attack. Of course, it is also pretty dang hard to use too.

Now that Ed and I have spent all of our energy going after each other, Hal is going to show up and mop the floor with our tired carcases.

Disclaimer: No Eds where harmed in the making of this episode.

Hal’s Analysis:

Why are Ed and Tim so angry all the time? It couldn’t have anything to do with the platform they’ve chosen to work on, could it? Hey guys, don’t worry, be happy! You can always install Linux for free, or even just use Cygwin.

When Ed first proposed this topic, I was pretty stoked because I thought it was going to be a cake-walk for me with my little friend lsof. But not all of Ed’s challenges that could be answered purely with lsof. Some required a bit more shell fu.

Let’s start with the simple stuff first. The “infinite loop with 1 second delay” idiom for bash is something we’ve seen before in previous Episodes:

# while :; do [...your commands here...]; sleep 1; done

In this case, the commands we put into the while loop are going to be a kill command and usually some variant of “`lsof -t …`” we’ll be using to select the PIDs we want to kill. Remember from previous Episodes that “lsof -t” causes lsof to print out just the PIDs of the matching processes, specifically so we can use the output as arguments to the kill command.

For example, let’s suppose we want to kill all of Ed’s processes. We can use lsof’s “-u” option to select processes for a particular user:

# while :; do kill -9 `lsof -t -u skodo`; sleep 1; done

Or we could nuke all the bash shells on the machine, using “-c” to select commands by name:

# while :; do kill -9 `lsof -t -c bash`; sleep 1; done

Of course, this would hammer our own shell, so it pays to be more selective:

# while :; do kill -9 `lsof -t -a -c bash -u^root -u^hal`; sleep 1; done

Here I’ve added the “-a” flag which means do a logical “and” on my selection criteria. Those criteria are “all commands named bash” (“-c bash”) and “not user root” (“-u^root”) and “not user hal” (“-u^hal”). Note that lsof’s negation operator (“^”) only works when selecting user names, PIDs (with “-p”), process group IDs (with “-g”), command names (with “-c”), and protocol state info (“-s”, as in “-s^TCP:LISTEN”).

Another one of Ed’s challenges was killing processes where the binary is in a particular directory. Again we can do this with lsof:

# while :; do kill -9 `lsof -t -a -d txt +d /home/skodo`; sleep 1; done

Here we’re looking for process binaries using “-d txt”. In the lingo, the binary is what’s used to create the “text segment” of a process (where the executable code lives), hence “-d txt” for lsof. The “+d” tells lsof to look for open files under a particular directory. Yes, lsof has so many command line options that the author had to start doubling up on letters using “+” instead of “-” (there’s a reason the lsof manual page is nearly 50 pages long when printed out).

Note that “+d” only searches “one level deep”. So if Ed were running “/home/skodo/evil”, then our loop above would whack that process. But if Ed were running “/home/skodo/bin/evil”, then we wouldn’t catch it. If you want to do full directory recursion, use “+D” instead of “+d”. lsof distinguishes these with separate options because full directory searches are so time-consuming.

However, as I mentioned earlier, Ed had challenges that I wasn’t able to come up with a “pure lsof” solution for. For example, while lsof has the “-R” option for displaying parent process ID (PPID) values, there aren’t any switches in lsof to select particular processes by PPID. So we’ll need to resort to some awk:

# while :; do kill -9 `lsof -R -d cwd | awk '($3 == 8552) { print $2 }'`; sleep 1; done

Here the lsof command is outputting PPID values (“-R”) in addition to the normal lsof output, and we’re only outputting the lines showing the current working directory of each process (“-d cwd”). The “-d cwd” hack is a good way of ensuring that you only get one line of lsof output per process– so we don’t end up outputting the same PID multiple times and generating spurious error messages from kill. The awk code simply matches against a particular PPID value in column #3 and outputs the PID value in column #2.

Even though I had to resort to a bit of awk in the last example, you have to admit that having lsof makes this challenge unfairly easy for us Unix/Linux folks. Ahhh, lsof! How I love thee! Let me count the ways…

admin | Jan 19, 2010 | Comments 0

Say Hello to My Little Friend Netstat

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed greets 2010:

Many times on our little blog here, I’ve spoken appreciatively of some of my favorite Windows command-line friends, including wmic, netsh, and sc. Yes, I’ve fought the desire to personify them, but I can’t help but think of them as buddies… Buddies with annoying quirks, a bad attitude, and flaws that sometimes let you down, but compadres nonetheless.

While spending some time over the holidays hanging out with these trustworthy sidekicks, something occurred to me: We haven’t spent nearly enough time on this blog with the good ol’ netstat command. Sure, we touched upon him a little bit in analyzing protocol statistics, checking out whether we’ve got a SYN flood, and even as an example of ways to get help from Windows commands. But, so far, I’ve truly let one of my most faithful and useful commands down by not giving him the spotlight he deserves. So, without further adieu, let’s look at some really useful invocations of netstat:

Detecting when a scan reaches a given target box: So, Mike Poor and I were at a client’s facilities conducting a large-scale vulnerability scan of thousands of hosts. Mike was running the scanning tool from the corporate headquarters, while I was located at another building sitting with the client in front of one of their Windows servers that was included in Mike’s scan. The client and I wanted to know when Mike’s scan would reach us. I asked the client if I could install a sniffer on their server, or connect my laptop to a span port on the switch so I could use my own sniffer to see when Mike’s scan reached us. “No dice,” said the client, who wasn’t authorized to let us install any software or get access to the switch. I asked the client if we could logon to the Windows server and run a simple single netstat command that would show us when Mike’s scan reached us, with approximately 1 second accuracy. He said, “Sure thing,” so I ran:

C:\> netstat -na | find /i "Listening"

Here, I’m just checking to make sure there is at least one listening TCP port, piping netstat’s output through the find command in a case-insensitive fashion (/i) because I didn’t want to hold down the shift-lock key to type LISTENING. Sure enough, on this Windows server, I saw several listening ports, including TCP 445. When Mike’s scanner reached us, it would open a connection to that port.

Then, I ran:

C:\> netstat -na 1 | find "[MikePoorIPaddr]"

That little space followed by a 1 means that Netstat will re-run approximately every 1 second. I scrape through its output looking for the string of Mike’s address. If I see no output, there is no connection from Mike. When I start seeing output, I’ll know that Mike’s scan has reached me. Sure, this isn’t perfect. It’s quite possible that the scanning tool will make and drop connections so quickly that they will fall in between the 1-second interval we’re working with here. However, that’s quite unlikely. And, sure enough, we were able to determine when Mike’s scan reached us, and finished with us. We then headed for lunch.

Detecting when a piece of malware starts interacting with the network: I was in my lab, fully clothed, analyzing some malware. After it finished some initialization actions, this little gem would start listening on TCP port 47145. Again, I wanted to know when it would start to listen, with 1-second accuracy… I ran:

C:\> netstat -na 1 | find ":47145"

I then wanted to go a little further, and find when the bad guy actually made a connection to this listener. For that, I executed:

C:\> netstat -na 1 | find ":47145" | find /i "Established"

Finding the mysterious cause of an ICMP Host Unreachable Message: A student who was taking my SANS 504 class told me that he was getting a mysterious ICMP Host Unreachable message coming from their border firewall back into their network a couple times per day, and he wanted to know what was triggering it. I asked him what the destination address of the packet was. It was one of their Windows 2003 servers. So, the Windows 2003 server was trying to send a message out through the firewall, but the firewall was responding that it didn’t know how to reach the destination machine. The student wanted to know which program on the Win2K3 box was sending the original message. I had him run:

C:\> netstat -nao 1 | find "[DestIPaddr]"

As long as that program was trying to make a TCP connection, we’d see output from netstat for the half open connection, and could see the process ID number (the -o in netstat gives me the PID of the process using the port). We were then able to run the wmic command to get all kinds of info about that process:

C:\> wmic process where processid="[pid]" list full

Playing process whack-a-mole: I was playing a king-of-the-hill style capture the flag game, and my adversaries were trying to listen on a given TCP port on the target box, which I had already compromised. I wanted to simply kill any process that started on that port as quickly as possible. The netstat invocations listed above are quite nice, but they don’t really help much for this need. You see, putting the integer after the netstat invocation makes netstat run continuously, showing us its output every 1 second. We can scrape through that output with the find command, but we cannot then run any other command after that, such as a taskkill. That’s because netstat never stops running, so we cannot kickoff another command. For this, we’ll have to wrap things in a FOR /L loop to get our continuity of operation, and in a FOR /F loop, to parse out the PID from our netstat output, as follows:

C:\> for /L %i in (1,0,2) do @for /f "tokens=5" %j in ('netstat -nao ^| find ^":47145^"') do @taskkill /f /PID %j SUCCESS: The process with PID 3152 has been terminated. SUCCESS: The process with PID 2432 has been terminated. SUCCESS: The process with PID 4068 has been terminated.

Here, I’ve simply set up a FOR /L loop to count from 1 to 2 in steps of zero to implement a continuous loop. Then, at each iteration through the loop, I run a FOR /F loop set to parse the fifth column (the PID) from the output of a netstat -nao command, whose output itself is filtered by the find command looking for our port. Once I’ve got my PID, I then use taskkill to forcefully (/f) kill that PID. The evil listening process cannot listen for very long now.

Note that I’ve eliminated the 1 second delay — this thing will run as fast as it can, useful in a CtF game as long as you aren’t worried about being a performance hog. If you want a delay, we can simply add “& ping -n 2 127.0.0.1>nul” inside our FOR /F loop to ping ourselves twice (which takes about 1 second), as follows:

C:\> for /L %i in (1,0,2) do @(ping -n 2 127.0.0.1>nul & for /f "tokens=5" %j in ('netstat -nao ^| find ^":47145^"') do @taskkill /f /PID %j)

Create a real-time network sentinel by adding a date and timestamp and dumping process info: Now, let’s really get jiggy with it. Instead of killing processes or using the above commands in an interactive mode, wouldn’t it be nice if we could run a command that would just watch a given TCP port, and when it finds something using that port, it displays the date/time and various vital aspects of that process? Of course it would!

But, we’d only want it to print information when a new processID starts to listen on the given port. That way, if one process starts and listens for a while, and then stops, and another starts up, we would capture on our output the time and details of each process that interacted with the port, a diligent sentinel gathering data for us. That would allow us to run a command that can watch for given network activity over night, and then come back to see what happened and when it happened later. I use this concept and the resulting command a lot during investigations. Check this out:

C:\> cmd.exe /v:on /c "set status=0 & for /l %i in (1,0,2) do @for /f "tokens=5" %j in ('"netstat -nao ^| find ":47145""') do @if NOT !status!==%j echo !date! !time! & wmic process where processid="%j" list full & set status=%j"
Sun 01/03/2010 5:36:47.79

CommandLine=nc -l -p 47145 CSName=FRED2 Description=nc.exe ExecutablePath=C:\tools\netcat\nc.exe ExecutionState= Handle=3116 HandleCount=33 InstallDate= KernelModeTime=0 MaximumWorkingSetSize=1413120 MinimumWorkingSetSize=204800 Name=nc.exe OSName=Microsoft Windows XP Professional|C:\WINDOWS|\Device\Harddisk0\Partition1 OtherOperationCount=208 OtherTransferCount=124718 PageFaults=475 PageFileUsage=540672 ParentProcessId=3584 PeakPageFileUsage=540672 PeakVirtualSize=14987264 PeakWorkingSetSize=1953792 Priority=8 PrivatePageCount=540672 ProcessId=3116 QuotaNonPagedPoolUsage=2376 QuotaPagedPoolUsage=29780 QuotaPeakNonPagedPoolUsage=2376 QuotaPeakPagedPoolUsage=29924 ReadOperationCount=5 ReadTransferCount=15977 SessionId=0 Status= TerminationDate= ThreadCount=1 UserModeTime=300432 VirtualSize=14987264 WindowsVersion=5.1.2600 WorkingSetSize=1953792 WriteOperationCount=1 WriteTransferCount=72

In this command, I start by running a cmd.exe with /v:on to enable delayed variable expansion. I’ll need that, because I need to store status to track if the PID using the port changes. I then set my initial status to zero, since no process should have that PID. I then start a FOR /L loop to run everything continuously. Then, I run a FOR /F loop to invoke my netstat command, piping its output through the find command to look for that port (47145, although you could also look for an IP address, as shown earlier in the article). I take the fifth column of output, the PID, and store it in iterator variable %j. Then, in the do clause of my FOR /F loop, I use an IF command to see if my status has changed (if the status variable is NOT the same as my PID, it has changed since the last iteration). When I detect such a change, I print out the date and time (note that !status!, !date!, and !time! have the exclamation points because I want their values to float over time with delayed variable expansion). I also run wmic to get full details of the given process. Finally, I set my status to the current PID. That way, I can check to see if it changes in the next go-round. This may look complicated, but it is darn useful. I’m simply dumping output to the screen here, but you could append it to a results file if you’d like by using >>results.txt at the end.

As you can see, there are a lot of fun things my little friend netstat can do using these building blocks.

Hal jumps in:

It’s always been interesting to me how much Windows completely ripped off the Windows netstat command is like the Unix version. With some minor changes, most of Ed’s examples work perfectly well on most Unix systems. Obviously, you end up using “grep” instead of “find” to filter the output, but the netstat options are almost 100% compatible. Here’s the Unix equivalent of Ed’s first netstat command as an example of what I’m talking about:

$ netstat -na | grep -i listen

Note that the output that Ed wants to match in this example is “LISTEN” in the Unix universe, not “LISTENING”.

One important difference between Unix and Windows netstat is the way you get netstat to run continuously rather than producing one set of output and then exiting. With the Unix version, you need to use “-c” to specify continuous mode:

$ netstat -nac | grep <Mike's IP addr>

The interval for “-c” is fixed at one second– you can’t specify your own interval like you can with Windows. If you want to use a less frequent interval, you’ll have to write your own loop and use “sleep”, like so:

$ while :; do netstat -na | grep <Mike's IP addr>; sleep 5; done

Frankly, though, I think I’d just use our old friend the “watch” command instead of either “netstat -c” or the loop option:

$ watch -n 1 'netstat -na | grep <Mike's IP addr>'

The watch command is not only quicker to type than the loop, it allows us to easily choose our preferred monitoring interval, which we can’t do with “netstat -c”.

But really I wouldn’t use netstat for this at all as long as I was on a machine that had the lsof command installed. With lsof, not only can I do this in less keystrokes, but I’ll also get more information than I would from the netstat. lsof is *my* little friend and the friend of Unix admins everywhere.

The downside is that you usually have to be root to get complete output from lsof:

# watch -n 1 lsof -nPi @<Mike's IP addr>

“lsof -i @<ipaddr>” show you all connections related to the specified IP address. As with netstat, “-n” suppresses IP address to hostname mappings. However, while “netstat -n” also suppresses port number to port name mappings, lsof wants you to use the “-P” option to show numeric port numbers.

We can also use lsof to do something similar to Ed’s port monitoring examples:

# watch -n 1 lsof -nPi :47145

Notice that you prefix port numbers with a colon when using lsof, while IP addresses are prefixed with “@” as we saw above. This allows us to stack IP address and port combinations to be even more specific. For example, suppose I was interested in SSH connections to a particular box, I could write:

# watch -n 1 lsof -nPi tcp@<ipaddr>:22

The above example also shows how to specify a particular protocol type (usually “tcp” or “udp”) along with the IP address and port information. If I just wanted SSH connections to all IP addresses, I could do:

# watch -n 1 lsof -nPi tcp:22

Another advantage to using lsof instead of netstat is that lsof’s “-t” option makes “process whack-a-mole” dead easy:

# while :; do kill -9 `lsof -t -i :47145`; done

As we discussed back in Episode #69, “lsof -t” just outputs the PIDs of the matching processes. This means it’s easy to kill those processes by using backticks and the kill command as we’re doing here.

Ed’s final monitoring loop is interesting. Here’s how I’d do something similar on a Linux machine:

# while :; do pid=`lsof -ti :47145` [[ "X$pid" == "X" || "X$pid" == "X$lastpid" ]] && continue lastpid=$pid echo -n '*** ' date ps e --cols 1600 -fl -p $pid lsof -p $pid done

I’m first getting the PID of the process we’re interested in using “lsof -t…”. If there’s currently no PID associated with this port, or if the current PID is the same as the previous time we ran the loop then we just start the loop all over again.

Otherwise I update the value of $lastpid and output some info about the new process. You could choose to use whatever commands you want, but here I start by throwing out some asterisks with “echo -n” so that it’s easier to find the start of each section of output. Then I follow that up with the date command to get a timestamp. Since I used “echo -n”, the date output will appear on the same line as the asterisks.

Then I use the ps command to dump some information about the process. “ps e” will dump any environment variables that are set in the process context. Since this output tends to be long, I also add the “–cols 1600″ option to specify the screen width as a larger value so that the output doesn’t get truncated (on narrower displays, the lines will simply wrap). I also use “-fl” to get the most detailed output possible. The “-p” option is used to specify the PID we’re interested in. While the ps command will give us plenty of output, I’m also using lsof to dump information about all of the open files associated with the process.

It’s worth noting that both Ed’s loop and mine assume that only one process at a time is ever going to be associated with the port we’re monitoring. But if the port is a back-door login port, you might actually have multiple connections to the port at the same time. I’d finesse this in bash by using an array to track multiple PIDs at the same time, but I think it would be next to impossible for Ed to deal with this in cmd.exe.

Tim leaps in:

Of course we can use netstat in PowerShell, but it would be mostly redundant (with the exception of the all of Ed’s For Loops). Also, Hal seems upset about Windows “stealing” netstat so let’s try the same thing in PowerShell without using netstat. Without netstat we will have to use the .NET framework available to us through PowerShell.

Here is the PowerShell and .Net equivalent of the first netstat command.

PS C:\> ([Net.NetworkInformation.IPGlobalProperties]:: GetIPGlobalProperties()).GetActiveTcpListeners() AddressFamily Address Port ------------- ------- ---- InterNetwork 0.0.0.0 80 InterNetwork 0.0.0.0 135 InterNetwork 0.0.0.0 445 InterNetwork 192.168.1.5 139 ...

Most of the information we are going to need is accessed via the IPGlobalProperties object’s methods and properties. Since we are going to be using it a lot let’s dump it into a variable so we can save keystrokes in future commands.

PS C:\> $a = [Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties()

Now we can run the first command again like this:

PS C:\> $a.GetActiveTcpListeners()

The name of this method is pretty self explanatory, it returns the active TCP listeners. Not much to talk about, so we’ll move on.

The next task is to find out when Mike connects.

PS C:\> while (1) {$a.GetActiveTcpConnections() | ? {$_.RemoteEndPoint.Address -eq "[MikePoorIPaddr]"} }

We use a while loop that will run forever since “1″ is always true (except for very small values of 1). Then we get the active TCP connections and look for Mike’s IP address.

This will really hit the processor, so we probably should put in a delay of a at least a half a second.

PS C:\> while (1) {$a.GetActiveTcpConnections() | ? {$_.RemoteEndPoint -like "[MikePoorIPaddr]:*"; sleep -milliseconds 500}}

…or maybe two seconds

PS C:\> while (1) {$a.GetActiveTcpConnections() | ? {$_.RemoteEndPoint -like "[MikePoorIPaddr]:*"; sleep 2}}

So no we move on to the malware that Ed was analyzing. In the example, the port to be monitored is TCP 47145.

PS C:\> $a.GetActiveTcpListeners | ? {$_.Port -eq 47145}

We get the active tcp listeners and then filter the results for connections where the local port is 47145.

Now we have the problem of killing the process. Without using netstat we don’t have a way to determine the PID. It looks like we will have to cave in an use our old friend netstat (even though Hal is trying to poison this friendship).

PS C:\> netstat -ano | ? {$_ -like "*:47145 *"} | % { $_ -match "\d+$"; stop-process $matches[0] }

The output of netstat is filtered for connections on port 47145. Using a regular expression inside the ForEach-Object loop we look for a number at the end of the line. In regular-expression-land \d+ means at least one numeric digit and $ means it has to be at the end of the line. The number that we found is represented by the $matches object so we use the first and only match, $matches[0], with stop-process to kill it.

Ed’s grand finale, the monitoring loop done in PowerShell.

PS C:\> while(1) {netstat -ano | ? {$_ -like "*:47145 *"} | % { $_ -match "\d+$"; Get-Process -id $matches[0] | Format-List *; (Get-Process -id $matches[0]).WaitForExit() } }

While it ain’t pretty, I think we can safely say that the PowerShell version is the easiest to read of the bunch (finally), but I did add a few extra line breaks and spaces to make it even easier to read. How does it work?

First, we have our friendly infinite while loop. Nested inside the loop is our netstat command and our filter to search for port 47145. As before, we use the match operator to find the PID. Then we get all of the properties for the process. Piping the Get-Process command into Format-List * will display all the properties, not just the most commonly used properties. Finally we wait for the process to end and then our loop continues from the beginning. By using the WaitForExit method we know when that process dies and we can look for a new process. We don’t have to worry about finding, keeping track of, and comparing the PID. It makes it much easier and cleaner. It also has the added benefit of being very light on system resources while it is waiting.

admin | Jan 05, 2010 | Comments 0

Yule Be Wanting an Explanation Then

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket
Hal returns to the scene of the crime

I opened last week’s post saying there would be no “explanations or excuses”, but apparently that wasn’t good enough for some of you. So at the request of our loyal readers, we’d like to revisit last week’s episode and explain some of the code. Especially that crazy cmd.exe stuff that Ed was throwing around.

Of course the bash code is completely straightforward:

$ ct=12; while read line; do
[ $ct == 1 ] && echo -n Plus || echo -n $ct;
echo ” $line”;
((ct–));
done < keyboards drumming
... lines removed ...
command line hist-or-y!
EoLines

First we're initializing the "ct" variable we'll be using to count down the 12 Days of Fu. Then we start a while loop to read lines from the standard input.

Within the body of the loop, we use the quick and dirty "[...] && ... || ..." idiom that I've used in previous Episodes as a shortcut instead of a full-blown "if ... then ... else ..." clause. Basically, if we've counted down to one then we want to output the word "Plus"; otherwise we just output the value of $ct. Notice we use "echo -n ..." so that we don't output a newline. This allows us to output the text we've read from the standard input as the remainder of the line. Finally, we decrement $ct and continue reading lines from stdin.

The interesting stuff happens after the loop is over. Yeah, I could have put the text into a file and read the file. But I was looking for a cool command-line way of entering the text. The "EoLines" syntax at the end of the loop starts what's called a "here document". Basically, I'm saying that the text I type in on the following lines should be associated with the standard input (of the while loop in this case). The input ends when the string "EoLines" appears by itself on a line. So all I have to do is type in the 12 lines of text for our 12 Days of Fu and then finish it off with "EoLines". After that we get our output. Neat!

Everybody clear? Cool. I now throw it over to Tim to get the lowdown on his PowerShell madness.
Tim goes back in time

Let's unwrap what we did last week.

PS C:\> $ct=12; “keyboards drumming
admins smiling
… lines removed …
command line hist-or-y!”.split(“`n”) |
% { if ($ct -eq 1) {“Plus $_”} else {“$ct $_”}; $ct– }

Here is the break down:

First, we initialize the $ct variable to begin our count down.

ct=12;

That was easy. Next, we take a multi-line string and split it in to an array using the new line character (`n) as the delimiter.

“keyboards drumming
… lines removed …
command line hist-or-y!”.split(“`n”)

We could have just as easily read in content from a file using Get-Content, but I wanted to demonstrate some new Fu that was similar to Hal’s Fu. The nice thing is that reading from a file would be an easy drop-in replacement.

Once we have an array of Fu Text, we pipe it into ForEach-Object so we can work with each line individually.

if ($ct -eq 1) {“Plus $_”} else {“$ct $_”}

Inside the ForEach-Object loop we use an IF statement to format the output. If the count is one, then we output “Plus” and the Fu Text, otherwise we output the value of $ct and the Fu Text.

Finally, we decrement the value of $ct.

$ct–

Simple right? That was pretty straightforward. But sit back, and grab some spiked Egg Nog before you proceed further.
Ed Surveys the Madcap Mayhem:

I really liked this episode because it required the use of a few techniques that we haven’t yet highlighted in this blog before. Let’s check them out, first reiterating that command:

c:\> copy con TempLines.txt > nul & cmd.exe /v:on /c “set ct=12& for /f
“delims=” %i in (TempLines.txt) do @(if not !ct!==1 (set prefix=!ct!) else (set prefix=
Plus)) & echo !prefix! %i & set /a ct=ct-1>nul” & del TempLines.txt
keyboards drumming
—-snip—-
command line hist-or-y!
^Z (i.e., hit CTRL-Z and then hit Enter)

OK… we start out with the copy command, which of course copies stuff from one file to another. But, we use a reserved name here for our source file: con. That’ll pull information in from the console, line by line, dumping the results into a temporary file, very cleverly named TempLines.txt. Of course, there is the little matter of telling “copy con” when we’re done entering input. While there are several ways to do that, the most efficient way to do so that has minimal impact on the contents of the file is to hit CTRL-Z and then Enter. Voila… we’ve got a file with our input. By the way, I throw away the output of “copy con” with a “> nul” because I didn’t want the ugly “1 file(s) copied.” message to spoil my output. By the way, it kinda stinks that that message is put on Standard Output, doesn’t it? Where I come from, that is much more of a Standard Error thing. But, I’m sure we could get into a big philosophical debate about what should go on Std Out and what should go on Std Err. But, let’s just cut the argument short and say that many Windows command line tools are all screwed up on this front, regardless of your philosophy. That’s because there are no reasonable and consistent rules for directing stuff to Std Out versus Std Err in Windows command line output.

Anyway, back to the point. I then run “cmd.exe /v:on” so I can do delayed variable expansion. That’ll let me use variables with values that can change as my command runs. Otherwise, cmd.exe will expand all variables to their values instantly when I hit Enter. I need to let these puppies float. I use the cmd.exe to execute a command, with the /c option, enclosing the command in double quotes.

So far, so good. But, now is where things get interesting.

To map to Hal and Tim’s fu, I then set a variable called ct to a value of 12, just a simple little integer. But, what’s with that & right after the 12? Well, if you leave it out, you’ll see that the output will contain an extra space in “12 keyboards drumming”… it’ll look like “12[space][space]keyboards drumming”. Where does the extra space come from? Well, check this out:

c:\> cmd.exe /v:on /c “set var=foo & echo !var!bar”
foo bar

c:\> cmd.exe /v:on /c “set var=foo& echo !var!bar”
foobar

Do you see what’s happening here? In the first command, the set command puts everything in the variable called var, starting right after the = and going up to the & (which separates the next command), and that includes the space! We have to omit that space by having [value] with the & right after it. For a more extreme example, consider this:

c:\> cmd.exe /v:on /c “set var=stuff & echo !var!blah”
stuff blah

I typically like to include a space before and after my & command separators when wielding my fu, for ease of readability. However, sometimes that extra space has meaning, so it has to be taken out, especially when used with the set command to assign a string to a variable, like the ct variable in that big command above.

Wait a second… earlier I referred to ct as an integer, and now I’m calling it a string? What gives? Just hold on a second… I’ll come back to that in just a bit.

We have to deal with our FOR loop next. I’m using a routine FOR /F loop to iterate through the contents of the TempLines.txt file. I’m specifying custom delimiters, though, to override the default space and tab delimiters that FOR /F loops like to use. With “delims=”, I’m specifying a delimiter of… what exactly? The equals sign? No… that’s actually part of the syntax of specifying a delimiter. To make an equals a delimiter, I’d have to use “delims==”. So, what is the delimiter I’m setting here? Well, friends, it’s nothing. Yeah. I’m turning off parsing, because I want the full line of my input to be set to my iterator variable. In the past, when I was but a young cmd.exe grasshopper, I would turn off such parsing by setting a delim of a crazy character I would never expect in my input file, such as a ^, with the syntax “delims=^”. But, I now realize that the most effective way to use FOR /F loops is to simply let them use you. I turn off parsing by making a custom delimiter of the empty set. Do not try and bend the spoon… That’s impossible. Instead try to realize the truth…. that there is no spoon.

Anyway, so where was I? Oh yea, with my delimiterless lines now being assigned one by one to my iterator variable of %i, I’m off and running. In the DO clause of my FOR loop, I turn off the display of commands (@) and jump right into an IF statement, to check the value of my ct variable. I expand ct into its value with a !ct!, because I’m using delayed variable expansion. Without delayed expansion, variables are referred to a %var%. My IF statement checks to see if !ct! is NOT equal (==) to 1. If it’s not, I set another variable called prefix to the value of ct.

Then, I get to my ELSE clause. Although I use ELSE a lot in my work, I have to say that this is the first time I’ve had to use it in one of our challenges on this blog. The important thing to remember with ELSE clauses in single commands (not in batch scripts) is that you have to put everything in parentheses. So, if !ct! is equal to 1, my ELSE clause kicks in and sets the prefix variable to the string “Plus”. That way, later on, I can simply print out prefix, which will contain the ct number for most of the days, but the word “Plus” for the last day.

And, here we back to that string/integer thing I alluded to above. The cmd.exe shell uses loosely typed variables. No, this is not a reference to how hard you hit the keys on your keyboard when typing. Instead, like many interpreted languages, variable types (such as strings and integers) are not hard and fast. In cmd.exe, they are evaluated in real time based on context, and they can even change in a single command. My ct variable behaves like an integer, for the most part. I can add to it, subtract from it, and store its value in another variable. But, when I defined it originally with the set command, if I had used “set ct=12 &…”, it would have been a string with the trailing space until I used it in some math, and then that space would disappear. Also, the prefix variable is given the value of ct most of the time, which is just an integer. But, when ct is one, I give the prefix variable a string of “Plus”. I’m an old C-language programmer, so this loose type enforcement kinda weirds me out. Still, it’s quite flexible.

Then, I echo the prefix (!prefix!) and the line of text (%i). I then subtract one from my ct with the “set /a ct=ct-1″, throwing the output of the set command away (>nul). Note that I want to show the prefix and the text on the same line, so I use a single echo command to display both variables on the same line. Most cmd.exe command-line tools actually put their output on standard out with a Carriage Return Line Feed (CRLF) right afterward. Thus, two echo commands, one for each variable, would have broken the prefix and the file content on separate lines, a no-no when trying to reproduce exactly the output of Hal and Tim. When formulating commands that need to display a lot of different items on a single line, I often chunk them into variables and then echo them exactly as I need them on a single line with a single echo statement.

Now, there is one interesting counter-example to the general rule that cmd.exe command-line tools insert a CRLF at the end of their output: the “set /a” command. It does its math, and then displays the output without any extraneous return, as in:

c:\> set /a 8-500 & echo hey
-492hey

I used that little fact in this fun command to spew Matrix-like gibberish on the screen from Episode #58:

C:\> cmd.exe /v:on /c “for /L %i in (1,0,2) do @set /a !random!”

When I first was working on this 12-days challenge, I was thinking about using set /a to display !ct! and then the line from the file. It would all be on the same line because of that special “no-CRLF” property of “set /a”. But, I ran into the little problem of the “Plus” for the last line of input, so I instead introduced the prefix variable and played on the loose typing. There are dozens of other ways to do this, but I tried to focus on one that, believe it or not, I thought made the most sense and was simplest.

Oh, and to close things out, I delete the TempLines.txt file. Can’t litter our file system with crap, now can we?

So, as you can see, there were a bunch of ideas we haven’t used in this blog so far that popped out of cmd.exe in this innocuous-seeming challenge, including empty-set delims, an ELSE clause, weak type enforcement, and variable building for a single line of output. That’s a lot of holiday cheer, and it makes me happy.

With that said, all of us at the Command Line Kung Fu blog would like to wish our readers a Happy and Prosperous New Year!

admin | Dec 30, 2009 | Comments 0

Getting the perfect Perm(s)

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Tim unwraps:

One of the things I find myself doing on a regular basis is creating a new directory structure and setting the permissions. The permissions are different for each folder and are based on who in the organization needs access to it. We could just write a script to create the directories and the permissions, but let’s say we want to copy permissions from one directory structure to another. For this example let’s assume we have a project folder structure that looks something this.

Prjs +-Project1 (Managers - Full Access, Consultants - Full Access) |-Budget (Consultants - Deny, Finance - Full Access) |-Data +-Docs |-ForRelease (AdminStaff - Full Access) +-InProgress
Included above is the appropriate permissions on each folder. All permissions are inherited, so consultants and managers would have access to the Data directory.

We can verify these permissions by using Get-ChildItem (aliases gci, dir, ls) and piping the results into Get-Acl.

PS C:\> ls Prjs -recurse | Get-Acl | fl Path,AccessToString

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1 AccessToString : WINXP\Consultants Allow FullControl WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Budget AccessToString : WINXP\Consultants Deny DeleteSubdirectoriesAndFiles, Modify, ChangePermissions, TakeOwnership WINXP\Consultants Allow FullControl WINXP\Finance Allow FullControl WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Data AccessToString : WINXP\Consultants Allow FullControl WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Docs AccessToString : WINXP\Consultants Allow FullControl WINXP\Managers Allow FullControl
Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Docs\ForRelease AccessToString : WINXP\AdminStaff Allow FullControl WINXP\Consultants Allow FullControl WINXP\Managers Allow FullControl

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project1\Docs\Working AccessToString : WINXP\Consultants Allow FullControl WINXP\Managers Allow FullControl

So now we want to create a second project, Project2, and we want to make sure we have the same permissions. We could copy just the directories without files, but there may be more subdirectories further down that we don’t want. So let’s create the folders.

PS C:\> mkdir Prjs\Project2\Budget PS C:\> mkdir Prjs\Project2\Data PS C:\> mkdir Prjs\Project2\Docs\ForRelease PS C:\> mkdir Prjs\Project2\Docs\Working

Note, when the Budget directory is created it also creates the Project2 directory since it doesn’t exist.

What are the permissions on the new folder?

PS C:\Prjs> Get-Acl Project2 | Get-Acl | fl Path,AccessToString

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project2 AccessToString : BUILTIN\Administrators Allow FullControl NT AUTHORITY\SYSTEM Allow FullControl WINXP\myuser Allow FullControl CREATOR OWNER Allow 268435456 BUILTIN\Users Allow ReadAndExecute, Synchronize BUILTIN\Users Allow AppendData BUILTIN\Users Allow CreateFiles
Those are not the permissions we want. The permissions need to be copied from Project1 to Project2, but how? The Get-Acl and Set-Acl commands will do it.

PS C:\Prjs> Get-Acl Project1 | Set-Acl Project2

Let’s verify.

PS C:\Prjs> Get-Acl Project2 | Get-Acl | fl Path,AccessToString

Path : Microsoft.PowerShell.Core\FileSystem::C:\Prjs\Project2 AccessToString : BUILTIN\Administrators Allow FullControl WINXP\Managers Allow FullControl WINXP\Consultants Allow FullControl

Looks good. Now the subfolder permissions need to be copied as well.

PS C:\Prjs> ls Project1 -Recurse | Get-Acl |% { Set-Acl $_ -Path ($_.Path -replace "Project1","Project2") }
First we do a recursive directory listing and get the Acl on each folder. We then take that Acl and apply it to a different folder. In our case all we need to do is replace Project1 for Project2 in the Path. Let’s verify that the permissions match.

PS C:\Prjs> Compare-Object (ls Project1 -Recurse | Get-Acl) (ls Project2 -Recurse | Get-Acl) -Property PSChildName, Access

No output, that’s good, it means the permissions are identical. How did that work?

The Compare-Object cmdlet is used to find the differences between the collection of objects returned by these two commands:

ls Project1 -Recurse | Get-Acl ls Project2 -Recurse | Get-Acl
The Property parameter specified in the original command allows us to select the properties to be checked for differences. PSChildName is the directory name and the Access property contains the permissions on the folder. We can’t substitute the Path property for PSChildName since Path is the full path and it would always be different.

Copying permissions is pretty easy, I imagine it will be pretty easy for Hal since it isn’t as granular. Finally, a bit of a leg up on Hal.

Hal just copies everything:

Do I detect a trace of jealousy and bitterness in my colleague’s last comments? Better fix up that attitude Tim, or there will be nothing but coal in your stocking this year.

It’s interesting that Tim brings up this subject, because it’s another case where the differences in philosophy between Windows and Unix are apparent. In Windows, you need to fix up your directory permissions with an external tool after you copy the files. In Unix, it’s just a natural part of the file copying operation– particularly if you’re doing the copy as the superuser.

This is also an area where we’ve seen some historical evolution in Unix-like operating systems. When I first got started with Unix in the 1980’s, the “cp” command didn’t have a “-p” option to preserve permissions, ownerships, and timestamps. The way you would copy directories when you wanted to preserve directory permissions was with the so-called “tar cp” idiom (actually, real old-timers will remember doing this with cpio):

# cd olddir # tar cf - . | (cd /path/to/newdir; tar xfp -)
Here we’re running the first tar command to create (“c”) a new archive from the current working directory (“.”) and write it to the standard output (“f -”). We pipe that output to a subshell that first changes directories to our target dir and then runs another tar command to unpack the incoming archive on the standard input. The “p” option means preserve permissions, timestamps, and ownerships. Actually “p” is normally the default if you’re running the tar command as root, so you can leave it off, but I prefer being explicit.

These days, however, there are a couple of simpler options. Obviously, you could just use “cp -p”:

# cp -Rp olddir /path/to/newdir

I generally prefer rsync though:

# rsync -aH olddir /path/to/newdir

rsync not only allows you to copy directories within the same system, but also gives you the option of copying directories across the network. Also, if you just want to update the permissions on a directory, the rsync command will do that and not actually copy any file data that has previously been copied. For more information on rsync, see Episode #24.

One issue that Tim brought up was that sometimes you want to copy only part of a directory structure, but exclude certain files and/or subdirectories. This is another place where rsync beats cp. rsync has a couple of different ways of excluding content: the –exclude option for specifying patterns to exclude on the command line, and –exclude-from for reading in a list of patterns to exclude from a file. There’s no way of excluding files built into the cp command at all. For those old fogies like me out there who still occasionally use “tar cp”, the tar command typically has a switch like -X to exclude files and directories from the archive, and GNU tar has –exclude options very similar to rsync.

One thing you do need to be careful with for all of these copy options, however, is that they may not copy special permissions like extended attributes or file ACLs by default. Both cp and rsync have explicit options you can set to preserve these settings:

# cp -R --preserve=all olddir /path/to/newdir # rsync -aHAX olddir /path/to/newdir

There’s no way to do something similar with the “tar cp” idiom, because the tar archive format doesn’t preserve extended attributes and ACLs.

Oh dear. Now it’s Ed’s turn. I hope Tim and I haven’t spoiled his holiday cheer…

Ed Joyously Responds:

Ahhhh…. file permissions. They tend to be an absolute pain in the neck to deal with en masse in cmd.exe. Sure, we can use cacls or icacls to manipulate them on single files or directories just swell. But, synchronizing or applying changes to bunches of files using cacls or icacls is often dangerous and painful. When I first read Tim’s challenge, I thought to myself, “This is gonna get ugly on us… as ugly as that feud between Snow Miser and Heat Miser.” I immediately began to search my mind for a hook or trick to make this a lot easier, hoping to avoid a trip to visit Mother Nature.

Then, it hit me: we can use our little friend robocopy, the wonderfully named tool in Vista, Windows 7, and Windows 2008! Yeah, it’s not built in to XP or Windows 2003, but it’ll work for the latest version of Windows. We talked about robocopy in Episode #24.

To address Tim’s challenge, I’m going to assume that the directory structure where we want to replicate our file permissions does not already exist, avoiding the mkdir commands Tim uses. Robocopy will make those for us, dutifully placing the proper access controls on them if we run:

C:\> robocopy [directory1] [directory2] /e /xf *

All of the subdirectories in directory1 will be created in directory2 with the same file permissions. The /e will make it recurse through those subdirectories, copying both directories with stuff in them and empty directories. The /xf means I want to exclude a certain set of files, which I’ve selected as *, meaning to exclude all files — Only directories will be copied, including all of their luscious permissions.

Well, that’s all fine and good, but what about Windows XP and 2003? Well, you can download and install robocopy on either of them, which is a pretty good idea. Alternatively, there is a way to trick Windows into applying the permissions from one directory structure to another, which applies to Windows 2003, Vista, 7, and 2008 Server. For this trick, we’ll use the handy /save and /restore feature of icacls. Here, let’s follow Tim’s lead, and assume that we’ve got directory1 and directory2 already created, and we want to take the permissions from directory1 and its subdirectories and apply them to the already-existing directory2. Check out the following command:

C:\> icacls [directory1]\* /save aclfile /t /c

This command tells Windows to run icacls against directory1 and all of its contents (*), saving the results (/save) in a file called aclfile, recursing through the directory structure (/t), not stopping when it hits a problem (/c). Now, the resulting aclfile is not regular ASCII, but instead a unicode format that includes all of the permissions for the directories _and_ files inside of directory1.

Now, if there is a directory2 that already exists and has a similar overall structure to directory1, but perhaps without having any files in it, we can use icacls to restore the aclfile on a different directory! Wherever there is commonality in the directory structure, the permissions from directory1 will be used to overwrite the permissions on the given entity in directory2. The command to use is:

C:\> icacls [directory2] /restore aclfile /t /c

Voila! We’ve restored the ACLs from directory1 onto directory2! Now, that is a delicious holiday treat.

But, that leaves out poor little Windows XP, an operating system without robocopy and icacls built in. Sad, sad, sad little XP. Looks like it gets a lump of coal in its stocking this year, not only from Santa-Ed, but also from Microsoft, which has announced its impending withdrawal of support of this very near and dear friend.

admin | Dec 15, 2009 | Comments 0

The Many Evil Ways to Make Money Online

From http://www.consumingexperience.com/

Currently, the main evil ways people make money off the Internet (i.e. take money from innocent you & me!) are:

Phishing – impersonating bank web sites in order to steal people’s banking details
Selling pharmaceuticals online – Viagra, of course; selling prescription drugs to people who don’t have a prescription
Selling cameras etc online – they take your money by Western Union, you never see the goods
“High yield investment programs” – the pyramid or Ponzi scheme gone online, effectively
Getting people to write content for them unpaid, off which they get Google Ads income – this may in fact be perfectly legal, depending on how it’s done.

Exploiting free bandwidth offers to sell internet porn was the best evil way to make money 10 years ago, but that only nets about $10 at a time. Phishing is much more profitable now, and although it’s illegal it seems phishers don’t get caught. Gentle hint: if you do decide to go into internet pron, you’ll earn more from pics of naked people than naked aardvarks!

1. Phishing

Phishing’s been around since 1996, when people would phish for AOL details (login & password) so they could get online for free using someone else’s AOL account rather than pay ISP fees.

From 2003, bad guys have been phishing for banking website login details or other credentials by impersonating banking websites and persuading people to enter their login information in forms on the fake sites, which the baddies then capture and use to take money from the duped people’s accounts.

Phishing is mostly done through using basic standard “cookie cutter” phishing kits to send people phishing emails in order to persuade them to click links to go to the forged sites. There’s an exception, drive by phishing, where malware gets onto your computer, spots when you next go banking online and captures your keystrokes (keylogging software) or sends you to some other site when you think you’re going to your banking website – but that’s a rarity.

Usually they send people emails with scare stories like “Your bank account is about to be closed down, you must login at once, here’s the link”. And of course the link leads to their fake site, not the real bank site. In fact the most successful phishing Dr Clayton’s team have seen in the last few months hasn’t been “Your bank account is about to be closed” but “This is the IRS, we’ve been reviewing your taxes and you have a refund of $93.16 due to you, please visit our website” – and the supposed repayment will be by credit card so they have to enter their name, social security number and credit card details including the 3 digits on the back! (They tried that with HMRC-lookalike pages too but weren’t so successful as they asked for a zipcode instead of postcode on the phishing form..)

The phishers are getting more careful and more culturally aware. In the USA phishing attacks for credit union debit card details operate in very localised areas; they spam the local university or local ISP with phishing emails about that credit union, using cookie cutter kits. And phishing emails about NatWest, Nationwide etc are being sent to the UK, while emails about Italian banks are sent to .it email addresses. They’re beginning to understand that sending stuff all over the world doesn’t work very well (except for .com), and sending the spam is a major cost for them so they’re better off being more targeted in their approach. But the relationship between the numbers of spam and the numbers of phishing sites is still not understood.

If you ever decide to visit phishing site out of curiosity, do ensure your virus checker is up to date etc. Most of them fairly safe but one or two of them 1 or 2 try to upload malware to your computer too…

Phishing kits and the underground economy

Most phishers use phishing kits, their own or bought – it’s hard to monetise kits as they’re quite easy to write, so creators undercut each other, offering 3 kits for $30 etc. “Mr-Brain” even gave kits away for free but, if you check the underlying php code, they arranged for a copy of the credentials to be sent to them too, which of course is why they gave them away: the security industry knew this but it recently came out on a blog so now no one will use Mr-Brain anymore, which has annoyed law enforcement officers who were exploiting Mr-Brain kits for tracking and will now have to get on top of the new kits.

On the underground economy you can buy information on compromised machines, phishing kits etc. Phishers will keep the high value cards, and sell the rest – usually for 50% of what the buyer gets (the underground economy operates on trust, a person’s “good name” is important – if you rip someone off, they won’t deal with you again). $10 is a lot of money in Romania, so it’s worth their while selling credentials for just $10. The Times last year got hold of 30 account details including an assistant judge’s in Newcastle – because they were posing as a new buyer so someone gave them 30 for free to check the merchandise and if it was good they’d be expected to go back and buy the rest of the batch. The judge didn’t even know his details had been taken.

All of this is done on IRC; law enforcement don’t close down the channels as they’d just be put straight back up again, and they can also monitor them. Plus, most of them are hosted in places where it would be difficult to take them down. Some IRC networks are hooked up to merchant accounts so that peoplle can check if credit card numbers are “good” are not (of course, the person running the list gets a copy of the number too!).

Use of domain names

At first the phishers used lookalike domain names for their fake sites, e.g. “barqlays.com”. Then they realised that as long as the bank’s name was somewhere in the URL (web address), most non-geeks would think it was the bank’s own site and wouldn’t know that e.g. “www.barclays.com.extrasecure.com” or “www.aardvark.com/~fred/wwwbarclays.com/phishingwebsite.html” weren’t in fact Barclays sites.

Now if they used “barclays” within a domain name, Barclays would go to the domain registrar and get the domain and get it transferred to them or removed for trademark infringement (on trademark risks for UK websites, see this post). Or if they used yahoo.com/~fred/barclays etc, the bank would get the site’s system administrator to remove that particular account or sub-site (obviously it’s not a good idea to try to get the domain name “yahoo.com” expunged!).

So what’s known as the Rock Phish gang (sometimes spelt Rockfish, which is behind about half of all phishing and is thought to be the Russian mafia!) started setting up innocuous-seeming domains which don’t infringe existing trademarks, notably “Lof80.info”.They’d use that domain name for impersonations of perhaps 20 banking websites (Barclays, Bank of America, Fifth Third Bank etc), with URLs such as www.barclays.com.lof80.info/barclaysphishing/whatever – but they’d cleverly make the bits at the end of the URL look like what you normally see when you login to the real bank site.

Imp note for non-techies: the domain name for a site, e.g. “mydomainname.com”, is separate from the computer where the files for the site are stored or hosted, i.e. the files which people see when their web browser downloads those files. You buy your domain name from a domain registrar, then you choose where to store or host the files for the domain (it doesn’t have to be where you bought the domain from). With a web address like “yahoo.com/~fred/phishing” the files are in fact stored under Yahoo’s domain name, but on a sub-part of their computer servers. You can change the storage location of the files associated with a domain name, as long as you change things behind the scenes so that the domain name “points” to the files at their new location. (For Blogger users: how to use your own custom domain but hosting your blog files on Google’s Blogger / Blogspot servers). So, in this context, banks can try to get the domain name cancelled so the phishers can’t use it anymore, or else they can get whoever is hosting the phishers’ files to pull the plug and delete their account or delete their phishing files – the domain name and file storage are separate things, strictly.

In other words, when fighting phishers basically banks can either try to remove domain names, or try to get the files removed, or both.

Hosting / storage of the phishing files

Phishers needn’t store the phishing website files on their own servers, and often they don’t – they can just hack into someone else’s website and use that. They needn’t compromise the whole machine, just one user’s account, e.g. someone with a blog running WordPress (which has had some big security vulnerabilities, see more on WordPress security issues or just a couple of them!), or an insecure photo site that’s not been updated for a while, where they can just break in, upload a photo with .php at the end, run the php and get in, and then put a phishing site on there.

They also used sites that provide free web space like Alice.it. They can register a name with the free host, like “bankname.alice.it”, and then put a phishing site on that webspace, and those sites tend not to take down phishing webpages very quickly. Yahoo free sites used to be quite popular with phishers but now their take down time is 20 minutes, the average takedown time being 23.8 hours because Yahoo don’t always get told about the phishing site immediately.

Dr Clayton has a graph from May 2007 showing that alice.it basically took down those phishing sites at once, only after 3 weeks of banks complaining – they’d basically been debating what to do for the 3 weeks! (Imp note: see this post about the alice.it story.)

From Prof Clayton’s research, on average in spring 2007 phishing websites stayed up for 62 hrs. However, Rock-phish domains stayed up for 95 hrs, because it’s harder to get domain names removed than it is to get a sysadmin to delete files or an account from their site where the files are kept on a third party website.

Domain registrars who’ve never encountered phishing sites before usually have no clue for about 3 weeks. E.g. last spring when phishers moved to using .hk as their favoured top level domain (moving over from .com and .info), the local police asked the domain registrar not to remove them in order to preserve the evidence (though efforts to trace them weren’t successful as they used cutouts or went through botnets or Tor). Banks asked them to remove the phishing sites, and they eventually did.

The RockPhish gang have been getting their domains to resolve to 5 or 10 IP addresses (or computers) in parallel, changing to a different 5 or 10 IP addresses every 20 minutes or so. In other words, if you try to go to that domain in your browser, it will take you to one of 5 or 10 different physical machines hooked up to the internet, and those machines would be different ones every 20 minutes. This technique’s known as fast flux.

Once they started using fastflux, it became pretty much impossible to physically locate the phishing website – they were using other people’s compromised machines just to relay people back to their “mother ship.” At first no one understood what they were doing, and their phishing sites’ uptime went up to 196 hrs. So the baddies have been very technically innovative, to avoid being taken down.

Not only did they keep changing their name servers in order to point to changing new IP addresses, but they’ve now started putting those name servers themselves on other lof80.info-style domains they’d bought, and arranged for those to work on fast-flux too – known as double fastflux or double-flux! Everything moves around at high speed, so, like in the shell game, it’s hard to tell where anything is.

It’s difficult enough to get a registrar to remove a domain name on the basis that it’s used only for phishing, imagine trying to get them to remove a domain name that’s only used to provide name services for a domain which is only used for phishing.

Imp note: see the paper on phishing website removal times etc, An Empirical Analysis of the Current State of Phishing Attack and Defence, by Tyler Moore and Richard Clayton. See also Phishing and the economics of e-crime by Tyler Moore which also goes into the mechanics of phishing, the Rockphish attacks, and fast-flux domains.

Moving the money – money mules, and the role of geeks

Dr Clayton is also interested in how the phishing “industry” works. It’s easy to compromise sites and send out spam. As with kidnapping, the hardest bit is arranging to receive the money without getting traced or caught.

Even after phishers get hold of bank account or credit card details etc, they must still be able to move the money in quantity at speed. If they move money to their account from 30 other accounts, the banks have programs that spot this sort of thing and move it back!

So what they do is to advertise for people to “Work from home! 2 hours a day” etc, people who have their own bank account and are regularly on the internet, to work as a “payment processor” for “The Sydney Car Centre” and the like. These people are known as “money mules“. Money goes into the mule’s legitimate personal bank account, and they send it out to the phishers over Western Union. When the fraud is discovered, the bank will move the money back from the mule’s account – but the mule can’t get it back from Western Union, so not only are they out of pocket, but they risk police accusations that they must have known fraud was involved, as they were getting say 10% just for moving money around. Despite Western Union warning people not to send money to strangers, it seems some people are still fooled, saying they’re not strangers they’re my employer, look here’s my contract of employment signed by the managing director!

SOCA have realised that there are other ways to make criminal activity unattractive than prosecuting people, and if they do prosecute they need to target the right people. Taking out Mr Big is no good as a lieutenant just takes over, and catching low level mules isn’t either as they’re expendable and know very little anyway. So SOCA now concentrate on taking out the people who know how to launder money, set up a phishing site, build a spam sending engine or viruses for a botnet. This is more effective as Mr Big can’t operate without his geeks.

Measuring phishing – tracking the figures

The server logs of phishing sites (except the Rock Phish ones) are often world readable, so researchers can get a list of their most visited pages from Webalizer, and it’s interesting to see a site which has had no traffic for months spike with everyone visiting the page bankofamerica.html! Dr Clayton checks the “Thank you” page (which redirects to the real bank’s own pages) as it tells how many people visited the site and gave their details (the phishers are very polite and send people to a “thankyou” page after they’ve filled in the form). Some sites also leave gathered credentials lying around on a file on the machine called e.g. results.txt so researchers can review them.

From this research it seems about half of the people who’ve filled in their details on phishing sites have email addresses along the lins of “diespammerdie” so about half haven’t been fooled and have deliberately filled in the wrong details. Recently he came across what seemed to be a valid American Express credit card but it was said to be registered to a Fred West of 25 Cromwell Street! And he’s seen an address in the USA which was supposed to be 45 Vagina Avenue!

It’s possible to construct a mathematical model that suggests that about 15-20% of banks’ losses are through phishing rather than keyloggers, malware or skimmers on the fronts of cash machines. Everyone concentrates on phishing but no one seems to have done any research on keyloggers, who can keep enjoying a field day!

Issues and problems with banking websites

Different banks use different security methods. But most of them tend to copy each other.

Using the mouse. Clicking a letter or number of your password from a drop down list or rotating keyboard etc (as per Lloyds TSB’s site) no longer works. It’s good against keyloggers, i.e. malware planted secretly on your computer which records keystrokes, exactly which keys you’ve pressed in what order – but these days malware will take a “snapshot” of the pixels, the area of the screen, around where you click, and then send the pic to the bad guys.

1st and 3rd characters from your password etc. Most phishers get round this by pretending it’s an emergency so you have to enter all the details, i.e. your full password. Also tests have shown that generally people get confused if they’re not asked for the characters in the “right” order (e.g. characters 7, 9 and 3 instead of 3, 7 and 9) (Imp note: although I think First Direct, and certainly ING Direct, don’t ask for them in numerical order). In fact after about 9 tries it’s possible to capture all the information anyway, with a keylogger. Banks set up their systems that way because they were more worried about shoulder surfing than keyloggers, so the systems are vulnerable to keylogger attack. It’s a question of the threat model: if they had engineered things another way keyloggers may have struggled but it would have been easier for shoulder surfers to steal banking details.

Find the face. Looking for a face or picture that you chose when you originally signed up (as per Bank of America’s site) is supposed to confirm that you’re on the correct banking site, not a fake site.

But the bad guys can social engineer around that. They can email you and say, very sorry the main system is down as we’ve had a break-in, please would you confirm your data immediately so we can check that the data we recovered for you is accurate – and then provide a helpful link to the alternative site. Some people will accept that explanation, go to the fake site which doesn’t show any face (the system’s down isn’t it, so of course they can’t show the usual face), and dutifully enter all their details. There’s a story, who knows if it’s true or not, that an Australian bank had a huge DOS attack which took its site offline for a few days – during which a phishing attack was launched against it (or rather its customers) asking customers to visit an “emergency server”! It’s not known whether the phishing was in response to its real site being offline, or had been planned to coincide with it.

Also, the “put up a face” method doesn’t work against “man in the middle” attacks where the bad guys intercept what’s passing between you and the banking website, although such attacks are less common as they’re much harder to set up and get to work properly with phishing kits. In fact, generally almost none of the methods will work against man in middle attacks. (Imp note: another example of man in the middle.)

However, fortunately MITM attacks are not very common, there’s only a handful of them because they’re more complicated to set up. It doesn’t work very well to just compromise a random machine and try to arrange for it to work properly with a kit, you have to be very geeky to do it successfully. Most phishers use kits and none of kits do man in middle, which is why they’re not very popular.

But as mentioned they don’t need MITM. None of the security techniques will work if the phishers can produce a plausible explanation for why the security mechanism is not working. That will help them persuade the cleverer people that it’s different today for some reason, but most people won’t even notice the fact that it’s not working.

Issues with security indicators; and is your personality a factor?

Most academic research indicates that the security indicators or authentication are immaterial – no one takes any notice because they’re just concentrating on going to the bank website, even if there was something flashing red in the corner “This is dangerous don’t go here”, they won’t pay any attention because they’re focussed on logo.

A paper The Emperor’s New Security Indicators (Imp note: Symantec summary) reported on lab experiments with people which showed that, apart from physically stopping you from going to the site (which IE7 does), nothing works; even with Internet Explorer 7 you can click on one line of text to go to the site.

When explaining to someone how they’d know if a webpage is secure, it’s too difficult to explain how URLs work, how the web works, what precautions they should take to prevent malware on their computer or ADSL router, the way they should use passwords etc – so the normal explanation is “Look for the lock icon”, because that’s easy. In fact it should be “Look for the lock icon at the bottom right in the grey bar, not in the page”, but that’s not true for IE7 where it’s at the top!

Paypal works on the basis that “It’s OK because it’s green”. But a security vulnerability enables people to produce extra floating windows and put them over the browser address bar, and they could easily make their background green.

Dr Clayton used to say, hover the mouse over the link and check the address that comes up at the bottom, as there used to be tricks with very long URLs with spaces etc because they didn’t wrap properly, or tricks with @ in the URL e.g. www.barclays.com@aardvark.com (which so concerned Microsoft that they stopped it working, although it still works in a handful of ftp phishing sites). However, the hovering tip doesn’t work because of a frame bug in both IE and Firefox. If a frame doesn’t terminate properly, the browser tries to guess what it should show for the URL. The code that decides what to show when hovering over a link shows a common field, but the code for deciding where to go when you clicked the link shows another field, so it doesn’t work – when you hover over a link it might show microsoft.com, but when you click it it may take you to aardvark.com!

So there’s a big problem with explaining security to people – the problem with rules of thumb is they can be got round, and if the bad guys work out a way round them, then you’re toast because you think it’s safe when it’s not.

Dr Clayton is hoping to conduct some research this year on whether security mechanisms (which don’t work very well anyway) work better with some people than others; are there gender differences, so that mechanisms that work well for women differ for those which work well for men? If you score people on an autism / techie scale, might more autistic types work better with indicators, do certain personality types just ignore indicators when more picky people would check the lock icon?

There’s also very little research on people who may be more susceptible to social engineering or fraud. Banks, eBay etc have told Dr Clayton that they do see the same people getting suckered in time and time again.

So why aren’t banks’ security measures as good as they could be?

A couple years back, British banks lost £33 million through phishing. One bank alone lost £31 million of that, not because they were attacked more often but because they were poorer at defending (it was Barclays – who then rushed around introducing new security measures, as a result of which in the following year total British bank phishing losses were reduced to £26 million; so at least all the banks can can say they’re better than average, except for one bank!).

But for a bank, £30 million isn’t much money. To issue SecurID tokens to all its customers would cost a major bank £50 to £100 million, so it could actually take 2 or 3 years of phishing losses before it was financially worth it for it to do that.

Banks’ policies and procedures compound the problem. One attendee’s address wouldn’t validate on the form she submitted, so the bank sent her an email saying there was a problem verifying her details, please would she email back with the correct details! She wrote back to point out they were reinforcing phishing by legitimately asking for details, and why couldn’t they have say asked her to login to her account to correct her details there? Another stupid thing – one bank which shall remain nameless decided it would be a good idea to offer free webmail for all its customers – using the bank’s own domain name! Talk about helping phishers to masquerade as bank employees etc…

Also, the legal position doesn’t sufficiently incentivise banks to take better security precautions. Dr Clayton thinks we should move to a position where the banks are responsible for phishing losses. If someone forges your signature on your cheque and the bank pays out against it, it’s the bank’s problem since the 1882 Bills of Exchange Act, which is why banks concentrate on cheque fraud – because it’s their money.

But with things electronic, customers’ only protection is the UK Banking Code, which says banks will repay them if they’ve not been fraudulent. In practice the bank will argue with you and you’ll get your money back you’re if male, white, middle class and articulate! One of Dr Clayton’s colleagues Ross Anderson, who testifies in court cases on computer security, has noted a disproportionate number of cases that end up with him, where the system hasn’t worked so experts have had to be brought in to explain the position etc, appear to be not white middle class men. There seems to be some bias in the system, and Dr Clayton thinks it’s because it’s just the Banking Code (which banks voluntarily agree to), rather than a statute making it clear that the banks have to repay the money.

If the position was such that it was the bank’s money, if they were to bear the loss should they accept instructions to move money which didn’t in fact come from you in circumstances where they couldn’t prove fraud on your part, then Dr Clayton believes there would be a sea change in banks’ approach and how careful they are. At the moment many people working for banks do try to be careful, but the pressure just isn’t on banks in the right way, and things need to be changed in order to get the senior managers to care.

Why do the banks appear to try anyway? It’s reputational. None of the security indicators work, it’s all theater to persuade customers that banking sites are secure, because the really horrible thing for banks isn’t how much money they’re losing (the amounts are peanuts as far as banks are concerned), but whether people will stop trusting them. Currently about 60% of the population do their banking online; if people suddenly lost confidence and decided they didn’t want to do internet banking anymore, the banks would have to pay expensively to buy back all those trendy wine bars to turn them back into branches, and hire employees to staff them, which would cost them a lot more.

Facebook and other social networking sites

Concern was expressed about social networking sites and the like which ask you to enter your Gmail or other webmail login and password after you sign up witht them, so they can invite your friends or pull in their contact details etc.

An uncrupulous operator could easily set up a legitimate-sounding site just to phish details like that, even though it doesn’t seem to have happened yet (Imp note: I refused to give my Gmail details to Facebook, myself. They make it hard if not impossible to enter individual contacts manually (or through copy/paste or the like), I assume precisely because they want to put pressure on you to give in your webmail details and let them spam all your contacts!)

There’s another specific risk with Facebook. You know all those Facebook apps that let you throw sheep at people, send them kisses etc? Most of these apps are now created by third parties who have nothing to do with Facebook. If baddies create a compelling enough Facebook app that people will want to install (games etc), once you install that app it (and the people behind it) will then be able to access everything that Facebook knows about you. Because for Facebook apps the basic model is – “Let the application access everything that Facebook knows about me”? Yes or No. Period. (Imp note: incidentally this explains Facebook privacy settings well, see also How to prevent Facebook applications from spamming your mini-feed; and for anyone interested, you should soon be able to get out of Facebook’s Hotel California, at least if you’re in the UK).

So you might want to be very selective about installing Facebook apps!

eBay

eBay runs on trust and 99% of the time it works. But it’s not a good idea to buy flatscreen TVs off Ebay without looking hard at the seller’s feedback! People phish Ebay to get hold of high value accounts with good feedback, especially where it’s blurred whether it’s as buyer or seller. So you might find someone who used to buy and sell tea cosies for years suddenly having flat screen TVs and laptops to sell. Someone in California bought and sold Ferraris, and he got very good feedback until he sold the same car to 40 people in parallel. When he tried run away they caught him within 3 days..

Educating the public about scams and social engineering

People fall for scams because they don’t understand how they work. But educating people will never fix social engineering. People have been conning others for hundreds of years, and people have been falling for it. The notorious Kevin Mitnick was not some uber hacker but very good at social engineering, getting people to do things – see his book The Art of Deception (Amazon: The Art of Deception: Controlling the Human Element of Security). E.g. to break into a telephone company he needed a SecurID number, so he rang the company, said he was a telephone engineer up a pole in Kansas in a blizzard and he’d left his SecurID by his bed 20 miles away. They said it’s OK the manager has one for emergencies, and got it from the drawer and read the number out to him over the phone!

People do that because they’re helpful. Some other scams involved ringing up a switchboard and getting the name of someone in the corporation, then ringing that person up and saying “Hi I’m from HR I’m new here”, and they’d say “Oh you must work for X”, and then he’d ring up the next person and say “I’m X from HR” and just keep building on it. The only hacking thing he did was to go into their reception so that phone number on their caller ID would be internal.

The only way to stop someone from social engineering your company is to make all receptionists very unhelpful and rude to everyone and never tell them anything. Of course, some UK companies already along those lines! But seriously, it’s very difficult to protect a company from social engineering.

Educating the public is good (and Egg used to run late night ads explaining some Net scams), but it has significant limitations. It may seem an obvious scam that an African dictator has $17m to give you, but the web is full of fake banks, eg the “Nation Buildingwide Bank”, which have been set up solely to help fool the “marks“, who are given login details. They can login to check a supposed bank account to which “their” money has been transferred, though if they try to transfer it out it will say sorry can’t do that because you’ve not paid X fee, etc. But you can in fact login to these “banks” and see the “money” sitting there!

These scams are called 419 scams after section 419 of the Nigerian code because people believe they came out of Nigeria. [Imp note: in fact earlier this year three West Africans were convicted of 419 scams in New York.] These scams are actually a variation on the “Spanish prisoner” scam which dates back to the 1600s! In the 17th century conmen would wander aroud England talking about a nobleman locked up in a castle in Spain. Naturally, he had lots of money, and the conman would say he was trying to raise money to form a group of mercenaries to rescue the nobleman, and when he was rescued he would be very grateful and reward everyone who contributed to his rescue – so would you like to cough up please?

It worked in the 1600s, it works today – social engineering will be almost impossible to get rid of. Banks must be made to concentrate, by changing the liability laws, and equally they must rely on the fact that most would be bad guys try to do “cookie cutter” stuff, they move money in the same way, and it’s possible to pick up the patterns, see what’s going on and then stop it in order to reduce these losses.

2. Selling pills online

The pharmaceuticals sold online from the “better” pill sites are in fact real, e.g. sleeping tablets, because these sites are mainly selling to addicts who can’t get the drugs off their doctors anymore, and if what they are sent doesn’t work they won’t buy from that site again. In business it’s easier to sell more things to existing customers than go find new customers, so that’s what these sites do and that’s why they are still around. They still send spam email etc to find new customers, but they keep selling to old customers.

The take down period for pill sites is usually months. Some of them are also on fast flux networks so the only way to remove them is to remove the domain name, and it’s very hard to get registrars to understand that the name is only being used to host a Canadian pharmacy which is illegal under the laws of X, Y etc.

3. Selling cameras etc online

These sites are often run out of China. They tempt people with cameras etc at bargain prices e.g. Nikon D70S with lens for 150 euros! Some of them are quite plausible in that the offered prices are quite close to the going rate. They even have chat facilities on their sites where punters can ask about the products etc.

But they say they don’t take credit cards, you have to send the money to them by Western Union to order the goods. And of course, that’s the last you see of your money and you never get the goods. The admonition to never send money by Western Union to “strangers” seems not to work on people intent on a discount, particularly if they’ve chatted with the site concerned so they don’t think they’re “strangers”!

4. “High yield investment” programs – make money fast!!

Pyramid schemes or Ponzi schemes are common in the real world, e.g. about 2 years ago there was “Women helping Women” run from the Isle of Wight in the UK. And about 10 years back most of the economy of Albania was involved in a Ponzi scheme, before it collapsed.

These schemes work very well for the perpetrators. They offer to pay people a few percent on their investment – that’s a few % per day – but of course they pay those who invested earlier from the money they get from those who join later. And they can do this over the internet.

It seems everyone who takes part knows it’s a Ponzi scheme but they’re still getting their few per cent. a day and if it runs for another 50 days or so they’ll make their money back – so in a sense you could say it’s not really a Ponzi scheme, but gambling. Punters are gambling that the scheme will do well enough to get them their “investment” back and maybe more. Dr Clayton calls them “post-modern Ponzi schemes” because everyone who plays, at least earlier on, knows it’s gambling!

These schemes are very common, especially in Russia, as kits are readily available for about £50. A recent search on a particular phrase turned up 12,000 schemes. However in the UK it’s illegal to run pyramid schemes.

Those who set up these schemes will buy domain names and hosting at the start, run the scheme for 20 days or so before collecting the money then moving on to buy another domain name, etc. Some of them go as far as to buy https certificates for their website security!

There are even reputation services which provide statistics (independent or depending on bribes from scheme owners!) on which schemes are paying out ad worth investing in.

5. Google Ads – the red-blooded American Privila way…

Getting an AdSense account then putting Google Ads on your website is another way to make money online. But to make more than a minimal amount of money, you need to write a very interesting page (e.g. Markus Kuhn’s page on A4 paper is the most popular page on the Cambridge University Computer Laboratory security group’s site), or else arrange to be high up in the search results.

Now a mob called Privila sent Dr Clayton spam asking him to link to them (more links to them improve how much they get per ad click). So he investigated them. They have a clever technique for, how shall I put this, “leveraging” Google Ads in order to make money. (Imp note: before anyone asks why I’m seemingly helping them out with the link in the first sentence, note the rel=nofollow!)

Privila’s business model is to buy up existing domain names that have expired, paying attention to links in to those domains, their ranking with Google etc. They then get people to create content that fits the domain name, e.g. kitchencabinetswisconsin.com (kitchen cabinets!) or theaccidents.com (car accidents) etc, and they fill those pages with advertisements.

Here’s the extra cunning thing – Privila get people to write content for them for free, by advertising for (unpaid) “interns”, e.g. recent graduates from university, journalism courses etc. Interns are given assignments like 3 articles a week, which are posted on the Web under their bylines, the attraction for them being that they can supposedly build up their CVs to show to potential employers. A great way to get people to work for you for free and make money off their content from Google Ads at the same time! (Whether the writers can write well or know anything about the diverse random subjects they’re asked to write on, e.g. computer security, is another matter…)

Dr Clayton’s team built a model of this, and found that Privila have about 300 or 400 domains, and about 100-150 writing for them as “interns”. (UPDATE Imp note: for more by Dr Clayton’s team on Privila, from Light Blue Touchpaper see this and this.)

Although some might think this sort of thing was rather evil because it could be seen as exploitative, it’s at the legal end of the spectrum, and indeed is probably very all American.

A very illuminating talk, indeed. I wish I’d had the chance to ask questions there. I’d like to know things like:

Why is it expensive for phishers to send spam? I thought email was pretty much free…
How do the phishers avoid being tracked down through their domain name purchases?

admin | Dec 08, 2009 | Comments 1

That Special Time of Year

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Tim plays Santa:

A merry listener in the PaulDotCom IRC channel asked:
[Dear Santa]…is there a way to delete certain characters in a for loop from cmd.exe (such as nul, tab, etc)?

Santa slightly nods and begins his exclaims, “Now, Dasher! Now, Dancer! Now, Prancer, and Vixen! On, Cmd! On, For Loop! On, Donner and PowerShell! To the top of the terminal! to the top of the screen! Now bash away! bash away! bash away all!”

Anyway, enough of that crazy old guy.

The question was how to do it from the standard Windows command line, but indulge me for a minute and let’s see what PowerShell can do.

Santa has a text file containing the data below, the bits in brackets represent special characters that we want to remove.

Grandma[space]Got[tab]Runover[*]By[']A["]Reindeer[0x06]Last[nul]Night.

So the goal is to delete these special characters. The question is, “How does PowerShell handle special characters?” The answer is the backtick character (by the Esc and 1 keys), and here is a list of special characters and the associated escape sequence:

`n New line `r Carriage return `t Tabulator `a Alarm `b Backspace `' Single quotation mark `" Double quotation mark `0 Null `` Backtick character

These can be used in our regular expression to remove the special characters from our file. Here is how we do it.

PS C:\> Get-Content test.bin | % { $_ -replace " |`t|\*|`'|`"| |\0x06|`0", "" } GrandmaGotRunoverByAReindeerLastNight.

Inside our ForEach-Object (alias %) we use the replace operator to find all of our special characters and replace them with nothing (a.k.a. deleted). For those of you not familiar with regular expressions, the pipe character (|) is a logical OR, so any/all the characters will be replaced with an underscore. To represent the ASCII ACK character (0×06) in the regular expression we use \xNN, where NN is the hexadecimal ASCII code.

We removed the special characters from the text that was read from the file, but we didn’t actually change the file. Here is how we do that:

PS C:\> (Get-Content test.bin) | % { $_ -replace " |`t|\*|`'|`"| |\0x06|`0",""} | Set-Content test.bin

There is one very importantly subtlety that can be very easily overlooked. Notice the parentheses used in the first portion of the command. This is necessary so that all of the content is loaded in to memory before it is passed down the pipeline. If we don’t do that the command will attempt to write to the file which it is currently reading and will throw an error.

PS C:\> Get-Content test.bin | % { $_ -replace " |`t|\*|`'|`"| |\0x06|`0",""} | Set-Content test.bin Set-Content : The process cannot access the file 'C:\test.bin' because it is being used by another process.

It is pretty easy with PowerShell, now lets take a look at the Windows command line.

Windows command line

We will start off by using the same file as above; and we will use the standard Windows command line parser, the For Loop.

To see how the For Loop and variables handle the special characters, we will do a quick test of the For Loop without using any delimiters.

C:\> for /f "delims=" %a in ('type test.bin') do @echo %a Grandma Got Runover*By'A"Reindeer[0x06]Last

Oh No! Notice that we lost the last word. This happened because in the Windows command line variables are null terminated, meaning that the NUL character is deemed to be the end of the string so nothing beyond it will be processed. So we can’t work with the NUL character, first strike on Santa’s list.

Now, lets try to remove those other pesky characters.

C:\> for /F "tokens=1-8 delims=*^' " %a in ('type test.bin') do @echo %a%b%c%d%e%f%g%h GrandmaGot RunoverByA"Reindeer[0x06]Last

So we can’t represent the tab character, the double quote, or the special character either! Usually the caret character can be used to escape special characters, like the single quote. But for some reason it won’t work to escape the double quote. Second strike on Santa’s list.

However, we do have a work around for the tab character. We can tell cmd to disable file and directory name completion characters so we can type the tab character. All we have to do is tell cmd to F off.

cmd.exe /F:off

Unfortunately, this can’t be prepended to our other command and has to be a separate instance. But now we can type the tab character. All we have to do is add it as a delimiter and we are good to go.

C:\> for /F "tokens=1-8 delims= *^' " %a in ('type test.bin') do @echo %a%b%c%d%e%f%g%h GrandmaGotRunoverByA"Reindeer[0x06]Last

One more problem, we can only remove so many characters from a line. Why? Because only the variables a-z are available to represent the tokens.

Given a file with this content:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

To remove the space we would have to use this command:

C:\> for /F "tokens=1-26" %a in ('type test.txt') do @echo %a%b%c%d%e%f%g%h%i%j%k%l%m%n%o%p%q%r%s%t%u%v%w%x%y%z 1234567891011121314151617181920212223242526

Uh Oh! We lost 27 and 28 since we don’t have a way to represent them. Strike three on Santa’s list.

We can preserve the rest of the line, but we can’t remove the remaining “space” characters.

C:\> for /F "tokens=1-25*" %a in ('type test.txt') do @echo %a%b%c%d%e%f%g%h%i%j%k%l%m%n%o%p%q%r%s%t%u%v%w%x%y%z 1234567891011121314151617181920212223242526 27 28

In the above command the first 25 tokens are now represented by a-y. The 26th token is the remainder of the line, and is represented by z. Ugh!

Too bad cmd.exe is missing a nice easy way to do this, but there are a lot of things missing from cmd. I wish it had a compass in the stock and a thing which tells time, but if it did I would probably shoot my eye out with it.

Hal says ‘Oh Holy… Night’:

Um. Wow. After reading Tim’s fearsome fu I am ever more thankful this holiday season that I spend the vast majority of my time working with the Unix shell. Oh and by the way, Tim, “Bash away, bash away, bash away all!” is my line.

Let’s review some basics. Producing “special” characters in the Unix shell is generally pretty easy using some common conventions. On such convention is the use of backwhacks (“\”) to protect or “escape” special characters like quoting characters from being interpolated by the shell. Backwhacks can also be used in special escape sequences such as “\t” and “\n” to represent characters like tab and newline. And failing all of those options you can use “\xnn” or “\nnn” to represent arbitrary ASCII codes in either hex or octal.

All of these conventions can be demonstrated with a simple “echo -e” command to output Tim’s example input. The “-e” option is necessary here to force proper interpolation of the various backwhacked sequences:

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | cat -A Grandma Got^IRunover*By'A"Reindeer^FLast^@Night.$

The space is represented by a literal space and the tab by “\t”. I’ve enclosed the expression in double quotes so we don’t need to backwhack either the “*” or the single quote “‘”, but we do need a backwhack in front of the “literal” double quote that we want to output (so that it doesn’t terminate the double-quoted expression too early). The remaining special characters are produced using their ASCII values in hex and octal. I’ve piped the output into “cat -A” so that you can more easily see the special characters in the output.

The general quoting rules and escape sequences can vary slightly with different commands. For example, one way to strip characters from our sample input line is with “tr -d”. However, while tr understands octal notation for specifying arbitrary characters, it doesn’t handle the “\xnn” hex notation. This is not a huge deal, and we could just write:

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | tr -d " \t*'\"\006\000" | cat -A GrandmaGotRunoverByAReindeerLastNight.$

Again I’m using “cat -A” to confirm that I really did strip out all the characters we wanted to remove.

If you really, really insist on having the “\xnn” hex escape sequence, you could use the special $’…’ quoting notation in bash that forces interpolation using the same rules as “echo -e”:

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | tr -d $' \t*\'"\x06\\000' | cat -A GrandmaGotRunoverByAReindeerLastNight.$

Notice that I now had to backwhack the single quote in my list of characters, but I was able to drop the backwhack in front of the double quote. Also notice that while you can write “\x06″ inside of $’…’, you need double backwhacks in front of the octal.

Another way to remove characters from a line is with sed. However, while sed understands the “\xnn” hex notation, it doesn’t grok specifying characters with octal:

$ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | sed "s/[ \t*\'\"\x06\\000]//g" | cat -A GrandmaGotRunoverByAReindeerLast^@Night.$ $ echo -e "Grandma Got\tRunover*By'A\"Reindeer\x06Last\000Night." | sed "s/[ \t*\'\"\x06\x00]//g" | cat -A GrandmaGotRunoverByAReindeerLastNight.$

Even with these annoying little inconsistencies, life with bash, tr, and sed is infinitely preferable to the lump of coal my co-authors have to deal with.

So Merry \x-Mas to all, and to all a good night!

admin | Dec 08, 2009 | Comments 0

Malware that operates as a Firefox extension

Here’s a good overview of malware that operates as a Firefox extension. The PDF is by Symantec via Lenny Zeltser.

Firefox Malware (33)

admin | Dec 07, 2009 | Comments 0

Joining Up

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal fields a question from IRC

Mr. Bucket passed along the following query from the PaulDotCom IRC channel:

What functionality is available to loop through multiple files, and write the output to a single file with some values on the same line? Ex: If one program gives me the hash of a file, and the other program outputs the name/size/etc of a file, can I output to the same file HASH-FileName-Size

I couldn’t resist chortling with glee when this question came up, because it’s another one of those “easy for Unix, hard for Windows” kinds of tasks. I never can resist sharing these “learning experiences” with my fellow co-authors.

First let’s review our inputs. I’m going to use the openssl utility for generating checksums, since it’s fairly generic to lots of different flavors of Unix at this point:

$ openssl sha1 * SHA1(001.jpg)= a088531884ee5eb520e98b3e9e18283f29e13d25 SHA1(002.jpg)= 77febb1498b2926ee6a988c97f3457e38736456d SHA1(003.jpg)= 922bcb001d025d747c2ee56328811a4270b62079 ...

As you can see, it’s pretty easy to generate a set of checksums over my directory of image files, but there’s a bunch of cruft around the filename that’s not really helpful. So let me get rid of that with some quick sed action:

$ openssl sha1 * | sed -r 's/SHA1$(.*)$= (.*)/\1 \2/' 001.jpg a088531884ee5eb520e98b3e9e18283f29e13d25 002.jpg 77febb1498b2926ee6a988c97f3457e38736456d 003.jpg 922bcb001d025d747c2ee56328811a4270b62079 ...

That’s better! In the sed expression I’m using the “(.*)” sub-expressions to match the file name and the checksum in each line, and the substitution operator is replacing the original line with just the values of the sub-expressions. Slick.

Now that we’ve got the checksums, how do we produce the file sizes? I could just use “ls -l” of course. But since the questioner seems to only want “HASH-FileName-Size”, I may as well just use “wc -c” to produce simpler output:

$ wc -c * 4227504 001.jpg 4600982 002.jpg 4271719 003.jpg ...
Now that I know what my inputs are going to be, the question is how to stitch them together? Luckily, Unix includes the join command for putting files together on arbitrary fields (we last saw the join command back in Episode #43). Now I could save the checksum output and the file sizes to separate files and then join the contents of the two files, but bash actually gives us a cooler way to handle this:

$ join -1 1 -2 2 <(openssl sha1 * | sed -r 's/SHA1$(.*)$= (.*)/\1 \2/') <(wc -c *) 001.jpg a088531884ee5eb520e98b3e9e18283f29e13d25 4227504 002.jpg 77febb1498b2926ee6a988c97f3457e38736456d 4600982 003.jpg 922bcb001d025d747c2ee56328811a4270b62079 4271719 ...

See the “<(…)” syntax? That’s a little bit of bash file descriptor magic that allows us to substitute the output of a command in a place where a program would normally be looking for a file name. In this case it saves us the hassle of having to create intermediate output files to join together. The join command itself is pretty simple. We’re telling the program to join the output of the two commands using the file names in the first field of input #1 and the second field of input #2. The only problem is that the join command isn’t producing the “HASH-FileName-Size” output that the original questioner wanted. That’s because join always outputs the joined field first, followed by the remaining fields from the first input (the checksum in this case), followed by the remaining fields from the second input (the file size). We’ll have to use a little awk fu to re-order the fields:

$ join -1 1 -2 2 <(openssl sha1 * | sed -r 's/SHA1$(.*)$= (.*)/\1 \2/') <(wc -c *) \ | awk '{print $2 " " $1 " " $3}' a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504 77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982 922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719

… Mmmm, that’s a tasty little bit of shell magic, isn’t it? Let’s see what Ed and Tim are cooking up.

Ed retorts snidely:

Choosing a topic just because you think it’s hard for us Windows guys, huh, Hal? Well, aren’t you just a big ball of sunshine, a command-line Scrooge this holiday season? When I first read this one, I though… “Ugh… this is gonna be hard.” Perhaps I was psyched out by your juvenile trash talk. Or, maybe I’ve just been hanging around in cmd.exe too long, and have gotten used to hard problems.

But, this one turned out to be surprisingly straight-forward and even non-ugly (well, beauty is in the eye of the beholder, I suppose). Here’s the skinny:

C:\> FOR /f "tokens=1-2" %a in (name-hash.txt) do @for /f "tokens=1,2" %m in (length-name.txt) do @if %a==%n echo %b %a %m a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504 77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982 922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719

I’m assuming that name-hash.txt contains, well, names and hashes, one pair per line. Likewise, length-name.txt contains lengths and names, again one pair per line.

As we know, FOR /F loops can parse through all kinds of crap, including the contents of files. I use a FOR /F loop with two tokens (giving me two variables) of %a (for the file name) and %b (allocated automagically, holding the hash). For each of the files described in name-hash.txt, I then construct the body of my FOR loop. It contains another FOR /F loop, again with two variables (the original question mentioned “etc” for extra stuff there… if you have more stuff, just up the number of tokens and echo the proper variables at the end). My inner FOR /F loop iterates through the length-name.txt file, placing its values in the variables %m (length) and %n (name).

Now, if I just echoed out %a %b %m %n, I’d be making all of the possible combinations of every pair of two lines in the original files. But, we want to pare that down. We only want to generate some output if the name from name-hash.txt (%a) matches the name from length-name.txt (%n). We do this with a little IF operation comparing the two variables. If they match, we then echo out hash (%b), name (%n), and size (%m).

Admittedly, the performance of this little command isn’t great, as I have to run through every line of name-hash.txt, comparing the name by running through the entirety of length-name.txt. I don’t stop when I’ve found a match, because, well, there could be another match somewhere. Also, if there is no match of the name between the two files, my command ignores that name, not issuing any output. But, I think that makes sense given what the questioner asks.

So, Tim… does PowerShell have a nifty little built-in or something to make this easier than running through a couple of FOR loops? Inquiring minds what to know.

Tim tags in for Ed:

For loops! We don’t need no stinking For loops!

The first thing to do is import the files. Since there is a space between the columes we can use Import-CSV with a delimiter of the space character. Also, there is no header information so we have to specify it.

PS C:\> Import-Csv length.txt,hash.txt -Delimiter " " -Header File,Data File Data ---- ---- 001.jpg 4227504 002.jpg 4600982 003.jpg 4271719 001.jpg a088531884ee5eb520e98b3e9e18283f29e13d25 002.jpg 77febb1498b2926ee6a988c97f3457e38736456d 003.jpg 922bcb001d025d747c2ee56328811a4270b62079 ...

We have all the data, so now it can be grouped by the file name using Group-Object (alias group).

PS C:\> Import-Csv length.txt,hash.txt -Delimiter " " -Header File,Data | group file

Count Name Group ----- ---- ----- 2 001.jpg {@{File=001.jpg; Data=4227504}, @{File=001.jpg; Data=a088531884ee5eb520e98b3e9e18283f29e13d25}} 2 002.jpg {@{File=002.jpg; Data=4600982}, @{File=002.jpg; Data=77febb1498b2926ee6a988c97f3457e38736456d}} 2 003.jpg {@{File=003.jpg; Data=4271719}, @{File=003.jpg; Data=922bcb001d025d747c2ee56328811a4270b62079}} ...

We have the data grouped like we want, but we still need to massage it a bit so we can get the formate we want.

PS C:\> Import-Csv length.txt,hash.txt -Delimiter " " -Header File,Data | group file | Select @{Name="Hash";Expression={$_.Group[1].Data}}, Name, @{Name="Length";Expression={$_.Group[0].Data}} Hash Name Length ---- ---- ------ a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504 77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982 922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719 ...

The Select-Object (alias select) cmdlet allows for custom expressions which was used to get the hash and the length. The “Group” object contains multiple items and each can be access by its index value, 0 is the length and 1 is the hash.

Fileless PowerShell

The initial task was to get the file name, length, and hash from separate files and combine them in to one. Let’s try this again without using files.

This would be very easy if powershell just had a hashing cmdlet, but it doesn’t. However, we can do hashing by using the .NET library and some very ugly PowerShell. Maybe in v3 we will get a Get-Hash cmdlet, but it seems as likely as the addition of Get-Unicorn or Get-MillionDollars.

So we need some hash, but not the kind that is illegal in 49 states, we need the hash of a file. Here is how we get it.

PS C:\> PS C:\> gci 001.jpg | % { (New-Object System.Security.Cryptography .SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) }

We use the SHA1CryptoServiceProvider .NET class, but it adds another bump since it doesn’t take files as input and will only take a stream. It isn’t hard to get the stream though, all we need to use is the OpenRead method of our file object. If that wasn’t enough, there is another problem, the output.

PS C:\> PS C:\> gci 001.jpg | % { (New-Object System.Security.Cryptography .SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) } 160 136 83 24 ...
The result is an array of bytes. So we have to convert that to hex and combine it together.

PS C:\> gci 001.jpg | % {$hash=""; (New-Object System.Security.Cryptography .SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) | % { $hash += $_.ToString("X2") }; $hash} a088531884ee5eb520e98b3e9e18283f29e13d25

We use the ToString method with the format string X2 to convert each byte to hex. The X converts it to hex, and the 2 will make sure the output is two characters wide (0A vs A). We then use the variable $hash to stitch our bytes together to get the full hash.

Now let’s see the full command.

PS C:\> gci *.* | select @{Name="Hash";Expression={$hash=""; (New-Object System.Security.Cryptography.SHA1CryptoServiceProvider).ComputeHash($_.OpenRead()) | % { $hash += $_.ToString("X2") }; $hash}}, name, length Hash Name Length ---- ---- ------ a088531884ee5eb520e98b3e9e18283f29e13d25 001.jpg 4227504 77febb1498b2926ee6a988c97f3457e38736456d 002.jpg 4600982 922bcb001d025d747c2ee56328811a4270b62079 003.jpg 4271719 ...

The first thing we do is get all the files in the currect directory using Get-ChildItem (aliased as gci or dir). That is piped in to Select-Object (aliased as select) to get the hash, filename, and size. The Select-Object cmdlet allows us to get properties of the pipeline object as well as creating a custom expression. In our case we will use the custom expression to calculate the hash.

Our results are in object form and can be piped to a file with Out-File or Out-Csv.

So the task is complete, but let’s pretend for a second we had the fictional Get-Hash cmdlet. If we had our leprachaun our command might look something like this:

PS C:\> gci *.* | select @{Name="Hash";Expression={Get-Hash $_ sha1}, name, length

If only getting hash was easier in Windows.

admin | Dec 01, 2009 | Comments 0

The Tangled Web

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal gets a soft one this week

Lately we’ve had some of our loyal readers– mostly the Windows folk– asking about command-line tools for accessing web pages. When these questions come up, I just smile serenely, because it’s easy to do this in Unix. Ed and Tim on the other hand turn a little green and start sweating.

Among the Unix tribes, there seem to be two popular command-line tools for grabbing web pages. Some folks prefer curl, but I generally stick with wget since it’s the tool I learned first. Both curl and wget support HTTP, HTTPS, and FTP as well as proxies of various types along with different forms of authentication. curl also supports other protocols like TFTP, FTPS (FTP over SSL), and SCP and SFTP.

Using wget couldn’t be simpler:

$ wget blog.commandlinekungfu.com --2009-11-18 18:47:51-- http://blog.commandlinekungfu.com/ Resolving blog.commandlinekungfu.com... 74.125.113.121 Connecting to blog.commandlinekungfu.com|74.125.113.121|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `index.html'

[ <=> ] 173,805 18.3K/s in 9.3s
2009-11-18 18:48:01 (18.3 KB/s) - `index.html' saved [173805]

Or, if you don’t like all the noisy chatter, you can use the “-q” option:

$ wget -q blog.commandlinekungfu.com $ ls index.html index.html.1

Notice that wget doesn’t overwrite the first index.html file we downloaded. Instead it appends a uniqe number to the file name. If we downloaded another index.html file, it would end up as index.html.2 and so on. You can also use “-O” to specify an output file name.

Maybe my favorite feature, however, is the “-r” (recursive) option that allows me to grab an entire tree of content in a single operation. So all I have to do when I want to back up the content here at Command Line Kung Fu is just run “wget -r -q blog.commandlinekungfu.com” and wait a few minutes for everything to download.

There are lots of other neat features to wget, but I think I’ve already demoralized my Windows brethren enough for one Episode. Let’s see what Ed and Tim cook up.

Tim orders take-out:

Unfortunately (again), there isn’t a built-in cmdlet to do the equivalent of wget or curl, but we can access the .NET libraries and recreate some of the functionality of the Linux commands. By default, the .NET Framework supports URIs that begin with http:, https:, ftp:, and file: scheme identifiers, so it isn’t quite as full featured as Linux, but it is all we have.

PS C:\> (New-Object System.Net.WebClient).DownloadString( "http://blog.commandlinekungfu.com") ...
This will grab files in text format and it can be used further down the pipeline, such as saving the file by piping it in to Out-File. What if we want to grab non-text files or just download the file? We can use the DownloadFile method and specify where we want to save the file.

PS C:\> (New-Object System.Net.WebClient).DownloadFile( "http://blog.commandlinekungfu.com","c:\downloadedfile.html")
What happens if the file doesn’t exist? It raises an 404 error and the file (obviously) isn’t opened.

PS C:\> (New-Object System.Net.WebClient).DownloadString( "http://blog.commandlinekungfu.com/NonExistant.file")

Exception calling "DownloadString" with "1" argument(s): "The remote server returned an error: (404) Not Found." At line:1 char:49 + (New-Object System.Net.WebClient).DownloadString( <<<< "http://blog.commandlinekungfu.com/NonExistant.file")

Sadly, there isn’t way to recursive requests without doing some heavy duty scripting.

Ed Sits Here Trying Not To Think About What Hal Called His Part of this Article:

Ahhh… an age-old question. I remember about a year ago, I was having a discussion with Kevin Johnson, bon vivant and web attacker extraordinaire, when this issue came up. I posed the following question to KJ0 (as we call him): “Suppose you’ve just hacked a Windows box in a pen test. You have command shell access of said machine. Tell me what you want to do now, using only built in command-line capabilities.” I went further, boldly mixing my metaphors: “Consider me as /dev/WindowsCommandLine… where do you want to go today?”

Kevin shrugged, smiled, and said, “Easy… wget.”

I felt a piercing pain stab at my heart. “The wget tool has sooo many options… can we scale it back a bit? What do you really want to do?” Kevin said, “Sure… I just wanna fetch a web page and write it to the file system. How can I do that at the command line in Windows?”

I spent a little time fitzing around with various Windows commands, and ultimately settled on using the built-in Windows telnet client to formulate HTTP requests manually. It’s crude, but works.

“But, Ed,” you argue, “Microsoft removed the telnet client from Windows Vista and some of the 6 million versions of Windows 7.” I respond: Yes, it’s no longer installed by default, but on the professional versions of Vista and 7, you can run:

C:\> start /w pkgmgr /iu:"TelnetClient"

Even if it doesn’t finish, kill that cmd.exe, and you should have the Telnet Client ready to run. Note that the installation package for the telnet client is built-in, but the tool itself just isn’t installed. The machine doesn’t have to reach out to the Internet to get it. Also, note that /iu stands for install update, you need that :, and you should observe the case of the Cap-T and Cap-C in TelnetClient. Oh, and by the way… if you want the Telnet service, change “TelnetClient” to “TelnetServer”. To remove either after you’ve installed, it, run the same command, except substitute /uu: (uninstall update) for /iu:.

So, now equipped with the telnet client, let’s make an HTTP request.

Now, you might think that we should simply echo an HTTP request and pipe it into the telnet client, right? Bzzzzzt. Sorry, my friend, but the Windows telnet client doesn’t like anything to come into its Standard Input at the command line, and it certainly doesn’t send it to the target machine. Try sniffing it sometime. Another annoying part about the Windows telnet client is that it doesn’t like you to touch its Standard Output. The telnet client is somehow clairvoyant, and if you follow it with a > or a |, it knows and refuses to send any packets. Gee, thanks, Microsoft.

For this one, we’re going to have to go interactive.

C:\> telnet -f log.txt

I’m logging output to the file log.txt, where our output will reside.

At our new telnet prompt, we can open a connection to port 80 on our destination web server. We’ll use the “o” option to open a connection:

Microsoft Telnet> o www.google.com 80 Connecting To www.google.com...

Now, depending on the version of Windows you are using, it may or may not look like a connection is made. Either way, just trust that it has, and start typing your HTTP request. Typically, you’ll be getting a page, which you can do by entering:

GET [pagename] HTTP/1.1 Host: [TheHostName]

Hit Enter Enter at the end, and you’ll have the page displayed on your screen. Hit Enter again, and you’ll get back to your telnet prompt. Type “quit” to exit.

More importantly, you should now have your page in the output file log.txt. Yes, you’ll have the HTTP response header in front of it, but that’s not so bad. It’s ugly, but it’ll let you grab a file quickly from a server.

admin | Nov 24, 2009 | Comments 0

Man-in-the-middle attacks demoed on 4 smartphones

Security researchers from SMobile Systems have released a paper detailing successful man-in-the-middle attacks against several smartphones.

The SSL enabled log in sessions on the tested, Nokia N95, HTC Tilt, Android G1 and iPhone 3GS devices was sniffed using the publicly available SSLstrip tool, with the attack taking place over insecure Wi-Fi network, now prevalent literally everywhere. Here’s the scenario they used, and possible mitigation approaches:

“The attacker visits the same cafe that offers a free Wi-Fi hotspot and decides to employ basic host, network identification and enumeration tools from the laptop to enumerate all the active devices connected to the Wi‐Fi hotspot. From the results, the attacker notices a MAC address referring to a Nokia smartphone. The attacker know that there is little to no detection capabilities present on an overwhelming majority of smartphone’s in use today, so the owner would likely never find out about a successful man-in-the-middle- attack (MITM).

The well-informed attacker creates a successful MITM attack. In the meantime, the smartphone owner accesses the online bank website and enters the login credentials required to gain access to the banking information. In this scenario, all of the communication between the smartphone and the online bank site is routed through the attacker’s machine and the attacker can see the login details in plain text, as well as can capture all the sites accessed by the victim.”

The awareness-raising test aims to educate users on approaching convenient and free, public Wi-Fi networks with caution, emphasizing on how their mobile service provider’s 3G connection, or the one offered by a trusted Wi-Fi network should always be considered as their first choice.
Anyway, just how insecure or susceptible to compromise are the majority of Wi-Fi networks found on high-trafficked locations such as airports or international cities? The answer is sadly, self-evident with data backing it up available publicly.

Go through related posts: GPU-Accelerated Wi-Fi password cracking goes mainstream; D-Link router’s CAPTCHA flawed, WPA passphrase retrieved; Survey: 88% of Mumbai’s wireless networks easy to compromise

Last year, AirTight Networks conducted a major wireless network security study by visiting 14 airports (11 in the U.S and 3 in the Asia-Pacific) and found out that a huge percentage of the 478 Wi-Fi Access Points analyzed are either open, or using outdated encryption protocols. Even more interesting was the fact that users were falling victims to “viral” Wi-Fi networks using descriptive and lucrative names seeking to establish legitimacy.

The prevalence of such “handy”, but easy to compromise Wi-Fi networks internationally, is virtually the same. For instance, similar wardriving tests conducted in Paris; Santiago, Chile; China; Monterrey — Mexico, Sao Paulo – Brazil, Caracas (Venezuela), Warsaw, and London offer similar insights into the “security” of such public networks.

Possible mitigation practices? According to Marlinspike, the author of the tool:
Cautious users, for example, have been advised to explicitly visit https URLs or to use bookmarks in order to protect themselves from sslstrip, while other SSL/TLS based protocols such as imaps, pop3s, smtps, ssl/irc, and SSL-based VPNs never present an opportunity for stripping. This talk will outline some new tools and tricks aimed at these points of communication, ultimately providing highly effective attacks on SSL/TLS connections themselves.
How often do you face the trade-off of using a public, and possible insecure Wi-Fi hotspot, for the sake of convenience instead of sticking to your 3G data plan, even when traveling abroad?

Have you ever avoided using your mobile device and instead used your laptop at an airport, due to your host-based firewall’s better ARP filtering features — if any — enabling the detecting of changed MAC address for a (trusted) gateway network adapter in order to detect possible MItM attempts?

How EV SSL-aware is your E-banking provider, especially if you’re E-banking over a mobile device? Or do you simply “VPN-and-forget” over a public Wi-Fi network?

admin | Nov 17, 2009 | Comments 0

Destroy All Connections

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Ed looks out on the serene waters of Tokyo Bay:

Mr. Byte Bucket sent in a request from the ever insightful Pauldotcom IRC channel:

Can anyone suggest a Windows cmd to disconnect a specific socket?

Nice question! Unfortunately, Windows doesn’t offer much in the way of built-in tools that are fine grained enough to operate at the socket level. But, we can either restart the service handling the connection, or, if you want to be a little more violent, just kill its service.

We’ll start by taking the violent rout. First off, we need to figure out what processid is associated with the connection you seek. You can get a list of all connections to a given port using the following command:

C:\> netstat -nao | find ":[port]"

The right-most (i.e., fifth) column is the processid number.

If you have multiple different clients clients connected to that same destination port, you can select out the given process that is associated with a specific client connection using:

C:\> netstat -nao | find ":[port]" | find "[ClientIPaddr]"

You can then kill that process using wmic. However, be very careful! That process may be really important, and killing it could end you up in a world of hurt. In fact, a lot of Windows built-in features (such as file sharing and IIS) are all associated with the “System” service (with a PID of 4 or 8 depending on the version of Windows you are using).

C:\> wmic process where processid="[PID]" delete

You can wrap this all up in one very dangerous command… but I really don’t recommend doing this. Again, if you inadvertently kill the wrong process, your system could come crashing down around you. I recommend figuring out what process is at issue first, investigating it, and only then killing it.

C:\> for /f "tokens=5" %i in ('netstat -nao ^| find ":[port]" ^| find "[ClientIPaddr]"') do @wmic process where processid="%i" delete Deleting instance \\WILMA\ROOT\CIMV2:Win32_Process.Handle="[PID]" Instance deletion successful.

Our second approach involves restarting the service associated with the process. Based on the processID information we retrieved above, we can get a list of the associated services by running:

C:\> tasklist /fi "pid eq [PID]" /svc

That command tells us all of the services that are associated with a given process. We’ll then have to do a little research to figure out which specific service it is that is handling our given connection. Unfortunately, Windows doesn’t give us the ability to map a connection directly to a service, but we must instead map a connection to a process, which we then map to a set of services running from within that process. Now, if there’s only one service in that process, we’re golden. But, if there are more, we have to do this research step to find out which service it really is that we need to focus on.

Once you’ve discovered the appropriate service, you can then restart the service using:

C:\> sc stop [service] & sc start [service]

It’s kind of annoying, I think, that there is no “restart” option explicitly here, so we have to stop the service and then start it. Not a huge liability, but still a little frustrating.

Tim Sees Something Starting to Stir in the Otherwise Tranquil Waters of the Bay:

PowerShell doesn’t (yet) have any cmdlets similar to netstat. The Grand Poobah of PowerShell (@halr9000 on twitter) has a PowerShell script to “objectize” netstat, but that crosses in to scriptland so we will steer clear.

To find the connection and its associated process you will have to refer to the first portion of Ed’s section. But once we have the process id we can use PowerShell cmdlets.

So we have the PID, now we can get the process object by using the aptly named command Get-Process. The default method for Get-Process is the process name, so we need to use the Id parameter.

PS C:\> Get-Process -Id 3004 Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName ------- ------ ----- ----- ----- ------ -- ----------- 210 7 5116 13344 61 6.67 3004 Evil

The process can be killed using PowerShell, but as Ed said, “Be very careful!” The Stop-Process cmdlet’s default method is the process Id so we aren’t required to use Id parameter (but you can if you wish).

PS C:\> Stop-Process 3004

If the offending process is a service, we can’t retrieve the service from the process id using by using just Get-Proccess since it doesn’t include the Id property. However, we can use wmi in conjunction with Get-Service to “get” the service.

PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv ice $_.Name }

To paraphrase Ed (again), you will have to do some research to since there can be multiple services running from within that process (multiple services be in one process).

Once we find the service that needs a kick, we can stop it using Stop-Service. We also have the ability to restart the service using Restart-Service.

If you felt gutsy you could pipe the command above into Restart-Service or Stop-Service.

PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv ice $_.Name } | Restart-Service

PS C:\> Get-WmiObject win32_service | ? { $_.ProcessID -eq 3004 } | % { Get-Serv ice $_.Name } | Stop-Service

..or you could do it manually.

PS C:\> Stop-Service [Service]

PS C:\> Restart-Service [Service]

It would be nice if there was a facility in PowerShell to just kill a connection. Maybe we can get that in version 3, but while we wait for the additional capability I get the uneasy feeling Hal is getting ready to squash us like bugs.

And the waters of Tokyo bay begin to boil:

Oh dear. I really am going to have to open my Godzilla-sized can of whup-ass on my poor Windows-using bretheren. But first, let me try to make them feel not so bad by doing a quick netstat-based solution.

On Linux systems at least, netstat has a “-p” option to display process information. Let’s take a quick look at some sample output:

# netstat -anp --tcp -4 | grep :22 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 15805/sshd tcp 0 0 192.168.1.4:60279 192.168.1.2:22 ESTABLISHED 18054/ssh tcp 0 0 192.168.1.4:32921 192.168.1.2:22 ESTABLISHED 19409/ssh

Here I’m listing all TCP sockets that are using IPv4 (“–tcp -4″), and using “-n” so that the socket numbers are not converted into human-readable names. Anyway, as you can see, the 7th column is “PID/name” (this is what the “-p” option does).

So with the help of awk and cut, we can pull out just the PID of the master SSH daemon:

# netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f1 -d/ 15805

Killing that process just requires appropriate use of backticks:

# kill `netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f1 -d/`

We could also use cut to pull out the second field, if we wanted to shut down the process using it’s init script:

# /etc/init.d/`netstat -anp --tcp -4 | awk '/:22/ && /LISTEN/ { print $7 }' | cut -f2 -d/` stop bash: /etc/init.d/sshd: No such file or directory

Rats, it’s /etc/init.d/ssh on this system, and not /etc/init.d/sshd, but you get the idea.

But really, the 300 foot giant mutated atomic lizard we want to employ here is lsof. You might be aware that we can get lsof to show us just the port 22 stuff with it’s “-i” flag:

# lsof -i :22 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 15805 root 3u IPv6 37028 TCP *:ssh (LISTEN) sshd 15805 root 4u IPv4 37030 TCP *:ssh (LISTEN) ssh 18054 hal 3u IPv4 44514 TCP elk.deer-run.com:60279->deer.deer-run.com:ssh (ESTABLISHED) ssh 19409 hal 3u IPv4 53249 TCP elk.deer-run.com:32921->deer.deer-run.com:ssh (ESTABLISHED)
If we wanted to kill the SSH daemon, we could easily adapt the awk fu from the netstat example to pull out the appropriate PID value and then use backticks to feed this value into the kill command.

But we don’t need awk, because lsof has the “-t” (“terse”) option, which just spits out the PIDs:

# lsof -t -i :22 15805 18054 19409

The “-t” option is specifically designed so that you can do things like “kill `lsof -t -i :22`”. Of course, the problem with doing that is that I’d also end up killing my SSH sessions to the remote machine deer.deer-run.com– PIDs 18054 and 19409– which I don’t really want to do.

So how do we pull out just the daemon PID? Well looking at the lsof output above, I can see that I want to kill the sshd processes that are listening on port 22 but leave the regular ssh processes alone. I could use the “-c” option to select the PIDs by command name:

# lsof -t -c sshd 15805

But that means I’d have to already know the command name associated with the port in question– in this case by having already run lsof once to search by port number. And if I knew the command name already, why wouldn’t I just use pkill (see Episode #22) instead of lsof?

What I really want is a single command-line using just kill and lsof that lets me reliably destroy the master server process as effectively as Godzilla destroys Tokyo. Luckily, lsof has some more atom-age madness up its sleeve. You see, lsof’s “-c” option isn’t just limited to simple substring matching: you can use “/…/” to specify a full-on egrep-style regular expression. Because Unix server processes almost always end in “d” (think sshd, httpd, ftpd, and so on), we can construct a similar regular expression to match the daemon process name associated with an arbitrary port number:

# lsof -a -i :22 -c /d$/ COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 15805 root 3u IPv6 37028 TCP *:ssh (LISTEN) sshd 15805 root 4u IPv4 37030 TCP *:ssh (LISTEN)

The “-a” option means to logically “and” your search criteria together (the default is logical “or” for some strange reason, although this is almost never what you want). So here we’re looking for everything using port 22 and where the process name ends with “d”.

Adding the “-t” option and some backtick action, we have our final answer:

# kill `lsof -t -a -i :22 -c /d$/`

And with that, I shall swim home to Monster Island and await my next challenge.

admin | Nov 17, 2009 | Comments 0

How to Disrupt a Botnet

How to Disrupt a Botnet by Lenny Zeltser

The following note is inspired by the steps the folks at FireEye Malware Intelligence Lab took to disable the Mega-d/Ozdok bot network. People often wonder what it takes to shut down a botnet. Here are the key steps, which apply to “traditional” botnets, which don’t rely heavily on peer-to-peer protocols for their command and control (C&C) implementation; the number of hosts and domains that such botnets use can be sufficiently small that a group or an individual can disrupt the botnet by getting these IPs or domain names shut down.

Note that attempting to interfere with operations of a profitable botnet can be dangerous, as your actions may cause attackers to retaliate. Therefore, consider these steps as informational thoughts, rather than an encouragement to follow FireEye’s footsteps.

Obtain a copy of the bot through forensic analysis of a compromised system. It helps to get hands on several instances of the malicious program, in case multiple variants possess meaningful behavioral differences.
Understand the bot’s command and control mechanism. How does the attacker control the botnet? Reverse-engineer the malicious program to understand the C&C protocol and to get a sense for the commands the botnet understands. You may find a way to authenticate to the botnet and, posing as the attacker, commandeer it. (Warning: As Andre posted in the comments, “Logging on to network that is not your own, and issuing commands to take it over could potentially be considered illegal access.”)
Identify which systems, if taken off line, could disrupt the botnet. To accomplish this, look for weaknesses in the command and control implementation, such as the reliance on a small set of servers to distribute commands or weakness in the C&C servers’ IP or domain names generation algorithm. (You may recall how researchers at UC-Santa Barbara gained control over an instance of the Torpig botnet.)
Contact ISPs hosting suspected C&C servers. In your correspondence with them, present documentation that supports your claim that the systems they are hosting are being misused. Be specific about which IPs violate the ISP’s policy by acting maliciously and should be disabled.
Contact registrars of C&C domains. In your correspondence with them, present documentation that supports your claim that the domains they are hosting are being misused. Be specific about which domains violate the registrar’s policy by being used for malicious purposes and should be disabled.
Consider registering unused domains that the botnet’s C&C mechanism may attempt to use later. This can be expensive, depending on the number of domain names associated with the botnet’s C&C implementation.

Botnets come in different shapes, sizes, and flavors. The steps above don’t apply to all of them, but they should give you a sense for how defenders can take action against traditional botnets. For an example of these steps in the context of a specific botnet, see the “Smashing the Mega-d/Ozdok botnet in 24 hours” write-up by FireEye.

admin | Nov 10, 2009 | Comments 0

Help Fu

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Tim hits lead off:

This week we get help with help. Everyone needs help, and some in different ways than others. But we aren’t going to discuss Ed’s problem with crunchy peanut butter or Hal’s issue with furry animals with tails. Anyway, back to the issue at hand.

Instead of jumping right in to getting help on commands, let’s use PowerShell to find commands by using the Get-Command cmdlet. This command will retrieve all the cmdlets, aliases, and functions available to us. In order to use this command effectively, let’s go over a bit of the very basics of PowerShell. As you all probably know, the nomenclature for PowerShell cmdlets is verb-noun. Microsoft’s approved verb list defines the verbs so you know which verb to use and to ensure consistent use by cmdlet developers. That way the verbs are consistent so “get” is always used instead of a random choice of get, retrieve, grab, or obtain. The noun part of the name identifies the entity on which the action is performed.

Ed and Hal discussed the command history in episode #27, so we’ll use that as the focus of our research.

Let’s see a list of all the available nouns.

PS C:\> Get-Command -Type cmdlet | Sort-Object noun | Group-Object noun

Count Name Group ----- ---- ----- 2 Acl {Set-Acl, Get-Acl} 5 Alias {Import-Alias, Get-Alias, Set-Alias, Export-... ... 4 History {Invoke-History, Clear-History, Get-History,... ...

You probably could have already guessed, but the noun we want to use is “History.” To get a lits of the commands available using that noun we can use this command:

PS C:\> Get-Command -Noun history

CommandType Name Definition ----------- ---- ---------- Cmdlet Add-History Add-History [[-InputObject] ... Cmdlet Clear-History Clear-History [[-Id] Cmdlet Get-History Get-History [[-Id] ... Cmdlet Invoke-History Invoke-History [[-Id] Get-Command -Verb Get

Back to the subject at hand…

Now we have a list of cmdlets we are interested in. Let’s see how we can use the command by using the Get-Help cmdlet.

PS C:\> Get-Help Get-History

NAME Get-History

SYNOPSIS Gets a list of the commands entered during the current session.

SYNTAX Get-History [[-Id] ] [[-Count] ] [] ...
We have a bit of a problem here, the basic help isn’t super useful since it doesn’t provide a good description of the parameters or any examples of the cmdlet’s use. However, it does provide syntax and a list of parameters. To see the full help including examples and parameter descriptions use this command:

PS C:\> Get-Help Get-History -Full

To see just the examples:

PS C:\> Get-Help Get-History -Examples

As you have probably noticed in past episodes there are a lot of aliases available in PowerShell. The Get-Alias cmdlet can be used to “get” the list of aliases. Specifically, we can see the aliases for the commands using the History noun.

PS C:\> Get-Alias | Where-Object {$_.Definition -like "*history"}

CommandType Name Definition ----------- ---- ---------- Alias clhy Clear-History Alias ghy Get-History Alias h Get-History Alias history Get-History Alias ihy Invoke-History Alias r Invoke-History

The Get-Command cmdlet can be used to get the same results.

PS C:\> Get-Command -Type Alias | Where-Object {$_.Definition -like "*history"}

That’s about it for PowerShell, let’s see what Ed has for us with the Windows command line: Get-Help -FuMaster Ed -Subject Windows

Ed’s Output:

You guys with your fancy help capabilities in your shells are a real hoot. Yes, your shells include intuitive and easy-to-access help features. Where’s the fun in that? Or, more to the point, where’s the fu in that? I’ve spent many countless hours sitting at a cmd.exe trying to reverse engineer… uh… I mean conducting detailed design recovery of cmd.exe functionality to try to figure out what the heck the little bugger was really doing. Ah… good times.

In all seriousness though, cmd.exe offers an embarrassingly paltry amount of help for using it and its commands. Really, it’s kind of stinky. Let’s explore the few skimpy options we have.

First off, we can run the help command, followed by the command we want to analyze, as in:

C:\> help findstr

The good news? Well, we’ll see a brief description of findstr functionality, as well as its command flags.

The bad news? Well, where to start? How about:

Bad News #1) The help command doesn’t provide help for all of the commands available in cmd.exe. It only covers a little less than 80 of them.

To get a list of the commands that help has information about, simply run:

C:\> help

Some of the most interesting and useful commands of all, like netstat, netsh, reg, sc, and wmic, result in this unhappy little message:

C:\> help reg This command is not supported by the help utility. Try "x /?".

Bad News #2) The output of the help command doesn’t include any examples or much detail at all. It’s just really basic usage, without any interesting fu included.

Bad News #3) Note the little suggestion in the output of my command in Bad News #1: Windows tells us to try running the command with a /? argument for help. As an old Unix guy (shhhh… don’t tell anyone), I’m very hesitant to run a command that I don’t understand just so I can get help. It scares me to do so without knowing what just running the command might do. And, here’s a dirty little secret of the help command itself. It actually runs the command you are trying to get help about! Yes, check this out…

First, start Task Manager, move to the Processes tab, and sort it alphabetically by clicking on the Image Name column.

Now, in a separate cmd.exe, run the following command, which will get help for the findstr command, repeatedly:

C:\> for /L %i in (1,0,2) do @help findstr

Look in your Task Manager window… see it? Every once in a while, the findstr command pops up! So “help findstr” is actually invoking the findstr command itself to scrape out its help options. Yowza. That’s just painful for me.

Bad News #4: Well, with only around 80 commands covered by the help command, we often have to turn to that old stand by, “[command] /?” for more help. But the help included here is spotty at best as well, with major inconsistencies between commands and some commands supporting context-specific help for particular sub-areas of the command. For example, “wmic /?” does offer some help, but there is a wmic-specific option for getting more detailed help:

C:\> wmic /?:full

Besides wmic, none of the other commands I know about support this /?:full option.

Also, some commands have specific help for certain contexts within the command. Consider the following examples:

C:\> wmic process get /?

This shows you a list of all the process attributes wmic supports.

Or, try this:

C:\> wmic process call /?

This one shows you a list of all the process methods you can call from within wmic.

Or, how about this one:

C:\> net /?

And then:

C:\> net use /?

Likewise, we can run “reg /?” to get an overview of the reg command, followed by a “reg query /?” to get more details about the specific syntax for querying the registry.

Want another bewildering inconsistency? Try running this:

C:\> netstat /?

Note that the output actually shows the command flags with dashes instead of slashes (i.e., “netstat –na” instead of “netstat /na”). Thankfully, the actual netstat command on modern Windows boxes lets you use dashes or slashes in its flags.

So, how can we wrap our heads around all of this inconsistency? I’ve found that mastering Linux involves learning certain general principles and then seeing how they are applied throughout the operating system. Whereas, in Windows, mastering the command line involves memorizing a bunch of complex, counter intuitive, and often inconsistent options scattered throughout the operating system without rhyme or reason. So, why bother? Because, once mastered, the Windows command-line is incredibly useful in analyzing systems at a fine-grained level. Also, cmd.exe is pretty much everywhere, pre-installed, so there’s a lot less hit-and-miss than you get with installs of PowerShell. At least, there is for now… this is changing as PowerShell gets more widely deployed.

The bottom line here? When I’m actually looking for help with a command, I apply the following process. I start by trying “help [command]”. If that provides no satisfaction, I proceed to “[command] /?”. If that doesn’t give me what I want, I try to look for context-specific help with “[command] [option] /?”. And, if that doesn’t get me where I need to go, I turn to the web to research commands. One of my absolutely favorite sites for researching Windows (as well as Linux and Unix) commands is the awesome ss64 site. It includes a detailed list of most Windows commands, including their various flags, example usage, and mapping to rough equivalents in Linux, OS X (sometimes), and PowerShell (occasionally). That’s awesome. And, finally, there’s Microsoft’s own command line reference, worth a check from time to time.

Hal cleans up:

There are actually a number of different help systems available on a typical Unix system. First there’s the standard on-line manual pages accessed via the “man” command. The Unix manual pages include not only information about programs that you can run from the command line, but also documentation on programming APIs, device interfaces, and even the format of important configuration files on the system.

The trick sometimes is getting to the right manual page. For example, suppose I wanted some information on how to use the chmod() system call in my C program. If I just typed “man chmod”, I’d get the manual page for the chmod program and not the documentation on the system call. To distinguish these two different manual pages, the Unix manual has traditionally been organized into sections. Sections 1 (user commands) and 8 (administrator commands) are devoted to command-line tools, while sections 2 (system calls) and 3 (library routines) are devoted to programming APIs. FYI, section 4 is device interfaces, section 5 is configuration file formats, section 6 is games, and section 7 is “miscellaneous” but includes useful tidbits like the table of ASCII values (“man ascii”) and additional details on various IPC mechanisms. When a Unix document refers to “chmod(2)” it means “the documentation on the chmod system call in section 2 of the manual”.

But how does on pull up the chmod(2) manual page instead of the default chmod(1) page? The man command takes a “-s” option to specify the section: “man -s 2 chmod”. But on many Unix variants you can simply drop the “-s” and just type “man 2 chmod”.

Suppose, however, that I didn’t know which section the chmod() system call was documented in. One of the most useful features of the man command is the “-k” (keyword search) option:

$ man -k chmod chmod (1) - change file mode bits chmod (2) - change permissions of a file fchmod (2) - change permissions of a file fchmodat (2) - change permissions of a file relative to a directory f...

You don’t have to search on command names, of course. Any related keyword will work. For example, if you’re new to Unix and don’t know how to rename files:

$ man -k rename dpkg-name (1) - rename Debian packages to full package names lvrename (8) - rename a logical volume mmove (1) - move or rename an MSDOS file or subdirectory mren (1) - rename an existing MSDOS file mv (1) - move (rename) files prename (1) - renames multiple files rename (1) - renames multiple files rename (2) - change the name or location of a file rename.ul (1) - Rename files renameat (2) - rename a file relative to directory file descriptors vgrename (8) - rename a volume group

By the way, the “apropos” and “whatis” commands are equivalent to “man -k”. However, all of these commands operate on special “databases” of information that has been extracted from the manual pages themselves via the “makewhatis” command (aka “mandb” on Linux). Very often your system will have a cron job that builds the whatis databases automatically every week or so, but on some Unix systems you have to build the databases manually (or create your own cron job).

Another tricky issue with the Unix on-line manual is that sometimes you have multiple repositories of manual pages. For example, you might have the standard OS manual pages under /usr/share/man and manual pages for third-party software in /usr/local/man. The man command lets you use the “-M” option to specify a path to use for finding manual pages, but far it’s far easier to set MANPATH in your .bashrc file:

export MANPATH=/usr/share/man:/usr/local/man

MANPATH works just like PATH or LD_LIBRARY_PATH. In the above example, the man command will check the system manual page directory first and then look in /usr/local/man if it doesn’t find the manual page you’re requesting.

The man command is also the source of one of the most famous jokes in Unix:

$ man 'furry animals with tails' No manual entry for furry animals with tails

OK, so the traditional joke is “man rear”, but you get the idea.

Sadly, man pages seem to be dieing off. A lot of Open Source projects have either very scanty or completely non-existent manual pages. It’s actually pretty tough to write a good manual page, both in terms of content and also because you need to know nroff/troff formatting commands to format the text. But writing (and reading) manual pages properly is still an important Unix skill IMHO.

Another source of on-line documentation on Unix is the Free Software Foundation’s “info” system. The most complete documentation on packages like emacs, gcc, GNU tar, et al is found in the info system– the manual pages are mostly just stubs extracted from the docs in the info pages. Run the command “info info” to get more information on how to navigate the curses-based info interface. Frankly, I think the user interface for the info system is terrible, but it was mostly designed to be run from within emacs. I suppose that it makes sense if you’re one of those people who essentially lives inside of emacs and never touches an actual terminal window.

But the info system never really caught on outside of the FSF. With traditional manual pages being marginalized as well, the main documentation interface these days seems to be the “-h” or “–help” options supported by many commands. Typically, one or both of these options will generate at least a terse summary of command options:

$ info --help Usage: info [OPTION]... [MENU-ITEM...]
Read documentation in Info format.

Options: --apropos=STRING look up STRING in all indices of all manuals. -d, --directory=DIR add DIR to INFOPATH. --dribble=FILENAME remember user keystrokes in FILENAME. -f, --file=FILENAME specify Info file to visit. -h, --help display this help and exit. --index-search=STRING go to node pointed by index entry STRING. -n, --node=NODENAME specify nodes in first visited Info file. -o, --output=FILENAME output selected nodes to FILENAME. -R, --raw-escapes output "raw" ANSI escapes (default). --no-raw-escapes output escapes as literal text. --restore=FILENAME read initial keystrokes from FILENAME. -O, --show-options, --usage go to command-line options node. --subnodes recursively output menu items. -w, --where, --location print physical location of Info file. --vi-keys use vi-like and less-like key bindings. --version display version information and exit.

The first non-option argument, if present, is the menu entry to start from;
it is searched for in all `dir’ files along INFOPATH.
If it is not present, info merges all `dir’ files and shows the result.
Any remaining arguments are treated as the names of menu
items relative to the initial node visited.

Examples: info show top-level dir menu info emacs start at emacs node from top-level dir info emacs buffers start at buffers node within emacs manual info --show-options emacs start at node with emacs' command line options info --subnodes -o out.txt emacs dump entire manual to out.txt info -f ./foo.info show file ./foo.info, not searching dir

Email bug reports to bug-texinfo@gnu.org,
general questions and discussion to help-texinfo@gnu.org.
Texinfo home page: http://www.gnu.org/software/texinfo/

Not as useful as a traditional manual page, IMHO, but at least it’s something. The biggest problem is that with the documentation being tied up in each individual command’s “–help” output, there’s no “man -k” equivalent for doing global keyword searches if you’re not sure what command you’re looking for.

admin | Nov 10, 2009 | Comments 0

Time Lords

From: COMMAND LINE KUNG FU: PaulDotCom, Ed Skoudis, Hal Pomeranz, byte_bucket

Hal is still adjusting:

Mr. Bucket suggested a time-related Episode this week in honor of the shift back to Standard Time here in the U.S. Generally, we try to humor Mr. Bucket as long as his requests aren’t too deviant (or if Ed is the only one who has to dress up), so here goes.

First I’ve got a couple of tricks up my sleeve related to the NTP time synchronization service. My favorite command-line tool for checking up on my NTP servers is “ntpdc -p”:

$ ntpdc -pn server.example.com remote local st poll reach delay offset disp ======================================================================= =216.14.97.75 192.168.1.2 16 1024 0 0.00000 0.000000 0.00000 =173.45.227.99 192.168.1.2 2 1024 377 0.06984 -0.003889 0.13681 *64.73.32.135 192.168.1.2 2 1024 377 0.09087 0.002323 0.12178 =127.127.1.1 127.0.0.1 5 64 377 0.00000 0.000000 0.03093

The “st” column that shows the remote server’s stratum level. This is useful for detecting downed time servers, because they show up as stratum 16 like on the first line above. The “poll” (polling interval in seconds) and “reach” (displayed in octal notation and capped at 255, the number of recent successful polls) columns give you and idea of how well your server is synchronizing with other time servers. “delay” tells you how far away the remote server is from you, and “offset” is the difference between your local clock and the clock on the remote server (if things are working right, the offset should be less than a hundredth of a second). Note that in the above example, I checked on the remote server.example.com machine, but you can leave off the host name portion to check the time server running on your local box.

In addition to checking status with ntpdc, you can also set the time on your box from the command line using ntpdate. Normally you would just run ntpdate and specify the server you wanted to synchronize against, but here’s a cute little bit of awk fu for pulling the server names out of your current ntp.conf file:

# ntpdate `awk '/^(server|peer) / && !/127.127./ {print $2}' /etc/ntp.conf` 29 Oct 19:42:43 ntpdate[13376]: adjust time server 66.187.233.4 offset -0.004818 sec

Here I’m pulling off the host name or IP address specified on the “server” or “peer” lines in your ntp.conf file. I’m skipping any “addresses” that start with “127.127.” since these are merely place holders for clock driver definitions and not real IP addresses.

Note that you can only run ntpdate if your NTP daemon is not currently active. If ntpd is running at the same time you execute ntpdate, you get a message like this:

# ntpdate `awk '/^(server|peer) / && !/127.127/ {print $2}' /etc/ntp.conf` 29 Oct 19:42:18 ntpdate[13362]: the NTP socket is in use, exiting

Typically you only need to call NTP date by hand if the NTP daemon is not running and your clock has gotten out of synch. After you use ntpdate to jump your clock to something approximating the right time, you should start your NTP daemon to keep it synced (usually “/etc/init.d/ntpd start”).

Getting away from NTP now, let me show you a little environment variable hack that’s particularly useful for Forensic Analysts. Suppose you’ve captured a disk image from a system that was being used in a time zone other than the one your analysis machine is set to. You’d like to mount a copy of the image and be able to observe the timestamps on the files relative to the time zone the image was taken from. You could change the global time zone setting on your analysis workstation, but that could cause you lots of hassle.

It turns out that bash allows you to set the TZ environment variable in your shell to alter the local time zone setting for only that shell. For example, my local time zone is US/Pacific, but suppose I had an image from a machine from the East Coast:

$ date Thu Oct 29 17:59:13 PDT 2009 $ ls -ld /mnt/image/tmp drwxrwxrwt 23 root root 4096 Oct 29 17:28 /mnt/image/tmp $ export TZ=US/Eastern $ date Thu Oct 29 21:00:07 EDT 2009 $ ls -ld /mnt/image/tmp drwxrwxrwt 23 root root 4096 Oct 29 20:28 /mnt/image/tmp

Cool! Now I can look at the dates on files in my image and not constantly have to be doing mental arithmetic to convert the time stamps.

So there are three of my favorite time-related hacks. If we can get Ed out of those leather chaps that Mr. Bucket had him wear, I’m sure he’ll have some cool Windows Fu to show us…

Ed chimes in:

Ahhh… time. A great idea for an article, Mr. Bucket. Much better than your last idea… you know… the one about the chaps. Anyway, I digress.

Let’s walk before we run. To check the time on your local computer, you can simply run the following command:

C:\> time /t 06:05 AM

Want more precision? We can get that, as discussed in Episode #49, by displaying the %time% environment variable.

C:\> echo %time% 6:05:22.75

Alternatively, the “net time” command can be used to pull the time from a remote (or even local) Windows box:

C:\> net time \\[computername] Current time at \\FRED2 is 10/31/2009 6:04 AM The command completed successfully.
Note that, depending on whether you are a member of a domain and the privileges of your current account, you may need to first initiate an SMB connection with that machine, by running:

C:\> net use \\[computername] [password] /u:[username] The command completed successfully.

If you happen to trust that server, you can set your local machine’s time by its clock through running:

C:\> net time \\[computername] /set /yes

Well, that’s all well and good, but Hal was working with NTP, which offers many more time sources than, well, our local host or domain controller. How can we pull time from an NTP server in Windows? First, please note that officially, with built-in capabilities, Windows relies on the Simple Network Time Protocol (SNTP), a stripped down NTP implementation that is often used in applications where time accuracy isn’t as vital. That’s why you’ll see sntp throughout our commands below.

Let’s first look to see which NTP server our Windows machine is configured to use:

C:\> net time /querysntp The current SNTP value is: time.windows.com,0x1 The command completed successfully.

The time.windows.com server is the default. Do you trust Microsoft to accurately track the time on their server? After missing the release date of Windows Vista by… uh… something like 3 years, maybe we shouldn’t (darnit… I couldn’t resist pouring salt in those old wounds.) We can change the NTP server we’re configured to use as follows:

C:\> net time /setsntp:pool.ntp.org The command completed successfully.
Note that you can put a whole list of NTP servers there by following the colon with a space-separated list of NTP servers enclosed in double quotes, resulting in something like /setsntp:”pool.ntp.org ntp.colby.edu tick.gatech.edu”

OK… now, once we’ve set ourselves up to use an NTP server, let’s try to check the time:

C:\> net time Could not locate a time-server.

More help is available by typing NET HELPMSG 3912.

What? Why doesn’t “net time” work here? Note that the HELPMSG is not at all helpful, as it explains that we could not locate a time-server, which we already saw in the error message itself. Gee, Microsoft… thanks for nothing.

It turns out that we can’t pull time via “net time” using NTP (really SNTP) unless we’re a domain controller with the Flexible Single Master Operation (FSMO) role. Lovely. But, what if we’re just a lowly client, maybe not even in a domain? Can we sync with an NTP server using only built in tools? As Bob the Builder and Barack Obama might say: Yes, we can!

We can rely on that happy old stand-by for configuring time on Windows boxen, the intuitively named w32tm command. And in homage to Heinz Doofenshmirtz, by “intuitively named”, I mean, of course, “completely non-intuitively named.”

To sync with an NTP server, we start by configuring our Windows box (whether it’s a client or server version of Windows) to allow us to pull time manually from an NTP server we’d like to use (note that you have to specify this with w32tm even if you’ve already run “net time” with the /setsntp option):

C:\> w32tm /config /syncfromflags:manual /manualpeerlist:pool.ntp.org The command completed successfully.
Now that we’re configured, just about ready to sync times, let’s do a quick time check before the sync:

C:\> time /t 06:47 AM

Oh… that looks dreadfully wrong. I think my clock is fast. Let’s cause Windows to read our new configuration now:

C:\> w32tm /config /update The command completed successfully.
And finally, let’s resync:

c:\> w32tm /resync Sending resync command to local computer The command completed successfully.

Boom! The clock in my tool tray is now synchronized! Let’s double check that we have the new and proper time:

C:\> time /t 06:13 AM

We’ve now synced with a remote NTP server at the command line. Yummy.

Also, there are a lot of other fascinating time-related options available with w32tm. For instance, to display your timezone, you could run:

c:\> w32tm /tz Time zone: Current:TIME_ZONE_ID_STANDARD Bias: 300min (UTC=LocalTime+Bias) [Standard Name:"Eastern Standard Time" Bias:0min Date:(M:11 D:1 DoW:0)] [Daylight Name:"Eastern Daylight Time" Bias:-60min Date:(M:3 D:2 DoW:0)]

We can even convert an NT system time (the number of 100 nanosecond intervals that have elapsed since Jan 1st, 1601, the date DOS was first written by monks) to a human readable date and time:

c:\> w32tm /ntte 128904420852883740 149194 22:21:25.2883740 - 6/25/2009 5:21:25 PM

This is especially useful if you have a Vista box (my condolences) and are researching the installation of patches using “wmic qfe”. In XP Pro and Windows 7, “wmic qfe list full” returns a list of patches with an “InstalledOn” field in a human readable form. But, in Vista, this date is now a hexadecimal number of an NT system time. Convert that hex number into decimal using you favorite calculator, and then fire up w32tm /ntte to convert it into a date you can read. Also, the “w32tm /ntpte” command converts NTP times (seconds from Jan 1st, 1900, the date that Nicola Tesla implemented the first Unix kernel) into a human readable format.

But, wait, there’s more! Hal showed how to add a timestamp into the output of your commands, useful in forensics. I sometimes will put a date and time into my command prompt itself, so that I can then copy and paste my session to use as evidence:

C:\> prompt $D$T$S$P$G Sat 10/31/2009 6:15:29.97 C:\>

So, there’s a bunch of time-related stuff. Have you got any other fun time kickers, Tim?

Tim clocks in:

Let’s take a look at the cmdlets available that use the Time or Date nouns.

PS C:\> Get-Command -Noun date CommandType Name Definition ----------- ---- ---------- Cmdlet Get-Date Get-Date [[-Date] ... Cmdlet Set-Date Set-Date [-Date] ...

PS C:\> Get-Command -Noun time [nothing]

Unfortunately, there aren’t any built-in cmdlets for NTP. So to answer Ed’s question, “No, no interesting kickers this week.” But we can still take a look at the Get-Date cmdlet and its properties.

PS C:\> Get-Date | fl

DisplayHint : DateTime DateTime : Sunday, November 01, 2009 10:17:23 PM Date : 11/1/2009 12:00:00 AM Day : 1 DayOfWeek : Sunday DayOfYear : 305 Hour : 22 Kind : Local Millisecond : 206 Minute : 17 Month : 11 Second : 23 Ticks : 633927106432066338 TimeOfDay : 22:17:23.2066338 Year : 2009

The object has a number of methods available, and they can be seen by piping the object into Get-Member. The results aren’t shown here since it is a pretty long list. Since the reason for this episode is Daylight Savings Time we’ll take a quick peek at the relevant method.

PS C:\> (Get-Date).IsDaylightSavingTime() False

Ok, that wasn’t spectacular, but let’s carry on and recreate Ed’s command prompt with the date.

PS C:\> function prompt {"PS $(Get-Date) $(Get-Location)>"} PS 11/01/2009 22:43:53 C:\>

Since the prompt is defined by overloading the prompt function there are endless possibilities for setting a custom prompt.

That’s all folks, hope to see you next time.

admin | Nov 03, 2009 | Comments 0