Archive

Archive for the ‘Perl’ Category

Data::Dumper formatting

October 8, 2014 Leave a comment

I look this up about every couple weeks, so I’m posting it here for posterity. In order to nicely format Data::Dumper output…

I almost always set

$Data::Dumper::Indent = 1;
$Data::Dumper::Sortkeys = 1;

with Data::Dumper. The first statement makes the output more compact and much more readable when your data structure is several levels deep. The second statement makes it easier to scan the output and quickly find the keys you are most interested in.

If the data structure contains binary data or embedded tabs/newlines, also consider

$Data::Dumper::Useqq = 1;

which will output a suitable readable representation for that data.

Much more in the perldoc.

Advertisement
Categories: Perl, Tech

Perl on NetBeans

June 30, 2013 Leave a comment

This is a new, but very promising project to bring Perl support to NetBeans 7.x. I can’t really articulate *why*, but I’ve always preferred using NetBeans to other cross platform IDE’s  (e.g. Eclipse).

Both IDE’s are written in Java, but for some reason NetBeans always seemed more polished (and faster) to me. It could all be in my head, but I’m glad to see a push to bring Perl support back to NetBeans.

They also have a Facebook page that you can follow for more news and updates about the project.

From the Perl-on-NetBeans project website:

This project aims at providing support for Perl on the NetBeans platform. It aims at filling in the void for Perl programmers on NetBeans platform.

Initially, the following are the features that are planned to be integrated with this IDE:

1. Perl File Type Support
2. Perl Project Type
3. Code Execution
4. Syntax checking
5. Perl Tidy and Perl Critic (Source code formatting and analysis respectively)
6. Syntax highlighting
7. Brace matching
8. Real-time error parsing
9. Code completion
10. Code folding
11. Debugger Support

Categories: Perl, Tech

Perl’s predefined variables have long and short equivalents

July 16, 2012 Leave a comment

Have you ever seen (or written) something like this in your Perl scripts? 

if ( $^E eq "Access is denied" ) #...

I know that I have, and I also know that 2 years later I found that snippet much less readable.

$^E is one of Perl’s predefined variables that reports error information specific to the current OS. However, it has an equivalent/alias that is much more understandable:

$EXTENDED_OS_ERROR

That’s right, $^E  is the same as  $EXTENDED_OS_ERROR

So a better example of the original snippet would be:

if ( $EXTENDED_OS_ERROR eq "Access is denied" ) #...

I’d highly suggest that you always use the long-hand equivalent version of all the predefined Perl variables. Another option is to add the following to the top of your script:

use English qw( -no_match_vars ) ; # Avoids regex performance penalty

http://perldoc.perl.org/English.html

This module provides aliases for the built-in variables whose names no one seems to like to read. Variables with side-effects which get triggered just by accessing them (like $0) will still be affected.

For those variables that have an awk version, both long and short English alternatives are provided. For example, the $/ variable can be referred to either $RS or $INPUT_RECORD_SEPARATOR if you are using the English module.

See perlvar for a complete list of these.

Categories: Perl, Programming, Tech

PHP, Perl, Python, Ruby compared

March 3, 2012 Leave a comment

This is pretty handy for someone like me that knows Perl pretty well, but is interested in learning other languages like Python and Ruby. This is also great as a cheat sheet for a language you already know well: http://hyperpolyglot.org/scripting.

For instance, I’ve been using Perl for years and had no idea that they had a REPL mode…well, it’s more of a pseudo REPL mode but it lets me use Perl interactively:

perl -de 0

Categories: Perl, Programming, Tech

Perl and Unicode

January 15, 2012 1 comment

Great article on Perl and Unicode. http://www.perlmonks.org/?node_id=551676

The days of just flinging strings around are over. It’s well established that modern programs need to be capable of communicating funny accented letters, and things like euro symbols. This means that programmers need new habits. It’s easy to program Unicode capable software, but it does require discipline to do it right.

There’s a lot to know about character sets, and text encodings. It’s probably best to spend a full day learning all this, but the basics can be learned in minutes.

These are not the very basics, though. It is assumed that you already know the difference between bytes and characters, and realise (and accept!) that there are many different character sets and encodings, and that your program has to be explicit about them. Recommended reading is “The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)” by Joel Spolsky, at http://joelonsoftware.com/articles/Unicode.html.

A word of caution,  you don’t really need to care that Perl stores strings internally as UTF-8:

Please, unless you’re hacking the internals, or debugging weirdness, don’t think about the UTF-8 flag at all. That means that you very probably shouldn’t use is_utf8_utf8_on or _utf8_off at all.

Perl’s internal format happens to be UTF-8. Unfortunately, Perl can’t keep a secret, so everyone knows about this. That is the source of much confusion. It’s better to pretend that the internal format is some unknown encoding, and that you always have to encode and decode explicitly.

Steep .NET Framework disk space requirements

November 7, 2010 Leave a comment

I’m always a little wary when someone writes an application targetting the .NET Framework and then says it’s only 80k. Well, assuming you have the .NET Framework/runtime installed…which weighs in at hundreds of megabytes depending on the version and CPU architecture.

 

Details on the hardware requirements now that .Net 4.0 has been released. The minimum disk space requirements (as of September 2010) seem a little steep:

.Net 2.0                  x86: 280 MB              x64:     610 MB
.Net 3.5                  x86: 280 MB              x64:     610 MB
.Net 4.0 client         x86: 600 MB              x64: 1,536 MB (1.5 GB)
.Net 4.0 full             x86: 850 MB              x64: 2,048 MB (2.0 GB)

I know I’m a snob, but I’ll stick to native programming languages when it comes to writing desktop applications. And hack everything else together with Perl ;-)

Hardware requirements Version 4

full

Version 4

client

Version 3.5 Version 3.0 Version 2.0
Processor
Minimum 1 GHz 1 GHz 400 MHz 400 MHz 400 MHz
Recommended 1 GHz 1 GHz 1 GHz 1 GHz
RAM
Minimum 512 MB 512 MB 96 MB 96 MB 96 MB
Recommended 512 MB 512 MB 256 MB 256 MB
Disk space (minimum)
32-bit 850 MB 600 MB 280 MB 280 MB 280 MB
64-bit 2 GB 1.5 GB 610 MB 610 MB 610 MB
Categories: Delphi, Perl, Programming, Tech

The Programmer Competency Matrix

June 13, 2010 Leave a comment

A good reminder that you don’t actually know anything about programming ;-)

The Programmer Competency Matrix

In all seriousness, I ran across this and think it’s an excellent way to test yourself on your understanding and knowledge of not only a programming language, but operating systems and writing software as a discipline.

Categories: C/C++, Delphi, Perl, Programming, Tech

Getting the BackTrack menu structure (and tools) in Ubuntu

November 20, 2009 115 comments

NOTE: I have moved this code to GitHub.

UPDATE [11-7-2010]: It is *best* to use the actual BackTrack distribution due to the patches and optimizations they make to the kernel. Patching the standard Ubuntu kernel would make it no longer *Ubuntu*. I’d highly suggest you either:

  1. Install BackTrack on a real or virtual machine environment
  2. Manually download and install the security tools you need to use on your system.

————-

I recently outlined how to install all the BackTrack tools on your Ubuntu system. It’s a pretty easy process these days because BackTrack is Debian based. However, by default, BackTrack uses KDE, and therefore all of the tools, when installed in Ubuntu with Gnome show up in the “Other” folder as just one big really long list of tools.

There also were problems with launching the tools from the menu, because often times you’d get the error message below:

That’s not very helpful, so I have done a little work and gotten the BackTrack tools AND menu structure within Ubuntu 9.10. I’ve also written a Perl script that will fix the error message above.

So, here’s a brief overview of the steps in this lengthy blog post:

  1. Prepare the GNOME menu with the appropriate BackTrack menu structure
  2. Install BackTrack 4 tools within Ubuntu
  3. Run a Perl script to update the newly created menu entries so that they will launch a terminal correctly within Ubuntu

Here’s a quick preview of the finished product:

BackTrack 4 Menu in Ubuntu!

NOTE: This involves modifying your current GNOME menu settings, and could cause issues with your menu if done incorrectly. Make a backup first, and note that this worked on my system, but your mileage may vary!

The first step before installing any of the tools is to prepare the GNOME menu. Open up a shell:

sudo cp /etc/xdg/menus/applications.menu /etc/xdg/menus/applications.menu.original
sudo geany /etc/xdg/menus/applications.menu

Note that I am using the application Geany, which is a programmer’s editor for Linux, much like Notepad++ for Windows. You can use gedit if you’d prefer, but Geany may help you in modifying this XML file because it offers code folding and syntax highlighting.

If you opened it with Geany, click on the Document menu, Select Set Filetype, select Markup Languages, and then finally select XML Document.

Select file type within Geany

With Geany, I find it easiest to collapse the menu blocks for the other sub-menus within Gnome’s Applications menu.

After collapsing the System Tools section, paste the following XML directly after it. This will define the BackTrack menu’s and submenu’s:

Adding BackTrack menu structure with Geany

NOTE that because of the length, I have moved the XML code to a new post. Please obtain the BackTrack menu XML from this post.

After updating this with the BackTrack XML, you may now save and close the document.

At this point, we can follow the instructions in my previous post to install the BackTrack utilities.

Let’s begin by launching a root bash shell by typing:

sudo bash

The next step is to add the BackTrack repositories to your apt-get sources.list file:

1. Add the Backtrack repository:

sudo echo deb http://repo.offensive-security.com/dist/bt4 binary/ >> /etc/apt/sources.list

2. Import the Backtrack PGP key and update your sources (and set a proxy server to use if you need it):

export http_proxy="http://myproxyserver.com:8080"
wget http://repo.offensive-security.com/dist/bt4/binary/public-key && sudo apt-key add public-key && sudo aptitude update

3. Build your package list (NOTE that I am specifying a proxy server — remove this part from the command if you do not use a proxy):

links -http-proxy myproxyserver.com:8080 -dump http://repo.offensive-security.com/dist/bt4/binary/ | awk '{print $3}' | grep -i deb | cut -d . -f 1 > backtrack.txt

If you do not use a proxy server, then the command will look like this:

links -dump http://repo.offensive-security.com/dist/bt4/binary/ | awk '{print $3}' | grep -i deb | cut -d . -f 1 > backtrack.txt

4. Install packages:

for i in $(cat backtrack.txt); do sudo aptitude -y install $i; done

Credit for the BackTrack menu settings goes to or4n9e at Remote Exploit’s forums.

Next, we need to run a Perl script to ensure that the newly installed applications can be correctly executed from our GNOME Applications menu.

Copy the following code and save it to a file. I saved mine to a file called UpdateBTMenu.pl

#!/usr/bin/perl
#
##
# Written by Mick Grove
# https://micksmix.wordpress.com
#
#  [v0.1]		11/20/2009
##
#
# BSD Licensed
#
#       Redistribution and use in source and binary forms, with or without
#       modification, are permitted provided that the following conditions are
#       met:
#
#       * Redistributions of source code must retain the above copyright
#         notice, this list of conditions and the following disclaimer.
#       * Redistributions in binary form must reproduce the above
#         copyright notice, this list of conditions and the following disclaimer
#         in the documentation and/or other materials provided with the
#         distribution.
#       * Neither the name of the  nor the names of its
#         contributors may be used to endorse or promote products derived from
#         this software without specific prior written permission.
#
#       THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#       "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
#       LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#       A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
#       OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#       SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#       LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#       DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
#       THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
#       (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#       OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
#
use strict;
use warnings;
use Tie::File;

my $dir     = "/usr/local/share/applications";
my $section = "Desktop Entry";
my $in_section;
my @files;

opendir(BIN, $dir) or die "Can't open $dir: $!";
while (defined(my $file = readdir BIN))
{
    next if $file =~ /^\.\.?$/;    # skip . and ..
    if ($file =~ m/.*\.desktop$/i)
    {
        push(@files, $file);
    }
}
closedir(BIN);

foreach my $curfile (@files)
{
    open( FH, "<", "$dir/$curfile" ) or die "$!";
    chomp( my @fileparts =  );

    my $termval = TerminalStatus(\@fileparts);
    next if $termval eq 0;    # skip if this is not a terminal application

    #lets see if this is actually a BT program
    my $btprogram = IsBTProgram(\@fileparts);
    next if $btprogram == 0;    # skip if this is not a BT application

    my $ExecKey     = "Exec";
    my $TerminalKey = "Terminal";
    my @tiedfile;

    #open this file for editing
    tie @tiedfile, 'Tie::File', "$dir/$curfile" or die "$!";

    #read file line by line here
    # updating "Exec" line
    foreach my $fline (@tiedfile)
    {
        next if $fline =~ /^#/;       # skip comments
        next if $fline =~ /^\s*$/;    # skip empty lines

        if ($fline =~ /^\[$section\]$/)
        {
            $in_section = 1;
            next;
        }

        if ($fline =~ /^\[/)
        {
            $in_section = 0;
            next;
        }

        my $oldline;
        my $updatedline;
        if ($in_section and $fline =~ /^$ExecKey\s*=\s*(.*)$/)
        {

            # this means we have the "Exec key"
            $oldline = $1;
            next if $oldline =~ m/^.*xterm -e.*;bash.*$/i;    #skip
            $oldline =~ s/"/\\"/img;
            $updatedline = "Exec=xterm -e \"$oldline;bash\"";
            $fline       = $updatedline;

            print "New exec: " . $fline . "\n";
            next;
        }
        elsif ($in_section and $fline =~ /^$TerminalKey\s*=\s*(.*)$/)
        {

            # this means we have the "Terminal key"
            # we will set it to "0" to turn it off --- we are launching
            #   xterm ourselves, if we set to 1, we'll get an extra
            #   terminal opened
            #
            $oldline     = $1;
            $updatedline = "Terminal=0";
            $fline       = $updatedline;
            next;
        }

    }
    untie @tiedfile;
}

print "\n\nAll menu entries have been updated\n";

###
### Subroutines ###
###
sub TerminalStatus
{
    my @lines       = @{$_[0]};
    my $TerminalKey = "Terminal";
    my $ExecKey     = "Exec";
    my $termkeyval  = 0;            #default = 0 FALSE, 1= TRUE
    my $i           = 0;
    my $execkeyval = 0;  #default = 0 = this exec line probably wasn't set by us

    foreach my $fline (@lines)
    {
        next if $fline =~ /^#/;       # skip comments
        next if $fline =~ /^\s*$/;    # skip empty lines

        if ($fline =~ /^\[$section\]$/)
        {
            $in_section = 1;
            next;
        }

        if ($fline =~ /^\[/)
        {
            $in_section = 0;
            next;
        }

        if ($in_section and $fline =~ /^$TerminalKey\s*=\s*(.*)$/)
        {

            # this means we have the "terminal key"
            $termkeyval = $1;
            next;    #last;
        }

        if ($in_section and $fline =~ /^$ExecKey\s*=\s*(.*)$/)
        {

            # this means we have the "exec key"
            $execkeyval = $1;
            if ($execkeyval =~ m/^.*xterm -e.*;bash.*$/i)
            {
                $execkeyval = 1;    # this script likely set this value before
            }
            next;                   #last;
        }
    }

    if ($execkeyval eq 1)
    {
        # force this to true, because this can be updated by this script,
        #   b/c we appear to have modified this entry before.
        $termkeyval = 1;
    }
    return $termkeyval;
}

sub IsBTProgram
{
    my @lines    = @{$_[0]};
    my $key      = "Categories";
    my $isbtprog = 0;              #default = FALSE = 0
    my $i        = 0;
    foreach my $fline (@lines) {
        next if $fline =~ /^#/;       # skip comments
        next if $fline =~ /^\s*$/;    # skip empty lines

        if ($fline =~ /^\[$section\]$/) {
            $in_section = 1;
            next;
        }

        if ($fline =~ /^\[/) {
            $in_section = 0;
            next;
        }
        if ($in_section and $fline =~ /^$key\s*=\s*(.*)$/)
        {

            # this means we have the "terminal key"
            if ($1 =~ m/.*BT-.*/i)
            {
                $isbtprog = 1;
            }
            last;
        }
    }
    return $isbtprog;
}

I saved that file to my home folder at /home/mick

Next we need to run the script, but first we will backup all menu files in case something goes wrong. Open up a terminal:

cd ~/
mkdir menu_backup
sudo cp /usr/local/share/applications/* ~/menu_backup

Now we have made a backup of the menus, so it is safe to run our Perl script now:

sudo perl ./UpdateBTMenu.pl

That’s it! Your BackTrack tools (with menu structure) are ready to use within Ubuntu!

If for some reason there was a problem with executing the Perl script or your menu isn’t working, you can copy the backed up menu items to their original location:

sudo cp ~/menu_backup/* /usr/local/share/applications/
Categories: Linux, Perl, Tech

Extracting Data from Network Captures (pcap) with Perl

October 11, 2009 1 comment

A big part of malware analysis involves capturing network activity using Wireshark or Microsoft Network Monitor. Many online sandboxes, such as Anubis and CW Sandbox will actually provide you with a network capture in the standard pcap format.

With the pcap in hand, it is as easy as opening it within Wireshark. I often find that for a networking novice (like me), parsing through these pcap’s can be daunting and time consuming. This is where a project called chaosreader comes in to make our lives easier.

When I am analyzing network activity generated by malware, I am most interested in HTTP get/posts, the addresses the malware is communicating with, and the data that was actually sent or received.

Example GET/POST report

Example GET/POST report

Chaosreader is a Perl script that takes a pcap file as its argument and will create communication summaries in a report format. It will also pull data from the tcp streams (within the pcap) and re-assemble the actual files.

Here is an example of the primary report that chaosreader generates. With malware analysis, I find that the most useful report is the getpost.html file that is generated. It provides only a listing of the HTTP post/get activity in a very easy to read report.

Chaosreader works anywhere that Perl does, although I’ve only personally used it with ActivePerl 5.10 on Windows XP and Perl 5.10 on a Linux distribution.

Generating a standard report (with re-assembled files) from a pcap file called “malware_activity.pcap” is as simple as running the following from a cmd prompt or bash shell:

perl chaosreader.pl malware_activity.pcap

Here’s another guide to using chaosreader

Categories: Disassembly, Network, Perl, Tech

Format your Perl scripts using Perl-Tidy and Notepad++

September 24, 2009 21 comments

UPDATED on 8/15/2011 – The macro now automatically reloads the current file from disk, making this a 1 step re-tidy process.

I primarily use Notepad++ to write my Perl scripts because it is quick, offers code folding, syntax highlighting, and very useful keyboard shortcuts.

I’ve also begun using Perl::Tidy to “beautify” my Perl code. Perl::Tidy is available from CPAN, or available via Activestate’s Perl Package Manager (PPM).

To install via CPAN (if you use Strawberry Perl or Cygwin, this is the easiest method):

If you are not behind a proxy server (most people aren’t), just open a command prompt (or Cygwin shell) and enter:

perl -MCPAN -e shell

Once that has loaded, type:

install Perl::Tidy

Then wait a few moments for it to be installed. If you are behind a proxy server (such as at work), make sure that before you run “perl -MCPAN -e shell” that you set the environment variables in your command prompt. I make this simple by creating a batch file that I save as “proxy.bat” that contains the following:

REM Proxy.bat
set HTTP_proxy=http://myproxy.contosocom:8080
set FTP_proxy=http://myproxy.contosocom:8080

…and then I save that file (proxy.bat) somewhere in my PATH environment variable (such as c:\windows, or in my case I put c:\tools in my PATH environment variable). Then, I just run:

c:\tools\proxy.bat
perl -MCPAN -e shell

Once you have installed the Perl::Tidy module (either from PPM or CPAN), here is how you can integrate it into Notepad++ to re-format (or beautify) your current Perl script.

I am using Notepad++ v5.4.5, but you must ensure that you have the NppExec plugin installed with Notepad++. It is typically part of the standard installation.

Begin by opening Notepad++ and selecting the Plugins menu, NppExec submenu, and select Execute:

perltidy02

In the Execute dialog box, enter the following (note: change the path to perl.exe if your installation is different):

npp_save $(FULL_CURRENT_PATH)
perl -x -S perltidy -b "$(FULL_CURRENT_PATH)" -ole=win
npp_open $(FULL_CURRENT_PATH)

Then click Save…

NOTE, this command tells Perl-Tidy to create a backup of the current script (in the script’s directory) and clean up the code according to Perl-Tidy’s default formatting options. It will only create 1 backup file. If one already exists it is replaced. See here for other Perl-Tidy configuration options.


The second command tell Notepad++ to reload the file from disk once Perl::Tidy completes. This means changes are immediately updated in your active document.


Enter Perltidy as the script name when prompted.

Next select the Plugins menu, NppExec submenu, and select Advanced Options:

perltidy05

On the Advanced Options screen, first enter a menu item name, such as Run PerlTidy on current file. Then select PerlTidy on the Associated script drop-down box. Then be certain to click the Add/Modify button.

Now look at the top-left of the dialog box and click teh Place to the Macros submenu checkbox. Then click OK.

See the image below for a graphic that also shows the order of these operations:

perltidy06

You may receive a message from Notepad++ that you need to restart the program. Do that now.

Now you are ready to test it all out! Open up a Perl script that is in need of some formatting. If you aren’t sure if you have one, open up any Perl script, select all, and then press SHIFT+TAB several times to remove all indentation.

Now click on the Macro menu item and select Run PerlTidy on current file (or whatever you entered for Item Name from the previous step):

perltidy07

Now the last step is to reload the current script to see our change. Simply click on File and select Reload from Disk:

perltidy08

The above step is no longer necessary since I have updated this guide on 8/15/2011. The reason it isn’t necessary is because when creating our macro using NppExec, I specified the internal Notepad++ command, npp_open $(FULL_CURRENT_PATH), which reloads the file from disk once Perl::Tidy has executed.

npp_save $(FULL_CURRENT_PATH)
perl -x -S perltidy -b "$(FULL_CURRENT_PATH)" -ole=win
npp_open $(FULL_CURRENT_PATH)

Although this reloads the file from disk, when you click a different window on your desktop and then later click on Notepad++, you may receive a message stating that the file has been modified and it will ask if you want to reload it:

If you would like to suppress this message:

  • Click the Settings menu
  • Select Preferences
  • Select the MISC. tab.
  • Check Update Silently


Categories: Perl, Programming, Windows