NDiff_Quickstart - tutorial for ndiff and related tools.

This document provides a fast introduction to each of the ndiff tools:

ndiff - calculate the differences between two nmap scans

ngen - synthetically create baseline nmap results

nrun - execute nmap and ndiff, doing some housekeeping

The simplest invocation of ndiff is to using the machine-parseable output of two previous nmap runs on the same net, for example:

  # nmap -m first_scan.nm  10.0.0.0/24 ...

  ... later ...

  # nmap -m second_scan.nm 10.0.0.0/24 ...

OK, now we have two scans of the same net at different moments in time. Now to see the changes:

  # ndiff -baseline first_scan.nm -observed second_scan.nm

  ... ndiff outputs: ...

  missing hosts:
   < hosts present in first_scan, but missing in second_scan >

  new hosts:
   < hosts present in second_scan, but missing from first_scan >

  changed hosts:
   < hosts present in both scans, but whose port states have changed>
    [ for each host, a list of changes in port states]

Note the options passed to ndiff. We designate first_scan as the baseline for comparison. Changes are reported as differences from first_scan.

Ndiff has additional options and features for controlling output detail and format. See ndiff for more information.

Most parameters to the ndiff tools have both short and long versions. For example -baseline above is synonymous with -b. They are used interchangeably throughout this document. See the relevant man pages for full information.

Using the results of a previous scan as your baseline for comparison is fine for many purposes, but if you never knew or liked the state of the scanned net, the previous scan probably didn't yield a satisfactory baseline.

In that case, what you really want as your baseline is a description of your ideal net - one which reflects your firewall rules and/or security policy.

That is where ngen comes in. Ngen accepts host and port specifications, and outputs an equivalent nmap scan result which can be used as an ndiff baseline. Comparisons with this output then will show how your net varies from your ideal net.

Suppose we have a net 192.168.2.0/24. All hosts on this net are web servers, so port 80 is open, the rest should be closed. There is no firewall or filter device on the net. To generate the appropriate baseline:

  # ngen -o baseline.nm  -h 192.168.2.0/24:80o

The "-o baseline.nm" option specifies the output filename. The -h switch specifies a host specification. The host specification has two parts. The part before the colon gives the host IP range, in this case using CIDR notation. The part after the colon is the port specification, 80, followed by o. The o says the port's state is open. There are several accepted flags- o=open(default), c=closed, f=filtered, x=unfiltered, t=tcp (default), and u=udp.

Now suppose we actually have two nets to scan. Most should have only port 80 open. But a few hosts should also have port 22 (ssh) and port 53 tcp,udp (domain) open. The command-line would appear as follows:

  # ngen -o baseline_2.nm -h 192.168.2.0/24:80 \
        -h 10.0.2.128/25:80   \
        -h 10.0.2.144-150:22,53,53u

A few things are worth noticing in this example. First, the three host specifications. They are applied in order of their appearance, and are cumulative. So the 10.0.2.128/25 subnet net will end up with ports 80 open on every host, and ports 22/tcp, 53/tcp, and 53/udp open on hosts .144 through .150.

Second, host ranges can be specified with dash notation, that is 144-150 means every number in that range. This notation works for any quad in an IP address. Dash notation works for ports as well, so 10-12ux means udp ports 10, 11, and 12, each unfiltered (that's the x flag).

Third and this was just demonstrated you can specify both a protocol and a port state at the same time for a port, that is 53cu means port 53/udp, in a closed state. You never need to explicitly use the t and o flags, since they are the default.

You can also punch out parts of a host specification with the negation operator (!). For example, on a net with all web servers except for a few hosts which should never appear:

  # ngen -o baseline.nm -h 10.0.1.0/24:80 \
              -h '!10.0.1.0'       \
              -h '!10.0.1.45-53'   \
              -h '!10.0.1.255'

Again, the hosts specifications are applied in order of their appearance, so first the full host range is created, then the negations are applied to remove the specific ports and/or hosts. Also note that the negations are quoted to avoid interpretation by the shell.

Nmap shows ports it deems interesting, but this is somewhat ambiguous. What about the ports which were uninteresting, and what about those which were never scanned by nmap?

For each host, nmap reports an Ignored state, which is a shorthand. Every scanned port which wasn't explicitly reported is actually in that Ignored state. To know which ports these are, we need the set of all ports scanned. When running nmap, this is generated with the -v and -m switches together. On the other hand, ngen needs to be told explicitly which ports to include in the set-

 # ngen -o baseline.nm -p 1-1024,6000,53u -h 10.0.1.0/25:80

The -p switch above tells ngen that tcp ports 1-1024, tcp port 6000, and udp port 53 are the set of scanned ports. If no -p switch is specified, ngen will generate a list of ports from /etc/services and /usr/local/lib/nmap/nmap-services. However this may not be what you really want to happen.

By default, the Ignored state is closed, so all ports in the -p set above except for tcp/80 will be in the closed state. If you want to specify a non-closed default state for a given host, you must append it on to the host specification-

 # ngen -o baseline.nm  -p 1-1024,6000,53u -h 10.0.1.0/25:80:filtered

Now the default state for the 10.0.1.0/25 subnet is filtered. All scanned ports will be filtered, except for tcp/80 which will be open.

Acceptable Ignore states are: open, closed, filtered, and unfiltered. Ndiff will also report the psuedo-state unknown when a port was present in one scan but not in the other.

Ndiff and ngen give you a way to create an idealized picture of how your net should look, then compare the actual state of your net with that picture. Nrun goes the next step by taking care of bread-and-butter housekeeping tasks necessary to use ndiff in an automated fashion, such as with cron.

Specifically, nrun will-

execute nmap

name the results something reasonable

store the results somewhere reasonable

optionally run ndiff against those results to generate a report

A simple nrun session is as follows-

  # nrun -o my_net_scan.nm  -- -sS 192.168.2.0/26

  ... executes "nmap -sS 192.168.2.0/26   -v -m nmapXXX" ... 
  ...     "-v -m nmap$$" added by nrun ...
  ...     nmap output follows ...

All parameters after the -- string are passed directly to nmap. Nrun adds the -m and -v nmap switches to get output in a form suitable for ndiff.

Notice "-o my_net_scan.nm". This specifies the output file for the nmap scan. A literal name for the output file is fine for a one-off scan. But for monitoring your network you will probably want scans to run regularly, and results stored from each scan. A naming convention is necessary.

Nrun and the other tools allow %-style substitions for inserting time, date, and/or the local hostname dynamically. So if we were to have cron run a scan once a day, we might want a naming convention incorporating the date. For example:

  # nrun -o "192.168.2.0.26-%Y%m%D-syn-scan.nm"  -- -sS 192.168.2.0/26

If today's date is Dec 31, 1999, the name will expand to 192.168.2.0.26-19991231-syn-scan.nm.

The % codes are consistent with the gnu date command, with the addition of CW%F, which expands to the local hostname. See nrun for more information.

Nrun, ndiff, and ngen use nmap-format files on the local filesystem by default. This allows you the most flexibility in managing the nmap results data.

For the sake of future expansion, the ndiff tools have a mechanism to use other methods of storage and retrieval, for example pgp-encrypted nmap files, or a database engine.

Anywhere you specify a results file name in any of the ndiff tools, you can instead specify a string starting with a colon (:). When you do this, the string (sans colon) is regarded as a unique key into a data store. For example:

 # ngen -o :baseline-syn-192.168.2.0-net

 ...

 # nrun -o :daily-scan-for-192.168.2.0-net-%Y%m%D

 ...

 # ndiff -b :baseline-syn-192.168.2.0-net \
        -o :daily-scan-for-192.168.2.0-net-20000401

In all of the above cases, the tools will store/retrieve the results data from the configured data store. This can be very convenient, allowing you to think about the scan rather than managing nmap files.

Currently the only implemented data store type is an nmap file stored in special directory. That special directory is set in a global configuration file, ndiff.conf, which is read at runtime.

A sample file ndiff.conf.sample is installed into the same directory as the NDiff perl modules (PortScan subdirectory under perl extensions).

Copy and edit this file per the comments to configure data stores. The only thing you need to change is root attribute to point to the desired results storage directory.

The Ndiff tools will look for ndiff.conf in the following places, in order:

  ~/.ndiff/ndiff.conf
  the directory pointed to by the environment variable NDIFF_ROOT
  /usr/local/lib/ndiff/ndiff.conf
  ./ndiff.conf

A sample ndiff.conf is included below, but it is not definitive use the sample file stored with the installed NDiff modules.

NDiff.conf:

 #!/usr/bin/perl
 #
 # $Id: NDiff_Quickstart.pod,v 1.8 2000/12/17 09:40:26 levine Exp $
 #
 # Copyright (C) 2000  James D. Levine (jdl@vinecorp.com)
 #
 #
 #   This program is free software; you can redistribute it and/or
 #   modify it under the terms of the GNU General Public License
 #   as published by the Free Software Foundation; either version 2
 #   of the License, or (at your option) any later version.
 # 
 #   This program is distributed in the hope that it will be useful,
 #   but WITHOUT ANY WARRANTY; without even the implied warranty of
 #   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 #   GNU General Public License for more details.
 # 
 #   You should have received a copy of the GNU General Public License
 #   along with this program; if not, write to the Free Software
 #   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 
 #   02111-1307, USA.
 #
 ####################################################################

 package PortScan::ScanContext;

 #
 # Site-specific settings go here.  Currently there are only
 # two that need setting.
 #
 # The ndiff tools try to find ndiff.conf in order of the following:  
 # 
 #  $HOME/.ndiff/ndiff.conf
 #  the directory pointed to by the environment variable NDIFF_ROOT
 #  /usr/local/lib/ndiff/ndiff.conf
 #  ./ndiff.conf
 #
 # This order of precedence is not necessarily rational.
 #

 sub get_config
 {
     return {

         #
         # The configured DataStore type - currently only 
         # NmapFile is implemented
         #

         default_data_store_type => "PortScan::NmapFile",

         #
         # Properties to set in the default DataStore instance.
         # this is a comma-separated list of key=value pairs.
         #
         # For the NmapFile object, only "root" is used, which
         # should point to a directory where results and baseline
         # files will be stored.
         #
         # This directory must be initially created by hand
         # and have appropriate permissions.
         #

         default_data_store_props => "root=/tmp/ndiff_data",

     }
 };

1;