HTML::TableExtract - Perl extension for extracting the text contained in tables within an HTML document.

 # Matched tables are returned as "table state" objects; tables can be
 # matched using column headers, depth, count within a depth, or some
 # combination of the three.

 # Using column header information. Assume an HTML document with
 # tables that have "Date", "Price", and "Cost" somewhere in a
 # row. The columns beneath those headings are what you want to
 # extract. They will be returned in the same order as you specified
 # the headers since 'automap' is enabled by default.

 use HTML::TableExtract;
 $te = new HTML::TableExtract( headers => [qw(Date Price Cost)] );
 $te->parse($html_string);

 # Examine all matching tables
 foreach $ts ($te->table_states) {
   print "Table (", join(',', $ts->coords), "):\n";
   foreach $row ($ts->rows) {
      print join(',', @$row), "\n";
   }
 }

 # Old style, using top level methods rather than table state objects.
 foreach $table ($te->tables) {
   print "Table (", join(',', $te->table_coords($table)), "):\n";
   foreach $row ($te->rows($table)) {
     print join(',', @$row), "\n";
   }
 }

 # Shorthand...top level rows() method assumes the first table found
 # in the document if no arguments are supplied.
 foreach $row ($te->rows) {
    print join(',', @$row), "\n";
 }

 # Using depth and count information. Every table in the document has
 # a unique depth and count tuple, so when both are specified it is a
 # unique table. Depth and count both begin with 0, so in this case we
 # are looking for a table (depth 2) within a table (depth 1) within a
 # table (depth 0, which is the top level HTML document). In addition,
 # it must be the third (count 2) such instance of a table at that
 # depth.

 $te = new HTML::TableExtract( depth => 2, count => 2 );
 $te->parse($html_string);
 foreach $ts ($te->table_states) {
    print "Table found at ", join(',', $ts->coords), ":\n";
    foreach $row ($ts->rows) {
       print "   ", join(',', @$row), "\n";
    }
 }

HTML::TableExtract is a subclass of HTML::Parser that serves to extract the textual information from tables of interest contained within an HTML document. The text from each extracted table is stored in tabe state objects which hold the information as an array of arrays that represent the rows and cells of that table.

There are three constraints available to specify which tables you would like to extract from a document: Headers, Depth, and Count.

Headers, the most flexible and adaptive of the techniques, involves specifying text in an array that you expect to appear above the data in the tables of interest. Once all headers have been located in a row of that table, all further cells beneath the columns that matched your headers are extracted. All other columns are ignored: think of it as vertical slices through a table. In addition, TableExtract automatically rearranges each row in the same order as the headers you provided. If you would like to disable this, set automap to 0 during object creation, and instead rely on the column_map() method to find out the order in which the headers were found. Furthermore, TableExtract will automatically compensate for cell span issues so that columns are really the same columns as you would visually see in a browser. This behavior can be disabled by setting the gridmap parameter to 0. HTML is stripped from the entire textual content of a cell before header matches are attempted unless the keep_html parameter was enabled.

Depth and Count are more specific ways to specify tables in relation to one another. Depth represents how deeply a table resides in other tables. The depth of a top-level table in the document is 0. A table within a top-level table has a depth of 1, and so on. Each depth can be thought of as a layer; tables sharing the same depth are on the same layer. Within each of these layers, Count represents the order in which a table was seen at that depth, starting with 0. Providing both a depth and a count will uniquely specify a table within a document.

Each of the Headers, Depth, and Count specifications are cumulative in their effect on the overall extraction. For instance, if you specify only a Depth, then you get all tables at that depth (note that these could very well reside in separate higher-level tables throughout the document since depth extends across tables). If you specify only a Count, then the tables at that Count from all depths are returned (i.e., the nth occurrence of a table at each depth). If you only specify Headers, then you get all tables in the document containing those column headers. If you have specified multiple constraints of Headers, Depth, and Count, then each constraint has veto power over whether a particular table is extracted.

If no Headers, Depth, or Count are specified, then all tables match.

Text that is gathered from the tables is decoded with HTML::Entities by default; this can be disabled by setting the decode parameter to 0.

Make sure you fully understand the notions of depth and count before proceeding, because it is about to become a bit more involved.

Table matches using Headers, Depth, or Count can be chained together in order to further specify tables relative to one another. Links in chains are successively applied to tables within tables. Top level constraints (i.e., header, depth, and count parameters for the TableExtract object) behave as the first link in the chain. Additional links are specified using the chain parameter. Each link in the chain has its own set of constraints. For example:

 $te = new HTML::TableExtract
   (
    headers => [qw(Summary Region)],
    chain   => [
                { depth => 0, count => 2 },
                { headers => [qw(Part Qty Cost)] }
               ],
   );

The matching process in this case will start with all tables in the document that have Summary and Region in their headers. For now, assume that there was only one table that matched these headers. Each table contained within that table will be compared to the first link in the chain. Depth 0 means that a matching table must be immediately contained within the current table; count 2 means that the matching table must also be the third at that depth (counts and depths start at 0). In other words, the next link of the chain will match on the third table immediately contained within our first matched table. Once this link matches, then all further tables beneath that table that have Part, Qty, and Cost in their headers will match. By default, it is only tables at the end of the chains that are returned to the application, so these tables are returned.

Each time a link in a chain matches a table, an additional context for depth and count is established. It is perhaps easiest to visualize a context as a brand-new HTML document, with new depths and counts to compare to the remaining links in the chain. The top level HTML document is the first context. Each table in the document establishes a new context. Depth in a chain link is relative to the context that the matching table creates (i.e., a link depth of 0 would be a table immediately contained within the table that matched the prior link in the chain). Likewise, that same context keeps track of counts within the new depth scheme for comparison to the remaining links in the chain. Headers still apply if they are present in a link, but they are always independent of context.

As it turns out, specifying a depth and count provides a unique address for a table within a context. For non-unique constraints, such as just a depth, or headers, there can be multiple matches for a given link. In these cases the chain forks and attempts to make further matches within each of these tables.

By default, chains are elastic. This means that when a particular link does not match on a table, it is passed down to subtables unchanged. For example:

 $te = new HTML::TableExtract
   (
    headers => [qw(Summary Region)],
    chain   => [
                { headers => [qw(Part Qty Cost)] }
               ],
   );

If there are intervening tables between the two header queries, they will be ignored; this query will extract all tables with Part, Qty, and Cost in the headers that are contained in any table with Summary and Region in its headers, regardless of how embedded the inner tables are. If you want a chain to be inelastic, you can set the elastic parameter to 0 for the whole TableExtract object. Using the same example:

 $te = new HTML::TableExtract
   (
    headers => [qw(Summary Region)],
    chain   => [
                { headers => [qw(Part Qty Cost)] }
               ],
    elastic => 0,
   );

In this case, the inner table (Part, Qty, Cost) must be immediately contained within the outer table (Summary, Region) in order for the match to take place. This is equivalent to specifying a depth of 0 for each link in the chain; if you only want particular links to be inelastic, then simply set their depths to 0.

By default, only tables that match at the end of the chains are retained. The intermediate matches along the chain are referred to as waypoints, and are not extracted by default. A waypoint may be retained, however, by specifiying the keep parameter in that link of the chain. This parameter may be specified at the top level as well if you want to keep tables that match the first set of constraints in the object. If you want to keep all tables that match along the chain, the specify the keepall parameter at the top level.

Are chains overkill? Probably. In reality, nested HTML tables tend not to be very deep, so there will usually not be much need for lots of links in a chain. Theoretically, however, chains offer precise targeting of tables relative to one another, no matter how deeply nested they are.

What happens with the following table extraction?

 $te = new HTML::TableExtract(
                              chain => [ { depth => 0 } ],
                             );

Answer: All tables that are contained in another table are extracted from the document. In this case, there were no top-level constraints specified, which if you recall means that all tables match the first set of constraints (or non-constraints, in this case!). A depth of 0 in the next link of the chain means that the matching table must be immediately contained within the table from a prior match.

The following is equivalent:

 $te = new HTML::TableExtract(
                              depth     => 1,
                              subtables => 1,
                             )

The subtables parameter tells TableExtract to scoop up all tables contained within the matching tables. In conjunction with a depth of 1, this has the affect of discarding all top-level tables in the document, which is exactly what occurred in the prior example.

The main point of this module was to provide a flexible method of extracting tabular information from HTML documents without relying to heavily on the document layout. For that reason, I suggest using Headers whenever possible that way, you are anchoring your extraction on what the document is trying to communicate rather than some feature of the HTML comprising the document (other than the fact that the data is contained in a table).

HTML::TableExtract is a subclass of HTML::Parser, and as such inherits all of its basic methods. In particular, CWstart(), CWend(), and CWtext() are utilized. Feel free to override them, but if you do not eventually invoke them in the SUPER class with some content, results are not guaranteed.

The following are the top-level methods of the HTML::TableExtract object. Tables that have matched a query are actually returned as separate objects of type HTML::TableExtract::TableState. These table state objects have their own methods, documented further below. There are some top-level methods that are present for convenience and backwards compatibility that are nothing more than front-ends for equivalent table state methods.

new()

Return a new HTML::TableExtract object. Valid attributes are:

headers

Passed as an array reference, headers specify strings of interest at the top of columns within targeted tables. These header strings will eventually be passed through a non-anchored, case-insensitive regular expression, so regexp special characters are allowed. The table row containing the headers is not returned. Columns that are not beneath one of the provided headers will be ignored. Columns will, by default, be rearranged into the same order as the headers you provide (see the automap parameter for more information). Additionally, by default columns are considered what you would see visually beneath that header when the table is rendered in a browser. See the gridmap parameter for more information. HTML within a header is stripped before the match is attempted, unless the keep_html parameter was specified.

depth

Specify how embedded in other tables your tables of interest should be. Top-level tables in the HTML document have a depth of 0, tables within top-level tables have a depth of 1, and so on.

count

Specify which table within each depth you are interested in, beginning with 0.

chain

List of additional constraints to be matched sequentially from the top level constraints. This is a reference to an array of hash references. Each hash is a link in the chain, and can be specified in terms of depth, count, and headers. Further modifiers include keep, which means to retain the table if it would normally be dropped as a waypoint.

automap

Automatically applies the ordering reported by column_map() to the rows returned by rows(). This only makes a difference if you have specified Headers and they turn out to be in a different order in the table than what you specified. Automap will rearrange the columns in the same order as the headers appear. To get the original ordering, you will need to take another slice of each row using column_map(). automap is enabled by default.

gridmap

Controls whether the table contents are returned as a grid or a tree. ROWSPAN and COLSPAN issues are compensated for, and columns really are columns. Empty phantom cells are created where they would have been obscured by ROWSPAN or COLSPAN settings. This really becomes an issue when extracting columns beneath headers. Enabled by default.

keepall

Keep all tables that matched along a chain, including tables matched by top level contraints. By default, waypoints are dropped and only the matches at the end of the chain are retained. To retain a particular waypoint along a chain, use the keep parameter in that link.

elastic

When set to 0, all links in chains will be treated as though they had a depth of 0 specified, which means there can be no intervening unmatched tables between matches on links.

subtables

Extract all tables within matched tables.

decode

Automatically decode retrieved text with HTML::Entities::decode_entities(). Enabled by default.

br_translate

Translate <br> tags into newlines. Sometimes the remaining text can be hard to parse if the <br> tag is simply dropped. Enabled by default. Has no effect if keep_html is enabled.

keep_html

Return the raw HTML contained in the cell, rather than just the visible text. Embedded tables are not retained in the HTML extracted from a cell. Patterns for header matches must take into account HTML in the string if this option is enabled.

debug

Prints some debugging information to STDOUT, more for higher values.

depths()

Returns all depths that contained matched tables in the document.

counts($depth)

For a particular depth, returns all counts that contained matched tables. For a particular depth and count, return the table state object for the table found, if any.

table_states()

Return table state objects for all tables that matched.

first_table_state_found()

Return the table state object for the first table matched in the document.

The following methods are invoked from an HTML::TableExtract::TableState object, such as those returned from the CWtable_states() method.

rows()

Return all rows within a matched table. Each row returned is a reference to an array containing the text of each cell.

depth()

Return the (absolute) depth at which this table was found.

count()

Return the count for this table within the depth it was found.

coords()

Return depth and count in a list.

column_map()

Return the order (via indices) in which the provided headers were found. These indices can be used as slices on rows to either order the rows in the same order as headers or restore the rows to their natural order, depending on whether the rows have been pre-adjusted using the automap parameter.

lineage()

Returns the path of matched tables that led to matching this table. Lineage only makes sense if chains were used. Tables that were not matched by a link in the chain are not included in lineage. The lineage path is a list of array refs containing depth and count values for each table involved.

The following top level methods are alternatives to invoking methods in a table state object. If you do not want to deal with table state objects, then these methods are for you. The tables they deal in are actually just arrays of arrays, which happen to be the current internal data structure of the table state objects. They are here for backwards compatibility. Same as CWtable_state(), but returns the internal data structure rather than the table state object.

tables()

Same as CWtable_states(), but returns the data structures rather than the table state objects.

first_table_found()

Same as CWfirst_table_state_found(), except returns the data structure for first table that matched.

table_coords($table)

Returns the depth and count for a particular table data structure. See the CWcoords() method provided by table state objects.

rows()

rows($table)

Return a lsit of the rows for a particular table data structure (first table found by default). See the CWrows() method provided by table state objects.

column_map()

column_map($table)

Return the column map for a particular table data structure (first found by default). See the CWcolumn_map() method provided by table state objects.

HTML::Parser(3), HTML::Entities(3)

Matthew P. Sisk, <sisk@mojotoad.com>

Copyright (c) 2000-2002 Matthew P. Sisk. All rights reserved. All wrongs revenged. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

HTML::Parser(3), perl(1).