Text::Autoformat - Automatic text wrapping and reformatting

This document describes version 1.12 of Text::Autoformat, released May 27, 2003.

 # Minimal use: read from STDIN, format to STDOUT...

        use Text::Autoformat;
        autoformat;

 # In-memory formatting...

        $formatted = autoformat $rawtext;

 # Configuration...

        $formatted = autoformat $rawtext, { %options };

 # Margins (1..72 by default)...

        $formatted = autoformat $rawtext, { left=>8, right=>70 };

 # Justification (left by default)...

        $formatted = autoformat $rawtext, { justify => 'left' };
        $formatted = autoformat $rawtext, { justify => 'right' };
        $formatted = autoformat $rawtext, { justify => 'full' };
        $formatted = autoformat $rawtext, { justify => 'centre' };

 # Filling (does so by default)...

        $formatted = autoformat $rawtext, { fill=>0 };

 # Squeezing whitespace (does so by default)...

        $formatted = autoformat $rawtext, { squeeze=>0 };

 # Case conversions...

        $formatted = autoformat $rawtext, { case => 'lower' };
        $formatted = autoformat $rawtext, { case => 'upper' };
        $formatted = autoformat $rawtext, { case => 'sentence' };
        $formatted = autoformat $rawtext, { case => 'title' };
        $formatted = autoformat $rawtext, { case => 'highlight' };

 # Selective reformatting

        $formatted = autoformat $rawtext, { ignore=>qr/^\t/ };

Perl plaintext formatters just aren't smart enough. Given a typical piece of plaintext in need of formatting:

        In comp.lang.perl.misc you wrote:
        : > <CN = Clooless Noobie> writes:
        : > CN> PERL sux because:
        : > CN>    * It doesn't have a switch statement and you have to put $
        : > CN>signs in front of everything
        : > CN>    * There are too many OR operators: having |, || and 'or'
        : > CN>operators is confusing
        : > CN>    * VB rools, yeah!!!!!!!!!
        : > CN> So anyway, how can I stop reloads on a web page?
        : > CN> Email replies only, thanks - I don't read this newsgroup.
        : >
        : > Begone, sirrah! You are a pathetic, Bill-loving, microcephalic
        : > script-infant.
        : Sheesh, what's with this group - ask a question, get toasted! And how
        : *dare* you accuse me of Ianuphilia!

both the venerable Unix fmt tool and Perl's standard Text::Wrap module produce:

        In comp.lang.perl.misc you wrote:  : > <CN = Clooless Noobie>
        writes:  : > CN> PERL sux because:  : > CN>    * It doesn't
        have a switch statement and you have to put $ : > CN>signs in
        front of everything : > CN>    * There are too many OR
        operators: having |, || and 'or' : > CN>operators is confusing
        : > CN>    * VB rools, yeah!!!!!!!!!  : > CN> So anyway, how
        can I stop reloads on a web page?  : > CN> Email replies only,
        thanks - I don't read this newsgroup.  : > : > Begone, sirrah!
        You are a pathetic, Bill-loving, microcephalic : >
        script-infant.  : Sheesh, what's with this group - ask a
        question, get toasted! And how : *dare* you accuse me of
        Ianuphilia!

Other formatting modules such as Text::Correct and Text::Format provide more control over their output, but produce equally poor results when applied to arbitrary input. They simply don't understand the structural conventions of the text they're reformatting.

The Text::Autoformat module provides a subroutine named CWautoformat that wraps text to specified margins. However, CWautoformat reformats its input by analysing the text's structure, so it wraps the above example like so:

        In comp.lang.perl.misc you wrote:
        : > <CN = Clooless Noobie> writes:
        : > CN> PERL sux because:
        : > CN>    * It doesn't have a switch statement and you
        : > CN>      have to put $ signs in front of everything
        : > CN>    * There are too many OR operators: having |, ||
        : > CN>      and 'or' operators is confusing
        : > CN>    * VB rools, yeah!!!!!!!!! So anyway, how can I
        : > CN>      stop reloads on a web page? Email replies
        : > CN>      only, thanks - I don't read this newsgroup.
        : >
        : > Begone, sirrah! You are a pathetic, Bill-loving,
        : > microcephalic script-infant.
        : Sheesh, what's with this group - ask a question, get toasted!
        : And how *dare* you accuse me of Ianuphilia!

Note that the various quoting conventions have been observed. In fact, their structure has been used to determine where some paragraphs begin. Furthermore CWautoformat correctly distinguished between the leading '*' bullets of the nested list (which were outdented) and the leading emphatic '*' of *dare* (which was inlined).

The fundamental task of the CWautoformat subroutine is to identify and rearrange independent paragraphs in a text. Paragraphs typically consist of a series of lines containing at least one non-whitespace character, followed by one or more lines containing only optional whitespace. This is a more liberal definition than many other formatters use: most require an empty line to terminate a paragraph. Paragraphs may also be denoted by bulleting, numbering, or quoting (see the following sections).

Once a paragraph has been isolated, CWautoformat fills and re-wraps its lines according to the margins that are specified in its argument list. These are placed after the text to be formatted, in a hash reference:

        $tidied = autoformat($messy, {left=>20, right=>60});

By default, CWautoformat uses a left margin of 1 (first column) and a right margin of 72.

You can also control whether (and how) CWautoformat breaks words at the end of a line, using the CW'break' option:

        # Turn off all hyphenation
        use Text::Autoformat qw(autoformat break_wrap);
        $tidied = autoformat($messy, {break=>break_wrap});

        # Default hyphenation
        use Text::Autoformat qw(autoformat break_at);
        $tidied = autoformat($messy, {break=>break_at('-')});

        # Use TeX::Hyphen module's hyphenation (module must be installed)
        use Text::Autoformat qw(autoformat break_TeX);
        $tidied = autoformat($messy, {break=>break_TeX});

Normally, CWautoformat only reformats the first paragraph it encounters, and leaves the remainder of the text unaltered. This behaviour is useful because it allows a one-liner invoking the subroutine to be mapped onto a convenient keystroke in a text editor, to provide one-paragraph-at-a-time reformatting:

        % cat .exrc

        map f !Gperl -MText::Autoformat -e'autoformat'

(Note that to facilitate such one-liners, if CWautoformat is called in a void context without any text data, it takes its text from CWSTDIN and writes its result to CWSTDOUT).

To enable CWautoformat to rearrange the entire input text at once, the CWall argument is used:

        $tidied_all = autoformat($messy, {left=>20, right=>60, all=>1});

CWautoformat can also be directed to selectively reformat paragraphs, using the CWignore argument:

        $tidied_some = autoformat($messy, {ignore=>qr/^[ \t]/});

The value for CWignore may be a CWqr'd regex, a subroutine reference, or the special string CW'indented'.

If a regex is specified, any paragraph whose original text matches that regex will not be reformatted (i.e. it will be printed verbatim).

If a subroutine is specified, that subroutine will be called once for each paragraph (with CW$_ set to the paragraph's text). The subroutine is expected to return a true or false value. If it returns true, the paragraph will not be reformatted.

If the value of the CWignore option is the string CW'indented', CWautoformat will ignore any paragraph in which every line begins with a whitespace.

Often plaintext will include lists that are either:

        * bulleted,
        * simply numbered (i.e. 1., 2., 3., etc.), or
        * hierarchically numbered (1, 1.1, 1.2, 1.3, 2, 2.1. and so forth).

In such lists, each bulleted item is implicitly a separate paragraph, and is formatted individually, with the appropriate indentation:

        * bulleted,
        * simply numbered (i.e. 1., 2., 3.,
          etc.), or
        * hierarchically numbered (1, 1.1,
          1.2, 1.3, 2, 2.1. and so forth).

More importantly, if the points are numbered, the numbering is checked and reordered. For example, a list whose points have been rearranged:

        2. Analyze problem
        3. Design algorithm
        1. Code solution
        5. Test
        4. Ship

would be renumbered automatically by CWautoformat:

        1. Analyze problem
        2. Design algorithm
        3. Code solution
        4. Ship
        5. Test

The same reordering would be performed if the numbering was by letters (CWa. CWb. CWc. etc.) or Roman numerals (CWi. CWii. CWiii.) or by some combination of these (CW1a. CW1b. CW2a. CW2b. etc.) Handling disordered lists of letters and Roman numerals presents an interesting challenge. A list such as:

        C. Put cat in box.
        D. Close lid.
        E. Activate Geiger counter.

should be reordered as CWA. CWB. CWC., whereas:

        C. Put cat in box.
        D. Close lid.
        XLI. Activate Geiger counter.

should be reordered CWI. CWII. CWIII.

The CWautoformat subroutine solves this problem by always interpreting alphabetic bullets as being letters, unless the full list consists only of valid Roman numerals, at least one of which is two or more characters long.

If automatic renumbering isn't wanted, just specify the CW'renumber' option with a false value.

Note that numbers above 1000 at the start of a line are no longer considered to be paragraph numbering. Numbered paragraphs running that high are exceptionally rare, and much rarer than paragraphs that look like this:

        Although it has long been popular (especially in the year
        2001) to point out that we now live in the Future, many
        of the promised miracles of Future Life have failed to
        eventuate. This is a new phenomenon (it didn't happen in
        1001) because the idea that the future might be different
        is a new phenomenon.

which the former numbering rules caused to be formatted like this:

        Although it has long been popular (especially in the year

        2001) to point out that we now live in the Future, many of the
              promised miracles of Future Life have failed to eventuate.
              This is a new phenomenon (it didn't happen in

        2002) because the idea that the future might be different is a
              new phenomenon.

but which are now formatted:

        Although it has long been popular (especially in the year 2001)
        to point out that we now live in the Future, many of the
        promised miracles of Future Life have failed to eventuate. This
        is a new phenomenon (it didn't happen in 1001) because the idea
        that the future might be different is a new phenomenon.

Another case in which contiguous lines may be interpreted as belonging to different paragraphs, is where they are quoted with distinct quoters. For example:

        : > CN> So anyway, how can I stop reloads on a web page? Email
        : > CN> replies only, thanks - I don't read this newsgroup.
        : > Begone, sirrah! You are a pathetic, Bill-loving,
        : > microcephalic script-infant.
        : Sheesh, what's with this group - ask a question, get toasted!
        : And how *dare* you accuse me of Ianuphilia!

CWautoformat recognizes the various quoting conventions used in this example and treats it as three paragraphs to be independently reformatted.

Block quotations present a different challenge. A typical formatter would render the following quotation:

        "We are all of us in the gutter, but some of us are looking at
         the stars"
                                -- Oscar Wilde

like so:

        "We are all of us in the gutter, but some of us are looking at
        the stars" -- Oscar Wilde

CWautoformat recognizes the quotation structure by matching the following regular expression against the text component of each paragraph:

        / \A(\s*) # leading whitespace for quotation (["']|``) # opening
        quotemark (.*) # quotation (''|\2) # closing quotemark \s*?\n #
        trailing whitespace after quotation (\1[ ]+) # leading
        whitespace for attribution
                                #   (must be indented more than
                                #   quotation)
          (--|-) # attribution introducer ([^\n]*?\n) # first
          attribution line ((\5[^\n]*?$)*) # other attribution lines
                                #   (indented no less than first line)
          \s*\Z # optional whitespace to end of paragraph /xsm

When reformatted (see below), the indentation and the attribution structure will be preserved:

        "We are all of us in the gutter, but some of us are looking at
         the stars"
                                -- Oscar Wilde

Note that in the last example, CWautoformat broke the line at column 68, four characters earlier than it should have. It did so because, if the full margin width had been used, the formatting would have left the last two words by themselves on an oddly short last line:

        "We are all of us in the gutter, but some of us are looking at
        the stars"

This phenomenon is known as widowing and is heavily frowned upon in typesetting circles. It looks ugly in plaintext too, so CWautoformat avoids it by stealing extra words from earlier lines in a paragraph, so as to leave enough for a reasonable last line. The heuristic used is that final lines must be at least 10 characters long (though this number may be adjusted by passing a CWwidow => CIminlengthCW argument to CWautoformat).

If the last line is too short, the paragraph's right margin is reduced by one column, and the paragraph is reformatted. This process iterates until either the last line exceeds nine characters or the margins have been narrowed by 10% of their original separation. In the latter case, the reformatter gives up and uses its original formatting.

The CWautoformat subroutine also takes a named argument: CW{justify => CItypeCW}, which specifies how each paragraph is to be justified. The options are: CW'left' (the default), CW'right', CW'centre' (or CW'center'), and CW'full'. These act on the complete paragraph text (but not on any quoters before that text). For example, with CW'right' justification:

         R3>     Now is the Winter of our discontent made
         R4> glorious Summer by this son of York. And all
         R5> the clouds that lour'd upon our house In the
         R6>              deep bosom of the ocean buried.

Full justification is interesting in a fixed-width medium like plaintext because it usually results in uneven spacing between words. Typically, formatters provide this by distributing the extra spaces into the first available gaps of each line:

         R7> Now is the Winter of our discontent made
         R8> glorious Summer by this son of York. And all
         R9> the clouds that lour'd upon our house In
        R10> the deep bosom of the ocean buried.

This produces a rather jarring visual effect, so CWautoformat reverses the strategy and inserts extra spaces at the end of lines:

        R11> Now is the Winter of our discontent made
        R12> glorious Summer by this son of York. And all
        R13> the clouds that lour'd upon our house In
        R14> the deep bosom of the ocean buried.

Most readers find this less disconcerting.

Even if explicit centring is not specified, CWautoformat will attempt to automatically detect centred paragraphs and preserve their justification. It does this by examining each line of the paragraph and asking: if this line were part of a centred paragraph, where would the centre line have been?

The answer can be determined by adding the length of leading whitespace before the first word, plus half the length of the full set of words on the line. That is, for a single line:

        $line =~ /^(\s*)(.*?)(\s*)$/ $centre =
        length($1)+0.5*length($2);

By making the same estimate for every line, and then comparing the estimates, it is possible to deduce whether all the lines are centred with respect to the same axis of symmetry (with an allowance of E<plusmn>1 to cater for the inevitable rounding when the centre positions of even-length rows were originally computed). If a common axis of symmetry is detected, CWautoformat assumes that the lines are supposed to be centred, and switches to centre-justification mode for that paragraph.

Note that this behaviour can to switched off entirely by setting the CW"autocentre" argument false.

The CWautoformat subroutine can also optionally perform case conversions on the text it processes. The CW{case => CItypeCW} argument allows the user to specify five different conversions: This mode unconditionally converts every letter in the reformatted text to upper-case; This mode unconditionally converts every letter in the reformatted text to lower-case; This mode attempts to generate correctly-cased sentences from the input text. That is, the first letter after a sentence-terminating punctuator is converted to upper-case. Then, each subsequent word in the sentence is converted to lower-case, unless that word is originally mixed-case or contains punctuation. For example, under CW{case => 'sentence'}:

        'POVERTY, MISERY, ETC. are the lot of the PhD candidate. alas!'

        'Poverty, misery, etc. are the lot of the PhD candidate. Alas!'

        'Poverty, misery, etc. are the lot of the PhD candidate. Alas!'

        'What I Did On My Summer Vacation In Monterey'

        'What I Did on my Summer Vacation in Monterey'

You can select which paragraphs CWautoformat actually reformats (or, rather, those it doesn't reformat) using the CW"ignore" flag.

For example:

        # Reformat all paras except those containing "verbatim"...
        print autoformat { all => 1, ignore => qr/verbatim/i }, $text;

        # Reformat all paras except those less that 3 lines long...
        print autoformat { all => 1, ignore => sub { tr/\n/\n/ < 3
        } }, $text;

        # Reformat all paras except those that are indented...
        print autoformat { all => 1, ignore => qr/^\s/m }, $text;

        # Reformat all paras except those that are indented (easier)...
        print autoformat { all => 1, ignore => 'indented' }, $text;

The Text::Reform module

Damian Conway (damian@conway.org)

There are undoubtedly serious bugs lurking somewhere in code this funky :-) Bug reports and other feedback are most welcome.

Copyright (c) 1997-2000, Damian Conway. All Rights Reserved. This module is free software. It may be used, redistributed and/or modified under the terms of the Perl Artistic License (see http://www.perl.com/perl/misc/Artistic.html)