Bric::Hacker - A guide for Bricolage hackers.

$LastChangedRevision$

$LastChangedDate: 2005-10-13 13:21:43 -0700 (Thu, 13 Oct 2005) $

This document is designed to provide information useful to Bricolage developers. If you've got questions about hacking Bricolage that aren't answered here please post to the Bricolage developer's mailing-list (see below) and tell us about it.

Bricolage has a number of mailing-lists that are relevant to developers:

users@lists.bricolage.cc

Questions, installation problems, bug reports, job and site announcements, other general discussion. Posts to this list are subscriber-only, because otherwise we would be inundated with spam. Please subscribe before posting. Subscribe: <http://www.bricolage.cc/support/lists/> Archives: <http://marc.theaimsgroup.com/?l=bricolage-general>

announce@lists.bricolage.cc

Official announcements from the Bricolage team. These are always Cc'd to the users list, so you need not subscribe to both. Very low traffic. (read-only except for core developers) Subscribe: <http://www.bricolage.cc/support/lists/> Archives: <http://marc.theaimsgroup.com/?l=bricolage-announce>

devel@lists.bricolage.cc

In-depth discussions about new development directions, proposed features, implementation details. Posts to this list are subscriber-only, because otherwise we would be inundated with spam. Please subscribe before posting. Subscribe: <http://www.bricolage.cc/support/lists/> Archives: <http://marc.theaimsgroup.com/?l=bricolage-devel>

commits@lists.bricolage.cc

Subversion context diffs for all checkins to the Bricolage repository. (read-only). Subscribe: <http://www.bricolage.cc/support/lists/> Archives: <http://marc.theaimsgroup.com/?l=bricolage-commits>

bugs@lists.bricolage.cc

Bricolage Bugzilla activity. This list is read-only. To report a bug in Bricolage, please use our Bugzilla server. Bugzilla: <http://bugs.bricolage.cc/> Subscribe: <http://www.bricolage.cc/support/lists/> Archives: <http://marc.theaimsgroup.com/?l=bricolage-bugs>

You can often find folks hanging out and occasionally discussing development issues on the #bricolage channel on the MagNet IRC network.

If you're developing Bricolage then you should be working with the latest code from the Bricolage Subversion repository. You can browse the Subversion repository at <http://viewsvn.bricolage.cc/>.

Information on connecting to the repository to checkout a working copy is at <http://www.bricolage.cc/dev/subversion/>.

Bricolage has a Bugzilla server dedicated to it:

   http://bugzilla.bricolage.cc/

You should use this system to report bugs in Bricolage. If you're looking for something to do you can also use the system to find open bugs and fix them. For more things to do see Bric::ToDo.

Patches should be generated using the CWsvn diff command on each of the files modified, from the root directory. For example, if you made changes to lib/Bric/Changes.pod and comp/foo.mc, you would generate a diff by running this command from the root of your Subversion checkout:

  svn diff lib/Bric/Changes.pod comp/foo.mc > patch.txt

If you created one or more new files in your changes then you'll have to add them to the patch separately using normal CWdiff against CW/dev/null. For example, if you created the file CWinst/upgrade/1.9.1/solve_fermat.pl then you would add this to patch.txt with:

  diff -u /dev/null inst/upgrade/1.9.1/solve_fermat.pl >> patch.txt

Always create patches using an up-to-date Subversion checkout if possible. Send your patches to the devel list mentioned above.

Patches created using the method above can be applied using CWpatch with the CW-p0 option from the root of your Subversion checkout:

  patch -p0 < patch.txt

Make sure you check the results with CWsvn diff before committing.

Try to follow the style of the existing Bricolage code. Except where it is bad, of course. Basically, write in the style you see in most Perl books and documentation, particularly perlstyle.

Although historically the Bricolage code has not enforced whitespace rules, we now request that you use 4-space indents (2 spaces for continued lines) and discourage the use of tabs. The following settings for some of the more popular editors are thus recommended while editing Bricolage source code.

Emacs

We strongly recommend that you use CWcperl-mode while editing Bricolage sources in Emacs. Grab the latest version from the CPAN, install it, and then place the following in your CW~/.emacs file:

  (setq-default indent-tabs-mode nil)
  (custom-set-variables
   '(case-fold-search t)
   '(cperl-close-paren-offset -4)
   '(cperl-continued-statement-offset 2)
   '(cperl-indent-level 4)
   '(cperl-indent-parens-as-block t)
   '(cperl-tab-always-indent t)
   '(indent-tabs-mode nil))
   ;; let hashes indent normally; I think this requires
   ;; at least version 4.32 of cperl-mode.el
   '(cperl-indent-parens-as-block t)
   '(cperl-close-paren-offset -4)

Also, if you'd like to take advantage of the full functionality of CWcperl-mode and have it automatically parse all Perl source files, add these settings, as well:

  (defalias 'perl-mode 'cperl-mode)
  (setq auto-mode-alist
        (append
         '(("\\.\\([pP]\\([Llm]\\|erl\\)\\|al\\|pod\\)\\'" . cperl-mode))
         auto-mode-alist))
    (setq cperl-hairy t)
    (setq interpreter-mode-alist (append interpreter-mode-alist
       '(("miniperl" . cperl-mode))))

When editing Bricolage Mason components, CWmmm-mode can help. It's Mason mode will parse Mason component files and use CWsgml-mode in HTML spaces and CWcperl-mode in Mason blocks. Grab it from <http://mmm-mode.sourceforge.net/>, install it, and then add the following to your CW~/.emacs file to have it automatically parse your Bricolage Mason component files:

  (add-to-list 'load-path "/usr/local/share/emacs/site-lisp")
  (require 'mmm-mode)
  (require 'mmm-mason)
  (setq mmm-global-mode 'maybe)
  (add-to-list 'auto-mode-alist '("/usr/share/bricolage/comp" . sgml-mode))
  (mmm-add-mode-ext-class 'sgml-mode "/usr/share/bricolage/comp" 'mason)

And if you need to examine the Mason object files created by Bricolage in order to chase down bugs and such, you can use the CWcperl-mode in those files by adding this to your CW~/.emacs file:

  (add-to-list 'auto-mode-alist '("/usr/share/bricolage/data/obj" . cperl-mode))

Vim

Rafael Garcia-Suarez has written a Vim indent macro which (for the most part) duplicates the behavior of Emacs CWperl-mode. It is now, as of Vim 6.0, included in the Vim distribution and should be found in CW$VIMRUNTIME/indent/perl. The easiest way to use it though is to place the following line in your .vimrc file:

    source $VIMRUNTIME/indent.vim

You'll also need to add these lines.

    set tabstop=8
    set softtabstop=4
    set shiftwidth=4
    set expandtab

The first three lines make Vim duplicate the behavior of Emacs in creating the appearance of 4 space tabs with a mix of tabs and spaces. This is necessary for reading older Bricolage files which were written this way, and haven't yet been re-tabbed.

The expandtab setting does the Right Thing under the new rules, in that it doesn't use tabs at all, only spaces.

SQL

The standard for writing SQL in Bricolage is pretty straight-forward: format the SQL in 80 columns or less, and use heredocs where possible. For example:

  my $sql = "<<    END_SQL";
      SELECT id, name, description, publish_date, cover_date, current_version,
             publish_status, active
      FROM   story
      WHERE  id = ?
      END_SQL

Also, when writing your queries, please follow the following rules with regard to table aliases:

•

Do not alias any table names in queries, unless those table names are longer than 8 characters, or are referred to in the FROM clause more than once. Under this rule, for example, story and member would never be aliased unless referenced twice.

•

For table names over 8 characters, abbreviated aliases are acceptable provided that they are still long enough to be informative. i.e. st_cat instead of sc for story_category.

•

For multiple links to the same table, use an alias which is long enough to be informative, such as story2 or inst_2 for the second reference to story or story_instance.

The reason for these rules is that single-letter aliases for queries are as unreadable and unmaintainable as single-letter variables names. You get halfway down the page, and you can no longer remember to what tables they refer.

The rules above are actually taken from O'Reilly's Introduction to PL/SQL Programming, which has an excellent chapter on code cleanliness in SQL and SQL-extension languages.

Keep subroutines short. Each subroutine should handle one task. For example, if you have a subroutine CWget_foo, and getting foo requires getting bar and qux, then make two more subroutines CWget_bar and CWget_qux and put them in the private functions section. Also if you duplicate some code in several subroutines, factor it out into another subroutine. It's easier to maintain one subroutine than to maintain three.

Limit the width of your code to 80 characters if possible. This makes them easier to read. Keeping subroutines short helps to meet this goal, as well.

Make enough comments so that someone maintaining your code can understand what is going on. Comments shouldn't be redundant with the code. They should explain non-obvious code. Comments can be bad in some cases if code changes and the corresponding comment is not kept in sync. So keep the comments in sync!

Use Perl idioms if it is clear and concise, but use them with care. Implicit variables can be slick, but hard to understand; add a comment if you use uncommon ones.

Use Mason components only for display, and put business logic into library modules and callback components. But only for as long as callbacks are in components! We hope to move them into libraries soon, at which time this rule will become even more rigid.

For POD, generally you copy/paste it from another module. Recently we are trying to cut out some of useless things like CWB<Side Effects:> NONE. For a canonical example, see CWBric::Biz::Site.

Avoid magic values, i.e. don't hard-code numbers into code. It tends to become a maintenance problem if you put the number 1023 throughout a module and then somewhere else you have 1021; is 1021 related to 1023 (1023 - 2), or is it just another random number? And what does 1023 mean? Put a constant at the top of the module and use that, instead.

In the end, just try to follow the existing code style, and take into consideration any feedback you get from the mailing list when patches are submitted.

All Bricolage patches should be accompanied by the necessary tests. You are strongly encouraged to write comprehensive tests that thoroughly test whatever changes or additions you make to the Bricolage API. You are also strongly encouraged to write tests for the current API, if you find that adequate tests have not yet been written (and this is true in a great many places, unfortunately). Although Bricolage does not yet support UI tests, we take our API testing seriously, and so should you. Here's what you need to know to write tests for Bricolage.

Bricolage contains a test suite based on Test::Class. The test classes live in the t/Bric directory, and all subclass Bric::Test::Base. If you're familiar with Test::More, then the syntax for how the Test::Class-based classes work should be pretty readily apparent. See the Test::Class documentation for details. It's definitely worth a read. If you haven't used Test::More, its documentation is also a must-read.

Bricolage has two different sets of tests, those run by CWmake test and those run by CWmake devtest. The former are stand-alone tests that don't rely on the presence of a Bricolage database to be run. They can thus be run before CWmake install. The idea here is that we offer a basic set of tests that can be run via the standard Perl installation pattern of

  perl Makefile.PL
  make
  make test
  make install

The tests run by <make devtest> are intended to be run during development. They require that a Bricolage database be installed and running. They will CWSELECT, CWINSERT, CWUPDATE, and CWDELETE data from that database, so CWmake devtest should never be run against a production database. Indeed, you should in general have a clean, fresh database installation to run the tests against. Although the test suite makes every effort to clean up after itself by CWDELETEing data it has added to the database, it is unfortunately imperfect, and extra data will be left, especially in Group-related and Attribute-related tables.

So, once more, CWmake devtest should never be run against a production database. 'Nuff said.

You can get a clean database without reinstalling Bricolage by doing

  sudo make db
  sudo make db_grant

Answer yes when it asks you to drop the database. (Note: if there were any database changes, you'll also need to run upgrade scripts in inst/upgrade/.) Then to get useful debugging output, do verbose testing with something like

  sudo make devtest TEST_VERBOSE=1 > test.out 2>&1

All of the tests are run by the inst/runtests.pl script, which in turn tells Test::Harness to execute t/Bric/Test/Runner.pm. This file will find all the necessary test classes, load them, and then run their tests. inst/runtests.pl takes a number of arguments to simplify the running of scripts, including a list of test files or classes to run, so that you can just run the tests you need to run while you're developing new tests. CWperldoc inst/runtests.pl for more information.

Be sure to use the testing base classes in your test classes. For non-database dependent tests (which are always in files named Test.pm, use Bric::Test::Base. For development tests, use Bric::Test::DevBase, which inherits from Bric::Test::Base. Be sure to read the documentation in these classes, as it will help you to write your tests more effectively. Pay special attention to the methods added to Bric::Test::DevBase, as they're there to help you clean up any new records you've added to the database.

And finally, when you do provide patches to Bricolage, along with the new tests to test them, make sure that all existing tests pass, as well. This means that you should always run CWmake devtest on a fresh database build with your changes. Furthermore, if your patch involves changes to the database, you should provide the necessary upgrade script in inst/upgrade/, and also run the tests against a database that has been built from Bricolage sources untouched by your patch, and then upgraded by your upgrade script. This will help to ensure that your upgrade script modifies the database in the same way as your patch modifies the Bricolage SQL files.

And with that said, happy testing!

Bricolage is a complex application and debugging can be difficult. Here are some tips to help you find bugs faster:

•

Turn on CWQA_MODE in your bricolage.conf file. This will cause the UI to display a bunch of useful data at the bottom of every page, including session state, cache state, cookies, and GET and POST args. CWQA_MODE turns on PerlWarn (CWuse warnings) and causes error messages to include more information. CWQA_MODE also enables Apache::Status at the URL /perl-status which allows you can to examine the state of the Perl interpreter inside Apache.

•

Set the CWNO_TOOLBAR directive to 0 or Off in bricolage.conf to keep Bricolage from popping up a new browser window without a toolbar.

•

Run Bricolage under the Perl debugger using the debug command with bric_apachectl:

   bric_apachectl debug

This will start Apache in single-process mode (CWhttpd -X) and setup Apache::DB to start the debugger on the each hit to the server. You'll need to install the Apache::DB Perl module to use this command. To run Bricolage under DDD (<http://www.gnu.org/software/ddd/>), start ddd as root and load bin/bric_apachectl. Give it the argument debug and run it. When you issue a hit to the server the debugger will stop on the first line of CWBric::App::Handler::handler(). From there you can set breakpoints inside Bricolage and debug normally. If you prefer to run without the debugger using only the Apache single-process mode, then run Bricolage using the command

   bric_apachectl single

•

Set CWDBI_DEBUG and CWDBI_CALL_TRACE to 1 in your bricolage.conf file. CWDBI_DEBUG records every database call in the logs complete with SQL and arguments. CWDBI_CALL_TRACE adds a a subroutine call trace for each statement showing where the database call originated. This generates a lot of data but it can be very helpful.

•

Look at the database directly using psql. Many bugs in Bricolage can only be successfully diagnosed by examining records created in the database.

Bricolage has two separate profiling systems that you can use to extract performance data:

•

To run Bricolage under the Devel::Profiler module set the CWPROFILE variable on in your bricolage.conf file and restart your server. This will create a profiler/$$/tmon.out file in your Apache log directory for each server process. You can use dprofpp to analyze these files after the server has been stopped with CWbric_apachectl stop. See the Devel::Profiler documentation for more details.

•

The database profiler is activated by enabling the CWDBI_PROFILE option in bricolage.conf. This causes database profiling traces to be written to the Apache error log during requests. You can then use bric_dbprof to analyze these trace. See bric_dbprof for details.

CAUTION: Neither of these options is appropriate for a production system.

A relatively simple way to contribute to Bricolage is to provide actions and movers. These are plugin modules that can add new functionality to Bricolage without needing to make changes to the existing API.

An action is an act that is performed on files before they are distributed. Say you want to clean the HTML of all of your HTML files before they're distributed. You'll need to create an action to do this. Consult Bric::Dist::Action for information on how to create actions.

Say you need to distribute files via a protocol that Bricolage doesn't currently support say, an new variant of FTP called FooTP. You'll need to create a new mover. Consult Bric::Dist::Action::Mover for details on how to do that.

If you're a Bricolage developer with permission to commit to the Subversion repository, you may occasionally have to merge changes from a release branch (where bug fixes are generally committed) into the trunk. Here's how to do it with a minimum of hassle.

1

Make sure that both the branch you are merging from and the trunk are updated and have no uncommitted changes. You can check for these differences by using the Subversion CWstatus and CWupdate commands. Running CWsvn status will tell you if there have been any uncommitted changes to the local copy of the branch or trunk. Running CWsvn update will merge in any updates from the copy on the Subversion server. So we recommend running both of these commands in both the branch and the trunk and committing any necessary changes before continuing.

  % cd bricolage-trunk
  % svn status
  % svn update
  At revision 6123.

  % cd bricolage-branch-checkout
  % svn status
  % svn update
  At revision 6123.

2

To merge the changes from the branch into the trunk, you need to tell the Subversion CWmerge command to use the difference between two revisions in the branch to update the trunk. The two revisions to compare are, on the one hand, either the revision when the branch was created, or the revision when the branch was last merged into the trunk; and on the other hand, the current revision in the branch. For the latter number, you can simply use the CWHEAD keyword, which represents the latest revision number in the branch. For the former number, we have implemented a standard way of recording a merge: we create a new tag for the branch. In Subversion, branches and tags are exactly the same; the difference is only in convention. So the convention for Bricolage is to store tags in the tags subdirectory of the project directory, and branches in the branches subdirectory. The other half of the convention is that no one should ever commit changes to a tag after it has been created. Tags are named for the branch name, the word merge, and the revision number up to which the merge was made. For example, if a merge was committed to the trunk from the rev_1_8 branch in Subversion revision # 5694, the branch would be tagged rev_1_8_merge_5694. So to get the revision number of the last merge, look in the Bricolage ViewCVS interface for the last merge tag created for the branch. Or use the Subversion CWlist command to find it:

  % svn list http://svn.bricolage.cc/bricolage/tags | grep rev_1_8
  rev_1_8_merge_5694/
  rev_1_8_merge_5825/
  rev_1_8_merge_5902/
  rev_1_8_merge_5995/

If the branch has never been merged into the trunk you can get the revision number at the time the branch was created using CWsvn log command:

  % svn log --verbose --stop-on-copy \
    http://svn.bricolage.cc/bricolage/branches/rev_1_8
  ------------------------------------------------------------------------
  r5525 | theory | 2004-05-10 19:16:48 -0700 (Mon, 10 May 2004) | 1 line
  Changed paths:
     A /bricolage/branches/rev_1_8 (from /bricolage/trunk:5524)

So now we know that we want to update the trunk with all of the changes between revision 5995 and CWHEAD. If no merge tag exists, then use CWsvn log to find out revision number at which the branch was created and use that number.

3

Now simply switch to the trunk checkout and perform the merge. Using the above examples, the merge would look like this:

  % cd bricolage-trunk
  % svn merge -r 5995:HEAD http://svn.bricolage.cc/bricolage/branches/rev_1_8
  U  lib/Bric/App/ApacheConfig.pm
  C  lib/Bric/Hacker.pod

If you're nervous about doing the merge (don't be, it's just your local copy you're updating at this point), you can use the CW--dry-run option to see what files would change without changing them. Or run CWsvn diff with the same arguments to see what the differences are:

  % svn merge -r 5995:HEAD --dry-run \
    http://svn.bricolage.cc/bricolage/branches/rev_1_8 > test.diff

4

Resolve any conflicts (hopefully none, but they do happen occasionally). Notice above that the result of the merge is a conflict in lib/Bric/Hacker.pod. To resolve the conflicts, follow the directions in the Subversion book, published online here: <http://svnbook.red-bean.com/en/1.1/ch03s05.html#svn-ch-3-sect-5.4>.

5

Rebuild the database and run all of the tests:

  % make devtest TEST_VERBOSE=1

If there are any test failures, you'll need to fix those, too.

6

Commit the changes to the trunk. Be sure to specify what revisions were merged in, and the name of the new tag you'll create for the branch. The final revision number (represented by HEAD) happened to have been printed out when we ran CWsvn update in step one, so it's easy to find and use:

  % svn commit -m 'Merged rev_1_8 changes 5995-6123. Will tag rev_1_8_merge_6123.'

7

Tag the branch with the last revision, so that future merges can find it and know to merge only from that revision on.

  % svn cp http://svn.bricolage.cc/bricolage/branches/rev_1_8 \
    http://svn.bricolage.cc/bricolage/tags/rev_1_8_merge_6123 \
    -m 'Merge tag for rev_1_8 to revision 6123.'

By following this methodology we should be able to minimize the number of conflicts we get between merges. See book Version Control with Subversion, chapter 4 at <http://svnbook.red-bean.com/svnbook/ch04.html> for a more detailed explanation of the whys and wheres of this approach to merging Subversion branches.

If you are a Bricolage release manager and you're getting ready to release a new version, here are the steps you'll need to take to create a distribution tarball. First, proof-read the list of changes in Bric::Changes. Also, be sure to add today's date to the header for the new release:

  =head1 VERSION 1.x.x (2003-11-29)

If this is a major release, you'll need to create a new branch, so that it can be maintained for bug fixes separately, and then tag the release in that branch. In Subversion, branches and tags are exactly the same; the difference is only in convention. So the convention for Bricolage is to store tags in the tags subdirectory of the project directory, and branches in the branches subdirectory. The other half of the convention is that no one should ever commit changes to a tag after it has been created.

So to create a branch, make the copy from the trunk:

  % svn cp http://svn.bricolage.cc/bricolage/trunk \
    http://svn.bricolage.cc/bricolage/branches/rev_1_8 \
    -m 'New branch for maintenance of Bricolage 1.8.x.'

Now export the branch and create the distribution tarball.

  % svn export http://svn.bricolage.cc/bricolage/branches/rev_1_8
  % cd rev_1_8
  % make dist

This will create a distribution tarball in the rev_1_8 directory. Copy this tarball somewhere, and do a full test with it, to make sure that Bricolage does indeed build and properly install itself.

  % cp bricolage-1.x.x.tar.gz /tmp/src
  % cd /tmp/src
  % tar zxvf bricolage-1.x.x.tar.gz
  % cd bricolage-1.x.x
  % perl Makefile.PL
  % make
  # Shut down PostgreSQL.
  % make test
  # Start PostgreSQL.
  % sudo make install
  % make devtest
  % sudo bric_apachectl start
  # Log in to the UI and test it a bit.
  % sudo make uninstall

Be sure to run CWmake devtest after everything is installed to ensure that it is all functioning correctly. Users won't be running these tests, as they muck up the database. But that's okay for our validation of the release tarball. Use CWmake uninstall to clean up the mess. If you have to go back and make any changes, be sure to commit them to the Subversion repository and then do the whole thing over again. Once everything appears to be working properly, release!

Now that the new version of Bricolage is out free in the world, it's time to tag the release. Again, it's a simple copy:

  % svn cp http://svn.bricolage.cc/bricolage/branches/rev_1_8 \
    http://svn.bricolage.cc/bricolage/tags/rel_1_8_0 \
    -m 'Tag for the 1.8.0 release of Bricolage.'

The tag is mainly kept for the purposes of record-keeping. And since copies are cheap in Subversion, it's worthwhile.

Once the new version of Bricolage has been released, you'll need to regenerate the documentation on the Web site. There is documentation for each major version of Bricolage. The documentation for each version is generated from the latest minor release. So if you've just released, say, version 1.8.4, then the documentation should be generated for the 1.8 Documentation section. If you've just released 1.10.0, then you'll need to create a new 1.10 section and add a link to it on the documentation page of the Web site.

The documentation is generated with the pod_browser script, which is in the distutils/PodBrowser in Subversion. Read its instructions carefully, and then generate the documentation on the Web server.

Mail Lists

announce@lists.bricolage.cc

users@lists.bricolage.cc

devel@lists.bricolage.cc

mason-announce@lists.sourceforge.net

mason-users@lists.sourceforge.net

pgsql-announce@postgresql.org

pgsql-general@postgresql.org

announce@perl.apache.org

modperl@perl.apache.org

html-template-users@lists.sourceforge.net

templates-announce@template-toolkit.org

templates@template-toolkit.org

general@oscom.org

Web Sites

<http://www.bricolage.cc/>

<http://www.cmsmatrix.org/>

<http://freshmeat.net/projects/bricolage/>

<http://use.perl.org/>

<http://bugs.bricolage.cc/>

Sam Tregar <stregar@about-inc.com>

David Wheeler <david@wheeler.net>

Bric::Admin, Bric::ToDo