TclCommandWriting - Writing C language extensions to Tcl.

This document is intended to help the programmer who wishes to extend Tcl with C language routines. It should also be useful to someone wishing to add Tcl to an existing editor, communications program, window manager, etc. C programming information can also be found in the *.3 manual pages in the doc directory of the Berkeley distribution, and in the *.3 manpages in the man directory of Extended Tcl.

All C-based Tcl commands are called with four arguments: a client data pointer, an interpreter pointer, an argument count and a pointer to an array of pointers to character strings containing the Tcl arguments to the command.

A simple Tcl extension in C is now presented, and described below:

    #include "tcl.h"

int App_EchoCmd(clientData, interp, argc, argv) void *clientData; Tcl_Interp *interp; int argc; char **argv; { int i;

for (i = 1; i < argc; i++) { printf("%s",argv[i]); if (i < argc - 1) printf(" "); } printf("\n"); return TCL_OK; }

The client data pointer will be described later.

The interpreter pointer is the ``key'' to an interpreter. It is returned by Tcl_CreateInterp and is used extensively within Tcl, and will be by your C extensions. The data structure pointed to by the interpreter pointer, and all of the subordinate structures that branch off of it, make up a Tcl interpreter, which includes all of the currently defined procedures, commands, variables, arrays and the execution state of that interpreter. (For more information on creating and deleting interpreters, please examine the CrtInterp(3) manpage in the Berkeley Tcl distribution. For information on creating interpreters that include the commands provided by Extended Tcl, check out the TclX_Init(3) manpage of Extended Tcl. For a manual page describing the user-visible fields of a Tcl interpreter, please look at Interp(3) in Berkeley Tcl.)

The argument count and pointer to an array of pointers to textual arguments is handled by your C code in the same manner that you would use in writing a C main function -- the argument count and array of pointers works the same as in a C main call; pointers to the arguments to the function are contained in the argv array. Similar to a C main, the first argument (argv[0]) is the name the routine was called as (in a main, the name the program was invoked as).

In the above example, all of the arguments are output with a space between each one by looping through argv from one to the argument count, argc, and a newline is output to terminate the line -- an ``echo'' command.

All arguments from a Tcl call to a Tcl C extension are passed as strings. If your C routine expects certain numeric arguments, your routine must first convert them using the Tcl_GetInt or Tcl_GetDouble function, Extended Tcl's Tcl_GetLong or Tcl_GetUnsigned, or some other method of your own devising. Likewise for converting boolean values, Tcl_GetBoolean should be used. These routines automatically leave an appropriate error message in the Tcl interpreter's result buffer and return TCL_ERROR if a conversion error occurs. (For more information on these routines, please look at the GetInt(3) manpage in the Berkeley Tcl distribution.)

Likewise, if you program produces a numeric result, it should return a string equivalent to that numeric value. A common way of doing this is something like...

	sprintf(interp->result, "%ld", result);

Writing results directly into the interpreter's result buffer is only good for relatively short results. Tcl has a function, Tcl_SetResult, which provides the ability for your C extensions to return very large strings to Tcl, with the ability to tell the interpreter whether it ``owns'' the string (meaning that Tcl should delete the string when it's done with it), that the string is likely to be changed or overwritten soon (meaning that Tcl should make a copy of the string right away), or that the string won't change (so Tcl can use the string as is and not worry about it). Understanding how results are passed back to Tcl is essential to the C extension writer. Please study the SetResult(3) manual page in the Tcl distribution.

Sophisticated commands should verify their arguments whenever possible, both by examining the argument count, by verifying that numeric fields are really numeric, that values are in range (when their ranges are known), and so forth.

Tcl is designed to be as bullet-proof as possible, in the sense that no Tcl program should be able to cause Tcl to dump core. Please carry this notion forward with your C extensions by validating arguments as above.

In the command below, two or more arguments are compared and the one with the maximum value is returned, if all goes well. It is an error if there are fewer than three arguments (the pointer to the ``max'' command text itself, argv[0], and pointers to at least two arguments to compare the values of).

This routine also shows the use of the programmer labor-saving Tcl_AppendResult routine. See the Tcl manual page, SetResult(3), for details. Also examine the calls Tcl_AddErrorInfo, Tcl_SetErrorCode and Tcl_PosixError documented in the Tcl manual page AddErrInfo(3).

    int
    Tcl_MaxCmd (clientData, interp, argc, argv)
        char       *clientData;
        Tcl_Interp *interp;
        int         argc;
        char      **argv;
    {
        int maxVal = MININT;
        int maxIdx = 1;
        int value, idx;

if (argc < 3) { Tcl_AppendResult (interp, "bad # arg: ", argv[0], " num1 num2 [..numN]", (char *)NULL); return TCL_ERROR; }

for (idx = 1; idx < argc; idx++) { if (Tcl_GetInt (argv[idx], 10, &Value) != TCL_OK) return TCL_ERROR;

if (value > maxVal) { maxVal = value; maxIdx = idx; } } Tcl_SetResult (interp, argv [maxIdx], TCL_VOLATILE); return TCL_OK; }

When Tcl-callable functions complete, they should normally return TCL_OK or TCL_ERROR. TCL_OK is returned when the command succeeded and TCL_ERROR is returned when the command has failed in some abnormal way. TCL_ERROR should be returned for all syntax errors, non-numeric values (when numeric ones were expected), and so forth. Less clear in some cases is whether Tcl errors should be returned or whether a function should just return a status value. For example, end-of-file during a gets returns a status, but open returns an error if the open fails. Errors can be caught from Tcl programs using the catch command. (See Tcl's catch(n) and error(n) manual pages.)

Less common return values are TCL_RETURN, TCL_BREAK and TCL_CONTINUE. These are used if you are adding new control and/or looping structures to Tcl. To see these values in action, examine the source code to Tcl's while, for and if, and Extended Tcl's loop commands.

Note the call to Tcl_SetResult in the above command to set the return value to Tcl. TCL_VOLATILE is used because the memory containing the result will be freed upon the function's return.

In the command below, one list is passed as an argument, and a list containing all of the elements of the list in reverse order is returned. It is an error if anything other than two arguments are passed (the pointer to the ``lreverse'' command text itself, argv[0], and a pointer to the list to reverse.

Once lreverse has determined that it has received the correct number of arguments, Tcl_SplitList is called to break the list into an argc and argv array of pointers.

lreverse then operates on the array of pointers, swapping them from lowest to highest, second-lowest to second-highest, and so forth.

Finally Tcl_Merge is calleds to create a single new string containing the reversed list and it is set as the result via Tcl_SetResult. Note that TCL_DYNAMIC is used to tell Tcl_SetResult that it now owns the string and it is up to Tcl to free the string when it is done with it.

Note that it is safe to play around with the argv list like this, and that a single call to ckfree can be made to free all the data returned by Tcl_SplitList in this manner.

int
Tcl_LreverseCmd(notUsed, interp, argc, argv)
    ClientData notUsed;			/* Not used. */
    Tcl_Interp *interp;			/* Current interpreter. */
    int argc;				/* Number of arguments. */
    char **argv;			/* Argument strings. */
{
    int listArgc, lowListIndex, hiListIndex;
    char **listArgv;
    char *temp, *resultList;

if (argc != 2) { Tcl_AppendResult(interp, "wrong # args: should be " list return TCL_ERROR; }

if (Tcl_SplitList(interp, argv[1], &listArgc, &listArgv) != TCL_OK) { return TCL_ERROR; } for (lowListIndex = 0, hiListIndex = listArgc; --hiListIndex > lowListIndex; lowListIndex++) { temp = listArgv[lowListIndex]; listArgv[lowListIndex] = listArgv[hiListIndex]; listArgv[hiListIndex] = temp; } resultList = Tcl_Merge (listArgc, listArgv); ckfree (listArgv); Tcl_SetResult (interp, resultList, TCL_DYNAMIC); return TCL_OK; }

To install your command into Tcl you must call Tcl_CreateCommand, passing it the pointer to the interpreter you want to install the command into, the name of the command, a pointer to the C function that implements the command, a client data pointer, and a pointer to an optional callback routine.

The client data pointer and the callback routine will be described later.

For example, for the max function above (which, incidentally, comes from TclX's tclXmath.c in the TclX7.4/src directory):

    Tcl_CreateCommand (interp, "max", Tcl_MaxCmd, (ClientData)NULL, 
                      (void (*)())NULL);

In the above example, the max function is added to the specified interpreter. The client data pointer and callback function pointer are NULL. (For complete information on Tcl_CreateCommand and its companion routine, Tcl_CommandInfo, please examine the CrtCommand(3) command page in the Berkeley Tcl distribution.)

Dynamic strings are an important abstraction that first became available with Tcl 7.0. Dynamic strings, or DStrings, provide a way to build up arbitrarily long strings through a repeated process of appending information to them. DStrings reduce the amount of allocating and copying required to add information to a string. Further, they simplify the process of doing so. For complete information on dynamic strings, please examine the DString(3) manual page in the Berkeley Tcl distribution.

The client data pointer provides a means for Tcl commands to have data associated with them that is not global to the C program nor included in the Tcl core. Client data is essential in a multi-interpreter environment (where a single program has created and is making use of multiple Tcl interpreters) for the C routines to maintain any permanent data they need on a per-interpreter basis. Otherwise there would be reentrancy problems. Tcl solves this through the client data mechanism. When you are about to call Tcl_CreateCommand to add a new command to an interpreter, if that command needs to keep some read/write data across invocations, you should allocate the space, preferably using ckalloc, then pass the address of that space as the ClientData pointer to Tcl_CreateCommand.

When your command is called from Tcl, the ClientData pointer you gave to Tcl_CreateCommand when you added the command to that interpreter is passed to your C routine through the ClientData pointer calling argument.

Commands that need to share this data with one another can do so by using the same ClientData pointer when the commands are added.

It is important to note that the Tcl extensions in the tclX7.4/src directory have had all of their data set up in this way. Since release 6.2, Extended Tcl has supported multiple interpreters within one invocation of Tcl.

Sometimes you need to have a data element that isn't readily representable as a string within Tcl, for example a pointer to a complex C data structure. It is not a good idea to try to pass pointers around within Tcl as strings by converting them to and from hex or integer representations, for example. It is too easy to mess one up, and the likely outcome of doing that is a core dump.

Instead we have developed and made use of the concept of handles. Handles are identifiers a C extension can pass to, and accept from, Tcl to make the transition between what your C code knows something as and what name Tcl knows it by to be as safe and painless as possible. For example, the stdio package included in Tcl uses file handles. When you open a file from Tcl, a handle is returned of the form filen where n is a file number. When you pass the file handle back to puts, gets, seek, flush and so forth, they validate the file handle by checking the the file text is present, then converting the file number to an integer that they use to look into a data structure of pointers to Tcl open file structures, which contain a Unix file descriptor, flags indicating whether or not the file is currently open, whether the file is a file or a pipe and so forth.

Handles have proven so useful that, as of release 6.1a, general support has been added for them. If you need a similar capability, it would be best to use the handle routines, documented in Handles(3) in Extended Tcl. We recommend that you use a unique-to-your-package textual handle coupled with a specific identifier and let the handle management routines validate it when it's passed back. It is much easier to track down a bug with an implicated handle named something like file4 or bitmap6 than just 6.

Occasionally you may write code that scribbles past the end of an allocated piece of memory. The memory debugging routines included in Tcl can help find these problems. See Memory(TCL) for details.

To add your extensions to Extended Tcl, you must compile them and cause them to be linked with TclX. For the routines to be linked into the tcl and wishx executables, they must be referenced (directly or indirectly) from TclX. For these extensions to be visible as Tcl commands, they must be installed into Tcl with Tcl_CreateCommand.

Application-specific startup is accomplished by creating or editing the Tcl_AppInit function. In Tcl_AppInit you should add a call to an application-specific init function which you create. This function should take the address of the interpreter it should install its commands into, and it should install those commands with Tcl_CreateCommand and do any other application-specific startup that is necessary.

The naming convention for application startup routines is App_Init, where App is the name of your application. For example, to add an application named cute one would create a Cute_Init routine that expected a Tcl_Interp pointer as an argument, and add the following code to Tcl_AppInit:

    if (Cute_Init (interp) == TCL_ERROR) {
	return TCL_ERROR;
    }

As you can guess from the above example, if your init routine is unable to initialize, it should use Tcl_AppendResult to provide some kind of useful error message back to TclX, then return TCL_ERROR to indicate that an error occurred. If the routine executed successfully, it should return TCL_OK.

When you examine Tcl_AppInit, note that there is one call already there to install an application -- the call to TclX_Init installs Extended Tcl into the Tcl core.

TclX's infox command can return several pieces of information relevant to Extended Tcl, including the application's name, descriptive name, patch level and version. Your application's startup can set these variables to application-specific values. If it doesn't, they are given default values for Extended Tcl.

To set these values, first be sure that you include either tclExtend.h or tclExtdInt.h from the source file that defines your init routine. This will create external declarations for the variables. Then, set the variables in your init route, for example:

    tclAppName = "cute";
    tclAppLongName = "Call Unix/Tcl Environment";
    tclAppVersion = "2.1";

Note that the default values are set by TclX_Init, so if you wish to override them, you must call your init routine in Tcl_AppInit after its call to TclX_Init.

When Extended Tcl exits, Tcl_DeleteInterp may be called to free memory used by Tcl -- normally, this is only called if TCL_MEM_DEBUG was defined, since Unix will return all of the allocated memory back to the system, anyway. If TCL_MEM_DEBUG was defined, it is called so that any memory that was allocated without ever being freed can be detected. This greatly reduces the amount of work to detect and track down memory leaks, a situation where some piece of your code allocates memory repeatedly without ever freeing it, or without always freeing it.

It is often necessary for an application to perform special cleanup functions upon the deletion of an interpreter as well. To facilitate this activity, Tcl provides the ability to perform a function callback when an interpreter is deleted. To arrange for a C function to be called when the interpreter is deleted, call Tcl_CallWhenDeleted from your application initialization routine. For details on how to use this function, read the CallDel(3) manual page that ships with Berkeley Tcl.

Suppose you are in the middle of coding a C extension and you realize that you need some operation performed, one that would be simple from Tcl but possibly excruciating to do directly in C. Tcl provides the Tcl_Eval, Tcl_VarEval, Tcl_EvalFile and Tcl_GlobalEval functions for the purpose of executing Tcl code from within a C extension. The results of the call will be in interp->result. For more information please consult the Eval(3) manual page within the Tcl distribution.

Tcl variables and arrays can be read from a C extension through the Tcl_GetVar and Tcl_GetVar2 functions, and set from C extensions through the Tcl_SetVar and Tcl_SetVar2 functions. They can also be unset via the Tcl_UnsetVar and Tcl_UnsetVar2 functions. For complete information on these functions, please refer to the SetVar(3) manual page in the doc directory of the Berkeley Tcl distribution.

Tcl_LinkVar and Tcl_UnlinkVar can be used to automatically keep Tcl variables synchronized with corresponding C variables. Once a Tcl variable has been linked to a C variable with Tcl_LinkVar, anytime the Tcl variable is read the value of the C variable will be returned, and when the Tcl variable is written, the C variable will be updated with the new value.

Tcl_LinkVar uses variable traces to keep the Tcl variable named by varName in sync with the C variable at the address given by addr.

Whenever the Tcl variable is read the value of the C variable will be returned, and whenever the Tcl variable is written the C variable will be updated to have the same value.

Int, double, boolean and char * variables are supported. For more information, please examine the LinkVar(3) manual page in the Berkeley Tcl distribution.

As of Tcl version 7.0, math functions such as sin, cos, etc, are directly supported within Tcl expressions. These obsolete the Extended Tcl commands that provided explicit calls for these functions for many releases.

New math functions can be added to Tcl, or existing math functions can be replaced, by calling Tcl_CreateMathFunc.

For more information on adding math functions, please examine the CrtMathFnc(3) manual page in the Berkeley Tcl distribution.

The Tcl_TildeSubst function is available to C extension writers to perform tilde substitutions on filenames. If the name starts with a ``~'' character, the function returns a new string where the name is replaced with the home directory of the given user. For more information please consult the TildeSubst(3) manual page in the Berkeley Tcl distribution.

Tcl has a preset recursion limit that limits the maximum allowable nesting depth of calls within an interpreter. This is useful for detecting infinite recursions before other limits such as the process memory limit or, worse, available swap space on the system, are exceeded.

The default limit is just a guess, however, and applications that make heavy use of recursion may need to call Tcl_SetRecursionLimit to raise this limit. For more information, please consult the SetRecLmt(3) manual page in the Berkeley Tcl distribution.

If an event such as a signal occurs while a Tcl script is being executed, it isn't safe to do much in the signal handling routine -- the Tcl environment cannot be safely manipulated at this point because it could be in the middle of some operation, such as updating pointers, leaving the interpreter in an unreliable state.

The only safe approach is to set a flag indicating that the event occurred, then handle the event later when the interpreter has returned to a safe state, such as after the current Tcl command completes.

The Tcl_AsyncCreate, Tcl_AsyncMark, Tcl_AsyncInvoke, and Tcl_AsyncDelete functions provide a safe mechanism for dealing with signals and other asynchronous events. For more information on how to use this capability, please refer to the Async(3) manual page in the Berkeley Tcl distribution.

The Tcl_Backslash function is called to parse Tcl backslash sequences. These backslash sequences are the usual sort that you see in the C programming language, such as \n for newline, \r for return, and so forth. Tcl_Backslash parses a single backslash sequence and returns a single character corresponding to the backslash sequence.

For more info on this call, look at the Backslash(3) manual page in the Berkeley Tcl distribution. For information on the valid backslash sequences, consult the summary of Tcl language syntax, Tcl(n) in the same distribution.

Hash tables provide Tcl with a high-performance facility for looking up and managing key-value pairs located and maintained in memory. Tcl uses hash tables internally to locate procedure definitions, Tcl variables, array elements, file handles and so forth. Tcl makes the hash table functions accessible to C extension writers as well.

Hash tables grow automatically to maintain efficiency, rather than exposing the table size to the programmer at allocation time, which would needlessly add complexity to Tcl and would be prone to inefficiency due to the need to guess the number of items that will go into the table, and the seemingly inevitable growth in amount of data processed per run over the life of the program.

For more information on hash tables, please consult the Hash(3) manual page in the Berkeley Tcl distribution.

The C extension writer can arrange to have a C routine called whenever a Tcl variable is read, written, or unset. Variable traces are the mechanism by which Tk toolkit widgets such as radio and checkbuttons, messages and so forth update without Tcl programmer intervention when their data variables are changed. They are also used by the routine that links Tcl and C variables, Tcl_LinkVar, described above.

Tcl_TraceVar is called to establish a variable trace. Entire arrays and individual array elements can be traced as well. If the programmer already has an array name in one string and a variable name in another, Tcl_TraceVar2 can be called. Calls are also available to request information about traces and to delete them.

For more information on variable traces, consult the TraceVar(3) manual page in the Berkeley Tcl distribution.

Tcl has the ability to call C routines for every command it executes, up to a specified depth of nesting levels. The command Tcl_CreateTrace creates an execution trace; Tcl_DeleteTrace deletes it.

Command tracing is used in Extended Tcl to implement the cmdtrace Tcl command, a useful command for debugging Tcl applications.

For complete information on execution tracing, please look at the CrtTrace(3) manual pages in the Berkeley Tcl distribution.

Tcl_ExprLong, Tcl_ExprDouble, Tcl_ExprBool, and Tcl_ExprString can be called to evaluate Tcl expressions from within a C routine. Depending on the routine called, the result is either a C long, a double, a boolean (int with a value of 0 or 1), or a char * (pointed to by interp->result).

For complete information on evaluating Tcl expressions from C, you are invited to examine the ExprLong(3) manpage in the Berkeley Tcl distribution.

The Tcl_StringMatch function can be called to see if a string matches a specified pattern. Tcl_StringMatch is called by the Tcl string match command, so the format for patterns is identical. The pattern format is similar to the one used by the C-shell; string(n) describes this format.

More information about Tcl_StringMatch is available in the StrMatch(3) manpage in the Berkeley Tcl distribution.

Tcl_RegExpMatch can be called to determine whether a string matches a regular expression. Tcl_RegExpMatch is used internally by the regexp Tcl command.

For more information on this function, please consult the RegExp(3) manpage in the Berkeley Tcl distribution.

The C extension writer often needs to create, manipulate and decompose Tcl lists. Tcl_SplitList parses a list into an argv and argc like to the way command-line arguments are passed to a Tcl extension. Tcl_Merge, likewise, creates a single string (pointer to a char *) from an argv and argc.

Two routines, Tcl_ScanElement and Tcl_ConvertElement, do most of the work of Tcl_Merge, and may also be of use to the C programmer.

For more information on these commands, please consult the SplitList(3) manual page in the Berkeley Tcl distribution.

Tcl_Concat concatenates zero or more strings into a single string. The strings are space-separated. Tcl_Concat works like Tcl_Merge, except that Tcl_Concat does not attempt to make the resulting string into a valid Tcl list.

Tcl_Concat is documented in the Concat(3) manpage in the Berkeley Tcl distribution.

C routines that collect data to form a command to be passed to Tcl_Eval often need a way to tell whether they have a complete command already or whether they need more data. (Programs that read typed-in Tcl input such as Tcl shells need this capability.) Tcl_CommandComplete can be used to tell whether or not you have a complete command.

For more information examine CmdCmplt(3) in the Berkeley Tcl distribution.

Tcl has a history mechanism that is accessed from Tcl through the history command. To propagate commands into the command history, your extension should call Tcl_RecordAndEval. This command works just like Tcl_Eval, except that it records the command as well as executing it.

Tcl_RecordAndEval should only be called with user-entered top-level commands, since the history mechanism exists to allow the user to easily access, edit and reissue previously issued commands.

For complete information on this function, please examine the RecordEval.3 manual page in the Berkeley Tcl distribution.

Tcl_PrintDouble converts a C double into an ASCII string. It ensures that the string output will continue to be interpreted as a floating point number, rather than an integer, by always putting a ``.'' or ``e'' into the string representing the number. The precision of the output string is controlled by the Tcl tcl_precision variable.

For complete information on Tcl_PrintDouble, examine PrintDbl(3) in the Berkeley Tcl distribution.

Tcl_CreatePipeline is a useful procedure for spawning child processes. The child (or pipeline of children) can have its standard input, output and error redirected from files, variables or pipes. To understand the meaning of the redirection symbols understood by this function, look at the exec(n) Tcl command. For complete information on Tcl_CreatePipeline, please examine CrtPipelin(3).

Files opened from your C code can be made visible to Tcl code via the Tcl_EnterFile function. Likewise, Tcl filehandles passed to your C extension can be translated to a Posix FILE * structure using the Tcl_GetOpenFile function.

For complete explanations of these commands, please look at EnterFile(3) in the Berkeley Tcl distribution.

When a Posix system does a fork to create a new process, the process ID of the child is returned to the caller. After the child process exits, its process table entry (and some other data associated with the process) cannot be reclaimed by the operating system until a call to waitpid, or one of a couple of other, similar system calls, has been made by the parent process.

The C extension writer who has created a subprocess, by whatever mechanism, can turn over responsibility for detecting the processes' termination and calling waitpid to obtain its exit status by calling Tcl_DetachPids.

Tcl_ReapDetachedProcs is the C routine that will detect the termination of any processes turned over to Tcl, permitting the processes to be fully reclaimed by the operating system.

For complete information on these routines, please look at DetachPids(3) in the Berkeley Tcl distribution.

In addition to the documentation referenced above, you can learn a lot by studying the source code of the commands added by Tcl, Tk and Extended Tcl. The comp.lang.tcl Usenet newsgroup is read by tens of thousands of Tcl people, and is a good place to ask questions. Finally, if you have interactive Internet access, you can ftp to ftp.aud.alcatel.com, the site for contributed Tcl sources. This site contains quite a few extensions, applications, and so forth, including several object-oriented extension packages.

Extended Tcl was created by Karl Lehenbauer (karl@neosoft.com) and Mark Diekhans (markd@grizzly.com).