PIPE Module

The CommuniGate Pro PIPE module creates the Submitted folder inside the CommuniGate Pro base folder.

The PIPE module scans this folder periodically, and processes the files with the .sub file name extension. When such a file is found, the module copies it into a message queue file and submits that file to the Server kernel for processing.

The .sub text files should contain messages in the RFC822 format. The module uses the data in the RFC822 header fields to compose the message envelope.

• The RFC821 "channel format" should NOT be used: submitted files should use the system native "End of Line" character(s), the dot (.) symbol in the first position should not be doubled, and the file should not end with the dot (.) symbol.
• Addresses specified in the To:, Cc:, Bcc: header fields are used to create the message envelope (recipient addresses).
• The Bcc: header fields are removed from the submitted message.
• If at least one Envelope-To: header field is detected, message envelope (recipient) addresses are formed using the addresses specified in these header fields, and addresses in all remaining To, Cc, and Bcc headers are not placed into the message envelope. The Envelope-To: header fields are removed from the submitted message.
• If no Envelope-To: header field exists, and the Envelope-Ignore field or fields exist, the addresses specified in To/Cc/Bcc header fields and also listed in the Envelope-Ignore fields are NOT included into the message envelope.
• If the Sender: and/or From: header fields contain addresses without the domain part, the Server domain name is added to those addresses.
• If the Return-Path: header field exists, the address specified in that header is used to compose the Return-Path envelope address, and this header field is removed from the submitted message.
• If the Return-Path: header field does not exist, the address specified in the From: or Sender: header field is used to compose the envelope Return-Path address.
• If the Envelope-ID: header field exists, its content is used as the message Envelope ID. This header field is removed from the submitted message.
• If the Envelope-Notify: header field exists, it should contain one or several keywords SUCCESS,FAILURE,DELAY,RELAY separated with comma symbols, or it should contain one keyword NEVER.
  This header field specifies the DSN parameters applied to all envelope addresses composed after this field is met. This header field is removed from the submitted message.
  Several Envelope-Notify: header fields can be used to specify different DSN parameters for different addresses.

If processing of a .sub file fails (for example, if the file does not have any recipient address), the module places a record into the System Log, and changes the file extension to .bad.

If a .sub file is submitted successfully, the file is deleted from the Submitted folder.

Because of the way the PIPE module processes the Submitted folder, it is recommended to compose messages in a different file directory and then move the composed .sub files to the Submitted folder, or to compose messages in the Submitted folder, in files with the .tmp file name extension, and then change the file name extension to .sub.

Messages submitted via the PIPE module are marked as "received from a trusted source", so they can be relayed without restrictions.

The Submitted folder is used for Legacy Mail Emulation.

The PIPE module accepts all messages directed to the pipe domain.

The local part of the message address specifies the external application to launch. The part can contain parameters, and can be enclosed into the quotation marks.

Note: the name of the external application must be registered in the Child Processes Manifesto.

Example:

A message directed to the "execjoe -l store"@pipe address will be sent to the application execjoe started with the -l store parameters.

You usually use the PIPE delivery via the Router: this Router record will direct messages sent to the joe@somedomain address to the execjoe application.

this Router record will direct messages sent to the joe@somedomain address to the execall application started with the -u joe parameters.

To limit the set of applications that can be started via the PIPE module, the external application directory is specified as one of the PIPE module settings. The application names specified in message addresses can not include the slash (/) or the backslash (\) symbols, and they cannot start with the dot (.) symbol, and it specified the name of the application (program) file in the external application directory.

The message text (including the message headers and the message body) is passed to the external application as its standard input.
Note: the application must read the entire stdin data stream, otherwise message processing fails.

When the external application completes, the PIPE module reads and discards the application standard output. Make sure that your application does not write anything to its standard output, so it is not blocked when the communication channel (pipe) buffer between the application and the Server is full.

When the external application completes, the PIPE module reads its standard error channel. If it is not empty, the message delivery fails, and the text written to standard error is sent as an error report to the message sender.

Serialized Delivery

In order to allow several PIPE processors to deliver messages simultaneously, the PIPE module creates a separate queue for each message it has to deliver. If you want to serialize processing, you can use the following form of the PIPE address:
"queue[name] application parameters"@pipe
All messages directed to these addresses will be placed into the name queue, and a single PIPE processor will send the enqueued messages to the application(s) specified in those addresses. You can use any alphanumeric string as a queue name, and you can specify as many queues as you need.

The following Router records can be used to maintain two serialized PIPE queues (the PROC1 and ARCH queues):

All messages sent to the <incoming@maindomain.com> and <control@maindomain.com> will be processed one-by-one using one PIPE processor.

For PIPE addresses that do not have the queue[name] prefix, the PIPE module creates separate queues with numeric names.

Command Tags

The message text (the header and the body) is sent to the task as that task standard input (stdin).

An application name can be prefixed with the [FILE] tag:
[FILE] application parameters
When this prefix is used, the application standard input will be empty (closed), or it will contain only the message header fields added by Rules. The string -f Queue/fileid.msg (the -f flag and the Message file name, relative to the base directory) will be appended to the end of the application parameters:
-f Queue/12002345.msg

Note: The beginning of a Queue file contains the service information (envelope, options, etc.). The application should ignore this information skipping the file data till the first empty line. The message itself starts after that first empty line in the Queue file.

Note: Header fields added to the message by Server-wide and Cluster-wide Rules are not stored in the message Queue file, they are sent via the task standard input.

Note: this prefix should not be used on MS Windows platform, as the Server keeps the message file open, making it impossible for an external Task to read the file.

An application name can be prefixed with the [RETPATH] tag:
[RETPATH] application parameters
When this prefix is specified, the string "-p" followed by the message return-path address is added to the end of the application parameters:
-p address@domain.com

An application name can be prefixed with the [RCPT] tag:
[RCPT]application parameters
When this prefix is specified the string "-r" followed by the original recipient address is added to the end of the application parameters:
-r address1@domain1.com

An application name can be prefixed with the [STDERR] tag (see below).

An application name can have several prefix strings, and they can be specified in any order. If several of [FILE], [RETPATH], and [RCPT] prefix strings are specified, the -f flag and its parameter are added first, followed with the -p flag and its parameter, followed with the -r flag and its parameter.

If the [STDERR] prefix is specified and the external application completes sending some data to its standard error channel, the standard error data is used to compose the error report text.

Use the WebAdmin Interface to configure the PIPE module. Open the Mail pages in the Settings realm, then open the PIPE page.

Processing
Log Level: Crashes OnlyFailuresMajor & FailuresProblemsLow LevelAll Info	Processors: 0135102550other
Application Directory:
Processing Time Limit:	1 second5 seconds10 seconds15 seconds30 secondsminute2 minutes3 minutes5 minutes10 minutes15 minutes30 minuteshour2 hours3 hours6 hours12 hours

Log

Use the Log setting to specify what kind of information the PIPE module should put in the Server Log. Usually you should use the Major (message transfer reports) or Problems (message transfer and non-fatal errors) levels. But when you experience problems with the PIPE module, you may want to set the Log Level setting to Low-Level or All Info: in this case protocol-level or link-level details will be recorded in the System Log as well. When the problem is solved, set the Log Level setting to its regular value, otherwise your System Log files will grow in size very quickly.

The PIPE module records in the System Log are marked with the PIPE tag.

Processors

This option specifies the number of threads used to deliver messages. If some of your external applications are slow, you may want to use several PIPE delivery threads, so several messages can be processed at the same time.

Application Directory

This option specifies the directory with the applications the PIPE module can launch. If an empty string is specified for this option, all messages directed to the PIPE module are rejected.

Processing Time Limit

This option limits the time an external application uses to process a message. If an external application does not complete within the specified period of time, the application process is interrupted and the message is rejected.

Submitting
Check Directory every:	1 sec2 sec3 sec5 sec10 sec15 sec20 sec30 sec40 sec60 sec90 sec2 min3 min5 min10 min15 min20 min30 min40 min60 min2 hour(s)3 hour(s)4 hour(s)6 hour(s)10 hour(s)12 hour(s)other

Check Submitted Directory

This option specifies how often the PIPE module should scan the Submitted directory and deliver the .sub files stored there.

In emergency situations you may need to process an additional Queue directory without stopping your server. These situations include a server hardware failure when the rescued Queue files should be processed with some other, already running server.

To process an additional Queue directory, move it to the base directory of a running server as the ForeignQueue directory. If you prefer to use symbolic links, make sure that the Queue and ForeignQueue directories are created on the same file system.

Every 3 minutes the PIPE module checks if the ForeignQueue directory exists in the server base directory. If the ForeignQueue directory is found, the module moves all .msg files from the ForeignQueue to the Queue directory (it can rename the files in the process), and deletes all .tmp files found in the ForeignQueue directory. Files with other extensions are left in the ForeignQueue directory.

All moved .msg files are submitted to the Server kernel, into the ENQUEUER queue, and the Server starts to process them in the same way it processes all submitted messages.

When the scanning process is completed, the ForeignQueue directory can be removed from the base directory.