CommuniGate Pro
Version 6.3
 

Remote Poll Module

The CommuniGate Pro Remote Poll Modulr implements E-mail message retrieval from remote sources via TCP/IP networks.

The Remote Poll Module can retrieve messages for each CommuniGate Pro Account from several remote mailboxes via POP3 and IMAP protocols using RPOP and RIMAP components, respectively. This way users can collect mail from several external accounts in one place in their CommuniGate Pro Account.

The RPOP Component of the Remote Poll Module can retrieve mail for your entire Domain using "Unified Domain-wide accounts" and distribute retrieved messages to their recipients.

The Remote Poll Module can be used when the CommuniGate Pro Server has a dial-up connection with dynamically assigned IP address, and thus the Server cannot receive mail via SMTP.


Configuring the Remote Poll Module

Use the WebAdmin Interface to configure the Remote Poll Module. Open the Mail pages in the Settings realm, then open the Remote Poll pages.

Processing
Log Level:   Processors:

Delay Failed Hosts for: Use APOP
Delay Failed Accounts for: Allow Self-Poll
Source IP Address: Use Domain IP Addresses
Log
Use the Log setting to specify what kind of information the Remote Poll Module should put in the Server Log. Usually you should use the Major (message transfer reports) or Problems (message transfer and non-fatal errors) levels. But when you experience problems with the Remote Poll Module, you may want to set the Log Level setting to Low-Level or All Info: in this case protocol-level or link-level details will be recorded in the System Log as well. When the problem is solved, set the Log Level setting to its regular value, otherwise your System Log files will grow in size very quickly.

The Remote Poll Module records in the System Log are marked with the RPOLL tag.

Processes
When you specify a non-zero value for this setting, the Remote Poll Module starts to connect to the remote hosts and retrieve mail from accounts on those hosts. The setting is used to limit the number of simultaneous connections the Remote Poll Module can initiate.
Use APOP
The RPOP Component can use the secure APOP authentication method when connecting to hosts that support this feature. If for any reason you want the RPOP Component to always use the "clear text" passwords, disable the Use APOP option.
Source IP Address
This option selects the default source network address for outgoing connections. You can allow the server OS to select the proper address or your can explicitly select one of the server IP addresses as the default source network address.
Use Domain IP Addresses
This option selects source network addresses for outgoing connections. If this option is selected, the Remote Poll Module will use the first Assigned IP Address for the Domain the RPOP or RIMAP record belongs to, and if the Domain Assigned IP Addresses can be used for outgoing connections.
If this option is not selected, or if the Domain does not have any Assigned IP Address, the Remte Poll Module uses the default source network address.
Delay Failed Hosts
When the Remote Poll Module fails to connect to an external host, it marks the host as "failed" and stops polling all accounts on that host. The option specifies when the Remote Poll Module should try to poll the failed host again.
Delay Failed Account
When the Remote Poll Module fails to open a mailbox (wrong password, remote mailbox is locked, etc.), or if the connection fails when the Module retrieves messages from a remote account, the Module delays polling of this Account for the specified period of time.
Allow Self-Poll
Very often CommuniGate Pro users misunderstand the concept of remote account polling and specify their own CommuniGate Pro accounts as the "remote" accounts to be polled. This creates message loops and wastes Server resources. If this option is not selected, the Remote Poll Module checks the network address of the remote server it has to connect to. If that address is one of the CommuniGate Pro Server own network addresses, the "remote" account is not polled.

Click the Update button to modify the Remote Poll Module settings.


Configuring Account RPOP Records

The RPOP Component of the CommuniGate Pro Remote Poll Module can poll POP accounts on remote hosts on behalf of the CommuniGate Pro Account users. For each CommuniGate Pro Account several external POP accounts (RPOP records) can be specified. RPOP records can be specified by Server and Domain Administrators, using the WebAdmin Interface.
Open the Account Settings pages and open the Remote Poll page in the Mail section:

Name Poll Every Account at Host Password Leave APOP TLS Mailbox Last
firstISP 12:34:56
 

If an Account has the CanModifyRPOP setting enabled, the Account user can modify the Account RPOP records via using the WebUser Interface, or a XIMSS client.

Name
This is the RPOP record name: any text specified when the record was created.
Poll Every
This option specifies how often the Remote Poll Module should poll the remote account.
Set this option to Never to remove this RPOP record.
If you set this option to Disabled, the RPOP record is not removed, but the remote account is not polled.
Account
This option specifies the mail account name at the remote host.
at Host
This option specifies the exact name of the POP server that should be polled. Please note that this could be the name of a specific computer (as specified in DNS A-records), not just a generic domain name of the provider system. For example, if the provider has the domain name provider.com, its POP server is usually named mail.provider.com or pop.provider.com. Consult with your provider.

Note: Standard POP servers accept incoming connections on the TCP port 110. If you need to poll an account on a remote POP server that uses a non-standard port, specify the port number after the host name, using the colon (:) symbol as the separator:
pop.provider.com:111

Password
The password to use to log into the remote account.
Leave
If this option is selected, the RPOP Component does not delete messages from the remote mailbox. Instead, it remembers the UID (Unique IDentifier) of the retrieved messages, and the next time the RPOP Component polls this remote account, it does not retrieve messages that have the same UIDs.
If you want to use this option, verify that the remote POP server supports the UIDL command.

Note: messages UIDs are stored in the Account File Storage, in the private/rpopids/name text files, where name is the RPOP record name.

APOP
If this option is selected AND the UseAPOP Remote Poll Module option is enabled AND the target host advertises APOP capability in its initial prompt, the RPOP Component uses the secure APOP method for authentication on that remote host.
TLS
If this option is selected, the RPOP Component tries to establish a secure (SSL/TLS) connection with the remote host.

Note:Standard POP servers accept incoming secure connections on the TCP port 995. If you need to poll an account on a remote secure POP server that uses a non-standard port, specify the port number after the host name, using the colon (:) symbol as the separator:
pop.provider.com:9786

Mailbox
This option can specify some Account Mailbox name. If some name is specified, then the retrieved messages are immediately stored in this Mailbox, without any additional processing.

If this option is not specified, the retrieved messages are sent to the Account via the CommuniGate Pro Queue, so all Server-Wide and Account-Level Rules (including External Filters) are applied to these messages.
These messages are flagged as 'do not report failures', so if delivery to the Account was unsuccessful, no error report is sent to the original message sender.

Last
If the last attempt to retrieve mail from the remote account was successful, this field tells when (in the server local time) this attempt took place.
If the last attempt was not successful, the field contains the error code.

To remove an RPOP record, set its polling period value to Never.

To create a new RPOP record, enter its name in the last table row and select some valid polling period value.

Click the Update button to modify the RPOP record set.

Note: The RPOP Component supports non-standard MSN POP3 servers: if the remote host (server) domain name ends with .msn.com, the Component uses the non-standard AUTH MSN method to log into that server.


Configuring Account RIMAP Records

The RIMAP Component of the CommuniGate Pro Remote Poll Module can poll IMAP accounts on remote hosts on behalf of the CommuniGate Pro Account users. For each CommuniGate Pro Account several external IMAP accounts (RIMAP records) can be specified. RIMAP records can be specified by Server and Domain Administrators, using the WebAdmin Interface.
Open the Account Settings pages and open the Remote Poll page in the Mail section:

Name: my other account
Poll Every:
Account:
at Host:
Password:
Connection Security:
Authentication Method:
Filter:
Last: 15-Mar Messages: 3 (62K)

If an Account has the CanModifyRPOP setting enabled, the Account user can modify the Account RPOP and RIMAP records via using the WebUser Interface, or a XIMSS client.

Name
This is the RIMAP record name: any text specified when the record was created.
This name will be used as the root folder name of the folder hierarchy retrieved from the remote account.
Poll Every
This option specifies how often the Remote Poll Module should poll the remote account.
Set this option to Never to remove this RIMAP record.
If you set this option to Disabled, the RIMAP record is not removed, but the remote account is not polled.
Try Now
Pressing this button will create an out-of-schedule poll order. Scheduling the order may take several minutes. While the order is active the button is not displayed.
The button may be used even when the Poll Every is set to Never.
Account
This option specifies the mail account name at the remote host.
at Host
This option specifies the exact name of the IMAP server that should be polled. Please note that this could be the name of a specific computer (as specified in DNS A-records), not just a generic domain name of the provider system. For example, if the provider has the domain name provider.com, its IMAP server is usually named mail.provider.com or imap.provider.com. Consult with your provider.

Note: Standard IMAP servers accept incoming connections on the TCP port 143. If you need to poll an account on a remote IMAP server that uses a non-standard port, specify the port number after the host name, using the colon (:) symbol as the separator:
imap.provider.com:1143

Password
The password to use to log into the remote account.
Connection Security
Sets the outgoing connection security type.
None
Connection to port 143 without encryption.
STARTTLS
Connection to port 143 with encryption via STARTTLS.
SSL/TLS
Connection with encryption to port 993.
Authentication Method
Sets the outgoing connection authentication type.
Normal Password
Clear text password.
Encrypted Password
CRAM-MD5 encrypted password.
Filter
A parameter of LIST command of IMAP protocol. Allows to reduce the number of folders being retrieved from the remote account.
Error
Contains the error message if the last poll attempt had failed. If there was no error the field is not displayed.
Last
If the last attempt to retrieve mail from the remote account was successful, this field tells when (in the server local time) this attempt took place.

To remove an RIMAP record, set its polling period value to Never.

To create a new RIMAP record, enter its name in the last input form and select some valid polling period value.

Click the Update button to modify the RIMAP record set.

Note: In order to re-synchronize the contents of a particular folder with the folder from the remote account you can delete the local folder; the folder will be recreated and its contents will be filled on next polling.


Processing Unified Domain-Wide Accounts

A Domain-wide account is an account on the ISP or any other host that collects all messages sent to your Domain. The RPOP Component can be instructed to retrieve mail from such an account and distribute it to the local users. The RPOP Component can poll several Unified Domain-Wide accounts.

When a message is sent via the Internet, the information about the sender and the message recipients is sent in the so-called mail envelope. If mail is sent via SMTP, the envelope is sent as a sequence of protocol commands.

The information in the envelope is usually the same as the information in the message header fields, but it is not always true. The most important exceptions are:

  • the message header fields do not contain the addresses of the Bcc recipients
  • the header fields of a mailing-list message do not contain the mailing list subscriber addresses.

When a message is stored in a mailbox, the envelope information about the sender is added to the message headers as the Return-Path header field. Usually, the envelope information about the recipients is not added to the message headers.

When the RPOP Component retrieves a message from a Unified Domain-Wide Account, it has to recompose the message envelope and deliver the message to its final recipient. If the message contains the Return-Path header field, the address in that field is placed in the new envelope as the sender's address, and the header field is removed from the message (it will be recreated when the message is delivered to its final destination).

If a Unified Domain-Wide Account is created with the mail system that can copy the recipient addresses from the envelope into some message header field, then the delivery via RPOP is as reliable as SMTP delivery.
Enter the name of that header field into the Unified Account RPOP record settings, and the RPOP Component will look for that field in all messages retrieved from that account. The addresses from that field will be placed into the new envelope and the messages will be directed to those addresses. The header field itself is removed from the message. All accepted addresses get the 'report on failure' flags, so if message delivery fails, the original message sender (the address in the message Return-Path field) will receive an error report.

Unified Domain-Wide Accounts can be provided with a CommuniGate Pro Server running on the provider side. For messages stored in those accounts, the envelope recipients are added to the message headers as the X-Real-To fields. To learn how to provide Unified Domain-Wide Accounts with CommuniGate Pro, check the Local Delivery Module section.

A legacy sendmail system can be configured to add X-Real-To header fields, too. See the Appendix A below.

RPOP records for Unified Domain-Wide Accounts should be created for the postmaster Account in the Main Domain.

The WebAdmin RPOP page for this Account contains the Special field:

Name Poll Every Account at Host Password Leave APOP TLS Special Last
firstISP 12:34:56
 
Special
The name of the messages header (RFC822) field that the provider host inserts into the messages stored in the Unified Domain-Wide Account.

Mail Distribution without Special Header Fields

Many ISPs still use various legacy mail systems that cannot store envelope recipients in message headers. If you have to host your Unified Domain-Wide Account on such a system, enter the star (*) symbol into the Special field.

The RPOP Component will search for all To:, Cc:, and Bcc: header fields in retrieved messages. It will use the addresses from those header fields only if that address is routed to any existing local CommuniGate Pro Account.

If an address is routed to the SMTP or some other Module, or an address cannot be routed at all (unknown user name error, etc.), the RPOP Component does not send any error messages to the sender. The Component simply ignores that address.

All accepted addresses get the 'do not report failures' flags, so if the message delivery fails for any reason, no error report is sent to the original message sender.

If none of the message To:, Cc:, or Bcc: addresses has been accepted, the RPOP Component sends that message to the postmaster Account in the Main Domain.

As explained above, the method based on To:/Cc: header field parsing can cause problems when the actual envelope addresses are not the same as the header field addresses. Besides, some systems do not process the Unified Accounts correctly, so if a message is sent to three users in your domain, those systems may store three copies of the message in the Unified Domain-Wide Account Mailbox. Since each message header contains the addresses of all three users, the RPOP Component will deliver three copies of the message to each user.

The problems with Bcc, mailing lists, and duplicated message can be very annoying, so we strongly recommend you to ensure that the provider's mail system adds the envelope information to the messages stored in your Unified Domain-Wide Account, and you can use the Special Header Field feature.


RPOP Record Format

The RPOP records are stored in the Account database as a dictionary. An RPOP record name is used as the dictionary key, and the correspoding value is a dictionary, containing the following elements:
domain
the domain name or the IP address of the remote mail system.
authName
the mail account name in the remote mail system.
password
the mail account password in the remote mail system.
mailbox
(optional) the name of the mailbox to store retrieved messages in.
special
(optional) the name of the "envelope address" messages header field (see above).
leave
(optional) if the element is present and it has the YES value, mail messages will not be deleted from the remote mail system after they have been retrieved.
TLS
(optional) if the element is present and it has the YES value, connections to the remote mail system must be established securely, using the SSL/TLS protocol.
APOP
(optional) if the element is present and it has the YES value, the APOP login method should be used with the remote mail system.

RIMAP Record Format

The RIMAP records are stored in the Account database as a dictionary. An RIMAP record name is used as the dictionary key, and the correspoding value is a dictionary, containing the following elements:
domain
the domain name or the IP address of the remote mail system.
authName
the mail account name in the remote mail system.
password
the mail account password in the remote mail system.
connectionSecurity
(optional) the type of connection. Possible values are: None, STARTTLS, "SSL/TLS"
authMethod
(optional) the authentication method. Possible values are: Normal, Encrypted
listFilter
(optional) the parameter of LIST command of IMAP protocol.
extOrder
(optional) the time stamp of the out-of-schedule poll order.

Appendix A. Configuring sendmail for Unified Domain-Wide Accounts

The following file can be used to force the freeware sendmail program to store the envelope information in message headers.

# This file should be placed into the directory cf/feature from
# the sendmail.8.X.XX.cf.tar.Z archive.
# To add special headers, the macros `FEATURE(xrealto)' should be
# added to the main configuration file in the directory cf/cf,
# and the flag T should be added to the mailer description.
#
# This file adds special headers with the `X-Real-To' keyword.
# The special headers will be added to all messages routed to the
# mailer marked with the `T' flag in the sendmail configuration.
divert(0)
VERSIONID(`@(#)xrealto.m4 0.1 1/4/96')
 
divert(9)
# add the X-Real-To: header field to the message
# if the mailer is marked with the `T' flag
H?T?X-Real-To: $u
divert(0)

After these updates are applied, make sure that sendmail delivers all mail for your domain to one account on the sendmail system. The sendmail configuration for that unified account should list the 'mailer' marked with the 'T' flag.


CommuniGate Pro Guide. Copyright © 2020-2023, AO StalkerSoft