Exchange Email Ingestion

This article describes how to use the onboarding tool (OT) for exchange server.

Introduction

Introduction

The onboarding process consists of following steps:

  • Retrieve a list of users to ingest from Mailsphere.
  • Iterate through all users and their messages
  • Process the messages by sending to Mailsphere

Prerequisites

  • Java 7 or above installed
  • Download the Exchange Email Ingest tool
  • Enable Exchange web services on Exchange server
  • Open ports to Mailsphere on port 443
  • Credentials of ingest user in Exchange who can read messages in all mailboxes (see 'Settings Permissions' below)
  • Credentials of a Mailsphere RestAPI account
  • Run the Exchange User Sync script first

Setting Permissions - Exchange 2010 & 2013

To facilitate the email ingestion from Exchange 2010 and 2013 we need to ensure that Exchange Web Services are running on the target exchange server and that a user is set up with access to all mailboxes.

  1. Configure a new ingest user in the AD and Exchange and record the username and password
  2. Add the username to the onboarding tool configuration file - the password will be asked when you run the script
  3. Run the following Exchange shell script to give the necessary permissions to the new ingest user
New-ManagementRoleAssignment -Name:exchangeImpersonation -Role:ApplicationImpersonation -User:IngestExchangeUser
Get-Mailbox | Add-MailboxPermission -User “IngestExchangeUser” -AccessRights FullAccess -InheritanceType All

Note: where it states IngestExchangeUser please use the new user name that you just created in AD and Exchange

Setting Permissions - Exchange 2007

To facilitate the email ingestion from Exchange 2007 we need to ensure that Exchange Web Services are running on the target exchange server and that a user is set up with access to all mailboxes through impersonation.

  1. Configure a new ingest user in the AD and Exchange and record the username and password
  2. Add the username to the onboarding tool configuration file - the password will be asked when you run the script
  3. Run the following Exchange shell scripts to give the necessary permissions to the new ingest user
Get-ExchangeServer | where {$_.IsClientAccessServer -eq $TRUE} | ForEach-Object {Add-ADPermission -Identity $_.distinguishedname -User (Get-User -Identity IngestExchangeUser | select-object).identity -extendedRight ms-Exch-EPI-Impersonation}
Get-MailboxDatabase | ForEach-Object {Add-ADPermission -Identity $_.DistinguishedName -User IngestExchangeUser -ExtendedRights ms-Exch-EPI-May-Impersonate}
Get-Mailbox | Add-MailboxPermission -User “IngestExchangeUser” -AccessRights FullAccess -InheritanceType All

Note: where it states IngestExchangeUser please use the new user name that you just created in AD and Exchange

For more information on setting up EWS impersonation on Exchange 2007 please follow this link:

http://msdn.microsoft.com/en-us/library/bb204095(v=exchg.80).aspx

Testing your Exchange configuration

Testing your Exchange configuration

Now that you have configured your Exchange server it is recommended to test that the Exchange Web Services are operating correctly and that the user you have configured can login.  Go to the following URL in a browser of your choice on the Exchange server:

https://localhost/EWS/Exchange.asmx

The Exchange Web Service will ask for authentication.  Enter the username and password.  Depending on your Exchange configuration this may require a domain name, the email address or just the username.  Enter the password and click OK.

If the ExchangeWS is set up and the user configured it will display an XML output reporting the web service schemas available.  You may now close your browser.

Running the tool

The onboarding script is run using following command where 'mail-sync.jar' is the name of the ingest tool jar file and path/to/config/file is the path and name of the properties file.

java -jar mail-sync.jar ingest path/to/config/file

If the supplied jar is called 'mailsphere-mail-sync.jar' and both it and the properties file is called 'mailsync.properties' and both are stored in the same folder then use the command prompt to access the folder holding these files and run the following:

java -jar mailsphere-mail-sync.jar ingest mailsync.properties

Example configuration file

Contents of an example configuration file:

# Example configuration of MailSync ingestion tool
# host and port of MailsphereUI server
mailsync.mailsphere.host=https://portal.mailsphere.co.uk
mailsync.mailsphere.port=443
# User with privileges to access Mailsphere REST interface
mailsync.mailsphere.user=api
# Password for mailsphere, if left commented out or empty it will be requested
mailsync.mailsphere.password=password
# Exchange server EWS endpointUrl
exchange.endpoint.url=https://localhost/EWS/Exchange.asmx
# Credentials for test user accessing mailboxes
# This user must have permission to access exchange.userName
exchange.credentials.username=admin.user
# Password for exchange, if left commented out or empty it will be requested
exchange.credentials.password=admin/user/password
# Ignored folders
# System - users don't have access to this folder
# Deletions - contains permanently deleted messages
exchange.folders.ignored=Drafts,Deletions,System
# Used if you want to limit which accounts to ingest from
#exchange.mailbox.use-whitelist=false
#exchange.mailbox.whitelist=test.user1@demolab.co.uk,test.user2@demolab.co.uk,test.user3@demolab.co.uk
# Set how many items to retrieve in one query when listings folders' contents
exchange.items-per-request=100
# Set delay in ms between two queries to Exchange servers, set to 0 to turn off delay
exchange.query-interval=0

If you wish to limit the ingest to specific mailboxes then the whitelist configuration can be used by changing the following lines:

# Used if you want to limit which accounts to ingest from
exchange.mailbox.use-whitelist=true
exchange.mailbox.whitelist=ingest.mailbox1@demo.com,ingest.mailbox2@demo.com

Possible Error Codes

The following error codes can be reported if the configuration file has incorrect entries:

401 - Check the Mailsphere API user credentials.  It appears that either the user ID or the password is incorrect.  Confirm these are correct by loggin on to the portal.

500 - Check the Mailsphere Host details are correct. This should be https://portal.mailsphere.co.uk using port 443

Could not find index folder - This results from the ingest tool being unable to access the Exchange Web Services. It can result from any of the following:

  • The Exchange credentials are incorrect - the error will be repeated for all mailboxes being attempted
  • The Exchange user does not have impersonation enabled - the error is reported for all mailboxes except the mailbox for that user
  • The Exchange Web Services are not responding in a timely fashion - try adding an exchange query interval using the parameter at the end of the configuration file.