Data migration and replication

Given current BlueMind-Outlook stability with the MAPI protocol, migrating data through a PST upload in Outlook is not an option. As a whole, server-side data migration is safer and the result will be more consistent.

Recommended data migration solutions include:

• Exchange migration tool
• server-side PST migration
• IMAP synchronization tools with imapsync (see recommendations below)
• Domino migration tool

Replication extracts and stores mail spool information into BlueMind objects that must exist beforehand. For replication to work properly, only data known by BlueMind must be fed into Cyrus: domains and mailboxes must therefore be created in the admin console (or via API), before the mailboxes are filled with data. 

Currently, as admin0 (BlueMind super-administrator), you can migrate BlueMind data without worrying about BlueMind objects and mail storage rules. With admin0 privileges, mail data can be stored on the Cyrus server without undergoing any rights or consistency checks. This is why BlueMind may see the data as inconsistent and may cause the replication to fail. We are therefore advising you against importing data through imapsync as the admin0 user.

To avoid this, if you are planning an imapsync transfer of BlueMind data, it is important for the imapsync to be carried out with the user's login ID. By performing operations as a mail user, you are assured that an account exists, with the correct partition, etc.

To generate an API token for a specific user:

https://forge.bluemind.net/stash/projects/BA/repos/bluemind-samples/browse/python-api-examples/generateUserToken.py

This link shows and example of data migration which you can adapt depending on the source server and the accounts/data you want to transfer:

https://forge.bluemind.net/stash/projects/BM/repos/production/browse/4.0/migration/mailsync_kerio.py

As a whole, and in particular for version 4.0, we strongly recommend that you test data migration on a test server, which will be re-installed or destroyed later. The migration process, once verified, can be done on a blank server (or domain), with no trace of the operations carried out during testing. The LDAP/AD connection, Imapsync or Exchange data migration, once prototyped successfully, can be replayed on the production server.

Verifying that replication works properly

In the bm-tick monitoring console, you can watch the "Mailspool & Replication" dashboard. Two graphs are particularly relevant:

Number of messages replicated per hour:  

Number of active replications:

This number must be 1 per server with the mailbox role and therefore the bm-cyrus-imapd service. If this number drops, this means that the replication is no longer working.

On the contrary, if this number is higher than the number of IMAP backends, it usually means that the role has been given one – or several – separate storage server(s) but the service is still running on the main server. In that case you need to force-disable them by creating the following files on the bm-core and then stop the services:

touch /etc/bm/bm-cyrus-imapd.disabled
touch /etc/bm/bm-lmtpd.disabled
systemctl stop bm-cyrus-imapd ; systemctl stop bm-lmtpd

To see if the replication is working, you can perform an operation on an email (e.g. change it to unread) and using a tail command, check whether, at the same time, a line looking like the one below is added to the /var/log/bm/replication.log log file:

{{APPLY MAILBOX (.... UserloginID ) }}

Replication progress

We are planning tick improvements in future versions which will allow you to check the replication process's progress.

In the meantime, you can compare the number of messages in the mail spool and archives with the number of entries in the message storage table. They won't match exactly, but it gives a pretty good idea of progress. 

To find out the number of emails to be synchronized:

# Number of messages in the spool:
find /var/spool/cyrus/data/ -type f|wc -l

# Number of messages in the archives:
find /var/spool/bm-hsm/cyrus-archives -type f|wc -l

The sum of the two should be close to the result form the query on the bj-data database:

select count(*) from t_message_body;

Note that the standard replication flow only watches "live" mailboxes. This means that if the replication delta is significant, then the replication has almost stopped, all active users have been properly replicated and have access to related features (webmail search, EAS, etc.). Also note that the count is approximate: if an email is sent to N users, it will be counted once in t_message_body but it will be present N times in the spool.

To start the replication on idle (unused) mailboxes, you must place them in the replication queue using the following command – after having cleaned some Cyrus logs:

#cleaning cyrus
service bm-cyrus-imapd stop
rm /var/lib/cyrus/sync/core/log*
service bm-cyrus-imapd start
#running the replication
bm-cli maintenance repair --ops replication.parentUid $DOMAIN_UID$

$DOMAIN_UID$ being the domain name, e.g.: bluemind.net

BlueMind

without MAPI

Outlook

Sans le Without the MAPI service MAPI, Outlook continue de fonctionner avec connecteur de la même façon qu'en , Outlook continues to work with the connector like in version 3.5. Les administrateurs doivent effectuer la même procédure de Mise a disposition du connecteur Outlook pour que les utilisateurs puissent le télécharger et l'installer sur leurs postesAdministrators must carry out the same provisioning procedure for the Outlook connector so that users can download and install it on their machines.

Cyrus

À partir de la From version 4.1, une vérification de l'arborescence Cyrus est réalisée au démarrage de BlueMind et alerte (au moyen d'un warning dans les logs) si l'arborescence n'est pas cohérentethe Cyrus folder structure is checked on BlueMind startup and an alert – a warning in logs – is sent if an inconsistency is found.

BlueMind avec MAPI

Autodiscover

À partir de la version 4.1, une vérification de l'autodiscover est réalisée sur tous les domaines et alias de l'installation : si aucun autodiscover ne fonctionne alors le service MAPI ne démarre pas, si au moins un autodiscover est bon alors le service démarre afin de servir les domaines accessibles.

Ainsi pour chaque domaine et alias, le serveur mapi tente une requête vers domain.loc/autodiscover et autodiscover.domain.loc/autodiscover et vérifie qu'il reçoit bien lui-même la requête.

Une vérification est aussi effectuée sur le serveur DNS pour vérifier l'enregistrement _autodiscover._tcp.domain.loc et _autodiscover.<tous les alias>.

Cyrus

À partir de la version 4.1, une vérification de l'arborescence Cyrus est réalisée au démarrage de BlueMind et alerte (au moyen d'un warning dans les logs) si l'arborescence n'est pas cohérente.

Outlook

Creating a blank Outlook profile

To enable connector-free Outlook, first make sure you follow the server-side implementation steps described in our documentation:

Mise œuvre de MAPI pour Outlook

In particular, make sure you read the section about server communications prerequisites: l'autodiscover doit fonctionner correctement pour qu'Outlook puisse communiquer avec BlueMind.

Then, for each workstation, follow our instructions on creating an Outlook profile:

Synchronisation avec Outlook

In this case, make sure you first ensure url accessibility from the workstation.

Outlook with BlueMind limitations

Known limitations with Outlook are listed in our ad-hoc page on compatibilité de BlueMind avec les logiciels clients et appareils

Known limitations

You can find known limitations in the following page on BlueMind compatibility.

Mise à jour de BlueMind 4.0 vers 4.1

Dossiers sous la boîte de réception

Dans les versions 4.0.x de BlueMind (4.0.x), les dossiers créés sous la boîte de réception par Outlook ne sont pas des dossiers de messagerie mais des dossiers virtuels.

BlueMind 4.1 apporte le support des sous-dossiers de la boîte de réception (inbox).