Migrating GMail to Exchange (part 1) - Configure GMail

The first step of your migration process is to configure your GMail account.

1. IMAP Access

In order to see your messages in Outlook, you'll need to configure your GMail account first.  You can do this under  => Settings => Forwarding and POP/IMAP.

Select Enable IMAP.

OPTIMIZATION #1 (recommended)

In order to speed up the conversion process, it makes sense to decouple deletion of a message in Outlook's cache from the actual expunge of that e-mail.   This way, Outlook will mark messages for deletion first, and communicate them in one shot to the server at the end of the conversion. You do this by selecting "Auto-Expunge off".

Once you do this, "Archive the message" should be selected below.  You must do this so that all labels can be migrated over.  This allows to migrate each label individually.

OPTIMIZATION #2 (consider)

I've found that the conversion speeds up considerably, if the number of messages in a folder are small.  It also may be that Outlook uses more memory traversing large folders, slowing the processor, and the conversion process down.  This also may be related to the size of the Outlook cache file - where Outlook downloads the IMAP folders' contents.  Either way, one way to control the size of the cache file and the required resident memory is to limit the number of messages in each folder.

I measured about 15-30 seconds/item/label conversion speed when the folder size was unlimited.  This reduced to 1-5 seconds/item/label when limiting folder size at 1000.

NOTE: If you limit the folder size, you will need to do the conversion in multiple iterations:

1. download messages (up to the limit) to local Outlook cache
2. convert downloaded messages
3. purge the messages on the GMail server, so new messages can be downloaded

NOTE: there is a link at the bottom of this page with instructions to set up your Outlook to access your GMail messages.

2. Labels

Control your Outlook cache size

Another way to control the size of the Outlook cache is to select which labels show up in the IMAP cache.  Under   => Settings => Labels, you can select for each label, whether or not to "Show in IMAP".  Here you can also see how many conversations are labeled with a certain label.  This allows you to gauge the size of the Outlook cache.

E.g. if you have 60000 conversations (this you know by checking the number when looking at all your mail.  It will say something like 1-50 of 60,000), you are using 6GB (this shows at the bottom of your GMail page), and you have a total of 60,000 conversations under all labels, you can estimate your total cache to be 12GB.  However, you also need to include the number of messages in your Inbox, Starred and Important, as these are not listed on the Labels page.

Exchange compatible label names

Outlook 2010 does not allow category names to start/end with space, or to contain comma (,) or semicolon (;).  Rename all of your labels that contain any of these letters.  Gmail labels already cannot start/end with space.

3. Decide what to do with your Trash

A. Do not migrate trash

The easiest option is to not migrate your trash.  Then, for best performance, go to your trash, and click "Empty Trash now".

B. Migrate trash with all labels

Messages in the trash retain their labels; however, these are not exported to the IMAP interface (deleted messages will not appear under their labeled directories).  If you want to migrate the messages in the trash with all their labels, you will need to add your own "trash" label.  You can do this by

1. Go to your trash

2. Select all mail on the scree on the checkbox

3. If you have more than 50 messages, a link will appear "Select all X conversations in Trash".  Click that link.

4. Move all mail in the trash under a new label (e.g. "T").  Make sure you move to a newly created label so that you can keep messages from your trash separately.

During conversion, you can designate this label to go to your Trash folder.

NOTE: you can use this trick to separate messages with certain labels in your trash to delete them permanently.  It is because now that you freed up your trash, you can selectively put only messages that you want to delete permanently into it.

NOTE: if you are migrating your trash with labels, this will increase the size of the Outlook cache similarly to having multiple labels on regular messages, as these messages will now appear in the folders of all of their labels.

C. Migrate trash without labels

If you want to migrate your trash without your labels, you can keep them where they are in GMail.  You will need to set your trash label to "Trash" during migration.

Next... configure your Outlook client

Migrating GMail to Exchange (part 0)

Update: I have now posted a quicker procedure to migrate your GMail to MS Exchange here.

Recently I took on the project of migrating from GMail to MS Exchange.  This is being an arduous process.  While I see no sane reason why to move from GMail with it universal accessibility and ease of interface to Exchange, sometimes there are external reasons.  E.g. your IT department decided after their Google Apps trial that it is not for them.

This series of posts will document a way to do this, while at least keeping all your GMail labels with your messages.  There may be other, easier ways, but a simple Google search did not yield a useable result.

You will need to have Outlook (part of MS Office) for this process.  Exchange (or Outlook at least) allows you to attach 'categories' to each item.  We will use these instead of GMail's labels.  Good news: I was able to translate all of my GMail filters to Outlook rules, albeit these have to run on the client (e.g. Outlook).  I'm sure you'll be able to do the same.

This guide is written for Outlook 2010, but should also work similarly for Outlook 2003 or 2007.  Note of caution: the conversion will take a LOT of time.  In my case it is taking 2.5 seconds/email/label.  You will not be able to use Outlook during the conversion. I recommend using Outlook Web Access if you have to access Exchange during the conversion, unless you have access to another machine.

We will use GMail's IMAP interface to access your old mail.  In this interface, there is a folder for each label, and a message will appear in ALL of the folders of its labels.  Outlook will see each of these copies as separate emails.  However, after the conversion, all of these copies will be consolidated into a single email with the labels represented as categories.

Limitation

Chat messages are not exported to IMAP, so they will not be migrated to Outlook.

Update: I have now posted a quicker procedure to migrate your GMail to MS Exchange here.

Let's begin... 

Debug utility function caveats

What's wrong with the following code assuming there is no danger of null dereference?

int debug;
module_param(debug, int, 0644);

...

void dump_xyz_info(struct xyz *obj)
{
    if (!(debug & DEBUG_INFO))
        return;

    dev_info(mydev, "[a=%d b=%d c=%d]\n", obj->a, obj->b, obj->c);
}

While I though it was a good idea to incorporate the debug flag check in my debug utility method, I learned later that it prevents me from using this method for debugging, and having to resort to ugly tricks like this:

void dump_all_info(struct the_obj *obj)
{
    int old_debug = debug;
    debug = DEBUG_INFO;
    dump_xyz_info(obj->xyz);
    dump_pqr_info(obj->pqr);
    debug = old_debug;
}

OK, so I did not actually do this and fixed the issue at the first place. But how many times have we seen ugly tricks in the code like the above? Now I cherish expressions like:

if (debug & DEBUG_INFO)
    dump_xyz_info(obj->xyz);

Cheers!

Run checkpatch automatically on git commit

This is a script I wrote that hooks into git-commit, and automatically runs checkpatch and “git diff –check” on a commit. If any of these fail, the commit is rejected. (In case you are not already using it, checkpatch looks for common coding mistakes, and it facilitates better kernel code.)

In addition, the script sets file permissions for common files (c, h, makefile, readme) to 0644, and executables (directories, shell scripts) to 0755. This is a common issue when editing files in Windows.

To install, save the script in your kernel GIT repository/ies under the $GIT_DIR/hooks directory (usually .git/hooks) as .git/hooks/pre-commit.

If you want to bypass the check (e.g. if the failure is intended), use the -n flag for git-commit to ignore the checks. Just be sure that you still resolve any failures that were correct, and there really wasn’t a better proper way to write your code without checkpatch errors. Now there is a whole set of discussion about the 80 character limit that sometime forces less understandable code (you can read about the pros/cons and find a patch to checkpatch to make this optional) BTW, Linus prefers in many cases to ignore the 80-char-limit warnings.

But wait! You can do even more. Since you are already modifying files, why not run checkpatch on the whole file (and not just on the lines you are changing)? To see if there are any already existing checkpatch errors in the files that you modified, set up these two bash aliases that will run checkpatch on the whole files that are about to be committed (just run ‘chkp’ - NOTE: this will not work in an empty repository.):

alias gitd='git rev-parse --show-cdup'
alias chkp='`gitd`scripts/checkpatch.pl -f `git diff --name-only HEAD | awk -v i=$(gitd) "{print i\\$0}"`'

Let me know if you have any questions/suggestions.

#!/bin/bash
#
# Run checkpatch.pl and fix permissions on what is about to be
# committed.
#
# Written by Lajos Molnar  based on the sample git
# pre-commit script.

if git-rev-parse --verify HEAD >/dev/null 2>&1
then
  against=HEAD
else
  # Initial commit: diff against an empty tree object
  against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

# Change access modes on committed files - do this first so
# we fix permissions even if there are errors
for i in `git diff-index --cached --name-only $against`
do
  name=${i##*/}
  # set permissions for common kernel files
  if echo $name \
  | grep -Eq "^(README|Kconfig|Kbuild|Makefile|HOWTO|TODO|"\
"\.gitignore|.*\.(c|h|cpp|txt|S|xml|mk))$"
  then
    chmod 0644 $i
    git add $i
  # directories and scripts are executable
  elif [ -d $i ] \
  || (echo $name | grep -Eq "^.*\.(sh|pl|py)$") 
  || (head -1 $i | grep -Eq "^#!")
  then
    chmod 0755 $i
    git add $i
  fi
done

# run simple diff checks
git diff-index --cached $against --check -- || exit $?

# run checkpatch if there is actual code difference
if git diff --cached $against | grep -Eq "^---"
then
  (git diff --cached $against \
  | $GIT_DIR/../scripts/checkpatch.pl --no-signoff -) || exit $?
fi

exit 0

#1 C Macro Pet Peeve: use do while 0

I cannot fathom why I still see compound macros like:

#define TRACE(fmt, ...) { \
printf(fmt, ##__VA_ARGS__); fflush(stdout); \
}

Please, use do while 0:

#define TRACE(fmt, ...) do { \
 printf(fmt, ##__VA_ARGS__); fflush(stdout); \
} while (0)

Don't worry. Your (any remotely recent) compiler will not generate any assembly code for the "do while 0". For an explanation, just search do while 0. The basic idea is that you want these kind of macros to behave like normal C void methods. However, if you use the original macro in an expression such as "if (expr) TRACE("yes"); else TRACE("no");", the semicolons after the expansion will cause syntax errors.

For more options to use instead of do while 0, see G_STMT_START/END macros in glib.h, depending on what your compiler allows:

#define TRACE(fmt, ...) if(0) { \
 printf(fmt, ##__VA_ARGS__); fflush(stdout); \
} else (void)0

#define TRACE(fmt, ...) (void) ({ \
 printf(fmt, ##__VA_ARGS__); fflush(stdout); \
})

C Macro School

C is language I use most commonly. One of the least developed features of C (and coincidentally my favorite feature) is its macro preprocessor. Luckily, during the past decade GCC has implemented many improvements that ease the creation of very powerful macros.

Over the course of my projects, I very often work with someone else or on someone else's code, and I am surprised that even after 15 years, I keep seeing the same mistakes. So I am going to list my list of "My 10 Greatest Pet Peeves with C Macros". Well there may not be 10, but there are many. So let's begin...

Hello

Hi, welcome to my new blog on all things software. I am a software engineer. My friends tease me that I only attended an Institute. I am here to assert myself. Keep on reading...