Don't store invalid objects in cache

For example, before the change it was possible to end up with
a task object in a calendar folder cache.