Documentation:CacheInvalidation

Read the background: https://blog.mdriven.net/cache-invalidation-a-real-problem-for-us-all/

Update 2022-11-29: To Clear Cache Invalidation Data and Start Over

To clear every cache module, you need to delete in the database accordingly:

delete from MDrivenCache_ManifestHead
delete from MDrivenCache_ManifestRow

Deleting the heads will make objects of relevant classes re-create the cache modules still marked for cache invalidation.

Admin and Update: Update January 2021

CacheInvalidation maintance view in MDrivenServer:

In this UI, you can find out the Latest or oldest invalidated and updated cache modules. You can list all the manifest rows for one module. You can delete a module if needed.

Update to Strategy 20200312

Changes to cache invalidation. Save writes to the recentlyupdatedtable, and then this table was joined with a cache manifest that could take time - and this locked the recentlyupdatedtable table that in turn limited other saves for the complete system.

To fix this, a new table has been added to recentlyupdatedtableWorkPart. We quickly move seen rows in the recentlyupdatedtable to recentlyupdatedtableWorkPart to minimize the time recentlyupdatedtable is in a transaction. Then we do the potential heavy work from the recentlyupdatedtableWorkPart table.

The EnsureCacheInvalidationTables function adds the recentlyupdatedtableWorkPart table, even if it is the only one missing.

Original Strategy

MDrivenCacheInvalidationAdmin

• LastInvallidationLoop
• LastUpdateLoop

MDrivenCacheInvalidationRecentlyUpdated

• id
• classname
• membername
• time

MDrivenCacheManifestHead

• ViewModel
• RootId
• Class
• viewmodelMetaChecksum // set on update
• priority // set by ViewModel property - this way information can control importance!
• Updated     // set on update
• Invalidated

MDrivenCacheManifestRow

• Id
• Class
• MemberName

Upon Normal Save

New rows indicating member level change are inserted in MDrivenCacheInvalidationRecentlyUpdated.

Special treatment of newly created and deleted objects - they get the **created and **deleted flag

For changed embedded links (single end), we also add the dirty of the other end.

Upon InvalidationLoop

We look for direct hits on id and attribute and invalidate.

We look for any objects of a class that has been changed recently that should now be part of the set.

(ie other end of OptionalAssociationEndNameForBeingIncluded , does it point to id -> invalidate).

We look for manifest rows indicating the use of allinstances and match to **created and **deleted and invalidate.

We look for meta checksum changes of the ViewModel.

Upon UpdateLoop

We look for the Invalidated which has the diff between Now and Invalidated higher than allow oldness and we sort these by priority.

For each cache found, we instantiate the named ViewModel with the given root object.

We execute all actions on the ViewModel.

We save all the used and hence dependant model usage during cache - allowing us to set persisted calculated fields and create persisted derived objects.

We ignore the use of common superclass members to avoid invalidating caches by un-precise things i.e. changetime.

Things to Think About

How do we ensure the initial creation of the cache when a new cache is created that already has existing root objects?

Are the current polled criteria-based SSVM-Run helpful? Is it the criteria-based expression that discovers new cache needs or is it based on type alone?! Type-alone criteria will be hidden in UI and ignored in RT.

Is the cache type-wise complete - or only on matching the polling criteria?! Type complete - all objects for type will get a cache manifest

If we want to involve the polling criteria (ocl-ps) with creation need - do we also want it with refresh need? ! No

- If we involve the polling criteria in refresh decision we may end up with a invalidated caches that never gets updated and that is not good.

- It may be easier for user to completely separate the polled criteria ViewModels from the auto invalidated, mixing the two reasons for running may get confusing

- We still want the polling periodicity for update discovery to be controllable by user and also the chunking ! Yes - chunking and periodicity still applies

Anyway we look at it - we need someway to join in the cachehead to make use of the invalidated value in order to select viewmodels to create/refresh

select d.a0_detailid from a0_detail d where CAST(d.a0_detailid AS nvarchar(30)) not in (select rootid from a0_mdrivencache_manifesthead where viewmodel='ViewModelServerSideDetail')

This limits the possible RootObjects in cached viewmodels to having a single key column.

The discover of missed cached roots only creates invalidated heads - then the update process discovers, orders and executes updates.

Special ViewModel considerations

The ViewModel used in cache has the option to use the following attributes with special meaning:

CacheIgnoreClasses - set of strings that corresponds to name of classes to ignore in manifest and hence in invalidation. Example: Set{User.asstring,Singleton.asstring,SysSingleton.asstring,AcoSuper.asstring}

Priority - integer written to head and used with order by by next update loop 1 is higher priority than 1000 (default on update is 1000, on new cacheneed found prioritu is 1)

Practical queries

What has changed but not yet made it to check if it will invalidate anything

select count(*) from MDrivenCache_RecentlyUpdated

Same as above grouped on class and member:

select count(*),classname,membername from MDrivenCache_RecentlyUpdated group by classname,membername
order by classname,membername