View source for BestPractices:Complexity shows up almost immediately–get the tools to manage it

Listening to business requirements is the only way to help a business with information management. However, listening to the business will take you places you would never have thought of yourself.

It always starts small and neat: can you help us with our product development projects? We only need to keep track of what articles the project contains and their state of development. It will be fun and easy! I promise! We know almost exactly what we want and are more or less done…As a software developer, you know the drill.

These past few years, I have reminded myself: Be a pro-business developer – do what they ask – do not let practical technical issues stand in the way of progress.

Well. Where did that lead me?

In this particular view – one of 500 in the system – the requirements have kept trickling in one by one over the years. Everything has gone smoothly with no major issues. Since we do everything with UML and declarative ViewModels and as we model everything in the same lingo the business uses, it is no issue to add a few new things now and then.

Building things this way lets the complexity sneak up on you.

==== '''The View:''' ====
[[File:Complexity - 1.png|frameless|644x644px]]

It has five main levels: ''the project'', ''the articles within'', ''the packaging components for the selected article'', ''the logistic solution'', and ''the barcodes for the selected article''.

The main grid has columns to show some details for each article, same for the grid for the used components - nothing fancy and all very relaxed and moderate.

The users in the innovation department love this view. They should – they designed it.

So why do I use this simple sample to talk about complexity? I will tell you why.

Using the “Analyze expressions” button in the ViewModel dialog, I get this auto-generated diagram:

[[File:Complexity- 2.png|frameless|644x644px]]

A quick calculation shows that I have accessed 6x5=30 classes in one view. 30 CLASSES IN ONE VIEW!

And look at all those relations!

And this is not even the most complex view in the system – it is pretty typical. How did I end up here?!

'''My conclusion is:''' if you listen to the business (you know, the guys that pay your bills and that you are on a mission to help), then you end up here.

Of course, you can listen to best practices and NOT DO COMPLEX THINGS – but then again, you might as well stay home with your mother your whole life.

If my conclusion is right – how can we deal with this? It would be hard work to optimize a way to fetch objects for 30 classes in a good and speedy fashion. It is even harder to update that strategy whenever something is added or removed. This would probably require one developer to become an expert on this single view… But, hey – we have 500 views like this… and 2 developers… So 250 views like this each… Did I mention that we release to production every 4 weeks? And we have never missed a release for the last 8 years? And that we do about 50 different updates on each sprint?

=== We have(had) a Problem ===
MDriven uses the declarative ViewModel and the OCL expressions it consists of to look ahead to fetch data efficiently. In 2018, we found that it did not do it efficiently enough. It turned out that it got tricked by derivations in attributes and associations. Derivations were skipped and they resulted in lazy fetch as the UI was drawn, which resulted in a lot of single queries sent to the database. This is the same basic problem that usually kills high abstraction framework ambitions.

In our case, it became obvious we had a problem once we got users further away from the server. As long as the user had a latency of 10ms, it was fine – but at 100ms, it sucked. Since every single query sent to the DB got the latency penalty of 100ms, the number of queries hit the user experience straight in the face.

I started testing an app with a simulated latency of 400ms – that is what you can expect from Sweden to Australia – not at all our use case – but it made the problem very obvious.

We needed to rethink how the MDriven Server can help the application.

Many things were researched. The solution we came up with is what we call a '''Query Plan'''. 

=== We have a new thing to speed up the load: Query Plan ===
MDriven uses the declarative ViewModel to search for the primary keys and foreign keys needed for a particular view. It resolved OCL-based derived attributes and associations to find out persistent things that must be fetched to get the information onto the screen. It also resolved all the opt-in constraints in the classes shown and the DefaultStringRepresentation of each class used the ReadOnly, Visible, and Style expressions.

From this, MDriven retrieves all the persistent single, multi, and inner links (links to association classes). It sends this information to the MDriven Server along with the root of the part of the ViewModel being fetched. It treats the structure-building expressions (the nesting creators) separately from the column expressions.

The MDriven Server uses the information to request the set of objects needed and sends back all that data in 1 go.

This approach reduces the load on the SQL-Server – since now we can fetch multiple foreign keys for multiple multilinks in 1 query (not possible before). It also reduces the round trips needed to the server and by doing that effectively, addresses the issue we had with latency. Tests confirm a 20-time improvement in time when having 400ms latency.

Phew! We HAD a problem. Now, we have an engine that does it better than you can expect from a developer on their best day. And it works every time – for every view – even if I changed it to use 5 more classes tomorrow.

'''The view above has this in its Query Plan:'''<blockquote>''ProjectPackagingView''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ArticleChange.Project''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleChange.Article''</blockquote><blockquote>''ArticleRevision.Article''</blockquote><blockquote>''ArticleRevision.ComponentUsageConstellations''</blockquote><blockquote>''ArticleRevision.ActiveLogisticSolution''</blockquote><blockquote>''ArticleRevision.Changers''</blockquote><blockquote>''LogisticSolutionRev.LogisticSolution''</blockquote><blockquote>''Article.ArticleCollection''</blockquote><blockquote>''Article.PartOf''</blockquote><blockquote>''ArticleCollection.ConsistsOf''</blockquote><blockquote>''ArticleCollection.RepresentedAs''</blockquote><blockquote>''ArticleChange''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''User''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ArticleChange.Project''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleChange.Article''</blockquote><blockquote>''ArticleChange.Responsible''</blockquote><blockquote>''ArticleRevision.Article''</blockquote><blockquote>''ArticleRevision.Filler''</blockquote><blockquote>''ArticleRevision.Bulks''</blockquote><blockquote>''ArticleRevision.ShelfLife''</blockquote><blockquote>''ArticleRevision.LogisticsVote''</blockquote><blockquote>''ArticleRevision.SafetysVote''</blockquote><blockquote>''ArticleRevision.FormulationsVote''</blockquote><blockquote>''ArticleRevision.PackagingsVote''</blockquote><blockquote>''ArticleRevision.ComponentUsages''</blockquote><blockquote>''SalesArticle.MarketingCompany''</blockquote><blockquote>''Article.SalesArticles''</blockquote><blockquote>''Article.ArticleRevisions''</blockquote><blockquote>''Article.RegulatoryDomain''</blockquote><blockquote>''Article.ArticleCollection''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevisionLogisticsVote''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevision4''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevision3''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevision2''</blockquote><blockquote>''ComponentSpecificationRev.ComponentSpecification''</blockquote><blockquote>''ComponentUsage.UsedRevision''</blockquote><blockquote>''ComponentUsage.ArticleRevision''</blockquote><blockquote>''ComponentUsage.ComponentSpecification''</blockquote><blockquote>''ComponentSpecification.PackagingSupplier''</blockquote><blockquote>''ProjectRole.Members''</blockquote><blockquote>''Structure to reach this level from ProjectPackagingView''</blockquote><blockquote>''Project.ArticleChanges''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleRevision.Filler''</blockquote><blockquote>''ArticleRevision.ComponentUsages''</blockquote><blockquote>''ArticleRevision.Article''</blockquote><blockquote>''ArticleRevision.PackagingsVote''</blockquote><blockquote>''ArticleRevision.LogisticsVote''</blockquote><blockquote>''ArticleRevision.SafetysVote''</blockquote><blockquote>''ArticleRevision.FormulationsVote''</blockquote><blockquote>''ComponentUsage.ComponentSpecification''</blockquote><blockquote>''ComponentSpecification.PackagingSupplier''</blockquote><blockquote>''User.FavoriteArticles''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevisionLogisticsVote''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevision4''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevision3''</blockquote><blockquote>''ArticleRevisionApprovalVote.ArticleRevision2''</blockquote><blockquote>''Article.ArticleRevisions''</blockquote><blockquote>''Components''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''User''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ComponentSpecificationRev.ComponentSpecification''</blockquote><blockquote>''ComponentUsage.UsedRevision''</blockquote><blockquote>''ComponentUsage.ArticleRevision''</blockquote><blockquote>''ComponentUsage.ComponentSpecification''</blockquote><blockquote>''ComponentUsage.ComponentUsageConstellations''</blockquote><blockquote>''ArticleRevision.Article''</blockquote><blockquote>''ComponentSpecification.PackagingSupplier''</blockquote><blockquote>''ComponentSpecification.Replaces''</blockquote><blockquote>''Structure to reach this level from ArticleChange''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleRevision.ComponentUsages''</blockquote><blockquote>''ComponentUsage.ComponentSpecification''</blockquote><blockquote>''ContractManufacturer''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Structure to reach this level from ProjectPackagingView''</blockquote><blockquote>''Project.ArticleChanges''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleRevision.Filler''</blockquote><blockquote>''PackagingSupplier''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Structure to reach this level from ProjectPackagingView''</blockquote><blockquote>''Project.ArticleChanges''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleRevision.ComponentSpecifications''</blockquote><blockquote>''ComponentSpecification.PackagingSupplier''</blockquote><blockquote>''LogisticSolutionRev''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''LogisticSolutionRev.LogisticSolution''</blockquote><blockquote>''LogisticSolutionRev.PalletLogSpec''</blockquote><blockquote>''LogisticSolutionRev.TransportLogSpec''</blockquote><blockquote>''LogisticSolutionRev.ExtraRetailLogSpec''</blockquote><blockquote>''LogisticSolutionRev.RetailLogSpec''</blockquote><blockquote>''Structure to reach this level from ArticleChange''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleRevision.ActiveLogisticSolution''</blockquote><blockquote>''AffectedArticleRevision''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''User''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ArticleRevision.Article''</blockquote><blockquote>''ArticleRevision.Bulks''</blockquote><blockquote>''ArticleRevision.ActiveLogisticSolution''</blockquote><blockquote>''ArticleRevision.FormulationsVote''</blockquote><blockquote>''ArticleRevision.PackagingsVote''</blockquote><blockquote>''ArticleRevision.ArticleOwner''</blockquote><blockquote>''ArticleRevision.LogisticsVote''</blockquote><blockquote>''ArticleRevision.SafetysVote''</blockquote><blockquote>''ArticleRevision.Filler''</blockquote><blockquote>''ArticleRevision.ShelfLife''</blockquote><blockquote>''ArticleRevision.SalesArticles''</blockquote><blockquote>''ArticleRevision.Changers''</blockquote><blockquote>''ArticleRevision.DeclaredUnit''</blockquote><blockquote>''LogisticSolutionRev.LogisticSolution''</blockquote><blockquote>''LogisticSolutionRev.ExtraRetailLogSpec''</blockquote><blockquote>''LogisticSolutionRev.TransportLogSpec''</blockquote><blockquote>''LogisticSolutionRev.RetailLogSpec''</blockquote><blockquote>''LogisticSolutionRev.PalletLogSpec''</blockquote><blockquote>''Article.RegulatoryDomain''</blockquote><blockquote>''Article.SalesArticles''</blockquote><blockquote>''SalesArticle.MarketingCompany''</blockquote><blockquote>''MarketingCompany.Country''</blockquote><blockquote>''ContractManufacturer.CMValuesPerRegulatoryDomains''</blockquote><blockquote>''CMValuesPerRegulatoryDomain.AppliesTo''</blockquote><blockquote>''CMValuesPerRegulatoryDomain.OnlyInCountries''</blockquote><blockquote>''CMValuesPerRegulatoryDomain.ExtraRetailBarcodeFormat''</blockquote><blockquote>''CMValuesPerRegulatoryDomain.RetailBarcodeFormat''</blockquote><blockquote>''CMValuesPerRegulatoryDomain.TransportBarcodeFormat''</blockquote><blockquote>''CMValuesPerRegulatoryDomain.PalletBarcodeFormat''</blockquote><blockquote>''ShelfLife.ShelfLifeMarking''</blockquote><blockquote>''Structure to reach this level from ArticleChange''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ComponentDocRefs''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ComponentSpecificationRev.ComponentSpecification''</blockquote><blockquote>''ComponentDocRef.ComponentSpecificationRev''</blockquote><blockquote>''DocumentReference.DocumentType''</blockquote><blockquote>''Structure to reach this level from Components''</blockquote><blockquote>''ComponentUsage.UsedRevision''</blockquote><blockquote>''ComponentSpecificationRev.ComponentDocRefs''</blockquote><blockquote>''self''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''User''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ComponentSpecificationRev.ComponentSpecification''</blockquote><blockquote>''ComponentUsage.UsedRevision''</blockquote><blockquote>''ComponentUsage.ArticleRevision''</blockquote><blockquote>''ComponentUsage.ComponentSpecification''</blockquote><blockquote>''ArticleRevision.Article''</blockquote><blockquote>''ArticleRevision.ComponentUsageConstellations''</blockquote><blockquote>''ArticleChange.Project''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''Structure to reach this level from Components''</blockquote><blockquote>''ComponentUsageConstallations_PickListPresentation''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Structure to reach this level from self''</blockquote><blockquote>''ArticleRevision.ComponentUsageConstellations''</blockquote><blockquote>''Members_PickListPresentation''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Members''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ArticleChangeDetails''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''User''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ArticleChange.Project''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleChange.Article''</blockquote><blockquote>''ArticleRevision.Article''</blockquote><blockquote>''ArticleRevision.ActiveLogisticSolution''</blockquote><blockquote>''ArticleRevision.Changers''</blockquote><blockquote>''ArticleRevision.BatchCodePosition''</blockquote><blockquote>''ArticleRevision.ExpiryDatePosition''</blockquote><blockquote>''LogisticSolutionRev.LogisticSolution''</blockquote><blockquote>''Structure to reach this level from ArticleChange''</blockquote><blockquote>''DocumentTypesForFilter''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''ActiveLogisticSolution_PickListPresentation''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Structure to reach this level from ProjectPackagingView''</blockquote><blockquote>''ArticleChange.AffectedArticleRevision''</blockquote><blockquote>''ArticleRevision.LogisticSolutionRevs''</blockquote><blockquote>''BatchCodePosition''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''BatchCodePosition''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Structure to reach this level from ArticleChangeDetails''</blockquote><blockquote>''ExpiryDatePosition''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''ExpiryDatePosition''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Structure to reach this level from ArticleChangeDetails''</blockquote><blockquote>''DeclaredUnitPickList''</blockquote><blockquote>''AllInstances on:''</blockquote><blockquote>''DeclaredUnit''</blockquote><blockquote>''Members on this level per type:''</blockquote><blockquote>''Structure to reach this level from AffectedArticleRevision''</blockquote>
[[Category:Architecture]]