COMPLEX PROBLEMS, COMPLEX SOLUTIONS, COMPLEXITY ?

About a year ago I saw a presentation from a group of telco business analysts. One particular slide caught my eye which contained a classic statement “this seems to be a complex problem, we need to design a complex model for it”.  Right there and then the system engineering part inside me got a heart attack.  Recently I have seen LinkedIn blogs on “We will solve your complex problems”, or  “This is amazing software to solve your complex data problems “ etc. So I visited the companies, downloaded the software and read all the books, articles and comments. From a practical perspective I look back and I have the following questions and comments as a practitioner trying to optimize organization assets using data science as a core toolbox:

1. Why would you spend at least 10 000 euro’s per year on software that draws nice pictures but can be modeled through something like PostGIS/Postgre/MySQL and R?
2. Do we define solving complex problems as having the ability to process a MxN matrix with thousands of variables and find correlations from it ?
3. How can a piece of software help you to solve “complex” if you don’t understand what “complex” really means?
4. Is the saying true “a fool with a tool is still a fool?”
5. Shouldn’t software provide you with the ability to understand and solve “complex” problems in an open environment which is fed by global intelligence rather than single company intelligence?
6. Shouldn’t you start with the basic skills and understanding of data before trusting software solutions?
7. How will you know the software produces correct answers?
8. Can you interpret the results from these “magic tools” ?
9. If you can do 7 and 8, why do you need software that “does everything for you?”
10. If the vendor had a magic bullet then why do they sell software and not change the world with their own competitive weapons instead?

So when we deal with “Complex problems”, “Complex Solutions” and “Complexity” then I think one should master the following principles as practised in complexity management:
11. Have a standard work approach for data science so that you can understand how to select, prepare, analyse, model and visualise data.
12. Think in terms of multidimensional data-sets.
13. Expand your multidimensional thinking to include spatial data.
14. Expand spatial data to include geographical data, images and dynamic object movements.
15. Think spatial data as how it changes over time – temporal insight.
16. Multidisciplinary insight rather than functional insight solves complex problems.
17. A large MxN matrix with thousands of variables do not capture or solve “complex” – rather it reflects on the tool jockey not understanding insight into the problem required to be solved.
18. Kill MS Excel ideas in the data science domain.

This means that if we borrow from Bioinformatics and gene mapping analytics, we can use that knowledge to build data solutions which can assist in the identification of fraud patterns inside a large ERP system. Some of my favorite “business fractal images” shows how these multidimensional data sets can be used to find optimal trade densities from ERP and geospatial data – definitely not possible through a propriety system – but possible through multiple open-source collaboration efforts.

Is big data really changing things ?

As an industrial engineer I used to lack data for business models and optimization; suddenly things have changed and we can access huge amounts of data. But what is the impact of this ? A few examples of how things are changing:

Example - Marketing
No more sample base data for marketers; in one exercise we analyzed 30 million transactional events
of 300 000 customers and with the help of a non-supervised learning algorithm discovered three natural customers segments. This leaves everybody to discuss the output and not argue about what they think the customer does.

Example - Business Architecture
Using the same 30 million lines of code we were able to construct all possible customer process flows using process mining. No more mapping energy required but rather discussion time on patterns, trends and possibilities.

Example - BIS/GIS
Merging GIS data with customer data provides an immediate picture of where customers spend their
time. No need for corner modelling, but rather facilitating the integration of operations, supply chain, sales and marketing around the data.

Example Logistics/Fleet/Route Planning
 Use sample sets from 50 to 100 000 vehicles to understand what the travel time impact will be between any points in the supply chain network. No more guessing, optimal estimates can be derived.

With the help of my colleague Carmen van der Merwe, we have created an Integrated Design Framework for Value propositions which shortens product development as it integrates large design teams, but even more exiting about this is that Big Data directly impacts a number of key design areas which even should shorten this more drastically. The following picture shows where Big Data directly impacts on the design.