Techniques of Mind Control

https://upload.wikimedia.org/wikipedia/commons/c/cd/Brainwashing_3-with_hearts.jpg

Introduction

What all structures of power have in common is the problem to enable a minority to rule the majority. This is a very old problem, a classical one. Different cultures have addressed it in different ways. Some using techniques of brute-force domination, others adopting more elaborated methods of mind control. The realm of modern corporations has focussed on the latter approach, taking the most effective ideas from each and every culture. In this respect, modern structures of power are pretty similar across geographies. What are these techniques? It would be difficult to give an exhaustive account, because there are so many, and they keep changing. In this essay I will introduce some ever greens, which will give an overview of some of the methods used by powerful groups to exert their control over their subjects.

The first category of mind control techniques is Language Techniques. Examples of these techniques are: Manipulative Communication, Definition and Enforcement of Official Vocabulary, and Misuse of Metaphors.

The second category of mind control techniques is Cognitive Techniques, such as the use of Why/What/How questions, Partial Truths / Half-Truths, Misuse of KPIs, Information Overload, Misuse of Variable Compensation, Misuse of Gantt Charts, and Misuse of Time Management.

The third category is Emotional Techniques, such as Foment Perpetual Fear of Losing Job, The Superman CEO, Forbid Errors, Disconnection of Personal Contribution from the Outcome, Prevent Formation of Consent, Get Creative Contribution from Consultants, not Employees, Reference Letter/Work Certificate, and Misuse of Lifelong Certifications.

We will now start this inquiry into the world of scientific mind control with Language Techniques.

Language Techniques

There is a view in the literature, according to which the use of language affects formation of ideas, and influences behaviour:

“The principle of linguistic relativity holds that the structure of a language affects its speakers’ world view or cognition. Popularly known as the Sapir–Whorf hypothesis, or Whorfianism, the principle is often defined to include two versions. The strong version says that language determines thought, and that linguistic categories limit and determine cognitive categories, whereas the weak version says that linguistic categories and usage only influence thought and decisions.”

Source: (n.d). “Linguistic Relativity”. Retrieved January 4th, 2017, from https://en.wikipedia.org/wiki/Linguistic_relativity

Based on this principle, a number of manipulative techniques has been developed.

Manipulative Communication

Manipulative Communication is aimed at obtaining something from someone through insidious techniques of control. Manipulative Communication is done by using manipulative language. A good description of manipulative language is available at: http://www.clairenewton.co.za/my-articles/the-five-communication-styles.html

I reproduce an excerpt here, for convenience’s sake [accessed on 21 December 2016]
———————————————
The Manipulative Style
This style is scheming, calculating and shrewd. Manipulative communicators are skilled at influencing or controlling others to their own advantage. Their spoken words hide an underlying message, of which the other person may be totally unaware.

Behavioural Characteristics

Cunning
Controlling of others in an insidious way – for example, by sulking
Asking indirectly for needs to be met
Making others feel obliged or sorry for them.
Uses ‘artificial’ tears

Non-Verbal Behaviour

Voice – patronising, envious, ingratiating, often high pitch
Facial expression – Can put on the ‘hang dog” expression

Language

“You are so lucky to have those chocolates, I wish I had some. I can’t afford such expensive chocolates.”
“I didn’t have time to buy anything, so I had to wear this dress. I just hope I don’t look too awful in it.” (‘Fishing’ for a compliment).

People on the Receiving end Feel

Guilty
Frustrated
Angry, irritated or annoyed
Resentful
Others feel they never know where they stand with a manipulative person and are annoyed at constantly having to try to work out what is going on.

Source: The Anxiety and Phobia Workbook. 2nd edition. Edmund J Bourne. New Harbinger Publications, Inc. 1995.
———————————————
Manipulative communication may be used to obtain something from other people, through canning manipulation of their thought-formation processes . How can one detect manipulative language? Luckily, there are some indicators, some words which are usually reliable markers of a manipulative communication:

all, every, none, everyone, no one, always, never, best, worst, etc.

Examples:

“You always make silly mistakes”
“You never get it right at the first attempt”
“Are you sure this is the best solution?”
“This is the worst presentation I have ever seen”

If one wants to develop resilience against manipulative communication, the first thing to do is to learn to spot this manipulative words.

Definition and Enforcement of Official Vocabulary

Imposing use of acceptable vocabulary on others is an effective way of exerting influence on how things are perceived. One thing is saying: “I’ll give you this problem to solve, and you’ll have to complete it and resolve all impediments proactively. If the task is not done, you will pay the consequences.”; quite another is saying “I think you have the right skills for owning this challenge“. The apparent meaning is exactly the same, but the second form is malicious, because it hides the negative aspects behind a layer of shiny fresh paint. The first source of manipulation is done using “challenge” instead of “problem”. A problem exists independent of the observer. For example, finding a computationally efficient algorithm which determines if a given integer is a prime number or not. But if we call this a challenge, the fact that it can be solved is implicitly attributed to the ability of the resolver, not to the objective complexity of the problem. Calling problems “challenges” is a subtle way to put pressure on the person to whom the task is given. The second manipulation is done using the expression “to own a challenge”. The notion of ownership implies that the owner of an object can, among other things, do the following:

refuse to receive the object
sell the object
donate the object
destroy the object
dispose of the object
exchange the object with another one

Think about it, this is certainly true of a car, a house, a book, a pair of shoes, etc. Now the point is, can one really “exchange a task or assignment” with something else? (e.g. another task one likes best?). Can one decide to reject it? The answer is no. When a manager gives an assignment, the assignee is clearly not free to reject the assignment, if not once or twice, and then s/he is shown the door. Saying that someone who has been given a task is “owning a challenge” is a gross misrepresentation of what is going on.

Misuse of Metaphors

Metaphors are a very powerful type of figurative language. According to Merriam Webster (https://www.merriam-webster.com/dictionary/metaphor, accessed on 21 Dec 2016) a metaphor is:

a figure of speech in which a word or phrase literally denoting one kind of object or idea is used in place of another to suggest a likeness or analogy between them (as in drowning in money); broadly : figurative language

How are metaphors used to control people’s mind and behaviour? The method is based on implicitly attributing qualities to an object or notion, which it does not possess. This is achieved by replacing such an object/notion with a metaphor. The metaphor shares some qualities with the original object, but not all. The manipulation is done by tricking the audience into believing that the properties of the surrogate object/notion (the metaphor) also apply to the object/notion replaced by it. A few examples will clarify. The examples below are taken from article “Twenty business metaphors and what they mean by Tom Albrighton” (http://www.abccopywriting.com/2013/03/18/20-business-metaphors-and-what-they-mean, accessed on 21 Dec 2016)

(…)

Organisations = machines
This is another product of the Industrial Revolution, when businesses were characterised like the machines around which they were built. Applying the concept of mechanisation to human work led to things like organisational diagrams (=blueprints), departments (=components), job specifications (=functions) and so on.
All these concepts were productive in their way, but they obscured the reality of an organisation as a group of people. Unlike uniform components, people have very different abilities and aptitudes. And unlike machines, they can’t be turned up, dismantled or tinkered with at will.

(…)

Products = organisms
This metaphor is expressed in phrases like ‘product lifecycle’, ‘product families’, ‘next generation’ and so on. It draws a parallel between the evolution of living things and the way products are developed. This nicely captures the idea of gradual improvement through iteration, as well as the way in which products are ‘born’ (=introduced) and eventually die (=withdrawn) – always assuming they’re not killed off by the forces of ‘natural selection’ in the market.
One drawback of this metaphor might be the way it downplays human agency. Products and services don’t actually have an independent life, nor will they evolve independently. They are the result of our own ideas and decisions – reflections of ourselves, for better or worse.
Progress = elevation
This idea encompasses such well-worn phrases as ‘taking it to the next level’ and ‘onwards and upwards’. If we visualise this metaphor consciously at all, we might think of a lift going to the next floor, or perhaps someone ascending a flight of stairs. While ‘up’ is usually associated with ‘more’ and ‘better’ (think of growing up as a child, or line graphs), not everyone likes heights.
Careers = ladders
Thinking of careers as ladders embodies the same ‘up is good’ idea as ‘progress = elevation’. The higher up the ladder you go, the more you can see – and the more you can ‘look down on’ those below you.
Since ladders are one-person tools, there’s also an implication that you’re making this climb alone. So what will happen if others do try and join you? Will the ladder break, or fall?
More to the point, what if you reach the top and discover that your ladder was leaning against the wrong wall? Jumping across to another ladder is dangerous, so you might have to go all the way back down and start again.
A more useful metaphor in this context might be ‘careers = paths’. Paths fork, implying choice. Overtaking, or being overtaken, is no big deal. You can step off the path for a while, if you like – there might even be a bench to sit on. And while some might feel the need to ‘get ahead’, it’s clear that the most important thing is not how far you go, but whether you’ve chosen the right path for your destination.

(…)

To conclude,

Metaphors are meant to create an impact in the minds of readers. The aim of this literary tool is to convey a thought more forcefully than a plain statement would.

They are exaggerated expressions no doubt, but they are exaggerated because they are supposed to paint a vivid picture, or become a profound statement or saying.

source: http://examples.yourdictionary.com/metaphor-examples.html, accessed on 21 Dec 2016

Cognitive Techniques

Why/What/How

Most people are used to defining what are the things which are important to them. Some want to pursue higher education, others want to become rich, some want to do something good for the poor or the sick. Everyone has the chance to define what is her life strategy. I will call this kind of questions, the “why” questions. When there is a vision, and objectives are defined, one has to identify what need be done. For example, if one wants to pursue higher education she has to study hard, make sacrifices, define an area of specialisation based on her interests and abilities, and so on and so forth. I will refer to questions like these as the “what” questions. When it is clear what need be done, one has to define how each activity can be accomplished. Here it is necessary to define the details. See what is the available budget, what are the universities to target, what their admission criteria, etc. I will refer to questions like these as the “how” questions.

When one works for someone else, and that is the condition of employees, depending on her role, she has access to some kind of questions but, generally, not all. Those who define the “why” questions are the CEO and the Board of Directors. In so doing, they have to interpret the intentions and priorities given by the most important shareholders. One level down, directors and general managers usually define the “what” questions, and define what need be done to achieve the set objectives. Everyone else in the organisation, is actually dealing with “how” questions. Ordinary employees are overwhelmed by details and specialist work. The need to implement and execute “what” their directors have defined.

Dealing with people in a way that carefully filters out more interesting questions and focusses on the details of the things to do is yet another way of controlling their behaviour and ensuring they will not contribute creatively to the definition of the strategy. Their action will be constrained in well defined “tasks” which someone else has created for them. If they fail, such an outcome will be easily attributed to their action, not a wrong plan, because they do not even have visibility on anything else than their micro tasks. Ordinary employees, usually, do not have the elements to understand what is the value of what they are doing, because the ultimate objective is oftentimes shared only with a few people higher up in the organisation hierarchy.

There is a lot of rhetoric about being creative, proactive and so on and so forth. But a lot of organisations, do not really expect people to engage in “what” questions (let alone in “why” questions). All they want from their employees is that they get things done independent of the inefficiencies, politics, and oddities of their working conditions. This is what is really meant, oftentimes, by “being proactive”. The skeptics are invited to try propose a change which makes a process more efficient, or simplifies a procedure, or increases employee satisfaction. And then share the outcome of their proactive and creative behaviour.

Partial Truths / Half-Truths

Salami-technique

I learnt about this expression from a manager some jobs ago. He consciously used it to obtain things from employees, keeping them in a constant condition of dis-empowerment. The “salami-technique” consists in splitting the information necessary to understand a request and execute the tasks in small items, and only share what is strictly required to obtain the execution of tasks by individuals and teams, without giving them the context. The receivers of the “information slices” will know enough to accomplish their duties in a monkey-like fashion, without gaining any insight into what their are doing. Managers practicing the “salami-technique” pride themselves of being the only ones who see the full picture making themselves indispensable.

A variant of this technique is a misused form of the “need to do/need to know” principle. This principle is applied in security in order to reduce exposure to confidential data and reduce risk of disclosure. However, the same principle is sometimes misused to justify the practice of keeping people unnecessarily in the dark and giving them only the minimum information required to execute a task.

Information Funnel

This technique has similarities with the salami-technique illustrated above, but is systematically applied in a hierarchical fashion, reducing the information content shared, every level down the organisation. It is one way in which the why/what/how principle explained above is enforced.

Misuse of KPIs

Key Performance Indicators (KPI) may be useful in defining and measuring quantitative objectives. Several good books explain how to use them wisely. In reality, many actual uses of KPIs are misguided and aimed at manipulating teams into behaving in desired ways. The way this is done is through the exploitation of a cultural bias and equivocation. I have explained this in detail in my post The False Myth of Scientific Management, to which the interested reader can refer to. Here I will give a brief summary. KPIs are expressed in figures. Figures are associated with mathematics, which is the language of science. This association is intended to implicitly claim and evoke rigour and credibility. However, KPIs are not Science, and not even science. They are a technique. Different than observables of scientific laws, KPIs are not bound by mathematical laws which allow to describe and predict phenomena in a way which makes ti possible, for example, to know what is the precision of the estimate, say, the error function, the epsilon. Not being experimental laws, they cannot be disproved if wrong. You have to stay content with the strait faces of the holy priests of this technique, its true believers. Nothing else can substantiate the truth or significance of what is measured by defined KPIs. This is not to say the cannot be used wisely. There are many good ways of doing it described in the literature. Here, the point I want to make, is that a misguided use of KPIs (not KPIs per se) can be used to manipulate people.

Information Overload

Influential book “The Net Delusion: The Dark Side of Internet Freedom” by E. Morozov explains how totalitarian regimes which initially invested their energies in censorship, have soon learnt how to apply a cheaper and more effective technique: information overload. The traditional technique based on censorship is based on the creation and maintenance of a catalogue of forbidden content and resources. This approach was effective in the pre-Internet era, when content was produced and disseminated at lower rates, and this process happened by physical means (e.g. pamphlets, books) whose distribution and access could more easily be controlled. Nowadays, maintaining a black list of undesired content is pretty impossible. In order to prevent people from accessing content which could inspire them to bring about change, regimes soon found that a cheaper and more effective way was already available: flooding the Internet with gargabe content like entertainment sites, porn, etc. When one is overwhelmed by this content one is less prone to engage in discussions and debates on how to change the world.

The question arises, given the focus of this essay on corporate mind control, how does this method developed by regimes relate to the corporate agenda? It relates indirectly. Think about a motivated employee who thinks she can promote her ideas and bring about change in her organisation. What will a change-resistant organisation do? As we have seen above, it will no longer try to formally forbid this. Quite to the contrary, the organisation will pay lip service to innovation and invite proactive employees to check on the intranet what are the processes made available to submit proposals. This information will be buried in a sea of content, with primitive search functionality and the proactive employee will have to check hundreds of hits one by one manually, in a never-ending endeavour. Very soon more mundane tasks will take priority and divert this employee’s effort to tasks closely related to “how” questions (see above), and the time for proactivity will soon be over.

Misuse of Variable Compensation

There are very good reasons why a percentage of the total compensation can be variable. As I did above, I will not articulate here on the good uses of this practice. My focus is on how a malicious use of variable compensation is part of the tool set of the professional mind manipulator. When criteria for the recognition of variable compensation are set in a way which leaves room for interpretation, a manager can easily put the necessary pressure on her employees to obtain what she wants. Although employees are often advised to base their spend on fixed compensation and use variable compensation on non critical things or services, most people do otherwise. If the variable part of their compensation should be reduced, they may have very practical issues, like paying installments, a leasing, or failing to go on holiday. Giving a manager or a handful of individuals the power to decrease the salary of employees clearly gives them an extraordinary power to obtain exceptional performance. What is the problem with this technique of manipulation? Well, there are many. For example, some people focus on the objectives bound to the bonus, neglecting the rest. They become less collaborative and aim at achieving objectives in a formal way, not necessarily generating the expected value. Secondly, when people are treated like greedy individuals who only do something because otherwise they will get less salary, they will behave like ones. Appealing to the desires of the first order, like greed, is a sure way to make a good person behave like a fool.

Misuse of Gantt Charts

Gantt chart are a very popular representation of project plans. Despite the emergence and hype of agile methods, which are based on entirely different concepts, many enterprises still use Gantt charts because internal processes have been shaped over decades based on traditional project management practices. According to Wikipedia,

A Gantt chart is a type of bar chart, devised by Henry Gantt in the 1910s, that illustrates a project schedule. (…) One of the first major applications of Gantt charts was by the United States during World War I, at the instigation of General William Crozier.

source: “Gantt chart”. Retrieved January 14th, 2017, from https://en.wikipedia.org/wiki/Gantt_chart

Gantt charts can be very effective and are a powerful tool when it comes to planning activities like the ones for which they were first introduced. Clearly, planning the activities of the army and planning knowledge work is not exactly the same thing. Part of the equivocation is innocent in its nature: thinking that the advancement of tasks is proportional to the effort invested into it, and the time needed to complete them is inversely proportional to the resources allocated. This is probably true if one builds a wooden table. At least, to an extent. But if one is trying to find an efficient algorithm to solve a problem, this is usually not true. The professional manipulator transforms a problem to solve in a plan, and then describes the solution to the problem in terms of such a plan. Having transformed a problem in its representation, the manipulator feels legitimised to drag tasks here and there, and think (or want others to think) that the corresponding activities can also be completed sooner or later, correspondingly. However, this is blatantly false, as the nine women paradox explains very well:

Nine Women Can’t Make a Baby in a Month

Gantt chart true believers will counter this argument saying that “if all the dependencies are correctly modelled, and the tasks correctly classified (fixed work, fixed duration, fixed units, etc.), than the representation is very close to reality indeed”. However, real life experience, proves that this is seldom the case. And the reason is simple to understand. Modelling all such dependencies correctly is very difficult. One would require infinite knowledge of the problem, at the time the plan is written. But knowledge is not infinite. It’s partial. And with a model based, necessarily, on partial information one cannot expect total reliability. The good planner knows this and uses Gantt charts cum grano salis. The manipulator or the imbecile, firmly believe that what they see is what they will get and want others to believe the same. Sadly, this people contribute to making the statistics of successful projects what it is.

Misuse of Time Management

Let us start with defining what is Time Management:

Time management is the act or process of planning and exercising conscious control over the amount of time spent on specific activities, especially to increase effectiveness, efficiency or productivity. (…) The major themes arising from the literature on time management include the following:

Creating an environment conducive to effectiveness
Setting of priorities
Carrying out activity around prioritization.
The related process of reduction of time spent on non-priorities
Incentives to modify behavior to ensure compliance with time-related deadlines.

It is a meta-activity with the goal to maximize the overall benefit of a set of other activities within the boundary condition of a limited amount of time, as time itself cannot be managed because it is fixed.

source: “Time management”. Retrieved January 14th, 2017, from https://en.wikipedia.org/wiki/Time_management

While it is certainly true that, being time a finite resource, only a number of tasks can be done in any given finite period, that does not imply that people can be made to achieve more by switching tasks continuously based on a set of activities having ever-changing priorities. The human brain is simply not a CPU: context switch proves extraordinarily expensive as a mental process. When knowledge workers are focussed on their tasks and have maximum efficiency, they are in a so-called state of flow.

In positive psychology, flow, also known as the zone, is the mental state of operation in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity. In essence, flow is characterized by complete absorption in what one does.

source: “Flow (psychology)”. Retrieved January 14th, 2017, from https://en.wikipedia.org/wiki/Flow_(psychology)

Whenever a knowledge worker is interrupted, this state of flow ceases and it takes a certain amount of time to be re-established. The time required to get again into the state of flow following an interruption, is one of the reasons why misuse of time management actually reduces efficiency and efficacy, rather than increase them. And one can only undergo a number of cycles before her efficiency is completely spoiled for the whole day.

Professional manipulators use Time Management as an excuse to abuse their victims, so that they can interrupt them continuously whenever they need or want to have something immediately done at the expense of others. They will use a changed priority as a justification, forgetting or neglecting the fact that urgency and importance are two distinct and different things.

Emotional Techniques

Foment Perpetual Fear of Losing Job

In a job market shaped by the tenets of neo-liberalism, people are constantly compared with professionals from all around the world. Nobody is so good in her profession to be safe, when compared with professionals on a global scale. There is always someone who is better, cheaper, younger, has a better health and is more ambitious and driven to take anyone’s job. Companies restructure not only when they do bad but, increasingly, when they make good results. People are constantly confronted with cost cutting, digitization (see my essay The Dark Side of Digitization ), increasing use of Artificial Intelligence, etc. The effect on employees is that everyone feels replaceable. If one says no to a request, however irrational, someone else will immediately be ready to take her job. When forces like the above are used to intentionally foment fear of losing jobs, this is a mind control technique.

The Superman CEO

Years ago I was impressed to see a CEO call employees by their first name in occasion of a business party. He would address people like if he truly knew them and would ask questions like if he cared for what they were doing. That was an amazing thing. It had a great effect on me and I thought this was genuine. Years after, I met another CEO, who prided himself of having learnt Italian pretty fast and amazingly well. It was a remarkable achievement and employees went saying that they might not have been able to do the same. This CEO had an image like a super man. Employees started thinking, “there’s a reason why he is the CEO and I’m not, if he can learn a language in no time, God only knows what else he can do. He is cleverer than us, he is gifted.”

Year after year I have known a lot more CEOs and I have noticed that this is a pattern. While it may certainly be the case that most CEOs are indeed very gifted and smart individuals, probably well above the average, it is also true that some seem to have a clear urge to prove this alleged superiority with spectacular demonstrations like the above. When abilities are showcased in a very spectacular way, this is a form of manipulation and mind control. It is intended at creating the myth that the company is guided by extraordinary individuals and their decisions must just be trusted, even when not understood. The subliminal message is that there’s a reason why such decisions may seem irrational sometimes: it is that ordinary people like you and I can’t just understand them. The argument goes, “in the same way as we can’t remember the first name of all our colleagues (neither they really can, by the way) or learn another language so fast, we can’t just grasp the depth and full meaning of their decisions because we are not as smart as they are.”

Forbid Errors

Some organisations pursue practices which aim at forbidding errors. I remember a colleague managing a technical team shouting to them “this quarter you have to achieve the zero-ticket objective, understand!!!??”. Management practices like this have the objective to keep people focusing of the “how” questions (see above), and forget the “what” or “why” questions. When people cannot make a mistake they cannot innovate, cannot learn, cannot be truly proactive. All they can do is work like monkeys, day in and out.

Disconnection of Personal Contribution from Outcome

Human beings need to have feedback on their work. When a job is well done, normal people like this to be recognised. When negative feedback is received, one likes to be able to understand what went wrong, in order to be able to improve and do better next time. A mind control technique is based on the idea to make the connection between individual contribution and result opaque, so that people can no longer claim recognition for a job well done and can, at the same time, be blamed if something went wrong, even if they did not have anything to do with it. Tasks are defined in such a way that no individual contribution can easily or directly be linked to the overall outcome. Everything becomes “a team exercise”, with the implication that anyone in the team could have been replaced, and the same result still be achieved. The only way one can try to defend themselves from this, is to try to work on tasks whose deliverable has a clear value even when evaluated individually.

Prevent Formation of Consent (aka Divide and Conquer)

A family of manipulation techniques has the objective to prevent formation of consent. The reason is obvious: if a minority is to rule the majority, the most dangerous thing is when people gain awareness of having like interests and stakes in the situation. Because, if they do, they can join forces, and bring about change. But the ruling minority does not want this to happen. Therefore, it engages in a series of manipulation techniques which I will refer to as divide and conquer.

Competition

An example is fostering competition between individuals, teams, locations, etc. The alleged purpose is to stimulate healthy energies with the intention to let the best ideas emerge. The actual effect is oftentimes that people reduce or stop collaboration, pursue personal gratification instead of team objectives, try to claim ownership of achievements, cast a dark light on less talented individuals in order to shine better, and engage in a multitude of variations of rogue behaviour.

Forced Bell curves

This techniques is canny and sophisticated in the way it achieves deception. The idea is sometimes articulated along these lines. “Since there is statistical evidence that the performance of individuals in teams and organisations is distributed like a Bell curve, there is no reason, (true believers say), while the performance of your particular team should violate this scientific truth. Science in person tells us that performance has a Bell distribution: who are you, poor manager, to challenge this dogma?!”

This idea is plagued by many conceptual and pragmatic issues. The conceptual issue is that statistics is about big numbers. It may actually be true that in an organisation with 150’000 employees, performance overall is distributed like a Bell curve. But no mathematical law, and not even common sense, mandates that this be the case given any arbitrarily small subset of the overall sample space. In other words, if a manager has a team of six people, it is usually not the case that there has to be a complete idiot, two average individuals, two slightly smarter guys, and a genius. Enforcing Bell curves on arbitrarily small teams is like asking a player to use six dices: one with only ones, the second with only twos, … and the sixth one with only sixes, to make sure that using them one by one, the result will abide by the holy tenets of the Bernoully distribution. Only a madman would do that. But that is still a current management practice in a lot of companies nowadays. And people are tricked into believing it’s true, because it is allegedly based on statistics, which is assumed infallible like all branches of mathematics. But nobody should be allowed to claim mathematical truths without even grasping the basics. And if they do and still claim this technique is good, they are doing it maliciously, knowing to be tricking people.

Get Creative Contribution from Consultants, not Employees

An effective way to control individuals is to keep them constantly in a status of dis-empowerment. An instance of this technique is to give all creative work to consultants and treat employees like people who cannot possibly come out with innovative ideas. When people are treated like children, they will behave like ones.

Reference Letter/Work Certificate

A very common mind control technique is the practice which requires new employees to provide a reference letter from their previous employer. Knowing that your current employer will one day write a reference letter for you when you will leave (you want it or not), gives the employer an extraordinary power to limit the range of acceptable behaviour from you. If they ask you to work on a weekend and you would rather like to celebrate your daddy’s 80th birthday, they may write that you are not “flexible” or do not understand client priorities. And so on and so forth. In countries like Switzerland, the work certificate is particularly sophisticated because employers do not usually write negative statements. Instead, they omit to write positive statements. There exists a code, known by HR departments, which allows them to read between the lines in ways that no other reader can understand. For example, if they write an otherwise positive certificate, but omit the sentence “It is with regret that we received his resignations”, what they really mean is that that person may indeed be good and talented, but did not match their preferred behavioral cliche (maybe was too innovative or proactive), and they are not so displeased that she is leaving. On the receiving side, this apparently positive certificate will be interpreted as, “this is a smart guy, but she is going to create us headaches. Better get a less smart individual who stays quiet in her corner. If we should really need skills, we will take a consultant instead.”

Misuse of Lifelong Certifications

Another effective way to keep people quiet and focussed on the “how” questions, it to keep them busy learning some new technical tricks all the time. Getting certifications is a very good way of acquiring new skills and it is particularly useful in the current competitive job market. Certifications are usually required to technical specialists, less commonly to managers, especially senior managers.

When certifications are bound to promotions and professional models, there can be at least two effects. The first one is that there is more transparency in the promotion criteria, and this is clearly positive. The second one is that a manipulative manager can keep raising the bar, so that technical specialists are constantly in a condition of artificial deficiency. They will always lack a certification or two to be promoted, or even to keep their current band. To make things worse, modern certifications expire, and individuals have the burden to keep renewing them all the time. In addition to day job, people are constantly required to refresh their quickly obsolescent technical skills, and re-certify. These certifications are demanding and take a lot of time, usually in the evenings or weekends. When people are constantly busy certifying and proving themselves, they will not have the mindset, or simply the time, to seek new ways to change the world.

Conclusions

In this essay I have introduced the topic of corporate mind control. This is an evolving set of techniques used by a minority to control the majority. These techniques are aimed at limiting the set of acceptable behaviour, neutralise genuinely innovative ideas, disincentivise proactivity while claiming the contrary, and optimise profitability or other objectives through domination. These techniques are used by some corporations in various degrees. The range is from almost innocuous to very unethical. There is no such a thing as a corporate ranking in this regard. Or, at least, not one that I am aware of. The reason why I have researched this topic is that I argue that these techniques are unethical, because they violate the Kantian principle:

“Act in such a way that you treat humanity, whether in your own person or in the person of any other, never merely as a means to an end, but always at the same time as an end.”
— Immanuel Kant, Grounding for the Metaphysics of Morals

What this principle really means is subject to philosophical inquiry (see, for example http://philosophy.fas.nyu.edu/docs/IO/1881/scanlon.pdf). However, for the sake of my argument, we need not pursue sophisticated philosophical research. We only need to understand that human beings have priorities, intentions, ambitions, values, emotions, potential, limits, a need for social interactions, a sense of justice, self-esteem, etc. Therefore, they are very different from other “resources” utilised by corporations to produce goods or offer services. Mind control techniques are aimed at limiting the expression of the natural endowment of people, so that they pursue exclusively what is required to them to achieve goals and accomplish tasks defined by someone else. Extreme uses of these techniques deprive people of their dignity, because they are not treated also as ends, but only as a means to an end.

The techniques described in this essay have been developed over time and are utilised by all sorts of organisations, ranging from private enterprises, to political parties and regimes. This latter kind of organisation excels in use of media control, which is a subject not covered in this essay, because it has less relevance to private corporations. This is a separate topic very well covered, for example in:

Media Control, Second Edition: The Spectacular Achievements of Propaganda (Open Media Series), Sep 3, 2002, by Noam Chomsky

I hope to have stimulated some constructive meditations on this important topic, with this admittedly peculiar essay, and I will be keen on receiving feedback. Thank you for reading.

References

T. Albrighton, “Twenty business metaphors and what they mean”. Retrieved December 21st, 2016, from http://www.abccopywriting.com/2013/03/18/20-business-metaphors-and-what-they-mean

N. Chomsky, 2002, “Media Control, Second Edition: The Spectacular Achievements of Propaganda” (Open Media Series)

I. Kant, “Grounding for the Metaphysics of Morals”

E. Morozov, 2012, “The Net Delusion: The Dark Side of Internet Freedom”

C. Newton, “The Five Communication Styles”. Retrieved December 21st, 2016, from

http://www.clairenewton.co.za/my-articles/the-five-communication-styles.html, accessed on 21 December 2016

(n.d). “Flow (psychology)”. Retrieved January 14th, 2017, from https://en.wikipedia.org/wiki/Flow_(psychology)

(n.d). “Gantt chart”. Retrieved January 14th, 2017, from https://en.wikipedia.org/wiki/Gantt_chart

(n.d). “Linguistic Relativity”. Retrieved January 4th, 2017, from https://en.wikipedia.org/wiki/Linguistic_relativity

(n.d). “Metaphor”. Retrieved December 21st, 2016, from https://www.merriam-webster.com/dictionary/metaphor

(n.d). “Metaphor Examples”. Retrieved December 21st, 2016, from http://examples.yourdictionary.com/metaphor-examples.html

(n.d). “Time management”. Retrieved January 14th, 2017, from https://en.wikipedia.org/wiki/Time_management

Type-Parametric Segment Trees In Scala

stree_small

In my post Functional Solution to Range Minimum Query (RMQ) using Segment Trees in Scala I explained how to efficiently solve the problem of finding the minimum element of an arbitrary sub-vector of integers. In that article I used a Segment Tree in which each node and leaf has an integer value, which is the minimum of the sub-vector defined by the corresponding range. In this article, I will explain how to use Segment Trees to solve a more general class of problems. Before introducing the solution, we need to answer the question, what are the functions, other the min:Int⁺ -> Int, which can be computed efficiently using Segment Trees? Intuitively, some other functions are max(), sum(), avg(), probability(), etc. What do these functions have in common? It is the fact that they may be computed over each element in a given range or, more efficiently, using intermediate results, like their value computed over defined sub-ranges. In my article about RMQ we have seen that an efficient solution to the range minimum problem is based on a Segment Tree which stores the minimum of certain sub-ranges of the given integer vector. Using this technique, instead of iterating over the whole given sub-vector of n elements it is possible to execute only log₂(n) computation steps. The same applies to max(), sum(), avg(), probability() computed over a range, and other similar functions, when their computation is optimised using Segment Trees.

In general, the functions we are targeting with Type-Parametric Segment Trees can be defined like this.

Definition
Let v be a Vector of n elements of type X, say, val v: Vector[X], and f a function f:X⁺-> V. Function f can be solved efficiently using a Type-Parametric Segment Tree if and only if:
∃ g: (V x V) -> V, such that
∀n >= 2, ∀i ∈ {0, n-2} f(x₀, x₁, …, x_n-1) = g(f(x₀, x₁, …, x_i), f(x_i+1, …, x_n-1)) (1)
♦

Example: min()
Function min() clearly satisfies the definition above because the minimum of a set of elements can also be determined as the minimum of the minimum values computed over any two partitions of the same set.

Example: avg()
Let:
1) n ∈ Integer, such that n > 0;
2) val a = Vector[Integer], with a.length == n;
3) i, j ∈ Integer such that i, j >= 1 and i+j == n
4) avg_n() the average computed over all n elements of vector a
5) avg_i() the average computed over the first i elements of vector a
6) avg_j() the average computed over the last j elements of vector a

We define our function g as follows:
7) g(avg_i(), avg_j()) = (i/n)(avg_i()) + (j/n)(avg_j())

Optional proof (only for the interested reader)
Proposition 1: avg_n() = (i/n)avg_i()+(j/n)avg_j()
By definition of avg we have:
8) avg_n() = (1/n)∑x_n
Decomposing the sum:
9) (1/n)∑x_n = (1/n)(∑x_i+∑x_j)
Using the distributive property:
10) (1/n)(∑x_i+∑x_j) = (1/n)∑x_i + (1/n) ∑x_j
But 1 = (i/i), therefore:
11) (1/n)∑x_i = (i/n) (1/i) ∑x_i
By definition of avg we have:
12) (1/i) ∑x_i = avg_i()
Combining 11) and 12) we now have:
13) (1/n)∑x_i = (i/n)avg_i()
In the same exact way it can easily be proved that:
14) (1/n)∑x_j = (j/n)avg_j()
Combining 10), 13) and 14) we have:
15) avg_n() = (i/n)avg_i()+(j/n)avg_j()
This ends the proof.
♦

We see from definition 7) that avg of n elements can be expressed in terms of its value on any two splits of the vector. To make the concept clearer, we will now see an example in Scala. First, we will see how the SegmentTree data structure can be redesigned in order to become type-parametric. For convenience’s sake, I will use images in this article, but the interested reader may find source code in github:
Type-parametric Segment Tree
Example with function AVG

The first thing to notice is the creation of dedicated class Range. A range class is already available in Scala, but its intended use is different from what we need for our SegmentTree. The below is a very light-weight implementation which allows to perform useful operations on ranges like, for example, intersection.

classrange
This is the implementation of the Segment Tree. The value of each node or leaf of a segment tree is parametric (class V).

If is worth noticing that even the function f:(V,V)=>V, which computes value V based on the partial results contained in the left and right sub-trees is also parametric. As we will see, this is required to achieve the desired degree of flexibility.
classsegmenttree
The build method of singleton object SegmentTree now contains two functions. Function xToV:(X)=>V takes a value from the original vector and builds the value of its corresponding leaf in the associated SegmentTree. Function fun:(V,V)=>V allows to compute value of a node, based on the partial results contained in its left and right sub-trees.
objectsegmenttree
In order to utilise the parametric segment tree to efficiently compute the average AVG over any given sub-vector of a given vector of integers, the following definitions are required:

class Value which contains the average over a given sub-vector, and the length of such a sub-vector. We have seen above that this length is used by function fun:(V,V)=>V
Method intToValue which, given an integer, builds a value having as avg the conversion to Float of the integer, and length equal to one.
Method fun:(V,V)=>V, see proposition above.

If we now use the test application ATAvg with the following input:

10 5
10 20 30 40 11 22 33 44 15 5
0 5
1 2
8 9
0 9
4 6

We obtain the following output:

It is easy to verify that it is correct.

Remark
The first row of the input contains the size of the vector, and the number of queries. The second row is the integer vector. Each subsequent row is a query (start and end index of the sub-vector). The output contains the avg value computed over each sub-vector in the query list.

This ends the explanation of the parametric segment tree. The interested reader may find the mathematics of segment trees in my previous article:
Functional Solution to Range Minimum Query (RMQ) using Segment Trees in Scala

If you find other interesting uses of this data structure to solve computation problems, I will be happy to receive your feedback. Thank you for the read, and Up Scala!!!
♦

The Dark Side of Digitization

[https://commons.wikimedia.org/wiki/File:Eclipse20070304-2.JPG]

If we were to base our judgement of digitization on certain accounts, the future would look bright. The long promised efficiency which is expected from technology will eventually be achieved. No more tedious manual work will be necessary. People will be able to “focus on more interesting problems” and, as the story goes, the error rate of diagnoses, fraud prevention/detection and defense systems would soon become negligible. To those who are inclined to believe in fairy tales, this perfect picture of a super-efficient, technology-driven, future society will look great, won’t it? Yet, I think, there is a much overlooked dark side of the coin. It is the long-term social effects of replacing human work with automatons. Generally, technical solutions are desirable when they tackle problems in a way which is beneficial to the highest number of people. Is digitization one such case? It depends on how it is approached. I contend that the current hype may in some cases betray a number of dangerous misconceptions:

Misconception 1. Reducing manual work is always beneficial

Work is, among other things, vocation, occupation and profession. At work people socialize, build relations, nurture friendships. Right or wrong, western culture has elevated professional activity to the rank of a personality-defining aspect. When people will work less, (or not at all), something else will have to fill this gap. Personally I see this as beneficial. I have always perceived with suspicion the atomistic myth of professional success at the expense of a miserable social life. But changing this will require time and, if not properly managed, can lead to unrest. Are we prepared to manage the transition? Are our omniscient Artificial Intelligence robots giving us any insight on how to approach this problem? I don’t think so. And then, how will the future consumeristic economy be fuelled with an increasing number of unemployed people? Who will buy the goods which will be produced so efficiently? Who will be able to pay for these services? A new society need be shaped, before the much wanted digitization can be made sense of. The rest is only a lot of talk, solutionism and technology-centrism, the myth that technology is the answer to any problem.

Misconception 2. Artificial Intelligence will replace Human Intelligence to an ever increasing extent

Artificial Intelligence will enable new computing scenarios which will complement human intellect in many ways. This is particularly true in the context of cognitive intelligence. And this is indeed an exciting scenario. But mentation is a lot more than cognition. And only someone with a very limited understanding of what is a human being, can indeed think that a computable function can be a replacement for the wealth of psychological treats and social behaviours which are associated with the spontaneous action of people. The idea that mentation is a computable function has further socially destabilising consequences. Let us assume, for the sake of argument, that Artificial Intelligence fanatics are right. So, human mentation is a computable function. Functions are deterministic by design: given an input, they return an output. But most social contracts have been built over thousands years on the assumption that there is such a thing as personal accountability for our actions. And this can only exist if one has free will. But how can one make a free decision if mentation is nothing more than a computable function? A serial killer would be just someone whose mentation implements the wrong computable function. How could he be accountable for producing outputs which could not have been otherwise? If we think we are ready to accept the weird (yet, admittedly, theoretically possible) idea that mentation is a computable function, than we must also be prepared to renounce the idea of personal accountability, and delete the concept of free will from our dictionaries.

Misconception 3. The efficiency of digitization is universally beneficial

This efficiency is mostly beneficial to a few people, who can use digitization to optimise their businesses and increase profitability. But these people are a only tiny minority of those affected by it. The vast majority of people will be personally affected for the worse. Some say that more interesting jobs will be created. But this is a mystification. Let’s get real: in the past, the most relevant corporations, like General Motors, employed hundreds thousands people. Nowadays, corporations like Google and Apple make more money and employ a much smaller number of people. And not everyone is “up to” the level required to be employed by such companies. What this will amount to, I need not tell you. You do the math.

I advance that these misconceptions, and possibly more, are plaguing the current over-hyped idea of digitization and, as such, will prevent ordinary people from reaping benefits they can make sense of. One may object that it is easy to criticise ideas, but it is more difficult to devise alternatives. I accept the objection. My answer to this is articulated in my essay “Post-Anthropomorphic Sensorial Computing” which the interested reader will find at this address:

https://alaraph.com/2015/03/10/post-anthropomorphic-sensorial-computing/

Should then people resist digitization altogether? Simply put, no. But oversimplifications may kill the idea and make it difficult to get digitization right. Good digitization is about empowering individuals, not getting rid of them. Claiming that a computer can “outperform” a lawyer or a doctor is a sensationalist claim. But ask the bold defenders of it what they would do when seriously in trouble or sick; would they really put their lives in the cold hands of a robot? A more balanced way to put the question would be to contend that professionals equipped with state-of-the-art fast data advisors, would make a better job. In the end, however greedy a single human being may be, Homo sapiens are, overall, ethical animals. Should we give up all this in favour of a merciless computer program which will always pursue the interests of its owners? The decision is yours.

Lessons Learned from Medical Testing

lessons-learned-from-med-testing-01

[Wikimedia, 2015]

Application Testing is about determining whether or not a given software “behaves” as expected by its sponsors and end users. These stakeholders have a legitimate right to their expectations, because the system should have been engineered with the objective to satisfy their requirements. The diagram below represents the testing process as a black-box: In consideration of the above, it is easy to see that Application Testing is a particular instance of a more general mathematical problem: Hypothesis Testing. As such, it can be tuned to maximise its reliability either in terms of false positives or false negatives, but not both.

False Negative: this is the error in which one incurs when a buggy application wrongly passes a test.
False Positive: this is the error in which one incurs when a correct application wrongly fails a test.

This is a very well-known problem, and it finds application in other domains, including statistics and medicine.

lessons-learned-from-med-testing-03

Adapted from [Wikimedia, 2015]

Let us consider, for instance, the case of medical tests. How did the scientific community manage to strike the right balance between sensitivity and reliability? In order to answer this question, one has to answer a preliminary one: what is worse, wrongly detecting an illness or failing to detect it? Clearly, the latter case is the worst. Failing to detect contagion can result in rapid spread of infectious deseases, loss of lives, and high healthcare costs. For this reason, medical tests are designed to minimise false negatives. What is the downside? It is clearly the cost of managing false positives. Whenever a critical infection is detected, medical tests are repeated in order to minimise the chance of an error.

Now, let us come back to our domain, Application Testing. What is the worst scenario, false positives or false negatives? If an application fails a test, but it is actually ok, costs originate because the application needs be investigated by software development and, sometimes, business users. But that is the end of it. Conversely, if an application has a severe problem which remains undetected until deployment to production, this is an entirely different story, and can result in a full range of critical consequences, including compliance breach, financial loss, reputational damage, and so on and so forth. So, in the financial sector, just like in medicine, testing must be optimised in order to minimise the chance of false negatives.

A question naturally arises: how do we fare in this regard? Where are we? Luckily, I believe, in the financial sector we are doing alright. Application Testing in this sector is clearly aligned with this risk management strategy. However, there is still room for improvement. The fact is that, as we have seen above, false positives also have a cost. Do we have to be prepared to pay it in its entirety, or can we do something to reduce it? To go on with the parallel with medicine, I shall argue that false positives, like cholesterol, come in two flavours: the good ones and the bad ones. The good ones originate because of unavoidable causes, like technical glitches, human error in test case design or execution, or even up the software life cycle, like wrong requirements or wrong understanding of user requirements. These errors are part of the game, and can only be mitigated up to an extent.

But there is another category of false positives which I think is bad, and can be reduced. Those are the ones generated because of lack of domain expertise. Techies will base their judgement on the evidence collected during test execution. But this evidence is oftentimes not self-explanatory: interpretation is required. And this interpretation can only go as far as the business domain experience of the tester. Enterprise-wide application landscapes implement very sophisticated workflows and support complex business scenarios. The behaviour of these applications changes depending on user rights, user role, client type, and many more criteria. What is actually the intended behaviour of an application, can easily be mistaken for an error.

So, what is my pragmatic recipe to cut down bad cholesterol in application testing (reduce the cost of false positives)? I believe the answer to this question is closely related to the evolution of the testing profession. Nowadays, a good test engineer is someone with technical skills, and business domain knowledge. This unique blend of skills is certainly precious and makes application testing a science and an art at the same time. But new trends are changing this. Automation is increasingly being pursued because of cost pressure, and of the need for increased agility and reliability. But automation comes with skills challenges of its own, to the effect that, as it is generally recognised, more sw engineering savvy personnel will be required in testing. And this is good. But, at the same time, once the amount of manual activity will be reduced, a new opportunity will exist to inject more business-savvy personnel in testing. The transition, the way I see it, can be represented like this:

To sum up, more automation will pursue optimisation in terms of minimisation of false negatives, whereas business domain expertise will reduce the costs of false positives. This evolution of the testing profession in specialist roles is what is required to apply the lessons learned from medical testing to application testing in the financial sector. I will be happy to receive your feedback on this admittedly unconventional view on the future of the testing profession.

References Wikimedia, 2015, https://commons.wikimedia.org/wiki/File:Infant_Body_Composition_Assessment.jpg, accessed on 24.07.2015 Morelli M, 2013, https://alaraph.com/2013/09/26/the-not-so-simple-world-of-user-requirements/, accessed on 07.08.2015

Test Automation and the Data Bottleneck

[Wikimedia]

Introduction

The topic of automation has been revamped in the financial industry following the recent hype on industrialization of IT and lean banking. The rationale of the idea is that there are tasks which are better taken care of by automatons than by humans. This kind of tasks are composed by actions which are executed iteratively, and the quality of their outcomes can be negatively affected by even the smallest operational mistake. An interesting proportion of tests belong to this category. Not all of them, of course, because human intellect can still excel in more creative testing endeavours, like exploratory testing, to give just an example. But other kinds of tests, like regression-tests, would make a very good candidate for automation. Practitioners and managers know all too well that a well-rounded battery of regression tests can indeed prevent defects from being introduced in a new release with demonstrable positive effects on quality. But they also know that manual regression testing is inefficient, error prone, and expensive. Therefore, awareness is raising on the necessity to pursue an increase in the degree of test automation. In this essay I will argue that, before thinking about tooling, solutions, and blueprints, there is a key success factor that must be addressed: avoidance of the data bottleneck. I will first explain what it is, and why it can jeopardize even the most promising automation exercise. After that, I will introduce an architecture which can tackle this issue, and I will show that this approach will bring additional advantages along the way. I will now start the exposition introducing the topic of the data bottleneck.

The Data Bottleneck
If we make an abstraction exercise, we can see testing as a finite state automaton. We start from a state {s₁} and after executing a test case TC₁ we leave the system in state {s₂}. A test case is a transition in our conceptual finite state diagram.

In order to be able to execute a test case, the initial state {s₁} must satisfy some pre-conditions. When the test case is executed, ending state {s₂} may or may not satisfy the post conditions. In the former case we say that the test case has succeeded; in the latter case we say that it has failed. Now, what does this have to do with automation? An example will clarify it. Let us consider the case of a credit request submitted by a client of a certain kind (e.g. private client, female). The pre-conditions of the test case require that no open credit request can exist for a given client when a new request is submitted. From the diagram above we see that there is no transition between states {s₂} and {s₁}. What does it mean? It means that business workflows are not engineered to be reversible. If the test case creates a credit request and it fails, there is no way to execute it again, because no new business case can be created in the application for this client, until the open request is cancelled. Now, there are cases in which the application can actually execute actions which recover a previous state. But in the majority of cases, this is not possible. In banking, logical data deletion is used instead of physical deletion. Actions are saved in history tables recording a timestamp and the identity of the user, for future reference by auditors. In cases like this, the initial state of a test case cannot be automatically recovered. Sometimes, not even manually. What one would need, is a full database recovery to an initial state where all test cases can be re-executed. This is the only way; other approaches to data recovery are not viable because of the way applications are designed and of applicable legislation.
Above we have seen that data recovery is a key pre-condition for automation. Now we will see why legacy environments are oftentimes an impediment to tackling this issue efficiently. Oftentimes, business data of a financial institution is stored in a mainframe database. And when it is not in a mainframe, the odds are, it is in an enterprise-class database, such as Oracle. What do an Oracle database on Unix/Linux, and a DB2 on a mainframe have in common? Technology-wise very little. Cost-wise, a lot: neither one comes for cheap. The practical implication is that database environments made available for testing are only a few, and they must be shared among testing teams. This makes it impracticable to make available automatic database recovery procedures, because of the synchronization and coordination required. What happens in reality is that test engineers have to carefully prepare their test data, hoping that no interference from their colleagues will affect their test plan. And what is worst, after they are finished with their tests, the data is no longer in a condition suitable for re-execution of the same battery of test cases. Another round of manual data preparation is required.
One may wonder if it is indeed impossible to reduce the degree of manual activity involved. The point is that so long as access to databases is mediated by applications, and applications obey by the business workflow rules (and applicable legislation), recoverability of data is not an option. Are we indeed stuck? Isn’t it possible to achieve automatic data recovery without breaking the secure data access architecture? My contention is that there is indeed a viable solution to this problem. The solution is outlined in the following section.

Proposed Solution: On-Demand Synthetic Test Environments
Automated tests take place in synthetic environments, that is, environments where no client data is available in clear. Therefore, the focus of this solution will be on these environments, which are the relevant ones when it comes to issues of efficiency, cost-optimisation, regressions and, ultimately, quality.
The safest way to recover a database to a desired consistent state is using snapshots. A full snapshot of a database in a consistent state is taken and this “golden image” is kept as the desired initial state of a battery of automated tests. Using the finite state representation, we can describe this concept in the following way:

The diagram shows that any time a battery of automated test cases terminates, it can be executed again and again, just by recovering the initial desired state. To be more precise, the recovery procedure can take place not only at the end of the test battery, but at any desired intermediate state. This is particularly useful when a test case fails, and the battery must be re-executed after a fix is released. The diagram can be amended like this:

So, we have solved the problem of recovering the database to a desired consistent state which enables automatic (and manual) re-execution of test-cases. Is this all? What if other test engineers are also working on the same database environment? What would be the effect on their test case executions if someone else, inadvertently, swept their data away through the execution of an automatic database recovery procedure? It would simply be catastrophic, to say the least. This would bring about major disruption. How to fix this problem? What is needed is a kind of “sandboxing”: environments should be allocated so that only authorised personnel can run their test cases against the database, and no one else. Only the owner of one such environment should be in a position to order the execution of an automatic database recovery procedure. How can this be achieved? An effective way to do it is by offering on-demand test environments which can be allocated temporarily to a requestor. This sounds very much like private cloud. The below are the key attributes of an ideal solution to the on-demand test environment problem:

Test environments shall be self-contained.
Applications, data and interfaces shall be deployed as a seamlessly working unit

Allocation of test environments shall be done using a standard IT request workflow.
For example, opening a ticket in ServiceNow or what have you

Test environments are allocated for a limited period of time.
After the allocation expires, servers are de-allocated. After a configurable interval, data is destroyed.

During the whole duration of the environment reservation, an automatic database recovery procedure shall be offered.
This procedure may be executed by IT support whenever a request is submitted using a standard ticket. An internal Operational Level Agreement shall be defined. For example, full-database recovery is executed within a business day of the request.

The TCO of the solution shall be predictable and flat with respect to the number of environments allocated.
Traditional virtual environments are allocated indefinitely and can easily become expensive IT zombies. Zombie environments still consume licenses, storage and other computing resources. Conversely, the solution proposed prevents these zombie environments from originating in the first place.

The logical representation of the proposed solution infrastructure is the following.

To sum up, these are the advantages of the proposed approach:

It enables full test automation, making it truly possible to re-execute batteries of test cases using an automatic database recovery procedure in sand-boxed database instances.
It gives 100% control of TCO and allows to keep testing spend on IT within defined limits.
It allows to attribute testing spend to projects with high precision.
It increases overall quality of testing results
It eliminates cases of interference among independent test runs.
It allows to anticipate involvement of test teams in the software life cycle.
It saves infrastructure costs because computing clouds allow for transparent workload distribution, with the effect of running more (virtual) servers on the same physical infrastructure.

Conclusions

In this essay I have articulated the data bottleneck problem relating to test automation. First, I have given a general introduction to the topic. Second, I have explained why this problem may put in jeopardy test automation initiatives. Last, I have proposed a solution based on the concept of on-demand test automation environments and I have shown why I believe this is the way forward. The interested reader can contact me for sharing feedback or delving deeper in the discussion.

References

Wikimedia, 2015, https://commons.wikimedia.org/wiki/File:Buckeye_automatic_governor_%28New_Catechism_of_the_Steam_Engine,_1904%29.jpg, accessed 1 June 2015

Post-Anthropomorphic Sensorial Computing

I sense, therefore I am

Wikimedia[1]

Evolution of Computing

For decades, evolution of computing has been framed in terms of increase in processor performance, random access memory, and storage capacity. Millions of instructions per second has been a primary criterion for comparison. Moore’s Law successfully predicted the doubling of this performance indicator every eighteen months and, when processor vendors approached the physical limits of this growth, multi-core architectures were introduced, so that what could no longer be achieved with vertical scalability, could theoretically be obtained through horizontal scalability. The reason for this obsession with computing power was that a lot of interesting problems which admit of an algorithmic solution could not be efficiently treated with then-current technology. Nowadays, we have reached a point where further advances can only be pursued if software architecture allows for a high degree of parallelism. However, further increases in the number of cores per CPU will not automatically translate in improved performance, unless computer programs are designed to run intensive computations in separate threads which can be executed on different cores. But even software architecture has its own limits. Such limits are imposed by mathematics: parallelisation cannot be pushed beyond defined thresholds[2]. The good news is that with current technology, we can probably already solve most of all that mathematical complexity allows. Computer programs can interpret human voice more than decently. Human face recognition is a reality. 3D virtual reality is there to see. And the list could go on indefinitely. So what is the next evolutionary step on the horizon? There has been a lot of talk around mobile, big data, internet of things, wearables, and social. Is this all?

Emergence of a new Trend?

Social networks would not be so successful if mobile devices were not a reality. Try see people on a commuter train every day. They are hooked on their social networks accessed with a smart phone. If online access were limited to desktop internet surfing in the evening, interaction would drop dramatically. Big data became a hot topic with the explosion of computing devices feeding databases and analytical systems with an unprecedented amount of data, structured and unstructured. If social and IoT were not that ubiquitous, big data would probably not be the same hot topic that we all know. The point is that these trends are closely intertwined and dependencies among them exist, which can reinforce or inhibit certain evolutionary paths. The central contention of this essay is that IoT, mobile and wearables are giving rise to a new shift, which cannot be reduced to the joint use of these trends alone. I will call this new trend “Post-Anthropomorphic Sensorial Computing” (PASC).

Post-Anthropomorphic Sensorial Computing

First attempts to augment computing machinery with sensorial capabilities were inspired by the human body. Computers were made too “see”, “hear”, and “speak”. Then came robots, which can also move in 3D. Some even believed (or believe), that computers could one day be made to think. This of course reflects a very poor idea of mentation, because everyone who has had a normal interaction with other human beings knows all too well that rational thinking is only a small, yet critical, part of psychological processes of a healthy human being. I contend that the debate whether or not machines can be made to think like a human being is moot. The interesting point is that machines are increasingly endowed with a number of sensors which allow them to read the environment in ways which complement human senses. This is made possible by IoT, wearables, and mobile, but not only that. Try think when, after using a maps application on a tablet, one tries using the same application on a laptop, only to find out that the laptop does not know its current position, which has to be patiently typed the old way. Tablets which were once seen as the poor replacement for more powerful computing devices, can now do things which these very devices cannot do, no matter how powerful their processors are. These wonders are made possible by sensors. A GPS sensor does only make sense on a mobile device. A camera can very well be put on a desktop/laptop, but all it will shoot is the bodily features of the person in front of it. Still something, but nothing compared to what I can picture or film with my iPhone. Try use a password management application on a tablet. The master password can be as user friendly as a fingerprint. Touch the fingerprint sensor and you can access the secure vault without the pain of entering improbable combinations of characters. Thanks God. I would not like to type again and again passwords when I will be in my seventies. It would make for an unsurmountable barrier. Now try do the same on a desktop. It looks like a step one-hundred years back on the time machine. It is clear that it is sensors which will create value add in ubiquitous computing. But now that the utopian dream of replicating humans with automatons has shown its pathetic metaphysical overlook, it is time to think about Post-Anthropomorphic Sensorial Computing. This new wave of computing artefacts will be enriched with thermometers, infra-red cameras, multi-axis motion sensors, radioactivity sensors, earthquake detection sensors, accelerometers, and so on and so forth. They will complement human senses in ways that would not have been possible if we had kept focusing on artificial intelligence alone. Imagine having a wearable ultrasonic sound location system, the equivalent of a pocketable bat, which can aid the blind move safely in an unknown environment. Imagine having radioactivity sensors and PH sensors in the Oceans, in order to measure the effects of climate change and natural disasters in real time. One day, closer than one may think, we will be able to “feel” if our loved relative or friend is having a hard time, and prevent the worst to happen thousands of kilometres away, maybe only with a good-old reassuring phone call. Because PASC is not about creating fancy gadgets, it is about extending human faculties smoothly, with a human-centric viewpoint. For PASC to succeed, existing disciplines will have to further advance. For example, emergence of PASC will pose new challenges for big data. IoT will become a lot more than a community of refrigerators with an Ethernet card. The boundary between IoT and mobile will blur. PASC is about renouncing the mad scientist dream of replicating mentation on a silicon chip, and finding ways to expand and complement humankind‘s sensorial power with a new generation of ubiquitous, energy efficient, sensorially enriched computing artefacts. The challenge of PASC will be to find ways to translate this wealth of sensorial data in useful information for a human being. In their current evolutionary stage, humans are still endowed with five senses, plus other interesting abilities, like balance. Extending the reach of their biological sensorial endowment will necessarily require the ability to make this extension relevant to them, and usable. Let us consider the interesting experiment of Google Glasses. They have not been very successful so far, but I think this is mostly because they came too early. Secondly, they fell victim to Google’s religious allegiance to the myth of technology. Adding tweets sent by somebody located nearby to the visual perception of a human being is not extending her sensorial capability in the PASC sense. It is only creating a scarcely relevant, probably useless, distraction to vision proper. It’s about adding noise, not signal. PASC instead, is about adding signal. Let us imagine risk evaluation in the PASC age. The risk of buying shares will not only be assessed against the market, the portfolio of the buyer, and other classical criteria. It will also be assessed based on the psychological condition of the buyer. With wearables like Apple Watch or a future evolution of it, it will be possible to determine if a purchase is done lucidly, or as a surrogate compensation for a loss or a feeling of unfulfillment or, maybe worse, fuelled by sudden euphoria.

As a recap, the below sketches some relations between PASC and other existing trends, trying to highlight the dependencies and provide grounds for defending the worth of labelling this new computing trend. I am convinced that, independent of the success of this PASC concept, useful extension of human sensorial endowment, rather than imitation of human mental processes, is the way forward. There is a bright future ahead, made of achievable advances, which can make the life of people, healthy or otherwise, a lot more fulfilling.

Footnotes

[1] https://commons.wikimedia.org/wiki/File:Sobrecarga_sensorial_-_8_%2810862394325%29.jpg

[2] See, for example, Amdahl’s Law: https://en.wikipedia.org/wiki/Amdahl%27s_law

Functional Solution to Range Minimum Query (RMQ) using Segment Trees in Scala

Segment TreeRMQ

Given an array of N integers, computing the minimum over the whole array is an easy operation requiring N steps. For example, one can take the first value as a tentative minimum, and iterate through the array comparing the tentative minimum with the current value. Any time the current value is smaller than the tentative minimum, it becomes the new tentative minimum, until the end of the array.
Let us now consider the case where it’s not the minimum of the whole array which matters, but the minimum of an arbitrary sub-array of the original one. With the term sub-array I mean an array formed by M consecutive elements taken from the original array, with 1 <= M <= N
The brute-force approach would be to compute the minimum element for any given range. The Range Minimum Problem (RMQ) is about finding a more efficient solution to this problem. How can one do better than O(N)? One obvious way would be to pre-compute the minimum of sub-arrays and sacrifice space for performance. But, wait a moment, how many possible sub-arrays can be defined given an array of size N?
Given an array of size N there are N sub-arrays of size 1, N-1 sub-arrays of size 2, N-2 sub-arrays of size 3 (…), 2 sub-arrays of size N-1 and 1 sub-array of size N.
The total number of sub-arrays is therefore:

which is bad, because this is space complexity O(N²). Is there another smarter way to tackle the problem? The answer is, of course, positive. Actually, there are a number of ways documented in the literature. This article will illustrate the technique known as “Segment Trees”. This is the idea that it is not necessary to pre-compute all the possible sub-arrays, but only some of them, and the other ones can be more quickly computed starting with the available partial results. Let us consider an example to clarify the idea. Before I go on, I need to do some explaining regarding the choice of Functional Programming (FP) data structures. Arrays are an imperative data structure which is not suitable for FP proper. The idea is that FP is about immutable values allowing for side-effect free computer programs. Arrays, instead, mimic the way a computer memory is structured, i.e. as a sequence of modifiable cells. The functional equivalent of arrays, in Scala, is Vectors. From now on, I will abide by the FP rules and use Vectors. I will now get back to our example. Given the vector:

val data = Vector(2,5,3,0,8,9,5,3,7,5,9,3)

if we knew already:

val min1 = min(Vector(2,5,3,0,8,9)) // min[0:5] val min2 = min(Vector(5,3,7,5,9,3)) // min[6:11]

we would not need to scan the sub-arrays, because we could simply compute:

val min = math.min(min1, min2)

Segment Trees are balanced binary trees which are composed by nodes consisting of a range and a value, which is the pre-computed minimum of the sub-array corresponding to such a range. In order to see how this tree looks like, let us consider the following example:

val data = Vector(4,5,2,8,9,0,1,2,5,1,8,6,3) // N=13

Our Segment Tree would look like this (click on image to see it full size):

The first thing to notice is that there are 25 ranges (nodes or leaves) in this tree, which is a lot less than 13(13+1)/2= 91
With higher values for N the difference would be even more noticeable. This number can be approximated in excess with this formula:

Approximation in excess of the number of ranges in a Segment Tree (nodes and leaves)

In our case it would give:

2⁰ + 2¹ + 2² + 2³ + 2⁴ = 1 + 2 + 4 + 8 + 16 = 31

The approximation is in excess because the formula assumes that each node forks in two nodes, which is not always the case. To see how best is this idea than the brute force approach, let us consider a Vector with N=1000 elements. We have seen above that all possible sub-ranges are:

1000*(1000+1)/2 = 500'500

However, our Segment Tree will contain no more than:

1 + 2 + 4 + 8 + 16 + 32 + 64 + 128 + 256 + 512 + 1028 = 2’061

Remarkable achievement, isn’t it? We can now see how to query a Segment Tree in order to get the minimum value of any sub-range of the original vector. The query algorithm is very simple, if expressed recursively, which is the perfect way to write in a Functional Language like Scala. The idea is this: if the query range is the same as the root range, get the minimum value from there. Otherwise, query the sub-trees, each with the intersection of the query range and the respective left and right sub-ranges, recursively. The Scala code will speak for itself. Try it out and have fun! I will copy it here for convenience’s sake, but it is also available at this address:

https://github.com/maumorelli/alaraph/blob/master/hackerrank/src/com/alaraph/hackerrank/rmq/Solution.scala

RMQ-002
RMQ-003

The explanation of how to use the program, the input format and expected output can be found at:
https://www.hackerrank.com/challenges/range-minimum-query
Here is a brief excerpt:

Sample Input
10 5 10 20 30 40 11 22 33 44 15 5 0 5 1 2 8 9 0 9 4 6
Sample Output
10 20 5 5 11
We are now approaching the end of this article. I hope you had a good read. To conclude, I would like to draw your attention to the fact that the depth of the Segment Tree is only Log₂(N)+1. This clearly implies that a Segment Tree can be visited very efficiently with a moderate number of recursive calls.

References

Hackerrank, (2014), https://www.hackerrank.com/challenges/range-minimum-query, accessed 01.08.2014

Topcoder, (2014), http://community.topcoder.com/tc?module=Static&d1=tutorials&d2=lowestCommonAncestor, accessed 01.08.2014

Wikipedia, (2014), https://en.wikipedia.org/wiki/Segment_tree, accessed 01.08.2014

Identity, Equality and Software

Software Sameness

The notion of identity is implicit in everyday discourse on an incredible variety of subjects. When one says “my car”, what one means is not a car of the same make, model, colour, as the car one has left in a nearby parking lot. What one means, is exactly the same very car, the single one object which is spatially and temporally continuous with the car one has just driven. This relation is referred to in the literature as numerical identity. As a first approximation, “Numerical identity can be characterised, (…) as the relation everything has to itself and to nothing else. “, (Stanford; 2014a)

Everybody is familiar with this relation. We all can (barring abnormal circumstances) recognise our beloved relatives, friends and pets quite easily. But our recognition abilities are not limited to living beings. We are also usually able to tell inanimate objects from one another quite unproblematically. This ability is so common that sometimes one forgets on which principles it is grounded. So, the question arises, how do we do to determine whether two entities are identical or not? In order to answer this question, we need to start from the basics. The relation of identity is clearly an equivalence relation, because it satisfies the following properties:

for every entity e, e is identical to itself (reflexivity)
for any given two entities e and k, if e is identical to k, then k is identical to e (symmetry)
for any given three entities e, k, and z, if e is identical to k and k is identical to z, then k is identical to z (transitivity)

Is this all? Clearly not. There are plenty of equivalence relations which are not identity relations. To give just a couple of examples:

“Has the same birthday as” on the set of all people.
“Has the same cosine” on the set of all angles.

(Wikipedia; 2014)

So, what does the identity relation require in addition to being an equivalence relation? It requires Leibniz’s Law: the principle of the indiscernibility of identicals, which says that “if x is identical with y then everything true of x is true of y.” (Stanford; 2014a). To sum up, when in this essay I will refer to the relation of identity, I will intend the equivalence relation which satisfies Leibniz’s principle of the indiscernibility of identicals. Let us now try to apply the notion of identity to software. To be more specific, in this acceptation, software has to be intended as computer programmes¹. And now the fun will begin! Let us consider a given personal productivity application P₁ installed on one’s laptop at work. If the user of P₁ has another application P₂ installed at home, by the same vendor, with the same major version, are P₁ and P₂ one and the same software? If we stick to Leibniz’s Law, we must see if there is at least a property which is valid of P₁ which is not valid of P₂, or viceversa. Of course there is at least one. “Being installed on the same hardware” is clearly unsatisfied. And one can easily find many other properties which hold true of P₁, but not for P₂, and the other way round. For example, “having the same detailed configuration”. The installation at work will very likely be pre-configured in conformance with security policies and guidelines which at home may not apply in exactly the same way (thanks God). But this preliminary result is puzzling. Isn’t it? Most people would say that P₁ and P₂ are the same programme. And if we apply Leibniz’s Law in other special cases, the puzzlement can go even further. For example, let us consider a computer programme written in a given programming language, say C++. In order to be executed on a given hardware, this programme must be compiled. Compilation is a process whereby a computer programme P₃ undergoes a series of correctness checks (syntactical and, to some extent, semantic), and is then “translated” into a sequence of machine instructions which can be executed on that particular hardware. Now, let us call P₃-Win our computer programme compiled on a Windows machine, and P₃-Ux the programme compiled on a Unix system. Are P₃-Ux and P₃-Win the same programme? If we consider that the source code is the same line-by-line, statement by statement, character by character, we may be tempted to say yes. But property “being compiled in the same set of machine instructions” is clearly not satisfied. So, according to Leibniz’s Law, P₃-Win and P₃-Ux are not numerically identical². Is this really so puzzling, in the end? I don’t think so. Everybody with experience knows that programmes may behave differently on different platforms. So, even though P₃ is clearly identical to itself, its instantiations aren’t identical with one another.

At this point, one may think, the problem is with the instantiations of software, not software itself. In our examples above, it was the installed application, or the compiled programme which led us into trouble. Are we safe if we just consider source code? At first sight, yes. But if we look more closely, not even source code can get us out of trouble when it comes to identity. There are still two problems. The first one is that source code does not exist in the vacuum: is it either saved on a hard disk, in a computing cloud, a flash memory, or is printed on paper. Again, two instances of a programme P, which are saved in two storage systems, or two folders in the same hard disk, or are printed on different sheets of paper, fail to satisfy Leibniz’s Law. The second problem is that even if we could find a way to make sense of the relation of identity for software, we would still need to answer the related question of what it means for a programme P to continue to be the same programme over time. Let us consider a programme consisting of, say, 3’000 statements. Let us now assume that this programme is modified and only the following statement is updated³, and nothing else:

old:

i = i+1;

new:

i++;

Every person with an understanding of programming would say that after this simple edit the programme is still the same. After all, the algorithm computed is the same; the operational semantics is the same. Everything except the syntax looks the same. But as we have seen above, Leibniz’s Law would not be satisfied, because property “having the same syntax” would not hold true⁴.

We have collected plenty of evidence that the relation of identity is not applicable to software. It is simply undefined. We cannot speak of numerical identity of software in any meaningful way. But how can we live with this finding? How can a software vendor justify the same license cost for two packages of a computer programme, if these cannot be proven numerically identical? How can one treat this strange substance, software, without such a basic relation as identity, which we use unproblematically with a lot of other entities we interact with on a daily basis?⁵

Luckily, there is, I think, a way out of this puzzle. This way out is the relation of equality. When people say that a programme P₁ and a programme P₂ are the same, what they really mean is not that they be numerically identical. What they mean is that they are equal. Equality is a weaker relation than identity. Here, it is not necessary that two entities satisfy exactly the same predicates: Leibniz’s Law need not apply. Equality is about satisfying a set of relevant properties which are the only ones which really matter to the observer for the determination whether or not such two entities can be exchanged. It is interesting to see that, although equality is a weaker relation than identity, it is still an equivalence relation⁶. The following definition will help us understand the difference between identity and equality more in depth:

The terms “equality” (Gr. isotes, Lat. aequitas, aequalitas, Fr. égalité, Ger. Gleichheit), “equal,” and “equally” signify a qualitative relationship. ‘Equality’ (or ‘equal’) signifies correspondence between a group of different objects, persons, processes or circumstances that have the same qualities in at least one respect, but not all respects, i.e., regarding one specific feature, with differences in other features. ‘Equality’ needs to thus be distinguished from ‘identity’ — this concept signifying that one and the same object corresponds to itself in all its features (…)

(Stanford; 2014b)

For physical objects, people usually use implicit but reliable criteria of equality. For example, a five dollars banknote in my wallet is considered to have the same value as a five dollar banknote in my wife’s wallet, despite the fact that they may differ in some respects like year printed, colour alterations due to ageing, or status of preservation overall. However, as I will show, if we try to apply equality to software, finding a commonly agreed set of properties is not so easy. As every head of software development knows very well, the notion of equality used by a manager and the notion of equality used by a software engineer are at odds. Let us consider the following fictitious excerpt of a discussion between a manager, and the head of software development:

Manager: How much will it cost to develop the new e-commerce platform internally?

Head of Development: It will cost X $

Manager: No way I’m doing it internally: I can have it done by software house ACME for 30% of that amount.

What equality criterion is the manager using above? Very likely, her criterion is “the software provider claims the software will satisfy the requirements document”. This is indeed a risky bet on her part, but it is the way outsourcing decisions are made every day. Why do I contend that this is a risky bet? Well, for a start, it is difficult to compare two things which do not even exist at the moment the decision is made. Second, satisfying a set of requirements is seldom black and white: compliance with requirements comes in degrees. For instance, test plans do not usually all fail or succeed. Some test cases fail, some others succeed, and when the ones which are perceived (or contractually defined) as the most important ones succeed, we say that requirements are satisfied. Moreover, requirements tend to be amended during a project, because they are seldom 100% correct at inception. Furthermore, there are such things as implicit requirements (Morelli; 2013), which are not formally defined, but are assumed because it is safe to give them for granted in a given cultural environment. But this assumption is only safe so long as the cultural affinity of the extended team is high. And a software house needs not be in the same cultural area as its customers. Therefore, even if all the requirements in the requirements document were satisfied by two applications, it may still be the case that implicit requirements are not equally satisfied by both of them. In which case, the two applications cannot be said equal, not even from the short-sighted viewpoint of our fictitious manager above. But this is not all. Below is an admittedly limited sample of other properties which a software engineer can legitimately consider in order to compare two computer programmes addressing the same specification:

Having the same algorithmic complexity in time and space
Having the same scalability with regard to number of concurrent users
Having the same scalability with regard to number of concurrent transactions
Having the same maintainability in terms of side effects per change
Having comparable quality (e.g. cyclomatic complexity)
Having comparable results when exposed to security penetration tests
Having comparable levels of service availability when deployed to the same hardware
Supporting the same deployment options
Being compliant to the same applicable regulations
Respecting the same standards
Requiring the same skillset to be maintained
etc.

The list above is open-ended and could be expanded at will. The bottom line is that, even if we give up using the relation of identity and stick to the weaker notion of equality, it appears that when it comes to software, sameness is really in the eyes of the beholder. What are differentiating factors for the software engineer, are not differentiating factors for the manager. And, to be intellectually honest, the other way round. For example, a software engineer may not be so sensitive to issues of cost. Or, at least, not as much as a manager. What this story is teaching, in the end, is that criteria of equality are affected by such factors as education, role, cultural bias, an so on and so forth. Given that judgements on equality are done by human beings, we cannot escape this result: equality without formal definition of criteria is an undefined concept. So, is equality a relation which we can use in the comparison of software? My contention is that equality is indeed a useful and usable way to approach comparison of software. But it is only useful and relevant on condition that:

Equality criteria are formally defined, shared and agreed with
Equality criteria are measurable
The error function of the measure is known in advance

In our fictitious case above, the manager is not making an informed decision unless she has formally defined, shared and has stakeholders agree on a set of equality criteria. The sentence “they will do the same programme for 50$” or what have you, is completely indefensible if it is not qualified in abidance with the provisions of the findings above.

Conclusion

This essay has explored the subtle notion of sameness applied to software. As a first attempt to define the semantic of software sameness I have analysed the relation of identity. First, I have defined what is a relation of identity; second, I have shown that this relation cannot be applied to software in any meaningful way. After that, I have introduced and explained the relation of equality. This relation has proved more useful and I have advanced that it is the best candidate semantics for the notion of sameness in software. However, my contention is that this is only true if the criteria for equality are formally defined, shared and agreed on. This proviso is a critical one and it must be satisfied in order to be able to profit from the use of the equality relation in matters of software sameness.

Bibliography

Morelli, Mauro, (2013), https://alaraph.com/2013/09/26/the-not-so-simple-world-of-user-requirements/, accessed 17.03.2014

Parfit, Derek, (1971), `Personal Identity´, Philosophical Review, pp. 8-27, vol. 80, no.1.

────────, (1987/1984), Reasons and Persons, Oxford University Press, 1987.

Stanford, (2014a), http://plato.stanford.edu/entries/identity/, accessed 13.03.2014

Stanford, (2014b), http://plato.stanford.edu/entries/equality/, accessed 17.03.2014

Wikipedia, (2014), https://en.wikipedia.org/wiki/Equivalence_relation, accessed 13.03.2014

End Notes

(1) Some, like Enterprise Architects, distinguish between computer programmes, applications, and technical components. However, in this essay I will use these terms interchangeably, because for the metaphysics of software, an algorithm is an algorithm and the rest is irrelevant.

(2) One may object that if we consider interpreted languages or byte-code languages the problem does not arise. But this contention is wrong, because the interpreter or the virtual machine, are themselves compiled computer programmes. The problem has only been moved one level of indirection away.

(3) For the sake of simplicity, we must also assume that this statement is not nested in a control structure like, for instance, a “for” cycle.

(4) I’m not sure whether the machine code generated by the compiler would be the same either.

(5) There are other known problematic cases for the identity relation. One is personal identity. As Philosophy Professor Derek Parfit contends in (Parfit; 1971) and (Parfit; 1984), identity is not what really matters in issues of survival. In such cases, he says, we know everything we need to understand the situation, without having to appeal to the notion of identity.

(6) Because it is clearly reflexive, symmetrical, and transitive.

Humanistic Work-Life Alignment Test (HWLAT)

[Wikimedia; 2014]

A lot of people think that it is good to have a job matching their values and priorities. Common sense suggests that, since we spend a substantial part of our lives at work, so much the better if we do something we like. But when it comes to defining what this means more in detail, and how to quantify the goodness of such a match, opinions vary. For example, one may try to define the distinctive qualities of a good job, and measure the match based on how a given job is close to such a golden standard. This viewpoint, which I will call the sociological approach (SA), can be tempting in that it gives the reassuring impression that this complex subject can be factorized, after all, in a limited and standardized set of variables. Defenders of the SA think that one can reliably define a golden rule using theories such as Marlow’s hierarchy of needs (https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs), and quantify the goodness of a given job in terms of degree of alignment with them. In this essay I will neither argue in favor, nor against any particular theory of value. My point is another: I advance that a good work-life alignment test (WLAT) should not commit to any particular theory of value, yet be compatible with all of them. Proponents of the SA are driven by their blind allegiance to the objectivity Demigod, which I reject entirely. My view, which I will call the humanistic approach (HA) is that what really matters in issues of alignment, is how I perceive the alignment, here and now. It is no consolation if one feels miserable in her job, to find out that such a subjectively miserable job is in fact a good one, according to some objective theory of value. In order to give an answer to the need for quantifying work-life alignment compatible with the tenets of the HA, I propose a method based on the following principles:

The criteria which define what is a good job for person P1 may not (and need not) be the same as the criteria for person P2.
Criteria which are relevant to person P at time t1 may not be the same as the ones which are relevant for the same person P at a later time t2 .
The ideal WLAT method should allow to compare the alignment of job J1 with criteria C1, with the alignment of job J2 with criteria C2. In other words, WLAT(J1, C1) and WLAT(J2, C2) should have values in the same set S for every combination of job Ji and criteria Ck.
The ideal WLAT should allow for a graphical representation of the work-life alignment which is easy to understand without requiring knowledge of WLAT. In other words, the outcome of the WLAT shall be easy yo understand independent if the reader is WLAT savvy or not.
The graphical representation of the outcomes of WLAT shall allow for uniform comparisons of alignment between different jobs against the same criteria, the same job against different criteria, and different jobs/criteria.
The WLAT shall allow for sets of criteria of any size, with the only obvious constraint that there be at least one.
If a criterion is already satisfied, increasing the satisfaction further will not increase the alignment rate, ceteris paribus.

I will call my method the Humanistic Work-Life Alignment Test (HWLAT). For convenience’s sake, before proceeding with the definition of the HWLAT, I will introduce a set of default criteria.

Default Criteria

Salary
Fringe Benefits
Flexible Work Time
Work from Home
Work Hours
Education Opportunities
Career Opportunities
Business Ethics
International Context
Small Commute from Home
Positive Relation with Boss
Positive Relation with Colleagues
Opportunities of Internal Job Change
Possibility to have a Sabbatical

Remark: Default criteria are not mandatory. The reader can use them as they are, or complement them with her own, replace some of them, keep some others, or even drop some without replacement.

Remark: Some criteria may only make sense for a job one already has, but not for prospective one. For example, one may know all too well how is her relation with her boss, but she can only guess the relation of her prospective boss, and not very reliably.

For the sake of argument let us assume that there be custom criteria as well:

Custom Criteria

Custom Criterion 1
Custom Criterion 2
Custom Criterion 3
etc.

To sum up, let n be the size of the chosen set of criteria C. For every criterion Ci, with i ϵ [1..n], the reader will express a weight in the range [0..10]. Higher weight means higher importance. Weight 0 means that the criterion should better be dropped because it has no importance. After that, the method requires the reader to assess the job at hand based on its degree of alignment with every single criterion. Results will look like this:

Criterion	Subjective Importance (si)	Subjective alignment rate (sar) of job J with this criterion
Salary	8	2
Fringe Benefits	7	7
Flexible Work Time	9	10
Work from Home	10	2
Work Hours	10	10
etc.
etc.

Now that we have data, the question arises, how can we build an alignment rate compatible with the tenets of HWLAT? My proposal is to pursue a representation like the following:

HWLAT Concept
The bigger the angle, the lower the alignment. Angle Θ is in the range [0..π/2]. The alignment is maximum when Θ is 0 (employee and employer have fully compatible values and priorities) and it is minimum when Θ is π/2. Assuming that we are working in the trigonometric circle with radius 1, the overlap is measured as cos(Θ). cos(0) = 1; cos(π/2) = 0. I have now all the elements for defining the formula.

I will give the following preliminary definitions:

C = {C1, C2, … , Cn} is a set of n criteria, with n > 0
J is the job whose alignment with criteria C must be determined
ITM(Cj) is the subjective importance of criterion Cj with j ϵ [1..n]
SAR(J, Cj) is the subjective alignment rate of job J with criterion Cj with j ϵ [1..n]

Definition 1 – HWLAT%

Let C be the set of n criteria relevant to a person P, and J the job whose alignment with C, P wishes to determine. The Humanistic Work-Life Alignment Test in Percentage (HWLAT%) is the function defined below:

HWLAT% formula

⧫

In order to represent the alignment using the format introduced above, we need to compute the angle between the two lines. This can be easily done applying elementary trigonometry.

Definition 2 – HWLAT

Let C be the set of n criteria relevant to a person P, and J the job whose alignment with C, P wishes to determine. The Humanistic Work-Life Alignment Test (HWLAT) is the function defined below:

HWLAT(J, C) = arccos(HWLAT(J, C))

⧫

Remarks

HWLAT has values expressed in radians in the range [0..π/2]. Its semantics is angle Θ above.

Now that the method is fully explained it is possible to go through a practical example of its use.

Example

Let us consider the following situation:

Criterion	Subjective Importance (si)	Subjective alignment rate (sar) of job J with this criterion
Salary	8	2
Fringe Benefits	7	7
Flexible Work Time	9	10
Work from Home	10	2
Work Hours	10	10
Education Opportunities	10	6
Career Opportunities	7	2
Business Ethics	9	5
International Context	6	8
Small Commute from Home	8	3
Positive Relation with Boss	10	6
Positive Relation with Colleagues	10	10
Opportunities of Internal Job Change	7	7
Possibility to have a Sabbatical	6	6

Applying the formulas above we have:

HWLAT%(J, C) = ∼ 69%

HWLAT(J, C) = ∼ 0,806 radians = ∼ 46°

HWLAT Example

⧫

To sum up, in this essay I have argued against objective approaches to measuring work-life alignment, on the grounds of the inevitably subjective perception of one’s happiness with her job. After that, I have defined a new approach which I have called Humanistic Work-Life Alignment Test (HWLAT). Then, I have shown the graphical representation of results. Last, I have given formulas allowing to compute the HWLAT compatible with all the tenets above. I hope the reader will find it useful to apply this method for the determination of the degree of alignment of her current or prospective job with her values and priorities. Feedback is welcome.

Bibliography

Wikimedia; 2013, http://upload.wikimedia.org/wikipedia/commons/a/a9/Adi_Holzer_Werksverzeichnis_850_Lebenslauf-2.jpg, accessed 3 February 2014

On the Use of Job Interview Puzzles

Monty Open Door

[Wikimedia; 2013]

Hiring the right person for the job requires experience, craft, time and money. Even in the most favorable circumstances, the determination of the best candidate for an opening remains a very difficult one. Consequences of a wrong hire can be dire, and may range from failing to achieve project objectives, to ending up with a long-term incompatibility with the team or the company culture. Because of this, companies invest substantial resources on devising the best possible hiring process. The challenge is to define a repeatable process which, when executed by expert recruiters, leads systematically to good results. This is the theory. Now, how do hiring teams actually proceed with the selection of IT personnel? Methods vary, but a technique has become particularly popular among hyped technology companies: it is the use of job interview puzzles. This essay focuses on this very technique, with the intent to demystify it and, possibly, determine its value and applicability. Before we delve deep into the topic, I must define what I mean by puzzle problems. There are a lot of different such problems, but what they have in common is that they are at the same time easy to understand, and apparently difficult to solve. Secondly, they cannot be answered using the skills required for the position at hand, except, possibly, creative problem solving. Examples abound on the internet. The interested reader can use search keywords “sample job interview puzzles” on any search engine, like www.duckduckgo.com. For the sake of argument, I will consider job interview puzzles as a peculiar kind of test. As such, they can be tuned to maximise their reliability in either one of the following cases:

False Positive: this is the error in which one incurs when a bad candidate is selected.
False Negative: this is the error in which one incurs when a good candidate is rejected.

Even though people from HR (or Human Operations as someone with a lot of fantasy is calling them) like to tell the story that they “care for false positives and negatives”, classical logic and common sense advise quite to the contrary. One can only maximise either for false positives or false negatives, but not for both. This is a very well known problem, and it has application in several sciences, including statistics and medicine, to name a few. Medical tests, for instance, are carefully tuned in order to strike the optimal balance between sensitivity and reliability. Typically, such tests are optimised in order to minimise the risk that a disease is not detected. Hence, they can give false positives. It is for this reason that positive tests are repeated twice. Now, let us come back to our problem space: interview puzzle tests. Here, it’s easy to see that puzzles are optimised in order to minimise false positives: if one solves the given problem (without cheating), it cannot be the case that her cognitive intelligence comes from nowhere. She must have the required degree of cognitive intelligence. Conversely, if someone is unable to produce evidence of the required cognitive intelligence during the interview, she will not make it. But this can have two reasons: either the candidate actually fails to meet the set requirements, or she simply had a bad performance for some reason, despite meeting the requirements. Why can a good candidate fail a puzzle test she would have solved correctly in dozens of other circumstances? Well, it depends. She may have not slept the night before because she was excited for the interview of her life. Or she did not sleep enough because she had to catch that early morning flight. Or for any other reason, because she is human and humans fail sometimes. But the hiring team does not care for determining which case it is, perhaps giving her a second chance. Why should they, so long as they have enough good people in the pipeline? Among them, there will certainly be someone in a better shape, able to produce the required evidence. The problem is that it may very well be the case that, among the rejected good candidates, there are some who are a better match of Mr Rambo who on that particular day was in a better shape to solve a puzzle, but happens to be an annoying presence and a nuisance for any team. “Hey, wait a minute”, the recruiter will want to object by now. She will probably advance that: (1) the candidate is purposefully put under stress, to see how she can put up with it, like in a real work situation; (2) one must use a criterion after all, to tell the good from the bad or, in politically correct, to tell the best match from the rest. However, I reject both objections and I advance that they are untenable. Let us start with (1): if someone fails to produce evidence of her qualities under exceptional circumstances, this is no evidence whatsoever that after a life of successes but one unlucky interview, this candidate is going to start, for this very reason , a series of failures. This is simply a ridicule argument, and betrays an oversized ego: the recruiter is implicitly advancing that dozens of university professors, former satisfied employers and clients were all wrong about the qualities of this candidate, and the only absolute truth worth of consideration is the result of the holy puzzle test. So holy and so true that she feels like rejecting on these very grounds a candidate with a history of documented successes (and if she did not have it, why have they invited her for an interview in the first place?). I will now proceed with point (2). The argument that a criterion is a criterion neglects such a thing as ethics. Discarding a good candidate is bad for the candidate, even though it may be good for the hiring company, which just considers this as ineliminable part of a process optimised to avoid false positives and, with them, reduce risk. Ethics mandates that the hiring process shall not pose the hiring company in a position of unilateral advantage on the candidate. But optimising the test against false positives does just that. Is there a way out of this clash of viewpoints? Perhaps. There can be one only if the likeliness of rejecting a good candidate is reasonably small, statistically. So, before the last word can be said on the use of puzzles, let us determine the likeliness of the morally uncomfortable mistake. How can we possibly do this? A famous formula will help. I will use Bayes’ Theorem, proved by English mathematician and Presbyterian minister Thomas Bayes (1701-1761) ^{[Wikipedia; 2013]}. This is the formula, in its simplest form ^{[Wikipedia; 2013b]}:

The meaning is this [Wikipedia; 2013b; Bayesian interpretation]:

In the Bayesian (or epistemological) interpretation, probability measures a degree of belief. Bayes’ theorem then links the degree of belief in a proposition before and after accounting for evidence. (…)

For proposition A and evidence B,

P(A), the prior, is the initial degree of belief in A.
P(A|B), the posterior, is the degree of belief having accounted for B.
the quotient P(B|A)/P(B) represents the support B provides for A.

In our case, I will set:

A = ”candidate X is a bad match”
B = “candidate X has failed the job interview puzzle”

I will now apply Bayes’ Theorem in order to determine the probability that candidate X is a bad match, given that she has failed the puzzle problem. In order to apply the formula, I must first determine the value of P(B|A), P(A) and P(B). Let’s start with P(A), which is the prior probability that candidate X is a bad match, Candidates are only invited to a job inteview after a careful screening. Their resume is analysed, titles are (assuming the hiring company is a serious one) verified. Further, oftentimes a phone interview precedes the on-site puzzle surprise. If the candidate has a documented brilliant academic record, an history of successes, and validated written references, she is very likely to deserve, if not the job, at least to be shortlisted. P(A) is the probability that: (1) the academic record is fake or, dozens of university professors have been damn wrong over a prolonged period of time, spanning years; (2) references are fake, and former managers and clients are not respectable people and wrote false reference letters. Uhm, this is crazy, isn’t it? What can be the prior probability of all this? I would say zero, but I cannot, because that would be tantamount to begging the question. To give an advantage to the puzzle interview true believers, I will generously concede that there may be one case in 1’000. So, we have:

P(A)=0,1% [1]

Let us now determine P(B|A), that is, the probability that candidate x fails the test, if she is a bad match. The probability is fairly high, say 99.5%: it is the probability that someone without the necessary problem solving skills fails a problem specifically designed to stress them to the max.

P(B|A)=99,5% [2]

Now it’s the turn of P(B): the probability that a candidate, whose resume has been screened, whose academic record and references checked, who has successfully passed the preliminary job interview fails the puzzle test. The fairness of the selection process hinges on the apparently reasonable assumption that P(B) is very very low. But this assumption is false, because people with the requested endowment of intellectual resources may not be ok on that particular day, they are likely to have caught an early flight, they are techies, and oftentimes they need time to familiarise with a new environment where there are several new people. I will again give a generous concession, I will recognise that these exceptionally unlucky circumstances will not materialise in more than 2% of cases:

P(B)=2% [3]

Now we have got the figures. Let us compute P(A|B)=(0,995*0,001)/0,02=0,04975%, which is approximately 5%. Wait a moment, we have determined that P(A|B)=5%. What??? 5%??? This astonishing result is worth a pause: The probability that candidate X, whose resume has been checked successfully, whose academic titles are genuine, whose references are validated, who has successfully passed a preliminary phone interview, is a bad match, given that she has failed a given job interview puzzle, is about 5%. The puzzle test priests will now complain that they did not actually invest the time to check all these things beforehand, because they can only afford this effort for the shortlisted candidates. After all, they have an infallible collection of puzzle tests, which will filter out bad candidates before HR even thinks of validating their dossier (resume, references, academic record). Sure, but along the way, I contend, they may have filtered out very good candidates as well. True believers will now want to remark, with a pitiful expression, that they are very sorry for that, but their job is to select the best person for the job, not to save unlucky candidates, however good they may be. They are not a charity after all. The point is that, in so doing, they are running an unethical process which is biased, by design, in favour of the hiring company, and against the single applicant. Of course, this selection will find a very good match, but at the cost of potentially discarding more good candidates than ethics would allow.

Conclusions

One may be tempted to think that the implications of this essay are leading nowhere. One needs criteria to select candidates, however imprecise these may be, the true believer will say. And she will go on with “imprecision is a fact of life: nobody ìs perfect. But we are doing our best. If we have made it, you can make it too: your success is only a few puzzle tests away”. But, to be constructive, how can one define an effective hiring procedure which is, at the same time also ethical, rather than grossly biased in favour of the hiring company? My answer is that puzzle problems can actually be utilised, on condition that a candidate with the right credentials can only be discarded after failing a series of such tests, not just one. A popular interview scheme is this:

A popular, but unethical, interview process

Using the results from this essay, one may think of alternative, more balanced, implementations of the selection process. For example the following:

A revised interview process

The revised process above is only an example, but it contains important amendments with respect to the previous one. For a start, the preliminary resume screening is now in depth for every candidate with the required qualifications and experience. Second, there are two preliminary job interviews rather than one, but there is no single instance of an interview puzzle with the potential to interrupt the ongoing selection of a candidate. Last, if the candidate meets the criteria so far, she is invited for a run of on site interviews. If one is really in love with puzzle tests, here they can be used: one cannot be always unlucky. But the candidate has the right to spend the night before close to the hiring location, and the hiring company must pay the expenses within reasonable limits. Under this proviso, can puzzle tests be used, which do not constitute an unfair bias of the selection process in favour of the hiring company, and against candidates. To sum up, this essay has disproved the following misconceptions:

Misconception 1: Job Interview Puzzles can replace serious resume screening We have seen that technically speaking, they can, but at the cost of running an unethical process which is disreputable for a mainstream company and, actually, for any company.
Misconception 2: Tests can be optimised at the same time for false positives and false negatives Here there is little to say. Someone with this crazy belief is simply lacking the cultural background to understand what is a test. One can “really care for both false positives and false negatives”, but the act of caring can neither reverse, nor cancel the laws of mathematics and common sense.
Misconception 3: A candidate failing a puzzle test is a bad match As we have seen above, Bayes’ Theorem demystified this contention without mercy. This is a fairly wrong belief, and a real shame. Good people deserve to work for better companies than these.

Bibliography

Wikimedia; 2013, https://upload.wikimedia.org/wikipedia/commons/3/3f/Monty_open_door.svg, accessed 14 November 2013

Wikipedia; 2013, https://en.wikipedia.org/wiki/Thomas_Bayes, accessed 11 November 2013

Wikipedia; 2013b,https://en.wikipedia.org/wiki/Bayesian_probability, accessed 11 November 2013