The paradox of information systems[edit] Drummond suggests in her paper in 2008 that computer-based information systems can undermine or even destroy the organisation that they were meant to support, and it is precisely what makes them useful that makes them destructive – a phenomenon encapsulated by the Icarus Paradox.[9] For examples, a defence communication system is designed to improve efficiency by eliminating the need for meetings between military commanders who can now simply use the system to brief one another or answer to a higher authority. However, this new system becomes destructive precisely because the commanders no longer need to meet face-to-face, which consequently weakened mutual trust, thus undermining the organisation.[10] Ultimately, computer-based systems are reliable and efficient only to a point. For more complex tasks, it is recommended for organisations to focus on developing their workforce. A reason for the paradox is that rationality assumes that more is better, but intensification may be counter-productive.[11]
From Wikipedia page on Icarus Paradox. Example of architectural design/technical debt leading to an "interest rate" that eventually collapsed the organization. How can one "pay down the principle" and not just the "compound interest"? What does that look like for this scenario? More invest in workforce retraining?
Humans are complex, adaptive systems. Machines have a long history of being complicated, efficient (but not robust) systems. Is there a way to bridge this gap? What does an antifragile system of machines look like? Supervised learning? How do we ensure we don't fall prey to the oracle problem?
Baskerville, R.L.; Land, F. (2004). "Socially Self-destructing Systems". The Social Study of Information and Communication Technology: Innovation, actors, contexts. Oxford: Oxford University Press. pp. 263–285