Notes for Lev Manovich Software Takes Command

Key concepts: affordances, cultural software, grey software, media software, metamedium, personal dynamic media, remediaion, software studies, stretchtext.

Related theorists: Jay Bolter, Jerome Bruner, Douglas Engelbart, Matthew Fuller, Sigfried Giedion, Adele Goldberg, Richard Grusin, Stuart Hall, Alan Kay, Ted Nelson, Howard Rheingold, Claude Shannon, Alvy Ray Smith, Ivan Sutherland, Noah Wardrip-Fruin, Warren Weaver.

First encounter with text was free, online version.


Understanding media
(1-2) Welcome to the world of permanent change—the world that is now defined not by heavy industrial machines that change infrequently, but by software that is always in flux.
(2) Software has become our interface to the world, to others, to our memory and our imagination—a universal language which the world speaks, and a universal engine on which the world runs.
(2) This book is concerned with “
media software--programs such as Word, PowerPoint, Photoshop, Illustrator, After Effects, Final Cut, Firefox, Blogger, WordPress, Google Earth, Maya, and 3ds Max.
(4) More generally, how are interfaces and the tools of media authoring software shaping the contemporary aesthetics and visual languages of different media forms?
(4) What happens to the idea of a “medium” after previously media-specific tools have been simulated and extended in software?

What is media after software?

(4) In short: What is “media” after software?

Does “media” still exist?
(4-5) What is the intellectual history of media software? What was the thinking and motivation of the key people and research groups they were directing?--J.C.R. Licklider, Ivan Sutherland, Ted Nelson, Douglas Engelbart, Alan Kay, Nicholas Negroponte—who between 1960 and the late 1970s created most of the concepts and practical techniques that underlie today's media applications?

Situates work within software studies; title pays homage to Giedion Mechanization Takes Command.

(5) Its title pays homage to a seminal twentieth-century book Mechanization Takes Command: a Contribution to Anonymous History (1947) by architectural historian and critic Sigfried Giedion.
(5) My investigation is situated within a broader intellectual paradigm of “software studies.”

Software, or the engine of contemporary societies
(7) And this “cultural software”--cultural in a sense that is directly used by hundreds of millions of people and that it carries “atoms” of culture—is only the visible part of a much larger software universe.
(9) Even today, ten years later, when people are constantly interacting with and updating dozens of apps on their mobile phones and other computer devices,
software as a theoretical category is still invisible to most academics, artists, and cultural professionals interested in IT and its cultural and social effects.

What is software studies?
(10) But computer science is itself part of culture. Therefore, I think that Software Studies has to investigate the role of software in contemporary culture, and the cultural and social forces that are shaping the development of software itself.
(10-11) The publication of this groundbreaking anthology [
New Media Reader] laid the framework for the historical study of software as it relates to the history of culture.
(11) In February 2006 Matthew
Fuller who had already published a pioneering book on software as culture (Behind the Blip: essays on the culture of software, 2003) organized the very first Software Studies Workshop at Piet-Zwart Institute in Rotterdam.

Reference to peer-reviewed journal Computational Culture.

(11-12) To help bring this change, in 2008, Matthew Fuller, Noah Wardrip-Fruin and I established the Software Studies book series at MIT Press. . . . In 2001, Fuller together with a number of UK researchers established Computational Culture, an open-access peer-reviewed journal that provides a platform for more publications and discussions.
(12-13) Yet another relevant category of books comprises the historical studies of important labs and research groups central to the development of modern software, other key parts of information technology such as the internet, and professional practices of software engineering such as user testing.

Rheingold first to explicitly base computers as new media, not just new technology.

(13) My all-time favorite book, however, remains Tools for Thought published by Howard Rheingold in 1985, right at the moment when domestication of computers and software starts, eventually leading to their current ubiquity. This book is organized around the key insight that computers and software are not just “technology” but rather the new medium in which we can think and imagine differently.
(13) Beginning around 2000, a number of artists and writers started to develop the practice of software art which included exhibitions, festivals, publishing books, and organizing online repositories of relevant works.
(15) I think of software as
a layer that permeates all areas of contemporary societies. Therefore, if we want to understand contemporary techniques of control, communication, representation, simulation, analysis, decision-making, memory, vision, writing, and interaction, our analysis cannot be complete until we consider this software layer.

Insistence on new methodologies of software studies including humanities scholars who program and have technical experience to round out accounts of modern media and technology.

(15-16) At the same time, the existing work in software studies already demonstrates that if we are to focus on software itself, we need new methodologies. That is, it helps to practice what one writes about. It is not accidental that all the intellectuals who have most systematically written about software's roles in society and culture have either programmed themselves or have been involved in cultural projects and practices which include writing and teaching software—for instance, Ian Bogost, Jay Bolter, Florian Cramer, Wendy Chun, Matthew Fuller, Alexander Galloway, Katherine Hayles, Matthew Kirschenbaum, Geert Lovink, Peter Lunenfeld, Adrian Mackenzie, Paul D. Miller, William J. Mitchell, Nick Montfort, Janey Murray, Kaite Salen, Bruce Sterling, Noah Wardrip-Fruin, and Eric Zimmerman. In contrast, the scholars without this technical experience or involvement—for example, Manual Castells, Bruno Latour, Paul Virilio, and Siegfried Zielinksi—have not included discussions of software in their otherwise theoretically precise and highly influential accounts of modern media and technology.
(16) These programming and scripting languages and APIs did not necessarily make programming easier. Rather, they made it much more efficient. . . . Yet another reason for more people writing software today is the emergence of a massive mobile apps marketplace that, unlike the desktop market, is not dominated by a few large companies.

Despite of preference for retaining affordances of specific media over convergence in invisible interface, endorses hope that programming will become easy and lead to long tail democratization.

(17) Clearly, today the consumer technologies for capturing and editing media are much easier to use than even the most friendly programming and scripting languages. . . . But I do not see any logical reasons why programming cannot one day become equally easy.
(17-18) Although we are far from a true “long tail” for software, software development is gradually getting more democratized. It is, therefore, the right moment to start thinking theoretically about how software is shaping our culture, and how it is shaped by culture in its turn.

Cultural software
(20) This book is determined by my own history of engagement with computers as a programming, computer animator and designer, media artist, and as a teacher. . . . My first experience with computer graphics was in 1983-4 on Apple IIe.
(21) Thus, although I first learned to program in 1975 when I was in high school in Moscow, my take on software studies has been shaped by watching how during the 1980s GUI-based software quickly put the computer in the center of culture.

Grey software is that which is not directly used by most people, such as logistics and industrial automation software, although it regulates society.

Culture software enables cultural actions: creating artifacts, accessing and remixing them, creating knowledge online, communicating, engaging in interactive cultural experiences, participating in information ecology by expressing preferences and adding metadata, developing software tools and services.

(21) However, since I do not have personal experience writing logistics software, industrial automation software, and other grey” software, I will not be writing about such topics. My concern is with a particular subset of software which I used and taught in my professional life. I call it cultural software.
(21-23) These cultural actions enabled by software can be divided into a number of categories.
1 Creating cultural artifacts and interactive services . . .
2 Accessing, appending, sharing, and remixing . . .
3 Creating and sharing information and knowledge online . . .
4 Communicating with other people . . .
5 Engaging in interactive cultural experiences . . .
6 Participating in the only information ecology by expressing preferences and adding metadata . . .
7 Developing software tools and services.

Media applications
(27) While I will focus on media applications for creating and accessing “content” (i.e. media artifacts), cultural software also includes tools and services that are specifically designed for
communication and sharing of information and knowledge, i.e. “social software.”
(28-29) The challenge of software studies is to be able to use terms such as “content” and “software application” while always keeping in mind that the current social media/cloud computing paradigms are systematically reconfiguring the meaning of these terms.
(29) The interface category is particularly important for this book. I am interested in how
software appears to users—i.e. what functions it offers to create, share, reuse, mix, create, manage, share and communicate content, the interfaces used to present these functions, and assumptions and models about a user, her/his needs, and society encoded in these functions and their interface design.
(31) Should I not put my energy into promoting programming rather than explaining applications?

Focus on mainstream applications and create and access cultural content over promoting programming, which is an exceptional category.

(31) The reason for my choices is my commitment to understand the mainstream cultural practices rather than to emphasize (as many cultural critics do) the exceptions, no matter how progressive they may be.
(32) at the end of the twentieth century humans have added a fundamentally new dimension to everything that counts as “culture.” This dimension is software in general, and application software for creating and accessing content in particular.

From documents to performances
(33-34) Instead of fixed documents that could be analyzed by examining their structure and content (a typical move of the twentieth-century cultural analysis and theory, from Russian Formalism to Literary Darwinism), we now interact with dynamic “software performances.” . . . In other words, in contrast to paintings, literary works, music scores, films, industrial designs, or buildings, a critic cannot simply consult a single “file” containing all of the work's content.
(34-35) This shift in the nature of what constitutes a media “document” also calls into question well-established cultural theories that depend on this concept. . . .l Communication scholars have then the model of information transmission formulated by Claude
Shannon in his 1948 article A Mathematical Theory of Communication and his subsequent book published with Warren Weaver in 1949, and applied its basic model of communication to mass media.
(35) Classical communication theory and media industries considered such partial reception a problem; in contrast, in his influential 1980 article “Encoding/decoding” the founder of British Cultural Studies, Stuart
Hall, argued that the same phenomenon is positive. . . . But both the classical communication studies and cultural studies implicitly took for granted that the message was something complete and definite.

Actively managed model of communications replaces classical theory of Hall encoding decoding in which partial reception problematic.

(35-36) The interfaces of media access applications . . . encourage people to “browse,” quickly moving both horizontally between media (from one search result to the next, from one song to another, etc.) and vertically, through the media artifacts (e.g., from the contents listing of a music album to a particular track). . . . In other words, the “message” that the user “receives” is not just actively “constructed” by him/her (through a cognitive interpretation) but also actively managed (defining what information s/he is receiving and how).
(37) This shift from messages to platforms was in the center of the Web's transformation around 2004-6. The result was named Web 2.0.
(38) Continuously changing and growing content of web services and sites; variety of mechanism for navigation and interaction; the abilities to add one's own content and mashup content from various sources together; architectures for collaborative authoring and editing; mechanisms for monitoring the providers—all these mechanisms clearly separate interactive networked software-driven media from twentieth-century media documents. . . . And while “old media” (with the exception of twentieth-century broadcasting) also provided this random access, the interfaces of software-driven media players/viewers provide many additional ways for browsing media and selecting what and how to access.
(39) This media architecture enables easy addition of new navigation and management tools without any change to the documents themselves.

Why the history of cultural software does not exist

No reason to resurrect obsolete versions of most cultural software, in contrast to reissue of early video games.

Suggestion that cultural interest would be catalyzed if early software was widely available.

(41) It does not derive any profits from the old software—and therefore it does nothing to promote its history. . . . in contrast to the video games from the 1980s, these early software versions are not treated as separate products which can be re-issued today. . . . Although I am not necessarily advocating the creation of yet another category of commercial products, if early software was widely available in simulation, it would catalyze cultural interest in software similar to the way in which wide availability of early computer games, recreated for contemporary mobile platforms, fuels the field of video game studies.

With no preservation of obsolete versions of cultural software to study, no conceptual history or investigation of roles played by software in media production; compare to Campbell-Kelly and other software historians.

(41) we lack not only a conceptual history of media editing software but also systematic investigations of the roles of software in media production. For instance, how did the adoption of the popular animation and compositing application After Effects in the 1990s reshape the language of moving images?
(42) In summary, a systematic examination of the connections between the workings of contemporary media software and the new communication languages in design and media (including graphic design, web design, product design, motion graphics, animation, and cinema) has not yet been undertaken.
(420 By focusing on the theory of software for media design, this book aims to complement the work of a few other theorists that have already examined software responsible for game platforms and design (Ian Bogost, Nick Montfort), and electronic literature (Noah Wardrip-Fruin, Matthew Kirschenbaum).
(42) In this respect, the related fields of code studies and platform studies being developed by Mark Marion, Nick Montfort, Ian Bogost and others are playing a very important role.

Summary of the book's narrative
(43) What was the thinking and motivation of people who between 1960 and the late 1970s created the concepts and practical techniques that underlie today's cultural software? How does the shift to software-based production methods in the 1990s change our concepts of “media”? How have interfaces and the tools of content development software reshaped and continued to shape the aesthetics and visual languages we see in contemporary design and media? These are the key questions that I take up in this book.
(43) I will trace a
particular path through this history that will take us from 1960 to today and which will pass through some of its most crucial points.

Kay called computer first metamedium; foundations established in 1960s through 1970s so that by mid1990s media hybridization, evolution and deep remix are dominant concepts.

(44) Accordingly, [Alan] Kay calls computers the first metamedium whose content is “a wide range of already-existing and not-yet-invented media.”
(44) The foundations necessary for the existence of such metamedium were established between the 1960s and the late 1970s.
(45) I use three different concepts to describe these developments and the new aesthetics of visual media which developed in the second part of the 1990s after the processes of adoption reached sufficient speed. These three concepts are
media hybridization, evolution, and deep remix.

New historical stage of softwarization first affecting professional creatives, then the rest of us; would Manovich progression be orality, literacy, hybridity?

(45-46) Once they were simulated in a computer, previously incompatible techniques of different media begin to be combined in endless new ways, leading to new media hybrids, or, to use a biological metaphor, new “media species.” . . . In my view, this ability to combine previously separate media techniques represents a fundamentally new stage in the history of human media, human semiosis, and human communication, enabled by its “softwarization.”
(46) Deep remixability is central to the aesthetics of motion graphics.
(46-47) The next major wave of computerization of culture has to do with different types of software—social networks, social media, services, and apps for mobile platforms. . . . The 1990s' media revolution impacted
professional creatives; the 2000s' media revolution affected the rest of us.
(47) I decided that offering the detailed theoretical analysis of this new wave would be premature. . . . Instead, I am focusing on tracing the fundamental developments which made possible and shaped “digital media” before its social explosion: the ideas about the computer as a machine for media generation and editing of the 1960s-1970s, their implementation in the media applications in the 1980s-1990s, and the transformation of visual media languages which quickly followed.

Sutherland Sketchpad first computer design system presented publicly in 1961.

(47) To be more precise, we can frame this history between 1961 and 1999. In 1961, Ivan Sutherland at MIT designed Sketchpad, which became the first computer design system shown to the public. In 1999, After Effects 4.0 introduced Premiere import. Photoshop 5.5 added vector shapes, and Apple showed the first version of Final Cut Pro—in short, the current paradigm of interoperable media authoring and editing tools capable of creating professional media without special hardware beyond the off-the-shelf computer was finalized. And while professional media tools continued to evolve after this period, the changes so far have been incremental. Similarly, the languages of professional visual media created with this software did not change significantly after their radical transformation in the second part of the 1990s.
(48) I have chosen to focus on the desktop applications for media authoring most widely used today—Photoshop, Illustrator, InDesign, Dreamweaver, After Effects, Final Cut, Maya, 3ds, Max, Word, PowerPoint, etc. . . . I will also be making references to popular web browsers, media sharing services, email services and clients, web-based office suites, and consumer geographic information systems. Since I am interested in how users interact with media, another key softawre category for this book is media players and document viewing applications.
(50) Because I do not expect a typical reader of this book to have a working experience with these expensive systems, I will not be referring to them further in this book.
(50) I love and support open source and free access, and use it for all my work.

Foregrounds the most commonly used applications, which are likely commercial, regardless of personal ideological preference for free, open source options.

(50-51) The reason this book focuses on commercial media authoring and editing software rather than its open source equivalents is simple. In almost all areas of software culture, people use free applications and web services. . . . However, in the case of professional tools for media authoring and editing, commercial software dominates. It is not necessarily better, but it is simply used by many more people. . . . Since I am interested in describing the common user experiences, and the features of media aesthetics common to millions of works created with the most common authoring tools that are all commercial products, these are the products I choose to analyze. And when I analyze tools for media access and collaboration, I similarly choose the most popular products—which in this case includes both free software and services provided by commercial companies (Safari, Google Earth), and free open source software (Firefox).

Inventing media software
Alan Kay's universal media machine

Appearance versus function
(55) Between its invention in the mid-1940s and the arrival of PCs in the early 1980s, the digital computer was mostly used for military, scientific, and business calculations and data processing. It was not interactive. It was not designed to be used by a single person. In short, it was hardly suited for cultural creation.
(56) In short, it appears that the revolution in the means of production, distribution, and access of media has not been accompanied by a similar revolution in the syntax and semantics of media.
(56) Building on the already accomplished work of the pioneers of cultural computing, the Learning Research Group at Xerox PARC, headed by Kay, systematically articulated the paradigm and the technologies of
vernacular media computing, as it exists today.
(57) It is well known most of the the key ingredients of personal computers as they exit today came out of Xerox PARC: the Graphical User Interface with overlapping windows and icons, bitmapped display, color graphics, networking via Ethernet, mouse, laser printer, and WYSWYG printing. But what is equally important is that Kay and his colleagues also developed a range of applications for media manipulation and creation that also all used a graphical interface. They included a word processor, a file system, a drawing and painting program, an animation program, a music editing program, etc. Both the general user interface and the media manipulation programs were written in the same programming language, Smalltalk.
(57-58) When Apple introduced the first Macintosh computer in 1984, it brought the vision developed at Xerox PARC to consumers (the new computer was priced at USD $2,495). The original Machintosh 128K included a word processing and a drawing application (MacWrite and MacPaint, respectively). Within a few years these were joined by other software for creating and editing different media: Word, PageMaker and VideoWorks (1985), SoundEdit (1986), Freehand and Illustrator (1987), Photoshop (1990), Premiere (1991), After Effects (1993), and so on. In the early 1990s, similar functionality became available on PCs running Microsoft Windows.
(58) By around 1991, the new identity of a computer as a personal media editor was firmly established.

GUI software turned computer into remediation machine representing earlier media.

(58-59) By developing easy-to-use GUI-based software to create and edit familiar media types, Kay and others appear to have locked the computer into being a simulation machine for “old media.” Or, to put this in terms of Jay Bolter and Richard Grusin's influential book Remediation: Understanding New Media (2000), we can sat that GUI-based software turned a digital computer into a “remediation machine,” a machine that expertly represents a range of earlier media.
(59) There was definitely nothing in the original theoretical formulations of digital computers by Turing or Von Neumann about computers imitating other media such as books, photography, or film.
(60) While media theorists have spent considerable efforts in trying to understand the relationships between digital media and older physical and electronic media in the 1990s and 2000s, the important sources—the writing and project by Ivan Sutherland, Douglas Engelbart, Ted Nelson, Alan Kay, and other pioneers working in the 1960s and the 1970s—remained largely unexamined.

What is media after software becomes the new question; Kay personal dynamic media historically unprecedented affordances.

(60) In short, I want to understand what is “media after software--that is, what happened to the techniques, languages, and the concepts of twentieth-century media as a result of their computerization.
(61) Kay conceived of “
personal dynamic mediaas a fundamentally new kind of media with a number of historically unprecedented properties such as the ability to hold all the user's information, simulate all types of media within a single machine, and “involve the learner in a two-way conversation.” These properties enable new relationships between the user and the media s/he may be creating, editing, or viewing on a computer. And this is essential if we want to understand the relationships between computers and earlier media. Briefly put, while visually, computational media may closely mimic other media, these media now function in different ways.
(62) To use a different term, we can say that a digital photograph offers its users many “
affordancesthat its non-digital predecessor did not.
(62-63) In summary, we can say that only some of the “new DNA” of a digital photograph is due to its particular place of birth, i.e., inside a digital camera. Many others are the result of the current paradigm of network computing in general.
(63-64) While Vannevar Bush, J.C.R. Licklider and Douglas Engelbart were primary concerned with augmentation of intellectual and in particular scientific work, Kay was equally interested in computers as “a medium of expression through drawing, painting, animating pictures, and composing and generating music.”

Simulation is the central notion of the Dynabook”

Dynabook platform a metamedium, challenging prior understanding of media as separate from one another.

(64) In this article [“Personal Dynamic Media”] Kay and Goldberg describe the vision to create “a personal dynamic medium the size of a notebook (the Dynabook) which could be owned by everyone and could have the power to handle virtually all of its owner's information-related needs.”
(65) Rather, the goal was to establish a computer as an umbrella, a platform for
all existing expressive artistic media. (At the end of the article Kay and Goldberg give a name for this platform, calling it a “metamedium.”) This paradigm changes our understanding of what media is. From Gotthold Ephraim Lessing's Lacoon; or, On the Limits of Painting and Poetry (1766) to Nelson Goodman's Languages of Art (1968), the modern discourse about media depends on the assumption that different mediums have distinct properties and in fact should be understood in opposition to each other. . . . Some of these new connections were already apparent to Kay and his colleagues; others became visible only decades later when the new logic of media set in place at PARC unfolded more fully; some may still not be visible to us today because they have not been given practical realization. . . . All in all, it is as though different media are actively trying to reach towards each other, exchanging properties and letting each other borrow their unique features.
(70) It was only Kay and his generation that extended the idea of simulation to media—thus turning Universal Turing Machine into a
Universal Media Machine, so to speak.

Beyond remediation, creating magical paper: adding new properties and personal programming suggests another site for critical programming.

(70-71) Appropriately, when Kay and his colleagues created computer simulations of existing physical media—i.e., the tools for representing, creating, editing, and viewing these media—they “added” many new properties. . . . As Kay has referred to this in another article, his idea was not to simply imitate paper but rather to create “magical paper.”
(72) Studying the writings and public presentations of the people who invented interactive media computing—Sutherland, Engelbart, Nelson, Negroponte, Kay, and others—makes it clear that they did not produce the new properties of computational media as an afterthought. On the contrary, they knew that they were turning physical media into new media.

View control example of new media property intentionally highlighted by Engelbart demo, comparable to Nelson idea of stretchtext.

(72-73) Paying attention to the sequence of the demo reveals that while Engelbart had to make sure that his audience would be able to relate to the new computer systems to what they already knew and used, his focus was on new feature of simulated media never before available previously. . . . As Engelbart points out, the new writing media could switch at the user's wish between many different views of the same information.
(75) (In 1967 Ted Nelson articulated and named a similar idea of a type of hypertext, which would allow a read to “obtain a greater detail on a specific subject.” He named it “
(75) Since new media theory and criticism emerged in the early 1990s, endless texts have been written about interactivity, hypertext, virtual reality, cyberspace, cyberculture, cyborgs, and son on. But I have never seen anybody discuss “view control.” And yet this is one of the most fundamental and radical new techniques for working with information and media available to us today.
(78) While such historical precedents for hypertext are often proposed, they mistakenly equate Nelson's proposal with a very limited form in which hypertext is experienced by most people today—i.e., the World Wide Web.
(79) “What kind of structures are possible in hypertext?” asks Nelson in a research note from 1967. He answers his own question in a short but very suggestive manner: “Any.”
(80-81) The announcement for his [Nelson's] January 5, 1965 lecture at Vassar College talks about this in terms that are even more relevant today than they were then: “The philosophical consequences of all this are very grave. Our concepts of 'reading', 'writing', and 'book' fall apart, and we are challenged to design 'hyperfiles' and write 'hypertext' that may have more teaching power than anything that could ever be printed on paper.”
(82) Although Nelson says that hypertext can support any information structure and that this information does not need to be limited to text, his examples and his style of writing show an unmistakable aesthetic sensibility—that of literary modernism.
(82) The early twentieth-century avant-garde artists were primarily interested in questioning conventions of established media such as photography, print, graphic design, cinema, and architecture. . . . In contrast, Nelson and Kay explicitly write about creating new media, not only changing the existing ones.
(83-84) Instead of pa particular modernist “ism,” we get a file structure. . . . Instead, the new system would be capable of simulating all these media with all their remediation strategies—as well as supporting development of what Kay and Goldberg referred to as new “not-yet-invented media.” And of course, this was not all. Equally important was the role of interactivity. The new meta-systems proposed by Nelson, Kay and others were to be used interactively to support the processes of thinking, discovery, decision making, and creative expression. . . . Finally, at least in Kay's and Nelson's vision, the task of defining new information structures and media manipulation techniques—and, in fact, new media as a whole—was given to the user, rather then being the sole province of the designers. . . . Since the end of 2000, extending the computer metamedium by writing new software, plugins, programming libraries and other tools became the new cutting-edge type of cultural activity – giving a new meaning to McLuhan's famous formula “the medium is the message.”
(84) In either case, the need for new research is justified by a reference to already established or popular practices—academic paradigms which have been funded, large-scale industries, and mainstream social routines which do not threaten or question the existing social order.
(84) The invention of new mediums for its own sake is not something which anybody is likely to pursue, or get funded. From this perspective, the software industry and business in general is often more innovative than academic computer science.
(85-86) The newness lies not in the content but in the software tools used to create, edit, view, distribute and share this content. Therefore, rather than only looking at the “output” of software-based cultural practices, we need to consider software itself—since it allows people to work with media in a number of historically unprecedented ways.
(88) Rather than conceiving of Sketchpad as simply another medium, Sutherland presents it as something else—a communication system between two entities: a human and an intelligent machine.

Example of digital frame buffer as new creative medium.

(90-91) But even if we forget about SuperPaint's revolutionary ability to combine graphics and video, and discount its new tools such resizing, moving, copying, etc., we are still dealing with a new creative medium ([Alvy Ray] Smith's term). As Smith pointed out, this medium is the digital frame buffer, a special kind of computer memory designed to hold images represented as an array of pixels (today a more common name is graphics card).

The permanent extendibility
(92) In short,
new media” is “new” because new properties (i.e., new software techniques) can always be easily added to it.
(92) What used to be separate moments of experimentation with media during the industrial era became the norm in a software society.
(93) But this process of continual invention of new algorithms does not just move in any direction. . . . As new techniques continue to be invented they are layered over the foundations that were gradually put in place by Sutherland, Engelbart, Kay, and others in the 1960s and 1970s.

Malleability of software compared to other industrially produced objects.

(93) New programs can be written and existing programs can be extended and modified (if the source code is available) by anybody who has programming skills and access to a computer, a programming language and a compiler. In other words, today software is fundamentally malleable in a way that twentieth-century industrially produced objects were not.

Kay like Kemeny does philosophy of programming.

(94) This democratization of software development was at the core of Kay's vision. Kay was particularly concerned with how to structure programming tools in such a way that would make development of media software possible for ordinary users.
(94) This means that the idea that a new medium gradually finds its own language cannot apply to computer media. If this were true it would go against the very definition of a modern digital computer. This theoretical argument is supported by practice. The history of computer media so far has not been about arriving at some standardized language—as, for instance, happened with cinema—but rather about the gradual expansion of uses, techniques, and possibilities.
(95) To rephrase this example in more general terms, we can say that rather than moving from an imitation of older media to finding its own language, computational media was from the very beginning speaking a new language.
(96) The inventors of computational media had to question many, if not most, already established concepts and techniques of how both software and hardware function, thus making important contributions to hardware and software engineering. A good example is Kay's development of Smalltalk, which for the first time systematically established a paradigm of object-oriented programming.
(96-97) Looking at the history of computer media and examining the thinking of its inventors makes it clear that we are dealing with the opposite of technological determinism. . . . Similar to Marx's analysis of capitalism in his works, here the analysis is used to create a plan for action for building a new world—in this case, enabling people to create new media.

Example of interactive interface as non-deterministic development not latent in theoretical computing concepts of Von Neumann architecture.

(97) But the most important example of such non-deterministic development is the invention of the modern interactive graphical human-computer interface itself by Sutherland, Engelbart, Kay and others. None of the key theoretical concepts of modern computing as developed by Turing and Von Neumann called for an interactive interface.

Media must be thought beyond symbols, for even Platonic living writing ideal which implies invisible interface akin to direct manipulation, especially for learning; on the right track correcting ideology of direct manipulation, in which the medium disappears, with position leveraging material specific affordances of media.

Kay goes beyond Kemeny and others who focus on utilitarian uses interpellating adults of all ages by including experimentation and artistic expression; appeal to enactive, iconic and symbolic mentalities via mouse, icons and windows, Smalltalk.

Kay deliberate design guide thinking of computers as medium for learning, experimentation and artistic expression for children of all ages; user interface should appeal to enactive, iconic and symbolic mentalities as articulated by Bruner and Piaget.

Learning having enactive, iconic and symbolic components means removing need to program from interface unintentionally weakened human intelligence.

(97-98) According to Kay, they key step for him and his group was to starting thinking about computers as a medium for learning, experimentation, and artistic expression which can be used not just by adults but also by “children of all ages.” Kay was strongly influenced by the theory of cognitive psychologist Jerome Bruner. . . . Bruner gave slightly different names to these different mentalities [of Piaget]: enactive, iconic, and symbolic. While each mentality has developed at different stages of human evolution, they continue to co-exist in an adult.
(98) Kay's interpretation of this theory was that a user interface should appeal to all these three mentalities. In contrast to a command-line interface, which is not accessible for children and forces the adult to use only symbolic mentality, the new interface should also make use of emotive and iconic mentalities.
Mouse activates enactive mentality (know where you are, manipulate), Icons and windows activate iconic mentality (recognize, compare, configure.) Finally, Smalltalk programming language allows for the use of symbolic mentality (tie together long chains of reasoning, abstract.)
(99) In actual use, a contemporary GUI involves constant interplay between different mentalities.

PARC GUI designed as medium to facilitate learning, discovery and creativity.

(100) If we are to agree with Bruner's theory of multiple mentalities and Kay's interpretation of this theory, we should conclude that the new computational media that he helped to invent can do something no previous media can—activate our multiple mentalities which all play a role in learning and creativity, allowing a user to employ whatever works best at any given moment and to rapidly switch between them as necessary. . . . In short, while many HCI experts and designers continue to believe that the idea human-computer interface should be invisible and get out of the way to let users do their work, looking at the theories of Kay and Goldberg that were behind GUI design gives a very different way of understanding an interface's identity. Kay and his colleagues at PARC have conceived GUI as a medium designed in its every detail to facilitate learning, discovery, and creativity.

Commercially successful GUI designed as intuitive mimicry of physical world workspace.

(101) Unfortunately, when GUI became the commercially successful paradigm following the success of Apple's Mac computers, introduced in 1984, the intellectual origins of GUI were forgotten. Instead, GUI was justified using a simplistic idea that since computers are unfamiliar to people, we should help them by making interface intuitive by making it mimic something users are already well familiar with—the physical world outside of a computer (which in reality was an office environment with folders, desks, printers, etc.) Surprisingly, even in recent years—when “born digital” generations were already using computer devices even before they ever set foot in an office—this idea was still used to explain GUI.

The computer as a metamedium
(102-103) In other words, a computer can be used to create
new tools for working with the media types it already provides as well as to develop new not-yet-invented media.

Literacy implies reading and writing abilities; media editing applications provided with computers should inspire users to write their own programs.

Development of Smalltalk and applications provided with computer to encourage user modification and development to test hypotheses: a prototype of critical programming.

(103) Using the analogy with print literacy, Kay motivates this property in this way: “The ability to 'read' a medium means you can access material and tools generated by others. The ability to write in a medium means you can generate materials and tools for others. You must have both to be literate.” Accordingly, Kay's key effort at PARC was the development of the Smalltalk programming language. . . . In other words, all media editing applications that would be provided with a computer, were to serve also as examples, inspiring users to modify them and to write their own applications.
(104) Accordingly, the large part of Kay and Goldberg's paper is devoted to description of software developed by the users of their system . . . professionals, high school students, and children—in order to show that everybody could develop new tools using the Smalltalk programming environment.
(104) Just as a scientist may use simulation to test different conditions and play different what/if scenarios, a designer, a writer, a musician, a filmmaker, or an architect working with computer media can quickly “test” different creative directions in which the project can be developed as well as see how modifications of various “parameters” affect the project.

Is it only historical accident that the Macintosh did not ship with a user development environment, corrupting Kays vision?

Processing model language for everyday users developing their own media tools.

(105) Unfortunately, when in 1984 Apple shipped Macintosh, which was to become the first commercially successful personal computer modeled after the PARC system, it did not have an easy-to-use programming environment. . . . Only more recently, as the general computer literacy has widened and many new high-level programming languages have become available—Perl, PHP, Python, JavaScript, etc.--have more people started to create their own tools by writing software. A good example of a contemporary programming environment, very popular among artists and designers and which, in my view, is close to Kay's vision, is Processing.

Understanding metamedia
The building blocks

(107) During the years I was writing and editing the book, many important developments made Alan Kay's vision of a computer as the “first metamedium” more real—and at the same time more distant.

Manovich, Lev. Software Takes Command. New York: Bloomsbury, 2013. Print.