Wayne O Evans Consulting, Inc
The following article has been reproduced from an article by Wayne O. Evans who is an Independent Security Consultant. Wayne kindly granted us permission to use this text on the grounds that we make a donation to charity. We are proud to report we have made a donation of 50GBP to the NSPCC (National Society for the Prevention of Cruelty to Children) in the UK.
We would urge anyone who uses this information or information from Wayne’s web site to do the same. Together we can make a difference!
More compelling articles by Wayne can be found at www.woevans.com
A Glance Back in History to the Origins of OS/400
This article reviews some of the design decisions that were made early in the development of S/38. This early history is important because much of the architecture of OS/400 was inherited from S/38.
Over 20 years ago when the S/38 was being developed, the world had never heard of Microsoft or Netscape. But the struggle between the computer giant IBM and its competitors for market share was as fierce as today’s battles over the next browser. The IBM Corporation had experienced the loss of hardware sales to “look alike” vendors Amdahl and others. These hardware clones would run the IBM operating system and customers could get equivalent computing power at less cost by running a mixed shop. Hardware sales represented a major source of revenue to IBM therefore this loss of hardware sales to clone vendors was serious business to IBM executives.
On the large mainframe front, IBM had just canceled a project known as FS (Future Systems) that was going be a revolutionary replacement for the existing line of IBM mainframe computers. After several years of work, meetings, and more task force meetings and many train loads of paper specifications later the IBM management team determined the project was too ambitious even for IBM and IBM abandoned the project in 1975. (Does any one know the date this is a guess.)?
In the heart of the cornfields in Rochester, Minnesota the S/3 and its follow on systems S/32 and S/34 were nearing the end of a very successful life. The time for a replacement system was nearing. There was no hardware clone of the S/34 but the fear of a clone vendor replacing existing hardware was high on IBM management’s concerns. The news of FS demise was slow to reach Rochester so the group of planners, engineers and programmers continued on an independent approach to design the system for the future at least for the small and medium business customer. Every one wanted bigger, better, faster, and cheaper than the now aging S/34 but just how that would be accomplished was still an unknown.
Early in 1975, I joined the core group of less than 25 people who were developing the vision of the replacement for S/34. This small group of talented people with came from varied backgrounds. Some from the design teams of the S/3, S/32 and S/34 while others had large S/360 and S/370 backgrounds.
Abstract Machine Interface
When I joined the project there was a revolutionary idea being discussed. The words “objects”, “encapsulation” and abstract machine were new to most system programmers including me. Rather than having the instruction set of the computer being designed by the engineers, the programmers (the real users of the system) were defining an instruction set for an abstract machine. These instructions rather than be optomized for the hardware were being designed for writing software applications. This abstract machine interface would eventually become known as MI (Machine Interface) or what IBM currently calls TIMI (technology Independent Machine Interface). This abstract Machine Interface (MI) defined some very advanced concepts to make programming more efficient and less error prone. Programmers would write programs for an abstract machine using new concepts like not loading data into a hardware registers for processing. In this new abstract machine programmers would improve the reliability of programs by prevented programs from modifying storage to manipulate address registers. Using MI programs could manipulate pointers but the abstract machine would not allow a change that would cause a pointed to be corrupted. This abstract machine interface was being designed to protect programmers from them selves. On previous hardware the programmers knew the addressing structure of pointers and could even alter memory to change the address (which often was the cause of program reliability problems). On this abstract machine these pointers were encapsulated (actual pointer content was not presented to application programs) this meant that programs could not create a pointer by modifying storage and only pointer instructions could be used to modify pointers. Any attempt by the programmer to alter storage that contained a pointer would cause the pointer to become invalid. These abstract machine pointers were much larger than any hardware register and the system microcode would translate the use of pointers into actual hardware registers. The separation of the actual machine hardware from the abstract machine is responsible for the successful migration form CISC hardware to RISC hardware.
Other platforms are busy rewriting programs to get to 32 or 64 bit address (some
platforms are projecting completion in 2005). What is truly amazing is that every AS/400 application is 64 bit enabled without any modifications to customer or OS/400 programs.
A second major concept of this abstract machine interface was to move much of the traditional operating system functions (task management, security and even database) into the microcode. Since these functions are implemented close to
the hardware interface they could be programmed efficiently. More important to IBM was that this integration of operating system function into the hardware microcode would make copying of the S/38 hardware more difficult. This fact alone insured that the S/38 project got funding from IBM management.
Today our experience with AS/400 proves this abstract machine concept makes a lot of sense but to the system programmers of the 1970s this idea was radical.
THE BIG DEBATE
This idea of an abstract machine interface was not without its critics. There were several very qualified technical professionals that argued (with great passion) that such an abstract machine interface many be academically possible but would introduce so much overhead that the program execution would be poor. Much effort (and strong emotions) was being devoted to discuss and prove or disprove the performance of direct hardware or microcode implementations.I remember the day the issue was solved. A meeting of key technical staff was held one cold winter day in Rochester, MN. The proponents and opponents of the abstract machine were gathered in the conference room of Glen Henry the programming system manager. Glen was an extremely intelligent individual but like so many truly gifted people, he lacked polished human interaction skills.
When Glen Henry saw the obvious answer he told you so with few words and without any sugar coating. Glen proclaimed the end of the debate. With great emotion he explained that the abstract machine interface (later to be known as MI) was the only choice. Glen explained how this abstract machine would protect IBM investment from the clone hardware vendors. He concluded with “either get with the program or I will be happy to sign the transfer of any individual to another area”.
We can look back now and see that this abstract machine interface has been one of the most significant architectural advantages of the AS/400. Separation of the software from the actual hardware allowed radical change to the physical hardware with no modification to the software. Users of AS/400 were able to make the transition from CISC to RISC without any disruption in their applications because of the abstract machine interface.
Why the AS/400 Names are 10 Characters Long
Once the decision about the abstract machine was made then the less critical design decisions could be made. Since this was a new machine all of the design parameters were up for question. One of the decisions that had to be made was the length of object names. One proposal was an 8-character name that would be compatible with the S/34. Other proposed “self documenting names” of up to the 32 characters supported by the MI. The decision was made when the data base design team determined that they need to be able to store the file, library and member name in the 32 characters allowed by the MI. The 10-character name was determined by dividing the maximum name length (32) by 3 so S/38 names (and AS/400 names) are limited to10 characters long.
Selection of Command Naming
The consistency and structure of the CL command language is well recognized. The verb object structure of command names did not just happen but resulted from a problems discovered during the early development The commands were developed by individual teams of developers. They would name their commands in inconsistent ways and that the “terminology police” needed to impose very strict naming conventions of verb-object and use common abbreviations across the entire operating system. These consistent naming conventions were extended from the operating system to also include the Licensed Program Products. A verb object structure was not enough because it became apparent there was difficulty being able to recognize the breaks between abbreviations, so the standards were extended to standardized on 3 character abbreviations, with the exception of the last word in the command name. Thousands of AS/400users find the CL language easy to learn and used need to thank the efforts of the “terminology police” for selecting and enforcing command naming standards.
Prompter is Born
Usability tests showed that users could not remember all the command names and keyword and value naming. Initial efforts were to construct individual screens for each command and tried to keep them consistent. As the number of commands grew it was apparent that IBM couldn’t afford the cost of individually created screen interfaces this so then IBM planners tried to prioritize the commands, so there would be screens for some and not others. With the number of commands growing by the day, the user interface department could not develop all these screens or even keep track of them, therefore they proposed automated the generation of those screens based on the stored command definition objects.
The final result was the revolutionary command prompter which has proven invaluable to over 1500 IBM commands but is also available for customer commands. Using the question mark in front of a command name provides an easy way for CL programs to prompt for input.
Soon after the FS project was canceled by IBM management began to focus on the development efforts in Rochester. The question of the day was how could this small group in the corn fields of Minnesota (know by code name Silverlake) hope to accomplish what the massive FS design teams from the east and west coasts could not. A system review was held to determine the status of the Silverlake project. The future of the Silverlake project depended upon the outcome of this system review. Lead technical people from the former FS project reviewed the hardware and software design. All efforts in Rochester were focused on telling a good story about our progress. The detailed hardware and software plans were prepared and explained to the review team. Sometime the ink was not dry on the latest design when it was presented as fact to the review team. This was a time of major stress because the design team was aware that the fate of the system was in the hands of this review team. The Silverlake project survived the review and S/38 and AS/400 are the result.
The review served an important role in solidifying the design of the operating system. Prior to the system review, individual designers would change (“improve”) their designs on a weekly basis. The system review forced the design team to solidify and document design decisions.
The staffing of the engineering and programming group began to grow rapidly after the successful design review. The IBM facilities at the Rochester site could not hold all of the people so the complete Silverlake project (engineering and programming) was moved to an abandoned department store in Rochester. This kept the hardware and software design groups together and allows developers to walk down the hall to talk with designers in related areas. I believe this ease of communication was one of the reasons that the project in Rochester was successful where the multi-site attempts to implement FS failed.
There were three design teams (hardware, microcode, and operating system) all focusing on design and implementation. These design teams closely represented the machine architecture. The communication between these three different groups was excellent but often you thought that you were in a tower of babble. Different groups used the same data processing term (such as queue) to describe a similar concept but the details of the function were vastly different. This terminology conflict could be very confusing because when designers from different areas talked every one thought they understood but frequently there was often a vast difference in actual understanding. When the operating system programmers and engineers talked they often need to get one of the few individuals that could serve as a translator to explain the subtle differences.
The development of the hardware, microcode, and operating system was all proceeding in parallel. In the early phases of the project there was no hardware available for the programming team. A simulator of the machine hardware instructions was developed on the S/370. This simulator allowed the microcode developers to design and test their functions in the absence of physical hardware. The simulation of the hardware instructions was slow at best. I was the lead developer of the command analyzer and to simply syntax check a CL command using the simulator would take over 30-45 minutes when the computers were not loaded. If multiple simulations were running the same test would require up to 2 hours. As a result many other people and I started to come to work off-hours to get better response time. (If you can call 30 minutes to syntax check a command better response time.) Frequently the programs would fail, then it was a challenge to determine was it your code, microcode bug, or problem with the simulator. The programming language that was used to develop the operation system was PL/MI a derivative of PL/I. There was a similar language PL/S that developed code for the S/370 so our development team used the S/370 to do much of the initial program debug on the command analyzer. This was essential since the CL command analyzer needed to be done early so that it could be used by the development teams to the individual commands in S/38 operating system called Control Program Facility (CPF).
Testing on Hardware
Finally engineering began to make some of the hardware available and that became a new challenge. The tools used to step through code on the simulator did not exist on the hardware so you need to learn a whole new method of debugging a program. The debug tools did not exist during the early stages so setting hardware address stops became the method to test you programs. The most used program on the system was a utility call 2 to 1 copy. All of the microcode and operating system would fit on a single disk (Piccolo drive). The first step was a 15-30 minute copy of the entire system from the drive 2 to drive 1. Then you would add your programs and if you were lucky you would be able to add your programs to drive one and do a shut down of the system so you could then copy drive 1 back to drive 2. Frequently the system would hang and the only alternative was to start over running the 2 to 1 copy again. The copy of disk drives was running 80% of the time and only 20% being used for actual testing.
Every week the developers would get a new driver with the latest fixes and more of the microcode function complete. The save restore functions were not working so the driver build team would roll a disk drive up to your system connect the cables and run the famous 1 to 2 copy but this would be a reload of the complete
system. Then you would need to add any test programs to the system. Each week the system began to become more stable.
ITSWITCH saves the day
The most useful program other than the 2 to 1 copy was a tool call ITSWITCH. If it were not for this program the S/38 operating system, CPF would never have been completed. This program had many functions, setting the entry point table, determining the compile date and time of programs. The most critical function was the capability to trap an unexpected error and allow the operator to take manual recovery steps. There would be two terminals attached to each system one would be used for testing and the other to run ITSWITCH.
Since the command analyzer code was done early, I was able to move on the development of a demonstration of system function. This became the demonstration of S/38 function that was used in the first day announcement in
1978. The demonstration was a simple order entry application updating the database and displaying the results. By today’s standards it was a very simple application but during the development of CPF this was a big application.
Imagine running multiple terminals all accessing the same file and allowing updates to data files. Everything was carefully scripted to be sure that no order quantity ever reduced the amount on stock to less than zero or unexpected conditions occurred. The demonstration looked very impressive but it was fragile The system had a printer attached but if you attempted to print and do order entry at the same time the system would hang. The demonstration was carefully rehearsed to show users order entry and then move the audience to the printer to look at output (while no order entry was running). All the time ITSWITCH was monitored behind the scenes for a potential system failure.
S/38 Missed First Customer Ship
This application helped convince management that S/38 was ready for announcement before it was actually ready. As a result the approval for the announcement of the system was made and then the first customer ship date changed because the system reliability did not meet minimal standards. This was a black eye for the development team but an essential delay because customers did not have ITSWITCH to keep their system up a running.
Reflecting back on the S/38 on announcement day there were a lot of very elegant features, such as the integrated data base, subfiles and user defined CL commands and a CL language that could be compiled into a program. But there were a lot of features missing as well for example; the communications capability was not added until release 2.
One of the design mistakes the CL team made became apparent in during the development of CPF release 2. Adding a new keyword to an existing CL command required every program that used that command to be recompiled.
This caused a redesign of the method used by compiled CL programs that required that customers recompile all CL programs when release 2 was available but not in any follow on release. There was a lot of interest in S/38 but many customers had not made the purchase decision so only a few early adopters were impacted by the forced recompile of CL programs.
Copy File Interactive
In the first release of CPF IBM shipped what would not be called a wizard to simplify the copy file function. This was before the days of wizards. The copy file interactive function would prompt the user for a simplified copy file commands and based on the users response to those prompts other prompts were presented. This was implemented as multiple CL commands that would prompt the user for responses and simplify the CPFY command. As the number of options grew this alternative became impossible so CPYFINT (copy file interactive) was withdrawn in release 2. This is one of the few commands that has ever been withdrawn in the history of S/38 or AS/400.Program Size In the first release of S/38 the number of variables in a program were limited to 8192 by the MI architecture. This limit was thought to be larger than any reasonable program. Well soon after the S/38 was released to customers they began to convert large programs to S/38 and this limit was far to small. In release 2 the limit was significantly increased.
Other Interesting Thoughts
Main Memory Size
The cost of main memory was significant so the S/38 did not have the large memory sizes supported today on the AS/400. The early hardware systems were shipped with 512 K main memory and there was a significant design effort in Rochester to get the entry design point down to 256 K memory. We were never able to achieve this design goal. But in today’s Megabyte Memory World, the thought of running in 512K seems unreal. The performance of S/38 and AS/400 is significantly improved by addition of main memory an the architecture takes advantage of memory with no change to user applications. Today, it is hard to fathom that early systems could run (very slowly) in 512K.
Why Messages start with CPF
The AS/400 users may not recognize that the name of the S/38 Operating System was Control Program Facility (CPF). This abbreviation was used on all messages that user programs included in monitoring for messages. It became apparent that the number of messages required was larger than anticipated so additional variations using the starting two letters CP abbreviation. Next time you get a CPF message from the AS/400 recognize the early efforts of the talented design team of the S/38 that left the AS/400 a valuable heritage.
Much press was made about the successful conversion of S/36 and S/38 in record time to make the AS/400. In my view the technical challenges faced by the early developers of S/38 were much larger that those faced by the conversion of S/38 to AS/400. Most of the fundamental design ground rules for the architecture were already established by the S/38 and the AS/400 function could be developed and tested on the existing S/38 hardware.
To my former IBM collogues you may have a story that would help others understand the “Good Ole Days”. Send me a note and I would be delighted to add your thoughts to this article.
Wayne O. Evans
Why was S/38 not the successful system the AS/400 is?
I was in operating system development and not sales but I had a lot of contact with customers on the S/38. The question “Why did S/38 fail?” has been asked may times.
Here are my thoughts:
1. When the system (S/38) first as introduced it did not have a lot of available applications so that customers that want a solution in the box (turn key installation) would go to the S/36 which had many available applications. When AS/400 was announced there was a wealth of available applications
2. The first release of the S/38 system had no communications support. Remote communications was very limited and not available until release 2 of CPF. As a result, the system was limited to serving local workstations (5250s only). This was not state of the art in communications and many other systems had comparable or better support. When the AS/400 was announced it had full function communications because it was actually the next release of CPF renamed and repackaged.
3. Yes, the system was ahead of its time. The MI (machine interface) introduced an extra layer of confusion that most customers did not understand. The CPU and main store size (very significant) was restricted so the applications ran SLOW. By the time AS/400 was introduced the CPU and main storage and grown so performance was much improved.
4. The single level storage of the S/38 was very depended upon disk drives and power not failing. Should the S/38 go down unexpectedly, it could take 2 hours to 2 days to bring the system back. As a result the system got a bad image about reliability. Disk drives were prone to failure and the RAID technology did not exist so BACKUP, BACKUP, BACKUP was the order of the day.
THE REAL REASON
The marketing force of IBM had two solutions available for the small customer
- S/36 — Easy to install, good performance, many available applications,good usability reputation. EASY SELL
- S/38 — Required more computer knowledge, performance not best, limited application set, reported long recovery times on failures. Data Base was powerful but ahead of most customer needs. HARD SELL
The sales force would select the course of least resistance. When making a sale too often salesmen would choose the S/36 over the S/38.
The S/36 was less expensive though somewhat limited in capacity and easy for customers to install and get operational. Is was easy to sell the S/36 tosell a S/38 required education of customers. Sell customers the S/36 and move on rather than working to sell a S/38.
If the customer was larger, salesmen would often consider a S/370 solution because it had a growth path. The upper limit (capacity) of the S/38 was constrained by IBM to keep from impacting S/370 sales. The AS/400 was able to remove those limits so the growth of AS/400 processor power has not been constrained.
The S/370 and S/38 were comparable in price but the staff required to run the S/370 was much larger but customers would sometimes be sold S/370 simply because the salesmen knew more about that system that they did the strange box called a S/38.
When the AS/400 came out I considered it simply to be an extension of S/38 with the ease of use of S/36 added. The sales force was stuck with a S/36 that was old and limited in capacity or get on the band wagon with the AS/400. There was not other IBM solution so they encouraged customers to install the AS/400 had there been a S/36 follow on, I do not think the AS/400 would have been as successful in the early days.
Thanks for allowing me to think about the early days. The development team of S/38 was proud of what we had done but S/38 was truly before its time. We provided a state of the art computing system (data base, single level store, integrated security, new languages RPGIII, SEU, CL) when the customers want a business system (inexpensive, good performance, fast recovery after failure). Most customers did not want or need the powerful data base and advanced features of S/38 so for many years IBM considered dropping the S/38. Release 3 and 4 were also most never done. The development team and Rochester
management would fight to get out new improvements and keep the system going one more release. This is a case where the technical people were able to override the financial people in IBM and we kept the system alive until the AS/400 was announced.
The rest is history.
I wish you a secure life.
Hint: If your 400 is secure your life will be more secure.