A timeline spanning from 1966 to 2060 looking at breakthroughs from IBM using water to cool computers.
Created by ibm_research on 17/04/2012
Last updated: 15/08/12 at 09:41
Tags: supercomputing IBM
We address integration density in future computers based on packaging and architectural concepts of the human brain: a dense 3-D architecture for interconnects, fluid cooling, and power delivery of energetic chemical compounds transported in the same fluid with little power needed for pumping. Several efforts have demonstrated that by vertical integration, memory proximity and bandwidth are improved using efficient communication with low-complexity 2-D arrays. However, power delivery and cooling do not allow integration of multiple layers with dense logic elements. Interlayer cooled 3-D chip stacks solve the cooling bottlenecks, thereby allowing stacking of several such stacks, but are still limited by power delivery and communication. Electrochemical power delivery eliminates the electrical power supply network, freeing valuable space for communication, and allows scaling of chip stacks to larger systems beyond exascale device count and performance. We find that historical efficiency trends are related to density and that current transistors are small enough for zetascale systems once communication and supply networks are simultaneously optimized. We infer that biological efficiencies for information processing can be reached by 2060 with ultracompact space-filled systems that make use of brain-inspired packaging and allometric scaling laws.
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6044603&contentType=Journals+%26+Magazines&sortType%3Dasc_p_Sequence%26filter%3DAND%28p_IS_Number%3A6032764%29
Estimated timeframe 2024.
One of the main challenges to the continued improvement of integration density of information processing equipment is the removal of dissipated heat. This is caused by the increased power density resulting from smaller transistors and faster clock speeds. Thermophysical research helps to improve heat transfer from the transistor junctions through the chip and interfaces, and eventually to the ambient air or the coolant circuit. With microfabrication techniques we create interfaces and coolers with high aspect ratio patterns. Hierarchical nested structures use principles ubiquitous in nature to reduce the pressure drop that occurs when paste is squeezed out of gaps, thus creating high-performance thermal interfaces, as well as for liquid cooling with many tens of thousands of capillaries or jets.
Interlayer cooling was proved to be an effective heat removal concept that scales with the number of tiers in vertically integrated packages, therefore relaxing electrical constraints substantially to heat flux levels of 250 W/cm2. It also enables on-stack integration of optical communication as well as high-power RF devices, resulting in high-performance heterogeneous chip packages. The use of water as a coolant will minimize cooling power dissipated in data centers with respect to the inefficient air-cooling methods utilized today. Furthermore it makes high-grade heat available for reuse in residential heating systems. This reduces the effective CO2 emission of data centers to a minimum.
http://www.zurich.ibm.com/st/cooling/
Estimated time 2022.
The heat-removal capability of area-interconnect-compatible interlayer cooling in vertically integrated, high-performance chip stacks was characterized with de-ionized water as coolant. Correlation-based predictions and computational fluid dynamic modeling of cross-flow heat-removal structures show that the coolant temperature increase due to sensible heat absorption limits the cooling performance at hydraulic diameters ≤200 μm. An experimental investigation with uniform and double-side heat flux at Reynolds numbers ≤1,000 and heat transfer areas of 1 cm2 was carried out to identify the most efficient interlayer heat-removal structure. The following structures were tested: parallel plate, microchannel, pin fin, and their combinations with pins using in-line and staggered configurations with round and drop-like shapes at pitches ranging from 50 to 200 μm and fluid structure heights of 100–200 μm. A hydrodynamic flow regime transition responsible for a local junction temperature minimum was observed for pin fin in-line structures. The experimental data was extrapolated to predict maximal heat flux in chip stacks having a 4-cm2 heat transfer area. The performance of interlayer cooling strongly depends on this parameter, and drops from >200 W/cm2 at 1 cm2 and >50 μm interconnect pitch to
http://www.springerlink.com/content/l6q7m6263013ux25/
Estimated time 2018
We address integration density in future computers based on packaging and architectural concepts of the human brain: a dense 3-D architecture for interconnects, fluid cooling, and power delivery of energetic chemical compounds transported in the same fluid with little power needed for pumping. Several efforts have demonstrated that by vertical integration, memory proximity and bandwidth are improved using efficient communication with low-complexity 2-D arrays. However, power delivery and cooling do not allow integration of multiple layers with dense logic elements. Interlayer cooled 3-D chip stacks solve the cooling bottlenecks, thereby allowing stacking of several such stacks, but are still limited by power delivery and communication. Electrochemical power delivery eliminates the electrical power supply network, freeing valuable space for communication, and allows scaling of chip stacks to larger systems beyond exascale device count and performance. We find that historical efficiency trends are related to density and that current transistors are small enough for zetascale systems once communication and supply networks are simultaneously optimized. We infer that biological efficiencies for information processing can be reached by 2060 with ultracompact space-filled systems that make use of brain-inspired packaging and allometric scaling laws.
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6044603&contentType=Journals+%26+Magazines&sortType%3Dasc_p_Sequence%26filter%3DAND%28p_IS_Number%3A6032764%29
SuperMUC, the fastest supercomputer in Europe and fourth fastest in the world, officially went live on Friday, 20 July during a gala event at the Leibniz Rechenzentrum near Munich, Germany.
http://ibmresearchnews.blogspot.ch/2012/07/supermuc-gets-powered-up.html
SuperMUC will provide compute power in the Petaflop/s regime. With the operation of SuperMUC, LRZ will act as an European Centre for Supercomputing and will be Tier-0 centre of PRACE, the Partnership for Advanced Computing in Europe.
SuperMUC will also use a new revolutionary form of warm water cooling developed by IBM. Active components like processors and memory are directly cooled with water that can have a temperature of up to 45 degrees Celsius. The "High Temperature Liquid Cooling" together with very innovative system software promises to cut the energy consumption of the system. In addition, all LRZ buildings will be heated re-using this energy.
http://www.lrz.de/services/compute/supermuc/systemdescription/
Announced at CeBIT, the IBM System x iDataPlex dx360 M4, offers significant flexibility for supporting a wide range of high-performance computing workloads. The system is designed for datacenters requiring high performance with limited floor space, and power and cooling infrastructure. Available with specially designed exclusive IBM warm water-cooled technology based on the original Aquasar design from IBM Research - Zurich and IBM Boeblingen, this iDataPlex system is up to 40 percent more energy efficient than air-cooled systems, recovering 90 percent of heat per nodes.
http://www.ibm.com/systems/info/x/idataplex/
IBM has delivered a first-of-a-kind hot water-cooled supercomputer to the Swiss Federal Institute of Technology Zurich (ETH Zurich), marking a new era in energy-aware computing. The innovative system, dubbed Aquasar, consumes up to 40 percent less energy than a comparable air-cooled machine. Through the direct use of waste heat to provide warmth to university buildings, Aquasar's carbon footprint is reduced by up to 85 percent.
Building energy efficient computing systems and data centers is a staggering undertaking. In fact, up to 50 percent of an average air-cooled data center's energy consumption and carbon footprint today is not caused by computing but by powering the necessary cooling systems to keep the processors from overheating – a situation that is far from optimal when looking at energy efficiency from a holistic perspective.
The development of Aquasar began one year ago as part of IBM's First-Of-A-Kind (FOAK) program, which engages IBM scientists with clients to explore and pilot emerging technologies that address business problems. The supercomputer consists of special water-cooled IBM BladeCenter® Servers*, which were designed and manufactured by IBM scientists in Zurich and Boblingen, Germany. For direct comparison with traditional systems, Aquasar also holds additional air-cooled IBM BladeCenter® servers. In total, the system achieves a performance of six Teraflops** and has an energy efficiency of about 450 megaflops per watt. In addition, nine kilowatts of thermal power are fed into the ETH Zurich's building heating system. With its innovative water-cooling system and direct utilization of waste heat, Aquasar is now fully-operational at the Department of Mechanical and Process Engineering at ETH Zurich.
http://www.ibm.com/press/us/en/pressrelease/32049.wss
QPACE (QCD Parallel Computing on the Cell Broadband Engine) is pursuing the development of a massive parallel, scalable supercomputer for applications in lattice quantum chromodynamics (QCD). The machine structure is a three-dimensional torus of identical processing nodes, based on IBM's PowerXCell 8i processors. These nodes are tightly coupled by an Xilinx Virtex-5-based FPGA, application-optimized network processor attached to the PowerXCell 8i processor.
The three Identical QPACE supercomputers at Jülich Research Centre, University of Regensburg and University of Wuppertal are as of November 2009 topping the Green500 list of most energy efficient supercomputers in the world[1][2] and are at the same time ranking at 110, 111 and 112 place on the Top500 list of most powerful supercomputers.
Each of the QPACE installations comprise 4608 3.2 GHz PowerXCell 8i processors housed in four racks, reaching sustained performance of 43 TFLOPS while drawing 59.49 KW.
http://www.fz-juelich.de/portal/EN/Research/InformationTechnology/Supercomputer/QPACE.html
IBM Corp. has rolled out its first water-cooled Unix server, the Power 575 supercomputer, which includes water-chilled copper plates on top of the processors' heat sinks.
It includes 14 servers that include 16 dual-core 4.7 GHz Power6 processors each, with a total of 224 processors and 448 cores.
Up to 256 GB of RAM per server, for a total of about 3.5 TB of memory in a single server cabinet.
Water piping that threads through the rack and into the server, including water-chilled copper plates that sit on the processor heat sinks.
The ability to run AIX, Novell SUSE Linux Enterprise Server, and Red Hat Enterprise Linux.
http://www-03.ibm.com/systems/nz/power/hardware/575/index.html
3D-rendered image of IBM's "Zero Emission Data Center" showcase at CeBIT 2008. IBM researchers have introduced an intelligent water cooling circuit for computer chips that revolutionizes energy management in data centers. Compared to today's air-cooled data centers, the zero emission concept not only reduces energy consumption by 40%, but also makes waste heat available for direct reuse, i.e. for heating homes. Thereby, the data center can virtually operate with a zero emission footprint. The first prototype system already reuses three quarters of the energy needed to operate the data center. The showcase demonstrates in simplified models the cooling system and energy consumption for the IT and cooling of a conventional air-cooled data center (left) with the prototype of the new "Zero Emission Data Center" (right).
The goal is to reuse the waste heat generated for heating buildings, swimming pools, or into a municipal heating network. A requirement for the technology is waste-heat temperature must be above a certain threshold, which, for modern municipal heating networks, is about 50°C.
To accomplish this, researchers used water, which conducts heat by up to 4000 times better than air. Using water also enables cooling efficiency improvements of several orders of magnitude. Using hot (45°C) water, the cooling system cools a chip to the typical operating temperature of 85 °C, causing the cooling water to reach a temperature of more than 50 °C, and the water can then be used directly to provide heat to a subsequent user.
The entire cooling system is a closed circuit in which the cooling water is constantly heated by the chip and then cooled to the required temperature by delivering heat to a subsequent user. This not only eliminates the use of energy-intensive cooling systems, it also substantially reduces the data center’s power consumption.
The first prototype of the zero-emission data center already reuses about 0.75 of the electrical energy needed for IT operation and reallocates it, for example, to heat buildings. This corresponds to a capacity to heat up to 70 homes, in addition to a 40% reduction in energy consumption for a typical 1-MW data center.
http://www.eebeat.com/?p=794
At the IEEE Semi-Therm Conference 2007, IBM researchers unveiled details on a new technique to significantly increase the ability to cool computer chips.
The technique, developed by a team of scientists at IBM Research - Zurich in co-operation with Momentive Performance Materials, formerly GE Advanced Materials, overcomes a barrier in chip cooling by improving the application of the "glue" that binds chips to their cooling systems. The new technology could allow for faster computer chips to be cooled more efficiently.
In today's computer chips, as the circuits on chips get smaller and smaller, the chip puts out more heat than ever before. To remove the heat from the chip, a cooling system is attached to the microprocessor using a special adhesive or glue. This glue is necessary to bind the two systems together, yet it poses a real barrier in heat transport.
To improve the glue's heat-conducting properties, it is enriched with micrometer-sized metal or ceramic particles. These particles form clusters and build "heat-evacuation bridges" from the chip to the cooler to make up for the glue's shortcomings. However, even highly particle-filled pastes are still inefficient, consuming up to 40 percent of the overall thermal budget, i.e. of the cooling capacity available to draw the heat away.
IBM researchers now unveiled the reason and presented a novel technique to solve this problem. By observing how the glue spreads when attaching a chip with its cooling element, the scientists noticed a cross forming in the paste, where large numbers of particles were pilling up, inhibiting the ability to thin out the layers of glue. The scientists were able to trace the cause of this back to the flow behavior of the paste, which simply follows the path of least resistance. Along the diagonals, the particles are pulled in opposite directions and as a result they do not move anywhere and pile up on each other as the squeezing process continues — forming the "magic cross".
To overcome this problem, the team designed a special layout of micrometer-sized channels — or trenches — in a tree-like branched structure, consisting of larger and smaller channels, which functions like an irrigation system for the paste at exactly those spots where the particles would pile up. This allows the particles to spread more homogeneously, and reduces the thickness of the resulting paste gap.
The results obtained are impressive: The paste thickness could be reduced by a factor of three, and the pressure needed to squeeze the paste to the same bondline thickness could be reduced by a similar factor. These lower assembly pressures ensure that the delicate components and interconnects below the chip are not damaged as the chip package is created. The channels also allow pastes with higher fill factor and higher bulk thermal conductivity to be squeezed to thinner gaps, thereby reducing the thermal resistance of the paste interface considerably by more than a factor of three. The new technology allows air-cooling systems to remove more heat and helps to improve the overall energy efficiency of computers.
To further optimize the technology in real cooling systems and to demonstrate its feasibility, the IBM team co-operated with paste manufacturer Momentive Performance Materials, Wilton, CT, USA.
Together with other industry-leading suppliers tools are developed to define the surface channels through the same copper stamping process currently used to fabricate high volume chip lids. This will define a full supply chain of low-cost parts to quickly integrate the new technique into products.
The work is being published in the paper "Hierarchical Nested Surface Channels for Reduced Particle Stacking and Low-Resistance Thermal Interfaces" by R. J. Linderman, T. Brunschwiler, U. Kloter, H. Toy, B. Michel, Proc. 23rd IEEE SEMI-THERM Symp., 2007.
http://www.zurich.ibm.com/news/07/cooling.html
A Power and Cooling Summit here today, IBM (NYSE: IBM) researchers presented an innovative approach for improving the cooling of computer chips, an increasingly urgent need given the large amount of heat released by today's more powerful processors and the additional energy required for removing that heat.
The technique, called "high thermal conductivity interface technology," allows a twofold improvement in heat removal over current methods. This paves the way for continued development of creative electronic products through the use of more powerful chips without complex and costly systems simply to cool them.
As chip performance continues to progress according to Moore's Law, efficient chip cooling has become one of the most vexing problems for designers of electronic products. The IBM technique outlined today is one of several being explored by scientists from IBM Research - Zurich to address the issue.
"Electronic products are capable of amazing things, largely because of the more powerful chips at their heart," said Bruno Michel, manager of the Advanced Thermal Packaging research group at IBM's Zurich lab. "We want to help electronics makers keep the innovations coming. Our chip-cooling technology is just one tool at our disposal to help them do that."
The approach used by IBM addresses the connection point between the hot chip and the various cooling components used today to draw the heat away, including heat sinks. Special particle-filled viscous pastes are typically applied to this interface to guarantee that chips can expand and contract owing to the thermal cycling. This paste is kept as thin as possible in order to transport heat from chip to the cooling components efficiently. Yet, squeezing these pastes too thin between the cooling components and chip would damage or even crack the chip if the conventional technologies are used.
Using sophisticated micro-technology, the IBM researchers developed a chip cap with a network of tree-like branched channels on its surface. The pattern is designed such that when pressure is applied, the paste spreads much more evenly and the pressure remains uniform across the chip. This allows the right uniformity to be obtained with nearly two times less pressure, and a ten times better heat transport through the interface.
This unique and extremely powerful design for chip cooling is borrowed from biology. Systems of hierarchical channels can be found manifold in nature, e.g. tree leaves, roots, or the human circulatory system. They can serve very large volumes with little energy, which is crucial in all organisms larger than a few millimeters. Ancient water irrigation systems also used the same approach.
The demonstrated prototype is part of a large effort within IBM's Research and Development organizations to improve cooling performance of next and future generations of computer systems.
The cooling bottleneck results from the demand for ever more powerful computer chips and becomes one of the most severe constraints of overall chip performance. Today's high-performance chips already generate a power density of 100 Watts per square centimeter — one order of magnitude more than that of a typical hotplate. Tomorrow's chips may attain even higher power densities, which would create surface temperatures close to that of the sun when not cooled (approx. 6000 °C). Current cooling technologies, mainly based on forced air convection (fans) blowing across heat sinks with densely spaced fins, have essentially reached their limits with the current generation of electronic products. To make matters worse, energy needed to cool computer systems is rapidly approaching the power used for calculations, thus almost doubling the overall power budget.
"Cooling is a holistic challenge from the individual transistor to the datacenter. Powerful techniques, brought as close as possible to the chip right where the cooling is needed, will be crucial for tackling the power and cooling issues," states Michel.
Looking beyond the limits of air-cooling systems, Zurich researchers are taking their concept of branched channel design even further and are developing a novel and promising approach for water-cooling. Called direct jet impingement, it squirts water onto the back of the chip and sucks it off again in a perfectly closed system using an array of up to 50,000 tiny nozzles and a complicated tree-like branched return architecture.
By developing a perfectly closed system, there is also no fear of coolant getting into the electronics on the chips. What's more, the IBM team was able to enhance the cooling capabilities of the system by devising ways to apply it directly to the back of the chip and thereby avoiding the resistive thermal interfaces in between the cooling system and the silicon.
First lab results are impressive. The team has demonstrated cooling power densities of up to 370 Watts per square centimeter with water as coolant. This is more than six times beyond the current limits of air-cooling techniques at about 75 Watts per square centimeter. Yet, the system uses much less energy for pumping than other cooling systems do.
http://www.zurich.ibm.com/news/06/cooling.html
The IBM Enterprise System/9000 (ES/9000) provides the most extensive computing range ever offered within a single processor family. Based on IBM's newest technologies and proven systems architecture, this powerful processor family provides over 100-fold growth from the smallest rack-mounted systems to top-of-the-line, general-purpose computers.
The result is a new era in computing that redefines management of the computing environment and lifts computing power to new levels of performance, designed to generate new and better solutions to specific business needs.
The IBM ES/9000 family includes eight water-cooled models: the 330, 340, 500, 580, 620, 720, 820 and 900.
http://www-03.ibm.com/ibm/history/exhibits/mainframe/mainframe_2423PH9000.html
The IBM Model 3081 Processor Complex was a mainframe computer which was announced November 12, 1980 and withdrawn August 4, 1987. It introduced the System/370 Extended Architecture. (IBM c. 1982) It consisted of a 3081 Processor Unit and supporting units; the 3083 and 3084 were in the same family. (IBM 1982; Pittler, Powers, and Schnabel 1982).
Some key technological features of the 3081—compared to the previous most powerful processor, the 3033—were the following:
About 800,000 circuits implemented in Large Scale Integration, using up to 704 logic circuits per chip, which provided the required performance, reliability, and serviceability that were design goals.
"Elimination of one complete level of packaging—the card level." (p. 2)
Water cooling, which provides heat removal from chips beyond the ability of conventional air cooling.
A machine cycle time of 26 nanoseconds.
Reduced power consumption, 23 kilowatts for a 3081-D16 versus 68 kilowatts for a 3033-U16.
Approximately double the instruction-execution rate of the 3033.
Two central processor components. (Pittler, Powers, and Schnabel 1982, 2, 3, 8)
Both central processors have access to channels (as many as 24), and main memory (up to 32 megabytes[1]). (Pittler, Powers, and Schnabel 1982, 3, 4)
The elimination of a layer of packaging was achieved through the development of the thermal conduction module (TCM), a flat ceramic module containing about 30,000 logic circuits on up to 118 chips. The TTL chips (which were not compatible with the TTL chips sold on the open market by many manufacturers) were joined face-down (sometimes called "flip chip") to the TCM with an array of 11 x 11 solder pads. The TCM contains 33 metalized layers which distribute signals and power. "A module is connected to the next level of packaging through 1800 pins (1200 are available are available for signals, 500 pins are available for power, and 100 pins are spare)." (p. 7) The module is fitted with a helium-filled metal cap, which contains one piston per chip; the piston presses against the back of each chip to provide a heat conduction path from the chip to the cap. A water-cooled cold plate is attached to the cap; the water temperature is approximately 24 °C. This arrangement provides cooling of the module heat flux on the order of 105 watts per square meter, which is about a tenfold increase over the 3033 processor. (Pittler, Powers, and Schnabel 1982, 4–8)
The internal code name of the 3081 was Adirondack.
http://en.wikipedia.org/wiki/IBM_3081
The IBM System/360 Model 91 was introduced in 1966 as the fastest, most powerful computer then in use. It was specifically designed to handle high-speed data processing for scientific applications such as space exploration, theoretical astronomy, subatomic physics and global weather forecasting. IBM estimated that each day in use, the Model 91 would solve more than 1,000 problems involving about 200 billion calculations.
The system's immense computing power resulted from a combination of several key factors, including advanced circuits that switched in billionths of a second, high-density circuit packaging techniques and a high degree of "concurrency," or parallel operations.
To users of the time, the Model 91 was functionally the same as other large-scale System/360s. It ran under Operating System/360 -- a powerful programming package of approximately 1.5 million instructions that enabled the system to operate with virtually no manual intervention. However, the internal organization of the Model 91 was the most advanced of any System/360.
Within the central processing unit (CPU), there were five highly autonomous execution units which allowed the machine to overlap operations and process many instructions simultaneously. The five units were processor storage, storage bus control, instruction processor, fixed-point processor and floating-point processor. Not only could these units operate concurrently, they could also perform several functions at the same time.
Because of this concurrency, the effective time to execute instructions and process information was reduced significantly.
The Model 91 CPU cycle time (the time it takes to perform a basic processing instruction) was 60 nanoseconds. Its memory cycle time (the time it takes to fetch and store eight bytes of data in parallel) was 780 nanoseconds. A Model 91 installed at the U.S. National Aeronautics & Space Administration (NASA) operated with 2,097,152 bytes of main memory interleaved 16 ways. Model 91s could accommodate up to 6,291,496 bytes of main storage.
With a maximum rate of 16.6-million additions a second, NASA's machine had up to 50 times the arithmetic capability of the IBM 7090.
In addition to main memory, NASA's Model 91 could store over 300 million characters in two IBM 2301 drum and IBM 2314 direct access storage units. It also had 12 IBM 2402 magnetic tape units for data analysis applications, such as the processing of meteorological information relayed from satellites. Three IBM 1403 printers gave the system a 3,300-line a minute printing capability. Punched card input/output was provided through an IBM 2540 card read punch.
The console from a Model 91 has been preserved in the IBM Collection of Historical Computers, and is exhibited today in the IBM Technology Gallery in the company's corporate headquarters in Armonk, N.Y.
http://www.columbia.edu/cu/computinghistory/36091.html
Update Sources