|
|
|
Computer Network SupportA network is a group of computers, printers, and other devices that are connected together with cables. The sharing of data and resources. ..... Microsoft Exchange Server 2003 represents an important, continuing investment in enterprise technology for Microsoft. Exchange 2003 offers improvements required by enterprise messaging and collaboration customers. Many of the largest companies in the world run their messaging systems on Microsoft Exchange, including Microsoft.
The purpose of this document is to provide an overview of the architecture and design decisions made during the upgrade of Exchange Server 2003 at Microsoft. The paper focuses on the hardware selection and configuration aspects of the project. It also includes discussions on the key technology wins and best practices that emerged from the upgrade. Since OTG is a leading edge implementer of Microsoft technologies and products, the organization brings a unique set of requirements as well as innovative approaches to meeting the needs of its customers. This paper describes these requirements and approaches, as well as the way they affected design decisions for the deployment. The intended audience for this white paper includes technical decision makers, system architects, IT implementers, and messaging system managers.
OTG based its mission for migrating from Exchange 2000 to Exchange 2003 on achieving several objectives:
* To test and improve the product before Microsoft offered it to its customers.
* To consolidate Exchange server sites worldwide to reduce server maintenance and administration costs and workload.
* To simplify the messaging infrastructure based on standardized server and storage hardware for all deployment locations.
* To improve the ability of OTG to meet its SLA obligations for data backup and restore.
* To significantly improve the end-user experience with messaging services at Microsoft.
OTG met all these objectives when it deployed Exchange 2003.
Overview of Current Network Infrastructure
With all of the beta-level and test version software used in its production environment, the Microsoft corporate network is the world’s largest experimental computer network. The network is a confederation of functional backbones, spanning the globe. Each backbone is defined on regional boundaries with connectivity focused on the Main corporate campus located in the Puget Sound Metropolitan Area.
The network is architected following a multi-domain routing model. It is divided into four regional networks, with each network functioning as a single Open Shortest Path First (OSPF) routing and addressing domain. The four regions cover the following areas: 1. the Puget Sound metropolitan area in western Washington State; 2. Europe, Africa, and the Middle East; 3. Japan, the Pacific Rim, and the South Pacific, and 4. the remainder of North America and South America.
Each regional network consists of a backbone area (Area 0) and multiple areas to ensure scalability of each regional network. External Border Gateway Protocol (EBGP) is used to exchange routes between the regional networks to ensure the scalability of the network as a whole.
The Puget Sound Metropolitan Area Network (MAN) supports the bulk of data traffic on the global enterprise network providing gigabit rate connectivity between buildings and the main datacenters located in the area. The current campus is comprised of 70 separate buildings and two datacenters with a network infrastructure providing access to corporate resources, developer lab networks, and Internet connectivity to any location within the campus.
This network relies on Gigabit Ethernet and Packet over Synchronous Optical Network (SONET), using privately owned or leased Dark Fiber as the transport medium. In the metro area, efficient use of limited fiber resources is realized by leveraging Wave Division Multiplexing (WDM) technologies to provision multiple circuits across a single physical link.
The available network bandwidth is significant for applications like Exchange Server 2003 and site-to-site connectivity. As of June 2003, the network had grown to encompass:
* Three enterprise data centers, nineteen regional data centers worldwide
* 310 sites in approximately 230 cities in 77 countries
* The largest wireless LAN (802.1x EAP-TLS) in the world
* More than 24,000 wireless devices
* More than 4,000 wireless access points
* More than 250 wide area network (WAN) circuits
* More than 200 WAN sites in more than 70 countries
* More than 3,300 IP subnets
* More than 2,000 routers
* More than 2,600 network layer 2 switches
* More than 275 ATM switches
* More than 10,000 world wide servers
* More than 350,000 LAN ports
Overview of Current Messaging Infrastructure
Managing the complex messaging infrastructure at Microsoft is a team effort that involves many different groups within OTG. Organizationally, OTG is comprised of more than 2,500 staff members that are responsible for operations spanning more than 400 IT locations worldwide. In addition to providing the IT utility for the company, OTG plays a key role in helping Microsoft meet its main business objective of software development and marketing. As the first and best customer of Microsoft, OTG serves as an early adopter of new Microsoft software, such as Windows Server 2003, Microsoft Office 2003, and Exchange Server 2003. The result of this process is known in the industry as “eating your own dog food.”
In the “dog food” messaging environment of OTG, servers regularly receive software patches, operating system test releases and upgrades, Exchange server test releases and upgrades, and more. Each Exchange server is “touched” by OTG for these software upgrades on an average of two times each month. The changes to software are implemented to test new scenarios, meet specific requirements, and continually run the latest application concepts through real world, enterprise-level testing. The rate of change is very high in OTG.
Microsoft employees place a significant load on the messaging infrastructure. The average employee at Microsoft possesses three computers, typically all of which are used to synchronize with Exchange. In addition, a significant portion of that population also carries Pocket PC and Smartphone devices that also synchronize with Exchange. The average Remote Procedure Call (RPC) operations per second (a measurement of work) at Microsoft is significantly higher than at any other company known to OTG. Microsoft often works with customers and partners to benchmark their messaging infrastructure. The workload managed by the Exchange servers at Microsoft is typically more than double than the load measured at these companies.
At the time of this writing, the messaging environment at Microsoft consists of more than 200 servers, including 190 Exchange 2003 servers (113 of which are mailbox servers) in 75 locations worldwide, including servers in additional cross-forest test environments. This environment supports:
* Global mail flow of 6,000,000 messages per day, with 2,500,000 average Internet e-mail messages per day, 70 percent of which is filtered out as either unwanted spam e-mail, virus-infected, or to invalid e-mail addresses. Comparing bytes over the wire, the size ratio of blocked message content versus accepted message content received at Microsoft is 40:1. The average size of a typical e-mail message is 44 KB.
* Approximately 85,000 mailboxes, each being increased from a 100 MB to 200 MB limit. Average 100 MB mailbox was only 44 MB in size.
* More than 85,500 distribution groups.
* More than 230,000 unique public folders managed on public folder servers.
The OTG server infrastructure includes:
* Corporate standard client configuration comprised of Windows® XP Professional and Microsoft Office Outlook® 2003.
* Legacy, stand-alone mailbox server configurations of 500, 1,000, or 1,500 mailboxes on stand-alone servers. Stand-alone servers are being replaced by clustered SAN solutions worldwide and have been scaled per server to support 2,700 user mailboxes in regional locations and 4,000 user mailboxes in the headquarters data center.
* One centrally located support organization in headquarters supports all Exchange servers worldwide.
* In addition to the Main corporate Exchange Active Directory® forest, three additional forests are used to host Exchange mailbox servers at Microsoft:
o A Level A Test forest dedicated that runs development and test code for Exchange, operating in a frequently changing server software environment.
o A specialized Level B Test forest, serving as a limited-use production environment used by one product division that hosts a limited number of user mailboxes. Specialized hardware configurations and test scenarios can be run in this environment. Level B Test uses a two-node server cluster connected to a SAN scaled to support 5,000 user mailboxes.
o A legacy test environment forest that is used for testing Windows server operating system versions one version back from the currently released version (specifically Windows 2000 Service Pack-specific testing) with Exchange.
Note OTG uses both Level A Test and Level B Test forests to test cross-forest behavior and support with the Main Microsoft corporate production forest.
The OTG service levels include:
* The global service availability Service Level Agreement (SLA) goal in the Main corporate forest, calculated as the availability of mailbox databases per minute (including both planned and unplanned outages), was 99.9 percent for stand-alone server designs. This was increased to 99.99 percent for the new clustered server designs used with Exchange 2003.
* Worldwide e-mail delivery in less than 90 seconds, 95 percent of the time.
* Backup and restore operation SLA of less than one hour per database.
Note For security reasons, the sample names of forests, domains, internal resources, and organizations used in this paper are fictitious. They do not represent real resource names used within Microsoft and they are in this document for illustration purposes only.
Sites and Locations
Following the lead of the Exchange 2000 deployment, OTG continued the strategy of deploying Exchange servers in dedicated roles. Table 1 shows the distribution of Exchange 2003 servers by server role. OTG grouped the Exchange 2003 servers into 37 Exchange routing groups that were interconnected with 79 site connectors.
Table 1 Exchange 2003 Server Distribution by Server Role at Microsoft
Server Role Exchange 2002 Exchange 2003 (post-consolidation goal *)
Mailbox 113 38
Public Folder 20 11
Messaging Hub 12 7 **
Instant Messaging 4 0 ***
Internet Gateway 22 18
Dedicated Free/Busy 6 0 ****
Front-End ***** 14 12
Antivirus 9 7
* The mailbox server consolidation project is slated to be completed as of the end of the calendar year 2003.
** OTG will set up seven messaging hubs and four additional dual-purpose servers that will provide messaging hub services.
*** Exchange Instant Messaging servers will be eliminated as the messaging service is migrated to Windows Real Time Communications (WinRTC) servers.
**** All of the Free/Busy server services will be provided by existing Public Folder servers. OTG will not set up any dedicated Free/Busy servers at Microsoft.
***** Front-End servers were consolidated with the deployment of Exchange 2003 since the technology formerly included in Mobile Information Server (MIS) 2002 product was added into Exchange 2003. To increase system availability, each Exchange 2003 front-end server deployment site was configured with a pair of load-balanced servers.
Routing Group and Administrative Group Structure
In all Exchange deployments prior to Exchange 2000 (including versions 4.0, 5.0, and 5.5), OTG grouped Exchange servers into sites based on the network topology. For Exchange 5.5, OTG designed the environment to strike a balance between the need for large sites and the limitations of network bandwidth within those sites because of directory and public folder replication and message routing traffic.
Since the release of Exchange 2000 on Windows 2000, the limits and boundaries imposed by the Exchange 5.5 model were no longer a concern. The ability to place servers in routing groups independent of their administration group membership allowed OTG to optimize the routing topology without losing the advantages of large administrative groups.
Directory replication is now a function of Active Directory and is an operating system-level issue that is no longer a key concern of the Exchange deployment. Since routing groups and administrative groups need not be the same (as was the case in Exchange 5.5 and earlier versions), the OTG Messaging operations staff is free to place Exchange 2003 servers into groups that match their administrative and operational structure, and into routing groups that match the WAN topology. This leaves directory replication concerns to another OTG team specifically focused in that area. As of this writing OTG maintains 31 Exchange Server 2003 routing groups and 11 administration groups.
Exchange 2000 Legacy Architecture Back to Top
OTG began its deployment of Exchange 2003 when the product was still in an early beta version. To fully grasp the scope of this project, let us review the previous messaging infrastructure under Exchange 2000, the compelling reasons why OTG had to upgrade to Exchange 2003, and what OTG did to make the upgrade a success. Various challenges and discoveries made by OTG during this experience are included to provide some guidance and considerations as you plan your Exchange 2003 deployment.
Overview of Exchange 2000 Infrastructure
The Microsoft Exchange Server platform is the fastest selling Microsoft server product in history. Since 1996, when Exchange 4.0 was released, Exchange Server has sold more than 50 million seats. Table 2 provides an overview of the evolution of the internal deployment of Exchange Server at Microsoft since 1996 when Microsoft first released Exchange Server.
Table 2 The Evolution of Exchange Server Deployment at Microsoft
Exchange 4.0 Exchange 5.0 Exchange 5.5 Exchange 2000 Exchange 2003
Mailboxes/Server 305 305 1,024 3,000 4,000
Mailbox Size/User 50 MB 50 MB 50 MB 100 MB 200 MB
Restore Time/Database ~12 hours ~12 Hours ~8 Hours ~1 Hour ~25 minutes *
Total number of Mailboxes ~32,000 ~40,000 ~50,000 ~71,000 ~85,000
* It takes 25 minutes to restore a database from backup disks.
Legacy Server and Storage Design
OTG used stand-alone servers in both the headquarters data center and in all regional deployments. The servers were categorized into four basic mailbox server configurations as shown in Table 3.
Table 3 Microsoft OTG Exchange 2000 Server Configurations
Exchange 2000 Server Configuration Mailboxes
Small Configuration Regional Mailbox Server 500
Medium Configuration Regional Mailbox Server 1,000
Large Configuration Regional Mailbox Server 1,500
Data Center Configuration Mailbox Server 3,000
The storage design varied depending upon the requirements of each server configuration. All Exchange 2000 mailbox servers supported 100 MB mailboxes. The regional server configurations used direct attached SCSI storage disk arrays that were backed up over the 100 Mbps LAN. The data center configuration servers used three SAN arrays, each one comprising one SG. They were backed up over the Gigabit LAN.
OTG used best practice guidelines when designing their original Exchange servers with consideration towards maximizing system performance and availability with both the server and storage hardware. To optimize the disk input/output (I/O), each volume of an SG was designated as a Logical Unit Number (LUN). Since each LUN was assigned a drive letter, each server, hosting three SGs comprised of three LUNs each, used nine drive letters.
OTG configured each SG to maintain three separate LUNS. The mailbox data LUN using 24 18-GB disks and the Log LUN using six 18-GB disks were both configured using a striped mirror configuration, known as Redundant Array of Independent Disks (RAID)-10. The SAN also maintained a dedicated backup LUN utilizing 12 36-GB disks in a RAID-5 configuration. This LUN was used to support two days of online, disk-to-disk backup retention.
Each SG supported five databases, and each database supported 200 mailboxes, meaning that they could support up to 1,000 mailboxes per SG and 3,000 mailboxes per server.
Performance, Scalability, and Supportability Challenges
Exchange 2000 was a major upgrade from previous versions of Exchange. However, as powerful as Exchange 2000 was, OTG still had to work around some limitations.
Number of Servers to Manage Too High
Due to an inability to consolidate servers and sites effectively, the number of sites with servers drove support costs significantly higher and added complexity into the messaging environment. Some of the more common cost factors associated with the distributed environment included:
* More systems to backup
* Additional maintenance of backup systems at larger number of sites
* More personnel added to administer backup processes
* Greater power and cooling resources required at additional sites
* More onsite support staff added for hardware maintenance at multiple sites
From a complexity perspective, the larger number of systems meant more moving parts in a complex machine; i.e. the more backup jobs required, even with the same success rate, means a higher number of failures to troubleshoot and resolve. The planned 90 percent reduction in the number of sites with servers dramatically reduces the number of moving parts in the messaging machine, thereby reducing the exposure to failure on a number of fronts.
Recoverability of Databases within Service Level Agreement (SLA) Time Difficult
Even small efforts to consolidate resulted in higher scaling on servers in a number of sites. As the number of mailboxes on a server continued to increase with scalability improvements in the product, database sizes grew as well. More significantly, the initiative to increase the maximum mailbox size from 100 MB mailboxes to 200 MB mailboxes promised an immediate doubling in the size of databases.
Since Exchange 2000 does not offer support for new recovery options such as Recovery Storage Group (RSG) functionality or Volume Shadow Copy Service (VSS), a database outage due to corruption on an Exchange 2000 Server meant that the process of database restoration would result in an extended outage. In many sites, backups were managed across multiple computers in a datacenter, which resulted in backups and restores occurring over the 100 MB LAN, for which restore times averaged, at best, 16 GB per hour. The original restore SLA was full database restore in one hour, a goal that was quickly becoming unattainable.
Cluster Scalability Limitations
Windows 2000 Advanced Server supported two-node clusters and Windows 2000 Datacenter Server supported four-node clusters. With Exchange 2000 running on Windows 2000 Advanced Server, for an optimized configuration, OTG needed to have multiple drive letter volumes associated with each SG. There were also additional drive letters used in the server configuration, such as the Simple Mail Transfer Protocol (SMTP) drive (a dedicated inbound/outbound queue device). As a result, each virtual Exchange server within the cluster, after accounting for the collective SGs and the SMTP drive, used ten extended drive letters. This does not account for the required, reserved drive letters used by the server node itself, such as for the floppy disk, operating system volumes, and a CD drive. OTG could only use two servers in a cluster before it exhausted the supply of available letters assignable to disk volumes. The lack of available drive letters prevented OTG from adding additional instances of Exchange servers into a clustered environment.
Backup Infrastructure Inflexible
OTG processed a single-stage backup for regional servers. The regional servers used the 100 Mbps LAN to perform a direct, disk-to-tape backup. In Redmond, servers performed a two-stage backup process: first disk-to-disk within the SAN, and then disk-to-tape. To ensure that the backup process completed during non-business hours, OTG needed to deploy Gigabit Ethernet network adapters in each Exchange server to ensure that they could get the throughput necessary to push the data across the LAN and onto tape.
Data restoration required the creation of a temporary restoration server to serve as a staging server for retrieving data from tape. OTG learned that it in addition to the time it took to restore the data, before that process could start, a tape drive had to read and seek the starting point of that particular database on a tape. This process often entailed a wait of 90 minutes or more before any data actually transferred to disk. The typical throughput for data restoration (once data began to flow) on the OTG 100 Mbps network was approximately 300-350 MB per minute. With a selective restoration of a sample 15 GB database, the total time needed to complete the job was often more than two hours – far in excess of the SLA.
In the end, OTG based its entire architecture of Exchange 2000 on the technical requirements for meeting backup and restore efforts within the allotted SLA time window.
Reasons for OTG to Upgrade Back to Top
OTG had many compelling reasons to upgrade to Exchange 2003. Of course, in its special role as a group running Microsoft product group dog food software, OTG was committed to deploying Exchange 2003. This deployment was an effort to improve the product with real world, enterprise experience and feedback, long before any customers would receive the product.
In addition, Exchange 2003 resolved the Exchange 2000 challenges for OTG as described earlier. The deployment of Exchange 2003 enabled OTG to improve service to its customers and to reduce operations requirements. Microsoft realized the following business benefits:
* Reduced number of servers
* Improved server availability, reliability, and manageability
* Improved clustering support
* Improved security
* Improved data backup and recovery
* Improved support for mobile users
* Improved integration with Office 2003
Site and Server Consolidation
As of this writing, with the deployment of Exchange 2003 completed, OTG is in the process of implementing a long-planned consolidation of regional mailbox servers and locations. OTG had 113 mailbox servers in 75 locations around the world. The end goal of the consolidation plan is to reduce the number of locations by 90 percent, down to seven worldwide, using 38 clustered Exchange virtual mailbox servers. This level of server reduction will significantly reduce the administrative workload required of the messaging infrastructure in OTG.
Normally an increased number of mailboxes per server and a greater amount of data per SG would present an increased risk in the event of failure. Indeed, OTG measures database service availability as a factor of downtime multiplied by the number of databases affected. For example, a one-minute outage affecting a single SG of five databases on a server containing three SGs (containing 15 databases) is measured as five minutes of downtime. In addition, OTG studied its downtime incidents and learned that its planned downtime exceeded its unplanned downtime by a factor of 6:1.
Despite the fact that the number of mailboxes per server is growing, and that mailboxes are doubling in size, the site and server consolidation project is expected to improve OTG’s overall availability as well as its backup and restore performance SLAs. It is also expected to reduce the OTG server management workload significantly, thereby reducing costs.
For more information about OTG’s Exchange Server 2003 site consolidation plan, see the iT Showcase technical white paper titled, “Exchange 2003 Site Consolidation” at http://www.microsoft.com/technet/itsolutions/msit/default.asp.
Availability/Reliability/Manageability Enhancements
Exchange 2003 offers a variety of enhancements that make it a compelling upgrade.
Virtual Memory Management
The virtual memory improvements to Exchange 2003 reduce memory fragmentation and increase server availability. Specifically, Exchange is much more efficient in the way it reuses blocks of virtual memory. These design improvements reduce fragmentation and increase availability for higher-end servers that have a large number of mailboxes.
Virtual memory management for clustered Exchange servers is also improved. In Exchange 2003, when an Exchange virtual server is either moved manually or failed over to another node, the MSExchangeIS service on that node is stopped. Then, when an Exchange virtual server is moved or failed back to that node, a new MSExchangeIS service is started and, consequently, a fresh block of virtual memory is allocated to the service.
Exchange System Manager (ESM)
Administrator functionality using ESM has been enhanced in Exchange 2003 with these key updates:
* Improved method for moving mailboxes. The Exchange Task Wizard now allows you to select as many mailboxes as you want and then, using the task scheduler, to schedule the move to occur at some point in the future. You can also use the scheduler to cancel any unfinished moves at a selected time. Using the wizard’s multi-threading capabilities, you can move up to four mailboxes simultaneously.
* Improved Public Folder interfaces. To make public folders easier to manage, Exchange 2003 includes several new public folder interfaces in the form of tabs.
o The Content tab displays the contents of a public folder in Exchange System Manager.
o The Find tab enables searches for public folders within the selected public folder or public folder hierarchy. A variety of search criteria can be specified, such as the folder name or age. This tab is available at the top-level hierarchy level as well as the folder level.
o The Status tab displays the status of a public folder, including information about servers that have a replica of the folder and the number of items in the folder.
Windows Migration ServicesMicrosoft Strategic Alliance Windows Migration Services and Microsoft share the vision and commitment to deliver a ..... o The Replication tab displays replication information about the folder.
* New Mailbox Recovery Center. Using the new Mailbox Recovery Center, you can simultaneously perform recovery or export operations on multiple disconnected mailboxes.
* Enhanced Queue Viewer. The Queue Viewer improves the monitoring of message queues. Enhancements include:
o The X.400 and STMP queues are displayed in Queue Viewer, rather than from their respective protocol nodes.
o The Disable Outbound Mail option allows you to disable outbound mail from all SMTP queues.
o The refresh rate of the queues can be set using the Settings option.
o Messages are searchable based on the sender, recipient, and message state using Find Messages.
o Queues are clickable for displaying additional information about that queue.
o Previously hidden queues, DSN messages pending submission, Failed message retry queue, and Messages queued for deferred delivery, have been exposed.
* Enhanced control of message tracking log files. When using Exchange System Manager, you have greater control over your message tracking log files. Exchange 2003 automatically creates a shared directory to the message tracking logs and allows you to change the location of the message tracking logs.
* Improved error reporting. Error reporting allows server administrators to easily report errors to Microsoft. Although error reporting was included in Exchange 2000 SP2 and SP3, its implementation is improved in Exchange 2003. For example, if users do not want to view the standard error reporting dialog box, they can configure Exchange to send service-related error reports to Microsoft automatically.
Improved Cluster Support
Clustering in Windows Server 2003 provides a number of improvements that allows OTG to take full advantage of this technology to provide a solid clustered server standard to support its global Exchange mailbox server consolidation initiative. The new standard provides for a better level of scalability and availability over any previous deployment methodologies used for Microsoft’s corporate Exchange deployment.
Support for Up to Eight Nodes
Exchange has added support for up to 8-node active/passive clusters when using Windows Server 2003 Enterprise Edition or Windows Server 2003 Datacenter Edition. This enabled OTG to boost the number of servers in their Exchange Server 2003 clusters, thereby substantially improving server availability and reliability while reducing the number of Exchange deployments necessary to manage the Microsoft corporate messaging environment.
Support for Volume Mount Points
Exchange now supports the use of volume mount points when using Windows Server 2003 Enterprise Edition or Windows Server 2003 Datacenter Edition.
A volume mount point is a feature of the NTFS file system that allows linking of multiple disk volumes into a single tree, similar to the way the Distributed File System (DFS) of a server links remote network shares. Administrators can link many disk volumes together with only a single drive letter pointing to the root volume. The combination of an NTFS junction and a volume mount point can be used to graft multiple volumes into the namespace of a host NTFS volume.
Improved Failover Performance
Exchange has improved clustering performance by reducing the amount of time it takes a server to failover to a
exchange
|
|
|
