While few IT personnel are ever asked during job interviews how familiar they are with software vendor licensing, licensing ignorance is a substantial liability not just for an IT department, but for an organization as a whole. Failure to honor vendor licensing terms can lead to significant unbudgeted expenses, hefty fines from vendors, loss of negotiating power, and even Sarbanes-Oxley violations.
An IT department without licensing expertise is like a sailor without charts showing water depth. Heading from point A to point B in a straight line may be the obvious route to the destination, but sooner or later the hapless sailor will hit a rock or get stuck in the mud, often at a substantial financial reckoning.
At Directions on Microsoft, we focus on a single vendor’s licensing rules and structures, but because almost every organization with more than a few computers uses at least some Microsoft software, familiarity with the company’s licensing rules are necessary for every IT shop.
This article looks at licensing scenarios that can affect how IT performs in five general areas:
- Provisioning users
- Advanced service delivery
- Software development
- High availability
- Security and maintenance
One of the first tasks associated with provisioning a new user is putting a PC on his desk. That PC will likely have Microsoft Windows and Office on it, along with other software that might be critical to the user’s specific role, such as financial software for someone in the business office or project management software for a planner.
A common way to set new users up quickly is to unpack a new computer and to overwrite the OEM-supplied operating system and other applications (many of them so-called “crapware”) with a preconfigured software image using Norton Ghost or similar imaging software. An organization can prepare different images for different company roles and have a new computer on the user’s desk, configured with baseline software, in a few hours. The operating system (OS) used in all images is likely an edition of Windows purchased through volume licensing.
Using reimaging to prepare new computers, though legal and approved by Microsoft, can nevertheless violate Microsoft’s licensing rules in several ways.
Installation on Bare Machines
Many organizations assume that they are covered for use of Windows on all of their computers because they have an agreement such as a Microsoft Enterprise Agreement that licenses Windows on all of their computers. They can save money if a computer builder sends them a “bare” computer with no OS, such as Windows, preinstalled on it, since it will immediately be overwritten with a corporate-approved image.
But this practice is not legal. Microsoft requires that any computer used to run Microsoft software come with either an OEM or retail license-bare machines may not be imaged with volume software.
A computer may only be reimaged with same edition of the OS for which it is already licensed, unless the customer has purchased an upgrade license (an odd thing to do with a new computer, since it is less expensive to purchase the device with the preferred OS in the first place).
For instance, an organization that wants to create a standard Windows 7 Professional desktop on all new computers must purchase them with an OEM Windows 7 Professional license, not a less expensive Windows 7 Home Premium license.
One exception: if the customer has purchased upgrade rights for the desktop OS – either through a volume agreement, or the purchase of Software Assurance (SA), Microsoft’s upgrade subscription add-on for the new computer within 90 days of the computer’s purchase, then the customer is also entitled to reimage Windows 7 Professional with Windows 7 Enterprise, which has additional features.
Another scenario involves replacement employees. To give a new employee a fresh start, the IT department may reimage the former employee’s computer. However, if the former employee had been using an older OS, such as Windows XP Professional, and the organization is now imaging with Windows 7 Professional, overwriting the old OS with the new image may be illegal.
Many Microsoft server products require the purchase of two types of license: a license to run the server itself (such as Exchange Server), and a Client Access License (CAL) for each user who accesses the server. Some servers have discrete sets of features or functions, each of which requires a different type of CAL.
CALs are a daunting barrier to easy license management and configuration. Although Microsoft requires that CALs be “assigned” to a device or user, CALs in use are impossible to count, since they are only licenses and are not stored in a computer’s file system or registry.
Microsoft servers can’t detect or count CALs either, so an Exchange Server will happily deliver e-mail to 1,000 users even if not one of them has an Exchange CAL.
Ensuring that an organization’s CALs are compliant requires cooperation from many parts of the organization. The purchasing department knows which and how many CALs have been purchased; user departments should know which servers and server functions their employees require access to; and the IT department determines which server features are implemented. IT architects, product specialists, and network managers may also be involved in designing systems that properly segment user requirements to make CAL compliance easier.
Not Enough CALs
Only a handful of customers (we actually have yet to meet one) bother with formal assignment of CALs. Assigning CALs formally reduces flexibility, requires meticulous record keeping, and at the end of the day, doesn’t do much good: Microsoft will demand more data than a simple list of CAL assignments in the event of an audit. Most organizations try to ensure that they have enough CALs on hand to equal the number of people or devices that access the servers and services that require them. This usually involves buying more or more expensive CALs than they need.
Sometimes rough proxies are useful. An organization that counts the number of mailboxes on the Exchange Server and ensures that it owns at least that many Exchange CALs will be within hailing distance of the number of Exchange CALs it needs. But Exchange is unique in having a proxy like mailboxes that can approximate the number of people accessing the server. Most other servers have no obvious way to count users.
Wrong CAL Versions
CALs must match not only the server, but the server version. When the organization upgrades from, say, Windows Server 2003 to Windows Server 2008, its CALs must be upgraded to Windows Server 2008 CALs as well.
Installing just a single instance of an upgraded server can trigger a requirement for an organization-wide CAL upgrade. If, for example, an organization has 10,000 users and 100 Windows servers and just one of those servers is upgraded to the next version, the organization may be required to upgrade everyone’s Windows Server CALs, even if they never access the upgraded server directly. Because the upgraded server may deliver services such as Domain Name Service lookups to any computer that requests them, Microsoft deems that anyone on the network could access the server and must therefore have the appropriate CALs. Similar rules are in place for Exchange.
Thus, IT departments should be careful when planning new deployments or developing new systems that rely on updated Microsoft servers. If CAL upgrades for every employee are not in the budget, the server must be very carefully isolated.
Wrong CAL Editions
Where Microsoft once had a single CAL for any given server type – an Exchange CAL for Exchange Servers, a SharePoint CAL for SharePoint servers, and so on – the company now offers “tiered” CALs that add a new dimension of complexity. In these scenarios, a Standard CAL licenses basic product features, while an Enterprise CAL licenses more advanced features. The two CAL editions license discrete feature sets, so the Standard CAL is always required, and the Enterprise CAL may be required.
To use certain features in Exchange, such as storing voice mail in an e-mail inbox or archiving e-mail (usually for regulatory purposes) in specific mailboxes, customers must have an Enterprise CAL in addition to a Standard CAL. Similarly, to use SharePoint features such as server-based spreadsheets or to access data from an SAP enterprise resource planning system through SharePoint’s Business Data Catalog, a user must have both Standard and Enterprise SharePoint CALs. Communications Server 2010 may have (its licensing has not been finalized) three tiers of CALs, one for basic instant messaging and presence, another for Web conferencing, and a third for voice-over-IP telephony. Users can easily and inadvertently cause CAL noncompliance. Any user who initiates a Web conference with Communications Server or shares a SharePoint-based spreadsheet could trigger a requirement for an Enterprise CAL for themselves or for those with whom they are sharing.
Microsoft’s answer to this dilemma is simple: buy every employee both Standard and Enterprise CALs for everything. But this solution has a cost, and any organization that wants to avoid it will need to examine its network design (e.g., set up a separate Exchange server for those who need features that require an Enterprise CAL) or configuration (to deny some employees access to SharePoint server-based spreadsheets, for example) in an effort to avoid overlicensing of CALs.
Ultimately it will be up to IT to minimize the risk of such licensing violations, by configuring the network or the technology to prevent them.
However, many organizations may elect to overlicense, reasoning that the cost of effectively managing CALs may be higher than the cost of buying more than needed.
Programmers who use Microsoft tools to write software usually subscribe to the Microsoft Developer Network (MSDN), which offers per-developer licenses for most Microsoft products. Products licensed this way can be installed on many devices as long as they are used only by the assigned developer for development and testing purposes.
A common scenario for MSDN violations involves a development team assigned to develop a custom application. They use MSDN licenses to create and test the application, after which they move on to the next project.
If users continue to work with the system that the development team built, however, all of the Microsoft software used in the production application may be licensed improperly.
In many cases, these errors are not detected until an audit or a year-end asset management report. If that audit discovers that the company is running 50 instances of SQL Servers but has licensed only 45 for production use, it may require US$80,000 or more to legalize the application or to rebuild it using production rather than development licenses.
Security and Maintenance
Microsoft sells many server products whose role is to secure, protect, and optimize the performance of other servers and devices, such Configuration Manager (CM), Operations Manager (OM), Data Protection Manager (DPM), and Virtual Machine Manager (VMM).
These products are sold under the System Center brand and are generally licensed by “operating system environment” (OSE) which can refer to an OS installed on a physical machine or in a virtual machine (VM). Similar to server-CAL licensing, each server managed by a management server must have a management licenses (ML).
These products install software called a management agent on each server that is managed and this software can be detected with software asset management tools. Nevertheless, managing System Center licensing is particularly complex.
Free or Not
In some cases, such as when System Center MLs are bundled as part of a suite of management products (e.g, the System Center Server Management Suite Enterprise), the customer is purchasing MLs but gets the right to run the console and management server at no additional cost. In other cases, such as when a customer wants to use CM on its own, the customer must purchase a separate license for the server, as well as the MLs.
Because asset management software cannot tell the difference between an instance of CM that requires a license and an instance that does not, the IT organization will need to work with the business office and asset management teams to ensure that a software asset scan is supplemented with additional notes to substantiate the licensing rights that come with various bundles.
Two-Tiered Management MLs
A very thorny licensing issue is the proper use of the two tiers of MLs for the System Center CM, DPM, and OM products. The Standard MLs license only specific types of workloads while the Enterprise MLs (which, unlike Enterprise CALs, license a superset of Standard ML rights) license the full set of management functions. Analyzing workloads to determine the appropriate CAL requires significant IT skills.
Almost no procurement officers (and few IT professionals) can specify which workloads require only the US$157 Standard ML, versus those that require a US$431 Enterprise ML. Asset management tools cannot help. They can detect the presence of a management agent and thus the need for an ML, but not the functions actually being managed. This may require advanced management expertise, special scripts, or a visit to the physical server.
Microsoft’s proposed solution for this complexity is that customers purchase only Enterprise MLs, or Enterprise editions of the suites, even if many OSEs might require only Standard MLs.
Companies that reject this advice and want to control ML inflation must ensure that everyone involved in designing and administering an IT infrastructure that uses System Center products knows how their functions map to ML requirements.
An IT architect may be able to design a server architecture that carefully segments workloads among servers and reduces the risk of getting out of compliance. But if System Center products are bolted onto an existing system by personnel unfamiliar with the fine distinctions in Microsoft’s Product Use Rights documentation, a company will almost certainly become noncompliant.
Technologies like clustering and virtualization have significantly reduced the likelihood of complete system failures in production systems. They also require redundant systems that, happily for software vendors, require additional licenses.
Most software vendors see high-availability systems as not only an opportunity to sell more licenses, but to sell higher-value licenses. Spend a few more bucks and you might move from 99.8% to 99.9% reliability, which knocks nearly nine hours a year off your unplanned downtime.
In Microsoft’s case, carefully constructed restrictions make high-availability more complex. Ordinary recovery or fail-over measures may be subject to restrictions that require more or different licenses.
A common scenario: a server fails and you temporarily move its workload to a new or perhaps a spare old machine while you rebuild the original machine, reinstalling the OS, the applications, and testing the machine before restoring it to production. While you’re doing this, your substitute machine labors away a few racks down – illegally, because you are running two instances of the same licenses simultaneously.
A few escape hatches exist however, if you’re familiar with the licensing rules.
SQL Server failover rights. SQL Server permits installation of passive fail-over instances for disaster recovery scenarios.
Permanent hardware failure. You can move a license to another server at any time if the original server experiences a “permanent hardware failure.” With a little flexibility on the word “permanent,” e.g., “it failed permanently until we fixed it,” you might be clear. However, that only got the license onto the backup server. The trip back could be 90 days longer, since Microsoft won’t let you transfer many licenses more often than every 90 days.
“The dog ate my purchase order.” In some volume licensing programs you can install a program without purchasing a license first, as long as you buy the necessary license by the end of the month. If you can get the workload shifted back and forth within a calendar month, then the second license that you require during the recovery period can quietly fall off the monthly license requisition without ever being needed. In this scenario, the license was never really transferred to the second machine: the backup got its own license, but before the bill had to be paid it disappeared.
Cold Disaster Recovery Rights. Microsoft’s preferred solution applies to licenses covered by Software Assurance (SA). These licenses can be used on a “dedicated disaster recovery server” while a production server is being recovered. Unless you have some other reason to purchase SA however, such as an imminent version upgrade, this is expensive insurance: 25% of the license price each year, plus the cost of hardware that is turned off for about 99.9% of the year. It would be less expensive in many cases to purchase an additional license purely for the purpose of disaster recovery, a cost that could be put off until it is actually needed.
Virtual Machine Mobility
Virtual machines (VMs) can substantially improve an organization’s uptime. VMs can be restored from backup much faster than physical machines and all modern commercial VM systems can now move a VM from a failing physical server to a replacement so quickly that users never see an interruption.
In the Microsoft world, however, moving the VM and moving its license are two different things. The company requires that licenses – even those used in VMs – be assigned to physical machines, and licenses generally cannot be reassigned from one physical machine to another less often than every 90 days. This creates a situation in which a customer might, by the numbers, be correctly licensed – the number of instances of products in service does not exceed the number of licenses – but because of an IT operation the organization has become noncompliant.
Some Microsoft products waive the 90-day requirement, but it applies to the Windows Server OS. In other words, even though the application running in a VM might be eligible to be moved at a moment’s notice, the OS on which it runs could keep it anchored to a physical machine.
Customers have several solutions.
OS overlicensing. Customers who license every processor in their data center with the most expensive edition of Windows Server, Datacenter edition, can transfer VMs as frequently as their application workload permits. Organizations that use other editions of Windows Server have two options: they can buy extra, but unused, licenses that are assigned to any servers to which a VM might be moved, or they can be careful not to move VMs within the 90-day freeze. In general, overlicensing is not necessary to deal with infrequent problems, such as hardware failures, because the 90-day limit will long have passed when an OS is moved, but moving VMs for performance reasons, such as load balancing, could require OS overlicensing.
Application overlicensing. Most server applications are licensed per instance. If one of these applications is subject to the 90-day constraint, customers who want the option to move it more frequently must purchase and assign extra licenses for that product to any server to which the VM might be moved. The license count must remain higher than the count of running instances in order to ensure this flexibility; any IT professional who decides to run more instances because “spare” licenses are available will potentially put the organization’s compliance in jeopardy.
Application upgrades. Because Microsoft waives the 90-day transfer limit for many of its newer server products, companies may get more mobility by upgrading to the latest products. If upgrading can buy the right to move an application without regard to the 90-day limit it will usually be less expensive than purchasing extra licenses and keeping them in reserve for applications that could not otherwise be moved frequently.
An organization’s IT group can minimize some of these constraints in the design of its data center. For example, it can purchase extra licenses only for specific servers designated for load-balancing. Server hardware with more powerful processors and large amounts of memory, combined with a VM hypervisor that can maximize the number of VMs per processor, squeeze maximum value out of each instance of Windows Server Datacenter.
But IT may also have the arduous responsibility of keeping track of the date that any given OS or application instance is migrated to a different physical server. This information is not generally available except through manual record keeping, a sad prospect for anyone who imagined that virtualization and VM mobility could create a “dynamic” data center that could balance loads, recover from failures, and automatically optimize operations without a human having to look data up in a spreadsheet.
To promote use of highly managed VMs, Microsoft offers the Enrolment for Core Infrastructure (ECI). The three suites available under this program provide discounted licenses for Windows Server, System Center management licenses, and Forefront antimalware protection. However, the ECI creates some dangerous exceptions to normal licensing rules that could foul up efforts at effective asset management.
The ECI Standard Suite includes one copy of Windows Server; one ML each for CM, DPM, and OM servers; and Forefront. The ECI Enterprise Suite includes the Enterprise edition of Windows Server and the System Center Server Management Suite Enterprise; and the ECI Datacenter Suite includes the Datacenter edition of Windows Server and the System Center Server Management Suite Datacenter.
The difficulty lies in how the components are licensed: the Standard and Enterprise components are normally licensed per device, but within the ECI they are licensed by processor. Since the actual software is the same in either case, neither asset management software nor an IT professional can tell whether a given instance has been licensed per processor or per device without looking up purchase and management records. As a result, an IT professional who is unfamiliar with the ECI licensing channel might add more VMs than are permitted on a server licensed by the ECI, since the four Enterprise server and management licenses assigned to that machine would normally support many more VMs. That would break the rules under which the product was purchased, even though the product use rights for those licenses normally permit the additional VMs.
Typically, IT professionals are hired for their technical knowledge, and procurement professionals are hired for their business skills. Neither group is likely to have the depth of licensing and operational expertise that can guide them safely through the shoals of software licensing. Furthermore, operational responsibility in IT organizations is often distributed among many people (to cover 24×7 operations in a data center or help desk, for example), and operational decisions are often made at a relatively low level of the IT organization.
The many undiscoverable and unmanageable attributes of IT assets limit the utility of automated management systems and puts the onus for compliance on people and organizations. Some level of licensing expertise must be an input for both long-range and daily IT decision making. Organizations should ensure, at a minimum, that individuals with a distinct technical role – a SQL Server or an Exchange administrator, for example – are not only intimately familiar with the rules that govern those products, but also stay up to date as the rules, licenses, and delivery channels change.
IT organizations should also insist that solution developers, whether internal staff or external contractors, keep asset management and licensing issues in mind from the very beginning of the design stage; their choices can have a dramatic impact on license manageability and costs.
Finally, organizations as a whole should push back on their software vendors. Most, like Microsoft, don’t license their own software and make no effort to manage their internal-use licenses. As a result, their awareness of asset management issues is negligible, and ease of asset management plays little or no role in product development, channel management, and licensing.
This environment is not likely to change until customers give asset management a higher profile. Only when vendors see customers adding asset management costs to decisions like “buy or wait,” “Windows or Linux,” “VMware or Hyper-V,” or “Dell or HP,” are vendors likely to change their ways.