Friday 25 December 2009

WSUS - Operations

When deploying server bear in mind that the server will probably use about 15-30GB of space. The server and clients use BITS to download updates.
Synchronizing does not cause the server to start downloading updates - it just downloads metadata of all available updates. The server will start downloading actual updates only once updates are approved. The actual size depends on the type of download and the languages you have selected. You have Express downloads which basically only transfers delta changes to clients rather than full updates - this reduces the amount of data transferred to clients - but increases download sizes from MS Update by about 300-400%!! The normal setting transfers full updates to clients but does not increase download sizes from MS. This is critical if customer servers are on an unshaped + capped service (Telstra Business) - if this is the case it is worthwhile getting hold of the Custdata login so bandwidth utilization can be tracked.

NB!!!:
The settings on the options panel don't seem to get applied retrospectively if you accidentally select the incorrect settings and then synchronize - so changed settings are only applied to newly synchronized updates - Your options are to temporarily stop all update downloads (see below) and selectively download each one you need and decline the rest - or reset the entire WSUS system entirely.

Recommended Settings:
In the console:
Options

Update Source
- Set to Synchronize with MS Update

Products and Classifications
- Select only the products you need to keep track of.
- For Classifications select:
Critical Updates
Definition Updates
Security Updates
Service Packs
Update Rollups
Updates

Update Files and Languages
NB: The options on this panel are critical to the download sizes and volumes downloaded from MS Update!!
- Set to Store update files locally on this server
- Select Download update files to this server only when updates are approved
- NEVER select Download express installation files!!
- NB!!! go to the next tab: Update languages - and select only the languages you need...The default is to download all languages!!!

Synchronization Schedule
- Set to automatic and once a day - early morning is best.

Automatic Approvals
- Check the box to enable
- Edit and select the classifications appropriately - I would not select Service Packs here - large download sizes.

Computers
- Assign via Group Policy and create the relevant GPOs and link them in.

To track pending downloads and current downloads:
Go to Updates > All Updates
for Approval - Any Except Declined
for Status - Failed or Needed
Enable the File Status, Release Date and Arrival Date columns
Apply to all views
Sort by File Status to see queued downloads - To temporarily stop downloads - Select all downloading updates and Cancel Download - about 30 sec later the BITS queue will be empty.
From command line run
bitsadmin /list /allusers (/verbose gives more info)
and you will see the BITS download queue

To track bandwith utilization from MS update
Open Resource Monitor
In the Network area - look for: svchost.exe (netsvcs) and under Address something like: a72-247-247-26 - this will correspond to an Akami server.

To force clients to check for updates run from commandline
wuauclt.exe /detectnow

WMI Troubleshooting

Source URL:
http://blogs.technet.com/configmgrteam/archive/2009/05/08/wmi-troubleshooting-tips.aspx

WMI Troubleshooting Tips

The Windows Management Instrumentation (WMI) subsystem has evolved over time to become a key dependency for many applications, Configuration Manager being one of them. We've been intertwined since the .698 build of WMI shipped with Systems Management Server 2.0 (there was even a WBEM Inventory component in SMS 1.2).

Given this history and relationship it's important to know what to do when something goes wrong.

I've spent many years investigating / troubleshooting WMI related issues - especially as they relate to SMS / Configuration Manager. Based on that experience I've compiled a few tips and general observations for the community. This list is by no means comprehensive.
Assumptions are made regarding a basic understanding of WMI, such as general structure, terms and usage of the WBEMTest tool.

Don't delete the repository (though it may make problems seem to go away).

Rebuilding the WMI repository is a destructive operation that can lead to data loss, applications breaking, and a whole host of slow to appear, difficult to diagnose problems.
Generally speaking, the only time this operation should ~really~ be necessary is in the case of true corruption as indicated by tools such as WMIDiag or Winmgmt /verifyrepository.

Can lost WMI data be recovered?
Probably, but that's never a good state to be in. Therefore, I say avoid this operation whenever possible.
On the flip side, I certainly recognize there is a tradeoff between operational needs and individual investigations.
Over time some customers have seen that rebuilding the repository makes a problem seem to go away quickly. Typically this also comes with a loss of ability to find root cause, could mask other problems, and may not actually solve anything long term. On the whole I strongly recommend against deleting the repository folder as a means to resolving WMI issues.

What can I do other than rebuild the repository?

One low risk, potentially high gain operation that can be performed is to recompile MOF files, and register component DLL's associated with WMI operations. If an important class or component registration needed for WMI operation was somehow removed you can put the needed structure back.

These steps can be automated easily, but aren't generally recommended on a large scale as they too can mask issues. This is just one more option to try short of rebuilding the repository. There are variations of the steps below available between XP and Vista, but this most basic version should work for either.

1. Open a CMD prompt on the server and change directory to %windir%\System32\WBEM (\SysWOW64\WBEM on x64)

2. Execute the following:
FOR /f %s in ('dir /b /s *.dll') do regsvr32 /s %s
Net stop /y winmgmt
FOR /f %s in ('dir /b *.mof *.mfl') do mofcomp %s
Net start winmgmt

Note: Don't attempt to compile the MOF files in the \bin\i386 folder on a site server, as we contain stub files (names start with an underscore character such as _smsprov.mof) that need to be populated with site specific data through other means.

Outside of that, there are many troubleshooting options available, depending on the amount of time you can spend investigating.

Where can I find the log files and error codes?

Start here: the WMI Troubleshooting page on MSDN. This page serves as a jumping off point for many important details such as logging and tracing information, WMI Error constants, and more.

Common Errors

These errors are referenced in greater detail on the WMI Troubleshooting page and subsequent links but I still wanted to mention them here.

WBEM_E_NOT_FOUND – 0x80041002
The Not Found message was very common in XP log files, a little less so in Vista and up. Without context this one isn't very helpful, as you have no way of knowing if the requested data is supposed to be present. Simply put, it may not always be a bad thing.

Access Denied
Echoing the troubleshooting page, if you're seeing 0x80070005 (E_ACCESS_DENIED) when connecting you're being turned away by DCOM, not WMI. Similarly the 0x800706BA (RPC_S_SERVER_UNAVAILABLE) means you're being turned away before you've talked to DCOM or WMI. A Network capture is often the quickest way to get to make progress for the RPC error.
There's also a bit more info in the Remoting and Security blog entry from the WMI team.

WBEM_E_PROVIDER_LOAD_FAILURE - 0x80041013
The Provider Event Troubleshooting Classes are a great resource, but may be a little overwhelming. The MSFT_WmiProvider_LoadOperationFailureEvent class is one that I've found useful quite often. Most Provider Load Failures I've encountered have been the result of bad component registration (either in the registry or WMI), or permissions related.
Could also be a corrupted cimwin32.dll

WBEM_E_INVALID_CLASS - 0x80041010 / WBEM_E_INVALID_NAMESPACE - 0x8004100E
Similar to the Not Found error, context is important here. Some operation was being performed against a class / namespace that isn't present on the target machine.
Is that bad? Depends on the situation. It may be perfectly normal. If investigation tell you it's not, the class or namespace can usually be recovered by recompiling the appropriate MOF file.

Generic Failure - 0x80004005
Among the least helpful errors, and not WMI specific. I only bring it up here as many people see this and mistakenly think it's an Access Denied message given the 5 at the end. Remember access denied is 0x80070005

WMIDiag
An invaluable tool for diagnosing WMI issues, even if it's a little dated.
http://www.microsoft.com/technet/scriptcenter/topics/help/wmidiag.mspx
It has many configuration options available and can be deployed via Configuration Manager. One of the more helpful features is the report that is generated at the end. It contains details on how to correct many common issues that are found when running the tool.

Tracking resource usage of WMI
By default the core WMI service lives in the shared Network Services instance of scvhost.exe. This can make debugging or identifying resource issues a little challenging. As a general rule of thumb I run (and recommend to customers) that they keep WMI separated into its own instance of svchost.

On XP/Server 2003 this can be accomplished automatically via the following case sensitive command:
RUNDLL32.EXE %Systemroot%\SYSTEM32\WBEM\WMISVC.DLL,MoveToAlone
For Vista and up this is done with
winmgmt /standalonehost

Where is the provider?
The WMI Provider host process (wmiprvse.exe) will create one instance for each different hosting (security) model defined. To find out which instance by PID a given provider resides in (such as smsprov.dll) you can simply run
Tasklist /m smsprov.dll
It is possible to isolate a provider into its own instance by changing the hosting model.

This is fairly rare and not necessarily a best practice, but if you're running into resource or performance problems that could be traced back to multiple providers running in the same instance, it may be worth investigating a split - at least for the purpose of issue isolation. The Provider Hosting and Security page has more information.

WMI configuration
There are quite a few options available for tuning WMI performance. Two that I'll cover here are important for Configuration Manager Site (provider) servers - the MemoryPerHost and HandlesPerHost values that can be found in the __ProviderHostQuotaConfiguration class in the root namespace.
First a little background:
For each instance of WMIPrvse.exe that is running, the classes above dictate how much virtual memory or handles that instance may consume. When exceeding that limit the process may terminate, or in some rare cases may hang.
As more providers for various applications are being used on server machines, and Configuration Manager environments get larger, it's expected to see increased resource usage with our provider.

Prior to Vista the limits were 128MB (134217728 bytes) and 4096 handles.

In a large Configuration Manager environment (in terms of number of objects that exist, such as collections, advertisements, AdminUI connections, as well as clients) you could definitely exceed those limits.
Quadruple the memoryperhost value to 512MB - 536870912 is the value to enter - is what I recommend to all my customers.

512 is even the default value now on Vista and above, further indication that a larger limit was needed.

If performance monitoring tools indicate that you're hitting or exceeding the 4096 handle limit, you can increase that as well but be a little more conservative since handles are a shared resource. It could likely be doubled but I usually recommend 5120, again if monitoring indicates an increase is needed.

It's important to remember that increased memory usage alone is not an indication of a problem state, or a leak - it's quite likely just normal behavior. In other words, many objects (and perhaps many objects from multiple remote connections) mean more resources required to handle everything.

If you see the Process ID (PID) of wmiprvse.exe that hosts smsprov.dll changing frequently, or multiple instances of smsprov.dll loaded you should definitely increase this value. Some customers have reported an increase helping with Administrator Console performance as well.

WMI repository stability fix
Lastly, if you're still on XP SP2 or Server 2003 SP1 or SP2 you should apply this fix to help further stabilize the repository files. Note it won't correct a system already having problems but makes for good preventative maintenance.
http://support.microsoft.com/kb/933062

In addition - If WMI does not work - SBS Monitoring and Reporting will not work either - Will probably need to be re-installed after repairing WMI.

Windows Media Player won't play DVDs

In order for WMP to play dvds it needs a DVD (MPEG2) decoder to be installed on the machine.
This can be done by installing PowerDVD or WinDVD, etc. The MPEG2 decoders are payware.

Windows Media Player - Backgrade options

Windows Media Player cannot be backgraded from version 10 to 9. It can however be backgraded from 11 to 10.
Version 9 is installed with XP SP3. Version 10 comes with Vista.

USB: USB2 controller cannot detect device

The problem here is using a USB1 cable to talk to a USB2 device or dirty connections.

Soultion:
1. Disable the USB2 controller - this will force the device into USB1.1 mode and the computers USB1 controller will pick the device up correctly.

or

2. Replace USB1 cable with a USB2 cable.

Task Manager - Killing processes

CAUSE
Stopping a process is done with a call to TerminateProcess (Win32 API). By default, it is not possible to kill a process that is running under a security context different than the one of the process who issued the call to TerminateProcess.

RESOLUTION
To work around this problem, use the KILL command line utility (available with the Windows Resource Kit) to stop processes that do not run under the security context of the logged on user.

This works except for really zombie processes on Win2k server...

Symantec Liveupdate Troubleshooting

Set LiveUpdate to manual mode:
Go to C:\Program Files\Symantec\LiveUpdate
Run the LuConfig.exe
Select Interactive Mode
Select ENHANCED Error Support

Set this back to Express Mode once system is working again.

On the Update Cache tab
Set the cache to a suitable size

On the Automatic LiveUpdate tab
Set this appropriately


To trigger a Liveupdate
run LUALL.exe

Go ahead and select all available updates and start the update
- watch the progress bars - if the download quits after about 4MB or so check the firewall you are using - there could be content filtering taking place. You should also receive a LU error stating it cannot download updates. You can click through the link and will open a Symantec support page. This may tell you to wait 4 hours - in the event LiveUpdate is down for maintenance, etc. It should also allow to update LiveUpdate - this will automatically uninstall\reinstall LiveUpdate.

For LiveUpdate cache, config and log location browse to
C:\Documents and Settings\All Users\Application Data\Symantec\LiveUpdate\

The log file is "Log.LiveUpdate"
The config file is "Settings.LiveUpdate"

Open the config file and set the following
PREFERENCES\INTERNET_CONNECT_TIMEOUT=600
PREFERENCES\INTERNET_READ_DATA_TIMEOUT=600
This will eliminate the possibility of a slow network being the cause.

The cache is the "Downloads" folder - clear this out to eliminate corrupted downloads being the cause.

Delete\Rename the log file "Log.LiveUpdate"

Run LiveUpdate again.
If you see "HR 0x802A0046 DECODE: E_UNABLE_TO_READ_DATA_FROM_SERVER"
This is a strong indication there is a firewall preventing access to LiveUpdate.

SSL Certificates

For Exchange 2007 you need a SAN (Subject Alternative Name) certificate allowing you to have many domain names represented by the one certificate.

e.g.

mail.domain.com
autodiscover.domain.com
exchange1.domain.com
exchange1.domain.local
exchange1

VeriSign supplies these for a cool $2891 USD ($150,000 guarantee)
- $395 for single domain 1 year
GoDaddy does the same for about $71.95 USD ($2000 guarantee)
- $29 for single domain 1 year


Wildcard certificates don't work very well for use with Exchange 2007 esp. if you have different internal and external domain names. Wildcard certificates equate to *.domain.com.

SMB error codes

System error 5 - Access is denied
This is a permission issue. If the net view command fails with a "System error 5 has occurred. Access is denied." message, 1) make sure you are logged on using an account that has permission to view the shares on the remote computer.
2) Need to cache credential: logon the same username and password on both computers or use net net use \\computername /user:username command.
3) Make sure the Netlogon service is running.

System error 8 - Not enough storage is available to process this command
System error 234 - More data is available.
Symptoms: If you attempt to start the server service manually, the following errors may be displayed: System error 234 has occurred. More data is available. Or system error 8 has occurred. Not enough storage is available to process this command. The event viewer shows "Event ID: 7023. Description: The Server service terminated with the following error: More data is available. Or Event ID: 7001. Description: The Net Logon service depends on the Server service which failed to start because of the following error: More data is available.
Resolutions: 1) apply (or reapply) the latest Windows NT Service pack.
2) remove any unnecessary entries from this value in the registry, HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanServer \Parameters\NullSessionPipes

System error 51 has occurred - The remote computer is not available
Symptoms: You may receive "System error 51 has occurred. The remote computer is not available" when using net use to map the computer drive.
Resolutions: 1. Make sure server service is running on the remote computer.
2. Enable file and printer sharing.

System error 52 - You were not connected because a duplicate name exists on the network.
Symptoms: you can ping a host but not net view it. When using net view \\hostname , you get system error 52 - a duplicate name exists on the network.
Resolutions: there are two host names or alias name (cname) are pointed to the same IP. 1) check the WINS records. 2) check DNS records. 3) Go to System in the Control Panel to change the computer name and try again.

System error 53 - The network path was not found.
Symptom: when using net view \\ip or \\computername , you get system error 53.
Resolutions: 1) if it is domain environment, check your WINS; 2) if it is peer-to-peer workgroup, enable NetBIOS over TCP/IP; 3) make sure the machine is running; 4) make sure file and Printer Share enabled on remote computer; 5) make sure client for ms networks is enabled on local computer; 6) make sure you type the correct name. 7) Make sure no firewall running or any security setting.
VPN Case Study - Can ping VPN server but receive System error 53 using net use
Case Study - System Error 53 - The network path was not found.

System error 67 - The network name cannot be found
Symptom: When using net view \\computer or net use \\IP, you may receive above error message.
Resolution: 1. Make sure you type the correct computer name or shared name.
2. Make sure the Workstation service is running on the local computer while Server service is running on the remote computer.
More troubleshooting here System error 67 has occurred

System error 85 has occurred. The local device name is already in use
Cause: net use /persistent:yes is default settings for NT and win2000/XP. If you have mapped some network drives and check the reconnect at logon, or your network uses logon script to map network drives, the mapped network drives may show red Xs. If you enable echo and pause the logon script or if using net use to map the same drive manually, you may get "System error 85 has occurred. The local device name is already in use." One thing you may want to try is using net use /persistent:no, for example, net use i: \\servername\folder /persistent:no.

System error 1219 has occurred - The credentials supplied conflict with an existing set of credentials
Symptoms: 1) When you log on to a domain from w2k client; 2) when attempting to join a domain, you may receive the following error message: The credentials supplied conflict with an existing set of credentials.
Resolutions: This may cause because of attempting to make two or more connections to the same server using two or more sets of credentials
1. Go to windows explorer and disconnect all network drives. Then re-logon.
2. Delete the profile or copy another profile. Note: you may lost all settings and data in My Documents when deleting or copying profile.
3. If solution 1 and 2 doesn't work, try this: 1) Log on as an administrator at any workstation and run regedt32. 2) Select HKEY_USERS, but do not open. 3) From the Registry menu, click Load Hive. 4) This will bring up a Load Hive dialog box. Locate the Ntuser.dat file for the user with the errors. Select the Ntuser.dat and click Open. You may enter any string for the Key Name. Use TEST for ease of use pertaining to the remainder of this article. 5) Locate the Username value under the following key in the registry: HKEY_USERS\TEST\Network\Username. 6) Delete the string for Username (leaving it blank is sufficient). 7) Select the TEST hive that you previously loaded, click the Registry menu, and then click Unload Hive. 8) Quit Registry Editor.
4. If you get this message when joining the domain, make sure 1) you have delete the computer from AD; 2) delete it from DNS; 3) delete it from WINS.

System error 1231 has occurred. The network location cannot be reached.
Symptom: When using net view \\computername , you may receive System error 1231.
Resolutions: 1) make sure Client for MS Networks is enabled, 2) make sure you have permission to access it.

System Error 1240 - The account is not authorized to login from this station.
Symptoms: 1. You may get the system error 1240 when using net view \\remotecomputer '
2. “Workgroup_name is not accessible… Account is Not Authorized to Log In to this Station” when attempting to browse the workgroup from a networking computer.
Resolutions: 1. Use Regedit to enable unencrypted (plain text) passwords for the SMB client.
2. Enable Send Unencrypted Password to Connect to 3rd Party SMB Servers under Local Security Policy.
3. Set the following policies as showing:
Digitally sign client communications (always) - disabled
Digitally sign server communications (always)- disabled
Digitally sign server communications (when possible) - disabled
LAN Manager Authentication Level set to Send LM and NTLM - use NTLMv2 session security if negotiated - (default) send LM & NTLM responses
Secure channel: Digitally encrypt or sign secure channel data (always) - disabled
Secure channel: Require strong (Windows 2000 or later) session key - disabled
4. Contact the third-party SMB server manufacturer if you have a third-party SMB server, such as DEC Pathworks, Samba or Linux.
5. If you are running Windows 9x, you may want to re-configure windows authentication for network logons.

System error 1311 - There are currently no logon servers available to service the logon request
Symptoms: The primary purpose of logging on with cached credentials is to enable you to access the local workstation. However, if you have logged on by cached credentials, you may be unable to access network resources because you have not been authenticated. For example 1) after you log on to a w2k/xp laptop by using cached credentials, you may be unable to access the network resources. This issue is commonly experienced by laptop users whose computer resides in a Windows Server domain and who log on to the computer by using cached credentials prior to being able to establish a remote access connection. 2) You log on to a w2k/xp laptop with a domain logon option in a workgroup network. After you establish the connection and you try to map the network drives, the operation may be unsuccessful, and you may receive the following error message: "System Error: (1311) There are currently no logon servers available to service the logon request."
Resolutions: To authenticate the cached credentials, 1) if it is w2k/xp, use net command, for example, net use \\servername\sharename /user:username. 2) if xp, open Windows Explorer>Tools>Map Network Drive. Click Connect using a different user name, enter the username and password.

System error 1326 has occurred - Logon failure: unknown user name or bad password.
Symptom: when using net use to map a network drive, you may receive "System error 1326 has occurred. Logon failure: unknown user name or bad password." message.
Resolutions: 1) create a user account on remote computer; 2) need to enable the guest account; 3) make sure the remote computer doesn't use auto-logon and blank password; 4) make sure you have a folder or drive shared on the remote computer. 5) use net use \\servername /user:username command. Make sure you type correct command (e.g. use net use \\servername \user:username will get this error too)

System error 1331 has occurred - Logon failure: account current disable
Symptom: When using net use \\computername command, you may receive above error message.
Resolutions: this is cache credentials issue. To fix this problem and cache the credentials, use net use \\computername /user:username command.

System error 1385 has occurred - Logon failure: the user has not been granted the requested logon type at this computer
Symptoms: When using net use \\remotecomouter\ahredname , you may receive above message.
Resolution: 1. The users do not have permission to connect to the remote computer. To resolve this problem: on the remote computer, select Administrative Tools>Local Security Settings>Local Policies>User Rights Assignment, right-click on Access this computer from the network>Properties>Add Users or Groups, add everyone or any users you want to be able to access the computer from the network.
2. refer to this case: Solved: System error 1385 - Logon failure -

System error 1396 has occurred - Logon Failure: The target account name is incorrect.
Symptoms: 1. when using net use, you may receive above message.
2. when using net view \\hostname, you may receive "System error 5 has occurred. Access is denied.". However, net view \\ip works fine.
3. You may receive above error while running logon script.
Causes: 1. SPN for the domain that is hosting the replica has not been propagated.
2. Incorrect target account name or the server is not online.
3. If you have DFS, make sure the DFSRoot is available.
Refer to RL060704

System error 6118 has occurred. The list of servers for this workgroup is not currently available
SYMPTOMS: 1) After enabling ICS/ICF, you can't see any computes on My Network places. If you try, you may get "workgroup is not accessible". 2) If you use the net view command, you may receive "System error 6118 has occurred. The list of servers for this workgroup is not currently available." message.
Resolutions:
1) This behavior can occur if you enable the ICF that will closes the ports for file sharing by default. To open these ports, right-click the network connection that is firewall protected> Properties>Advanced>Settings>Service Tab>Add, Enter 127.0.0.1) for the required Internet Protocol (IP) number. Enter UDP ports from 135 through 139, and TCP ports from 135 through 139 one by one (the external and internal port numbers should be identical).
2) This may occur if the workgroup name and the domain name are the different.
3) No master browser. Starting Computer Browser Service on one of w2k/xp computers should fix the problem

Smart Host configuration - Exchange 2003

Open Exchange System Manager in MMC or SBS Server Management
Under "Connectors" - there should be a "SmallBusiness SMTP connector" (on SBS - Otherwise it will just be called "SMTP connector").

On the "General" tab Select the option "Forward all mail through...." set this to the required SMTP server. This overrides the setting on the SMTP virtual server.

Shut down Windows in an instant

Shut down Windows in an instant

The other evening I turned off my Windows XP system and busied myself with other matters, only to find the machine churning away several minutes later as it worked through its shutdown process. I could've understood the delay if it were installing updates, or even if some program or service had hung the system. But this was a typical PC shutdown, and it was taking forever.
"There's gotta be a better way," I thought, and after doing a little research, I found a bunch of Registry tweaks that reset Windows to close shop like it's late for the bus ride home. Keep in mind, any changes to the Registry can be troublesome, so you may want to make these alterations one or two at a time just to make sure they don't futz up the works (it'll also be easier to diagnose any problems that may arise). And for sure back up the Registry by creating a restore point before you begin.

Kill your apps
Some programs just don't know how to say goodbye. To speed up force-closing them at shutdown
Start Regedit
HKEY_CURRENT_USER/Control Panel/desktop.
WaitToKillAppTimeout in the right pane, and change the value data (measured in milliseconds) to 1000 - if you're in a real hurry--or something larger, if you want to give your recalcitrant apps a little more time to call it quits.
Click OK when you're done.
Use this Registry key to reduce the number of milliseconds you want Windows to wait before killing apps at shutdown.

To reduce the wait before forcing hung applications to close, double-click HungAppTimeout in the right pane of the same key, and change the value data to 2000, or however many milliseconds you want to give the apps to unhang on their own. Of course, a better approach is to figure out why the app is hanging in the first place; I'll cover diagnosing hung applications in a future post: Stay tuned!

You may also want to change the value data of WaitToKillAppTimeout and HungAppTimeout in HKEY_USERS/.DEFAULT/Control Panel/Desktop key to apply the changes to all users on the system.

Clip your running tasks and services
There's another Registry key that automatically ends running tasks at shutdown. Navigate to HKEY_CURRENT_USER/Control Panel/Desktop, double-click AutoEndTasks in the right pane, and change the value data to 1.

Now to whack your slow-ending services, go to HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control, double-click WaitToKillServiceTimeout in the right pane, change the value to 1000, and click OK.

Some people will tell you that you can speed up shutdowns by telling Windows not to clear the pagefile when it closes. Unfortunately, this could compromise your system security because sensitive data may be stored in the file unencrypted. There's even some question about whether disabling this setting will save you any time at shutdown. For me, the possibility of saving a couple of seconds on shutdown isn't worth the risk. That's why I recommend that you leave the ClearPageFileAtShutdown value at HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/SessionManager/Memory Management at 1.
Some Vista systems (including mine) don't have all the Registry key entries described above. To add a missing entry, right-click in the right pane of its key, choose New > String Value, type the name, such as AutoEndTasks, double-click the new entry, add its value data (1 in the case of AutoEndTasks), and click OK.

Services not starting

Some programs that run services bound to specific ports may not run due to an issue with an update by Microsoft where DNS basically grabs all unused ports and doesn't allow other programs to use the ports, preventing services from starting.

I've noticed this issue with the Brian White site where they use a program called Leap and at CPO where the Multilink UPS service wouldn't start because the ports have been taken by DNS. What I have done to resolve is stop DNS momentarily and install/start the service, then start the DNS service. Not sure what other programs are affected by this, but keep this in mind if you are having similiar issues with services not starting that require certain ports to communicate.

Server 2008 or SBS 2008 hanging on boot

Source URL: http://achugh.wordpress.com/2009/06/17/small-business-server-2008-and-applying-computer-settings-message/

....checked its network properties…I remembered that I had unchecked IPv6 option…it was pretty strange and I didn’t really think that the problem could be due to this. I just checked it again and let the server reboot…within minutes, I was able to login… It seems that the problem occurs because IPv6 is supposed to be SBS2008’s primary protocol…due to which it gets stuck on the “Applying Computer Settings” screen....

SBS 2003: ISA + IIS

This is for the scenario where you are running SBS and running ISA in front of IIS for HTTP/HTTPS web publishing purposes. This needs to be configured correctly for Exchange ActiveSync and SSL OWA to work correctly.

Certificates
You need to create a self-signed certificate (or purchase a proper SSL cert) for the servers FQDN for external access. (e.g. mail.mginq.com.au)

Install the cert to the various cert stores (Trusted root, etc) on server.
Install the cert onto the HTTPS clients if its self-signed.

If server has multiple interfaces (virtual or real) for LAN and Inet access you need to configure IIS to listen only on the internal LAN iface using the httpcfg command.

Viewing all off the IP addresses that the HTTP API is listening on.
> httpcfg query iplisten

Adding an IP address to the iplisten store.
> httpcfg set iplisten -i 10.0.0.1

Deleting a record from the iplisten store.
> httpcfg delete iplisten -i 10.0.0.1

You also need to add the external FQDN to the hosts file - pointing to internal iface so that when the ISA redirects to SSL connections to the FQDN a loop does not occur.

ISA needs to be configured as follows:
Open the ISA management console
Go to properties of the server.
Go to Incoming Web requests tab
Choose 'Configure listeners individually per IP Address'
Select the listener, or create one that listens on the external iface only.
Configure the listener to 'Use a server certificate to authenticate to web clients'
Select Integrated
Configure TCP port to 80 and SSL to 443 and Enable SSL listeners.
Uncheck 'Ask unauthenticated users for identification'
Under web publishing rules
for all the generated 'Small Business ...' rules

On the 'Destinations' tab - make sure the FQDNs are correct and match the FQDN of the cert.
On the 'Action' tab select 'Redirect the request...' to the FQDN of cert.
Check the 'Send the original host header...'
Make sure the port mappings are correct

On the 'Bridging' tab
Redirect HTTP > HTTP
Redirect SSL > SSL
Check both 'Require secure channel' boxes

After that check configuration for conflicting rules in Accss Policy, etc

IIS setup
Make sure the sites configured for ports 80 and 443 are listening on the internal iface only.
Make sure the sites are using the new certificate.

Thats it!

Restart server at this point or stop and start the following processes:
IIS:
IIS Admin
WWW Publishing
SMTP
MS Exchange Routing Engine
MS Exchange POP3
FTP Publishing
HTTP SSL

ISA:
MS ISA Server Control
MS Web Proxy

SBS 2003 no network access after restart

The problem is that DNS is grabbing all the available TCP ports following a Windows Update

If IPSEC is affected it goes into Block mode which prevents any unsecured network traffic to\from the server.

The solution is to add ports to the ReservedPorts list in the registry.

Source URL:
http://support.microsoft.com/kb/956189

To resolve this problem, add the port that is needed by the service to the ReservedPorts registry value. This prevents the DNS Server service from listening on that port. The following ports are known to cause conflicts:
Collapse this tableExpand this table
Ports Program that uses the ports
1645-1646 IAS
1701-1701 L2TP
1718-1719 H.323 Gatekeeper (ISA 2000 only)
1745-1745 ISA Server 2000 or ISA Server 2004
1812-1813 IAS
2883-2883 AUTD
3500-3619 ISA Server 2000 only
4500-4500 IPSEC
To configure the ReservedPorts registry value, follow these steps:

1. Click Start, click Run, type regedit, and then click OK.
2. Locate and then click the following subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. Right-click ReservedPorts, and then click Modify.
4. Type the range of ports that you want to reserve.

Notes
* You must type the range of ports in the following format: xxxx-yyyy
* Do not replace the existing values. Instead, add the additional values.
* To specify a single port, use the same value for x and for y. For example, to specify port 4000, type 4000-4000.
* If you specify the continuous ports separately and if one port is reserved and is not used, the next port is not reserved correctly, and the port is used.
* ISA Server settings only apply to Small Business Server 2000 or to Windows Small Business Server 2003 Premium Edition.
5. Click OK.

Restart the server and things should be sorted.

RWW: Plugin problems

The problem presents when trying to access RWW. After login you select Workstations or Servers - A dialog box appears stating that IE is not able to download an activex plugin.

Cause: The activex plugin included with SP3 is incompatible with RWW.

Solution: Remove all system references to this plugin then re-download the plugin from the RWW site.

The plugin is mstscax.dll in the System32 folder.
Do a regsvr32 /u on the file
Then rename it.
Open regedit and remove all references to it.
When you attempt to access the site again you will be prompted to install a new plugin - probably named msrdp.ocx

RWW: Connections fail

The client could not connect to the remote computer by using Remote Web Workplace in Windows Small Business Server 2003

SYMPTOMS
When users try to connect to a Microsoft Windows Small Business Server 2003 computer by using a Remote Web Workplace (RWW), they receive the following error message:

"The client could not connect to the remote computer. Remote connections might not be enabled or the computer might be too busy to accept new connections. It is also possible that network problems are preventing your connection. Please try connecting again later. If the problem continues to occur, contact your administrator."

CAUSE
This issue may occur if there is a router or a firewall between the SBS 2003 and the client computer blocking the Transmission Control Protocol (TCP) port 4125.

RESOLUTION
To use Remote Web Workplace, you must configure the router or the firewall to forward TCP port 443 and 4125 to the server.

Rebulid the RSM database

Rebuild the RSM database by:

1. Stop the RSM Service
2. Delete the RSM databases located at %SystemRoot%\System32\NtmsData
3. Restart the RSM Service

RDP service not running on server

If the server is up and running - connect to a secondary server (Terminal).

Use MMC to open a computer management console on the server - check event logs of server

If you find the following event

Event Type: Error
Event Source: TermService
Event Category: None
Event ID: 1014
Date: 11/11/2009
Time: 3:32:10 PM
User: N/A
Computer: SERVER
Description:
Cannot load illegal module: C:\WINDOWS\system32\rdpwsx.DLL.

You need to run the following on the server to restart the RDP server service.
echo yes | reset session RDP-tcp

For this you need the PsExec command.
URL: http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx

Unzip and open a command prompt in that directory.

Run the following:
psexec \\servername cmd

You will then have a command prompt on the remote server. You should now be able to execute the command on the server.

RDP should now be working on the server.

This particular fix can appear after restarting following applying Windows Updates.

Problem accessing network shares via VPN - Vista

Hi all,

I recently got a new laptop from work with Vista. I have set up a microsoft VPN connection to my workplace and can connect to it. I can ping the file and print server on the network, however, I cannot access the share. I have my home PC with XP and there everything works fine. It is therefore a setting in Windows Vista. Can anyone help me with this?

>>>

I can't say this will fix the problem but is fast and simple so worth a shot. Open the Local Security Policy, select Local Policies, Security Options. Look for Network Security LAN Manager Authenicate Level and change it to Set to Send LM & NTLM responses. Not sure I have everything worded a 100% correctly but that should get you there.
Good luck.

DNS
Try adding the domain as a DNS suffix on your Vista PC. Because your PC isn't part of the domain you will not be getting DNS from the domain server therefore (basically) you wont be able to access shares.

1.) open regedit
2.) find HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\netbt\Linkage
3.) copy the value of the bind entry to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\smb\Linkage
4.) reboot

http://support.microsoft.com/kb/933468

Proactive Maintenance

Procedure for proactive maintenance

- Get CCleaner, Smartdefrag, Crystal Disk Mark, Windirstat and Belarc Advisor onto server
- Install apps (if TS then make sure shortcuts are only applied to administrator profile)
- Use Belarc to identify system model. Add this info to Managed Server configuration if applicable.
- Check if RAID tools are availble otherwise locate and install.
- Assess backups with customer or primary tech. Add this info to Managed Backup configuration if applicable.

- Check Application and System logs for unusual problems.
- Check for backup software - if SBS check for SBS backups. Check Tape drive/ free space on Ext HDD
- Check backup logfiles.
- Check that backup failure notifications are enabled.
- Refer to Managed Backup configuration in CW for info.
- Check SBS alert and notifications if applicable
- Check AV system installed - check for upgrades, definitions, licensing.
- Check Spam filtering - Trend/GFI
- Check RAID tools for failures. Check disk free space. Run CCleaner. Schedule SmartDefrag to run A\H.
- Check WSUS configuration - Automatic approvals, sync schedule, GPO settings for server/workstations.
- Check DFS replication status if applicable.
- Check for Windows Updates - Schedule for A/H restart - notify company if nessesary.

Performance: Task Manager

Introduction to Windows Server 2003 Task Manager

Do not neglect Windows Server 2003's Task Manager. There will be occasions when you just need a quick piece of system information, calling for your task manager can be just as effective as the performance logs but much faster.

Execute the Task manger by holding down the CTRL +SHIFT +ESC keys; if you have long fingers, try launching it with just your left hand. (As ever Microsoft provide 3 ways of doing everything, so you can right click the grey bar at the bottom of your screen and select Task Manger.)

Each of the five tabs has it's own personality, I recommend that you get to know the capabilities of each one. An application tab for zapping programs, a process tab for checking for viruses, a performance tab to calculate the pagefile, and now a network utilization tab to see how much activity there is on the LAN or WAN.

Topics for Task Manager in Windows Server 2003
* Applications Tab
* Processes Tab
* Performance Tab
* Networking Tab
* Thousand Separators
* Users Tab (Workgroup)
* Task Manager Trap

Applications Tab
No doubt you have already used task manager's Application Tab to end process for programs that hang or do not respond. Assuming that your group policy allows users access to this tab, why not send out an email reminding users of how to use this Applications tab when their machine hangs and programs are 'Not Responding'.

Another use of the Applications Tab is to discover the underlying Image Name of a program, simply right click and then select: Go to Process.

Tip: One of the joys of using any new program is setting the preferences. With Task Manager I like to remove the tick which says ' Always on top'. (Option Menu).

Processes Tab
Here is a tab for the support professional, in fact the more 'Image Names' that you can identify the better techie you are.

The more of these Image Names that you can match to programs or processes, the easier it is detect impostors such as viruses. Moreover, by identifying the 'good guys' you will learn how applications such as Exchange and SQL interact with the operating system.

At first svchost seems suspicious. Could you have been infect 7 times by a strange virus? No, svchost is the generic name for an image shared by the operating systems services, for example, Alerter, Net Logon, Print Spooler. The interesting fact about all these svchost is that some services would fight if put together in the same Image Name process, so the operating system separates incompatible services and puts them in separate svchosts.

Returning to the theme of identifying rogue programs; what would think if you saw Avgserv and Agvcc32 amongst the image names? When I saw these processes, I must admit my heart missed a beat. At first I thought my machine has been infected by a virus, but no, it was actually my virus checker which had installed itself as a process. Naturally I left that running!

How about msblast.exe? Was this a game that my nephew had installed? Well I tried a search in Google and up came W32/BlasterA virus. Here was a case where I needed to check the registry as the blaster virus cunningly re-infects those who are not diligent. Other viruses have more innocuous names like Tlntsvr.exe and Wina.exe, so this is why I urge you to know the Processes Image names.

View Menu, Select Columns
When using the Processes tab to troubleshoot be aware that you can add extra columns, for example Virtual Memory, IO Reads and Writes. Again my idea is to use the Task Manager to display information quickly without resorting to setting up the System Monitor and Log Counters.

Adjust Priority, Set AffinityTask Manager Processes Tab Windows 2003
If you right click any process then you can adjust Priority, typically you have a spreadsheet calculating in the background while you want to give a foreground application like Outlook more CPU time slices. Note, never set programs to Real Time, it will cripple your machine and require a reboot - you have been warned.

Affinity means that one program can be associated exclusively with one processor, naturally you need a multi processor machine for this setting to even appear. In truth, Affinity is a case of 'Mother knows best'. Guy says leave alone, unless someone has got in before you and incorrectly set affinity, leave it to the operating system.
Performance Tab

My main use of this Tab is to check memory. In particular to check on the pagefile. Now I have read many articles about how big to set the page file. Experts say use RAM x 1.5 whilst others say RAM x 2. Guy says suck it and see. By that I mean compare two figures: Commit Charge (K) Limit with Peak. You may have already guessed that the Commit Charge (K) is the sum of RAM + Pagefile

In Diagram 2 the Limit is 1280032
Whilst the Peak is 617564. My interpretation is that this machine has a big enough page file. However if the Peak was 1MB (1000000), then I would declare that the pagefile was not big enough and take action in the System Icon. If the Peak was below 400000 and I was desperate for the disk space, then I would even reduce the pagefile.

My attraction to ipMonitor is because it inhabits that zone of part work, part play; Guy just could not put the dashboard away. This excellent performance monitor will get you started in the quest to remove bottlenecks on your network. SolarWinds provides this fully-functioning product free for 21 days. So download and install ipMonitor, then start scrutinizing your computers CPU, memory and disk performance. You can also select from zillions more performance counters such as fan temperature and battery level.

Installing ipMonitor is a breeze, but learn from gung-ho Guy's mistake and install SNMP on each computer that you wish to monitor. What sealed my unreserved recommendation of SolarWinds is their support team, you will get expert help even when you are evaluating the ipMonitor.

My Pet Hate - No Commas
Jimmy May kindly wrote in showing me how to hack the registry and turn-on the thousand separators. This is how you do it:

Turning On Thousands Separators in Perfmon
Turn on thousands separators (commas or periods, depending on your locale) in Perfmon to make large numbers much easier to read.
[HKEY_CURRENT_USER\Software\Microsoft\SystemMonitor]
"DisplayThousandsSeparator"=dword:00000001

Turning Off Vertical Bars in Perfmon Charts
When a perfmon log contains many sample points, it may collapse multiple samples into a single point on the X axis, and graph a vertical bar that represents min/max/avg for the collapsed samples. This can be a useful feature, but it can also make the graph hard to read (trends are harder to discern) when viewing many counters. You can set the following registry value to turn off the vertical bars:
[HKEY_CURRENT_USER\Software\Microsoft\SystemMonitor]
"DisplaySingleLogSampleValue"=dword:00000001

Leaky Memory
Keep your eye on Kernel Memory, Nonpaged. These days servers need rebooting less frequently, if you have a 'leaky app' then Nonpaged memory will creep up over time. Confirm any suspicions with View, Select Columns, Non-Paged Pool.

Networking Tab
This is great utility to view network utilization. This is a new Task Manager interface in XP and Server 2003. In fact it is difficult if not impossible to create performance logs to measure % Network utilization.

Users Tab Task Manager Performance Tab Windows 2003
Call me a 'Luddite' but I have never had much use for this tab on my Windows 2003 Server. When I want to check on users connected to the server, I prefer the Shared Folder snap-in.

Another reason that I do not use the Users Tab is that with XP you only see this machine if it's part of a Workgroup and you are using 'Fast User Switching'.

Task Manager Trap
When I was a 'greenhorn, (many, many years ago) I fell into the trap of carelessly double clicking inside Task Manger, as a result the top menu with File, Option, Help disappeared. I could not understand what had happened until I double clicked near the top, and lo and behold the menu reappeared. It was a beginner's mistake; I mention it because there are about 3 or 4 other Microsoft programs that display the same menu behaviour.

Performance: Processor

Introduction to Processor Bottlenecks

Processor usage (also called CPU usage) is the easiest bottlenecks for performance monitor to detect. What we are looking for is the percentage time that the processor is in use rather than just ticking over by running an idle thread. If the CPU is so busy that it cannot respond to requests, then the whole server's performance soon deteriorates rapidly.

Database and email servers are the most likely to suffer from processor bottlenecks. On the other hand, file and print servers are less likely to be short of CPU power. However, large modern servers invariably have multiple symmetric processors so % Processor Time bottlenecks are becoming rarer than memory bottlenecks. That reminds me, always monitor the other major counters, Disk, Network and Memory.

If you discover a processor bottleneck, then use the process counter in performance monitor to identify which program or process is hogging the server. Also check out drivers and disk subsystems to pin point the source of the processor activity.

Note: there are two Performance Monitor counters with very similar names, processor and process, on this page we are investigating processor (CPU).
Processor Topics

* Processor: % Processor Time
* System: Processor Queue
* Multiple Processors
* Other Processor Counters
* Solutions to processor problems

As a quick way of checking processor usage, call for the Task Manger, Performance Tab. Ignore spikes but check for high continuous CPU Usage History.
Processor: % Processor Time

An overloaded processor has a distinctive and unmistakeable performance monitor profile. The % Processor Time trace looks like a curtain hanging down from an imaginary ceiling. See Diagram 1.Performance Monitor Processor bottleneck

Text books quote thresholds of between 70 - 85 percent for % Processor Time, the key point is that the counter is continuously high. It is normal for the trace to show a sharp increase when any program executes; you can safely ignore spikes.
System: Processor Queue

The hardest part of using this performance monitor counter is remembering to go to the System object (not the Processor object). What I love about any queue counters is that it is easy to remember the threshold. The rule of thumb is that the threshold for a queue bottlenecks is 2.

Other Counters

You may also wish to examine DPCs Queue/sec. This is where the server was busy so it deferred processing a request. High or intermittent bursts of Interrupts /sec could indicate a hardware problem or a loose component.

Multiple Processors

With multiple processors, it is reasonable to divide the System: Processor queue by the number of processors. So a twin processor could sustain a queue of 4.

The second rule of thumb is: you are allowed to divide the queue by the number of multiple components. (Processors, Disks or NICs)

As a point of monitoring technique, twin or quad processors give you a chance to compare Processor: %Processor Time Instances, rather than just recording the _Total.
Other Processor counters

1) Process and Thread
If you find a processor bottleneck you can pursue the cause by measuring the Process or even Thread object. What you are looking for is which instance of the Process is responsible for exhausting the processor.

2) System: %Privileged Time and Process: %Privileged Time.
The Windows 2003 operating system can execute, either in Kernel mode, which shows up as %Privileged Time, or User mode which corresponds to %User Time. This means that activities of programs like SQL or Exchange are charged to %User Time.
Here is a combination which would point to an I/O bottleneck, System: %Privileged Time > 20% and PhysicalDisk %Usage > 55%.

3) DPC
DPC means Deferred procedure calls - The processor is saying 'I am busy I will do this low priority task later.' Processor %DPC Time > 50% is suspicious and may indicate a network card bottleneck.

Solution to processor problems
Getting a second processor will work wonders for servers where the processor is being stressed. Upgrading the processor is another obvious solution for a stressed processor.
When you order the next server consider making it a quad processor. Talking of new kit, when next you spec a new system, consider clustering. Often you need two reasons to break new ground. In addition to the obvious advantage of fault tolerance, clustering can gain extra performance through load balancing across the multiple processors.

Performance: Network

Introduction to Network Bottlenecks

Running programs over the network is likely to result in network bottlenecks. Other causes of high network activity are, roaming profiles, file copying or large print jobs. Use performance monitor to create a log and calculate the network utilization.
Network Topics

* Key Network Counters
* Detecting a Network Bottleneck
* Problem with Network Segment / Network Utilization
* Network Monitor
* Additional Network Counters
* A Vision of Network Monitoring
* Solutions to Network Problems

Key Network Counters

Network Interface\ Bytes Total/sec
Network Interface\ Bytes Sent/sec
Network Interface\ Bytes Received/sec
Network Interface\ Current Bandwidth
Detecting a network bottleneck

There are several cautionary tales with this performance monitor chart. Let me see if I can convince you that the maths prove a network bottleneck.

The Bytes / Sec average 913,999. Beware, System Monitor never uses thousand separators which make the figures awkward to read. I have often revisited the data because I was out by a factor of ten. For example, at a glance you could mis-read 913999 as approximately 91,399 or even 9,1399,999.

Performance Monitor - Network Utilization

Diagram 1

You may think that just comparing the red line (Current Bandwidth) with the white line (Bytes /Sec) proves that the network is at full capacity. On closer inspection of the Scale, you realize that the red line is 10x bigger than the white line. Now you may revise you estimate and believe that the network is only running at 10% of capacity. However, there is one more factor, the red line is in bits while the white line is in bytes. 1 bytes = 8 bits. When you compute all these factors, the network is actually running at 73%.
Summary

White Line Bytes /Sec = 913,999 bytes x 8

White Line Bytes /Sec = 7,311,992 bits.

Red Line Current bandwidth = 10,000,000 bits

Network Utilization = 73.12%

One of the amazing features of the original ethernet network is that only one machine can transmit at a time. Once the network reaches 30% capacity, pure chance means that two machines try and send a packet at the same instant. The result is more and more collisions start happening, this leads to re-transmissions and a slow down of network traffic.

Networks bottlenecks occur at surprisingly low levels of utilization. 40% would normally be considered a bottleneck, and the only reason that I got a higher value was that there are only three machines on my test network. The more machines the greater the risk of collisions from two machines wanting to transmit at once.

However, with modern production networks two items of technology have alleviated the above bottleneck problem:
a) Switched networks replacing hubs
b) 100MB network cards replacing the old 10MB cards.

Problem with Network Segment \% Network Utilization

The problem is that it seems to have been discontinued in Windows 2000 onwards. Several respected sites claim you can install Network Segment via the Network Monitor, or Protocol Driver. For what its worth I have tried numerous techniques of getting the counter to appear in performance monitor without success - I agree with Microsoft Network Segment is no longer available.

Microsoft's TechNet article 253790 Says:

SUMMARY
In Windows 2000, installing Network Monitor does not add the Network Segment object in System Monitor as in Microsoft Windows NT.
MORE INFORMATION
To increase the stability and reliability of the Windows Management Instrumentation (WMI) interface, this object has been removed from System Monitor in the Performance tool in Windows 2000.
Network Monitor Network monitor Utilization

Microsoft have supplied Network Monitor from NT 4.0 onwards. I mainly use the Network monitor to troubleshoot connectivity problems, however you can use it to check network utilization. The diagram to the right shows a peak of about 85% and an instantaneous value of 81 or 82. This is merely a snapshot and unfortunately the Network monitor does not keep permanent records of network utilization.
Additional Counters to investigate

UDP\ Segments Received/sec
UDP\ Segments Sent/sec
TCP\ Frames Sent/sec
TCP\ Frames Received/sec


Server\ Bytes Total/sec
Server\ Bytes Received/sec
Server\ Bytes Sent/sec

Solutions to Network Problems

* Add another network card
* Segmentation - change your subnet mask, add routers
* Switches - install a packet switch
* Reduce Protocols - remove NWLink if you have no more Novell 3/4 servers
* If you must have more than one protocol, check the bindings order in the network icon

Performance: Memory

Introduction to Memory Bottlenecks

If ever your Windows 2003 server is running slowly, then the first place to look for a bottleneck is memory. Another way of looking at server performance, is that machines with plenty of RAM rarely give problems. A bonus of plenty of memory is, that to a degree, abundant RAM compensates for strain on other resources.

On old servers, lack of memory would give you the full sensory input, you could hear the disk paging, see the light flashing, and 'Mad' Mick swears you could smell the disk thrashing. Even with these sensory clues, it is still worth while monitoring memory with Performance Logs. Please also remember the big picture. So once you have had a quick look at memory, remember to check the processor and disk counters.

The servers most likely to suffer from memory shortage are pure database servers for example, Oracle or SQL. Email server also require plenty of RAM. Pure domain controllers are less likely to experience memory problems.
Memory Topics

* Available Bytes
* Three Paging counters
* Other Memory Counters
* Identify the root cause of a memory bottleneck
* Advice for application servers e.g. Exchange, SQL or Web
* Solutions to memory problems

Key memory counter: Available Bytes

The more available memory the faster the server can respond. When I check a server's memory with performance monitor, the first counter that I add to the log is Memory\Available bytes. As long as the trace indicates more than 10MB of free memory, then I conclude that the server has sufficient RAM.

Diagram 1 shows a white descending line, and the legend confirms that Available bytes are down to 3MB. Clearly this machine needs more memory.Performance Monitor, Memory bottleneck, Available Bytes

Suppose a spreadsheet wants to start a new thread or a database needs to sort data, what each needs is memory. The operating system provides this memory at least 100 times faster using RAM, than it could using a disk based pagefile. This is why a large pool of free memory is so important to an application server.
Three paging counters in performance monitor
1) Memory: Pages / sec

Take care to distinguish between these two paging counters:
1) Pages /sec (Hard page faults)
2) Page Faults /sec counter is likely to be at least twice the value of the above.

Two problems with monitoring in general, firstly no counter should be taken in isolation, secondly spikes should be ignored, or at least played down.

The less paging the better your server's performance. Most authorities agree that Memory: Pages / sec is a key memory counter. This counter measures 'hard' page faults, in other words the page in nowhere in memory, so the VMM (Virtual Memory Manager) has to fetch the data from the pagefile on the disk; in computing terms that takes an age.

I am reluctant to disagree with other authorities, but from my experience, I would put the threshold as high as 20 pages /sec, before blaming paging as the bottleneck. Moreover, I would not trust pages /sec as an indicator of a bottleneck without confirmation from low Available bytes. (see above)

In truth, if you put 5 experts in the same room, they could all spot a memory bottleneck, but when they wrote up their notes, they would use different time slices and different thresholds, consequently, it would seem that there was a conflict where none actually existed.

2) Memory: Page Faults / Sec

Page faults / sec is the sum of hard and soft page faults. Soft page faults are where the data is found elsewhere in RAM. For example, Word has opened the spellchecker, and now Outlook wishes to use it, there is no need for another call to the disk as the spellchecker is already in memory.

Hard page faults are generated whenever the VMM has to fetch data from the pagefile on the disk.
3) Page File: % Usage

While the pagefile is less likely to be a bottleneck it is easy to check, and satisfying to fix. You could also confirm that it is on the most suitable disk, and if possible split the pagefile over two disks and thus improve access times. (Note the object here is Page File not Memory)

Because the changes are so gradual, you are better off using this Page File % Usage counter as an alert, rather than a log. I suggest setting an alert on a limit of over 70.

Creating a Memory Bottleneck

If you really want to see a memory bottleneck that you can measure with performance monitor, then add the MAXMEM switch to your server's boot.ini. For example: multi(0)disk(0)rdisk(0)partition(1)\Windows="Windows Server 2003" /MAXMEM=256
Other memory counters
For leaky Applications
Memory: Pool Non-paged Bytes

Thanks to better programming, the problem of applications leaking memory is slowly disappearing. Programs need memory - fair enough, but when they close, if they fail to release the memory back to the pool - not fair.

Monitoring the slow build-up in the non-paged pool is a classic job for an alert. You are unlikely notice much creep in a day's logging, but over a month you could have lost a significant amount of memory. If you are in a hurry or just plain ruthless, a reboot solves memory leakage into the non-paged pool.
Memory: Cache Bytes

With Memory: Cache Bytes, you have the choice of monitoring the total or breaking down the cache into four categories.

Note Memory: Cache Bytes = Total of these four individual counters

* Memory: System Cache Resident Bytes
* Memory: System Driver Resident Bytes
* Memory: System Code Resident Bytes
* Memory: Pool Paged Resident Bytes

Cache: Data Map Hits %

Data Map Hits is the percentage of data maps in the cache that could be resolved without having to call the disk. It is a measure of how many data pages are already in physical memory. (Note the object is Cache not Memory like the others.)
Committed Bytes

If the value for committed bytes is greater than physical memory, then more RAM would help.
Identify the root cause of a memory bottleneck

Your goal is to discover which application (process) is using most of the memory. Either quickly check the Processes tab of Task Manger or better, create a log of the processes with performance monitor. Developers could consider investigating other tools such as PViewer from the Resource Kit.
Advice for application servers e.g. Exchange, SQL or Web

Many applications have their own settings for configuring memory usage. It is not that they can magically produce more memory but they can make the most of the memory the server has. Programs like SQL or Exchange have settings to control their caches. The key question is: should the memory be tied up in a file cache or would it better used by the application's own cache? I favour the latter. So adjust the applications cache or memory settings, and then see what effect your changes have by logging with System Monitor.

See more about Exchange Memory here
Solutions to memory problems Memory Bottlenecks

Not enough available memory? The easiest cure for memory problems is to open up the server and add another stick of RAM. Salesmen always seem particularly helpful and knowledgeable on the best type of RAM for my server.

I have a suggestion for future purchases - Always buy machines with more RAM than you need now.
More help for detecting bottlenecks

Performance: Exchange 2003

Introduction to Exchange 2003 Bottlenecks

Exchange Performance Counters

When you install Exchange 2003, not only does it expand the schema but it also adds specific exchange performance counters. Incidentally, you also see more services, for example Microsoft Exchange System Attendant.

As you look for bottlenecks, think how you could improve the service as cheaply as possible, for example move mailboxes to another server.
Exchange Performance Topics

* Tactics to Detect Bottlenecks
* Monitor Queues
* Other Exchange Counters
* Solutions to Exchange bottlenecks
* Summary of Exchange Performance Monitoring

Tactics to Detect Bottlenecks

Firstly remember best practice and monitor the Windows Server 2003 'big four' counters, memory, processor, disk and network. Next look for specific Exchange bottlenecks in these areas:

* MTA (Message Transfer Agent)
* SMTP
* Internet Mail Service
* Directory Service

The more you understand the way Exchange 2003 works, the more you can identify then predict where bottlenecks occur. Performance monitor will help you not only improve responsiveness, but also show you how the Exchange components route the mail messages.

Monitor QueuesExchange MTA Queue

The key to detecting Exchange 2003 bottlenecks is to find, then measure queues. Guy's rule of thumb is a queue bigger than 2 is evidence of a bottleneck.
MSExchangeMTA - Work Queue Length

This is the most important counter because it gives the total of all queues.

For more precise queue monitoring check

* SMTP Server -Local Queue Length
* SMTP Server Messages received /sec and delivered /sec
* MSExchangeMTA Connections - Queue Length
* MSExchangeIS - Send and Receive queues size
* MSExchangeIS Mailbox -Messages Sent/min Messages Delivered/min
* MSExchangeIS User Count

Solutions to Exchange 2003 Bottlenecks

* Increase memory, add another stick of RAM.
* More disks, with more disk controllers.
* Put Exchange on its own dedicated server.
* Add a second or third server so that each site has a local Exchange server.
* Deploy front-end / back-end servers. The front-end servers handle the connections and authentication, the back-end store the actual mailboxes.
* Lateral thinking and root cause analysis. Too much spam, get a good filter, swamped with the SoBig virus, better virus checker.

Summary to Exchange Health Checks

Monitoring Exchange Servers is probably the most difficult of all monitoring tasks. The secret is to start by examining the Windows Server 2003 'big 4' counters, Memory, Processor, Disk and Network. Once you have checked that they are not causing the bottleneck, then seek the Exchange specific counters, for example, MSExchangeIS and MSExchangeMTA.

Performance: Disk

Introduction to Disk Bottlenecks on Windows Servers

This page will explain how to use performance monitor to log disk counters. I will also recommend solutions to disk bottlenecks on Windows 2003 Servers.

Firstly, a homily to explain why you should always monitor these 'big four' objects: Memory, Processor, Disk and Network. Beware of monitoring one counter in isolation because that can lead to the wrong conclusions.

One company thought they had a problem with slow disks on a Windows 2003 Server. Performance monitor confirmed long queues and slow disk access times. Their conclusion was that the bottleneck was the disk and so they bought faster disks. Unfortunately, the slow response persisted and they called me in to investigate. By monitoring all the 'big four' performance objects, I found excessive paging, there was also less than 2MB of available bytes. The true ailment was lack of memory, high disk usage was a symptom and not the cause. The lesson: incomplete monitoring can mean a waste of time and money, so always record these four objects:- Memory, Processor, Disk and Network.

The Windows server roles most likely to experience disk problems are, web servers with lots of graphics and file servers. On the other hand, Domain Controllers, DNS, or DHCP servers are unlikely to have disk bottlenecks
Disk Topics

* Basic disk counters
* Disk Bottleneck - Queues
* Solutions to Disk problems
* Diskperf -y (New settings in 2003)
* Summary of Disk Monitoring

Basic counters to monitor disk activity
PhysicalDisk

1. PhysicalDisk: Avg. Read Queue Length Should be less than 2
2. PhysicalDisk: Avg. Write Queue Length Should be less than 2
3. PhysicalDisk: % Disk Time more than 50% indicates a bottleneck

Disk Bottleneck - Queues Performance Monitor Disk Queues

In Diagram 1 performance monitor shows classic symptoms of a disk bottleneck. My diagnosis is based on the Disk write queue counter, you can see that this queue averages more than 2. In fact the average is nearly 4 (with a peak of over 8).

I wanted to to be unbiased. So, to ensure that it was not a processor or memory bottleneck, I also recorded % processor time and available bytes. As you can see from Diagram 1, the processor's average was below 30%. If the processor were the bottleneck the trace would be over 80%. On the other hand, if there was a memory shortage, available bytes should drop below 10MB. The graph show there was always 70 MB of Available MBytes.Performance Monitor Disk Bottleneck

The performance bottleneck may be worse than the average figures above suggest. In Diagram 2, I have legitimately chopped the graph to isolate the period of intense disk activity. For these 5 minutes (4:46) the average is almost 6 against the bottleneck threshold of 2.

The other difference is that in Diagram 2 (taken from performance monitor), I have included % Disk Time, this exceeds 100% for the duration of the trace. In other words, the disk is working flat out writing data to to the hard drive.

There is one more deduction we can make from the queue data on the chart. If you compare the white line with the thick green line near the bottom, you can tell that the disk is writing more rather than reading. To see the diagrams more clearly, double click and expand the thumbnails into larger diagrams.

®
Solutions to Disk Problems
Defrag your disks

Once disks fill to 70% capacity they slow down dramatically. The other side of the coin is that a defrag can cut queues in half. Incidentally, I am always on the lookout for such cost-nothing solutions.

Starting with Windows 2000, Microsoft have licensed part of Diskkeeper. What you can do is defrag a server drive-by-drive. What you cannot do is schedule a defrag for the middle of the night, neither can you select multiple drives for defragging. So the answer is to get a good third party defragger like Diskkeeper's full product.
Faster disks

The logical solution is to buy faster disks. Go to your existing disk manufactures site and compare their figures with the data you collect for:

PhysicalDisk: Disk Read Byte /sec

PhysicalDisk: Writes /sec
Other Servers

Another cost-nothing solution would be to move the files or database to another server. Alternatively you could use the load-balancing properties of DFS.
Disk Striping

This would be my least favoured option. Technically it is a neat idea, to stripe data across two or more disks. The principle reminds of school days when I had to write out, 'I must not run across the school grass' 500 times. To speed up the process I wrote my lines with 3 pens at once. The multiple disk controllers, like my pens, write simultaneously across three disks. The reason I am wary of this method is that there is no redundancy, if any one disk fails you would lose all the data. Of course you could use hardware RAID 5, 10 or 20 which would protect your data against one disk failing.
Solarwinds IpMonitorGuy recommends: The SolarWinds ipMonitor

My attraction to ipMonitor is because it inhabits that zone of part work, part play; Guy just could not put the dashboard away. This excellent performance monitor will get you started in the quest to remove bottlenecks on your network. SolarWinds provides this fully-functioning product free for 21 days. So download and install ipMonitor, then start scrutinizing your computers CPU, memory and disk performance. You can also select from zillions more performance counters such as fan temperature and battery level.

Installing ipMonitor is a breeze, but learn from gung-ho Guy's mistake and install SNMP on each computer that you wish to monitor. What sealed my unreserved recommendation of SolarWinds is their support team, you will get expert help even when you are evaluating the ipMonitor.

Download SolarWinds ipMonitor (21 days eval)
Diskperf -y and Performance Monitor

Diskperf's overhead is very small and my advice is to leave it turned on. Another hint that this is the correct approach is that Windows 2003 has diskperf on by default. If you have Windows 2000 and you do not set diskperf -y then you are storing up a problem for when you ever do need to measure disk performance. The problem is that setting diskperf needs a reboot and it would be most inconvenient when you are keen to get on with the troubleshooting.
Perfmon situation 2000 and 2003

DISKPERF [-Y[D|V] | -N[D|V]] [\\computername]

-Y Sets the system to start all disk performance counters when the system is restarted.

-YD Enables the disk performance counters for physical drives. when the system is restarted.
-YV Enables the disk performance counters for logical drives or storage volumes when the system is restarted.
-N Sets the system to disable all disk performance counters when the system is restarted.

-ND Disables the disk performance counters for physical drives.
-NV Disables the disk performance counters for logical drives.
\\computername Is the name of the computer you want to see or set disk performance counter use.
The computer must be a Windows 2000 system.

NOTE: Disk performance counters are permanently enabled on for
systems beyond Windows 2000.
Summary for Disk Monitoring

Be aware that with Windows Server Disk monitoring there are both physical and logical disk counters. Disk activity could mask memory shortage, so always monitor the 'big 4' counters, Memory, Processor, Disk and Network.

Outlook File Locations

Autocomplete Cache:
On Vista - Users\(username)\AppData\Roaming\Microsoft\Outlook
On XP - C:\Documents and Settings\(username)\Application Data\Microsoft\Outlook

When using Windows 2000/XP, Outlook stores *.pst, *.ost, *.pab, *.oab, and extend.dat in
C:\Documents and Settings\(username)\Local Settings\Application Data\Microsoft\Outlook.
Extend.dat, Hotmail, IMAP, and MSN Connector message stores can not be moved.
All other Outlook configuration files are stored in
C:\Documents and Settings\(username)\Application Data\Microsoft\Outlook and cannot be moved.

Vista replaces Documents and Settings folder path with a Users path.
As a result, in Vista you'll find Outlook stores the PST and OST, in
C:\Users\(username)\AppData\Local\Microsoft\Outlook.
Hotmail, IMAP, and Live Connector message stores can not be moved but POP3 and Exchange offline or cache storage can be moved to other locations.
All other Outlook configuration files are stored in
C:\Users\(username)\AppData\Roaming\Microsoft\Outlook and cannot be moved.

If your mail profile is called "Outlook", your nk2-file is called outlook.nk2.
If you named your mail profile "Maslan" then your nk2-file is called maslan.nk2.

Network Performance Troubleshooting

There are 3 points of failure on this issue - OS - Filesystem - Network

To start we need to determine the raw max throughput of the network layer:
- run Iperf on source and destination servers. Source URL: http://sourceforge.net/projects/iperf/
This will be the baseline speed
- if this is really low for the network type (Wireless, 10\100, Gigabit, FC, etc) then network is the root cause.
Use http://www.ibeast.com/content/tools/band-calc.asp to convert output to something understandable.

Next we need a real world test:
- Do file copy of a large file using windows explorer from source and destination servers.
- If real world performance is far off baseline speed we need to rule the OS and filesystem out:

Start with the filesystem:
- Run CrystalDiskMark on both servers. Source URL: http://crystalmark.info/software/index-e.html
- If this exceeds the real world performance then the OS is the bottleneck
- if this matches real world performance - then the filesystem is the cause of the slowness.

Next step is to focus efforts on the worst performing of the 3 and things should improve.