I've just been working on a monitoring PC which have, which uses server perfmon to monitor the health of all our servers. Graphs include items such as CPU load, RAM available, and disk queue. I've also put a little system in place which will send an alert to twitter in the event of a server crash, or non-responsive ping;
It requires 2 batch files. The first one, looks like this;
--------------------------------------
@ECHO OFF
cls
REM Send 1 ping to server
ping %1 -n 1
IF %errorlevel% == 1 goto ServerFail
EXIT
:ServerFail
REM The next ping is just to force a few seconds of delay befor re-testing
@ping 127.0.0.1
@ping %1 -n 1
IF %errorlevel% == 1 goto ServerFail2
EXIT
:ServerFail2
CLS
@echo Server %1 Failed to ping on more than one occasion
tweetc "Server %1 was showing communication issues at %TIME% on %DATE%"
--------------------------------------
When calling the bat file it takes one parameter, which is the server name or IP address.
The second batch file simply calls pinger.bat for every server we want to run a check against.
You will notice a command in there "tweetc". This is a niftly little utility which allows you to tweet via the command line in windows.
If a server fails to ping, a delay is implemented, and then another ping test is done. If the second test fails, the message is sent to twitter saying "Server whatever was showing communication issues at %TIME% on %DATE%"
Its fairly crude in that its only relying on ping as a test, but its a good front line, and can be used in conjunction with more complex reports such as SMTP traps, or even using event logs.
If you want to see the live twitter go to http://twitter.com/bcolservers There isnt much on there at the moment aside from testing, and hopefully there never will be!
A little known feature which many of our users seem to find useful has been Shadow Copies. Often used from an Admin point of view, by systems such as Virtual Machine Manager, shadow copies can also be used by the end users. This allows people to restore their own lost/deleted files, and roll back to a previous version.
It does have its disadvantages, such as the number of available previous version available, and the overhead on storage and disk utilization, but overall it reduces calls to the helpdesk, provides users with a faster way to recover lost data, and minimizes the time your backup administrator has to spend sifting through tapes for lost files.
Video tutorial is here;
It does have its disadvantages, such as the number of available previous version available, and the overhead on storage and disk utilization, but overall it reduces calls to the helpdesk, provides users with a faster way to recover lost data, and minimizes the time your backup administrator has to spend sifting through tapes for lost files.
Video tutorial is here;
I thought it would be worth a post on my recent experiences with an iSCSI San, and a Server 2008 Cluster. First off this is using the Release 1 of 2008, not R2 which is currently in BETA and has extended support for iSCSI.
The situation we wanted to use iSCSI San is probably much the same as most, virtualization. We are using Hyper-v along with SC Virtual Machine Manager 2008.
I’ve managed to put this into a nine step process, but if you have questions about any of the steps drop me an email;
1.Setup your SAN, install Server 2008 on desired number of nodes in the cluster, and use the MS iSCSI initiator and MPIO to connect to the SAN. Your SAN provider may provide specific tools to assist with this, such as DELL’s host integration toolkit for the Equillogic units.
2.Validate and create the cluster using Failover Cluster Management
3.Create a very small (1.5GB) LUN on your SAN, this will be used for the Witness Disk by 2008, and holds important configuration information about the cluster. Also create a number of LUNS for hosting virtual machines. Size doesn’t matter too much at this stage as they can be extended using the SAN, and Server 2008 disk management.
4.Connect all the nodes in cluster to the LUNS. Bring online, initialise and format the LUNS. Your best bet is to use a GUID partition, without a drive letter. Otherwise you may end up running out of letters in the alphabet if you have a large number of VMs.
5.Using Failover Cluster Management, add the disks into the cluster, and ensure that the small 1.5GB LUN is set as the Witness Disk. Quorum settings will depend on the number of nodes in the cluster, but the wizard does advise accordingly.
6.Using SCVMM, add one of the nodes from the cluster as a host. VMM will automatically detect that this is a clustered node and add all the other nodes.
7.Ensure that the status of each node shows everything being upto date, and check the properties of the cluster and make sure there is available storage.
8.Create a VM and add it to one of the nodes in the cluster.
9.Test functionality by shutting down the node holding the new VM, and ensure that another node takes over.
When you create your VMs, be sure to set them as highly-available. The above should give you the ability to create highly available VMs, ensuring that if node or nodes in the cluster fail, the VMs will continue to function.
1.Setup your SAN, install Server 2008 on desired number of nodes in the cluster, and use the MS iSCSI initiator and MPIO to connect to the SAN. Your SAN provider may provide specific tools to assist with this, such as DELL’s host integration toolkit for the Equillogic units.
2.Validate and create the cluster using Failover Cluster Management
3.Create a very small (1.5GB) LUN on your SAN, this will be used for the Witness Disk by 2008, and holds important configuration information about the cluster. Also create a number of LUNS for hosting virtual machines. Size doesn’t matter too much at this stage as they can be extended using the SAN, and Server 2008 disk management.
4.Connect all the nodes in cluster to the LUNS. Bring online, initialise and format the LUNS. Your best bet is to use a GUID partition, without a drive letter. Otherwise you may end up running out of letters in the alphabet if you have a large number of VMs.
5.Using Failover Cluster Management, add the disks into the cluster, and ensure that the small 1.5GB LUN is set as the Witness Disk. Quorum settings will depend on the number of nodes in the cluster, but the wizard does advise accordingly.
6.Using SCVMM, add one of the nodes from the cluster as a host. VMM will automatically detect that this is a clustered node and add all the other nodes.
7.Ensure that the status of each node shows everything being upto date, and check the properties of the cluster and make sure there is available storage.
8.Create a VM and add it to one of the nodes in the cluster.
9.Test functionality by shutting down the node holding the new VM, and ensure that another node takes over.
When you create your VMs, be sure to set them as highly-available. The above should give you the ability to create highly available VMs, ensuring that if node or nodes in the cluster fail, the VMs will continue to function.
Thursday
Going back to another problem, I've decided that in order to create VMs on-the-fly, I'm going to need a list of MAC-Addresses which are already in our DHCP server. This will mean not needing to use the netsh command to populate DHCP, and will give us a little more control.
So I need a text file containing all the MAC's which are DHCP enabled. I need to use the first MAC in the file, then delete it from the file (so it doesnt get used again). Powershell to the rescue yet again....and I can even use this within my VM creation script to streamline the whole process.
In the following example the input text file is called c:\test.txt and contains a basic list.
-----------------------------------------------------
#load the file into a string called file
$file=get-content "C:\test.txt"
#$file[0] represents the first line of data......do with as your wish
delete the data which was in the first line
$file[0]=""
#output it to a tempfile
$file out-file "C:\tempfile.txt"
#delete the original file
Remove-Item c:\test.txt
#delete any empty lines from the temp file, and save it as the original name
cat c:\tempfile.txt where {$_ -notmatch "^$" } > c:\test.txt
-----------------------------------------------------
So I need a text file containing all the MAC's which are DHCP enabled. I need to use the first MAC in the file, then delete it from the file (so it doesnt get used again). Powershell to the rescue yet again....and I can even use this within my VM creation script to streamline the whole process.
In the following example the input text file is called c:\test.txt and contains a basic list.
-----------------------------------------------------
#load the file into a string called file
$file=get-content "C:\test.txt"
#$file[0] represents the first line of data......do with as your wish
delete the data which was in the first line
$file[0]=""
#output it to a tempfile
$file out-file "C:\tempfile.txt"
#delete the original file
Remove-Item c:\test.txt
#delete any empty lines from the temp file, and save it as the original name
cat c:\tempfile.txt where {$_ -notmatch "^$" } > c:\test.txt
-----------------------------------------------------
Tuesday
I've been to a few Microsoft conferences where people contantly bang on about PowerShell. Until now its been something I have tried to avoid, as it looks a little bit too much like DOS for my liking!
However, working on another project has lead me to look into powershell for creating Virtual Machines, and I've been blown away by what it can do.
The beauty of MS System Centre Products is that they all execute tasks using powershell, even if you use an admin console to create the task. You could quite happily never know that powershell was being used, and I imagine 90% of users never need to know. The brilliant thing is tha whenever you do something with the GUI, be it create a user in Exchange 2007 or deploy a Virtual Machine, a powershell script is generated, which you can use, reuse and tweek for your own requirments.
In my case I used it as a foundation for provisioning new VMs in Virtual Machine Manager 2007. Having created a VM Template (by sys-preping an XP install), I've been able to write a script which using one command can create a new VM, name it, install all our core software, generate a mac address & network card and join the domain!
I've spent the last 2 days trying to come up with a random MAC address generator in powershell, which I'm pasting below. It works a treat, and I'll be posting the full script (including the VM provisioning script) over the next couple of weeks. This random MAC generator is extreamly useful when wanting to give users the ability to create VMs on the fly, and after a lot of digging online I wasnt able to find anything like this...so hopefuly someone else can use it;
----------------------------------------------------------
param(
[int] $len = 12,
[string] $chars = "0123456789abcdef"
)
$bytes = new-object "System.Byte[]" $len
$rnd = new-object System.Security.Cryptography.RNGCryptoServiceProvider
$rnd.GetBytes($bytes)
#define the fields
$macraw = ""
for( $i=0; $i -lt $len; $i++ )
{
$macraw += $chars[ $bytes[$i] % $chars.Length ]
}
#add collons to the random macraw so that it is properly formatted
$macaddress = $macraw[0]+$macraw[1]+":"+$macraw[2]+$macraw[3]+":"+$macraw[4]+$macraw[5]+":"+$macraw[6]+$macraw[7]+":"+$macraw[8]+$macraw[9]+":"+$macraw[10]+$macraw[11]
$macaddress
----------------------------------------------------------
However, working on another project has lead me to look into powershell for creating Virtual Machines, and I've been blown away by what it can do.
The beauty of MS System Centre Products is that they all execute tasks using powershell, even if you use an admin console to create the task. You could quite happily never know that powershell was being used, and I imagine 90% of users never need to know. The brilliant thing is tha whenever you do something with the GUI, be it create a user in Exchange 2007 or deploy a Virtual Machine, a powershell script is generated, which you can use, reuse and tweek for your own requirments.
In my case I used it as a foundation for provisioning new VMs in Virtual Machine Manager 2007. Having created a VM Template (by sys-preping an XP install), I've been able to write a script which using one command can create a new VM, name it, install all our core software, generate a mac address & network card and join the domain!
I've spent the last 2 days trying to come up with a random MAC address generator in powershell, which I'm pasting below. It works a treat, and I'll be posting the full script (including the VM provisioning script) over the next couple of weeks. This random MAC generator is extreamly useful when wanting to give users the ability to create VMs on the fly, and after a lot of digging online I wasnt able to find anything like this...so hopefuly someone else can use it;
----------------------------------------------------------
param(
[int] $len = 12,
[string] $chars = "0123456789abcdef"
)
$bytes = new-object "System.Byte[]" $len
$rnd = new-object System.Security.Cryptography.RNGCryptoServiceProvider
$rnd.GetBytes($bytes)
#define the fields
$macraw = ""
for( $i=0; $i -lt $len; $i++ )
{
$macraw += $chars[ $bytes[$i] % $chars.Length ]
}
#add collons to the random macraw so that it is properly formatted
$macaddress = $macraw[0]+$macraw[1]+":"+$macraw[2]+$macraw[3]+":"+$macraw[4]+$macraw[5]+":"+$macraw[6]+$macraw[7]+":"+$macraw[8]+$macraw[9]+":"+$macraw[10]+$macraw[11]
$macaddress
----------------------------------------------------------
Monday
For the past week I’ve been looking at our new MD1000e blade enclosure. With the push towards virtualization and redundancy, blade architecture is obviously a major leap forward.... 2xBlade enclosures, clustered blade servers & SAN = 100% uptime..... in theory.
So far I’ve just been configuring the MD1000e chassis. For those who don’t know the chassis is the rack mountable unit which holds all the blade servers. It contains the PSUs, the fans and the network ports, which means your blades don’t contain any of the above and can be smaller in size. The MD1000e takes up 10u in a Dell cabinet, and can hold 16 servers. With standard poweredge servers we would like to get around 2 or 3 servers in a 10 u space.
The MD1000e has a myriad of features accessible from the web interface. My tip of the day is that when you first switch the MD1000e on and it asks “do you want to complete the configuration wizard now?”, DONT answer “no”. If you answer “no” the only way you can give the management port (CMC) and IP address is by hooking the unit up to a laptop and using a null modem lead.
Once configured, you can view the health of all your blades, start them up, shut them down etc etc. One beautiful feature is that you can assign IP addresses to the management port of any blades which might be plugged in. So you give the MD1000e 16 addresses to play with, and each time you plug a blade in, and IP will automatically be given to the blade’s iDRAC port. This is extremely important if you don’t have a kvm built into your MD1000e, or you don’t want to sit in front of 32 fans roughly equalling the power of a military class jet engine with a mouse and keyboard on your lap.
Using the iDRAC of each blade, you can connect to the server and install an OS as if you were sat in front of it.... actually you wouldn’t even be able to do it sat in front of it come to think of it. Using this you can load up an ISO or share your local drive to install 2008 or whatever OS you desire.
Getting the unit up and running initially isn’t a 2 minute job, but it’s worth the effort, and extremely impressive bit of kit!
Our blades will initially be getting server 2008 data centre edition installed, with the hyper-v role in clustered mode.
Wednesday
Whilst Microsoft seem to have published a lot of corporate vision statements about their VDI (Virtual Desktop Infrastructure) vision, they don’t actually seem to provide much in the way of technical roadmaps. As I am currently looking to deploy a full VDI, this is something I have been looking into closely, and here are some of my conclusions.
What we are looking to achieve; is a full desktop experience from a remote location. Ideally, this should be via a web browser, on any remote PC. The application for this is to allow staff to work from home or whilst on the move.
Currently a member of staff sits at their desk, and logs into an XP machine, and is presented with the pre-defined desktop, along with their redirected 'My Documents', their home area, and any crucial settings which we might choose to deploy. Of course they also have the core set of software such as Office, and MIS applications. What we want is an identical experience (or as near as possible) from a remote PC.
Microsoft's "vision" for this identifies several of their key technologies for this. Virtual Machine Manager, the System Centre Suite and Server 2008 Terminal Services. What they don’t yet seem to have is a way to put all this together to produce a working solution.
So what are the possible ways forward?
• Create a virtual PC for each member of staff who requires working from home, set the PC identical to the way any other staff PC is configured, and simply allow staff to Remote Desktop into the PC using ‘Remote Desktop. The PROS of this solution are that it’s simple to configure, and pretty much fool proof for the user. The cons are that it requires potentially a vast number of VMs, and systems management
• Allow staff to Remote Desktop into their own physical PC. PROS even easier to setup. CONS requires workstations to be left powered on, 110% chance of it not being workable!
• Use scripted HyperV, via the Virtual Machine Manager web portal, and powershell, to create VMs on the fly to start them up & shut them down according to usage. PROS sleek and sexy. CONS very complex to setup, potential for huge number of VMs
• Use Virtual Machine Manager / Hyper-v with Citrix. PROS Seems to have a reasonable amount of backing from MS partners. CONS Expensive, expensive & expensive!
We may have to revisit some of the above, but currently the leader is none of the above....it’s a product called MED-V which is currently in BETA. It’s another product “acquired” by M$ in a similar way to softgrid, so I can only assume it will become part of the System Centre Suite very soon.
The way it works (in principal) is that you create a Virtual PC (using Virtual PC 2007), and configure it as required. You then upload this to the Med-V server, which in turn clones it for clients over the net when the log onto the MedV server. This gives the user what appears as a virtual PC running on their desktop.... when running in full screen mode you are essentially sat in the office.
Obviously the setup is a little more complex as IP addressing, DNS and Active Directory all come into play to prevent conflicts, but the MED-V product does work... flawlessly so far once setup.
There are cons however. It requires a small client on the users machine, and the first time the user connects it takes an index of the users local PC. This is to reduce the amount of traffic which MEDV uses during connection. i.e. if you have a file called wibble.dll on your local pc, it won’t bother dragging it over your 500k connection every time a MEDV session requires it. This initial index takes a good 30 minutes on average. After that the user can expect to connect in the same time it takes to boot up a virtual PC.
Support for MEDV is currently non-existent, and you can expect to do a lot of self research if you are going to evaluate it. The setup process is complex and requires 2 server and at least 2 workstations, plus a virtual PC. I’m looking forward to an official release date, but with a bit of luck e may deploy the BETA to test users within a couple of months.
What we are looking to achieve; is a full desktop experience from a remote location. Ideally, this should be via a web browser, on any remote PC. The application for this is to allow staff to work from home or whilst on the move.
Currently a member of staff sits at their desk, and logs into an XP machine, and is presented with the pre-defined desktop, along with their redirected 'My Documents', their home area, and any crucial settings which we might choose to deploy. Of course they also have the core set of software such as Office, and MIS applications. What we want is an identical experience (or as near as possible) from a remote PC.
Microsoft's "vision" for this identifies several of their key technologies for this. Virtual Machine Manager, the System Centre Suite and Server 2008 Terminal Services. What they don’t yet seem to have is a way to put all this together to produce a working solution.
So what are the possible ways forward?
• Create a virtual PC for each member of staff who requires working from home, set the PC identical to the way any other staff PC is configured, and simply allow staff to Remote Desktop into the PC using ‘Remote Desktop. The PROS of this solution are that it’s simple to configure, and pretty much fool proof for the user. The cons are that it requires potentially a vast number of VMs, and systems management
• Allow staff to Remote Desktop into their own physical PC. PROS even easier to setup. CONS requires workstations to be left powered on, 110% chance of it not being workable!
• Use scripted HyperV, via the Virtual Machine Manager web portal, and powershell, to create VMs on the fly to start them up & shut them down according to usage. PROS sleek and sexy. CONS very complex to setup, potential for huge number of VMs
• Use Virtual Machine Manager / Hyper-v with Citrix. PROS Seems to have a reasonable amount of backing from MS partners. CONS Expensive, expensive & expensive!
We may have to revisit some of the above, but currently the leader is none of the above....it’s a product called MED-V which is currently in BETA. It’s another product “acquired” by M$ in a similar way to softgrid, so I can only assume it will become part of the System Centre Suite very soon.
The way it works (in principal) is that you create a Virtual PC (using Virtual PC 2007), and configure it as required. You then upload this to the Med-V server, which in turn clones it for clients over the net when the log onto the MedV server. This gives the user what appears as a virtual PC running on their desktop.... when running in full screen mode you are essentially sat in the office.
Obviously the setup is a little more complex as IP addressing, DNS and Active Directory all come into play to prevent conflicts, but the MED-V product does work... flawlessly so far once setup.
There are cons however. It requires a small client on the users machine, and the first time the user connects it takes an index of the users local PC. This is to reduce the amount of traffic which MEDV uses during connection. i.e. if you have a file called wibble.dll on your local pc, it won’t bother dragging it over your 500k connection every time a MEDV session requires it. This initial index takes a good 30 minutes on average. After that the user can expect to connect in the same time it takes to boot up a virtual PC.
Support for MEDV is currently non-existent, and you can expect to do a lot of self research if you are going to evaluate it. The setup process is complex and requires 2 server and at least 2 workstations, plus a virtual PC. I’m looking forward to an official release date, but with a bit of luck e may deploy the BETA to test users within a couple of months.
It's been 6 years since we were last hit by a major virus, but we have just been hit again (along with a few million others) by W32.Downadup.B
What did it do?: It removed ability of users to log-in to the domain. The following entry appeared in droves on domain controller; "The SAM database was unable to lockout the account of Administrator due to a resource error, such as a hard disk write failure (the specific error code is in the error data) . Accounts are locked after a certain number of bad passwords are provided so please consider resetting the password of the account mentioned above." with an event ID of 12294. At the peak we were probably experiencing around 100 per second.
The end result of the above was that due to the sheer number of requests, active directory was unable to authenticate genuine users....who were getting their account locked instead of getting logged in. Ironically this wasnt the main intension of the virus, just a side effect. In actual fact the virus looks to upload any passwords it finds to a remote server. In order to avoid detection the writers made it upload to a few thousand random domain names to avoid detection by the authorities.
Some more info;
http://www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.B
How did we get rid of it?: Firstly we had to disable every single account on the domain with local or domain/enterprise security rights. You can imagine the hassle this is going to cause in terms of service run-as issues. We also had to unplug any administrative workstations from the network and run all fixes direct from the servers.
Micrsoft have provided a tool for prevention, also provided in the form of a windows update (should you happen to do those on a weekly basis);
http://support.microsoft.com/kb/891716
They do however suggest that deployment via Group Policy is not suitable for servers, and kindly provide a handy 36!! step manual process. You can however reduce this to around 5 steps by using Symantec Antivirus, a bit of quick registry editing, and the removal tools.....probably 15 minutes per server. I'd also suggest regular password changing of any admin account you are using, just to ensure that it isnt grabbed and used by an infection.
The only method we have found thus far to remove the threat from an infected PC is to perform a full system scan using Symantec AV. Not great for the users given the load it puts on the workstations, but at least its a fix!
I'm sure this wont be the last such threat, and its been our first since moving to Active Directory.... but I'll be looking into how we can deploy windows updates using SCCM2007 on a regular basis!
Thursday
Subscribe to:
Posts (Atom)
