Open Simulator Server on Windows vs Linux

[Maxwell, Douglas CIV USARMY ARL (US)]

The MOSES project ( http://militarymetaverse.org/ ) has experimented with a number of different Open Simulator deployment methods. The grid is currently has access to a blend of 5 Dell PowerEdge R815 servers and one Dell PowerEdge R220. The R815's are virtual machine servers and the R220 is a small Linux host used to serve up to 10 sims. On the R815's we can stand up any operating system we wish to experiment with using virtual machines.



Currently, our favorite installation method is using Ubuntu Linux. This is only for convenience as all of the dependencies are available as packages. On Ubuntu, we can start with a bare OS install through completely functioning MOSES grid in about 3 hours, including compiling from source. Less time when dealing with binaries.



Our Complete Instructions Here:

http://militarymetaverse.org/wp-content/uploads/2014/10/MIAB_Installation_Instructions_using_Virtual_Box_and_Ubuntu-141006.pdf



We have worked with CentOS, but it is a lot of work. Mostly because of the bootstrapping needed to get Mono and its associated packages working. It is easy to make mistakes and you can spend hours troubleshooting. In the end, it seems to operate similarly as Ubuntu.



The version of Windows we typically work with is the Army Golden Master of Windows 2008 Server or Windows 7 AGM. The biggest issue we deal with is just setting up the environment and being forced to manually download and install all the dependencies. This can take days. The Army version of Windows also has extra registry values and network policies that pose configuration challenges.



Virtual Machine Performance Differences?



In our testing we have not *yet* found any measurable difference in performance between Open Simulator deployed via VM or bare host on any operating system. This has been independently verified by our industry partners and other military labs using their own internal hosts as well as Amazon EC2.



Operating System Performance Differences?



Unfortunately there are so many variables that affect open simulator performance that anecdotal evidence to support Windows or Linux simply don't hold up. Its not a simple matter of comparing .Net to Mono. The operating systems are fundamentally different, so a proper test would involve replicating the Open Simulator servers as closely as possible on each. This would require an initial benchmark test that has both Windows and Linux servers using the same version of Mono, same version of Apache, same version of MySQL, etc... on the same host configuration (i.e. hardware or VM profile).



Any deviations introduce variables to the test that must be accounted for. For example, only after the Windows vs. Linux test has been completed and analyzed for differences can you then introduce another variable of .Net vs. Mono and re-run the performance benchmarks. The original Windows vs. Linux benchmark is the baseline.



Douglas Maxwell, MSME
Science and Technology Manager
Virtual World Strategic Applications
U.S. Army Research Lab
Simulation & Training Technology Center (STTC)
(c) [phone]<tel:%28407%29%20242-0209>

[James Hughes]

On Sat, 2014-10-11 at 15:46 +0000, Maxwell, Douglas CIV USARMY ARL (US)
wrote:

> Virtual Machine Performance Differences?
>
>
>
> In our testing we have not *yet* found any measurable difference in
> performance between Open Simulator deployed via VM or bare host on any
> operating system. This has been independently verified by our
> industry partners and other military labs using their own internal
> hosts as well as Amazon EC2.
>

Thanks for the information about the interesting work you are doing
there. This is interesting about using virtual machines. Do you deploy
any of these in VirtualBox or similar on your hosts?

Thanks,
BlueWall

[Dahlia Trimble]

I have noticed decreased performance when running OpenSimulator in a
VirtualBox VM vs. running it on the host. Usually it's when there is a lot
of I/O operations, such as high database activity or networking load. I've
also seen issues when a lot of timers are used in scripts. This makes sense
as VIrtualBox adds significant overhead to such operations.

I've seen much better performance using OpenVZ, however OpenVZ is harder to
work with as it's not really a virtual machine but rather more of a
constrained operating environment. It does not add as much overhead to I/O
operations as a true virtual machine would. It also did not seem to suffer
from the timer issues I've seen in VirtualBox.

I've used OpenSImulator on a variety of virtual servers now for several
years and in general it does work reasonably well for smaller use cases. I
would not consider using anything but bare metal or OpenVZ for any regions
which were expected to have high performance and service many users
simultaneously.

In general I've found Windows+.NET to be much more robust than Mono. I do
see problems occasionally on Linux+Mono but I tend to blame Mono. I have
experimental code which can crash and burn any Mono version I've tried to
date, yet runs flawlessly on .NET. This code creates and dereferences
millions of very small objects and when run on a Mono installation will
leak memory and crash with out of memory errors. I've discussed it with
several Mono devs on IRC and the consensus was a possible bug in one or
more of the Mono runtime libraries. I've also seen a lot of random Mono
crashes that I just never see in .NET while running the same sim under
similar load conditions.

While in general I believe .NET to be a more robust operating environment
than mono, the lower cost and ease of use of Linux for server-like
applications usually means I use Linux servers for running regions. I
usually use Windows exclusively for OpenSimulator development, which may be
a reason why I tend to notice such differences. I don't really have any
documented empirical evidence to support my reasoning other than having
some code which will crash Mono. Most of my rationale is based on
observations during my OpenSimulator development and deployment activities
over the past several years.

On Sat, Oct 11, 2014 at 8:46 AM, Maxwell, Douglas CIV USARMY ARL (US) <
***@mail.mil> wrote:

> The MOSES project ( http://militarymetaverse.org/ ) has experimented
> with a number of different Open Simulator deployment methods. The grid is
> currently has access to a blend of 5 Dell PowerEdge R815 servers and one
> Dell PowerEdge R220. The R815's are virtual machine servers and the R220
> is a small Linux host used to serve up to 10 sims. On the R815's we can
> stand up any operating system we wish to experiment with using virtual
> machines.
>
>
>
> Currently, our favorite installation method is using Ubuntu Linux. This
> is only for convenience as all of the dependencies are available as
> packages. On Ubuntu, we can start with a bare OS install through
> completely functioning MOSES grid in about 3 hours, including compiling
> from source. Less time when dealing with binaries.
>
>
>
> Our Complete Instructions Here:
>
>
> http://militarymetaverse.org/wp-content/uploads/2014/10/MIAB_Installation_Instructions_using_Virtual_Box_and_Ubuntu-141006.pdf
>
>
>
> We have worked with CentOS, but it is a lot of work. Mostly because of
> the bootstrapping needed to get Mono and its associated packages working.
> It is easy to make mistakes and you can spend hours troubleshooting. In
> the end, it seems to operate similarly as Ubuntu.
>
>
>
> The version of Windows we typically work with is the Army Golden Master of
> Windows 2008 Server or Windows 7 AGM. The biggest issue we deal with is
> just setting up the environment and being forced to manually download and
> install all the dependencies. This can take days. The Army version of
> Windows also has extra registry values and network policies that pose
> configuration challenges.
>
>
>
> Virtual Machine Performance Differences?
>
>
>
> In our testing we have not *yet* found any measurable difference in
> performance between Open Simulator deployed via VM or bare host on any
> operating system. This has been independently verified by our industry
> partners and other military labs using their own internal hosts as well as
> Amazon EC2.
>
>
>
> Operating System Performance Differences?
>
>
>
> Unfortunately there are so many variables that affect open simulator
> performance that anecdotal evidence to support Windows or Linux simply
> don't hold up. Its not a simple matter of comparing .Net to Mono. The
> operating systems are fundamentally different, so a proper test would
> involve replicating the Open Simulator servers as closely as possible on
> each. This would require an initial benchmark test that has both Windows
> and Linux servers using the same version of Mono, same version of Apache,
> same version of MySQL, etc... on the same host configuration (i.e.
> hardware or VM profile).
>
>
>
> Any deviations introduce variables to the test that must be accounted
> for. For example, only after the Windows vs. Linux test has been completed
> and analyzed for differences can you then introduce another variable of
> .Net vs. Mono and re-run the performance benchmarks. The original Windows
> vs. Linux benchmark is the baseline.
>
>
> Douglas Maxwell, MSME
> Science and Technology Manager
> Virtual World Strategic Applications
> U.S. Army Research Lab
> Simulation & Training Technology Center (STTC)
> (c) [phone] <%28407%29%20242-0209>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-***@opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>

[Butch Arnold]

Hi All,

I haven't compared Linux vs. Windows in some time, but in the past when
comparing, a linux based instance seemed to require more ram than the
same region running on a windows machine.
We at 3rd Rock use only windows for our grid and centOS for our web server.
It is true that I personally am much more comfortable using windows than
I am Linux, but I am not against Linux.
In our early days back in '08 I had originally started 3rd Rock using
Linux, but changed over to Windows as I am more comfortable with that
platform.

This topic has all the dangers of quickly turning into a Ford vs. Chevy,
or coke vs. pepsi argument, so to be fair, some may prefer linux over
windows for different reasons, but the reason 3rd Rock runs windows is
that my personal observations and tests revealed to me that opensim on
linux uses more ram "And" I am more comfortable with the windows platform.

~Butch






On 10/11/2014 4:06 PM, Dahlia Trimble wrote:
> I have noticed decreased performance when running OpenSimulator in a
> VirtualBox VM vs. running it on the host. Usually it's when there is a
> lot of I/O operations, such as high database activity or networking
> load. I've also seen issues when a lot of timers are used in scripts.
> This makes sense as VIrtualBox adds significant overhead to such
> operations.
>
> I've seen much better performance using OpenVZ, however OpenVZ is
> harder to work with as it's not really a virtual machine but rather
> more of a constrained operating environment. It does not add as much
> overhead to I/O operations as a true virtual machine would. It also
> did not seem to suffer from the timer issues I've seen in VirtualBox.
>
> I've used OpenSImulator on a variety of virtual servers now for
> several years and in general it does work reasonably well for smaller
> use cases. I would not consider using anything but bare metal or
> OpenVZ for any regions which were expected to have high performance
> and service many users simultaneously.
>
> In general I've found Windows+.NET to be much more robust than Mono. I
> do see problems occasionally on Linux+Mono but I tend to blame Mono. I
> have experimental code which can crash and burn any Mono version I've
> tried to date, yet runs flawlessly on .NET. This code creates and
> dereferences millions of very small objects and when run on a Mono
> installation will leak memory and crash with out of memory errors.
> I've discussed it with several Mono devs on IRC and the consensus was
> a possible bug in one or more of the Mono runtime libraries. I've also
> seen a lot of random Mono crashes that I just never see in .NET while
> running the same sim under similar load conditions.
>
> While in general I believe .NET to be a more robust operating
> environment than mono, the lower cost and ease of use of Linux for
> server-like applications usually means I use Linux servers for running
> regions. I usually use Windows exclusively for OpenSimulator
> development, which may be a reason why I tend to notice such
> differences. I don't really have any documented empirical evidence to
> support my reasoning other than having some code which will crash
> Mono. Most of my rationale is based on observations during my
> OpenSimulator development and deployment activities over the past
> several years.
>
> On Sat, Oct 11, 2014 at 8:46 AM, Maxwell, Douglas CIV USARMY ARL (US)
> <***@mail.mil <mailto:***@mail.mil>>
> wrote:
>
> The MOSES project ( http://militarymetaverse.org/ ) has
> experimented with a number of different Open Simulator deployment
> methods. The grid is currently has access to a blend of 5 Dell
> PowerEdge R815 servers and one Dell PowerEdge R220. The R815's
> are virtual machine servers and the R220 is a small Linux host
> used to serve up to 10 sims. On the R815's we can stand up any
> operating system we wish to experiment with using virtual machines.
>
> Currently, our favorite installation method is using Ubuntu
> Linux. This is only for convenience as all of the dependencies
> are available as packages. On Ubuntu, we can start with a bare OS
> install through completely functioning MOSES grid in about 3
> hours, including compiling from source. Less time when dealing
> with binaries.
>
> Our Complete Instructions Here:
>
> http://militarymetaverse.org/wp-content/uploads/2014/10/MIAB_Installation_Instructions_using_Virtual_Box_and_Ubuntu-141006.pdf
>
> We have worked with CentOS, but it is a lot of work. Mostly
> because of the bootstrapping needed to get Mono and its associated
> packages working. It is easy to make mistakes and you can spend
> hours troubleshooting. In the end, it seems to operate similarly
> as Ubuntu.
>
> The version of Windows we typically work with is the Army Golden
> Master of Windows 2008 Server or Windows 7 AGM. The biggest issue
> we deal with is just setting up the environment and being forced
> to manually download and install all the dependencies. This can
> take days. The Army version of Windows also has extra registry
> values and network policies that pose configuration challenges.
>
> Virtual Machine Performance Differences?
>
> In our testing we have not *yet* found any measurable difference
> in performance between Open Simulator deployed via VM or bare host
> on any operating system. This has been independently verified by
> our industry partners and other military labs using their own
> internal hosts as well as Amazon EC2.
>
> Operating System Performance Differences?
>
> Unfortunately there are so many variables that affect open
> simulator performance that anecdotal evidence to support Windows
> or Linux simply don't hold up. Its not a simple matter of
> comparing .Net to Mono. The operating systems are fundamentally
> different, so a proper test would involve replicating the Open
> Simulator servers as closely as possible on each. This would
> require an initial benchmark test that has both Windows and Linux
> servers using the same version of Mono, same version of Apache,
> same version of MySQL, etc... on the same host configuration
> (i.e. hardware or VM profile).
>
> Any deviations introduce variables to the test that must be
> accounted for. For example, only after the Windows vs. Linux test
> has been completed and analyzed for differences can you then
> introduce another variable of .Net vs. Mono and re-run the
> performance benchmarks. The original Windows vs. Linux benchmark
> is the baseline.
>
> Douglas Maxwell, MSME
> Science and Technology Manager
> Virtual World Strategic Applications
> U.S. Army Research Lab
> Simulation & Training Technology Center (STTC)
> (c)[phone] <tel:%28407%29%20242-0209>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-***@opensimulator.org <mailto:Opensim-***@opensimulator.org>
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-***@opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev

[Amit Goel]

Dear All,

Performance of any product is of prime importance, since it results in user actually using or not-using the system.

My question to all is how do you collect and evaluate performance metrics and measures, specially when you are evaluating virtual vs. real hosts and linux vs. mac vs. windows ?

Is it just visual perception ? What about network configurations ? Can you please post exact specs of the system, network topology, and raw/processed performance metrics data ? Is it subjective or objective evaluation ?

I did post something similar on this list few weeks back, and Justin and Dan were kind enough to provide some links, however, those links lead me to believe that the performance testing was only restricted to few metrics such as CPU load, memory load and not much raw/processed data was available other then couple of tables listing some configuration and values.

Look forward to hear from you...

regards

-- amit

________________________________
From: opensim-dev-***@opensimulator.org [opensim-dev-***@opensimulator.org] on behalf of Butch Arnold [***@3rdrockgrid.com]
Sent: Saturday, October 11, 2014 5:43 PM
To: opensim-***@opensimulator.org
Subject: Re: [Opensim-dev] Open Simulator Server on Windows vs Linux

Hi All,

I haven't compared Linux vs. Windows in some time, but in the past when comparing, a linux based instance seemed to require more ram than the same region running on a windows machine.
We at 3rd Rock use only windows for our grid and centOS for our web server.
It is true that I personally am much more comfortable using windows than I am Linux, but I am not against Linux.
In our early days back in '08 I had originally started 3rd Rock using Linux, but changed over to Windows as I am more comfortable with that platform.

This topic has all the dangers of quickly turning into a Ford vs. Chevy, or coke vs. pepsi argument, so to be fair, some may prefer linux over windows for different reasons, but the reason 3rd Rock runs windows is that my personal observations and tests revealed to me that opensim on linux uses more ram "And" I am more comfortable with the windows platform.

~Butch






On 10/11/2014 4:06 PM, Dahlia Trimble wrote:
I have noticed decreased performance when running OpenSimulator in a VirtualBox VM vs. running it on the host. Usually it's when there is a lot of I/O operations, such as high database activity or networking load. I've also seen issues when a lot of timers are used in scripts. This makes sense as VIrtualBox adds significant overhead to such operations.

I've seen much better performance using OpenVZ, however OpenVZ is harder to work with as it's not really a virtual machine but rather more of a constrained operating environment. It does not add as much overhead to I/O operations as a true virtual machine would. It also did not seem to suffer from the timer issues I've seen in VirtualBox.

I've used OpenSImulator on a variety of virtual servers now for several years and in general it does work reasonably well for smaller use cases. I would not consider using anything but bare metal or OpenVZ for any regions which were expected to have high performance and service many users simultaneously.

In general I've found Windows+.NET to be much more robust than Mono. I do see problems occasionally on Linux+Mono but I tend to blame Mono. I have experimental code which can crash and burn any Mono version I've tried to date, yet runs flawlessly on .NET. This code creates and dereferences millions of very small objects and when run on a Mono installation will leak memory and crash with out of memory errors. I've discussed it with several Mono devs on IRC and the consensus was a possible bug in one or more of the Mono runtime libraries. I've also seen a lot of random Mono crashes that I just never see in .NET while running the same sim under similar load conditions.

While in general I believe .NET to be a more robust operating environment than mono, the lower cost and ease of use of Linux for server-like applications usually means I use Linux servers for running regions. I usually use Windows exclusively for OpenSimulator development, which may be a reason why I tend to notice such differences. I don't really have any documented empirical evidence to support my reasoning other than having some code which will crash Mono. Most of my rationale is based on observations during my OpenSimulator development and deployment activities over the past several years.

On Sat, Oct 11, 2014 at 8:46 AM, Maxwell, Douglas CIV USARMY ARL (US) <***@mail.mil<mailto:***@mail.mil>> wrote:

The MOSES project ( http://militarymetaverse.org/ ) has experimented with a number of different Open Simulator deployment methods. The grid is currently has access to a blend of 5 Dell PowerEdge R815 servers and one Dell PowerEdge R220. The R815's are virtual machine servers and the R220 is a small Linux host used to serve up to 10 sims. On the R815's we can stand up any operating system we wish to experiment with using virtual machines.



Currently, our favorite installation method is using Ubuntu Linux. This is only for convenience as all of the dependencies are available as packages. On Ubuntu, we can start with a bare OS install through completely functioning MOSES grid in about 3 hours, including compiling from source. Less time when dealing with binaries.



Our Complete Instructions Here:

http://militarymetaverse.org/wp-content/uploads/2014/10/MIAB_Installation_Instructions_using_Virtual_Box_and_Ubuntu-141006.pdf



We have worked with CentOS, but it is a lot of work. Mostly because of the bootstrapping needed to get Mono and its associated packages working. It is easy to make mistakes and you can spend hours troubleshooting. In the end, it seems to operate similarly as Ubuntu.



The version of Windows we typically work with is the Army Golden Master of Windows 2008 Server or Windows 7 AGM. The biggest issue we deal with is just setting up the environment and being forced to manually download and install all the dependencies. This can take days. The Army version of Windows also has extra registry values and network policies that pose configuration challenges.



Virtual Machine Performance Differences?



In our testing we have not *yet* found any measurable difference in performance between Open Simulator deployed via VM or bare host on any operating system. This has been independently verified by our industry partners and other military labs using their own internal hosts as well as Amazon EC2.



Operating System Performance Differences?



Unfortunately there are so many variables that affect open simulator performance that anecdotal evidence to support Windows or Linux simply don't hold up. Its not a simple matter of comparing .Net to Mono. The operating systems are fundamentally different, so a proper test would involve replicating the Open Simulator servers as closely as possible on each. This would require an initial benchmark test that has both Windows and Linux servers using the same version of Mono, same version of Apache, same version of MySQL, etc... on the same host configuration (i.e. hardware or VM profile).



Any deviations introduce variables to the test that must be accounted for. For example, only after the Windows vs. Linux test has been completed and analyzed for differences can you then introduce another variable of .Net vs. Mono and re-run the performance benchmarks. The original Windows vs. Linux benchmark is the baseline.



Douglas Maxwell, MSME
Science and Technology Manager
Virtual World Strategic Applications
U.S. Army Research Lab
Simulation & Training Technology Center (STTC)
(c) [phone]<tel:%28407%29%20242-0209>

[Justin Clark-Casey]

Actually, there's a lot of other data that OpenSimulator is recording. Did you see the "show stats" page? [1]. If you
type the "show stats all" console command on any server you'll see a whole bunch of data that can be inspected and
recorded periodically to file.

That page also has some explanation of a selection of the stats. If you want to know about specific ones that aren't
listed then please ask me on this list. I haven't been able to write them all up because of a large number of other
high-priority tasks taking up my time but I'll be happy to document individual ones as required. This facility is
currently undergoing evolution forced by the conference requirement for a high number of simultaneous grid users.

That said, I expect most people do not use this data, partly because it is comparatively recent, partly because it is
complicated to decipher and much of it isn't useful unless you're actually working on OpenSimulator in high server
stress situations.

This kind of monitoring is cutting edge for OpenSimulator (and any open-source virtual world implementation I expect).
You will not find a lot of studies of this stuff out there.

[1] http://opensimulator.org/wiki/Show_stats

On 14/10/14 13:10, Amit Goel wrote:
> Dear All,
>
> Performance of any product is of prime importance, since it results in user actually using or not-using the system.
>
> My question to all is how do you collect and evaluate performance metrics and measures, specially when you are
> evaluating virtual vs. real hosts and linux vs. mac vs. windows ?
>
> Is it just visual perception ? What about network configurations ? Can you please post exact specs of the system,
> network topology, and raw/processed performance metrics data ? Is it subjective or objective evaluation ?
>
> I did post something similar on this list few weeks back, and Justin and Dan were kind enough to provide some links,
> however, those links lead me to believe that the performance testing was only restricted to few metrics such as CPU
> load, memory load and not much raw/processed data was available other then couple of tables listing some configuration
> and values.
>
> Look forward to hear from you...
>
> regards
>
> -- amit
>
> ------------------------------------------------------------------------------------------------------------------------
> *From:* opensim-dev-***@opensimulator.org [opensim-dev-***@opensimulator.org] on behalf of Butch Arnold
> [***@3rdrockgrid.com]
> *Sent:* Saturday, October 11, 2014 5:43 PM
> *To:* opensim-***@opensimulator.org
> *Subject:* Re: [Opensim-dev] Open Simulator Server on Windows vs Linux
>
> Hi All,
>
> I haven't compared Linux vs. Windows in some time, but in the past when comparing, a linux based instance seemed to
> require more ram than the same region running on a windows machine.
> We at 3rd Rock use only windows for our grid and centOS for our web server.
> It is true that I personally am much more comfortable using windows than I am Linux, but I am not against Linux.
> In our early days back in '08 I had originally started 3rd Rock using Linux, but changed over to Windows as I am more
> comfortable with that platform.
>
> This topic has all the dangers of quickly turning into a Ford vs. Chevy, or coke vs. pepsi argument, so to be fair, some
> may prefer linux over windows for different reasons, but the reason 3rd Rock runs windows is that my personal
> observations and tests revealed to me that opensim on linux uses more ram "And" I am more comfortable with the windows
> platform.
>
> ~Butch
>
>
>
>
>
>
> On 10/11/2014 4:06 PM, Dahlia Trimble wrote:
>> I have noticed decreased performance when running OpenSimulator in a VirtualBox VM vs. running it on the host. Usually
>> it's when there is a lot of I/O operations, such as high database activity or networking load. I've also seen issues
>> when a lot of timers are used in scripts. This makes sense as VIrtualBox adds significant overhead to such operations.
>>
>> I've seen much better performance using OpenVZ, however OpenVZ is harder to work with as it's not really a virtual
>> machine but rather more of a constrained operating environment. It does not add as much overhead to I/O operations as
>> a true virtual machine would. It also did not seem to suffer from the timer issues I've seen in VirtualBox.
>>
>> I've used OpenSImulator on a variety of virtual servers now for several years and in general it does work reasonably
>> well for smaller use cases. I would not consider using anything but bare metal or OpenVZ for any regions which were
>> expected to have high performance and service many users simultaneously.
>>
>> In general I've found Windows+.NET to be much more robust than Mono. I do see problems occasionally on Linux+Mono but
>> I tend to blame Mono. I have experimental code which can crash and burn any Mono version I've tried to date, yet runs
>> flawlessly on .NET. This code creates and dereferences millions of very small objects and when run on a Mono
>> installation will leak memory and crash with out of memory errors. I've discussed it with several Mono devs on IRC and
>> the consensus was a possible bug in one or more of the Mono runtime libraries. I've also seen a lot of random Mono
>> crashes that I just never see in .NET while running the same sim under similar load conditions.
>>
>> While in general I believe .NET to be a more robust operating environment than mono, the lower cost and ease of use of
>> Linux for server-like applications usually means I use Linux servers for running regions. I usually use Windows
>> exclusively for OpenSimulator development, which may be a reason why I tend to notice such differences. I don't really
>> have any documented empirical evidence to support my reasoning other than having some code which will crash Mono. Most
>> of my rationale is based on observations during my OpenSimulator development and deployment activities over the past
>> several years.
>>
>> On Sat, Oct 11, 2014 at 8:46 AM, Maxwell, Douglas CIV USARMY ARL (US) <***@mail.mil
>> <mailto:***@mail.mil>> wrote:
>>
>> The MOSES project ( http://militarymetaverse.org/ ) has experimented with a number of different Open Simulator
>> deployment methods. The grid is currently has access to a blend of 5 Dell PowerEdge R815 servers and one Dell
>> PowerEdge R220. The R815's are virtual machine servers and the R220 is a small Linux host used to serve up to 10
>> sims. On the R815's we can stand up any operating system we wish to experiment with using virtual machines.
>>
>> Currently, our favorite installation method is using Ubuntu Linux. This is only for convenience as all of the
>> dependencies are available as packages. On Ubuntu, we can start with a bare OS install through completely
>> functioning MOSES grid in about 3 hours, including compiling from source. Less time when dealing with binaries.
>>
>> Our Complete Instructions Here:
>>
>> http://militarymetaverse.org/wp-content/uploads/2014/10/MIAB_Installation_Instructions_using_Virtual_Box_and_Ubuntu-141006.pdf
>>
>> We have worked with CentOS, but it is a lot of work. Mostly because of the bootstrapping needed to get Mono and
>> its associated packages working. It is easy to make mistakes and you can spend hours troubleshooting. In the
>> end, it seems to operate similarly as Ubuntu.
>>
>> The version of Windows we typically work with is the Army Golden Master of Windows 2008 Server or Windows 7 AGM.
>> The biggest issue we deal with is just setting up the environment and being forced to manually download and
>> install all the dependencies. This can take days. The Army version of Windows also has extra registry values and
>> network policies that pose configuration challenges.
>>
>> Virtual Machine Performance Differences?
>>
>> In our testing we have not *yet* found any measurable difference in performance between Open Simulator deployed
>> via VM or bare host on any operating system. This has been independently verified by our industry partners and
>> other military labs using their own internal hosts as well as Amazon EC2.
>>
>> Operating System Performance Differences?
>>
>> Unfortunately there are so many variables that affect open simulator performance that anecdotal evidence to
>> support Windows or Linux simply don't hold up. Its not a simple matter of comparing .Net to Mono. The operating
>> systems are fundamentally different, so a proper test would involve replicating the Open Simulator servers as
>> closely as possible on each. This would require an initial benchmark test that has both Windows and Linux servers
>> using the same version of Mono, same version of Apache, same version of MySQL, etc... on the same host
>> configuration (i.e. hardware or VM profile).
>>
>> Any deviations introduce variables to the test that must be accounted for. For example, only after the Windows
>> vs. Linux test has been completed and analyzed for differences can you then introduce another variable of .Net vs.
>> Mono and re-run the performance benchmarks. The original Windows vs. Linux benchmark is the baseline.
>>
>> Douglas Maxwell, MSME
>> Science and Technology Manager
>> Virtual World Strategic Applications
>> U.S. Army Research Lab
>> Simulation & Training Technology Center (STTC)
>> (c)[phone] <tel:%28407%29%20242-0209>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-***@opensimulator.org <mailto:Opensim-***@opensimulator.org>
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>
>>
>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-***@opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-***@opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>


--
Justin Clark-Casey (justincc)
OSVW Consulting
http://justincc.org
http://twitter.com/justincc

[Dahlia Trimble]

I haven't really done any rigorous performance evaluations myself, most of
what I know is qualitative based on trying many different configurations
during the time I've been involved in the OpenSimulator project. I have
reported on a few individual installations in the past which may be
archived on this list although they would be quite some time ago and
probably no longer relevant.

I believe there are several aspects to performance evaluation which might
be categorized as either user experience or technical attributes. Technical
performance attributes may or may not affect user experience. Some examples
of technical may be memory use, cpu use, script load, timer accuracy, #of
concurrent users, etc. Some examples of user attributes may be scene
loading time, number if concurrent users, action delays (i.e., time from
pressing a movement key until your avatar responds), etc. This is
obviously not a comprehensive list but I hope it illustrates the concept.
In my testing these are the kinds of things I've looked at.

Hardware and virtual machine performance varies widely and is always
evolving at a somewhat rapid pace. I'm not really sure how valuable a
rigorous, quantitative study would be, or how long it would remain
relevant. I do suspect such a study would spark interest in developer and
operator communities and probably uncover some potential areas for
improvement in OpenSimulator though.

On Tue, Oct 14, 2014 at 5:10 AM, Amit Goel <***@ucf.edu> wrote:

> Dear All,
>
> Performance of any product is of prime importance, since it results in
> user actually using or not-using the system.
>
> My question to all is how do you collect and evaluate performance
> metrics and measures, specially when you are evaluating virtual vs. real
> hosts and linux vs. mac vs. windows ?
>
> Is it just visual perception ? What about network configurations ? Can
> you please post exact specs of the system, network topology, and
> raw/processed performance metrics data ? Is it subjective or objective
> evaluation ?
>
> I did post something similar on this list few weeks back, and Justin and
> Dan were kind enough to provide some links, however, those links lead me to
> believe that the performance testing was only restricted to few metrics
> such as CPU load, memory load and not much raw/processed data was available
> other then couple of tables listing some configuration and values.
>
> Look forward to hear from you...
>
> regards
>
> -- amit
>
> ------------------------------
> *From:* opensim-dev-***@opensimulator.org [
> opensim-dev-***@opensimulator.org] on behalf of Butch Arnold [
> ***@3rdrockgrid.com]
> *Sent:* Saturday, October 11, 2014 5:43 PM
> *To:* opensim-***@opensimulator.org
> *Subject:* Re: [Opensim-dev] Open Simulator Server on Windows vs Linux
>
> Hi All,
>
> I haven't compared Linux vs. Windows in some time, but in the past when
> comparing, a linux based instance seemed to require more ram than the same
> region running on a windows machine.
> We at 3rd Rock use only windows for our grid and centOS for our web server.
> It is true that I personally am much more comfortable using windows than I
> am Linux, but I am not against Linux.
> In our early days back in '08 I had originally started 3rd Rock using
> Linux, but changed over to Windows as I am more comfortable with that
> platform.
>
> This topic has all the dangers of quickly turning into a Ford vs. Chevy,
> or coke vs. pepsi argument, so to be fair, some may prefer linux over
> windows for different reasons, but the reason 3rd Rock runs windows is that
> my personal observations and tests revealed to me that opensim on linux
> uses more ram "And" I am more comfortable with the windows platform.
>
> ~Butch
>
>
>
>
>
>
> On 10/11/2014 4:06 PM, Dahlia Trimble wrote:
>
> I have noticed decreased performance when running OpenSimulator in a
> VirtualBox VM vs. running it on the host. Usually it's when there is a lot
> of I/O operations, such as high database activity or networking load. I've
> also seen issues when a lot of timers are used in scripts. This makes sense
> as VIrtualBox adds significant overhead to such operations.
>
> I've seen much better performance using OpenVZ, however OpenVZ is harder
> to work with as it's not really a virtual machine but rather more of a
> constrained operating environment. It does not add as much overhead to I/O
> operations as a true virtual machine would. It also did not seem to suffer
> from the timer issues I've seen in VirtualBox.
>
> I've used OpenSImulator on a variety of virtual servers now for several
> years and in general it does work reasonably well for smaller use cases. I
> would not consider using anything but bare metal or OpenVZ for any regions
> which were expected to have high performance and service many users
> simultaneously.
>
> In general I've found Windows+.NET to be much more robust than Mono. I do
> see problems occasionally on Linux+Mono but I tend to blame Mono. I have
> experimental code which can crash and burn any Mono version I've tried to
> date, yet runs flawlessly on .NET. This code creates and dereferences
> millions of very small objects and when run on a Mono installation will
> leak memory and crash with out of memory errors. I've discussed it with
> several Mono devs on IRC and the consensus was a possible bug in one or
> more of the Mono runtime libraries. I've also seen a lot of random Mono
> crashes that I just never see in .NET while running the same sim under
> similar load conditions.
>
> While in general I believe .NET to be a more robust operating environment
> than mono, the lower cost and ease of use of Linux for server-like
> applications usually means I use Linux servers for running regions. I
> usually use Windows exclusively for OpenSimulator development, which may be
> a reason why I tend to notice such differences. I don't really have any
> documented empirical evidence to support my reasoning other than having
> some code which will crash Mono. Most of my rationale is based on
> observations during my OpenSimulator development and deployment activities
> over the past several years.
>
> On Sat, Oct 11, 2014 at 8:46 AM, Maxwell, Douglas CIV USARMY ARL (US) <
> ***@mail.mil> wrote:
>
>> The MOSES project ( http://militarymetaverse.org/ ) has experimented
>> with a number of different Open Simulator deployment methods. The grid is
>> currently has access to a blend of 5 Dell PowerEdge R815 servers and one
>> Dell PowerEdge R220. The R815's are virtual machine servers and the R220
>> is a small Linux host used to serve up to 10 sims. On the R815's we can
>> stand up any operating system we wish to experiment with using virtual
>> machines.
>>
>>
>>
>> Currently, our favorite installation method is using Ubuntu Linux. This
>> is only for convenience as all of the dependencies are available as
>> packages. On Ubuntu, we can start with a bare OS install through
>> completely functioning MOSES grid in about 3 hours, including compiling
>> from source. Less time when dealing with binaries.
>>
>>
>>
>> Our Complete Instructions Here:
>>
>>
>> http://militarymetaverse.org/wp-content/uploads/2014/10/MIAB_Installation_Instructions_using_Virtual_Box_and_Ubuntu-141006.pdf
>>
>>
>>
>> We have worked with CentOS, but it is a lot of work. Mostly because of
>> the bootstrapping needed to get Mono and its associated packages working.
>> It is easy to make mistakes and you can spend hours troubleshooting. In
>> the end, it seems to operate similarly as Ubuntu.
>>
>>
>>
>> The version of Windows we typically work with is the Army Golden Master
>> of Windows 2008 Server or Windows 7 AGM. The biggest issue we deal with is
>> just setting up the environment and being forced to manually download and
>> install all the dependencies. This can take days. The Army version of
>> Windows also has extra registry values and network policies that pose
>> configuration challenges.
>>
>>
>>
>> Virtual Machine Performance Differences?
>>
>>
>>
>> In our testing we have not *yet* found any measurable difference in
>> performance between Open Simulator deployed via VM or bare host on any
>> operating system. This has been independently verified by our industry
>> partners and other military labs using their own internal hosts as well as
>> Amazon EC2.
>>
>>
>>
>> Operating System Performance Differences?
>>
>>
>>
>> Unfortunately there are so many variables that affect open simulator
>> performance that anecdotal evidence to support Windows or Linux simply
>> don't hold up. Its not a simple matter of comparing .Net to Mono. The
>> operating systems are fundamentally different, so a proper test would
>> involve replicating the Open Simulator servers as closely as possible on
>> each. This would require an initial benchmark test that has both Windows
>> and Linux servers using the same version of Mono, same version of Apache,
>> same version of MySQL, etc... on the same host configuration (i.e.
>> hardware or VM profile).
>>
>>
>>
>> Any deviations introduce variables to the test that must be accounted
>> for. For example, only after the Windows vs. Linux test has been completed
>> and analyzed for differences can you then introduce another variable of
>> .Net vs. Mono and re-run the performance benchmarks. The original Windows
>> vs. Linux benchmark is the baseline.
>>
>>
>> Douglas Maxwell, MSME
>> Science and Technology Manager
>> Virtual World Strategic Applications
>> U.S. Army Research Lab
>> Simulation & Training Technology Center (STTC)
>> (c) [phone] <%28407%29%20242-0209>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-***@opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>
>>
>
>
> _______________________________________________
> Opensim-dev mailing listOpensim-***@opensimulator.orghttp://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-***@opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>