Benchmarks

Hey Nick,

I realize that with todays hardware and todays bandwidth, performance really isn't too critical on a mudclient (even the slowest client should have enough hardware behind it to handle a paragraph of text every 2 seconds).. But the benchmarks haven't been updated in 8 years.

Recently, a friend of mine asked me to check out Mudlet because he really wanted a native client on his mac, although he's happy with MC when he's not on battery power... Anyway, in their forums I've found comments like " MUSHclient also can't compare to mudlet's display or processing speed.." (http://forums.mudlet.org/viewtopic.php?f=9&t=1584&p=5433&hilit=mushclient#p5433) They also haven't actually published any benchmark numbers...

And, well, their client has some promising ideas in it (tho it performs a lot more like a version 0.7 than a 1.1), but a clean install of Mudlet on my machine is more sluggish than MC - with 9 years of plugins and triggers...

With out stepping into "fanboy" territory, it'd be nice if you still had your BSD box and updated the Benchmark... Let's have 'em put up or shut up.

That reminds me of a quote from Alice In Wonderland:



You notice Tsuujin said it "can't compare" not that it was faster or slower. ;)

I don't really want to get into benchmark-envy, and it is interesting that the implied claim that Mudlet is faster is not backed up by any factual claims. For example, if "processing speed" is claimed to be faster, processing what? Triggers? I am happy to test trigger matching against any other client. For one thing I am using the PCRE engine (itself very powerful). A quick browse seems to indicate Mudlet also uses PCRE, so the raw trigger matching should be comparable.

I remember a while back a client was claimed to be faster - indeed the claim was it "must be faster" because it was a console application, not a GUI application. However a test soon showed that this was false.

There is more to speed that being GUI or non-GUI, or written in C++ or some other language.

I just tested running on VMware on my Mac, with Ubuntu installed as a virtual machine, and under Ubuntu I ran Wine, and under Wine I ran MUSHclient. Now this is two virtual machines layered on top of a Mac, in effect, and the performance was still fast enough that a screen-full of description shot by faster than you could read.

Any Mac users out there could easily run MUSHclient under VMware if they had an old copy of Windows around to install on it, or failing that, install Ubuntu for nothing, and still run it, as I just did.

However as you note, performance isn't everything. There are claims MUSHclient is hard to use, and I can't really comment as I am used to it now. I try to make it simple and consistent, but perhaps I don't totally succeed. Version 4.54 improves things a bit by removing some of the extra windows that pop up if there is an error installing plugins.

Sometimes when I try other clients I think "I have to do *what* do make a trigger?" - but again, it is probably a case of getting used to the way a client works.

I wish the developers of Mudlet every success, after all it is cross-platform and open source.

I note that on their home page that "the text display is designed to handle thousands of lines under a second". Again I note that "designed to" and "actually does" are two different things, but for my own amusement I may just check out which client handles a few thousand lines of data faster. ;)

I have had 2500+ triggers active at the same time and still had a very respectable speed, although I could notice it getting a tiny bit sluggish. Then again, that's on a laptop that is now 5 years old, and I probably didn't help much by having a buffer that was about as large as my entire internal memory (I could go back a full month, as I always hibernated the silly thing instead of shutting down.)

Last time I saw Mudlet, which was in its very early stages, it didn't lift a candle to MUSHclient. They probably optimized a good deal since, but imo MUSHclient is still the fastest client out there (for me personally).

I just ran the first five tests on this page:

http://www.gammon.com.au/mushclient/benchmarks.htm

Both MUSHclient and Mudlet performed at the same speed, as far as I could tell with my stopwatch (basically about 1.8 seconds to process over 5,000 lines of coloured text, matching some triggers, not matching others).

According to the MUSHclient "info" page, it matched 8335 triggers in 0.47 seconds (however during that time it had to process another 10 triggers per line which did not match).

So I can't really agree that Mudlet "doesn't compare" to MUSHclient in display or processing speed. They seem to compare quite well, being pretty evenly matched. Since I believe that Mudlet uses Lua for scripting, I presume any scripting it does will operate at a similar speed (for whatever CPU you have), compared to MUSHclient Lua scripting.

So really, it comes down to personal preference, ease of use, cross-platform compatibility, forum support, and whether any bugs are fixed promptly. :)

Well you definitely have them beat on that last one...

They claim to be open source, but I've found no less than 4 git repositories, none of them up to date, some maintained by different people for different aspects of the program (isn't that what branches are for?)

Their client also has some annoying quirks with displaying tabs (messes up the whole display), and their color matching trigger is very annoying... It doesn't check that a line starts with a color combo (like every other client in the world), it checks that it's anywhere in the line... Plus their plugin/package distribution system is annoying (distribute an XML that a user basically imports into their world -yaaa, cluttered trampled _G variables).

That said, I like their idea of having a "lua function" pattern to trigger on (you know how much I like LPEG). I also like that they basically register callbacks for Event Handlers. It makes resize window routines simple for your package and eliminates worrying about trampling someone elses routine, since you're sharing one global environment.

So, I still prefer MC.
Sorry Nick, you're not getting rid of me. ;)

I almost wrote to you and suggested you do a JSON reader in LPEG, but I thought that was probably reinventing the wheel (there is one already but I thought it was a bit wordy). Still, if you get bored ...

Yes, LPEG *is* a gem isn't it? Trouble is, if you don't use it a lot you tend to forget how to. Well that happens to me anyway.

You should mud with WillFa. He won't let you forget LPEG. Ever.

:)

Lol. Is he in a guild called LPEG? And has a pet LPEG?

Ya know that t-shirt that reads:

"There are 10 types of people, those who can read binary"... ?




I just set my title to:

Hellbound lpeg.Cc("your attention") (good)

Well you just gave me an idea ...

Oh the nerdery!

As a basic user, I found Mudlet a little easier to use - there was a single row of large buttons, and pressing the first one ("Connect") allowed me to enter connection details and start playing. With MUSHclient I had to go to File->New World, which I found less intuitive. Like most users, I didn't bother reading any instructions, I just jumped straight in and started blundering around.

When it came to writing scripts, I gave up with Mudlet. I've seen some nice screenshots of what can be done, but I find their editor really uncomfortable to use, the documentation is pretty sparse, their graphical functionality is limited (eg I have to provide 1024 individual graphical terrain tiles, as opposed to a single terrain strip like I do in MUSHclient) and they don't support open protocols (so I had to hack together psuedo-ATCP support in order to trick Mudlet into reading MSDP data - which I've just discovered causes some strange side effects for CMUD users).

In terms of performance I've not noticed any difference between the two - both run nice and fast, and appear very stable. I've have had a few usage problems with Mudlet (vanishing text and session freezes), but as nobody else has reported them they're probably due to my "experiments" with psuedo-ATCP.

The main reason I'm interested in Mudlet is because of its cross-platform support. I know MUSHclient can be run under WINE and such, but as far as I know, none of the players who have tried have managed to get my plugin working. For first-time mudders, I really want to provide them something that's as easy as possible to get started with (although it looks like the PHudBase browser client might be a simpler option here, as it doesn't require any downloads).

Well, I've been a MUSHClient fan for a long time. I even paid for it, back in the day. I don't really want to get into a debate about it, but as an OSX user, I will say that there are a lot of problems with the OSX release of Mudlet that can be pretty frustrating. I think it'll be a solid client someday though.

I was excited by some things, like the mapper and whatnot. But now that Nick has been working on a general purpose one for MC, and now that WillFa is working on making it not hate 3k, if I'm honest.. I'm pretty thrilled to just stick with MC :)

Nick seemed to reply everything I would've liked to, thank you! A few points unaddressed:

opensource: Mudlet follows the same code model Lua has used successfully for the last 17 years. I'd say Lua is pretty successful at what it does; hence it's a good model to use for Mudlet development as well. The model is that all the code released is open-source, snapshots are provided during the development, community feedback is taken into all decisions, patches are accepted.

Mudlet on Mac problems: first time hearing this, never heard from other Mac users as well - so unless reported, chances are the specific problem isn't one or won't be fixed.

I'd like to have that benchmark server up to test with. For now, I just tried a localhost connection, made on an XP guest in Virtualbox w/ Ubuntu host:
http://dl.dropbox.com/u/84880/mushclient-display.ogv
http://dl.dropbox.com/u/84880/mudlet-display.ogv

With both clients being free and open-source, competition is moot - it's just what the user likes most :)

It looks like MUSHclient tries to actually render everything, whereas Mudlet renders a bit and then just hangs for a second or so before jumping to the end. Is that intentional or accidental behavior?

Hmmm, that's odd. You are right David, it looks like that, however I thought MUSHclient used to do what Mudlet seems to be doing right now.

Basically GUI applications tend to work like this:

• Some change happens (eg. new data arrives)
• Some or all of the output window is marked for redraw (ie. invalidated)
• The OS generates an Update event once it finds part of the window is "invalid"
• The client clears some or all of the output window and redraws the new information

The point is however, that the Update event is generally low priority. So you may get the first couple of steps (process a change, mark more and more of the window as invalid) many times, before, in a quiet moment (an Idle moment) the Update is generated.

This is generally much more efficient, so that if you have (say) 100 lines arriving, you don't redraw the whole screen 100 times, you add the 100 lines to the output buffer and draw it once.

Visually (and in my own tests too) it seems that MUSHclient is redrawing more often than I would expect, thus giving the effect of scrolling text, whereas Mudlet seems to be (arguably more correctly) drawing in batches. Of course, you may well prefer the scrolling look, but I thought that was what the "smoother scrolling" global option in MUSHclient was supposed to do.

I better look into that.

Mind you, in both cases the text is drawn really fast, and you could make out at case that the Mudlet approach of chunking it up actually makes it hard to see quickly-arriving text (which may be important). However as MUSHclient used to do the same, I wouldn't mark them down for that, and they may have an option to draw differently.

As for how I set up my test, these days it is hard to find a Unix box with telnet enabled (because of security concerns) and ordinary MUD servers tend to "throttle" output, so I adapted my "tiny MUD server" to just read a file and dump it to the client. If anyone wants a copy I can post it.

After setting some breakpoints, it seems that the code is not running unexpected UpdateWindow calls.

I'm not sure why it seems to be redrawing more often than I would expect. I have to guess that earlier versions did InvalidateRect (which invalidates part of the window) whereas now it does Invalidate (which invalidates the whole window).

The invalidation of the whole window is part of the change made for miniwindows, because you can't "scroll" a miniwindow upwards when you receive a new line.

Prior to miniwindows, this is what would happen. Say you had 25 lines of text on the screen...

• A new line arrives
• The output window was scrolled upwards (a bit-blit sort of scroll built into the operating system API) by one line height. This made the existing text move up a line (and the top line would disappear)
• The bottom line area rectangle was invalidated (with InvalidateRect) so that line only was marked for redraw.
• Eventually on an Update event the invalidated parts would be redrawn.

Of course, on more lines arriving the OS would scroll up the invalidated rectangles internally, so that the invalidated area would get larger.

I just have to guess that Invalidate must generate a higher-priority update event than InvalidateRect. A bit more testing might throw some light on this.

Well now I'm puzzled.

Adding an extra metric into MUSHclient shows how many times the OnDraw function is called to draw the output window.

Testing it with the high.txt file (5378 lines) puts 5416 lines into the output buffer (OK, some probably wrapped so that accounts for the extra lines).

However the OnDraw routine was called 159 times (that is, we got 159 Update events). So it certainly didn't wait until the end, nor did it do an update per line. Or even per screenfull, as the screen was about 58 lines in height.

Nor does there seem to be a direct relationship between the number of packets received and the times the screen was redrawn. It appears to be redrawn about every 2750 bytes, however the packet size was 1000 from this particular server.

My best guess is that Windows is noticing the Invalidate events and pumping out Update events to force a redraw at some sort of interval, not necessarily directly related to anything else. However not knowing the exact data flow at the TCP/IP level, maybe those 159 redraws happened to coincide with a short break in the incoming data, thus making the Update event the highest priority one at the time.

I think this is probably most likely, though I'm not sure about the "priority" bit. From what I've read so far, the message queue doesn't seem to be prioritized, though there are a handful of window messages that bypass the queue (and SendMessage() will also do this). I think you're just seeing the result of the order the events happened to be posted in, and I wouldn't be surprised if you got different results if you tested again.


edit-before-post: Also, MSDN says this about InvalidateRect():

Effectively, this make the WM_PAINT message a low priority, which is what I meant. It gets done if nothing more important needs doing. I didn't mean there was some sort of "priority sequence" just that updating the window is a low priority.

Yeah, so you were right. Hope I helped to at least clear up your puzzlement a bit, though.

Please do post the server script thing! Something for all to test against would be nice.

The version I amended is here:

http://www.gammon.com.au/files/muds/tinymudserver_v2.1.tgz (15745 bytes)

I updated tinymudserver_v2 to compile under recent versions of gcc. This one is only a single .cpp file (I think I overlooked I had a more modern one, oh well).

Just type "make" to build it, and then ./tinymudserver to run it.

Then fire up your client (eg. to localhost or wherever you have the server running).

Make a new character (type "new") and give a name and password.

Then once connected type "send xxx" to simply open file xxx and dump to the client.

I just grabbed the files mentioned in the benchmarks post, eg.



So to test in the client you connect as described above and type:

I wish this were true. Lua scripting performance is still far too slow for my liking, even on brand new hardware. :(

This is a rather surprising claim, to say the least, considering that Lua is used to satisfaction on embedded devices and in 3D games, which have far higher performance needs than MUD clients.

What kind of thing are you trying to do for which Lua is too slow?

It's more likely that the algorithm being used is inefficient or slow in some way (perhaps a hidden bottleneck). The core Lua language is extremely fast!

When I was playing around with rendering 3D scenes for Aardwolf, as described in http://www.gammon.com.au/forum/bbshowpost.php?id=10346&page=2, I was pleasantly surprised to see that not only was the scene rendered in under a second, I was getting quite a good frame rate (I think around 25 FPS). And this was individually addressing each pixel (with WindowSetPixel)! Not only was that a lot of pixels, but there was quite a bit of maths behind working out what pixel to set, where.

Other things I have done in Lua, like the spellchecker, also work very fast.

I suppose you can always say it is "too slow for my liking" but honestly, Lua in general is very fast, considering it is a scripting language.