I've been having a problem where after I have Mushclient running for a few hours, it will start to stiffen up, causing lag like effects and short freezes, and then it will ultimately freeze up for minutes at a time. The interesting part is that I can prevent this from happening by regularly reprocessing my script, and when I do this it causes the same freeze that I described before, but at least I can predict and control it. As part of my 'refreshing' routine I also bring up the configuration menu because that causes a freeze as well. After I do either of these things, I can do them again freely for a few hours until Mushclient once again shows signs of freezing.
I suspect I just might have too much on my world. 980 triggers, many of which are fairly complex, and a 111 kb Python script (I don't know an easy way to count the lines). The world itself is 497 kb. However, in between freezes it's fast and efficient.
I'm currently on Mushclient 4.01, but I've had this problem for a while so it also appeared on older versions. I run the world in question on two computers. One has a 2.4 Ghz processor and 512 mb of ram, the other has 1.4 Ghz and 512 mb. Both have Windows XP home edition. Both computers take relatively the same amount of time to freeze, though it's hard to tell for sure because the freezing time itself is variable.
So, I guess this is something I could live with because I have been for so long, but any help or insights would be appreciated.
A far more likely cause would be that some versions of Python had/have memory leaks, which, over time, can cause Python to slow down. Since scripting has the effect of interrupting the client (i.e. freezing it), anything causing issues in the script space is going to make it look like the client is getting messed up.
Mind you, I personally only have a 83k world file, 73k in the main script space and about 52k in plugins. But, none of them are in Python either, and I do know that some versions of that *are* known to have memory issues. Since you are dealing with 512M ram on both systems (You are actually trying to run XP anything on less than 1G?... Well, it will, but not well I suspect... lol), then its not a surprise that both would have the same problem at nearly the same time, especially if the issue causing it lies some place inside timed events, which happen at exactly the same intervals on both.
Mind you, this is a guess. I can't imagine however that someone else doesn't have bigger world and script files than you, or that if this same problem was happening, they wouldn't have reported it by now.
I have -far- larger scripts than you do (currently stuck with a 644kb script spread over some 40 odd files for curing alone, add a bunch of subs and other scripts to it and I can easily say I have 1mb worth of scripts alone), and I kind of recognise the freezes you mention. (Edit: on top of that, I have multiple worlds with the same plugins open at the same time. ^_^)
Best way to prevent them:
1. Put a lower limit to your output buffer. I'm a glutton for pain and have it set to 500,000 lines (even on my 512MB daily usage box), and I don't dare scroll up if I've been browsing too. Firefox and MUSHclient tend to fight for my RAM and as a consequence, turn it into swapping hell.
If you press Ctrl+Shift+Esc you will open the Windows Task Manager. Under the Performance tab you can see how much virtual memory you are using. If you have exhausted physical memory (I think Physical memory: available will be low), then when any program attempts to access memory - which is normally a fast operation - it has to access disk instead, which is slow.
You could also click on the Processes tab, sort by memory usage, and see which applications are using a lot of memory.
I tend to find that stuff you once installed, even if you aren't using it (like my G15 Gamers Keyboard) tend to put a heap of things into memory "just in case". The same with my printer.
I have 1.5 Gb of RAM, which is enough to run quite a few things at once, including MUSHclient, web browsers and so on.
Filling up your output window could certainly contribute to the problem. Check the world configuration -> Info tab, and see how much memory the output buffer uses. Reducing the number of lines there could help.
Also script languages may consume memory, possibly not freeing it all up.
You might defer the problem, as Worstje, says by buying more memory. It is cheap these days, and for $100 or so you might be able to bump your RAM up to 2 Gb.
The trouble with XP is that the OS itself uses something like 128 Mb, so around 25% of your available memory has gone before you even do anything. If you start with more memory in the first place, you will have most of it free initially.
Ok, I don't have anything much to contribute to this discussion, but I just have to say... WTF! What the heck are you doing that uses that much script? I regularly use like 7-8 plugins in my main world, and they only add up to like 46KB, and my world file is only 60KB. I only have 67 triggers(most of which I don't even use that much, and I think I have several disabled) and 2 timers. Every Plugin I /have/ even the example plugins only add up to 183KB. I mean seriously are you guys botting or something? With that many triggers/script.
I have noticed that firefox eats up a ton of memory, and I usually run it while MUSHclient is open, but I figured firefox is just a memory hog and would fight with anything else that was actively using memory, truth be told it fights more with FlashGet and my email program than it does with MUSHclient.
Also, do you use all those script/triggers all the time, or could you maybe split them into seperate plugins which could be enabled/disabled or installed/uninstalled to free up memory? Just a thought, don't know if it'd work.
IRE muds are kind of high maintenance, and curing is all but straightforward. The large curing script (622kb I think it was) deals with a lot of things: various curing balances, around 30 skillsets which have a varying amount triggers in them (some only 2, others like 50), prettifying output in some cases so it better represents what I actually want to know and so forth. To copy the first few lines of my 'debug stats' script (I kind of felt I needed statistics after a while)...
Besides that, I have plugins to deal with my offensive needs (aliases and such mostly, some sub stuff, too.). Others gag and replace my prompt, another fixes the extra new line issue the IAC EOR/GA option gives me. Then I have a singing script (wrote it for roleplay; got lazy typing stuff over.)
I am doing a sort of botting actually. I don't know if you're familiar with IRE, but in their games it's common practice to automate your defense and the 'system' you make can get very big very fast.
Anyway, I've done some investigation into the memory thing. First thing I checked was my output buffer, which is 10,000 lines long. I brought up the task manager and saw that Mushclient took about 20,000 kb of my memory. Then, I held down enter in Mushclient to spam myself with my prompt and fill up my buffer. As expected, Mushclient started taking up more and more memory, and if I stopped spamming, it would remain at it's higher memory usage level.
However, when the buffer filled up, Mushclient continued taking more memory as I continued spamming. I periodically checked info on the configuration menu and the memory used by output buffer remained steady at 2.2 mb, even though the memory used by Mushclient was going up by thousands of kb. When I got to about 350,000 kb, I went ahead and reprocessed my script and it did the normal freeze routine. Once it finished, Mushclient's memory use dropped to about 60,000 kb.
So yes, it's definitely a memory related problem, but why do more lines cause Mushclient to hoard more memory if they're not staying in the output buffer?
One possible explanation is that something (eg. a trigger) that matches is consuming some resource when a Python script is called, that is never freed.
It would be interesting to turn off scripting and rerun your test - that would isolate it to MUSHclient itself consuming resources, or the script consuming them.
However the fact that reloading your script seems to clear things points to the script engine (or the script itself) consuming something. I am not familiar enough with Python to suggest a solution.
Once the buffer fills up, Mushclient stops taking memory if triggers and/or scripting are disabled. So, I guess I have a trigger that's causing this, and considering the nature of my tests (and the nature of the problem itself, actually), it would make sense if it was a prompt trigger.
That narrows it down quite a bit, but I still have to find the 'resource'. Looking at my prompt trigger, I see nothing obvious that would cause the problem, but I'm not completely sure what I'm looking for. It would be something like a line that appends to a list every match, right?
Try and see what happens when you force a manual garbage collection:
/import gc
/gc.enable()
/gc.collect()
If that drops it, it just means Pythons garbage collector is not freeing up the memory. Of course, doing this from command line only affects the main script but not the plugins that may use python since those are seperate environments.
There are potentially a number of places where memory could be wasted:
MUSHclient allocating space for a string when calling a script and it not being freed - this *may* be happening, but I am using C++ classes for this and they should be freed when the destructor is called.
The script engine making a copy of the strings (eg. in its own format), and not necessarily freeing it as it should.
Some fixed overhead incurred by the script engine - erroneously - so that it gradually uses memory
The script itself doing something (like storing a variable into a table) in such a way that it consumes memory.
It could make a difference the way you are calling the script - are you using "send to script", or putting a script function name in the script box, and using a script file? The "send to script" method incurs a compilation step each time, and conceivably this is wasting memory in some way.
Garbage collection doesn't seem to help. In fact, Mushclient takes another few mb of memory when I collect.
Yes, I'm using "send to script".
One other thing I noticed that might be relevant. In the task manager's performance menu, there's a gauge for page file usage right below cpu usage. I'm not exactly sure what page files are, but as Mushclient's memory use increases, so does page file usage. If I minimize Mushclient, it's memory use will drop dramatically, but page file usage will remain where it was.
Minimizing, I believe, flags Windows to release the actual memory, and leave the copy in the page file. Thus I would expect the page file to stay the same, but the use of "real" memory to diminish.
You could try moving a trigger that is called a lot into the script file, rather than "send to script" - this may lower whatever overhead doing the "send to script" has.
I moved my prompt script to the script file and now once I fill up my output buffer, Mushclient almost completely stops taking new memory! Is calling the script file generally more efficient than using "send to script"?
Definately. With send to script, MUSHclient needs to parse the stuff in the Send field each time the trigger is fired (to take care of %1 and all that). After that, the code is being fed to the script engine of your world. Then the script engine has to translate it again.
Using send to script, the code is only read once and then stays within the script engine. MUSH simply tells the script engine 'execute function PromptCapture', which is definately less overhead.
Using send to script, the code is only read once ...
You mean, using the script file.
I am glad that solves the problem. "Send to script" shouldn't really consume lots of memory, but it sounds like Python is using something internally that it doesn't release.
Worstje is right, using a script file will incur less overhead. Doing "send to script" in triggers (etc.) is easier because you can see the code in front of your eyes, but less efficient.
I have a similar problem with my MushClient/Python, though I don't use Send To Script at all; everything is script calls.
Each function call on the script adds about 4KB to MushClient's footprint. Python itself doesn't leak memory in my experience. I use it for automation all the time.
However, Python's garbage collector is a refcounter, not mark-and-sweep. So it can't detect cycles.
My assumption has been that the World object handed to Python is not being destructed properly because it contains a cycle.
I don't create the world object each time, so I don't see how MUSHclient is doing it. If there is a fundamental problem, since all script engines, except Lua, are called using the Windows Script Interface, all languages would have a leak.
AFAIK, so is Javascript's, VBScript's and PerlScript's.
If the world object is being handed to Python with a cycle and on each script invocation the world object doesn't reuse the exact same Python objects... I think that's enough to make it leak.
One of these days I'll open up the MushClient source and see if I can work out what the problem is. It's definitely an integration/interaction problem rather than something that can be laid at the feet of either side :(
This effectively calls a function 1000 times, but when using Lua and doing this, the memory footprint seemed to rise about 40 Kb.
Doing this under VBscript (which uses COM) seemed to consume about 24 Kb. This is a lot less than then 4 Kb "per call" you reported.
You could probably make a case that the memory should not rise at all, so there may be a problem here.
In the case of GetInfo (1) a string is being created, so perhaps it is not being disposed of correctly. Looking at what is happening here, GetInfo (1) does this:
VARIANT CMUSHclientDoc::GetInfo(long InfoType)
{
VARIANT vaResult;
VariantInit(&vaResult);
vaResult.vt = VT_EMPTY; // in case dates are zero
switch (InfoType)
{
case 1: SetUpVariantString (vaResult, m_server); break;
// blah blah for other cases
default:
vaResult.vt = VT_NULL;
break;
} // end of switch
return vaResult;
} // GetInfo
So, clearly a string is being allocated, and the allocation of around 1000 strings might account for a lot of the memory usage I am seeing.
However as that string is being passed back to the script engine (the one that did the GetInfo) it seems to me that the script engine is responsible for freeing the string up when it is done with it.
Ah, I meant for every call from MushClient to Python. And I got the 4KB number from running a 0.1s timer for a minute and noticing the memory footprint increased by about 2.5MB
I added:
def test_memory():
for i in range(1000):
world.GetInfo(1)
to my script file and executed it. The first run seemed to add 4KB. Subsequent runs added 0KB to the footprint. (I should note that 4KB seems to be Task Manager's minimum granularity for memory usage.)
Recompiling the script randomly added from 20KB to 280KB :( Which smells like a bug in the WSH interface that the pywin32 extension implements.
Checking the result I mentioned in the first paragraph I watched the footprint increase by 3.8MB over a minute ... and then noticed that the timer was doing a Send-to-Script rather than a straight function call (because I defined my poll() function without arguments in the script, *sigh*). Changing it to a function call stopped the leak (memory usage stable at 28636K and script calls from timer > 4000 while I typed this post).
HTH.
EDIT: (I'm using Python 2.5.1 [2007-Apr-18] and pywin32 build 210 [2006-Sep-22], which are the latest versions of each.)
EDIT[2]: (Python 2.5 because I'm a neophile and the "with" statement.)
First of all, oops. I did mean script-file in that post far above. (Edit: I keep writing down the wrong one, bad addiction, gah!)
Second, I don't have too much experience with Python globbing memory like an insane beast. I have a plugin with 650 triggers in it (and growing), and I can reload it to my hearts content. While loading, it temporarily jumps up by around 1mb in memory, but immediately drops again. I tried kicking different triggers into action (both send to script ones and script-file variety), and I can easily say that neither makes a difference for me.
Then again; this computer uses a pretty old version of Python (2.4.2) and a pretty old version of pywin32. Maybe it is a more recent bug in either of those that is causing it?