Gammon Forum : MUSHclient : Lua : Garbage collection way too lazy?

Register forum user name Search FAQ

Gammon Forum

Entire forum

MUSHclient

Lua

Garbage collection way too lazy?

Garbage collection way too lazy?

It is now over 60 days since the last post. This thread is closed. Refresh page

Posted by

Larkin (278 posts)

Date

Sat 12 Feb 2011 03:55 PM (UTC)

Message

I've got a ton of RAM, so I didn't notice this at first, but a friend using my same scripts with a lot less RAM has so many problems with memory usage that he has to reload the system every 15 minutes or so.

After some investigating with a clean world and a couple of simple functions, I determined that Lua just wasn't collecting my garbage. I could set the variable to an empty table, set it to nil, make it local inside a function that returned (making it go out of scope), and the memory continued to go up, never down. Running collectgarbage("collect") manually does make it drop back down to what it was before my function to allocate, so I'm wondering if the automatic garbage collection is just not working in MUSHclient.

Anyone else have this sort of problem? If it's just me, I suppose I'll keep doing the manual garbage collection as a workaround, unless someone give me pointers on returning tables from functions and having them deallocated every time when they're not needed.

Posted by

Twisol USA (2,257 posts)

Date

Reply #1 on Sat 12 Feb 2011 11:41 PM (UTC)

Message

Any chance you could post your test scripts?

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

Posted by Larkin (278 posts) bio

Date Reply #2 on Sun 13 Feb 2011 12:08 AM (UTC)

Message


function leaker()
  local something = {}
  for i=1,50000 do
    table.insert(something, "blahblahblah")
  end
  return something
end

function testleak()
  local a = leaker()
end

And then I just executed "/testleak()" on the command line a couple times and watched the memory shoot up. Now, if I put a "collectgarbage('collect')" in that function, the memory remains the same.

Posted by Twisol USA (2,257 posts) bio

Date

Reply #3 on Sun 13 Feb 2011 03:47 AM (UTC)

Amended on Sun 13 Feb 2011 03:50 AM (UTC) by Twisol

Message

You're correct, but that's the behavior I'd expect. The GC doesn't run when the script ends; it runs when it needs to. Try this and watch the memory usage:

function testleak()
  for i=1, 1000 do
    local a = leaker()
  end
end

It goes up, then down, then up, etc., showing that Lua is perfectly capable of collecting garbage automatically. Now, I'd expect something different if MUSHclient itself called lua_gc() every time a script executed, but that's unnecessary overhead IMHO.

By way of analogy: The garbage trucks here only come on Wednesdays, not every time I put something in the trash.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

Posted by Nick Gammon Australia (22,982 posts) bio Forum Administrator

Date Reply #4 on Sun 13 Feb 2011 06:39 AM (UTC)

Message

Larkin said:


function leaker()
 ...
end

function testleak()
...
end

Now, if I put a "collectgarbage('collect')" in that function, the memory remains the same.

Which function? You posted two.

There are a number of points here. For one thing, the Lua garbage collection tries to adapt, I think. The first time it collects after a short time, then it collects later, and so on. It tries to balance time taken collecting with the need to reclaim memory.

Second, collecting garbage doesn't necessarily make your Windows memory go down. The C runtime libraries grab memory from a Windows pool when needed. Even if the C runtime libraries find they have some "spare" memory they don't necessarily give it back to Windows, particularly if it is fragmented.

I would be looking at your design, in particular can you re-use tables rather than always creating new ones?

Quote:

I determined that Lua just wasn't collecting my garbage.

Watching Windows memory usage isn't a reliable way of concluding that Lua isn't collecting garbage.

Anyway, the Lua garbage collector has parameters you can tweak.

Also it might be worth collecting garbage manually at a logical point (eg. in a mapper, after you draw the map). That way you always have a small overhead rather than a big pause from time to time that you notice.

Quote:

I'm wondering if the automatic garbage collection is just not working in MUSHclient ...

The Lua that comes with MUSHclient is basically standard Lua, maybe with a couple of compile options tweaked. I don't know why I would change it to disable garbage collection, and indeed I haven't.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

Posted by

Larkin (278 posts)

Date

Reply #5 on Sun 13 Feb 2011 02:47 PM (UTC)

Message

I didn't mean to imply you had messed it up on purpose, but my experience lately has been that the garbage collection never reduces my memory unless I run it manually, and I have no idea why that might be.

I can run the testleak function a dozen times and watch the memory allocation climb by several megabytes, but it never cleans it up. Is the threshold really that high?

And, if there's a better way to handle this sort of thing, I'm open to suggestions. I just don't want to have to make some sort of ugly hack to circumvent this problem when it could be handled more elegantly, I guess.

Posted by

Twisol USA (2,257 posts)

Date

Reply #6 on Sun 13 Feb 2011 08:44 PM (UTC)

Amended on Sun 13 Feb 2011 08:53 PM (UTC) by Twisol

Message

Larkin said:
I can run the testleak function a dozen times and watch the memory allocation climb by several megabytes, but it never cleans it up. Is the threshold really that high?

My MUSHclient process started at 9,000KB or so. It took about 50,000KB more before it stopped and collected (though of course the Task Manager only has 1 second of resolution).

I also found an old Lua-L email from 2003 mentioning a "used x 2" threshold [1].

[EDIT]: It seems that Lua 5.0 had C API functions for setting the threshold, but I can't find any trace of them in 5.1...

[1]: http://lua-users.org/lists/lua-l/2003-02/msg00540.html

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

Posted by

Larkin (278 posts)

Date

Reply #7 on Sun 13 Feb 2011 08:49 PM (UTC)

Message

That's not so much on a system with gigabytes of RAM, but my friend has 512MB, and he runs out faster than me. My MUSHclient usually sits around 90MB with my one world open (I have a lot of settings), and I suppose I expected it to clean up a little more aggressively than every 40-50MB.

Anyway, since I have access to the Lua garbage collector, I'm fine with just doing the collection a bit more often myself now. No big deal. I was wondering if others had bumped into the same sort of issue, like maybe it's a scripting inefficiency on my part.

Thanks.

Posted by

Twisol USA (2,257 posts)

Date

Reply #8 on Sun 13 Feb 2011 08:54 PM (UTC)

Message

It might also be useful to note that MUSHclient gives each plugin (as well as the world itself) its own Lua VM, meaning none of them know about eachothers' memory usage. If they all sit just below the threshold, you could definitely have a memory problem.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

Posted by

Larkin (278 posts)

Date

Reply #9 on Sun 13 Feb 2011 08:56 PM (UTC)

Message

I only use plugins when that's the only way to get access to certain functionality, like GMCP telnet suboptions. My mapper is entirely main world script, and it's definitely the big memory hog.

Posted by Nick Gammon Australia (22,982 posts) bio Forum Administrator

Date

Reply #10 on Sun 13 Feb 2011 09:28 PM (UTC)

Amended on Sun 13 Feb 2011 09:29 PM (UTC) by Nick Gammon

Message

Larkin said:


function leaker()
  local something = {}
  for i=1,50000 do
    table.insert(something, "blahblahblah")
  end
  return something
end

function testleak()
  local a = leaker()
end

OK, if you run "testleak ()" there is no reason, yet, for it to collect the garbage. Inside leaker it is validly using a lot of memory. After you run testleak the variable "a" goes out of scope, but the script ends. You would need to do something else afterwards to trigger off garbage collection, and I'm not sure what that would be.

Some tests seem to indicate that just running some simple scripts don't do it (of course, doing collectgarbage does).

Judging by what little I have read, there would be a trigger point. In your case you may have used 1 Mb of memory, the trigger point might be when another 1 Mb is used.

I seem to recall, but can't find it right now, that in my mapper I just collected garbage after drawing the map. All those temporary tables got cleaned up then. I don't see any harm in that, as you know you have used temporary things, you know you are finished with them, and a slight extra delay after drawing the map won't annoy people too much. However running out of memory later on, will.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

Posted by

David Haley USA (3,881 posts)

Date

Reply #11 on Tue 15 Feb 2011 08:48 PM (UTC)

Message

I suspect that you wouldn't actually run out of memory; the GC would kick in if it had to (but the penalty in delay would be higher). However, you would definitely get the perception of wasteful, ever-growing memory usage, which freaks some people out.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

Posted by

Larkin (278 posts)

Date

Reply #12 on Tue 15 Feb 2011 08:53 PM (UTC)

Message

I believe what my friend was seeing was overall system memory usage growing beyond the capacity for his physical memory, so he was really bogged down when it started paging to virtual memory all the time.

Posted by

Nick Gammon Australia (22,982 posts)

bio Forum Administrator

Date

Reply #13 on Tue 15 Feb 2011 08:56 PM (UTC)

Message

What version of MUSHclient did he have? Some older versions had a bug about normal memory leaking.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

36,943 views.

It is now over 60 days since the last post. This thread is closed. Refresh page

Go to topic: Search the forum

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.

Comments to: Gammon Software support
Forum RSS feed ( https://gammon.com.au/rss/forum.xml )