Gammon Forum : SMAUG : SMAUG coding : Log file size, bug file size, and their management....

Log on

Gammon Forum

Entire forum

SMAUG

SMAUG coding

Log file size, bug file size, and their management....

Log file size, bug file size, and their management....

It is now over 60 days since the last post. This thread is closed. Refresh page

Posted by

Nick Cash USA (626 posts)

bio

Date

Tue 09 Nov 2004 10:17 PM (UTC)

Message

Hello everyone!

I was wondering everyone's thoughts about a quota manager. While that is probably not an acurate term, there isn't much else to say. If you look around, you may notice (I usually notice it among SWRs) that sometimes you have a tendency to spew lots and lots of log file and bug file information. Now, when these files get large, you can reach your quota on your server, then you get pfile corruption and several other problems.

To prevent this, I was thinking of perhaps writing a snippet for those unfortunate people. It would scan through the log file directory, find the current log file and keep it, and delete all the rest IF there were say, more then 20 log files (to many? to few?), and manage the size of bugs.txt some how.

Now, I can figure out most of the stuff myself, but how would you all go about clearing out PART of the bugs file. I know how to delete log files (if there are a certain number), but if we opt for a certain size, would there be a way to determine the size of the directory? I know these are easily do-able, but I don't work with file size much on *nix systems.

Thanks everyone, lets try to help some people here :)

~Nick Cash
http://www.nick-cash.com

top

Posted by

Nick Cash USA (626 posts)

bio

Date

Reply #1 on Tue 09 Nov 2004 10:18 PM (UTC)

Message

Oh, and for those who may be wondering, it would be just like another update in the update section. And it would check say, every few minutes or so.

~Nick Cash
http://www.nick-cash.com

top

Posted by

David Haley USA (3,881 posts)

bio

Date

Reply #2 on Tue 09 Nov 2004 10:44 PM (UTC)

Message

I would set up a maximum 'size' e.g. 20 mb, and say 'who cares how many log files'.

Then, if you reach 20 mb, you kick out the oldest log file and continue. If you keep doing this and reach a point where only the current log file is active, you have to start doing filesize management on the current file. There should be file system calls that do this for you but I don't know them off the top of my head. The simple but bad way is to buffer the whole file, move forward however-many-lines in the buffer, and then re-write the file from then on . . . and finally, write out whatever you need to write.

I would suggest using a size count to kick out files, and a line count for the individual log file. After all, a partial line isn't very useful, and often lines are of a similar size.

I think that 20 files, with a cap of 10,000 lines should be enough. Assuming 250 chars/line, that's 2,500,000 chars per file which is ~2.5mb per file - at 20 files, that's a maximum of 50 mb of log files. Of course, tweak numbers as desired.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by

Robert Powell Australia (367 posts)

bio

Date

Reply #3 on Thu 11 Nov 2004 05:57 AM (UTC)

Message

I have set a cron job to remove them all, but if you wrote a snip and called it before the daily smaug reboot then this would make life easier for those who's host did not provide them with access to cron. you could expand it to also remove all those .bak files that seem to clutter everything we do as well.

Just a guy having a bit of fun. Nothing more, nothing less, I do not need I WIN to feel validated.

top

Posted by

Zeno USA (2,871 posts)

bio

Date

Reply #4 on Sat 13 Nov 2004 03:26 AM (UTC)

Message

After running into a problem where one of my log files was 18mb, and went over my quota, that caused some corrupt pfiles, I think this is a good idea. If you ever get to it, post it here. ;)

Zeno McDohl,
Owner of Bleached InuYasha Galaxy
http://www.biyg.org

top

Posted by

David Haley USA (3,881 posts)

bio

Date

Reply #5 on Sat 13 Nov 2004 04:03 AM (UTC)

Message

A cron job isn't necessarily enough. You might want/need to know "real-time" if you're filling up your logfiles - if your MUD runs a-muck (no pun intended) and starts writing out massive stuff, it might be too late by the time the cron job gets around to it.

Of course, it would be much safer to simply not overwrite the file until you're sure you can write out the whole thing, but that introduces an extra layer of complexity - not only in rewriting all the save functions, but in figuring out just when "I can write out the whole thing" is true.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by

Samson USA (683 posts)

bio

Date

Reply #6 on Sat 13 Nov 2004 02:22 PM (UTC)

Message

As far as things like bugs.txt, I've never seen a good use for it. I got rid of that file all together long ago and have yet to find a scenario where I say "Gee, I wish I hadn't done that now". Between core files and the general log, there is enough information to narrow down the cause of bugs. Why waste the space?

Up until now I haven't bothered with not saving .bak files since there has been the rare case where they come in handy. But not often. If you make a mistake early on and have saved more than once since making it, you're screwed anyway. I may just go ahead and stop that now as well. This will save more space, since those .bak files will often sit around forever until someone removes them.

So about all we have left now is a race condition where the logfile runs up multiple megabytes in seconds. Waiting for cron or the nightly reboot ( for those who even still have it ) isn't going to work in these cases. You can often exceed the typical 100-500MB quotas within 2-4 seconds. If you happen to be on a machine in which you have no quota, this can threaten the entire server in a matter of minutes. I once grew a logfile of well over 70GB when a file bug happened while I was down at the 711 stocking up on programmer food. Came back to see that and was amazed at how quickly it had happened. Scripts and timed jobs would not likely have stopped it unless they were checking constantly.

If you are ever caught in this type of situation, the best and fastest course of action is to delete the file which is growing out of control ASAP. The mud will still be attempting to log things, but with the file gone it's no longer a threat. You won't get much useful info out of one that's gone beyond a few megabytes anyway. At this point you can use GDB to attach to the process and find out where it got stuck.

top

Posted by

Zeno USA (2,871 posts)

bio

Date

Reply #7 on Sat 13 Nov 2004 05:16 PM (UTC)

Amended on Sun 14 Nov 2004 03:20 AM (UTC) by Zeno

Message

Speaking of useless things, I've ripped out any use of usage.txt and plevel.txt, I found both were far too large, and pretty much useless. I don't think my bug file has ever been that large. I've heard reports of logs files in the GB on my last host.

[EDIT] Now I may have to code this myself unless you release your code soon, because once again I went over my quota tonight. Darn logs.

Zeno McDohl,
Owner of Bleached InuYasha Galaxy
http://www.biyg.org

top

Posted by

Samson USA (683 posts)

bio

Date

Reply #8 on Thu 25 Nov 2004 01:30 AM (UTC)

Message

You can also add economy.txt to the list of files with no apparent use. Unless you like looking through tons of data on gold looting and such :)

top

Posted by

Zeno USA (2,871 posts)

bio

Date

Reply #9 on Thu 25 Nov 2004 04:12 AM (UTC)

Message

Ah, I wasn't sure if that was used in the economy code. As for the .bak files, I actually have found them useful, such if an area is lost when installing it, which has happened to me before.

Zeno McDohl,
Owner of Bleached InuYasha Galaxy
http://www.biyg.org

top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

20,438 views.

It is now over 60 days since the last post. This thread is closed. Refresh page

Go to topic: Search the forum

top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.