Automatic log file size limiting and splitting?

I usually have logging enabled and it's great, but at the end of the day I notice that each of my log files are HUGE.
Opening a TXT or (even more so) a HTML file that is over 3 megabytes is rather taxing on my computer and (most likely) the computers of other MU* players.

My suggestion:
Would it be possible to sit up a filesize limit?
It would work like this: when the log file being written hits the size limit (i.e. 500kb) that file would be ended and the logging would be written to a new file, a second file. Then If that second file also grew to (once again, for example) 500kb, it would also be ended and a 3rd file would be started.

What do you think?

You could easily script something to check the file size (FSO: obj.size) and then append the closing HTML tags, and open a new one. You could even make it add a link to the next page at the end.

If you're running on an NTFS filesystem (the default in WinXP and Win2000 I believe), then you can use the option 'compress folder to save disk space' on your log folder. To do so, right click on the folder in explorer, click 'advanced', and the option is there. Apply to folder, subfolders and all files.

I have 106mb of log files uncompressed - I'm a coder, not a player, so I don't have as many logs in game. :) Using this option, the logs take up only 32.6 mb. Depending on your log files (you might have much more repetitive text such as room descriptions or exit lists), the compression ratio may be even better.

Note that compressed files like this are not "zipped" and can be accessed just like any old file on your hard drive. Most importantly, MUSHclient can write to them 'live' and you don't have to do anything at all. The (de)compression is fast and completely automatic.

(edit)
Eh-heh. Sorry, I didn't notice that you were talking about opening the files, not storing them... yes, that is definitely a problem even for faster computers. Opening up a multi-megabyte HTML file is always rather tough on the computer. :)

A simple approach would be to make a timer that fires every hour or so and closes and re-opens the log file. If you open it with a blank name it should take the name from the configuration window, which can include the date/time.

Also, not to advertise or anything, but I've found that uhh, what's its name... TextPad can open my multi-megabyte logs (200+) without any difficulty at all... I use it for browsing through the logs, maybe you could give it a shot.

Textpad doesnt parse HTML. Or rather, it doesnt display/interperate/whatever HTML.

HTML logs are really the only problematic thing, plaintext is no big deal for large files.

I use TextPad for code writing when I need to deal with java or html or ANYTHING script related because it can (and does) match tags and whatnot quite nicely. Mind you, we aren't talking about the windows textpad which is um..... crap but a much more powerful downloaded editor with pretty nearly the same name. You may need to change settings a bit to get it to display code tags properly but it will do so, can even compile java for that matter.

The TextPad I'm refering to is put out by helios software and can be found at www.textpad.com and should afaik be available in an unlimited fully functional eval version.

gVim! *Real* coders use gVim! :) (Or maybe xEmacs...)

http://www.vim.org/download.php

There's a Windows version available that works very, very nicely.

If gVim is scary to you (understandable if you're not used to *nix environments), Crimson Editor is also a pretty good text editor.

http://www.crimsoneditor.com/

Don't know TextPad, so can't say much about it.

Eh, for that sort of stuff (the matching and such) you can use Arachnophilia. Completely customizable, although it is in java.

For macros and other massive replacements and such, I use PFE. I still havent found anything with a macro interface that I like besides it.


But point being guys, were not talking about text editors for text files, or even html files, were talking about VIEWING HTML, as in, <b> makes it bold, and such. Since, when you log in HTML, most of the time you dont view it in the HTML form, but as a webpage.

And he's looking for something distributable, shouldnt need to download anything. He just wants to be able to post logs on websites, or whereever, without spazzing out the unsuspecting users computer with a massive HTML file.

If you have a massive HTML file, well, you're pretty much screwed. Any program will just take a long time to render it (that's the term for what you're talking about). That's just how life goes... it's completely dependent on your computer speed.

The only option if you want to put it on websites, and if you don't want to have a massive HTML file, is to split it. There may be programs that do that.

I googled for 'html splitter' and the very first result looks promising:

http://www.rekenwonder.com/htmlsplitter.htm

Ah, I hadn't realized that we were talking about viewing html logs... I guess i didn't read that through :p I agree with Ksilyan on this one, it will take a lot of time, but you could use your text editor to split the file and add the header/footer to the new files and view those... By the way, Flannel, how do you replace stuff in PFE? I don't know if I have enabled it, but send me an email through the forum :p I find PFE's lack of regex etc makes it hard to do mass replacements, but I haven't looked into macros... Can you do more complex replacements than the "replace" dialog?

Yes, I would not post multi-megabyte log files for online viewing. But that is why the question was asked in the first place. :)

Probably a simple way would be to make a timer that fires every hour or two simply close and re-open the log, with the date/time in the name.

That way you always get (say) 24 logs per day, which makes more sense anyway when you are organising your web pages.

If they end up being too small, make it every two hours.

Alternatively have the timer do what Flannel suggested and have it find the log file size and close and re-open it based on that.

Or you could go through afterwards (with FSO and a small script, with some intelligence) and cut off every XXX lines, insert closing tags (and a link to the next file), and make a new file with appropriate opening tags with the rest.

You could do it either way, make a new file for the stuff after the split, or beforehand (it would be less to eat away and the stuff you eat, you put into a new file, rather than the other way around, less movement).

Thanks for all of the replies! (Even the ones that don't apply to my problem :).

The solution I like most is that one where a timer would check the size every hour or so and then close and re-open the log, with the date/time in the name.

I now have another question:


Where and how can I ask for some help with that... I have no idea how to write a script.