Page 2 of 4 FirstFirst 1 2 3 4 LastLast
Results 11 to 20 of 32

Thread: GZip Support for Session Logs

  1. #11
    Developer
    Join Date
    Apr 2006
    Posts
    922

    Default

    I do this with a post-run bash script. Here are some stats:

    Code:
    Size: 1 820 349 739
    Packed Size: 53 461 743
    Folders: 0
    Files: 3 213
    CRC: 80C172DA
    ----------------------------
    Path: Dropbox\kolmafia\sessions\Epicgamer.7z
    Type: 7z
    Physical Size: 53 501 753
    Headers Size: 40 010
    Method: LZMA2:26
    Solid: +
    Blocks: 1
    This is the solid archive for my main character, it has session logs from September 2008 until end of year 2017. It's has a 2.91% compression ratio.

    Code:
    Size: 271 421 367
    Packed Size: 29 471 797
    Folders: 0
    Files: 222
    CRC: B9FAF451
    ----------------------------
    Path: Dropbox\kolmafia\sessions\Epicgamer.zip
    Type: zip
    Physical Size: 29 499 535
    This is the ongoing zip that I update on a daily basis with my script that runs after mafia is closed. This has a 10.85 compression ratio which seems inline with what everyone else is getting. I re-compress the solid archive once a year and purge those logs from the daily zip.

    Here is my script if you're curious:

    Code:
    #!/bin/bash
    
    
    CHAR=$1
    
    
    DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
    
    
    pushd ${DIR}/../sessions
    
    
    chmod 664 ${CHAR}_*.txt
    
    
    7z u -mx9 ${CHAR}.zip ${CHAR}_*.txt
    
    
    oldest=$(ls -td ${CHAR}_*.txt | sort -r -n -t _ -k 2 | awk 'NR>7')
    if [ -n "$oldest" ]; then
        echo Removing old session files...
        rm ${oldest}
    fi
    
    
    popd
    Last edited by Rinn; 07-30-2018 at 05:36 AM.

  2. #12
    Senior Member
    Join Date
    Apr 2018
    Posts
    133

    Default

    Iím sorry if I insulted you. I was trying to explain that for the benefit of others reading this thread.

  3. #13
    Senior Member
    Join Date
    Apr 2018
    Posts
    133

    Default

    That’s almost exactly what I’d be doing if some scripts didn’t need to access the logs. I’d do it with a cron job at rollover.

    If everyone agrees this is a useful feature, I might code Gzip support myself and submit a patch. We can add support for archives with multiple files and different formats later, if anyone else wants to code it. For now, I want to keep this feature request simple.
    Last edited by Saklad5; 07-31-2018 at 04:22 PM.

  4. #14
    Senior Member
    Join Date
    Apr 2018
    Posts
    133

    Default

    I wrote and extensively tested a patch to implement support for individually compressed session logs. It’ll work the same for uncompressed files, while properly reading gzipped logs if they have the “.gz” extension instead of “.txt”. If both a compressed and uncompressed file exist, it will use the uncompressed file for now.

    Gzip Support.patch

    I feel it is ready to be implemented in the daily build.

  5. #15
    Senior Member Crowther's Avatar
    Join Date
    Nov 2006
    Posts
    1,404

    Default

    Itíll work the same for uncompressed files, while properly reading gzipped logs if they have the ď.gzĒ extension instead of ď.txtĒ.
    Originally Posted by Saklad5 View Post
    The normal convention in the UNIXish world it to have both extensions: file.txt.gz

  6. #16
    Senior Member
    Join Date
    Apr 2018
    Posts
    133

    Default

    Duly noted. I just changed the patch file accordingly.

    Gzip Support.patch

  7. #17
    Developer fronobulax's Avatar
    Join Date
    Feb 2009
    Location
    Central Virginia, USA
    Posts
    4,148

    Default

    Since you actually made the effort to compile and had success, I gave it a look. You did a few things that offended my sense of compartmentalization, primary introducing DataUtilities but after a couple tries to get around that I realized we either call DataUtilities or edit DataUtilities and FileUtilities. Given that choice your code didn't bother me as much. There were a couple things I did change so I would also like to test it. I'll do that ASAP but I might not be able until the weekend. But if it works for me, it will be in eventually.

  8. #18
    Senior Member
    Join Date
    Apr 2018
    Posts
    133

    Default

    Thanks for helping out, especially since it wonít help you personally.

    It is extremely trivial to add ZIP support for individual files (it works identically to gzip in Java), but handling archives is less straightforward. Youíd need to add support for nested folders, for a start. Ideally, everything would be broken into specific folders that may or may not be archives, so KoLmafia doesnít have to look through everything to determine something isnít there.

    For instance, there could be three valid locations for fronobulax_20180731.txt (or .txt.gz, etc):

    • sessions/
    • sessions/2018/
    • sessions/2018/07/

    Any level of folder could be an archive, and KoLmafia would decompress them as needed. If it doesnít find something, it wonít decompress a decade of log files before concluding it is missing. It will never check sessions/2017, for instance.
    Actually moving or writing things to those subfolders would be a breaking change, but looking in them if they exist would not be. And it sounds like many people are using this style of organization system already.

    Since DataUtilities and FileUtilities appear to be third-party libraries, I thought it would be a very bad idea to edit them. If we could edit those, it might be cleaner to actually change getReader() to look for compressed versions of any file it canít find. Or add another function there to do as such. Session logs are hardly the only thing that could benefit from compression, especially if we eventually decide to compress things when writing.

    I could tell DataUtilities wasnít really meant to be imported directly like that, but as you said, there wasnít really a better option. Besides duplicating that functionality wholesale, I suppose.

    Any other advice to keep in mind when contributing to KoLmafia in the future? I could certainly use the help.
    Last edited by Saklad5; 08-01-2018 at 01:17 AM.

  9. #19
    Developer
    Join Date
    Apr 2010
    Posts
    4,822

    Default

    Presumably tested on a range of O/S ?

    Personal opinion at present is that whilst it's useful to be able to unpack archived files (of one specific variant), it'd be more useful if the option to zip files automatically were also present (maybe zip session logs from before the current game day when Mafia runs it's new day code?)

  10. #20
    Senior Member
    Join Date
    Apr 2018
    Posts
    133

    Default

    fronobulax uses Windows, and I am running KoLmafia on both Ubuntu and macOS. So thatís fairly comprehensive.

    I donít see why the ability to compress files has to be added right now. It can be added later, though the variety of options and settings to choose from makes it harder to find the right balance for every system. And if we do want to add compression to KoLmafia, it would probably be better to do it in real-time, using GZIPOutputStream and such.

    Besides, we can implement any number of formats for reading, but we can only implement one for writing. Unless you want to add that many settings, that is. I think it would be best to let users automate that themselves for now, and just make sure KoLmafia can work with it. We can add ZIP next, and possibly import libraries for xz, 7zip, bzip2, and so on.

    If we add archive formats like ZIP and 7zip, we might need to add subfolder (and tar) support as well. It would be very confusing to only allow single files in those. The rest only support one file in the first place, so there isnít any potential for confusion there.
    Last edited by Saklad5; 08-01-2018 at 01:11 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •