xKiv
Active member
I have posted about this in the past already - when I "svn update", it takes a long time checking each script in turn, because I have many scripts. Often this ends up spending a minute or more just to say "nothing needed to be updated".
Updating in parallel would turn this into a few seconds, right?
But SVNKit as used by mafia isn't thread-safe. Well, we can always instantiate new SVNRepository for each thread ...
So I finally got off my lazy bum and coded a proof of concept.
This only parallelizes the initial "check which scripts' WC are at head", i.e. it splits the loop in SVNManager.doUpdate(), moving the check in a separate block (loop to create threads, then execute threads to see which scripts should be passed to the UpdateRunnable-executing loop).
Each thread gets its own instance of SVNRepository.
I didn't find an easy way to get new instances of SVNInfo, so I am calling SVNInfo.doInfo() sequentially. That shouldn't be a significant slowdown - the main slowdown is repeated waiting for internet replies.
I also put some slightly expanded diagnostic messages (in (my new class) CheckStatusRunnable):
1) I want to see the status of all scripts, not just the ones that are at-head
2) I want to see the name of the script for every updated script, even if SVN threw an exception. This is important, because mafia's current implementation SVN exceptions makes them seem to come from another script.
3) This should probably get a bit of reworking; perhaps put all messages in the a doReport() method, and call that sequentially from the main thread, in a known order (I currently print the messages when returning from the thread)
It *should* be possible to expand this so that the actual "update working copy from repository" part is also executed in the thread (parallelized). I didn't do this because
1) it's already in a different runnable, which is used in more places than this
2) I am not sure how to properly split-and-rework the code so that all the "fetch from repos to WCs" happens first, and *then* pushing *all* the updates at once (which has to be done sequentially, because it might push into the same file from different WCs, and might require users attention for each script separately).
(I can't rule out that it's as easy as new UpdateRunnable( f ).start(); at the end of CheckStatusRunnable, when the script should (and can) be updated)
My code is probably not mafia-standard-conforming, too.
So far, I have ran it once. It didn't destroy my svn directory. Then checking went quickly, and then it updated the three scripts that needed updating.
Patch attached:
View attachment 20160706-parallel-svn-update-wc-check.diff
Updating in parallel would turn this into a few seconds, right?
But SVNKit as used by mafia isn't thread-safe. Well, we can always instantiate new SVNRepository for each thread ...
So I finally got off my lazy bum and coded a proof of concept.
This only parallelizes the initial "check which scripts' WC are at head", i.e. it splits the loop in SVNManager.doUpdate(), moving the check in a separate block (loop to create threads, then execute threads to see which scripts should be passed to the UpdateRunnable-executing loop).
Each thread gets its own instance of SVNRepository.
I didn't find an easy way to get new instances of SVNInfo, so I am calling SVNInfo.doInfo() sequentially. That shouldn't be a significant slowdown - the main slowdown is repeated waiting for internet replies.
I also put some slightly expanded diagnostic messages (in (my new class) CheckStatusRunnable):
1) I want to see the status of all scripts, not just the ones that are at-head
2) I want to see the name of the script for every updated script, even if SVN threw an exception. This is important, because mafia's current implementation SVN exceptions makes them seem to come from another script.
3) This should probably get a bit of reworking; perhaps put all messages in the a doReport() method, and call that sequentially from the main thread, in a known order (I currently print the messages when returning from the thread)
It *should* be possible to expand this so that the actual "update working copy from repository" part is also executed in the thread (parallelized). I didn't do this because
1) it's already in a different runnable, which is used in more places than this
2) I am not sure how to properly split-and-rework the code so that all the "fetch from repos to WCs" happens first, and *then* pushing *all* the updates at once (which has to be done sequentially, because it might push into the same file from different WCs, and might require users attention for each script separately).
(I can't rule out that it's as easy as new UpdateRunnable( f ).start(); at the end of CheckStatusRunnable, when the script should (and can) be updated)
My code is probably not mafia-standard-conforming, too.
So far, I have ran it once. It didn't destroy my svn directory. Then checking went quickly, and then it updated the three scripts that needed updating.
Patch attached:
View attachment 20160706-parallel-svn-update-wc-check.diff