quick SVN primer (for scripters)

[roippi]

With mafia now supporting an internal svn client, some scripters are undoubtedly wondering what "svn" is, how it works, and what it can do.

What it is

Subversion ("svn") is a centralized version control system. A repository ("repo") exists at a url - this repo contains your project. It not only contains the files for your project, but every previous version of those files that's ever existed. Think of old versions like hitting ctrl-z on a Word document - you can revert changes all the way back to the start, or anywhere in between.

It is more than just a place to save files though. There is something called a Working Copy, which is a local version of a remote repository, plus any local changes you may have made. The process of creating this Working Copy is referred to as "checking out" the repo, or "checkout." You have probably also heard the term "commit", which is simply taking some/all local changes in a working copy and syncing them with the repo. Then other users sync their working copies with the repo, and voila!

How it works

A working copy contains a hidden folder, .svn, that contains all of the metadata needed to know how your local version differs from the repository's version. This is created for you when you checkout the project. You don't really need to know any of these details, really, for the most part using svn does not require knowing how it works under the hood. What you may be interested in is how to create your own repo.

I'll go over the easiest method under Windows - using TortoiseSVN. Under *nix you should probably just be using the command line "svn" client, though I imagine there are a million different graphical alternatives. There are alternatives in windows as well, I just prefer TortoiseSVN for its file system integration.

First, get a sourceforge account. Make a new project there. Download TortoiseSVN and install it. Make a new folder somewhere on your computer. Right-click, and choose SVN Checkout... from the context menu. You will want to give it the address that you see on your new sourceforge project code page - for now just grab the http:// address, it's easier. Ignore the commands in front of the address and after it, those are for command-line svn tools.

The rest of the default options are okay. Click OK and you should get a new working copy of your repo! Cool. Open it up and let's add some stuff. Add a new directory (call it whatever), within that directory add a folder called scripts/, and within scripts/ put a new text file. Now navigate back up to the root of your working copy, right click on the folder, and choose SVN commit... Tick all the files/folders (tick the "show unversioned files" box if it's not ticked), give it a quick commit comment up top. You should have to enter your sourceforge user information, save it if you want. If you've done everything correctly, your new project should have been committed.

Optional: to really play around with this, checkout a second working copy somewhere in another folder on your computer. Try committing changes from one working copy to the repo, then doing right-click SVN Update on the other working copy. Notice how TortoiseSVN helpfully changes the little file/folder icons for you when you make modifications to things. Hopefully you're starting to see how this whole thing works.

What it can do

svn is a very mature tool by now, so there's an absolute ton of stuff that I won't cover. You may for example notice that mafia itself has a "Latest SVN Changes" forum, which has .diffs of all the changes to files within mafia. There's also creating patches, merging repos, yada yada... Won't talk about it.

I'm more interested in talking about how mafia svn integration works. Doing svn checkout <repo> creates a working copy of the repo in your svn/ folder. Then, mafia "pushes" the contents of that working copy to the appropriate subfolders. For example, if a repo has a scripts/dostuff.ash file, that file gets copied to your scripts/ folder.

svn update goes through all your working copies and performs an update. There are other operations that can happen here. With a checkout, you can only be adding ("A") new files to a working copy. With an update you can update ("U"), merge ("G"), delete ("D") or also add ("A") files. Here, we warn the user whenever a file is added, due to security concerns. Note that "U"pdating only happens if the "pushed" local file still exists in the user's directory. I am bolding this because I'm sure it will confuse some people. This feature is a safeguard against svn re-installing scripts over and over that the user just wants to have deleted.

If you installed TortoiseSVN above and browse to the svn/ folder, you may notice something: the working copies checked out with svn checkout are fully-featured working copies, just like you checked out using TortoiseSVN. This means that you can make changes directly to this folder and commit them to the repo using TortoiseSVN - if you have repo commit permissions, of course.

And yeah. I meant this as a "quick" primer and it turned into a "moderately long" one, but them's the breaks.

 

[slyz]

Installment 1.5

I migrated a script to SVN, and I hit a couple of bumps I wanted to mention here:

Click OK and you should get a new working copy of your repo! Cool. Open it up and let's add some stuff. Add a new directory (call it whatever), within that directory add a folder called scripts/, and within scripts/ put a new text file.

Apparently the scripts/ folder has to be at the root. Putting it inside another directory led to the error "The requested repo failed validation. Complain to the script's author."

I think you should mention how dependencies.txt works too. It has to be at the root too (next to the scripts/ folder), and it should only contain the checkout URLs of the dependency projects, like this:

https://svn.code.sf.net/p/zlib/code
https://svn.code.sf.net/p/slyz-nemesis/code/

One last (very small) inconvenience was that I discovered after installing TortoiseSVN 1.8 that SVN 1.8 isn't backwards compatible. Mafia uses SVN 1.7, so you can't use TortoiseSVN 1.8 on Mafia's checked out copy of your script (the one in svn/).

The workaround is either (a) install TortoiseSVN 1.7 or (b) check out a separate copy of your project that you use for commits.

 

[roippi]

Installment 2: Branches

This is a condensed explanation of information found here. It's well written, I'd recommend reading it if you have time.

-----

Before We Begin

There are some things that are necessary to facilitate a good branch/tag setup. Setting up your repo in the right way is going to save you a lot of headache down the line.

Conventions dictate that in the top level of your repo, you want to create the following directories:

• trunk/
• branches/
• tags/

I won't go over what the difference between tags/ and branches/ is, they're basically the same thing. You can think of "tags" being like mafia's "point releases," if you're familiar with those. Not important, you don't really need to create tags.

The reason you want to have these folders in the root of your repo is that if you have users check out the root of your repo, there's no room to create separate branches - and with mafia's validation, even trying to do so will break things. So if you plan on using branches, at the very minimum make trunk/ and have the majority of your users checkout that, so you can later create branches/ if you want.

Some services like googlecode will even create these default folders for you when making a new repo. Like I said, it's just convention - you aren't forced to use these folders, but it's a good idea if you ever plan on making branches.

High Level Overview

Using branches simplifies your workflow when you want to work on a new feature. You can even have permanent development branches that periodically get merged into trunk. Also, there are space- and performance-related improvements when you create a branch. Physically, you are creating no new files on the repo server - you're just making symbolic links. If this is confusing or you want to read more, see the link at the top.

Sometimes you just want to develop a single feature in a separate branch, then when you're done merge it in and delete the feature branch. Note that with svn, data is never lost permanently - you can always resurrect deleted files and directories if you want.

Commands and TortoiseSVN

If you're using a CLI svn tool, you want to first use svn copy to make a new branch that is a copy of trunk/.

If you're using tortoiseSVN, have a checkout of trunk/ somewhere, then right click on it and do Branch/tag... The dialog is pretty straightforward. Up top, use the little repo browser to navigate to branches/ and give it a new folder name to create your branch. Give it a commit message and tell it you want to make the branch off of HEAD. If you want, tick the box in the lower left corner "switch working copy..." Like it says, this will turn your working copy into the new branch, instead of leaving it as trunk. Click ok and tortoiseSVN should make a new branch for you. Feel free to make as many random branches as you want - like I said above, there are no physical files being moved around, only symbolic links.

Now, you do all your development as usual in the new branch. When it's ready to go back into trunk, you right click on your trunk branch and choose Merge... There's a couple options of what you mean by "merge," and TortoiseSVN helpfully explains what they are, briefly. For your purposes, you are interested in the top two choices. If you want to keep the development branch alive, you choose the top option, "merge a range of revisions." If you're done with the development branch - say, you created it just to work on a single feature - you can choose the "reintegrate" option. Note that this won't actually delete the branch, it just will cause complications in the future if you continue to try to use the branch. Honestly the benefits of using the "reintegrate" option are pretty limited, and the downsides pretty extensive; for our purposes you can basically always use the top "merge a range of revisions" option.

Now, on the next screen, tell it that you want to merge your development branch into the working copy (trunk). SVN will go through and merge the differences from your branch into your working copy. Then if you're happy with the results, go ahead and commit the changes to trunk. If not, feel free to revert your local changes and try again. You can then go ahead and delete your dev branch if you want - or not.

Do you want this?

Some projects don't need anything other than a trunk branch. They can feel free to have users checkout the root of their repo and just not worry about it.

Others may want the flexibility to create a "beta" branch where advanced users can help them troubleshoot features before they go out to the masses. Or maybe you're just working on a particularly complex feature that deserves its own temporary branch - keeping your version history separate for that one feature can be useful when reviewing your code. Up to you!

 

[Winterbay]

A couple of questions:
1) Say you have already set up your repo, but not in accordance with these guidelines, and want to poke around with branches. Is there any way to do that without completely screwing it up for everyone who has already checked the script out?
2) If you have a branch and someone wants to help you test it, how do they switch between the different copies? I assume that you can check out the branch as well, but will that then get a new name in MAfia? And if not how does Mafia handle having two scripts with the same name from different branches of a repo?

 

[roippi]

A couple of questions:
1) Say you have already set up your repo, but not in accordance with these guidelines, and want to poke around with branches. Is there any way to do that without completely screwing it up for everyone who has already checked the script out?

If you're having users check out the root of your repo, not really. If they're checking out a folder, any folder, you can just pretend that is trunk/, but if they're checking out the root of the repo, not much you can do. Easiest way is to just make another repo and tell people to migrate to that one. Maybe have an informational line pop up in your script or something.

This is somewhat my fault that I didn't really educate people about the conventions ahead of time. The information is out there, so I don't feel too bad, but who reads the manual nowadays...

2) If you have a branch and someone wants to help you test it, how do they switch between the different copies? I assume that you can check out the branch as well, but will that then get a new name in MAfia? And if not how does Mafia handle having two scripts with the same name from different branches of a repo?

There is literally a command, "svn switch", that switches a working copy to a different location. If they're using an external svn tool, they can use that to just seamlessly move over to the dev branch. It's really straightforward to use.

Or they can just delete their project and checkout the new branch. There won't be any errors from checking out both at once, you just will get some.. goofy behavior. Since both trunk/ and your branch have the same files in them, whenever updates to either of them fire, those updates will update the local file.

example: you simultaneously checked out

svn/script1/trunk/scripts/myscript.ash
and
svn/script1/branches/dev/scripts/myscript.ash

Well, when updates to the dev/ version of the script fire, the dev/ version of myscript.ash gets pushed to your local copy. But if there are any updates to the trunk/ version, suddenly the trunk version of myscript.ash gets pushed to your local copy. Confusing. I would avoid this situation altogether and just make sure you only have one branch checked out at once.

 

[Theraze]

Sounds like if you want to simultaneously have two versions available, let's call them WHAM and WHAM beta for sake of example, you still do really need to have two repos set up and manually merge changes amongst the two.

 

[roippi]

Sounds like if you want to simultaneously have two versions available, let's call them WHAM and WHAM beta for sake of example, you still do really need to have two repos set up and manually merge changes amongst the two.

I.. what? No?

 

[Theraze]

In your example, you have two versions of the same script with the same name. You can have one or the other as the currently available one, but not both in the active folder currently. To keep two versions both available, for example so that you can do:

[default]
consult WHAM beta.ash
consult WHAM.ash

You'd need to have two different scripts. Or maybe it's possible, but your example would have WHAM.ash in either beta or release format, not both available simultaneously...

 

[roippi]

(can people get away from using spaces in directory/file names? It's technically not breaking anything here but it's really bad form.)

(and can we all use the same nomenclature? "active folder" means nothing to me. Do you mean working copy? Local copy?)

Your issue has nothing to do with keeping things in two repos. If you kept them in two repos and left them as the same name, you would still get the same issue. If you made a dev branch in the same repo, you can rename the file to whatever you want. However...

You can rename files when you make a dev branch; it's just not a good idea if you ever want to use merge, it creates a lot of issues. This is a limitation of svn - when you rename a file, it is the equivalent of deleting the old file and making a new one. So when you merge, some issues arise. I've not done it myself but I know that you're going to have to do some work to make it happen. Merging back into trunk would probably cause your file to be renamed. So I'd just.. not rename things. That is trying to do version control yourself by making multiple copies, you should be doing the VC through svn.

Perhaps I should have included "svn switch" in my explanation. It is sometimes useful to switch back and forth between a dev branch and trunk, and that is the right way to do it, not doing it with renamed files. If mafia needs the facilities to handle pushing the switched working copy, we can talk about adding those.

 

[roippi]

Bale - I stuck your thread and changed the title of this one. I didn't really have a target audience in mind for this thread, but it's definitely more for scripters.

 

[slyz]

I finally migrated a script to SVN, and I hit a couple of bumps I wanted to mention here:

Click OK and you should get a new working copy of your repo! Cool. Open it up and let's add some stuff. Add a new directory (call it whatever), within that directory add a folder called scripts/, and within scripts/ put a new text file.

Apparently the scripts/ folder has to be at the root. Putting it inside another directory led to the error "The requested repo failed validation. Complain to the script's author."

I think you should mention how dependencies.txt works too. It has to be at the root too (next to the scripts/ folder), and it should only contain the checkout URLs of the dependency projects, like this:

https://svn.code.sf.net/p/zlib/code
https://svn.code.sf.net/p/slyz-nemesis/code/

One last (very small) inconvenience was that I discovered after installing TortoiseSVN 1.8 that SVN 1.8 isn't backwards compatible. Mafia uses SVN 1.7, so you can't use TortoiseSVN 1.8 on Mafia's checked out copy of your script (the one in svn/).

The workaround is either (a) install TortoiseSVN 1.7 or (b) check out a separate copy of your project that you use for commits.

 

[roippi]

Apparently the scripts/ folder has to be at the root.

Not at the root of your repo, just at the root of the directory that you specify to checkout. You could have a project like

https://svn.code.sf.net/p/my_project/folder1/folder2/folder3/scripts

And as long as you specify that the url to check out is https://svn.code.sf.net/p/my_project/folder1/folder2/folder3/ it will pass validation.

I'm aware of the issue with SVN 1.8 and it frustrates me greatly. I think I might just upgrade the SVNkit sources, but I think that will make 1.7 clients unusable. It's a mess.

 

[roippi]

Actually it looks like svnkit doesn't yet have a stable version that supports 1.8, so I won't be doing that until they do.

 

[fronobulax]

Depending upon which computer I have 1.6, 1.7 and 1.8 repositories. It is a PITA but an unfortunate consequence of IDEs that don't stay up to date and licenses that only apply to my work computer.

I updated DCQuest by copying the appropriate KoL svn directory elsewhere, updating just the DCQuest svn to 1.8, doing my commit from that with a 1.8 client and then replacing the upgraded directory with the 1.7 copy that had been squirreled away.

 

[ungawa]

First, get a sourceforge account. Make a new project there. Download TortoiseSVN and install it. Make a new folder somewhere on your computer. Right-click, and choose SVN Checkout... from the context menu. You will want to give it the address that you see on your new sourceforge project code page - for now just grab the http:// address, it's easier. Ignore the commands in front of the address and after it, those are for command-line svn tools.

The rest of the default options are okay. Click OK and you should get a new working copy of your repo! Cool.

Hey, I'm trying to get my first script here set up with SVN, and so far I'm not having luck. I'm not sure I'm doing the "Make a new project there" step right. Does any one have more info? Also, it doesn't seem to be easy to change the initial settings from when I created a project or to delete a sourceforge project, and at this point I feel like I'm spamming them with new projects as I try to figure out the correct settings, so I figured it was time to ask.

Attempt 1: Project name CartoucheHunter
On the "Create Your Project Now" page, I saw an SVN checkbox and figured that had to be a good thing to check. I was able to use TortoiseSVN SVN Checkout to get my ash file into the sourceforge project, but when I tried to run "svn checkout https://svn.code.sf.net/p/cartouchehunter/svn/" in mafia, it said "The requested repo failed validation. Complain to the script's author." At that point, I noticed my https:// address is /svn instead of /code, which doesn't match the rest of the scripts here.

Attempt 2: Project name KoL_CartoucheHunter
I tried creating a new project without "SVN" checked. This one seemed better at first since there was a "Code" tab. However, when I tried to use TortoiseSVN to "Right click->SVN Checkout" on a new folder on my PC, after entering my sourceforge login, it gives an error: "Unable to connect to a repository at URL 'https://ungawa26@git.code.sf.net/p/kolcartouchehunter/code' 'p/kolcartouchehunter/code' path not found"
Poking around these forums, I saw Bale said "git is not an option for KoLmafia. It can only handle SVN". So I checked and saw that the "Git" checkbox is checked by default when creating a sourceforge project.

Attempt 3: Project name LoathingCartoucheHunter
I unchecked "Git" on the project creation page. Well crap, the resulting project doesn't have a "Code" tab.

Any thoughts?

 

[xKiv]

Attempt 1: Project name CartoucheHunter
On the "Create Your Project Now" page, I saw an SVN checkbox and figured that had to be a good thing to check. I was able to use TortoiseSVN SVN Checkout to get my ash file into the sourceforge project, but when I tried to run "svn checkout https://svn.code.sf.net/p/cartouchehunter/svn/" in mafia, it said "The requested repo failed validation. Complain to the script's author." At that point, I noticed my https:// address is /svn instead of /code, which doesn't match the rest of the scripts here.

I think this looks best of the three, but ...
1) https://svn.code.sf.net/p/cartouchehunter/svn/ contains one directory, and that directory is not scripts/; it should contain scripts/ directly and not in a subdirectory.
Alternatively, your checkou should be from "https://svn.code.sf.net/p/cartouchehunter/svn/ cartouchehunter-svn/", which leads to
2) the name of that directory, " cartouchehunter-svn/", starts with a space. Don't do that. Ever. Spaces in names of files and directories are not always guaranteed to be handled correctly, and they are especially evil at start or end, where some software could decide to just trim them away.
3) you should probably not have "cartouchehunter-svn/" in the same place where you have "scripts/". That looks like you tried to rename " cartouchehunter-svn/" to "cartouchehunter-svn/", but instead created the second in the first.

 

[ungawa]

the name of that directory, " cartouchehunter-svn/", starts with a space. Don't do that. Ever.

Ah, yea, that was a combo of me being unobservant and TortoiseSVN automatically creating folders. When I copied the https:// address from sourceforge, I removed some command line stuff from the start, but didn't notice the " cartouchehunter-svn" after, so TortoiseSVN interpreted that as a folder name.
It looks like I got the repository cleaned up, and I was able to "svn checkout https://svn.code.sf.net/p/cartouchehunter/svn/".

However, instead of making a "cartouchehunter" folder in mafia's SVN folder, it named the folder "72fd6657-9ca6-454e-98b5-a35926876534". During mafia's svn checkout, it also asked if I wanted to overwrite the script in my TortoiseSVN checkout development folder. I'm able to run CartoucheHunter from the CLI, but I'd like to figure out that folder name. I'm not sure if it's just doing that for me due to some interaction with the TortoiseSVN checkout development folder, or what.
Edit: I asked my bf to try downloading it and it had the same/similar junk folder name on his comupter. That got me thinking some more, and I guess I don't know how mafia picks the folder name. Is it parsing the https:// string, and since mine has "/svn" instead of "/code" it's getting thrown off?

 

[xKiv]

During mafia's svn checkout, it also asked if I wanted to overwrite the script in my TortoiseSVN checkout development folder. I'm able to run CartoucheHunter from the CLI, but I'd like to figure out that folder name. I'm not sure if it's just doing that for me due to some interaction with the TortoiseSVN checkout development folder, or what.

Is your development folder *inside* mafia's folder? I don't think that's supposed to work.
Either develop in a completely separate place (and copy files around), or inside .kolmafia/svn/, and .. there should be a command to automatically synchronize from there to "live" scripts ... "svn sync"?

However, instead of making a "cartouchehunter" folder in mafia's SVN folder, it named the folder "72fd6657-9ca6-454e-98b5-a35926876534".

That's problably the svn/ instead of code/ bit, I think I saw a request for mafia to recognize that.

 

[ungawa]

Is your development folder *inside* mafia's folder? I don't think that's supposed to work.

That's problably the svn/ instead of code/ bit, I think I saw a request for mafia to recognize that.

Thanks. Yea, moving my development folder out fixed the issue with mafia trying to overwrite it at checkout.

OK, cool. I'll live with the goofy folder name for now I guess. Thanks!

 

[Bale]

Someone was recently confused by a lack of updates in this thread. I am posting this old-ish news so that important information is up to date.

Actually it looks like svnkit doesn't yet have a stable version that supports 1.8, so I won't be doing that until they do.

That is no longer true. Heeheehee updated KoLmafia to support svn 1.8 in revisions 16072-16074 last year.