Beta testers

[Winterbay]

Since the thread in which this came up has a slight emotional tendency I thought I'd start a new thread for a discussion on this theme.

As slyz stated in the post linked above a group of beta testers, people experienced in the way of Mafia and willing to perhaps sacrifice some optimality to help out the devs in their testing. I think this is a great idea and it would be a potentially good thing for the community in that it could get more involved and possibly iron out minor bugs in changes before they hit a wider audience. I have however no idea how such a thing would be set up, but that's one of the things that this discussion can possibly help out with

 

[roippi]

I think this is a great idea. I believe the technical implementation involves forking a branch off the mainline trunk, and occasionally merging changes back in. Or some other technobabble that sounds like that.

Testers would need to be able to compile their own code off of sourceforge, and know how to create and apply patches. This is actually easier than it seems, especially once you get it up and running. You would help the devs out by testing out "experimental" code to both provide bug reports ("hey, this breaks this other thing"), or user feedback ("you could implement this GUI element a bit differently"). This simultaneously relieves a lot of the pressure of providing 100% perfect bulletproof code when we commit things, and also provides another set of eyes on upcoming features to head off possible user complaints.

I also think this is a great entry point for anyone trying to learn to code in Java, or experienced coders who want to code for mafia but can't wrap their heads around the codebase.

 

[slyz]

I don't know much about code repositories, but I remember from a conversation with a friend of mine that SVN isn't the most suited to maintain official branches of a program (he mentioned GIT, if I remember correctly). It would be great if people with more experience with this sort of thing could propose a simple way of making something like what I had in mind happen.

My idea was to provide an official "experimental" or "beta" release, available to everyone. If possible, from http://builds.kolmafia.us/.

I think that simply downloading KoLmafia-Experimental-9679.jar instead of KoLmafia-9679.jar would make the user aware that he really is beta-testing and is expected to report his impressions on a specific issue. It would provide a way for devs to gather more feedback before propagating a change to the rest of the user base.

Admittedly, this is in part a way to cut down on the nonconstructive criticism by giving an easy way to include more users in the development process. It would also be a way for me, a very inexperienced programmer, to have people test half-assed changes so I can weed out all the unexpected consequences before committing a change.

 

[fronobulax]

I think this thread misses the point. The problem with KoLmafia as a Project is not that things get committed and are subsequently found to be buggy or that there is not enough testing. It is that there is no clear process whereby someone in authority determines which requests get worked on and subsequently make it into the code base and which ones do not. At present if someone has commit access then their code makes it in. Anyone with commit access can undo someone elses work. The triumvirate of hola, veracity and jason implicitly keeps order because of the implied threat to remove the commit access from an individual who misbehaves. Devs who are looking for something to do either monitor the forums or just run with a good idea.

The problem is that monitoring the forums often exposes people to genuine Bad Ideas as well as good ideas and legitimate concerns that are poorly phrased or incomplete and that leads to the current situation. What seemed like a good idea to the dev whom implemented it might not seem like a good idea to the rest of the community.

If I was given KoLmafia to manage and wished to correct the situation with the lowest impact I would do two things. I would create and manage an "approved request" list. I would then require that all code committed contain a reference to one or more items on the approved request list. This being the case, devs who only wished to deal with code and not the user base would only have to scan the approved request list for something to do. Devs who had a good idea would not commit it without some community discussion or awareness (i.e "approval").

The biggest difference between the above idea and the status quo is that anyone with post privileges at kolmafia.us can make or comment on a feature request and the approval process is ad hoc and highly dependent upon which devs are interested in working on the feature.

So rather than enlist testers, I might enlist Mighty Minions who would be willing to "approve" requests and repost them in a controlled location. That could be a locked thread here or perhaps at the dev forums veracity maintains. The minions would only repost a FR if in their opinion it was something that needed to be done, wouldn't hurt, was best done in mafia as opposed to scripts and was detailed enough to be worked on without further questions.

 

[slyz]

I feel like the way devs choose what changes to work on functions well, mainly because of the voluntary aspect of the whole thing, and the issue I wanted to address was the way user feedback on some changes was expressed and/or interpreted.

Would filtering the requests help with this?

 

[Raijinili]

Can I request some context? I haven't really seen the "This change sucks" posts offsite. Is it on the Mafia thread in the forum?

 

[Winterbay]

The thread that sparked the problem is in the SVN-part of the forum in the discussion on the colouration of food in the item manager.

 

[lostcalpolydude]

Can I request some context? I haven't really seen the "This change sucks" posts offsite. Is it on the Mafia thread in the forum?

Search for the word "annoying" on probably any page of the mafia thread on the main KoL forum.

 

[Raijinili]

It seems there that there is a general sense of disconnect by the forum community from the development community. As in, because to them the Mafia developers are this "other", they're less likely to be treated with the same respect as a "real" person. It's easier to blah blah annoying if you can't attach names to the people working on it.

What are the thoughts on Bugzilla? I mean, as a way to report bugs without having to log in, not as a beta testing system.

 

[Winterbay]

The main problem with anonymous postings of any kind (and by that I mean where you can post as a general anonymous account) is that people tend to be even more dicks than they are otherwise...

 

[Raijinili]

What if it was sent through Kmail and posted to by a bot?

>>>
@5628
I don't think this would help me.
<<<
would have the bot find post 5628 and respond to it.

There may be a need for a thirty-day limit: if your account is less than thirty days old, your message will have to be approved.

However, it removes the barrier of registration.

 

[fronobulax]

What if it was sent through Kmail and posted to by a bot?

>>>
@5628
I don't think this would help me.
<<<
would have the bot find post 5628 and respond to it.

There may be a need for a thirty-day limit: if your account is less than thirty days old, your message will have to be approved.

However, it removes the barrier of registration.

More work than the problem is worth and an especially bad example to boot. Part of the problem, as I see it, is that when someone just says "I don't think this would help me" the comment is less than helpful. First, it puts people in attack or defense mode depending upon what they personally have invested in the feature and second, it contributes nothing to a discussion that would accept, reject or improve the proposed feature based upon its merits. Given the maturity level of much of the user community, a simple response of "Why?" would probably cause the original responder to become annoyed.

I don't think anonymity is necessarily a problem but the quality of the dialog certainly is.

 

[Raijinili]

Hey, not every example is excellent. Or serious.

I was simply giving a format.

 

[slyz]

I don't think anonymity is necessarily a problem but the quality of the dialog certainly is.

That's what I had in mind. Hence the proposal to find a way to get more people to test changes before they are experienced by all the regular users.

What I wanted was a practical way for people to download from http://builds.kolmafia.us/ and use a differently-labeled build that has a specific change, knowing that the change is there to be tested.

I hope this improves both the quality of the feedback, and that making things simple for the users will make a greater number of users want to try out these changes.

 

[Rinn]

Yeah you pretty much want to use features in git. The main issue here is a lot of people are expecting the daily builds to be safe, but they've never been guaranteed to be. I don't like the idea of an additional experimental build, the daily builds are already experimental.

 

[matt.chugg]

I don't like the idea of an additional experimental build, the daily builds are already experimental.

Agreed,

The way I see it is to restrict access to daily builds to beta testers, as big or as small a group as required, (and to people who are happy to build it themselves) and spin a more regular rtw, maybe every couple of weeks (or if something big happens in-game that requires a new release) giving the developers (who do this for free, in their spare time, and are all awesome) chance to respond to well thought out and appropriate comments, suggestions and requests before releasing changes to a wider community. With perhaps (insert credit to whoevers idea this was earlier in the thread) a group of minions to keep an eye on the bug/request forum and filter the information that gets to the developer level.

I'm willing to provide server/vps space/access, time, and rubber emo roes to keep the development process as pleasurable as possible for the people who have made this such an amazingly useful tool. I think I can safely say, that without kolmafia, I wouldn't have been able to enjoy this game for so long, even with the awesome content, and wouldn't have just received my 5th year anniversary gift box.

I'm sure i've done my part in making things difficult, filing badly written bug reports, unclear feature requests, and I hope these instances were not camel back breaking and straw like.

These are just my thoughts, they arn't necesarily right or wrong, just another line of discussion.

 

[roippi]

I think this thread misses the point. The problem with KoLmafia as a Project is not that things get committed and are subsequently found to be buggy or that there is not enough testing. It is that there is no clear process whereby someone in authority determines which requests get worked on and subsequently make it into the code base and which ones do not.

I respectfully disagree, at least on defining what The Biggest Problem is. To use an off-the-cuff example, having a designated someone greenlight the colorize food feature would result in no difference from the current system. A dev would pick up the ticket, implement it the way the dev thinks is best, and commit it. And a day later there's complaining. This is just one example, but I see where we differ: you believe that the problem lies in efficiently distributing jobs to the devs, and I believe the problem is effectively distributing features to the userbase.

Still, I do agree with much of your post. If there were a dedicated highly moderated (or yes, locked down) subforum for Official Requests, that would unarguably be good. I do think that anyone with commit access should be able to commit things that aren't on The List, though. The speed at which mafia is able to organically respond to changes is an amazing thing, and I wouldn't want to add any bureaucracy to slow that down.

 

[Theraze]

---

Regardless of whether you have items get tested by the developer or their 5-10 dedicated beta testers, people will still whine and complain. That's part of having an awesome, popular program... those who can't code, complain.

The question almost seems to be more, what happens when those complaints are made?
1) Do we immediately jump to try to make the code more in line with what the complaintant envisions? This will, of course, make other people complain about the new changes, but addresses the first problem at least.
2) Do we have a different forum for change comments, where people can say what they don't like about the changes and try to convince the developers that they should make changes? I'd suggest a different forum as opposed to adding this to Bug/FReq/NewContent, as those are likely to be less aggravating for the developers. If this did happen, anything that's an opinion on whether or not a change should happen should go there. It can be discussed, and if rejected, the thread can be locked. If approved by a minion or developer, a new FReq or Bug thread can be made on that forum... but at least the developers don't need the overwhelming flow of negativity, if we go with this strategy.
3) Do we continue with what we currently have, which is that sometimes complaints happen in the commit post, sometimes they happen as bugs, and sometimes they just scatter all over the place? The benefit is ease... it's what we currently do, and if people feel negative they can surely find SOMEwhere to put it. Problem is that it doesn't really make the developers feel appreciated, no matter how many "Devs, you're awesome!" threads we make.

But yeah, the problem as I see it is more one of collecting the feedback... the benefit of the current system is that we have a lot of people who can provide feedback in a constructive way leading to the issues getting quickly fixed. The negative of the current system is that we also have a lot of people, most of whom generally provide good feedback, who sometimes get caught in the "doesn't work" type of bug posts which doesn't actually help in solving the problem and which doesn't make you feel valued.

 

[fronobulax]

To use an off-the-cuff example, having a designated someone greenlight the colorize food feature would result in no difference from the current system.

Actually that's probably a bad example precisely because that is something I would not have approved if asked. (I did comment once I realized a 10 month old feature request had been worked on and implemented). The reason is that I have seen this before and I would certainly have asked about the impact on the visually impaired before I approved such a change. Ditto for things that fall under the general heading of internationalization. I have seen over the years of mafia development, cases where something was checked in that should not have been for any number of reasons. Whether that has happened (or will happen) enough to impose a process is debatable although I note that veracity was (and may choose to continue to be) the de facto approver.

A minor clarification - my comments were not based upon parceling out work to devs as much as they were based upon having a good answer based upon multiple opinions to the question - are things better if this feature is implemented?

 

[fronobulax]

1) veracity has not withdrawn from KoL and KoLmafia activities, just the communities associated with some of them. Thus if we can tap her wisdom and experience by giving her a "safe" place for BR/FR where she won't have to hear something called "annoying" that would be A Good Thing. An added benefit would be the community consensus implied by the Minion enabled questions and filtering.

2) There are always going to be whiners and many of them won't know there is something to whine about until they upgrade their version of mafia. If people believe that restricting daily builds to vetted individuals who are willing to participate in a feedback process will reduce the whining then I'm all for it. But I am concerned about the whining, of a different sort, that will occur when KoL changes something and people perceive that mafia is "broken" until support for the new content or feature is available.

3) As long as TPTB release ITOMs we can pretty much be assured that mafia will need monthly releases. If a case is made for more frequent releases then we need to make sure that someone besides veracity can spin them and post them to SourceForge.