collapse

* Arcadia Cabana

Refresh History
  • Brenna: Hey, Arcadians! I started a community on plug.dj for us. Think of it as a chatroom with a jukebox we can all take turns using. Whip up an account and let me know your account name (it's a private room).
    July 02, 2015, 04:14:52 PM
  • Brenna: I'm still here! Waiting... *hopeful*
    June 30, 2015, 02:26:07 PM
  • Edgewise: ;_; Thank you, very much.
    May 25, 2015, 08:27:28 PM
  • Alyss: I hope not!
    May 21, 2015, 03:32:13 PM
  • Edgewise: ...Please tell me this rpg hasn't been abandoned for the second time in six years. Please.
    May 18, 2015, 08:21:36 PM
  • Edgewise: Omg stay warm, everybody!  ;__;
    November 19, 2014, 01:00:04 PM
  • Ingrid: Maybe when they were kids XD
    October 19, 2014, 12:15:33 PM
  • Alyss: Phoebe wouldn't want Vaness to feel bad either, though, or dress up as something she really, really would rather not wear. Rapunzel and Sleeping Beauty have pretty dresses, so she'd be perfectly happy going as one of them. ^_^ (And the puppy dog eyes may end up on Roi to go as Flynn. XD)
    October 17, 2014, 01:04:57 PM
  • Alyss: She does - complete with a little bottom lip wibble. ^^;
    October 17, 2014, 12:59:37 PM
  • Ingrid: I bet Phoebe has the best puppy eyes
    October 17, 2014, 10:40:58 AM
  • Ingrid: Oh no ;_; Now Vaness feels bad
    October 17, 2014, 10:40:49 AM
  • Alyss: Phoebe would probably go as Rapunzel or Sleeping Beauty since they are both "hidden" princesses like her. (But only because Vaness probably won't go as Elsa to her Anna. XD)
    October 17, 2014, 10:35:10 AM
  • Brenna: Phia will dress as a butterfly, because its the last thing anyone will see coming out of the world's biggest tomboy.
    October 14, 2014, 10:54:38 PM
  • Brenna: Doc Lamya would go as the Ghost of Breakfasts Yet To Come, with a bacon and eggs print shroud.
    October 14, 2014, 10:53:31 PM
  • Ru: That is so awesome.  :D
    October 14, 2014, 06:06:48 PM
  • Pixie: Kivan would go as Liberace XD
    October 14, 2014, 05:50:22 AM
  • Ru: Kit I think might be a total geek and dress up as some historical figure her dad taught her about.  And then be a little confused that no one knows who she is.
    October 13, 2014, 06:52:34 PM
  • Ru: Hmm, I think Seren might go as a zombie, just to confuse the hell out of people.  She'd play it totally straight, and people would STILL find her attractive.
    October 13, 2014, 06:50:57 PM
  • Ingrid: Vaness goes as Flame Princess because of all the Elsa jokes
    October 13, 2014, 01:51:42 PM
  • Pixie: Pwyll would perversely go as his sister XD
    October 13, 2014, 07:05:13 AM
  • Brenna: Autumn Question Times! Cast Masquerade/Trick or Treating! What/Who/When is your character going as?
    October 11, 2014, 08:50:34 AM
  • Ru: Excellent, okay!  <3
    October 06, 2014, 06:36:01 PM
  • Brenna: Triso probably did. Seraita wasn't strong enough to swim, so she mostly waded.
    October 01, 2014, 01:59:40 PM
  • Ru: HEY so just a little question - for those of you with Lost Kids, do your kids know how to swim?
    September 25, 2014, 06:10:07 PM
  • Ingrid: Thank you <3
    September 23, 2014, 09:33:18 AM
  • Edgewise: More. Draw moarr
    September 20, 2014, 07:30:58 PM
  • * Edgewise is late to the party, as usual
    September 20, 2014, 07:30:31 PM
  • Brenna: Oh, how luscious!
    September 17, 2014, 04:28:14 PM
  • Alyss: Please do! :)
    September 15, 2014, 01:47:31 PM
  • Ingrid: Okay, mind if I improvise?
    September 15, 2014, 01:46:04 PM
  • Alyss: And I've been trying to find a good flower for Phoebe. I've come up with a few (white rose, lily of the valley, magnolia, etc), but haven't settled on one yet.
    September 15, 2014, 01:02:17 PM
  • Alyss: Pretty! :)
    September 15, 2014, 01:00:15 PM
  • Ingrid: I redrew Daki because reasons
    September 15, 2014, 11:58:03 AM
  • Ingrid: [link]
    September 15, 2014, 11:57:55 AM
  • Ingrid: Does Phoebe have a specific symbolic flower?
    September 15, 2014, 10:50:03 AM
  • Ingrid: Thank you :D
    September 14, 2014, 08:07:45 PM
  • Alyss: Yay for Ingrid arts! <3 They are awesome! ^^ <3
    September 14, 2014, 05:14:20 PM
  • Ingrid: Thank you <333
    September 14, 2014, 08:41:51 AM
  • Pixie: OMG those are awesome! <3
    September 14, 2014, 06:07:41 AM
  • Ru: And Daki is all sorts of gorgeous, and I love her hidey bangs.
    September 14, 2014, 05:03:13 AM

* User Info

 
 
Welcome, Guest. Please login or register.
Did you miss your activation email?

Author Topic: Review our review system!  (Read 64994 times)

Sage

  • Imperatrix
  • Posts: 94
  • Imperatrix
    • View Profile
Review our review system!
« on: December 17, 2013, 09:01:56 AM »
Hi ya'll!

So we've had about 4 apps go through our Google Forms based review process (half of them being Ru, so thanks to her for being a good sport).  I think this is a good time to ask for feedback on how you feel about the process, both as a reviewer and as someone who is being reviewed. 

Some sample questions, though feel free to comment on whatever you want:

- Is one week an appropriate turn-around time?  Is it enough time for reviewers to respond to an app, and is it a reasonable amount of time for an applicant to wait?

- Are the posted reviews effective?  Does it help to see the quotes?  Would reviewees rather see the whole review?

- Is Google Forms an appropriate template for reviews?  Are all of the questions appropriate? 

One change I will definitely make is making it so that reviewers can look over the reviews that other people have made so far.  I made it blind initially so that people won't be influenced by each other, but it seems like it will be far more valuable to reviewers to know what ground others have covered. 

As always, if you would rather keep your comments confidential, send me or Brenna a note!
« Last Edit: December 17, 2013, 09:03:35 AM by Sage »

Pixie

  • Posts: 18
    • View Profile
Re: Review our review system!
« Reply #1 on: December 17, 2013, 02:05:19 PM »
I for one really like the review system, having gone through it as both a reviewer and someone who's being reviewed. The peer review system really works here, I feel, though I agree it might be helpful to see what others have said in case there's something that got overlooked (and I do that all the time XD). To address specific questions:

- One week is a great turnaround time. It's not so long that it makes the person being reviewed sweat too much or feel like they've been forgotten, and it's enough time for everyone to get their reviews in. I'm happy with it on both sides of the equation.

- The posted reviews are way effective, and I like the quoted review excerpts -- they give an idea of how the character comes across and offer areas that need specific improvement.

- Google Forms is RAD. Very easy to use and edit!

In summation, TWO THUMBS UP. ^_^

Brenna

  • Mistress of the Tea Caddy
  • Countess of Mod
  • Posts: 41
  • Does this bustle make my butt look big?
    • View Profile
Re: Review our review system!
« Reply #2 on: December 18, 2013, 05:31:12 PM »
While I agree that a week is a good turnaround, it does make me worry about things like holidays, especially the long winter holidays.
"There are places out there where the skies are burning, the seas are asleep and the rivers dream. People made of smoke and cities made of song. Somewhere there's danger, somewhere there's adventure. Somewhere else, the tea is getting cold."
--The Doctor

Sage

  • Imperatrix
  • Posts: 94
  • Imperatrix
    • View Profile
Re: Review our review system!
« Reply #3 on: December 19, 2013, 09:12:24 AM »
While I agree that a week is a good turnaround, it does make me worry about things like holidays, especially the long winter holidays.

For example, maybe extending it to two weeks if there are reasons to believe reviewers might be delayed, like New Year's or a 3-day weekend?

Ru

  • Posts: 28
    • View Profile
Re: Review our review system!
« Reply #4 on: December 19, 2013, 07:51:42 PM »
Having been on both sides of the review system as well, I think it's working pretty well!  A week is a pretty good time for turning an app around, I think.  It keeps the applicant from having to wait too long, but it gives the reviewers a chance to respond.  I think stretching the turnaround time to two weeks for a holiday might be too much, though, because that's basically doubling the time for what may only be a couple of days of downtime.  That might require consideration on a case-by-case basis.

I actually kinda wonder if keeping the reviewers blind would be more helpful.  It does mean that reviewers will repeat things, but if everyone is saying there's x problem with the app, then that can help guide/prioritize suggestions for revisions.  If everyone tries to say something different, then you may not get that strength gauge, if that makes sense.

I like using Google forms, although I kind of wish there was a way to go back and call up your review, in case you want to make edits.  I know Google gives you the option to edit immediately after you've submitted, but it might be helpful to be able to recall it later in case you think of something else to add.  Is there a way to do that?

I think the posted reviews are very effective.  I definitely like the addition of quotes, as they give a stronger idea of how the character was received.

Overall, I'm liking this system!  <3

Sage

  • Imperatrix
  • Posts: 94
  • Imperatrix
    • View Profile
Re: Review our review system!
« Reply #5 on: December 20, 2013, 09:54:08 AM »
I like using Google forms, although I kind of wish there was a way to go back and call up your review, in case you want to make edits.  I know Google gives you the option to edit immediately after you've submitted, but it might be helpful to be able to recall it later in case you think of something else to add.  Is there a way to do that?

Okay, I'll have to go into the forms as a guest and see :D  I had thought there was a way to edit a few days after the fact, but maybe there is a trick to it, or it's not clear.  Even if I don't find a way, the review system isn't so strict that you can't jump in, write a new "review" and say that it is an extension of an earlier review, and then keep going with your new thoughts.

Alyss

  • Posts: 10
    • View Profile
Re: Review our review system!
« Reply #6 on: December 27, 2013, 05:53:56 PM »
So to hit the questions:

- One week does seem like a good turn around time. For holidays/long weekends maybe just add a caveat such as "due to the holiday, the review results may take an addition 3 [5?] [2?] days depending on the number of reviews received after the standard week review period."

- As a reviewer, I'd like to see the other reviews if possible to make sure I'm not missing anything and/or not being too lax/nit-picky in my comments. (So I'd like it more as a temperature check of sorts). ... As far as the reviewee seeing everything? I'm not as okay with that because seeing the whole review (instead of the quotes) might make it easy to figure out who wrote which review.

- Google forms is awesome, but like Ru said, I'd really like to be able to go back and re-read what I wrote and be able to edit it after submitting. Right now, if you don't save the link it gives you and you close the window, there is no way to get back to it.

- Another suggestion: maybe a day or two before the app review due date, have a chat session with everyone (except the reviewee, of course). That'll let folks who maybe don't have time to do a full review add some quick comments, as well as let the reviewers discuss questions they might have - for instance, for the Iron Angel apps, we can ask Sage how Sin might react to XYZ mentioned in the app to double check plausibility, etc. And(!) if the review verdicts come in at a tie maybe we can all get together and discuss it as well.

I do really like the system, though! :)

Kolamity

  • Chief Engineer
  • Posts: 10
    • View Profile
Re: Review our review system!
« Reply #7 on: January 06, 2014, 02:04:28 AM »
I have to admit, I was super nervous about the review system the first time I did a review because I was worried I was being too critical and harsh since I couldn't gauge the tone of the other reviews, but after that first one I feel a lot better and really like it :D.

And I do like the idea of doing a chat session, but considering how few chats I've been able to make it to, I think trying to wrangle everyone into a chatroom might not be the best option.

So as an alternative, perhaps keep the reviews private until the last day, then have it open for members to peruse to see if there's something they had a quibble with that no one else touched upon? There are times I don't have enough time to do a whole review, so something like that could be helpful, and it could help keep the balance between keeping things blind (with the benefit Ru mentioned, gauging the more problematic elements, still applicable) and making sure nothing gets missed.

Sage

  • Imperatrix
  • Posts: 94
  • Imperatrix
    • View Profile
Re: Review our review system!
« Reply #8 on: January 06, 2014, 09:32:32 AM »
Thanks for the feedback guys! 

So next time we have an app, we'll experiment by doing these things as well:

- For the first four-five days, reviews will be private (you cannot read other people's reviews).  On the fifth day, I will send a second PM/email that will open all of the responses for the other reviewers to read.  People are still free to submit new reviews throughout the whole 7 days.

- We'll try two things - we will have a chat on the fifth/sixth/seventh day to iron out differing opinions, and I will open a free blank G. doc where people who can't make it into the chat could throw in their opinions on decisive points.  The doc and chat are free for people who haven't written a review as well, so they can quickly throw in their POV.  I'll PM/email details about the chat and doc when I open the responses up. 

- I will research how to edit your response after you have submitted a review.  At worst, we may need to make it so people are logged into G. Drive to make a review, which I initially wanted to avoid cause I want be blind to the reviewer's identity too.

- For holiday extensions, let's play it by ear.  I think we are now past any holiday that would call for a 5-day extension, and the next big holiday I think would warrant an extension is Memorial Day weekend.  So if an app pops up around then, I can do a quick ask-around and see what people's availability is.   

Calling back to Alyss's points, I also prefer not to put a reviewer's whole review into the official review, so we will continue with my summaries and quotes. 

Also, I haven't said this explicitly, but so far, I have reserved the right to make the final verdict call in the case of a tie.  Hopefully by opening up a chat and a G. doc in the last few days of the review will help us avoid ties and get enough opinions to go for certain one way or another.   
« Last Edit: January 06, 2014, 10:07:04 AM by Sage »