The Alchemy of a Badge

We have been rolling out a gazillion things in the past month here at Mozilla, with the Summer Code Party campaign kicking off and it has been a while since I blogged here. Well this week it was time for a retreat as I traveled up to Maine to work with Erin on badges and assessments.

We took as a basis all the wonderful web literacy work Doug Belshaw and Michelle Levesque have been doing, as well as everything that  we have learned so far from people around the world using our tools. What we tried to do is develop an assessment system that will allow us to issue badges within the next months across our tools.

The process involved a lot of post-it and marker alchemy and for that reason I have named this post “The Alchemy of a Badge”. (which is probably me being biased as a game designer and somewhat inspired by playing too much of Magic The Gathering and Skyrim.)

In plain English that means I will cover some first ideas about assessment.

The Alchemy of a Badge

Alchemy is an art similar to chemistry or cooking, that involves mixing elements together to create potions. These potions might even result to some kind of inner revelation or even an enlightenment of some sort.

I have similarly imagined the art of creating a badge; There are many different elements you include in making a badge , many different ways you mix them all together and at the same time the whole process is part of something bigger, a system of other badges if you will. At the end you have discovered something new about yourself.

In this way badges become meaningful and valuable objects that demonstrate a rich learning experience and are worth pursuing and showcasing.

Poetic thoughts aside, it is time to directly dive into the types of assessments we are considering to create the alchemy of a badge.

1) Granular Skills Badges; These badges are assessed within tools and connect to specific skills i.e. in Thimble a learner can replace existing code, for example change the links or text within the following <img> tag, to get the Image Wrangler Badge.

The goal here is to get feedback as you go and keep you motivated. The metaphor we used with Erin when discussing this type of badge was linked to running, (it was no accident that we were actually running while we were having this discussion). Often when you run, you might feel the need to get cheered up when you have passed a high hill or you are half way to the end line. Well imagine these badges acting in a similar fashion. They are there to make sure you know what you are doing each step of the way and that you push forward.

Here are some ideas for granular badges, based on the embedded assessments we want to build in Thimble (names are placeholders).

2) Peer badges: For this first phase these badges are assessed by mentors and peers who give endorsements, ratings against rubrics and provide suggestions for improvements. For the second phase we are imagining granular badges being part of peer badges, for example when unlocking the Image, Text and Media Wrangler from above you can pledge for a bigger badge such as the HTML Tycoon.

A key element for the success of peer badges is to have mentorship baked in our platform, a great example being Army of Awesome. (this is something that I plan on writing more about soon.)

Below are some ideas for peer badges, based on Doug Belshaw’s latest list as it can be found here. (names are placeholders)

AUTHORING (Owl)

One.More.Time. Playtesting the Mozilla Webmaker Projects

I am a ____________ Webmaker, and other badges for the Summer Code Party

After last week’s sneak peek into the Web Arcade Projects ,that we have been busy prototyping with all the Learning & Tech Mozillians, the time has come for a sneak peek into the specific Badges for Mozilla’s Summer Campaign.

During the Summer Campaign we are expecting people everywhere who are joining the Summer Code Party to MEET up to MAKE something cool and LEARN how the code.

The badges suggested in the paragraphs below aim to seed a community of practice, celebrate creativity on the open web and showcase the different types and (ever changing) roles one can take upon as a webmaker.  (more on the badge system -and Caine’s arcade- at Carla’s blog)

The Webmaker projects will support a suite of digital and analog badges. In this blog post we will talk about some of the digital badges, which vary from badges awarded by the system when you publish your remixed webpage, to badges that you can email your peers, to badges that you can award to yourself. All badges will be using the Openbadger tools.

I will start with this last one type; a badge that you award to your self. I know this might sound a little bit strange; why would I want to give my self a badge? One could say that sounds like cheating or take it even further and call it straight up self-centered. But bear with me as I give a little bit of background.

The idea came about from talking to friends about what it means to be a “webmaker.” People would get mixed up when I told them that webmakers are not necessarily people who make things with technology, programmers if you will. For me, I argued, being a webmaker is a way of looking at things and an identity that we take upon when being active citizens of the web, and yes it might also mean that you get to learn how to code a bit. (Michelle Levesque who is THE expert has many answers about what it means to be a webmaker on her blog)

So what if I thought there was a self-assessment badge, a badge that called you to fill in the sentence I am a _______ webmaker, a way for you to reflect on your ever evolving identity as an active citizen of the web. 

In classic “play-testing” mode I asked a few friends that were in the Mozilla London offices today, to fill in the sentence above and here is what they came up with; multifaceted webmaker, creative webmaker, novice webmaker, webmaker TROUBLE, girl webmaker.

Enter “The Web Arcade”; teaching basic webmaking skills to teens

A few weeks ago I wrote a blog post about why learning on the web should look like the web. In the meantime I joined Mozilla’s Learning Team and together we have been working on something called “The Web Arcade” part of Mozilla’s Summer Code Party Campaign. The Web Arcade includes a set of projects aimed to youth, that can help anyone get started with HTML & CSS by remixing playful pages full of well… web goodness. 

The tool we are using for “The Web Arcade” is called Webmaker (read Jess Klein’s blog posts to find out more) and it has been build upon @iamjessklein and @toolness    Lovebomb, and is basically a brilliant editor and renderer all in one screen, that allows you to remix and share your web masterpieces with the world. 

When we originally perceived “The Web Arcade” with Jess Klein a few weeks back we imagined it as a series of mini games, what we later came about to calling “projects” specifically designed to scaffold the experience of learning how to code a webpage for the first time. 

Concept Art, in progress

The projects in “The Web Arcade” are designed to provide scaffolded yet open ended learning experiences; learners begin by hacking broken webpages to master basic HTML/CSS coding skills. As they progress in the Web Arcade they start learning about bits of code such as tags, elements and attributes. As they piece those bits of code together they have the choice of moving into more advanced projects, all while hacking webpages to create new ones, constructing theories of how the Open Web works and taking on the role of webmakers. 

Below is a rough sketch of how the system works:

Following this sketch, we started building different projects in the Web Arcade. Here is a sneak preview of what is currently work in progress: (for more projects you can also check out Laura & Jess fantastic blog posts) 

Welcoming Message:

“Enter the Web Arcade to explore the world of the Open Web ; a place made out of serious cats, planking videos and minecraft  empires! The Web Arcade offers a collection of mini games that help you level up your HTML & CSS skills and MAKE your very own web masterpieces.”

1. Project: HTML Card Game
   A creative and strategic card game that blends friendly competition with learning the building blocks of the web! Download and print a stack of game cards to get started.  Gameplay occurs as you strategically mix and match cards your hand to create unique snippets of code and defeat other players’ hands. May the best webmaker win! Learning Objectives: HTML syntax
    
2. Project: Map of the Web Arcade!
   This is the first project in the Web Arcade, a Map to guide you through your adventures. Not all the locations on the map though have a name and certain things look broken! Uh-Oh! Your goal in this project is to activate the map and even add your own flair to it. Get familiar with the syntax behind HTML by closing any open tags <> </> or fixing broken links that are missing a ” or two “”. Upon activating the map you reveal certain links that show you new projects. 
   Learning Objectives: <>, </>, what is a tag, how to open and close tags, what is an element.
    
   The Map Missing closing tags and links 
    
   Map of the Arcade ready to be edited in the Webmaker tool
   
   Map of the Arcade with fixed links
   
   Map of the Arcade hacked and personalized
    
3. Project: Passport-O-Matic: A unique blend of HTML agility,CSS and firepower. YOU are no ordinary character. Your goal in this project is to MAKE your undercover passport in order to explore the arcade defeating every challenge with your self-esteem and special powers. Learning Objectives: <title> <head> <body> <p> <img> <align>, src, privacy , CSS
    
4. Project:  The Bread Cat Bakery: The Bread Cat Bakery is throwing a party. Unfortunately the only attendees are undercover Serious Cats. Bummer! Your goal in this game is to change the <img> and <a> tags to include whoever YOU think belongs at  this party. Did someone say Brony? Learn about attributes such as src to make sure there is no undercover surprises this time around. Learning objectives: linking, <a href=”..”> </a>, src attributes, elements
    
5. Project: Open Webville: The Fantastic Open WebVille isn’t just your run-of-the-mill “Ville”. You see, it’s managed by Madame Lille a recent alumnus from the School of Farming, a former politician who is relaunching her career in a new direction so she can focus more on velvet bird painting and organic producing. In this project Madame Lille has started putting together a guide book for Open WebVille. Use your superstylin’ ways and HTML dexterity to hack her guidebook and help her fill it in. Learning objectives: what is the Open Web, <a href=”..”> </a>, src attributes, elements, <title> <head> <body> <p> <img> <align>, CSS.
    
6. Advanced Project: Internet Famous: Use your HTML and CSS swagger to mix and match elements to make a webpage featuring a meme to conquer all internet memes. Let your inner serious cat or Ryan Gosling hey girl shine  and claim your Webmaking powers by posting a video of how you did it. Learning objectives: synthesizing all previous HTML &CSS, design thinking and open web skills.
   
   Internet Famous Project ready to be hacked and become famous
   
   
   Internet Famous Project in the webmaker tool
    
7. Advanced Project: Web-structable. Have you always wondered “why people don’t get hand turkeys right? or why it is hard for some to sleep at night? (rhyme)” Well the time has come for you to make your contribution to the world of the open web and create a fine webpage tutorial using your HTML & CSS wisdom. Learning objectives: synthesizing all previous HTML &CSS, design thinking and open web skills.
   
   Web-structable Project ready to be hacked
    
   Web-structable Project in the Webmaker Tool
   
   Well that is where we are for now, I am imagining there will be plenty of iteration going on as these mockups become working prototypes, as we come up with badges and as we keep testing our work in the following weeks. Till then please check our  Flickr pool and Wiki for updates, and leave your feedback in the comments :)

Assessment and the Web; an illustration

For the past months I have been researching innovative approaches to “assessment” on the web; I have read a lot of articles and papers (I even helped write one), listened to many people talk about it and argue over it, and have come to one conclusion; Assessment has more first and last names than a secret agent in international missions; There is embedded assessment, formative assessment, evidence based, peer assessment, game based assessment (a personal favorite) stealth assessment and self assessment, the list goes on. As a result when discussing online assessment there is one thing that becomes very clear; nobody is quite sure what it really means! Which I personally think is a good thing, since I feel that when designing assessment for the web we should look at many different theories and apply whatever pieces fit the best. However, being the visual type I couldn’t help but draw a small illustration of assessment and the web. Let me know if you found it helpful!

Davidson, C. N. (2011). Now you see it: How the brain science of attention will transform the way we live, work, and learn. New York: Viking.

Educause (2011). Deeper Learning: Resources. Retrieved from  http://www.educause.edu/Resources/Browse/Deeper%20Learning/31407

Gibson, D. (2009). Designing a computational model of learning. In R. Ferdig (Ed.), Handbook of research on effective electronic gaming in education (2nd ed.). Hershey, PA: Information Science Reference.

Halavais, A. (2012). A genealogy of badges: Inherited meaning and monstrous moral hybrids. Information, Communication and Society 15(3), 354-373, doi: 10.1080/1369118X.2011.641992

Schwartz, D. L., & Arena, D. Choice based assessment in the digital era. Stanford University School of Education August 2009. Retrieved from http://aaalab.stanford.edu/papers/ChoiceSchwartzArenaAUGUST232009.pdf

Star, S. L. (2010). This is Not a Boundary Object: Reflections on the Origin of a Concept. Science Technology And Human Values, 35(5), 601-617. Sage Publications, Inc. doi:10.1177/0162243910377624

Learning online; Nyan cats and other web-animals

There are many things I love about the web. One of them (naturally) is cats.

So when last week this video  called “Catsplosion” - showing how you can make a cat fountain in Minecraft - (extract below) hit my radar it got me ( and many other Minecraft fans) really excited (and partially dizzy).

“271kochu created a “fountain of cats” in Minecraft by building a structure that extended to the top of the world, then exploiting the game’s simple flocking rules for virtual cats to entice the sprites to form a never-ending fountain that is a joy to behold.” from boing boing

271kochu’s video not only displayed a hilarious cat fountain (purring effect included) but also went through explaining to others how they can build a cat fountain in Minecraft themselves. As a result there have been different style cat fountain videos posted on youtube this week and some intense discussion involving terms like “despawning” (cats in outter space).

There are about 365,000 Minecraft tutorials on Youtube, some of them better than others, guiding you step by step through building your way through the game. The catsplosion video is pretty unique in that it also stands as joke by itself making the video appealing to (cat loving) audiences outside of the Minecraft community.

So what is it with cats? and how did they find their way to a blog post about online learning? Well… I have been thinking about learning online (ALOT) these past months due to the work I have been doing over at School of Webcraft and p2pu.org . This is an exciting time for (online) learning, especially with open initiatives such as MITx and Standford online courses as well as all the startups in the field; from Skillshare to Khan Academy. Despite all this happening, for me there is still a big problem with learning on the web; it doesn’t look like the web (which in turn, looks a lot like a catsplosion).  

Let’s begin by acknowledging that one of the problems with learning online is that it looks too much like learning offline; there are “courses” and “teachers” and “students.” And there are some brilliant “courses” and “teachers” and “students” out there, but at the same time the web does not look like a classroom; it is ruled by different norms and motivations. As a result learning online is often not engaging (unless its a Vi Hart tutorial) ,doesn’t scale (really scale, like in a catsplosion way) and there is a challenge, when it comes to assessing different types of learners (even if you are Codeacademy).

Now back to cats, and the webs’ most famous cat in particular.

The original Nyan Cat video has been viewed 66,747,131 times. A 100 hour version of it has been viewed 5,215,054 times- which is hard to believe but there you go- and there are around 4,000 remixes, including 168 flag versions of Nyan Cats (here is a hilarious greek one  in tribute to my motherland) and some versions of a cat looking at a cat looking at a Nyan Cat video.

So what is it about (Nyan) cats? Well for starters there is something relatively bizzare about them. Its a sense of humor that resonates with us regarding anything weird on the internetz. Moreover there is playfulness in engaging with these videos. As a viewer you take on a silly mindset and follow along the narrative of a tart looking cat with a rainbow flag flying into space - because within that context its really exceptionally engaging. So what if online learning had a sense of web-infused humor? what if it was playful and hilarious at times.

The other thing with cats, is that clearly their appeal has a massive scale. You tube has a nice function where you can see the stats for each video, including when it picks up traffic. It’s especially insightful when it comes to viral videos. The Nyan Cat for example otherwise known as Pop Tart Cat was posted as a gif animation on the daily comics site LOL-COMICS run by illustrator Chris Torres on April 2nd, 2011. On April 5th the animation was posted on Youtube by saraj00n  but it was only till April 12th, when the popular show Tosh.0’s blog linked to it that it became viral. (more on all the Nyan cat history here )

So what if online learning went viral? what if sharing and curating content were part of the actual learning process.

Finally, wether its Nyan cats, breading cats or catsplosions, one great thing about the web is that you can remix it and make it your own. The (great-er) thing about that, when it comes to video memes, is that people make remixes of an original meme really well - sometimes even better than the original ones. They spend time and put a lot of effort to add a new perspective, a new punchline and get assessed by the number of views, shares and likes that come along with the endorsements of their peers. So what if assessment was embedded within what people make on the web? within the remixes they post and the reputation they gain. 

Before you get all catted-out, what I m trying to get to in the post is that we have an opportunity here to do things differently, we don’t have to follow the (out-of date) classroom model when it comes to learning online. We can make learning on the web look like the web. And if that involves your occasional web-animal, so be it.

Reaching the end of these lazy Saturday scriblings, I leave you with another bunch of famous web-animals. Till next time-brohoof!

Athens Plaython; playing street games in a broken economy

2011 was a year of many changes in my life; firstly I switched jobs and went from working in a school to working in the field of online learning. Then I switched continents and went from being a devoted Brooklynite to splitting my time traveling between Europe and the US. But most importantly I launched a project called “Athens Plaython” in my hometown of -yes you guessed it right-…Athens, Greece, after seven years of absence (1 week long vacations dont really count), due to me living abroad.  So this blog post will be dedicated to that project and all the great things related to look forward in 2012.

First a bit of background; as you probably already know Greece has been in the middle of an international financial turmoil, often making headlines this past year -yes people take a very concerned look these days when I tell them I am Greek. Truth is things are not as melodramatic as the international press tends to depict them, but the morale is indeed low and the streets are often flocked with angry protesters marching against the austerity measures. So when me and my very talented friend Artemis Papageorgiou started discussing this past summer about working on a project together, doing something playful and FUN in the streets that are often flooded with demonstrations, was the first thing that popped into our mind. 

Following the footsteps of our friends at Come Out And Play in NYC and SF but also groups like Invisible Playground in Berlin, Hide and Seek in London and IgFest in Bristol we decided pretty much in a day to start our own street games festival in Athens- with a twist; not only have a group of awesome big sized games for Athenians to play, but also include a set of workshops that will help youth and adults design their own city sized games & learn about new technologies. 

In typical game design fashion we decided to “play-test” our idea and planned a quick launch in November to test peoples response. We invited friends from NYC, Finland and Athens to showcase some of their best games, mixed in DJ’s from Athens’ hippest indie radio station, partnered with the citys’ best online city guide and included a set of workshops for kids… et voila (many sleepless hours later) we had what you would call a “working prototype” within 2 months of planning! And it looked like the video below…

Around 1000 people attended our event from all kinds of backgrounds and ages, notably playing non-stop for four hours, filling up the space, mind you - during the evening of a highly anticipated soccer game between the two major leagues in the country. Big part of the reason was the great games we had the honor to host; from Gigantic Mechanic’s Shadowplay, to Kaho Abes’ Hit Me, to Christos Iosifidis & Douglas Wilson  Joust, to Eleni Savidou’s Moustag to Brendon Trombleys’ and Maria Mavroudis’ “Pickpocket”. We also included a workshop for kids in the mix (which sold out that week), were along with their parents, kids designed games for their city and learned how to use conductive materials and LED lights to build a basic circuit.  Moreover a bunch of 15 superhero volunteers helped us with everything that had to do with running the event to keeping our cool when things went wrong.

After all was done and we could take a deep breath and start focusing on a bigger festival to launch in September 2012, an unexpected set of good news arrived; we got an award for placing second in a competition organized by the local TEDxAthens conference, where we got to present our idea in a five minute (very stressful) talk. Once again it was amazing to see people’s positive response to games and it only encouraged us to keep “playing”. With that said, one of my favorite moments of the day was playing “Ninja” with a bunch of TEDxsters, one of them wearing a broken leg cast - who remained undefeated till the last round!

To sum up, our goal upon launching the project was to introduce a platform for the local community to come together through playing, interacting and learning. It was our attempt of proposing a lifestyle infused and surrounded by playfulness as a means to improve our everyday encounters and urban experiences. We asked Athenians to get out on the streets and re-imagine their cityscape, use a bench as their headquarters, play with projections on the walls and draw moustaches on their faces. And they did it! In the months to come our tiny team will be growing, we will be seeking more funds to build our festival the way we want it and most importantly; dear Athens we look forward to playing with you more :)

“hacking” with friends; a recap from Toronto

When we all met last November in Berlin for the p2pu.org annual workshop we agreed that one of the most remarkable things about working together has been the amount of work we are able to produce when we all find ourselves in the same coordinates. You see 90% of our time we work remotely dispersed in various and exotic locations of the world. I realize this reads a bit like marveling at how bright it gets when the sun is out but bear with me. 

This past week we met with our friends from Mozilla’s Hackasaurus in Toronto. For a while now we have been talking about the exciting possibility of using the “P2PU Challenges” as a model for scaling social learning for a range of Mozilla Webmaking projects. So we started by testing ”hacking” out how that would work, by spending three full days in Mozilla’s offices in Toronto (thank you for hosting us!) designing curriculum and developing new features that will allow us to launch a Set of Hackasaurus Challenges by the end of January. Even though we had been working remotely for a month prior to meeting , it was phenomenal to see the great work we accomplished in just a few days; 

Here is a summary;

• We designed a Set of fun Hackasaurus Challenges for teens, tailored for the p2pu platform, that will be first tested with members of the Hive Network. Here is a sneak preview of what they will be about. 

The  world of the open web is full of opportunities for you to create serious cats, planking videos and mindcraft empires. However deep within the trenches we are threatened by the evil nemesis of firewalls and parental controls, which prevent free and open webmaking for all. The world needs you, a superhero of the web to save the day! In the Hackasaurus Challenges you will go through basic super hero training to learn how to create your own webpages in the battle for the open web.

• We started developing some key new features that will allow for an improved learning experience; the Hackasaurus tasks will be interactive, users will be able to apply for a badge within the task page and there is going to be the ability to group challenges together. Here are a few screenshots of how it will look!
  
  

Apply for a Skill Badge within a Challenge Task

Card that includes a Set of Challenges

• We also tested a curriculum that can allow anyone to get started making their own challenge at p2pu.org. We realized that people have different approaches when putting together a challenge; some might want to devote more time to fleshing out their curriculum and some are looking for a “fast and furious” version of how to make a challenge. To cater to the later group, we got busy working on a series of story-lines that can help anyone understand a) what is a challenge b) how do I make one? 
  
  Screenshot from “What is a challenge storyline”. Check out the full deck  here
   
  Screenshot from “How do I make a Challenge” storyline. Check out the full deck here

• Some of the community members like Heather Payne, founder of Ladies Learning Code designed their first challenge during the meeting. In her challenge she used the context of making a “MadLips” game to have her peers learn how to code in Python. 
  
   
  Check out Heather’s Challenge  here

Our next steps after Toronto, include further development of the above key features, refining our curriculum, testing it with a group of teens from the Hive Network and releasing a set of animatics that guide you through what is a challenge and how do you make one.  As we move ahead in 2012 there are things in our to-do “Making Awesome Challenges” list, like adding new assessment features and improving the Challenge creation experience. So consider the above just a preview of all the good things that are yet to come.

In conclusion, we spent three days accomplishing more than we would have done by being remotely connected. But that was not the best part about taking a plane and heading to Toronto sometimes from places as far away as Germany or California. The best part was having the opportunity to work in person with friends and meet new ones like Matt Price, Luis Zarrabeitia, Greg Wilson, Sarah Wigglesworth and Heather Payne who so kindly offered their time volunteering to work with us during the meetup.

In P2PU we often talk about friendship -especially the one aligned with shared passions and interests -, as a driving force that helps us explore new ideas together, share the risk of making them and most importantly learn from each other. So If I had to make a wish for what I would like the new year to bring that would be more opportunities to come together and “hack” with friends. 

Challenges Model @p2pu - what we learned from London mozfest and the p2pu workshop in Berlin

As part of the work we are doing at p2pu.org we are creating a new feature for asynchronous online learning that uses game and project based learning principles; we call that a learning challenge. We have already launched a very rough beta of challenges with School of Webcraft that you can check out here. Additionally these last weeks we have been really focusing on getting our current work out there and testing it with different audiences, as well as iterating and redesigning the current model with the p2pu community in order to improve the experience.

In these past weeks we first run a bunch of workshops at the Mozilla festival “Media, Freedom and the Web” to test our existing “challenges model”. Depending on the session (we run a handful!) the participants were either novices in webmaking or expert programmers or simply people who had an interest in using the challenges model to teach about topics other than programming. During these sessions we had different types of comments, some regarding the interface and some the actual learning approach and content. Additionally during the P2PU Berlin workshop we looked more into new features that could improve the “learning challenge” experience. In this blog post I will try to sum up the feedback we got in both places; from creating a challenge, to completing the tasks, to applying for badges. Finally, we will share some of the wireframes we created in the p2pu workshop, including features the community would like to see or improve.

1. Creating a Challenge

Many participants were interested in creating their own challenges on p2pu, here is their feedback both in terms of content and user experience;

Feedback on the content

CONSTRAINTS=BETTER CHALLENGE IDEAS: we played a game during the mozilla festival were people were given constraints in order to come up with challenge ideas. They had to combine verbs, that correlated to “learning mechanics”, with artifacts, such as “video, recipe(!)”, with a word of their choice, that correlated to something they wanted to teach, for example “interactive narrative”. We found this activity to be really helpful for people to come up with ideas for challenges in a group setting. 

CHALLENGE 101 in 5 STEPS: we also spend time in the p2pu workshop explaining what are challenges, when we should use them and how we can make our own challenge. During these sessions we concluded in 5 steps that one can take to create their own challenge (see picture below). 

For more details on the content created for new challenges visit this etherpad.

Feedback on the UX

CREATE A CHALLENGE UX = CONFUSING: it is hard to navigate the system to create a ‘challenge’ as it is listed under study group. Many people asked “What is a “study group”? Why are we starting here?.”

INPUT FIELD= LONGER : “Wish that the input field was larger for the one sentence description.”

MISSING ABILITY TO UPLOAD BADGES: there is no way to upload badges as a challenge creator, not clear what the process is to do that.

2. Taking a Challenge: Many participants with various levels of expertise with programming took the challenges or tested the interface, here is their feedback both in terms of content and user experience; more detailed notes in John Britton’s etherpad here.

Feedback on Content

CREATIVE CHALLENGES vs ROTE MEMORIZATION: People thought that the HTML hunting in the world around you challenge was an inventive way of thinking about the meaning of tags; they found interactivity very motivating. “I really liked the challenge because it was fun, I like that I connected something abstract to something real (marking up the world around you)” On the other side there was a lot of controversy over the HTML by hand challenge that had learners copy code in a piece of paper. Some said things like “I didn’t do the first challenge, I hate copying things ” and for others copying HTML by hand was helpful to learn the different tags.

FINAL PRODUCT: Many participants asked for a final product that they can try out within the challenge mode. There were many ideas about this, from having a code editor embedded in the challenge to having the ability to better view blog posts or videos with peoples’ work. 

PUSH FOR A CO-OPERATIVE SPACE: even though the challenges encourage some peer interaction they can be completed alone. Many people said that they would prefer to work in a group in order to complete challenges and wouldn’t have done this on their own. 

INCLUDE EXAMPLES: some of the tasks are really general, it would be good to provide several examples. Maybe include pointers to external resources. “It was easy to ask for help since we are in the same place, don’t know how I would get help through the site.”

MORE BUILT IN INTERACTIVITY: include interactions such as voting for the most useful work link, battling and debating

UX Feedback

 

Assessment from the point of view of a designer #3: social, creative, dynamic and personal

It’s time to cover the part about “personal assessment”,  in the  ”assessment of deeper learning online” blog series that I am doing as part of my work at p2pu.org . So (if you haven’t done this already) and you are interested in the scope and context of these blogs please check out assessment related post number 1 (social)  here  and assessment related post number 2 (creative+dynamic) here.

Display them in a learner dossier :users are able to track their achievements (metrics), analyze their past interaction behavior and share their data with their friends. A common system that gives players these abilities is found in games and is known as a player dossier; Medler explains it in his paper Player Dossiers: Analyzing Gameplay Data as a Reward “a data-driven reporting tool comprised of a player’s gameplay data (..)These reports are mediators connecting players to the vast collections of gameplay data being recorded within games from Farmville (Zynga, 2009) to Halo: Reach (Bungie LLC., 2010). A player’s in-game actions, created content and achievements are organized into dossiers representing each player’s identity, morphing over time as the player continues to play. For example, Giant Bomb (Gerstmann and Davis, 2008), a game wiki and gamer community website, creates player dossiers by visually combining achievement data from a number of gaming platforms (Blizzard Ent., 2004; Microsoft Corp., 2002; Valve Corp., 2003). Achievements are broken down by game, used to rank registered players against the average community member’s achievements and aggregated to discover the rarity of achievements within the community (Figure 1). ”(Medler,2011) In a peer learning community we can rename a “player dossier” into a “learner dossier” were metrics work similarly to a control panel/dashboard found in a car, plane, spaceship. It will become a tool that helps learners understand and drive their own learning as that evolves and shapes over time.

Figure 1 from Medler’s paper: Giant Bomb aggregates player data from multiple achievement systems to display player dossiers that cross genres and platforms.

Create tools that allow discoverability: the design should allow for easy discovery of more resources or communities of practice to diversify ones’ interests and passions. Pandora was one of the first applications to help its users discover more music. Turntable.fm also helps one discover music but this time not through algorithms but instead through the contributions of its members who get recognition for playing good songs. A peer learning community could support learners in finding what they want to do next, through the contributions of the peers and also connections to other communities.

turntable.fm

Embrace point of view: this ties back to the earlier point about showcasing a diversity of autonomous learners. In order to do that we can imagine taking several perspectives on a problem or a situation and examining the problem from several sides, taking note of the benefits and limitations of each point of view. A Point of View tool can play an important role in assessment in both formal and informal ways. An informal example is when someone asks the crowd “Do you think this thing I’ve made addresses all the main criteria? Here’s is why i think it does…” and when friends like it, comment on it, or add ideas. At a formal level, the same tool can provide a debate and response to formal scoring of an artifact or piece of evidence, if desired. The space should provide channels for discussion that go beyond commenting to embrace diverse opinions. Debate structures are a good example for different points of view to be discussed. An example is the youth social network i-remix which has a debate feature that allows students to pose debates over topics of their interest. The voters have to provide arguments for why they prefer one over the other. Quora and Stack Overflow are other good examples were discussion is formed dynamically through a voting system (best answers get voted higher), often allowing for the rise of constructive and in depth answers.

Special thanks to David Gibson for helping me edit part of this post.

Assessment from the point of view of a designer #2: social, creative, dynamic and personal

In my previous post I talked about design of “social” assessment, one that happens within the discussions and debates that occur around a specific topic of interest in a community. In this post we will look at assessment of deeper learning as one designed in a space that is both “creative and dynamic”.

CREATIVE ASSESSMENT

Let’s start with “creative assessment”; To give you a little bit of background, as an avid gamer and designer of games myself I am very interested in the process of mastering play; how do you know that you are good at a game? Well for starters, you have finished it - which can mean that you have gone through a series of levels and defeated some evil boss at the end. It might also mean that you have created some exceptional structure along the way. And you can see in all those things you have made, whether they are strategic choices in a game of Chess or the creation of a virtual Matchu Pitchu Mountain in Minecraft, that you have mastered gameplay to some extend. 

Now, back to assessment; how is this “making” idea relevant to deeper learning in online settings? In the recent Mozilla Festival Tim Hunkin talked in his keynote about the power of “making things” and how traditionally makers have been people that might be “tinkering alone in their garage” but because of the accessibility of the online space they have been given the opportunity to come together. And because of that, we are witnessing the rise of a huge maker movement, with growing communities around the world such as maker fair and instructables.

On the other hand we have to make it clear that when we talk about “making things” that is not necessarily a process always tied to deeper learning. To assess deeper learning we need to ask our learners to make things by solving rich problems in real world situations. And they have to do that in a transparent setting. “In most current lists of “21st Century Knowledge and Skills” share some characteristics that make them hard to measure with standardized tests. They are complex, they are often multi-disciplinary, they manifest themselves as action-events rather than objects, and the learner has substantial autonomy and purpose in choosing the learning activity…working with others to solve complex real-world problems entails increased use of deeper learning capacities and a need for broad, multiple measures to validate learning and achievement. This occurs because solutions and decisions involved in real-world settings need to a) meet several criteria of success, b) involve and prioritize more than one viable outcome, c) be evaluated with negotiable values for individuals and the community, and have other characteristics that make them potentially as complex as the problems they address.” Moreover we need that process to happen in a transparent setting. Because “A strong assessment system provides not only quick understanding of what people know (a badge or token), but also deep evidence of that knowledge and of how that knowledge was formed. Access to this process is itself an educational resource for self- and other learners, as well as grounding the more visible markers of knowledge in a transparent, reliable record.”

In the paragraphs below we will look at what the design of the space can do to facilitate these parameters of what we have called “creative” assessment. The design needs to support learners to solve rich problems in real world situations and embrace transparent mechanisms and artifacts of practice such as portfolios and badges.  

To support the solution of rich problems in real world settings

Create complex problem settings:Learners are faced with problems that require them to navigate complicated systems, filter information, address questions that have more than one correct answer; to do so they have to create a hypothesis, debug a system, design a working model, perform a skill to communicate an idea. These problems are connected to real world settings making it clear to the learner how solving a problem could apply in reality. Moreover the design of the space allows for rich dialog and a flexible exchange of ideas. OpenIdeo is a nice precedent of such a structure; designed to facilitate collaborative work on solving world problems, the platform presents monthly “challenges”; such as “How might we use social business to improve health in low-income communities?”. The challenges sponsored by an organization such as Amnesty International and are open to the community; anybody can pitch ideas that are discussed in a simple forum. The process moves from conception, to applauding, to refinement, to evaluation, to completion. In each of this phase ideas are filtered through voting from the community and the challenge sponsor jury. The winning ideas are then sponsored and completed.

Foster evidence centered design: When asked to perform a task, such as building a website, a learner can easily tell us how to do it OR show us that she can do it. The completion of the task “demands further research, estimation, modeling of more than one field of knowledge, etc.” The system allows for the community to unobtrusively debate and settle the matter of linking each task to aspects of knowledge via mechanisms such as tagging and voting in the system.  Evidence-centered design suggests that the (assessor) person knows 1. what the task is (task model), 2. what good performance looks like (student model) and 3. can see a distance between an actual performance and the ideal (evidence model)). Ideally, the community should also promote this knowledge in the assessed as well, because the ideal is to have a mutually self-reinforcing system fueled by people who continuously play both roles; they assess themselves to self-improve, they use feedback from others to form self-improvement decisions, and they offer to other critical friendship advice that will help the other person improve. An example of that would be building creating a tutorial for others. Popular platforms for tutorials are sites such as Instructables but also Youtube tutorial channels and videos.

 Vi Hart’s Online Math Tutorials

To embrace transparent practices:

Have open criteria and artifacts: The criteria as well as the artifacts and process associated with the assessment are public. An Open Badge infrastructure (OBI) can provide transparency since the badge is connected to evidence that is open to all. Moreover there is the value of having a transparent “rubric” ; it guides the construction of something. “Does the gadget have a handle? Is the handle sturdy? Does the gadget do its job safely? etc.” As the user compares their created gadget (or any performance of any skill or demonstration of knowledge) can they see where on the rubric it would fit? Can they figure out how to make it better based on that self-assessment?

Open Badges in P2PU School of Webcraft

Provide clear pathways: being open and transparent creates clear pathways for learners who want to move from the periphery of a community to its core. Timelines or maps illustrate the idea of a pathway that indicates where does a user need to go first and where second, in order to achieve a goal in the community. We can find examples of such pathways in games, where the central game screen depicts locations that players can follow to progress in the game, unlock certain tools, complete quests. Another example is a “timeline”, popular in the new Facebook layout. A timeline shows the “story” of a user; one could imagine the idea of “timeline” translated into a learner’s story- showing were they were when they first started and were they are now. Other peers can look at that as a model for their own pathway in a community of practice. When thinking of badges, a puzzle of unlocked and locked badges such as in the game Glitch can help learners better understand what they need to do to get more involved in the community.

Glitch Game

Facebook Timeline

Embrace fairness and accountability: promoting real life connections for people in order for them to have accountability over their actions and be fair when assessing others. Having a real–world identity encourages fairness as learners recognize that their online behaviors will be used to judge their real–world reputations. In a peer community accountability can be reinforced when linking one’s real identity to external communities such as Facebook , Twitter and Google+.

Share success and reputation: the ability to share with your friends the success of completing something really hard or getting a rare item or even participating in something “bigger” like an event.  By sharing success you also build your reputation within the community. In Facebook games like Frontierville or Empires&Allies automatically allow players to track how well other players are doing, not only in terms of their skill level, but also in terms of their reputations. “Neighbors can automatically track each others’ progress, both in terms of their points, and in terms of their reputation level. These reputation levels play an important role in the game, providing transparency to which neighbors are more helpful and sociable, thus initiating friendly competition over who is the ‘friendliest’ neighbor.” (Whiston,2011). 

DYNAMIC ASSESSMENT

If you have made it so far in the post (congratulations you are a trooper and I feel personally obliged), it’s time to talk about assessment that is “dynamic”; First what do we mean with the term “dynamic” ? imagine assessment as something that changes over time. It is found within the actions and transactions between peers and the system; those actions can vary from receiving an automatic tip when you are stuck in a learning challenge or giving constructive feedback to each other in a discussion forum. ”Achieving deeper learning outcomes hinges heavily on assessment of learning by members of the community and making the skills for, and methods of, giving helpful feedback both public and social. (..) The well done peer assessment not only provides the subject of the assessment with a better idea of her skill level, and illuminates some path toward improvement, it also improves the assessor’s skill at discerning good work.”

In the paragraphs below we will look at what design elements can promote “dynamic” assessment. The design needs to support both feedback mechanics that help you move forward in your learning and that are embedded in the system as well as a space for learners to develop the habit of giving helpful feedback to their peers.

1. The system gives you feedback 

Feedback as you go and on demand. Feedback as you go is when you’re motivated to take an additional action or alter your previous action based on the assessment of your current action. For example  when you click on a button, you see a visual change or a sound which makes it clear that you’ve done something. Feedback on demand is when you request feedback to complete the assessment of your action in order to better understand what you need to do next. Games are perfect examples of feedback as you go (losing health points, getting attacked by an enemy, avoiding a block) and on demand (asking for a tip, requesting information on a leader board). In order to have a dynamic feedback system the platform needs to track metrics such as time taken to complete a task, attempts to complete it etc.

Leveling Mechanics & Power-ups: “the system can include “leveling” so that people understand early progress, completion of benchmark processes and products, and can see a developmental path for expanded reputation within the community. “Levels can unfold from bronze, to silver, to gold to platinum. The levels can be directly tied to badges, so for example there could be an “HTML Basic Silver Badge”. Additionally the levels and badges can connect to power-ups. Lange-Nielsen identifies four types of power-ups: “The Expendable”, which is stored as inventory,”The Temporary”, which is an instant power up, “The Constant” power up, which is tied to an “upgrade” and “The rechargeable” which is “gained then drained. ”(Lange-Nielsen,2011)

Negative consequences: the system allows for negative consequences in situations were the user is not providing helpful feedback or has been inactive. An example of this could be earning a “tourist” badge if you have not logged in for a long time. Another example would be loosing a badge or a certain other privilege after some time, the same way one can loose the “mayorship” privileges in Foursquare.

2. Your peers give you feedback

Create assessment as a core skill: learners develop the “habit of using feedback” to help other people see what they can do to improve their idea. When the receiver of feedback says “thanks, I’ll give that a try” then a mutual bond is created with rewards for both parties. The habit of helpful critical feedback is an act of “critical friendship” that repays one with higher levels of observation and skills of analysis while at the same time helps others improve. Coleman calls this “closure” and says that it creates “social capital” - the resources available to a participant - in the community. In a peer learning community the design should facilitate the development of “critical friendship” by providing plenty of opportunities to give feedback, creating a positive atmosphere were feedback is appreciated  and most importantly provide modules that help one learn “how to” give feedback, i.e. a system of appropriate tags or sentences that are suggested to users when leaving feedback.

Assessing the assessment: Users can express how assessment worked for them, if it was useful, how it felt to be both the assessor and the assessed. An example of such an experience is found in traveling sites such as tripadvisor. Under each review of a restaurant or a hotel, there is the option to evaluate the review itself by saying if you found it useful. 

Tripadvisor
Remixing Learning: learners have the ability to “mess around” by editing, redoing, and copying existing media (Ito,2009). A key part of learning in the community relies in the ability of its members to take it - “mess around with it” and make it  their own. The act of remixing an existing piece of media became very popular with youtube, were people would respond to video postings by making their own video. Soundcloud is another popular aplication that allows you to use others tunes, remix them and share them in a community. Popcorn.js also allows you to remix existing media and anotate them in HTML. In the p2pu community we could imagine “challenges” to be hack-able and remix-able, continuously evolving and shaping the “domain of knowledge.” 

popcorn.js

SoundCloud.com

References:

Social gaming for change: Facebook unleashed by Jennifer R. Whitson and Claire Dormann, 2011

The Power-up Experience: A study of Power-ups in Games and their Effect on Player Experience , Filip Lange-Nielsen , 2011

Peer-Based Learning in a Networked Age, Mimi Ito, 2011 

Social capital in the creation of human capital. American Journal of Sociology, Coleman, J. S. (1988) .

Framework for assessment of deeper learning in online learning communitiesWhite Paper in Progress, David Gibson, Alex Halavais, and Nils Peterson 2011 (coming soon)

Assessment from the point of view of a designer: social, creative, dynamic and personal

Since I joined p2pu.org I have been thinking of “assessment” a lot. A long often misinterpreted word if you ask me; brings to mind things like evaluations and tests. Well, this post is about putting a 21st century spin on assessment and talking in particular about assessment of deeper learning in online learning environments, like p2pu.org.
First a bit of background; the last two months we have been privileged to work with a group of “assessment gurus” such as David Gibson, Alex Halavais, and Nils Peterson who are writing a white paper that suggests a framework for assessment of deeper learning in online learning communities (stay tuned for the paper). This blog post is inspired by that work and aims to approach assessment of deeper learning from the point of view of a designer.  

 
What is deeper learning?  
 
The Hewlett Foundation defines deeper learning as; Learning that involves mastering core academic content, critical thinking and problem solving, working collaboratively, communicating effectively, and learning how to learn independently (Hewlett Foundation, 2010).
 
Why do we care?
 
Some weeks ago one of the p2pu.org community members, Caley, shared his story on John Britton’s blog here. Caley who has a background in systems management work, had always wanted to learn how to program. In his story he explains how he picked up programming in p2pu.org; he talks about working together with his mentors and how he grew to become not only a web developer but also an active member of the online open source community (he even runs an open source website to track goverment data in Missouri, where he lives). During this experience Caley became deeply involved in his learning and developed skills like problem solving and collaborating. However these skills can not presently be assessed within the P2PU community or credentialed outside of Caley’s circle of peers.
 
How can we do this?
 
So how can we provide assessment of those skills within an online learning community?First lets try to imagine what an online learning space like p2pu.org is made out off; there is the community of  learners, and then there are the actions they take and the things they make. The space is designed to be social, personal, dynamic and creative. To design assessment of deeper learning we have to think of it as interwoven within that space; we have to imagine a learning ecosystem were assessment is; 

• Social (the community perspective): it emerges out of the social practices within a community of practice.
• Personal (the learner perspective): it embraces a diversity of autonomous learners. It showcases what each learner knows, cares about, is able to do and wants to do.
• Dynamic (the assessment perspective): it is situated in the actions and habits of the peers and evolves over time.
• Creative (the domain of knowledge perspective): the design showcases the things people make, talk about and exchange within different knowledge domains. It encourages learners to solve rich problems in real world settings by creating transparent authentic artifacts of practice. 

 Assessment of Deeper Learning Ven Diagram

So how can we *really* do this?

SOCIAL ASSESSMENT 
In this blog post we will provide design guidelines for assessment that is social (stay tuned for personal,dynamic and creative assessment to follow in next posts). The question we want to answer is;  how can we design the social circumstances that allow assessment of deeper learning? We can achieve that by incentivizing participation and fostering social practices within a community of both learners and stakeholders who share common passions.

To incentivize participation:
 

• Create a sense of belonging: the users feel that they are part of something “bigger”. The popular crowd-source funding platform Kickstarter does that very well, once you have contributed to a project you feel a certain level of “special” having helped to get a project realized.  A peer to peer platform can be designed to reinforce this sense of “special cause” and “belonging to something bigger” ; sponsoring of newcomers, completing certain challenges before becoming officially a member and working towards common greater goals could strengthen that feeling.
   
  Kickstarter
  

• Embed social graphs: the users can invite people who are already in their various social networks (Facebook, Twitter, Google+) to join them to “geek out”.  For example, a learner should be able to invite friends who share a common passion to take the same course. Friends should simply have to ‘accept’ the invitation to join the course.
   
  Connected through twitter & facebook to klout.com 

• Design a co-operative space: the space conveys that objectives are much easier to accomplish together rather than alone. For example, while in games such as Empires & Allies the first stages of the game can be played alone, after progressing players cannot purchase desirable virtual items and certain missions cannot be completed without the help of a friend (or without going on a battle against a friend). Similarly in a peer to peer environment working together could be encouraged as part of the experience, for example certain challenges should be harder to complete alone.
  

• Have learners take on roles: the space encourages it’s users to explore multiple identities; in doing so they become committed to the space and the community in which the learning is situated. Those roles are reflective of the practices that take place within the community and develop over time. The signaling of change in roles (e.g. becoming a trusted helper to others) are part of creating learning communities.We can approach community roles as complimentary parts of a big jigsaw; for example, in order to complete a website project you would need to have a developer, a visual designer, a project manager etc. The approach is that one person can have multiple roles in various projects. Additionally it might be useful to consider negative roles within the community as well, for example a non-active user is often call in social networks a “tourist”.
   
• Make the social interactions engaging: The process and experience of belonging in the community is designed to be an enjoyable and enlightening experience. The actions and transactions between peers are meaningful and fun. Such interactions could encourage in depth conversation and socialization, as well as the coexistence of trivial actions such as sharing, gifting and reciprocity.

• Build social obligation: similar to the qualities of co-operative space above, social obligation is build around the action of exchanging and gifting aid; this is common in Facebook games were you ask the help of your “neighbors” in the game to obtain virtual items or to better complete a task. This action makes you feel “obliged” to return the favor. Similarly in social networks of shared practice, such as Little Big Planet users will use special terms such as “h2h” which translates to “heart my game so I can heart your game”. These mechanisms usually do not allow for deep assessment, however they have been proven to reinforce interaction.
  
  Empires & Allies

• Design referral mechanisms: make it easy for the learning experience to spread through word of mouth across social network channels bringing new users to the site. For example, in p2pu that could involve getting prompted to invite a friend after completing a challenge.
   
   
  Klout.com
  

To foster social practices within a community of both learners and stakeholders:
 

• Develop Specialist Language: the vocabulary and terms used by the community are created by and make sense within that community of practice. For example in github users will use the word “fork” to get a copy of someones code. A problem with that is when newcomers join the community, usually not fully understanding what everything means. Using mentoring as well as automatic systems like tagging or a taxonomy allows for the newcomers to make sense of the existing vocabulary, add to it and move faster from the outskirts of a group to its core.
   
  
  Github 

• Develop Smartools : A smartool is a tool that a learner can use again and again in various learning contexts. In unlocking smartools learners connect the skills they have gained to actual tools they can use and master. For example, Firebug (if Firebug was to be unlocked as a tool after completing challenge x) is a type of smartool. Smartools can also be user created, such as a collection of visual references to explain basic HTML elements and tags.
   
  Firebug 

• Integrate badges internally and externally: the badges provide validation that is both internal, comes from the peers who are taking the same course, but also external, comes from the stakeholders, who for example might want to hire someone for a job. “The badges celebrate not just the accomplishment of co-discovered goals, but the engagement of the community in assessing and guiding the progress of the learner. Traditionally, assessors are themselves guided at least in part by extrinsic factors, most often in the form of a salary or fee.“(Gibson,Peterson,Halavais,2011). An essential element assessment in a community of practice is that the act of assessment is beneficial to the community and by extension to both the “student” and the “expert.” The badges can be designed not only to be awarded to the assessed but also the assessor. Moreover making visible and celebrating the effort that went into acquiring the badge contributes to having the badge be part of the assessment.
   
  School of Webcraft Badges
  
  Check out this blog post to find out more about Creative +Dynamic assessment and this one for Personal Assessment.