Weekly Updates
Here I will keep updates on a weekly basis of my experiences throughout this internship.
Week 1
This week I began working at the Articulab. My first day, I received readings from Dr. Bai in order to begin collecting information for my literature review. I have been reading through these materials, as well as finding other materials that pique my interest. This week Dr. Bai and I also went over the description of my DREU experience on a week by week basis. I also did video transcription for group 6 of the outbreak game data, which is almost complete. The only challenges I faced this week were becoming accustomed to a new lab atmosphere without getting too intimidated. I now feel like I understand what is needed from me, and I am excited to carry out my responsibilities. Within the next week, I plan to update the finite state machine with child behaviors, finish all transcription of group 6 and group 4, and add to my literature review.
Week 2
My second week here at the Articulab was very interesting because I had the opportunity to get creative with the ways that we chose to analyze the data that had been collected. We have enormous files filled with transcripts of children talking during gameplay of a game called "Outbreak" which is a game created here at CMU by another lab. During gameplay, players take turns rolling the dice to move the game piece along the board, and they face a collection of ten obstacles along the way. The obstacles, or “rooms” as they are referred to in the game, have a challenge that is unknown to the participants. For each room, participants are given a period of time to ask questions about the room, and these questions are limited in format by question tokens they are given. After this question asking period, participants must discuss and choose up to three “skill cards” that must match up with the threats in the room that they discover during the question asking phase. If they can correctly cover each threat, then they pass that room.
There are 6 groups of participants that we are working with; that is 6 groups of up to 5 players with recorded sessions of gameplay. However, only groups 1-3 and 5 are fully and correctly transcribed. Another intern and I went through to put certain utterances into categores as we saw fit. This took much longer than one might think, because there must be a rhyme and reason as to why we put things where we did. Now that we have a big, categorized list of these statements, I must go through and find the specific information about each clause (the timestamps from the videos of gameplay, participant numbers, group numbers, etc.). We are doing this so that we can update the finite state machine that will be used to facilitate gameplay with a virtual agent.
[Virtual agent game companion in Gates Hall, CMU, 2017]
Week 3
This week I had the opportunity to pour over the data we have collected in a much more detailed manner. At this point in the project, we have a large corpus of clauses from the gameplay, and this corpus is categorized into phases that will be used to update the finite state machine. However, in order to update the finite state machine, we have to know very detailed information about each clauses and why it belongs to a certain category. So this week, I focused on providing detailed information to each clause that had been categorized. This detailed information includes timestamps in the video files that are kept on the server, preconditions, verbal, and nonverbal behaviors, and interactional context. We want to provide all of this information so that we can update the finite state machine meaningfully, and also provide video clips joined to each clause so that animation of the virtual peer is as accurate as possible.
Week 4
I really had a great time this week because I got to learn about a topic I am getting very interested in: natural language processing, or NLP. I am very interested in this topic because of how closely tied it is into artifical intelligence and machine learning, and these burgeoning fields are important to pay attention to. The task I got to participate in was assessing the corpus of clauses made during gameplay by the participants to understand patterns in the transcriptions and in turn write a definition for "representative example" for each category that we quantitatively discover. To work with the data in a more functional format, it had to first be preprocessed. What this means is that we removed the noise from the data, including clauses that contained less than two tokens (words), various transcription techniques which were comprised of annotations for exhales (“((exhale))”) inhales (“((inhale))”), along with a few others. The script created to do this preprocessing reads in the various transcription texts, performs the analysis described above, and then writes to a new text file with the lines of clauses we are interested in.
Another script that we created reads in the file we wrote from the previous script, and processes the file for n-grams. This script creates a python dictionary where the keys are an n-gram (n specified at the beginning of the script) and values are the frequency. An n-gram is “a contiguous sequence of n items from a given sequence of text or speech.” With these n-grams, we can curate methods for categorization based on quantitative data alerting us to the way in which children typically form their sentences, and how word tokens within clauses are linked to one another (frequency). We were able to process the data for any number n n-grams, but focused primarily on n=3, 4, and 5 because of the sentence length that is typical of the children.
We were able to come up with representative examples for n=3,4, and 5. By representative example, we mean a sentence fragment that is the most frequent for a given n-gram taken from the data set. The representative example for n=3 is “does the room.” This has a frequency of 75 occurrences. For n=4, the representative example is “does the room have” with a frequency of 33. Another n-gram of note where n=4 is “does the room look” that has a frequency of 23. When n=5, the representative example is “can we have a three” which occurs 3 times.
We are excited with the data we are working with, and have plans to do a much deeper analysis to extract quantitative categorization styles to help us extend the data set that we have organized. In order to do this, we have a plan to clean more of the noise and refine our scripts to show us possibly more well-defined results. We also plan to calculate the most common dependencies from the root of sentences. We will also find the verbs and other parts of speech using the POS technique to understand the sentence structures used by the participants in the study.
[A typical Articulab intern meeting, 2017]
Week 5
This week I got back to business with finding detailed information for behaviors within our list of selected utterances from groups 1-3 and group 5. Finding the detailed information is necessary because later I will be focusing on the nonverbal behaviors, and also will need to output specific video moments that are important to the animator so that they can get an accurate picture of what the real children look, act, and sound like.
Our plan is to use these transcriptions and detailed information to formulate sentences that are based on the gameplay state, and when my work is used in conjunction with the other interns working on AI, user interface, and natural language processing, we can have a functional virtual peer. We are working on a game reasoner to promote gameplay decisions, and a curiosity reasoner to influence curiosity of the participants. The gameplay states are those that take place in gameplay and support the flow of the game, such as suggesting a card, commenting on a game act, etc. These game states were based off of the original categories from weeks 2-3 but more refined.
Week 6
This week involved more transcription work, namely with group 4, which meant lots of observation of the video footage and typing/annotating behaviors that are taking place in the videos. I had to do this transcription because, although this group had been transcribed before, it wasn't as correct as we needed it to be. Times weren't precise, and something might have shifted the order of tiers of speech within the file. Finishing this transcription was pivotal because other parts of the project hinge on this being done, as it is the last transcription to be done for the entire corpus to be complete and ready for analysis. After this is finished, other group members can continue to finish the NLP analysis of the corpus, and this also gives us a more well-rounded data set (more data points) to make sure our categorical coding scheme holds true.
We also held our game test alpha run, where we tested the usability of the current modules we are all working on. I was acting as the virtual peer, and I was fed inputs by other group members and by this method we worked out what we still need to work on to make things more seamless. We often found the virtual peer quiet and with nothing to say, or too slow to work with the team, so we took that into account for next week's test.
Week 7
During week 7 I went through each transcription to mark exactly where a phase begins and ends. I had to mark each question asking phase, and each discussion phase of every transcription. This way, we can choose behaviors from each phase that represent every gameplay behavior state, and have a representative sample of data.
This week we also performed game test alpha 2.0, and it had minor improvements with fluidity of game play from the previous week.
Lastly, this week we held the Articulab's first Jam Session. In this session, we all shared brief presentaions of what we do outside of the lab, whether related to science or not. It was fun and I got to show another project that I am working on which can be read about under the CV section of my site.
[Photo taken by Yoichi Matsuyama, 2017]
Week 8
This week I had to work on getting more nonverbal behaviors, namely from groups 5 and 6, in order to create a more well rounded document describing the nonverbal behaviors within states. For the nonverbal behaviors, I am focusing on exactly what to send to the animator because that's how the animations for the virtual peer will get created. There are some animations currently supported by the current version of the virtual agent we have, so I am checking which occurring nonverbal behaviors we could already implement with current animations available, and which ones the animator will have to curate.
Week 9
This week I worked to compile what I've done to this point in my internship, and what else I must accomplish in order to make deadlines feasible and be sure that my work will be usable and understood for the lab after I have completed my time here. My exit tasks here at the Articulab icnlude delivering a final presentation, talking about my footprint at the lab, and carefully documenting all of my deliverables so that they can be undestood and adopted by other lab members and future interns.
Week 10
Week 10 is my final week with the Articulab, and I am sad to go. Throughout my final week, I started getting into some really cool details of the project that I wish I had more time to focus on, but unfortunately I will have to leave! One of those cool details has to do with considering interactional information from children in the Outbreak game. I have been calculating the amount of time each participant spends in overlapping speech, so talking while another participant is speaking simultaneously. I find this so interesting because there are papers with implications on what this could mean for a virtual agent to cut into speech or speak over a person, and how that affects the overall user experience. I wish that I had more time to work on this project! It is so cool, and I learned a lot.
One take away from this project that I am holding very close to me as I ready for the next academic step is how research is performed at this level. It doesn't involve a supervisor telling me exactly what to do, how to do it, and what results I should see; instead, I need to see an end goal and think about what I bring to the table and outline steps for myself to be and asset in a unique way. I am so grateful for this experience! It definitely changed me as a researcher and a critical thinker.