The following are the deliverables for the project. See the schedule for due dates. Once you’ve formed a group, create a folder in the class googledocs folder to keep the documents. Include the group name (if you have one) and your names in the folder title.
1. Deliverables: group contract: digital copy on github & signed hard copy submission
The group contract is a document that you and your teammates will use to assign responsibility for specific tasks within the larger project. These can be divided however you want, as long as everyone in the group feels that the tasks are shared equally. There are a few options for how to divide the work — some ways you might want to consider are:
- Divide by task: each group member gets a number of the tasks outlined in the grade section; equality is determined by grade percentage points
- Divide by skill: the group discusses what skills they bring to the group, and the work within the tasks is divided up accordingly.
- Everyone takes part: if you prefer that everyone have a say in all the aspects of the grade, the we recommend outlining roles within the group. For example, one person might be in charge of starting each task, one person will be the timekeeper, and the other will be the editor.
Please note that everyone in the group will be doing annotation, so that should not factor into the task assignment. Once you are all satisfied with the contract, everyone in the group must sign it and it must be turned in to John Vogel. The contract will be useful later in the peer evaluation section. Changes to the contract can be made throughout the semester, but must be submitted (signed) to John.
Example group contracts
2. Annotation goal
Deliverables: in-class presentation on possible topics and goals
There are many different topics in linguistics that you and your team can choose to examine over the course of the semester. Below is a list of possible topics—you can choose any one of these or select your own, as long as your task is approved by us.
- syntactic dependencies
- semantic role or semantic function labeling
- multimodal annotation
- Textual cohesion and coherence
- coercion (GLML)
- narrative structure
- temporal information (TimeML)
- spatial information (ISO-Space)
- presuppositions, implicatures, inferences
When writing up the annotation goal, your team should be able to express what you hope to achieve with the annotation, and how far-reaching the task will be. For example, if you want to do genre annotation of newspaper articles, how specific will your categories be? Will you do broad categories, such as “world news”, “local news”, “sports”, or will you be more specific: “sports: baseball”, “world news: Europe”?
3. Task Description and Corpus Selection
Deliverables: 1-2 pages write-up, put in Googledocs folder
The task description is a longer version of the annotation goal, where you will begin to define exactly how your annotation task will be done. This is closely related to the corpus selection, which will define the type of documents you will annotate (news articles, movie reviews, selected sentences displaying a phenomenon), the source of your documents (Internet, news archives, another corpus), and the size of the corpus (this will vary by task). The task description should be at least 1 page, typed.
4. Draft Annotation Schema
Deliverables: in-class presentation on tagset and idea behind them, put in Googledocs folder
The draft should be an intuitive design for tagset and attributes associated to each tag. This does not have to be empirically driven or completely formalized, as groups present their draft in class and get feedback from technical side. Given the time constraint, presentations should be concise and precise, not exceeding 10 minutes each.
5. MAMA cycle
Groups start to formally define the tagsets, that an annotation environment of their choice can read than in. Then, groups start their MODEL-ANNOTATE-MODEL-ANNOTATE cycle: internally do actual annotation on a subset of the corpus with the formal definitions, measure annotation agreement within group members (who invented this scheme!), inspect the complexity of the task, revise the task if necessary, then repeat over until satisfied.
6. Full Annotation Schema
Deliverables: DTD and descriptions write-up, put in Googledocs folder
(tagged as 1.0) by the beginning of the class period.
The schema document is what describes the tags and attributes that will be used for the annotation task. This description is in the form of a DTD-like document that can be input into the annotation environment so the annotators will be able to label the texts appropriately. Specifics for how these documents are created will be provided later in the semester.
7. Full Task Specification
- in-class presentation on the final corpus selection and a peek at the guidelines
- full fledged guidelines (typed) upload to github
- dataset distribution(at least for the first week of annotation)
The specification document contains the detailed instructions for the annotation task that will be provided to annotators. This document must be clear, contain relevant examples, and outline any exceptions or special cases that the annotators might encounter. There will be two versions of this: the initial version that is provided to the annotators, and the revised version that will be produced after the annotation is completed. Groups also prepare the dataset on which the annotators start to work from after the presentation.
Deliverables: weekly progress report, in-class oral brief and short write-up upload on github
Your group will not be performing its own annotation task—rather, each group will be given the schema and specification for a different group. Grading for annotation will be based partially on whether you as an annotator met the deadlines, and partly on how well you followed the instructions given by the other group.
9. Annotation Report
Deliverables: in-class presentation, submit the slides to github
dGroups present the final status of resultant annotation dataset, including
- Amount and quality of the dataset
- Difficulties during collecting the dataset (solved/unsolved)
- (If decided) Metrics used to measure the quality and justification behind
Plans for machine learning experiments
Adjudication is the process of creating a gold standard corpus from the documents provided by your annotators. No annotation effort is without mistakes, and part of how an annotation task is evaluated is how closely the annotators agree to each other and to the gold standard once it is completed. Adjudication is done by hand. Your group must write a program that can calculate precision and recall for each annotator when compared against the gold standard. A write-up of common annotator mistakes must be turned in, along with the code and the gold standard.
11. Gold Standard Release
Clean up your final data and release it through group repository along with the final version of guidelines and task scheme (DTD). If groups have issues with publicly releasing their data for any concern, consult with the instructor.
Each group will present their research findings to the rest of the class for up to 30 minutes. This will occur during the period allotted for final exams. Each group member must present the work that they did—if each member collaborated on all parts of the project, then each member still must give part of the presentation.
The presentation would cover following topics (not limited to)
- Review on task goals and annotation specification
- Characteristics of the dataset
- Difficulties during collecting the dataset (solved and unsolved)
- Possible improvements in the next iteration, if to go
- Annotation quality
- Metrics used to measure the quality and justification behind them
- Interpretation of numerics and error analysis
- Machine learning experiment
- Experiment design
- Baseline system and baseline features
- Features extracted from the annotation
- Experiment results
- Experiment design
13. Final Paper
Deliverables: pdf-output of paper uploaded to github
A report of the entire annotation process will be a major part of your grade. The paper should cover everything discussed in the final presentation, including any questions and feedback from the presentation. The format will be either LREC 2016 style guide or ACL 2016 style, whichever comes to your taste (they are mostly identical, but if you use ACL, be careful not to use the submission version – we want camera-ready style).
- If a group needs evaluation and review on their draft, bring in the draft at the final presentation (it’s okay even just for outlines).
- For those who want collaborative latex writing, Overleaf can be a good tool.
14. Peer Evaluation
Deliverables: Submit evaluation via Latte
Lastly, each student individually submit a short peer evaluation on the project and the teamwork. Evaluate these three aspects:
- Explain what each person in your team, including yourself, did for your project. What were they responsible for. What was the quality of their work.
- Give a grade for everyone in the team, including yourself, for their work on your project and explain why you think they should get that grade
- Write a short paragraph explaining the most important things you feel that you learned during the course.