by Mish
Goel did not disclose the robot until final grades were handed out. None of the students figured it out.
Each year, students have an option of selecting an assistant and next year Goel will do the same, but names will change. Students will have to guess if they are dealing with a robot or not.
Jill Watson
Please consider Professor reveals to students that his assistant was an AI all along.
To help with his class this year, a Georgia Tech professor hired Jill Watson, a teaching assistant unlike any other in the world. Throughout the semester, she answered questions online for students, relieving the professor’s overworked teaching staff.
Students were amazed. “I feel like I am part of history because of Jill and this class!” wrote one in the class’s online forum. “Just when I wanted to nominate Jill Watson as an outstanding TA in the CIOS survey!” said another.
Goel and his teaching assistants receive more than 10,000 questions a semester from students on the course’s online forum. Sometimes the same questions are asked again and again. Last year he began to wonder if he could automate the burden of answering so many repetitive questions.
As Goel looked for a technology that could help, he settled on IBM Watson, which he had used for several other projects. Watson, an artificial intelligence system, was designed to answer questions, so it seemed like a strong fit.
To train the system to answer questions correctly, Goel fed it forum posts from the class’s previous semesters. This gave Jill an extensive background in common questions and how they should be answered.
Goel tested the system privately for months, having his teaching assistants examine whether Jill’s answers were correct. Initially the system struggled with similar questions such as “Where can I find assignment two?” and “When is assignment two due?” Goel tweaked the software, adding more layers of decision-making to it. Eventually Jill reached the point where its answers were good enough.
“I cannot create chaos in my classroom. Jill had to be almost as perfect as a human TA or I am,” Goel said.
The system is only allowed to answer questions if it calculates that it is 97 percent or more confident in its answer. Goel found that was the threshold at which he could guarantee the system was accurate.
There are many questions Jill can’t handle. Those questions were reserved for human teaching assistants.
Goodbye TAs
Jill will not eliminate the need for all TA’s just 97% of them.
The real test comes next year. Students will ask all kinds of questions in attempts to figure out if they are dealing with a robot or not.
“Michelle Tutelage” better be prepared for ridiculous questions, and so must the real TAs. Precisely consistent answers and tones may help reveal the robot. I suspect human answers may vary a bit more than robots even for identical questions.
Perhaps Goel has already figured that out, or perhaps he will make a program change after reading this.
Secondly, if the same TAs help each year, then all of the TAs’ names must be disguised. So, why not just randomly assign the TAs?
“A really fun thing in this class has been once students knew about Jill they were so motivated, so engaged. I’ve never seen this kind of motivation and engagement,” Goel said. “What a beautiful way of teaching artificial intelligence.”
This kind of program will eliminate the need for TAs but it will also lower the cost of education making it more affordable.
Goel is forming a business to bring the chatbot to the wider world of education. “To me this is a grand challenge,” Goel said. “Education is such a huge priority for the entire human race.”
Mike “Mish” Shedlock
