The way we draw can reveal a lot about us. In fact, researchers have found that sketches can reveal how old a child is, if a person has had a stroke and more. Dr. Tracy Hammond, professor in the Department of Computer Science and Engineering and director of the Sketch Recognition Lab, researches these human motions to help improve strategies in a number of fields, including education.
In this episode, Dr. Hammond talks about her work and how it's impacting machine learning systems to better empower people.
The way we draw can reveal a lot about us. In fact, researchers have found that sketches can reveal how old a child is, if a person has had a stroke and more. Dr. Tracy Hammond, professor in the Department of Computer Science and Engineering and director of the Sketch Recognition Lab, researches these human motions to help improve strategies in a number of fields, including education.
In this episode, Dr. Hammond talks about her work and how it's impacting machine learning systems to better empower people.
The way we draw can reveal a lot about us. In fact, researchers have found that sketches can reveal how old a child is, if a person has had a stroke and more. Dr. Tracy Hammond, professor in the Department of Computer Science and Engineering and director of the Sketch Recognition Lab, researches these human motions to help improve strategies in a number of fields, including education.
Hannah Conrad:I'm Hannah Conrad, and my co-host, Steve Kuhlmann, and I met with Dr. Hammond to learn more about her work and how it's impacting machine learning systems to better empower people. Welcome to SoundBytes. This is Engineer This!
Steve Kuhlmann:We figured we'd start with a pretty broad question. What is sketch recognition?
Dr. Tracy Hammond:Sketch recognition is the automated understanding of your sketches. So in general, our lab does more than just sketch recognition. We do any sort of understanding of messy human motions, and sketching is one of them. We also do activity recognition, which again, it's like, maybe it's your wrist following a sort of a path or a drawing. So it's really looking at the paths and trying to understand them. It's looking at it from a variety of perspectives. So one perspective is that perspective of, you know, what did they draw, which is pretty straightforward. But then there's this other perspective of what are they intending to do? So what are they intending to draw for, still speaking about just sketching, but we can also learn a lot of other things about them. So like, looking a little bit about the forensics about them. We can identify like how old you are for kids, whether you've had strokes by sort of your, your sketching features. And so you do get a lot of interesting things that you can get sort of underlying that aren't obvious.
Hannah Conrad:And how did you get interested in these topics?
Dr. Tracy Hammond:I was a math major undergrad, but I also wanted to do medicine, but in particular, I wanted to do like neurology, and I actually entered into computer science kicking and screaming, so I had no, I had zero, zero interest at all in computer science. I didn't want anything to do with that black box, like it was this thing that you couldn't see into. And so I only agreed to take a computer science class like the second semester of my sophomore year, because my friends who knew me very, very well, were like, just take a course. I was like, pre-med, and I was like, I'm going to do neurology, or I'm going to do math. I finally took a class, and I came home and I said, "Oh, my God, computer science is mathematics, but with instant gratification." And so I was like, alright, well, then I guess this is where I'm going to be going. And then I learned that I could sort of try to imagine things that the brain might be doing. And I could code them in, I can try to find out right away if those ideas would work. And that led me to AI.
Steve Kuhlmann:There is a lot of variety that goes into the Sketch Recognition Lab. What are some of the real world problems that you and your students are working on?
Dr. Tracy Hammond:So we have a couple of systems that are already in classrooms right now and being used by multiple universities. So our mechanics software, it automatically corrects handdrawn homework problems. And it's specifically doing that for statics problems, like freebody diagrams and trusses. And the idea behind it is that there's lots of systems that can tell you if you have the right answer, or if, can tell you if you have written down the right equation, but none of them right now check your, except for ours, check your the process. Did you set up the problem correctly? And so it's really important, that sketch is incredibly important in solving the problem. If you don't have the sketch right, you're not going to get the answer correct. And so it, it's fundamental to be able to give this feedback on the process as well as that final answer. And we also give it on the equations and things like that. But we really focus on that first sketch a lot, making sure that that is correct, that they have framed the problem correctly. It's super important, this idea of having them sort of draw and produce the answer because you actually learn more from producing the answer. You learn a lot from particularly a short answer quiz, or, or a quiz where you have to produce the answer. Right now, almost all of our learning is done through like multiple choice, and that can actually hurt learning, which is because you, you end up remembering the wrong answer. If you choose that wrong answer, it gets further ingrained into your brain, which is not what you want. This is why it's so important for them to be solving the problem, us watching them solve the problem, and they're actually, you know, going through all the steps rather than just clicking whatever the ABCD answer is. We don't appreciate the value of quizzing as much as we should. When you're taking a quiz, you learn more, and there's literature on this, from taking a quiz than you do from listening to a lecture or from studying.
Steve Kuhlmann:Wow.
Hannah Conrad:Right? And where do we spend the most of our time in our classroom? And like, how is our educational system set up, right? We learn more from quizzes. So the whole idea of this mechanics software is because it can give this immediate feedback on your drawing, you could even do like a quiz at the start of every class. Then the instructor would get an idea of where you are, as well. And you learn from this practice. I, in all of my classes, I get a quiz every single day, because I've learned this and it's, so we need more things that can facilitate having a quiz every day, so you can facilitate the increase learning part. And that's where we need sort of this automated understanding of sort of these messy human inputs in sketching. And that system is used at Georgia Tech, obviously Texas A&M, San Jose State, LeTourneau and Texas State. So all four of, the five of those universities are using that in multiple classrooms now. So it's, it's really exciting. And I bet that's fantastic in classrooms that are, you know, 300 students, you know, these massive classrooms that otherwise a quiz would be impossible.
Dr. Tracy Hammond:We do a lot of control and test studies, we're only, you know, random half of them use it. And the students often beg to be able to use it after they're in the control, so they can still use that study for the exam. And we do let them do that. So that, that's really fulfilling that they, that they want to be using this and that they'd prefer to use this than the traditional methods.
Hannah Conrad:It's putting the application like right there in front of you, instead of, like you said, just taking a guess at a multiple choice.
Dr. Tracy Hammond:Yeah. Another system we have is called Sketchtivity. This is being used at TAMIU, Georgia Tech and San Jose State. And this one actually teaches you how to draw in 2D perspective, which for the most of us is drawing in 3D on a piece of paper. And Georgia Tech is of the opinion that every single one of their engineers needs to know how to draw.
Steve Kuhlmann:Hmm.
Dr. Tracy Hammond:And that's because they want when they're at the dinner table with their client, and they come in with their beautiful CAD drawing. And the person says, "You know, I really wish it were like this and not like this." The person is like, "Oh, yeah, how about like this?" And they sketch it up right there and are able to sort of innovate in any space, rather than having them saying,"You know what, I'm going to go back to the office, and I'll bring you back on Monday the updated CAD drawing," then the person next door hears them, who's on the other table says,"Oh, you know I just heard you're talking about. Is this what you're talking about?" And then suddenly they've lost that client. And so they really want them to be able to innovate successfully. And they think that one of the really important ways is for them to all to learn how to draw. And so we have that system that teaches them how to draw. It gives them immediate feedback on their drawing. It starts them with sort of simple shapes and then goes further into sort of complex shapes, eventually getting to like 3D cameras and other types of, city streets and really getting, giving them a lot of practice in their 3D drawing. And the other benefit that's come out of it is it significantly increases their spatial cognition skills. So just by using it a little bit. So you end up getting like a double benefit, like double whammy from learning how to sketch, you not only increase your sketching skills and visual communication skills, but you also increase your spatial cognition skills.
Steve Kuhlmann:That's fascinating.
Hannah Conrad:How important is immediate feedback in programs like that?
Dr. Tracy Hammond:That's a really good question. It depends. So we have two different stages of sort of creation and innovation. And I would say in the beginning phase, you don't want to give a lot of feedback. When you're trying to innovate and you're at the beginning of that brainstorming phase. In fact, sketching allows you to have this ambiguity that allows these like Deleuzian lines of flight. And what that means is that you can, like, accidentally misinterpret your own work and have a great idea from it. And so, at the beginning, you want to have it be really freeform, and you really want to, like allow, you know, especially when you're creating you want, you don't really want much feedback at all if you're trying to be creative. And then after you've produced lots of ideas, that's when you sort of want to pull things in a little bit. And that's when the systems can start to do some processing under the hood saying, you know, this is not really possible right over here. You know, maybe think about a little, you know, start to giving some some of the technologies that we can add.
Jenn Reiley:Howdy! It's your producer Jenn Reiley here with a quick note. I just wanted to add some extra information on Dr. Hammond and her research. So first off, if you didn't think Dr. Hammond was busy enough with her research, she also serves as director of the Institute for Engineering Education and Innovation. She's the chair of engineering education faculty, and she's the secretary for the Texas A&M Faculty Senate. I also wanted to know that her projects have been funded by a variety of institutions, including the National Science Foundation, the US Department of Defense, Google, Microsoft, Rockwell Collins, Adobe and more. Well, I don't want to keep you, so let's get back into the interview.
Steve Kuhlmann:How would you explain how this work is happening to a general audience?
Dr. Tracy Hammond:Here, let me give you an example of how we do ground things. So you know how I was talking about we could tell how old you are? Well, we have features that show up at various ages. Two of them have to do with distinguishing a, an arc from a corner, and it could look indistinguishable to the human eye. But actually, when we zoom in, you'll see that we as adults overturn on the corners. We actually like go too far, just like remember before power steering, you know, you do that same thing in the car, you'd sort of like, overturn and then cycle back. But when you're just doing an arc, you wouldn't actually do that. You would just sort of remain more constant. And we do the same thing with our pen. It's low level; we don't even think about it. And there's two features; one shows up at age four and one shows up at age six. Now, there's another feature that is really intuitive that you'll totally understand which is hooks. When you're drawing an 11. Right, you draw an 11, you draw, let's say, you draw down the first line, and then you're going to draw the next line down, but actually, you leave a little hook at the bottom of your first line. And it's pointing up to the second line, and you have another little hook at the top of your second line that's pointing back to the first line that you were drawing. And so it's these little artifacts that are left on the page, you don't see them, usually you have to zoom in, but in digital, it's super easy for us to see them because we can zoom in as much as we want. And these are evidence of low-level processing, that you're just... you're cognitively thinking about the next stroke before you finish the first stroke. And so your mind is working faster than your pen. You lift up your pen, but you're sort of turning as you're going to the next stroke, and so you leave that little like the tiny little hook, and those show up at age five, which is because that's the time when you're starting to think about the next stroke before you're done with the first one. You get these sort of little artifacts, and I like that we can, we can take them and they make sense cognitively. And then we do artificial intelligence. Remember how I didn't like that black box? Right? I still don't like that. I need to know what the computer is doing at all times. And so we do lots of different flavors of machine learning. But the end flavor always ends up being one that I knew exactly what it's doing. You know, some of them are more black box than others, some of these algorithms. And so we can use those to maybe do feature subset selection to reduce down to a certain number. But then we try to tease things out more and more to say, how much can we get to like an algorithm that someone in psychology would understand. We use these machine learning techniques to get us started but where we hope to end up as much as possible is things that are very, we call this sort of explainable AI is sort of a buzzword going around right now. But that's super, super important, not just because it helps us better understand the algorithms, and what it's doing and making sure that it's grounded in the theory. But AI can be incredibly biased, right? And so we have this huge problem that bias in is bias out. In fact, it's, it's exacerbated when you put it through machine learning program. And so if you want a system to act properly, you need to know what it's doing. I'm not comfortable with any algorithm that's produced unless I can see what's, what's going on.
Hannah Conrad:It is really interesting that you started really interested in medicine and neurology, and so you found medicine and neurology, in a sense, in computer science, looking at human patterns.
Dr. Tracy Hammond:Yeah.
Hannah Conrad:It's almost just, you found that little niche. What's it like working in an area of engineering that is so filled with psychology?
Dr. Tracy Hammond:Oh, it's so wonderful. I mean, I'm very interdisciplinary myself. And I guess if I had to have another job that I wanted to do is probably to be a psychologist or something like that. Because I am fascinated with how people work and how people interact. And I do feel that the intersection of these disciplines is really where you get that little magic little hooks and nooks and crannies and things that do make a huge difference in the world.
Steve Kuhlmann:So when you look at your own personal journey, what advice would you give to students?
Dr. Tracy Hammond:Know yourself, you know, I do think that self-awareness of what you're interested in, what your talents are, what your natural strengths are, and do that and do it with all your heart. That's, that's what I would say, you know, figure out what your're, what you're good at and what you really enjoy so that work is fun. I love, I love work. I love my job.
Hannah Conrad:We hope you enjoyed this episode of Engineering SoundBytes. Make sure to subscribe to stay up to date with what's happening within Texas A&M Engineering. Until next time, stay safe and Gig 'em.
Steve Kuhlmann:The views and opinions expressed in this podcast are those of the hosts and guests and do not necessarily reflect the official policy or position of The Texas A&M University System. SoundBytes is a part of the Texas A&M Podcast Network. To find more official Texas A&M podcasts, go to podcast.tamu.edu.