Guest: Kevin Schofield - Microsoft Research
Host: Robert Scoble - The Scoble Show

Robert Scoble - The Scoble Show
So, who are you?

Kevin Schofield - Microsoft Research
Who am I? I'm Kevin Schofield.

Robert Scoble - The Scoble Show
I'm still with UB4 on Channel 9, but this isn't Channel 9 anymore.

Kevin Schofield - Microsoft Research
This is not Channel 9 anymore. I'm in Microsoft Research and I handle almost everything for MSR, that does involve actually managing researcher, so...

Robert Scoble - The Scoble Show
Yup.

Kevin Schofield - Microsoft Research
Tech transferring of product groups, talking to customers, or academic programs, all sorts of those stuffs.

Robert Scoble - The Scoble Show
Why am I here, you guys just...

Kevin Schofield - Microsoft Research
Why are you here?

Robert Scoble - The Scoble Show
I don't know. I miss the rain.

Kevin Schofield - Microsoft Research
No, this is actually the cool thing about doing an MSR is, we're like the world's largest computer science department, we're very openly, we publish papers, we talk about the stuff we do, we love talking about the stuff that we do.

Robert Scoble - The Scoble Show
But you guys just celebrated you 15th birthday.

Kevin Schofield - Microsoft Research
We just celebrated our 15th anniversary, 15 years, we started in 1991.

Robert Scoble - The Scoble Show
So let's go in. You got the blue badge, I don't have one of those things.

Kevin Schofield - Microsoft Research
I'll get the blue badge. Let's actually go through, maybe we can cut through the cafeteria and go round to our other building next door, and you get to see all the deep, dark secrets of the Microsoft Research cafeteria.

Robert Scoble - The Scoble Show
Oh, do we really. And (Inaudible) belong with us too.

Kevin Schofield - Microsoft Research
Super secret high tech potato chips, and oh yeah, oh yeah very, very secret raps (Inaudible). We have the coolest cafeteria staff in any cafeteria. Smile, wave. They rock, they're totally, totally awesome.

Robert Scoble - The Scoble Show
You're taking me out today.

Kevin Schofield - Microsoft Research
I'm taking you out, yeah, I should go out and just briefly (Voice Overlap).

Robert Scoble - The Scoble Show
I should know, we're going to Eric Horvitz office, right?

Kevin Schofield - Microsoft Research
We're going to Eric Horvitz office. So one of the -- I'll just tell you a quick story here, so I'm trying to put together sort of an event around our 15th anniversary, and invite some people and then show some stuff. Yeah, its nice little rainy day in Seattle. Its been amazing (Voice Overlap).

Robert Scoble - The Scoble Show
I miss the rain. That's why I came here.

Kevin Schofield - Microsoft Research
We're going to have something special for you tonight.

Robert Scoble - The Scoble Show
You've got to open this door.

Kevin Schofield - Microsoft Research
So, its really -- I spent a couple of months of the summer trying to figure out, how do we tell the story of what we've done over 15 years in Microsoft Research. So we actually went and did bunch of math and we started counting stuff, and we found that, we've published over 37,000 papers in conferences over 15 years, that's like, almost one paper a day for every day that -- for every business day in 15 years, which is just amazing. If you figure, if it's like an average of ten pages per paper, that's like almost 200 reams of paper. So we figured, okay, how do you represent that? So we actually got office depot to bring over 37,000 pieces of paper, and we got a shot of Rick Rashid, our Senior Vice President sitting there on a living doc with 37,000 pages of paper, which is pretty amazing. Okay, right behind you, this is another big thing, we did sort of represent what's, what we've really done, or what's sort of been the path for 15 years of Microsoft research. We got this whole wall, maybe you can get the shot here.

Robert Scoble - The Scoble Show
Rocking.

Kevin Schofield - Microsoft Research
This whole wall here started in 1991. We used to love this whole picture of Eric here from 1991, back when big glasses were in. Then if you turn around and look, it's sort of stretches down the wall here for 15 years of everything that we've done. Early graphics work that we're doing, we've been doing all along. Jim Grey starting BARC back in 1995, there's Roger Needham in our Cambridge lab, work that we started doing with SQL Server. It just goes on and on - actually, that end was really hard to do, the first five years, because we didn't -- one of the things you learn (Inaudible). Oh, we didn't keep good records for the first five years. We were so busy doing great work and research and papers and everything that -- there's Tony Hoare doing things on Quicksort, works in our Cambridge lab, he joined us...

Robert Scoble - The Scoble Show
I know that guy.

Kevin Schofield - Microsoft Research
Yeah, I know... A.J. Brosh, working calendaring stuff, Jian Wang in our Beijing lab working on Ekeing (ph) and handwriting, Marc Smith on our social computing group.

Robert Scoble - The Scoble Show
Is there one thing that you think is the defining moment for research so far in the first 15 years?

Kevin Schofield - Microsoft Research
So, somebody asked Rick that, when we did a processing last week, and his sort of defining sea change moment was around 1996, summer of 1996, when at SIGGRAPH, the big annual graphics conference, we had...

Robert Scoble - The Scoble Show
Keep going.

Kevin Schofield - Microsoft Research
We had about 20%, I think its about 20% of the papers on the top, which was just unheard of any research lab to get that large of the percentage of the papers, and like on of the really big highly peer reviewed conferences. He sort of said, right after that happened like, later that day, when he find out that he had a meeting with Nathan Myhrvold, and two of them sat down, and they just spontaneously started laughing, because it felt like, okay, we've actually arrived, this crazy thing that we started doing five years before, its real, and we're making this huge impact on the field. Now, when you look across, OSDI, The Operating Systems Conference and SIGGRAPH and SIGIR, the big search and information retrieval one, tied to the user interface. We're seeing that, we have a huge amount of papers at every single one of those conferences, and the fact that -- it's amazing to do that for one conference, to run the table of all this conferences, its almost an embarrassment of riches. We should go upstairs.

Robert Scoble - The Scoble Show
Lets go. So what's happened since I've been here - it's been a few months since I've seen you.

Kevin Schofield - Microsoft Research
What's happened? Well, let's see, apart from the 15th anniversary, we just had a ton of researchers, super, super busy, we're always, sort of at the end of every school year (Voice Overlap).

Robert Scoble - The Scoble Show
What you do here?

Speaker
Record Research.

Kevin Schofield - Microsoft Research
He eats lunch.

Speaker
Yeah, I eat lunch though. Text mining and research.

Robert Scoble - The Scoble Show
Oh wow! That's important, we'll come back and see you later.

Kevin Schofield - Microsoft Research
We'll come back and see you later. So let's see, we had 220 interns here over the summer, right. We didn't have enough office space for 220 interns, so we took...

Robert Scoble - The Scoble Show
You had 220 interns just in research?

Kevin Schofield - Microsoft Research
Just here in Redmond.

Robert Scoble - The Scoble Show
You mean Microsoft overall, or just in research?

Kevin Schofield - Microsoft Research
Just in research, just in MSR, in this building, and the next building over, 220 interns.

Robert Scoble - The Scoble Show
You must have had some good Friday afternoons.

Kevin Schofield - Microsoft Research
So if you look, that used to be a lounge. We love our intern program, its just this huge wonderful thing, we've got all this great energy and enthusiasm in the lab over the summer doing this, but we didn't have enough office space for this, so we took every single lounge in our two buildings and we converted them all to something cubical. Let me just make sure I'm going to (Voice Overlap).

Robert Scoble - The Scoble Show
Isn't Eric up here in the corner?

Kevin Schofield - Microsoft Research
Yeah.

Robert Scoble - The Scoble Show
I thought I had it right.

Kevin Schofield - Microsoft Research
Yeah, so it's like, every spare place we could find where we could actually (Voice Overlap).

Robert Scoble - The Scoble Show
Now the people don't know who you are, but you're in charge of moving stuff out of the research and into real products, right.

Kevin Schofield - Microsoft Research
I spend a lot of time working on how we get...

Robert Scoble - The Scoble Show
So you know everybody on the research team?

Kevin Schofield - Microsoft Research
I know a lot of them, we're so big now, we don't know everybody. Hey, there is Eric.

Eric Horvitz - Microsoft Research
Hello.

Robert Scoble - The Scoble Show
What's the coolest thing you've seen?

Kevin Schofield - Microsoft Research
What's the coolest thing.

Robert Scoble - The Scoble Show
What's the top secret close thing, because the guy behind me, which you can see it, is probably doing it.

Eric Horvitz - Microsoft Research
Boy! You dress the same way, you don't wear a suit now to work daily?

Robert Scoble - The Scoble Show
No, they kicked me out of the country club in Silicon Valley the other day, if you read my blog.

Kevin Schofield - Microsoft Research
Do you want me to give him the mike?

Robert Scoble - The Scoble Show
Yeah, that will be great.

Kevin Schofield - Microsoft Research
This is Eric Horvitz. He is a senior researcher and runs our Adaptive Systems and Interaction Group, and I'm going to give the mike to him.

Eric Horvitz - Microsoft Research
Thanks Kevin.

Kevin Schofield - Microsoft Research
He is way, way cooler and smarter that I am.

Eric Horvitz - Microsoft Research
Yeah right.

Robert Scoble - The Scoble Show
He's way cooler and smarter than you then. I know you're supposed to be standing there.

Eric Horvitz - Microsoft Research
We need to get this what if shirts for everybody in my group actually. They're very, very cool.

Robert Scoble - The Scoble Show
Yeah, I like these.

Kevin Schofield - Microsoft Research
It's actually MSR work, sexy MSR work.

Robert Scoble - The Scoble Show
You've got to put that on a Valleyschwag, yeah.

Kevin Schofield - Microsoft Research
Actually, I'm wearing my (Voice Overlap).

Robert Scoble - The Scoble Show
Those would be, if you send them into - Valleyschwag's a company down in Silicon Valley that sends schwag around the world for 15 bucks a month, you get all sorts of schwag, but that would be like, that would get people a sign up just for that shirt, because...

Kevin Schofield - Microsoft Research
I like it. We're actually going to do another -- we've got a small handful. Actually, I think I can get you one of these before you leave today.

Eric Horvitz - Microsoft Research
So Kevin asked me just yesterday, he said, guess what! remember that guy Scoble, he's still doing this stuff, and he said, he's coming by to see (Voice Overlap).

Robert Scoble - The Scoble Show
I've got a bigger and heavier camera now.

Eric Horvitz - Microsoft Research
Oh my God, this guys really know how to treat you well.

Robert Scoble - The Scoble Show
Yeah, we got VC money, Bill Gates, he didn't have real money, the VC money, they buy stuff like this.

Eric Horvitz - Microsoft Research
Maybe they're less -- little bit more free with their money. We have to run here a little bit careful and directed.

Robert Scoble - The Scoble Show
Yeah, that's one way to put it.

Eric Horvitz - Microsoft Research
Not to install your VC company (Voice Overlap). Anyway, let's just sort of take you around and show you a couple of -- three new things, three new things today. People work that you've seen before, I think on your Channel 9 in the old days. The first person I want to show you...

Robert Scoble - The Scoble Show
But we should talk about who you are first, because my new audience might not have seen those earlier. You have the most patents here at Microsoft, right?

Eric Horvitz - Microsoft Research
I hate when people mention that.

Robert Scoble - The Scoble Show
Why?

Eric Horvitz - Microsoft Research
It sounds like we're a patent house or something. As I tell people, in our group, we have a very high regard for patents, we only patent things, where we think that, at a leading campus in our field, we really impress and surprise our colleagues in the field, so its -- there's a whole story here, but read about Ben Franklin and what he said about patents, it's very interesting.

Robert Scoble - The Scoble Show
What did he say?

Eric Horvitz - Microsoft Research
Well, he and Thomas Jefferson, who actually originated the patent system. It turns out, had some very good ideas that are set at the basis of it all, which is sharing. The idea back then was that people who had really good ideas were not necessarily incented to share them, they'd rather keep them trade secrets at companies, and the whole patent system was set up initially for sharing concepts and ideas. The idea was, you trade a little bit, you get little bit back for your interest and novelty, but you actually shared out the concept with the world, and others could build on them, which was kind of a nice thing. So we actually think a lot about that, we think about academic publication, we think about just publishing and showing ideas, without necessarily patterning them, so on. It's a broad portfolio of the kinds of things we do to get ideas out in the world.

So anyway, this group is called Adaptive Systems and Interaction. We do quite a bit of research in the search area, browsing, we do quite a bit of work in machine learning, so we're one of the teams at Microsoft Research that's very interested in the prospects that, computing might change dramatically some day if we see further maturations of technologies that can do decision making autonomously, and reason about what you want to happen next, and your goals, to allow you to maybe someday ask office to just, show me that pivot table like he did last time, but this time, show it to me with the computation highlighted in a new way, natural language and applications, speech vision.

Kevin Schofield - Microsoft Research
So, in another words you really want to make computers smarter, a lot, lot smarter.

Eric Horvitz - Microsoft Research
We think that computers might someday be sort of like talking to a very bright person you're working with. Like Kevin just now, you get the sense that, you know Eric, you can communicate this much more crispy to this audience, and literally this is called mixed initiative interaction, because he's helping me out, and a computer should someday also work with humans in some ways (Voice Overlap).

Robert Scoble - The Scoble Show
You think we're going to have a computer embedded in our head someday that's going to say, hey.

Eric Horvitz - Microsoft Research
No, but people do think about the idea that its not just sitting on a desk but its a big opportunity for software, computing will be kind of in the course of daily life, somehow where the computer system understands your goals and needs and recognizes how it can best help you -- potentially to remind you of things you might forget, like what was Robert's wife's name again, we had that party, and just sort of time in and so on, and there have been some prototypes of these things, and to date they're rather clunky, but the vision is there.

Kevin Schofield - Microsoft Research
Let's get you some close up.

Robert Scoble - The Scoble Show
So first of all, you don't have any implants on you?. You were the most connected guys at Microsoft Research, but neither of you guys have implants or ...

Eric Horvitz - Microsoft Research
I have the PC pocket implant.

Robert Scoble - The Scoble Show
Pocket PC implant.

Eric Horvitz - Microsoft Research
Now, Robert you met Raman Sarin in the past, when you came in here and saw some really cool stuff, about maybe a year ago now, and some of that technology led to some of the search techniques we're shipping with Windows Vista, desktop search. You also saw, I think SmartPhlow and JamBase, the traffic forecasting work we were doing, and now Raman continues to work in this area, a very promising area of small devices, and what can be done with them. Today I'll tell you a little bit about advances in sharing technologies, so Raman Sarin who is a developer on our team.

Raman Sarin - Microsoft Research
Hello.

Robert Scoble - The Scoble Show
How are you doing?

Raman Sarin - Microsoft Research
Hi Robert, how's it going? Yeah, there's a certain amount of hardware in this room.

Robert Scoble - The Scoble Show
Is it so hot in here, because there's so much (Voice Overlap).

Raman Sarin - Microsoft Research
Actually with the LCDs, not so much of a problem. I can remember way back when I use to have CRTs that, yeah that would be a huge problem, and these are all laptops, so they don't really do much, and then this display is for doing tablet, its basically a potent tablet. What I'm going to show you is this product prototype...

Robert Scoble - The Scoble Show
First of all, who are you?

Raman Sarin - Microsoft Research
Who am I? I'm me. My name is Raman Sarin and I'm a developer. I work for Ken Hinckley who works for Eric Horvitz, and we have a small sub groups under Adaptive Systems Interaction called the Adaptive Systems Interaction Focus Group, and we focus on interaction prototypes for tablet PCs, some prototypes for search and retrieval and some prototypes for smart phone.

Robert Scoble - The Scoble Show
I think you're one of the few Microsoft employees that could be the guy from Engadget, I got tour of his house the other day. I think you have more gadgets in your office.

Raman Sarin - Microsoft Research
Well, I hope I don't have more than he does. I do have the records at Zune and...

Robert Scoble - The Scoble Show
We just interviewed them an hour ago.

Raman Sarin - Microsoft Research
Yeah, very cool, very cool. I'm definitely going to get a good piece of hard work. So, what about 'Blue Rendezvous'? So 'Blue Rendezvous' came about, because I was in the car with my wife, who also has a smart phone, because I'm a gadget guy, and I got her one, and I wanted to send her an address of one of our friends, and I discovered that it wasn't really an easy thing to do. So I though, well, why isn't it easier to share information between these devices, I mean this thing right here is got a 2 gigs of storage in it, and its like better than most of the PCs I had until about 2000. So, why not make it really, really easy to share between these, without having to do what I called the infrared tango, where you try and keep the ports lined up and sort of move back and forth. So what I've done -- can you see that?

Robert Scoble - The Scoble Show
Yeah.

Raman Sarin - Microsoft Research
So this app, basically you're going to start out, and I'm going to touch a button, and I'll just push one, doesn't really matter which one I push, and over here on this device, I'm also going to push one, and then here I'm going to hit fine partner. What this thing is doing is, its now going out and its looking for any Bluetooth discoverable device that understands the Blue Rendezvous Protocol, and when it finds such devices, it's asking, hey, did somebody just press the number one on you, and when it finds somebody that says yes, somebody just press the number one on me. It goes, well, you're the guy I want to talk to. Now I have this nice little menu popped up where I can pick what it is.

Robert Scoble - The Scoble Show
I just got a new BMW, and hooking up my Bluetooth phone to the BMW, it took somebody who was trained. I couldn't even do that.

Raman Sarin - Microsoft Research
So, you understand why this is a problem.

Robert Scoble - The Scoble Show
Can you guys sell this technology to the car company, so we can just walk into Microsoft...

Raman Sarin - Microsoft Research
Well, we can sell it to anybody who wants to buy it. So yeah, from here I can now just -- I'm going to tap this little icon of a camera, just actually make a tap there. Tap, screen seems to now be out of the line, and I'm going to pick this picture of this goofy looking guy here, and over here what you're going to see, is it says, do you want to receive that picture? So if I say yes, you'll see it transferring, and there it is, now I have it on this...

Speaker
The new confirmation signal is on there.

Raman Sarin - Microsoft Research
So by the same token I got files, and I've got contacts, and then the last thing I have is this, down here it's this thing called business card swap. What it does is, in the program you can define this business card as me, so if I were to meet somebody like, let's say I wanted to swap business cards with you, we could just run this and hit that button, and it will say, exchange business card. The cool thing it does is, unlike just a contact swap, it is bidirectional, I'll get yours, you get mine, but where you define the business card, you can also define which fields go across. So when I send you my business card, you're not going to get my home address and my home phone number and my home email address, all of which are in there, which usually people sort of accomplish this by having multiple copies of their own contact in there, and then they end up sending the wrong copy all the time, so we're trying to just avoid that little problem. When you exist the application, what will happen is that, if the Bluetooth radio was off, it'll go back off, if it was on, it will stay on. There was no pairing that took place. You're not going to find that these two devices are still paired together, so somebody can't comeback later and reconnect you without your knowledge, it was a one shot deal.

Robert Scoble - The Scoble Show
Now, you have to load a little software in here to do this?

Raman Sarin - Microsoft Research
Yeah, there is an app that I wrote called Blue Rendezvous. The technology really is the connection stuff not so much the transfer, but the transfer is a useful thing that you can do with that connection that we just looked at.

Robert Scoble - The Scoble Show
Can people in the real world get this and download it?

Raman Sarin - Microsoft Research
Not yet, we're working on that.

Robert Scoble - The Scoble Show
So, you'll put up on the research site sometime?

Raman Sarin - Microsoft Research
Hopefully, yeah.

Robert Scoble - The Scoble Show
Okay. Very cool, what else have you guys -- what else have you been working on -- you do some really cool stuff.

Raman Sarin - Microsoft Research
I do. I have a great job, it's a lot of fun.

Robert Scoble - The Scoble Show
People should know that you wrote the cool little traffic app that I -- when I was here in Seattle, I would use it everyday to figure out which route I should take to work.

Raman Sarin - Microsoft Research
Yeah, and Eric's team is still working on doing some really cool extensions to that. You know, we're doing some really cool stuff, and all I can say is, come back in a couple of months.

Robert Scoble - The Scoble Show
Okay.

Raman Sarin - Microsoft Research
And we'll take to you about it then.

Robert Scoble - The Scoble Show
I will, thanks.

Kevin Schofield - Microsoft Research
I'm going to drive you down the hallway there.

Kevin Schofield - Microsoft Research
Have you met Robert Scoble before?

Robert Scoble - PodTech
Hey, how are you doing?

Ed - Microsoft Research
Hi, there, hi.

Kevin Schofield - Microsoft Research
That's another researcher in our office.

Eric Horvitz - Microsoft Research
I'm just going to mention him briefly that, we're very interested in attention -- attention modeling in our group and now we're going to show you some of the leg breaking stuff we're doing in terms of, getting a sense for gauge patterns during search and browsing applications. Patterns of looking at screen and browsing search results and what that all means and Ed, why don't you just get ready (Voice Overlap)....

Ed - Microsoft Research
We'll just show you what we're doing So, basically what we're trying to do here.

Robert Scoble - The Scoble Show
Hold on, just one second. Let me put the mike up here and get this tripod.

Ed - Microsoft Research
Well, yeah so I'm going to try to find a good spot here that I wanted to show you what we're looking for, we'll just stop it there. So, the idea behind this is that, we've done a series of studies to try to understand how people search, for both web search and for personal search, and other kinds of things, and one of the ways of getting a peek at how people are searching is, well I mean, the obvious way, like Google, and Yahoo Search et cetera that Yahoo! uses, is that they have massive and massive amounts of click through data. So, you know what people are clicking on, you vary things around, and you watch what they do. Now the problem with that is that you don't understand what people are doing when they're not clicking.

Robert Scoble - The Scoble Show
Yeah.

Ed - Microsoft Research
So, there is no information there. So what we tried to do is to say, okay, lets look at what people are actually looking at, and so that's actually some thing that some people have been doing for little while, you use an Eye-tracker, and you just watch what people look at. Now the next piece to that is, let's say, let's start changing what we show people, and see how that affects how they look for stuff. So one of the things that we've done with this experiment was, we went through and we started changing the tasks that people had, so we had some very explicit kinds of informational and navigational kinds of tasks, these are sort of standard things people do on the web, and then we vary things like, oh let's change where the number one result is. Let's start moving around how the ranking works, or better yet, let's start changing the kinds of metadata we show with it, and watch how people look for what they're doing. So, I'll just show you here a little...

Robert Scoble - The Scoble Show
So what's this device here?

Ed - Microsoft Research
This device here, this is a Tobii Eye-tracker. This is a thing that's made in Sweden. It's one of the top of the line kinds of units here, and what it does...

Robert Scoble - The Scoble Show
How does this work?

Ed - Microsoft Research
Yeah, so what it does is, that it has -- there are some IR LEDs in here, and there's a camera here. What it does is, it looks for the bright spot of the pupil. So that the reflection -- so whenever you take a photo, you get that big red eye thing, all that is, it's the light reflecting back of off the pupil in the cornea. So this does that, but it does it in IR, so that people can't see the lights flashing at them. As you move your eye around the screen, it's able to detect what's on axis and what's not on axis, and it can infer where the eye is looking on the screen, because it's registered to all the position there, so you calibrate it, and then you basically are able to see what people are looking at.

Robert Scoble - The Scoble Show
That's awesome. So, it follows your eye.

Ed - Microsoft Research
Yeah, I'll show you a movie, there's somebody actually doing a -- this is from our experiment, so here what you're seeing is, there task is that...

Robert Scoble - The Scoble Show
So this isn't on right now is it?

Ed - Microsoft Research
This is just a movie of somebody else.

Robert Scoble - The Scoble Show
Okay, because I think if you turn that on, the camera actually sees IR, so you're (Voice Overlap).

Ed - Microsoft Research
It would be flashing, you would see some flashing stuff here. We might be able to turn it on here in a minute, but I'll show you. So, the deal is that -- so what that person's supposed to do here is that, they're supposed to find when the Titanic set sail for it's only voyage, and what port it left from.

Robert Scoble - The Scoble Show
Yeah.

Ed - Microsoft Research
And then the query is just Titanic. So, what you see is, this little dot that's moving around the screen is their gaze point. and you see their queries, so they clicked Titanic, and their eyes are moving around, the size of the thing as it moves, as it gets bigger and bigger is (Voice Overlap).

Robert Scoble - The Scoble Show
Duration...

Ed - Microsoft Research
Yeah, its duration of looking. So you can see that they're reading. Now, they're reading a bunch of the details about the Titanic, or moving down to the next piece, they clicked on it up there, and then they're like waiting, oh, what happened. They go up, and then -- so now they clicked on that piece, so now then they're going to the Titanic piece. They're reading along, and you can see as they're reading, what they're looking for?

Robert Scoble - The Scoble Show
Now it doesn't look like it's completely calibrated into the text, is that...

Ed - Microsoft Research
It's a little bit off. You get some noise in there, and so what you have to do, let me find another spot in here, maybe down here, around 14 piece...

Robert Scoble - The Scoble Show
So this is how people create Heatmaps, right?

Ed - Microsoft Research
Exactly

Robert Scoble - The Scoble Show
They run hundreds of people through...

Ed - Microsoft Research
In fact I can show you a Heatmap of what we did here.

Robert Scoble - The Scoble Show
Okay.

Ed - Microsoft Research
I can actually go ahead and just show you that. Let's do -- this could take a second, because this is a very long, this was about an hour long search or session, and so it's having to build this up.

Robert Scoble - The Scoble Show
Now was it your team that figured out that people don't click the next button on searches. Like Live.com has a new internet scroll.

Ed - Microsoft Research
Well, actually they have rolled that back now.

Robert Scoble - The Scoble Show
Oh, they did.

Ed - Microsoft Research
They've rolled that back, that wasn't the beta. Now the reason is, because they weren't able to get the purf (ph) to be completely smooth.

Robert Scoble - The Scoble Show
Yeah.

Ed - Microsoft Research
I think that that's got huge benefit down the road, but right now, because you've got this, it's all Ajax, and they weren't able to get that to be completely smooth, and you'd get this jaggedness to it that would start confusing people, so they rolled it back for the current version that's out there. I think that they're trying to get the bugs worked out of that, so if they can (Voice Overlap).

Robert Scoble - The Scoble Show
I thought that was brilliant, because they found using Eye-track research that people would keep scrolling down the page.

Ed - Microsoft Research
I think it's brilliant. The other thing about it is, that it is still there for image search, and for image search it makes a lot of sense, because in fact with image search what you're doing is, you're just looking, basically you're just sort of scanning what's going on and you just rescroll, rescroll, rescroll.

Robert Scoble - The Scoble Show
On my new little start up company, I actually use that research to convince the team that they shouldn't care about what's on the page, on the first part of the page, that they should make it, so the page can scroll for a long way, because people will scroll through information, and fish, like they're fishing through a river something like that (Voice Overlap).

Ed - Microsoft Research
We did some research a while back that showed that people actually prefer to scroll and to scan the information as they're scrolling, over having to, to have more items on the screen, but having to go and get a little bit of information with Hovering for instance. You're better off just actually showing the stuff in line, and I could -- let me show you some of the Heatmaps here, you were just talking about it. I think they're kind of sexy, people love seeing these.

Robert Scoble - The Scoble Show
Only a geek finds Heatmap sexy.

Ed - Microsoft Research
So, what you see here?

Robert Scoble - The Scoble Show
That's a definition of a geek right there.

Ed - Microsoft Research
What you see here is this Titanic site that we just saw a minute ago, and you can see that the person, first off they're looking up here at the query, and then these little Xes have to do with where they've clicked. So they've hit the Titanic, and you can see the ones that they've kind of ignored, like they read this piece, and they're like, oh, there's no way, Titanic in 30 seconds with Bunnies is actually what I needed, so they didn't even bother to read the information here. This one, which is I believe, I don't know what this is exactly, but you can tell there's a lot of detail there, and they're really reading quite a lot of information in that piece. As we move down the page, you can see as they get passed the fold, they didn't even look at anything.

Robert Scoble - The Scoble Show
That argues my theory though, that people don't scroll.

Ed - Microsoft Research
Well it does, but the thing here is that, if they find what they think they need, for instance in this case Wikipedia, they have no reason to go any further. So, what happened in this case...

Robert Scoble - The Scoble Show
So treat it like a milk, like a grocer does with the milk, put the important stuff at the bottom of the page and make them scroll through all your ads. You're going to change the whole row.

Ed - Microsoft Research
Oh yeah, actually, it works. So in this case, one thing that's interesting about the different kinds of tasks that this research showed us is that, how people scan across the page is really varying depending on what they're doing.

Robert Scoble - The Scoble Show
Right.

Ed - Microsoft Research
A lot of people have tried to fit search results to be 'one size fits all', and be all things to everyone. A lot of our research now is suggesting that to the extent you can understand what people are doing, you can really tune what your results are showing you for what people are looking for, and you can have a much better experience with that. A lot of challenge now is now trying to do two things - one, figure out what people are doing and then do that optimization, and the other piece is maybe there are ways to wiggle the stuff around a little bit better, so that it performs better for all those people, so you don't have to worry about it so much. So that's the story here.

Robert Scoble - The Scoble Show
Very cool.

Kevin Schofield - Microsoft Research
Hey, so before we move on, actually I'll probably just a word about this room, because it's kind of a goofy looking room. This is actually a room where we do usability task, not only do our product groups do usability tasks, and they have a room similar like this...

Eric Horvitz - Microsoft Research
Oh yeah, we have a one-way mirror here, behind this (Voice Overlap).

Kevin Schofield - Microsoft Research
There is a one-way glass here, and usability tasks are equally important for doing research, so we bring in prototypes in this room and we're actually bringing real outside folks and sort of sit down with them, and we got this (Voice Overlap).

Eric Horvitz - Microsoft Research
For all the decision making and machine learning we do -- our whole team is arrayed around, what I would call the womb of our usability study labs, and this is the really way that we wrote is how people react to our things, or how it's actually build.

Andy Wilson - Microsoft Research
Exactly.

Robert Scoble - The Scoble Show
Unfortunately I can't show this demo, because you don't have it, but -- I mean that was mind blowing for me, it was like, wow, computers could be used for something completely wacky and the stuff that we saw in movies could actually come true.

Andy Wilson - Microsoft Research
Right, so one of the research questions is, how are these interactions actually going to work, how does it -- does it make sense to be able to put your hands directly on the data and manipulate it, and these kinds of natural gesture base interaction systems are interesting, because you have a real nice feel, fluid feel, for what you're doing. Right now, we have these systems with their base in the mouse, where you take all the richness of human emotions and you've reduced it to one 2D coordinate, right, you want more to (Voice Overlap).

Robert Scoble - The Scoble Show
How did you figure out -- who came up with the idea of putting two cameras underneath the piece of glass, how did you guys come up with that, because it seems so obvious once you see it, but it probably wasn't obvious before you did it or before you came up with that idea.

Andy Wilson - Microsoft Research
I went to SIGGRAPH a few years ago, and the company that sells that projection screen material was there demoing there -- they were just running a small demo of it, just passively displaying, advertising images on it, and as soon as I saw that, I knew immediately that I wanted a piece of that, because I knew exactly what we were going to do with it.

Robert Scoble - The Scoble Show
How did you get...

Speaker
Do you have something to show?

Andy Wilson - Microsoft Research
Yeah.

Robert Scoble - The Scoble Show
If you actually start showing me stuff, I'm going to he here all day, because out of all the stuff I saw at Tech Fest a year ago, a year and a half ago, that was the coolest stuff, so I'm honored at meeting you. I'm almost out of battery, so we have 13 minutes of battery left.

Andy Wilson - Microsoft Research
So what I'm going to show you here is just a real simple desktop version of some of these same ideas, and I'm just going to hop right into it. We have a camera here, this is just a USB webcam, and its actually looking down at the surface.

Robert Scoble - The Scoble Show
Right. I was looking at your hands

Andy Wilson - Microsoft Research
Yeah. So here I am, I'm just doing a standard sort of desktop computing kind of thing, and suppose I'm typing or using a mouse, whatever, so in this system if I put my thumb and forefinger together, the system notices that and allows me to move the cursor.

Robert Scoble - The Scoble Show
Do that again. How did you do that?

Andy Wilson - Microsoft Research
So if you look at this view here, this is actually showing what the camera sees and...

Robert Scoble - The Scoble Show
How did it sense -- did you program it to recognize that shape of your hand?

Andy Wilson - Microsoft Research
Well, so the way that it works is kind of neat. It actually reasons about the topology of the image, so it determines what it is the background, what is the foreground, and basically when you put your thumb and your forefinger together, you're pinching off a piece of the background, if you will, so all its doing is counting the number of shapes in the image. So by putting your thumb and your forefinger together, you're creating a new shape and image. Does that make sense?

Robert Scoble - The Scoble Show
Yeah.

Andy Wilson - Microsoft Research
So it doesn't know anything about hands or fingers or anything like that, and so right now we're just driving the Windows cursor around.

Robert Scoble - The Scoble Show
You're just dragging that Window around with your hand.

Andy Wilson - Microsoft Research
That's right, exactly, not using the mouse.

Robert Scoble - The Scoble Show
You're not doing anything underneath the desk?

Andy Wilson - Microsoft Research
No.

Robert Scoble - The Scoble Show
There's nobody, no remote control?

Andy Wilson - Microsoft Research
No, no, no guy under the desk.

Robert Scoble - The Scoble Show
When I first saw this, I thought it was so mind blowing, it's so magical, I guess that's good software, right, when software is really doing something cool, you think it's magic, you think there is a guy behind it.

Andy Wilson - Microsoft Research
(Voice Overlap) Magic, right, right. So here's hooking the same stuff up to Virtual Earth, and so occasionally that screws up a little bit, it says research code. Let's try that again. So you have this nice fluid kind of interaction, where if I rotate my hands, rotates the view, yeah, a little bit of motion there. Now, I can also extend the algorithm to look at the use of two hands, so I can zoom in and out very fluidly. Now, this program is pulling tiles off the Virtual Earth server, of the Virtual Earth web service in real time, so I can zoom in and out as much as you can on the...

Robert Scoble - The Scoble Show
This has deep implications for music, I could be a, what do you call that guy, conductor, and I could conduct an orchestra remotely using this kind of technology. As a surgeon, I can do really complex stuff. Now, could you come up with new algorithms that look at other gestures on your hands?

Andy Wilson - Microsoft Research
Well, sure...

Robert Scoble - The Scoble Show
Could I give the computer the finger?

Andy Wilson - Microsoft Research
So this is really -- what I'm showing you here is just sort of a really quick hack, like I'm skipping around all sorts of problems, recognizing fingers or hands or gestures just by exploring a very particular property of the image as it changes when I put my thumb and a forefinger together. So right now, this doesn't go very much further than that, but in general, yeah, we can do all sorts of stuffs like that. We can bring in other techniques, other pattern reorganization techniques to understand that, here, I'm doing a particular shape with my hand, and maybe I'm changing the axis on the CAD package or something like that, or I'm resizing something like this, these kinds of.

Robert Scoble - The Scoble Show
And this is just a $100 camera?

Andy Wilson - Microsoft Research
More like 30 bucks, so it's all just software.

Robert Scoble - The Scoble Show
Have you shown this stuff off in public before?

Andy Wilson - Microsoft Research
Have I? No, (Voice Overlap).

Robert Scoble - The Scoble Show
Are you going to go to SIGGRAPH and show this off?

Andy Wilson - Microsoft Research
I'm going to be showing this at a conference in a couple of weeks, for the first time really.

Robert Scoble - The Scoble Show
Okay. Do you want me to hold the video until...

Andy Wilson - Microsoft Research
No, no, it's fine.

Robert Scoble - The Scoble Show
I love software that just blows your mind, even though, once you see it, you understand how you can right an algorithm that does that, but it's just magical.

Andy Wilson - Microsoft Research
Yeah, it just has a really nice fluid feel, and the thing that -- I think the thing that's really nice about this particular -- for me, as a vision guy, who's into writing these kinds of algorithms, so it was really nice to be able to write this code very quickly, such a simple technique, just observation that, when you put your thumb and a forefinger together, you're making a new shape in the image. It's kind of a hack, if you will, kind of an inspired hack. So I like it when these kind of things happen.

Robert Scoble - The Scoble Show
Do you get called to consult with CSI. and help them there?

Andy Wilson - Microsoft Research
No, not really.

Robert Scoble - The Scoble Show
Because that's the kind of stuff. This is CSI style stuff, it's so mind blowing, unless I was seeing it right now, I wouldn't believe it.

Kevin Schofield - Microsoft Research
MSR actually has this thing called (Inaudible) a couple of times...

Andy Wilson - Microsoft Research
Could be a fun thing to do, to see what they think about it, and get a little exposure.

Robert Scoble - The Scoble Show
So what other kind of things do you do, I mean that's so mind blowing, I should probably stop, but...

Andy Wilson - Microsoft Research
These kinds of methods are just one example of other -- in general, just how do you do fluid sensing of motion and hands and that sort of thing, and the idea of manipulating maps is nice one, because you get this sort of very nice rotation and translation and scaling, pretty analogous to the way real maps on a table work. So those are the kinds of things that excite me the most, is when you're borrowing from what you know already, in reality based interaction, so some of the other systems that I have gathered are odd tabletop systems, and walls, and these kinds of things, we're doing largely the same, the same kind of thing.

Robert Scoble - The Scoble Show
How long do you, best guess, do you think its going to be before you get a system that has enough completeness to it, because obviously that's what -- one little algorithm, you probably need a group of algorithms to make a product, right?

Andy Wilson - Microsoft Research
Yeah, it needs to be a little bit more...

Robert Scoble - The Scoble Show
To get into my house.

Andy Wilson - Microsoft Research
Yeah, I mean that's a complex question, I don't know, it seems to me like these couple of -- this thing in particular might be the kind of thing that could be a power toy, like if you buy a camera, like you would ship with that, and that might actually be an interesting way to sort of test the waters and see how it works. Now the X-box 360 has a camera, and we've been working a little bit with that and trying to figure out what we can -- what new scenarios we can support with that.

Robert Scoble - The Scoble Show
A guy in Silicon Valley I met has actually bought some really cheap color bands that he puts on you, and then he can study your baseball sling or your golf sling. So you can do other things with colored images, even on your fingers, you can put little colored bands, it probably would help a lot (Voice Overlap).

Andy Wilson - Microsoft Research
Or a little ring or something like that. It would help a lot actually. I think we have lots of interesting ways to ship this technology, and I hope we do it. That will be fun. I think the next few years -- right now we're in this phase where we finally have the computational horsepower to do this kind of stuff without completely wrecking the rest of the machine, so with dual core processors, and these kinds of things, this really opens up a lot of possibilities.

Robert Scoble - The Scoble Show
Cool. We have 5 minutes of battery left, so do we have one more thing to see, or is this...

Kevin Schofield - Microsoft Research
No, I think this is it.

Robert Scoble - The Scoble Show
This is it.

Kevin Schofield - Microsoft Research
Wasn't that enough?

Robert Scoble - The Scoble Show
This was awesome. I could talk this guy all (Inaudible).

Kevin Schofield - Microsoft Research
You can talk to all these guys.

Eric Horvitz - Microsoft Research
You need kind a menu in, you can come back (Voice Overlap)

Kevin Schofield - Microsoft Research
My job is to make sure they actually get work done.

Eric Horvitz - Microsoft Research
Come back again, it was great seeing you. It's really great, we actually miss you around here.

Robert Scoble - The Scoble Show
Thanks.

Kevin Schofield - Microsoft Research
It was great to see you.

Robert Scoble - The Scoble Show
Thanks for giving me your -- I'm really honored to meet you, because the stuff I saw was just so cool, I wish I could tell you what I saw in the Tech Fest. That's one of the advantages of being a Microsoft employee, you get to come and see all the cool stuff you guys are building.

Kevin Schofield - Microsoft Research
Do you know what, there's a chance we can bring you back for the Tech Fest this year.

Robert Scoble - The Scoble Show
Yeah, but they don't let press into all the areas.

Kevin Schofield - Microsoft Research
Actually, because it's our 15th anniversary, we're actually going to do a bigger kind of open thing for Tech Fest this year.

Eric Horvitz - Microsoft Research
There's some real nice stuff coming out of Tech Fest. Very nice, thanks.

Kevin Schofield - Microsoft Research
Thanks.

Robert Scoble - The Scoble Show
Thanks.

Copyright ©2006 PodTech.net. All rights reserved. Privacy policy