Guest: Gary Flake - Microsoft
Host: Robert Scoble - The ScobleShow

Robert Scoble - The ScobleShow
So, who are you?

Gary Flake - Microsoft
My name is Gary Flake. I'm a technical fellow at Microsoft and founder and head of 'Live Labs'.

Robert Scoble - The ScobleShow
'Windows Live.'

Gary Flake - Microsoft
No, no, no, 'Live Labs.'

Robert Scoble - The ScobleShow
I heard.

Gary Flake - Microsoft
Yeah, it's just Microsoft Live Labs.

Robert Scoble - The ScobleShow
So, you are the head of the O'Reilly Web 2.0 Summit conference this week?

Gary Flake - Microsoft
Well, I am glad to hear you say that.

Robert Scoble - The ScobleShow
Well, I arrived this morning and like ten people came up to me, did you see Photosynth?

Gary Flake - Microsoft
That's good; I am glad to hear that.

Robert Scoble - The ScobleShow
So, that's from Live Labs?

Gary Flake - Microsoft
Yup, from Live Labs.

Robert Scoble - The ScobleShow
Okay and you are going to show it to me?

Gary Flake - Microsoft
Yeah, so, it's right here. So, let me, let me -- before we dive into this, let me give you a little bit of back story. The project is actually the merger of two different technologies that were developed independently. One was this Seadragon Technology, which came from a company that I acquired back in January. What Seadragon does is, it's basically a Client Server Technology that allows for very efficient streaming of high-resolution, big, big chunks of data, but in a very efficient manner. That's one piece. The other piece was a breakthrough piece of Machine Vision Research that was done in collaboration between University of Washington and MSR. And the researchers there -- one was a graduate student, by the name, Noah Snavely; his professor was Steve Seitz, and our researcher within MSR is - I'm blinking on people's names -Rick Szeliski. Thank you. Anyhow, when we saw what that research team had done in terms of this project that they called 'Photo-tourism', it was doing some very beautiful things in terms of stitching photos together automatically and especially relating them to one another. What really got us exciting was -- excited was, when we thought about combining that to the Seadragon Technology in combination, it came together as sort of a complete solution for -- how would you make this as a web service, where you had gigapixels of data that was remote on the server, but you want to give a very fluid environment. So, I'm actually, I'm going to start off in a home position right now.

Robert Scoble - The Scoble Show
Let me just set down my tripods. So, I can have...

Gary Flake - Microsoft
Yeah, you can steady.

Robert Scoble - The ScobleShow
Get a nice steady image for you. Actually, hold on just a second.-
I'm ready now.

Gary Flake - Microsoft
So, what we're going to do is we're going to fly around first, just to kind of get an overall perspective. And, what you see here is a Point Cloud that is constructed by taking a look at all of the photos and finding out what points that they have in common. So, that's where the Point Cloud comes from. I'll do that one more time, just so that you can get the bird's eye view, I'll pause it. You could see that it constructs a rough approximation of a 3D model that stays. Now, we can continue flying down and drop ourselves into the middle here and basically navigate the space. Now, I'm just going to pick an arbitrary spot here and start walking around the square a little bit. And in fact, I can...

Robert Scoble - The ScobleShow
That way - this is taken from one image?

Gary Flake - Microsoft
No, no. I'm sorry. These are hundreds of photos that were taken by a photographer...

Robert Scoble - The ScobleShow
All right.

Gary Flake - Microsoft
...and, what we've done is using that back-end technology, that was built by University of Washington and Microsoft Research. We figured out, how all those different photos, spatially relate to one another to effectively create a 3D Model of the world, and then, how those photos actually relate to that 3D Model. So, you're looking at the Point Cloud, which represents the 3D Model, and then now you're looking at individual photo, and if I want to...

Robert Scoble - The ScobleShow
And, how many photos -- if I wanted to go out to the Golden Gate Bridge and do one of these or something like that, how many photos do you need to really do this well?

Gary Flake - Microsoft
Well, right now -- it depends on how big the space is. Right now we're looking at a rather big space and we've probably on the order of about 200 photos right here, I am guessing. And so, the larger the space, and more photos that you need; however, dozens of 200's, 2000's of photos actually do nice things, and so....

Robert Scoble - The ScobleShow
How do these get in? Did somebody upload them with your tool?

Gary Flake - Microsoft
No, so it's actually wider way to net (ph). So, one of our program managers went on vacation. He took a whole bunch of photos. We dropped that into our servers to allow the algorithms to crunch on them and that is the step-work. It figures out, how they spatially relate to one another, the Point Cloud is one of the outputs of that process. How the photos, physically relate to that Point Cloud, is another output. And, it's from that, that's all the information we need, in order to reconstruct this 3D environment.

Robert Scoble - The ScobleShow
So, how long did it take to put 200 photos in, like this guy did? What do you guys think? You guys have estimates?

Gary Flake - Microsoft
It's a long time, I don't know whether I want into the details. It's about eight to ten hours of CPU time.

Robert Scoble - The ScobleShow
Eight to ten hours.

Gary Flake - Microsoft
Yeah. So, it's currently -- it can be an expensive operation. We know that there are lots of ways of actually speeding it up and doing much faster things with it. So, I'm just going to, kind of, walk around a little bit here, and we can take a look, and for all these things -- remember I mentioned the Seadragon Technology before which is about really efficiently transmitting big objects over tight pipes, well, you are seeing it right here because we can zoom in on and get a whole lot of detail, which is pretty fun. We can continue to step back, and step back, and step back, and so this gives a real a merced (ph) experience with respect to exploring a space. And, just maintaining that visual continuity has a really nice effect of helping the user to maintain contacts, where they are. The contact switches are no longer hard; they are actually quite soft. So, take a look at this, I'm just going to tour around here, walk around, I can go back, I can occasionally step back, do a whole bunch of things like that. We call this thing right here splatter mode, where, basically, when we're going to splatter mode, it takes the current image that you are on in 3D setting, puts them into the center and are ordered around the spiraling out from the center, are photos that are visually related to that one. So, you can see we have a whole bunch of photos here, that sort of capture part of that square and we can pick a different one and see a different perspective. For any of these photos, we can do that nice Seadragon eyes thing that I showed you before -- we're basically - now we're zooming in, and getting a lot of data all at once. Again, these are like 8 megapixel images. Now we can go in, we can go out...

Robert Scoble - The ScobleShow
And are these coming over the wire live or are these locally stored?

Gary Flake - Microsoft
For the purpose of this kiosk, we set this up so it's a local store, because we are sharing bandwidth with lot of other people, but the experience that you just saw, can be had by anyone using a broadband connection. So, I'm not showing you anything that isn't available today just by having a broadband connection.

Robert Scoble - The ScobleShow
Wow, and I can download this now, right.

Gary Flake - Microsoft
You can download this now, so if you go to labs at live.com/photos, and that's where the Add effect (ph) is downloaded.

Robert Scoble - The ScobleShow
Can I upload my own photos to it?

Gary Flake - Microsoft
We're not there yet and the...

Robert Scoble - The ScobleShow
So, it takes eight hours of process or so...

Gary Flake - Microsoft
Yeah, what we're working on right now is a number of things. One, is a longer term store, in terms of, how we want this to overlap with lots of different products, and what's the true potential of this here because whenever people look at this, automatically, there is a whole bunch of used cases that they come up with. I see this, and I want to take my vacation photos, and I want to see, how they relate to everyone's vacation photos. Someone else looks at this and they say, 'Oh my gosh!, it's the new way of selling real estate'. Someone else says,' I want to take the photos of my child, as they've grown up, and watch them grow up in my house.' So, there are so many different ways of how this could evolve, that instead of taking the shortest path, or like the quickest product, we really want to take a slightly more thoughtful path to a better product.

Robert Scoble - The ScobleShow
Okay.

Gary Flake - Microsoft
So, that's our approach; so clearly, what's in the cards is, we had to figure out, how do we make the community of photos available for many people, and I think, that's when a lot of the real mind boggling potential of this will come through. So, I'm also, what I'm going to do right now is, I'm going to show you a different collection, okay? So, right now, we're diving into the artist studio of a guy by the name of Barry Fegan (ph), and he's an artist local to Seattle, and he was kind enough to let us just come in to his studio and watch him work. And so, we can take a look at his studio, walk around, see what's going on. There all sorts of things going on, so I'm just kind of taking a gander of what's going on in the studio, and we've come full circle now. But, now I'm going to turn back and notice, that as I move the mouse over, this indicates that some of these photos we actually have, is much higher resolution; that they are in fact registered in the same process. So, I can take a look at this, step back or go back in, and then zoom in and get just an amazing amount of rich detail.

Robert Scoble - The ScobleShow
Wow!

Gary Flake - Microsoft
And so, again, like, Zoan, tell me how big are these scans of this photos?

Zoan
80 megapixels.

Gary Flake - Microsoft
Yeah you are looking at 80...

Robert Scoble - The ScobleShow
Now, who are these guys behind you?

Gary Flake - Microsoft
Oh, these guys want to introduce themselves (Voice Overlap).

Robert Scoble - The ScobleShow
Well, they can't hear because you've got to introduce them because you've got the mike .

Gary Flake - Microsoft
Oh! We have Adam Shepherd here, he is a group program manager that heads up a lot of product planning and program management within Live Labs. We've Jonathan Duggy (ph) who is a program manager working specifically on Photosynth. We have Blaise Agueray Arcas who is an architect within Live Labs.

Robert Scoble - The ScobleShow
Excellent.

Gary Flake - Microsoft
Thank you.

Robert Scoble - The ScobleShow
Thanks. Awesome stuff, guys.

Gary Flake - Microsoft
Yeah, so you're looking now at a 80 megapixel image, streamed over. We can now step back and see it in its context again. I'm going to zoom back in just for a second, and we can continue to walk around, and occasionally and we might say, oh, here's another photo over here that I want to dive into, and again, 80 megapixels. Now, you're getting the idea of that. Okay, we can do this for an artist studio. Imagine, if we did this for a major museum, what sort of experience could you have walking through the museum, seeing the 100 megapixel version of something that you could only really get the -- a really suitable experience, if it was done in person, but now, I can look at 80 megapixel objects over my VSL on.

Robert Scoble - The ScobleShow
Right.

Gary Flake - Microsoft
That sort of thing. Again, there's just really, really rich detail throughout this whole collection, and a sense of spatial continuity, that's preserved. I'm going to go back to -- towards Gary Fagan, and see, if I can navigate a little bit closer, and see what's going on with him. And here, we see a little bit of what's going on as he's working, and this is really neat, because you start to see things that look a little bit like time-lapse photography, because he's sitting there working and painting in his art wok, and each time we go through a transition, again, we're preserving this spatial continuity of all the different photos. So for fun if you're interested, I'm going to do a couple of more collections. This one is St. Peter's Vatican, and this collection is really nice, because from this collection we can get a sense of depth, that doesn't really exist in the other collections. So, we started from afar, we work our way in, we're going further, further, and so now you're seeing that you can capture orders of magnitude of depth within one experience, and navigate through that experience in a fairly seamless way. That's incredibly compelling in terms of, how you can connect the whole thing together. So, now I'm going to do something that's really unusual here, we're taking a little bit of a 3D, I'm sorry, a 360o tour of this. I'm now going to fly above, see what's going on here. I can take a look at what's on this different pieces, I'm going to keep moving, I'm looking for something special here, hold on. Okay, I want to get to the other side, all right. So, we have a little feature here that we call frusta, where it allows you to see where the photographers were standing when they took the photo, and what photos they were taking out. So you see here, I highlight this, and I'm seeing where the camera was pointed, and what they're taking the photo of. Well notice up over here, and over here, there's some people that would -- some shots that were taking from up high. So, we've now navigated this space, like taking a bird's eye view stepping back, seeing that there's a photographer up there, and now I'm going to click on this, and get that view of above. Now, I'm going to turn off the frusta by clicking on this camera icon, and now we can just see, some of these other images that are from the higher level view. So one more collection I want to show, and I think this will kind of do it all justice.

Robert Scoble - The ScobleShow
Can I get to all these collections online when I download this?

Gary Flake - Microsoft
We've put four of them up initially, so this one is not available, but I believe the other ones that I shown are.

Robert Scoble - The ScobleShow
Okay.

Gary Flake - Microsoft
So this is Grassy Lake, and what's really -- actually let me -- I want to get to a particular spot here, so I'm looking, I want to see these climbers in action. I'm going to go ahead and pick this one, and then I'm going to go into 3D view, which I'm going to do right now. Now what's really interesting is, again you have this sense of time lapse photography, because you could see the progress of the climbers. Now, even though the camera has moved a little bit, we're preserving the spatial continuity in terms of what the climbers were going through, and now we can actually take a longer, larger view of what it was they were climbing, and how it's spatially it relates to the rest of the world. So again we're given entirely new experiences to what's possible with respect to photos, and being able to see that context more seamlessly and more continuously. So you've been suspiciously silent, what does that mean, is that bad or good?

Robert Scoble - The ScobleShow
Stunned.

Gary Flake - Microsoft
Oh good, I'm glad to hear that, and again...

Robert Scoble - The ScobleShow
You stunned me into silence.

Gary Flake - Microsoft
That's great, that's good, I've never known you to be silent, so this is -- I consider this (Inaudible) renowned accomplishment. Anyhow, you see what's going on here, there's just -- again, lots of rich information, lots of fun. We think that there's a lot of potential here, we obviously haven't thought through all the different use cases, and the reason why we put this out seven months after starting it was, because we wanted to actually start something -- the dialogue between our team and the rest of world to figure out what were the more compelling things to do.

Robert Scoble - The ScobleShow
Very cool.

Gary Flake - Microsoft
Thank you.

Robert Scoble - The ScobleShow
Very cool guys. This rocks.

Gary Flake - Microsoft
Thanks.