There are cupcakes with Stamen maps on them. Each one has a single tile printed on it. Cups and Cakes Bakery baked and prepared them. They were made for the Where 2.0 conference, where Mike is talking about how Old Is The New New and I am talking about Drawing Outside the Lines, which taken together is a pretty good indication of where Stamen's collective thoughts are.
THERE ARE CUPCAKES WITH STAMEN MAPS ON THEM.
Here's the final entry of a week of posts about the maps.stamen.com
launch. As the launch of our new maps neared, I created a tool for us to look at how people are using the maps.
To speed up the loading of the maps, we try to pre-cache the map imagery. One problem. The Earth is big, and, no matter how popular, only a small percentage of it will actually be looked at on our maps. It would take our server years to render the entire world down to zoom 13
which still leaves 5 zooms levels uncached
. We have some tricks to address this problem like only pre-caching areas with land, but this topic is one Nate is going to cover in the near future. Now ten days after launch, we wanted to see how people are using the map so in the future we can improve our pre-caching.
First some background. Currently, most interactive maps on the internet are made of 256x256 pixel images called tiles (map tile background
). Using tiles allows maps to load dynamically as you pan and render out the geometries for a bounded area making the image generation faster. Here we have a 3 tiles from our new basemaps at zooms 11, 12, and 14 respectively.
Like any other page on the Internet, as people browse the maps, server logs are generated which give us a list of tiles that were downloaded. I rendered this data out showing each tile at its location and area it takes. The smaller squares represent the 256x256 pixel images but at a higher zoom level. The colors cycle through the zoom levels.
This is the world for Toner which has been in testing for a few months. Europe and Africa are in the middle. For North America, Florida and the bi-costal densities are the only defining features. South America is absent.
Here is the same dataset but focused on the US. The Bay Area has been looked at heavily, likely due to internal testing use on local projects. The green smile across the bottom is likely someone following a highway, maybe I-20/30.
This map is the terrain map pre-launch showing the western US. The coastline from San Francisco to Mexico has been examined. Also, a large amount of Utah was looked in fine detail. This area is very sparse of population but likely someone looking at the hill-shading.
We were intrigued. This map of what people look at on maps was something we hadn't seen before. There was more detail than the initial static images could show so I turned it into a tile-server so we could make it interactive progressively showing more detail as you zoom in.
Here, viewing the grid of tiles, someone appears to have followed a river.
The Northeast corridor for all 3 tilesets.
In New York City, Wall Street, Dumbo, and Park Slope are getting looked over most closely.
These are the log maps for just Watercolor over the last 10 days. They represent 1.2 million tiles served of which 350,000 were unique. As we saw previously, the US and Europe get a lot of viewing.
Not surprisingly, the map viewing correlates with city labels. The exceptions are along the coasts north of Los Angeles and of the Carolinas and Georgia.
This is a follow up post to yesterday's post about watercolor textures, Tuesday's about watercolor process, and Monday's announcing the launch of maps.stamen.com.
Terrain Layer has been on my mind since 2008 when I first started to experiment with digital elevation data, but it’s only really come together in the past year when Nelson Minar and I started kicking ideas around for making an open source answer to Google’s terrain layer. As an amateur pilot and an iPad owner, Nelson was interested in something that would make sense seen from high above. I was interested in something that would make sense at medium zooms, with all the crazy data-munging that implies.
An image as simple as this, for example, requires so much more than plain OpenStreetMap data can provide:
Obviously, there are the hills. The streets present a problem, too: many large roads that you might see at this scale are modeled as “dual carriageways” in OSM, which means that they’re actually two one-way roads as far at the database is concerned, so you can end up with a lot of doubled-up street names. The route numbers are often hidden away in weird tags with extra junk attached, and even picking the right color for the ground is a challenge.
Happily, the U.S. Geological Survey has our back. They publish absolute mountains (heh) of fascinating and useful data, including high resolution elevation models of the entire country that can be transformed into shaded hills, and types of land cover that can be colored according to vegetation.
Gem Spear helped me develop a custom color palette for the landcover, explaining the meaning of different plant classes, five kinds of forest, combinations of shrubs, grasses and crops, and a few tundras and wetlands and how they should appear together on a map.
The ground and hill renderings are about imitating the work of Eduard Imhof, whose use of color derived from grayscale relief simulated the appearance of hills in sunlight. My version is drastically toned-down from his, but the hint of warm and cool are there:
Back in the foreground, we’re making three adjustments to the base OSM data to improve the look of these maps.
First, High Road is a framework for normalizing the rendering of highways from OSM data, a critical piece of every OSM-based road map we’ve ever designed at Stamen. Deciding exactly which kinds of roads appear at each zoom level can really be done just once, and ideally shouldn’t be part of a lengthy database query in your stylesheet. High Road sorts it all out, giving you good-looking road layering despite the zoom level.
The shields and labels are both driven by some work I’ve been doing with Schuyler Erle on Skeletron. It’s an attempt to generalize complex linework in code using a range of techniques from the straight skeleton to Voronoi tesselation. I’m pre-processing every major road in the U.S. at a variety of zoom levels, so that big, complicated, doubled-up and messy roads are grouped together into neat lines that can be labeled using big, legible type:
The highways are a special beast. I’m using a combination of Skeletron and route relations to add useful-looking highway shields for numbered freeways. One particular OpenStreetMap contributor, Nathan Edgars II, deserves special mention here. I feel as though every time I did any amount of research on correct representation or data for U.S. highways, NE2’s name would come up both in OSM and Wikipedia. He appears to be responsible for the majority of painstakingly organized highways on the map, which means that maps like this of the East L.A. freeway system can look more legible:
The final assembly takes place in TileStache’s Composite provider, inspired by Lars Ahlzen’s TopOSM to do exactly the kind of raster post-processing and compositing that makes this terrain map possible. Everything we’re using is 100% open and available via Github.
I’ll close with some images of my favorite spots:
Following up on Zach's post from yesterday and Eric's post on Monday, here's a peek into my process in creating watercolor textures for watercolor maps.
Below is a breakdown of the steps, a mix of the hand and the computer in all:
My first concern was color. I started painting even swatches of colors using a very wide flat brush with vertical strokes.
Watercolor Maps, February 13
We weren't very happy with the results and moved the opposite direction.
I went through many rounds of painting and testing, experimenting with other types of brushes as I went along, as well as painting on smooth to rough varieties of paper. For the final product, a variety of brushes and papers were used depending on the desired result. For example, the rougher papers added that extra texture needed for ground areas. It was fun to mix it all up to convey a variety of terrain. ( See my set of textures available for use on Flickr under a Creative Commons Attribution-NonCommercial license. )
The process of going back and forth from painting to the computer became a continuous cycle. Midway through as I became more and more
familiar with the outcome of how the actual texture would appear on the screen when tiled, my painting process became more specific to achieve
the desired texture, color, darkness, stroke, range of value that I wanted for each feature on the map.
For oceans, I painted a range of blues mixed with turquoise and violet.
For parks, I mixed yellow greens, forest greens, muddy greens in dark and light ranges.
It was like this for every feature on the map.
Oranges, reds and browns for motorways with major roads fading in and out underneath.
Towards the end I was using original textures with no color retouching, Photoshop was mainly used for cutting seamless tiles.
The process of creating seamless tiles was just as painterly as the painting on paper. I quickly learned how to use that tiling feature in Photoshop and enjoyed the whole process of hiding the seams. Even at this step the desired color and texture was always in mind.
Some sample tiles below.
Watercolor Maps, March 2nd
This is Zach and Mike's domain. Zach and I would discuss and tweak qualities of line and
darkened edges, which are just some of the filters he created for this mapstyle. He would
set me up with the desired filtered effects leaving me to focus primarily on the textures. For the final product, a texture was created for each
zoom level and for each feature of the map. So for example, parks have
a green texture tile for each zoom level and so on. This worked
particularly well for zooming in and out of water bodies since it
conveys the movement of the oceans.
Yesterday, we announced a trio of new maps on maps.stamen.com - we've had a lot of interest in how the watercolor map gets generated, so here's the play-by-play. We'll be using a section of London as an example:
To begin with, we wanted to capture some of the irregularities of a truly handmade object. Although we're working with OpenStreetMap vector data, we decided to do all of our styling in rasterspace - the main reason was to have access to the overlaps of objects, but there are a couple of really great side benefits: the process doesn't increase in time for denser areas (i.e. cities, which is mostly what people look at), and it's entirely deterministic (important when you're stiching together tiles). We also wanted to be able to treat ground, water, green space, and so on differently, so we're actually running this process multiple times for each tile.
First, we render out some extraordinarily vibrant Mapnik images:
These can then be split up by grabbing all the yellow (for instance) and using that as a mask:
Because of how OSM treats water, we're actually subtracting the blue area from the mask as well:
Next, we do a gaussian blur:
And add a perlin noise image. This is just a 1024x1024 image that we tile:
Now we do a threshold to get a new, fuzzier, wobblier outline:
And do some antialiasing to get our final mask:
Which gets applied to a tileable watercolor image:
The other fancy bit of code adds a slight dark outline to the painted areas, to mimic the watercolor drying at different rates, and depositing more pigment at the edges. To get this sort of edge darkening, we do a gaussian blur of the mask:
And only then use the original mask to get just the interesting bits:
But that ends up giving undue weight to small or narrow areas. So we invert that and use it as an alpha channel on the gaussian blur, which evens things out:
This is then added as an overlay onto the masked image from before:
And we have one layer! This process is repeated for the other layers (in this case, water, green spaces, civic spaces, and highways) and composited into the final image:
Tomorrow, Geraldine'll be posting about the textures that went into the map.
EXTRA BONUS CONTENT:
As you can imagine, there are a lot of variables that go into this algorithm - so, to help ourselves visualize what tweaking each of those did, we created any number of samples. As a peek behind the curtain, here are a couple of examples:
maps.stamen.com, the second installment of the City Tracking project funded by the Knight News Challenge, is live. These unique cartographic styles and tiles, based on data from Open Street Map, are available for the entire world, downloadable for use under a under a Creative Commons Attribution 3.0 license, and free.
*takes deep breath*
There are three styles available: toner, terrain, and watercolor:
- Toner is about stripping online cartography down to its absolute essentials. It uses just black and white, describing a baseline that other kinds of data can be layered on. Stripping out any kind of color or image makes it easier to like focus on the interactive nature of online cartography: when do different labels show up for different cities? what should the thickness of freeways be at different zoom levels? and so forth. This project is the one that Nathaniel is hacking on at all hours, and it's great to be seeing Natural Earth data get more tightly integrated into the project over time.
- Terrain occupies a middle ground: "shaded hills, nice big text, and green where it belongs." In keeping with City Tracking's mandate to make it easier for people to tell stories about cities, this is an open-source alternative to Google's terrain maps, and it uses all open-source software like Skeletron to improve on the base line cartographic experience. Mike has been heading up this design, with help from Gem Spear and Nelson Minar.
- Watercolor pushes through to the other side of normal, bending the rules of traditional legibility in order to explore some new terrain. It incorporates hand-painted textures and algorithmic rule sets into a design that looks like it's been done by 10,000 slaves in the basement, but is rendered on the fly. Geraldine and Zach did the lion's share of the design and development on this one. This design is a mixed bag for me: I'm delighted to see it out in the world, but it's the thing that's pretty much kept me from looking at anything else for the last month and a half.
The code that runs Toner and Terrain is available for download and use at the Citytracking GitHub repository; watercolor we're going to wait on a little while until we can get some of the kinks ironed out. We talked about waiting to launch until watercolor was all buttoned up but what with all the attention that Open Street Map has been getting we decided to just bite the bullet and go for it.
We'll follow up this week with some posts on how everything works and how the sausage is made, and I've got a lot more to say about what I think this implies for what can be done with online maps and data visualization. In the meantime, have you seen how awesome Los Angeles, Washington DC, The Forbidden City, Massachussets Bay, Key West, London, New Orleans, New York, Versailles, and every other city in the cotton-pickin' world look when you point this thing at it? Holy heck.
The Forbidden City
We've been working with the smarties and do-gooders at Climate Central for the better part of a year now, designing and programming and planning and rendering and otherwise embiggening the idea of a map that could bring the reality of climate change to people's doorsteps. As of last week, the project is available at http://sealevel.climatecentral.org.
We started with two ideas:
- most maps of sea level rise are generic. How could we bring home the idea that this isn't about "those people" but about you and your neighborhood? and
- most maps of sea level rise are about the shrinking land. As a percentage of total land area, these always feel small and unsatisfying. Why don't we focus on the land that's going to be underwater, and try to make it clear that this is the land that we're going to lose?
The context for this work is: while there are a great many papers, scientific studies, meteorological surveys and other things that fall under the rubric of things that normal people accept as true, there remains a persistent and nagging unreality to the idea that, in something like a normal human timescale, we'll see and have to reckon with large-scale changes to the world as we know it. It's one thing to say "the world is changing and all of us will have to deal with it." It's quite another to say "7.6% of the people and 9.1% of the homes may very well be underwater in Boston, and so you'll need to start thinking about that pretty damn soon, is that cool?"
The political reluctance is certainly predictable — telling people about a long-term existential threat like this is just not a job that your average politician, elected for a term of a few years, may want to tackle. For that matter, it’s a hard subject for the rest of us to think about intelligently.
Although the risk of coastal flooding is slowly worsening year by year, it’s true that the worst consequences of sea-level rise, if they ever materialize, are still a long way off. Most people simply have trouble contemplating risks to their great-grandchildren. We’re a lot more interested in our own skins!
(As proof, I will tell you that two Times editors came by my desk this morning wondering about the near-term risks to their homes in low-lying areas of Long Island.)...
The Web site of Climate Central, where these sea-level studies were mainly done, has a great deal more useful information, including the ability to search by ZIP code and get a sense of your own risk.
(Justin Gillis, The New York Times)
Which is to say: if you're doing science and you want to have an impact on public discourse, pubish a rigorous scientific paper in the New York Times, and no mistake. But also: let people search for their own zip code, because what's personal matters.
Also: when the New York Times puts you on their front page, a couple dozen other news outlets do the same, and both Al Gore and Tim O'Reilly link to your stuff, it's probably a sign of two things: a) the alpha nerds are paying attention, and b) you're about to have a big traffic day.
At times on Wednesday the Climate Central computer servers were overloaded, so if you have trouble getting the search to work, try again a while later. (NYT)
This past January saw the first meeting of ConvergeSF, which hosts events to explore the intersection of design, art and technology here in San Francisco. Stamen's resident mathematician, Rachel - (how cool is that?) - presented along with
Doug Winnie from Lynda.com.
And while slides are a good record of the main points of any talk, I wanted to hear more directly from Rachel, so I asked her a few questions to follow up...
George: What do you think is the main reason people have moved away from Flash over the past few years?
G: What's your favourite Stamen HTML5/JS project?
R: My favourite pet project is the European Music Awards [Twitter Tracker].
G: Is that because you got to meet The Hoff?
[The EMA site] is the nicest thing that I've made here at Stamen so far. One of the major hurdles that we've had is getting the motion to work as well as it did in Flash. I think that the EMA site was great in terms of creating constant ambient motion that was interesting and fluid. The animations are short enough to be interesting, and long enough to hold a narrative... The new Esquire project at http://migration.stamen.com/ is great too - it's so fast, so responsive.
G: Stamen has been using a variety of JS libraries like d3.js and paper.js on projects here, but I understand you've been doing a bit of research around new potential tools we could be using. What have you discovered?
R: I guess the main thing that everyone's talking about and trying to figure out how to use would be WebGL and specifically three.js. I am a bit of a WebGL curmudgeon, though, because it doesn't work and crashes all the time. The success rate is too low for me to be excited. It's a different way of thinking about objects, primarily built out for game rendering and environments. You build an object then color and shade it, then look at it. This isn't necessarily what I'm interested in on the web. Alien masks, shading, light sources...? Make a scene then move the camera around?
I'm more excited about building things that people can play with, interact with and change. Seems like you can simulate that with WebGL, but that's not what it's best at.
Sencha is another JS library that offers UI and animations and recognizes different touch events. Then you can write JS to respond to that. Finger is not longer just mouse pointer, but we can start poking at gesture support. Fun to think about websites as mobile native applications...
G: Thanks, Rachel! Holy crap, there are some braniacs around here!! Is it OK if I call you a brainiac?
There are a few other things Stamen is, or has been, out and about doing:
- On January 23, Mike & Eric gave a presentation at Code for America about Data Visualization as part of the Big Data for the Public Good seminar series. You can watch videos of those presentations on Greenplum.
- Mike & Shawn have been teaching a Visualizing and Mapping Data course at the Grey Area Foundation for the Arts. This round of classes is about to finish up, but there's talk of another run if there's interest.
- Rachel is presenting at Designers and Geeks, trying to convince you that "The Future will be made of Screens" on Thursday, February 23 at 7pm. Update, March 2, 2012: Here are Rachel's slides from that presentation.
- I'm headed to SxSW in March to be on a panel about "Creating an Internet of Entities" with Tyler Bell from Factual, Drew Vogel from the Sunlight Foundation and Pete Warden from Jetpack. We'll be talking about the difficulty of finding resolute entities (like, say, a person) in the ocean of data we find ourselves in, and how important it is to give these entities a place to call home if you do manage to find one. That talk is on Tuesday, March 13 at 11am. I'm also looking forward to enjoying a few Shiner Bocks and as much barbecue as any one young lady should eat.
- Eric is sitting on the jury for the Cooper-Hewitt's National Design Awards, so will be in New York City March 12-13 (missing out on the barbecue in Austin).
In the coming weeks, I'll be reporting more from the trenches, as it were, in the form of a quick video or another interview or a screenshot and a story about some work in progress. Stay tuned!
We keep detailed project blogs for the duration of all our projects, where we post work-in-progress, links to related material, and so forth. We do this both as a way to share with our clients where things are at, but also as a way of maintaining a record of the thinking that went into the work (after all, "You can control time when you can see it."). I've often felt a sense of sadness that it's only the final piece that sees the light of day; there's a lightness to the experimentation that goes into the early parts of projects, when you're not worried so much about final implementation and instead can just play. We're going to start exposing some of this process, and this post is about the thinking that went into http://migration.stamen.com/, a recent project for Esquire Magazine.
Slicing & dicing
We started thinking about different ways to slice through the country. 1&, 5%, 99% are all in the news a lot lately, as is the idea that the American Dream of work hard & save your money & get a house & you'll be fine is true for less and less people all the time.
Our first idea was to break up all the zip codes across the country into different forms, based on demographics. Kind of like Jenny Odell's collages of [tankers from google maps]:
But instead of ships, there'd be the shapes of all the zip codes where, say, more that 10% of all homes are in foreclosure. I started sketching out maps of where some number of people have lost their jobs, or where the economy is strong; some aspect of economic life that matters to people now. And then, perhaps, you could arrange those back into a more visually-like-the-US map:
Slicing & splicing
We thought about a different kind of slicing: take a latitude line across the country, and find all the towns along that line. Every time you get to a town, you measure a series of demographics: age, income, education.
Stack enough of these on top of one another, you start to see patterns in the way the country is arranged: places with high populations of young people and lots of poverty, right next to places with the opposite. You get a literal slice through the country.
Maybe those lines aren't just latitude or longitude lines. Maybe they're lines along roads from one place to another. Find the poorest places in America, plot the routes between them, map obesity along the way:
Reverse this idea: instead of starting from one poor place and driving to another, find a route that goes only through poor places. Would you be able to get from Allen, South Dakota to Elmo, Montana, only driving through [poor places]? What would those routes look like?
How far could you get, if you could only drive through towns where the majority of people who lived there are under 40? Are there age sheds, just like there are watersheds and foodsheds?
(map by [Ryan Alexander])
Comparing rich to poor, we might find that some of these routes overlap quite closely; or that they never touch at all.
We could start to think of these routes as weather fronts: places that push up against one another without quite touching, but which affect each other as they roll past:
Maps of where the haves and have-nots live and work might start to emerge as distinct patters of relationships, depending on how you turned the dials. The 1% and the 5% might have more in common than they think. Or not; depending. Let's find out! Also, George has just pointed out to me that "iso chubbs" is not nearly as neutral a term as I had thought when I wrote it down, so that's a thing.
First, Mike plotted the routes people likely use to leave California, and where they end up:
Second, Jeff did this for all the roads in the US that people have moved along; thicken them by how often they're used (and note the crazily-well-traveled road up to Alaska):
Compare the roads that the top 1% of the money travels along:
to the roads the bottom 1% of money travels along. Notice that even though the amounts of $ are the same in both graphs, the road networks are a lot more extensive in the second, because there are alot more people moving at the lower levels of the scale:
George and Geraldine started thinking about how these maps might overlay and interact with one another, and came up with a couple different ideas that we liked:
Ultimately, though, none of these were as satisfyingly crisp and direct as Jeff's original sketches of the richest and poorest 35 counties, respectively. This idea of little archipelagos clustered along major travel routes, and the non-intersection of rich and poor, was more what we were looking for:
At some point Mike wrote "Can you make, like, a hundred more?"
LIMIT 35 OFFSET 35, LIMIT 35 OFFSET 70, LIMIT 35 OFFSET 105
and we were off and running. The final piece lets you move through the whole range from the top 1% to the bottom, and the different constellations and nooks and crannies of the moving American economic landscape are laid out like cracks in so much financial ice.
More at http://migration.stamen.com/
Last year, we were challenged by Esquire magazine to re-imagine a map of America to depict the country in a new light. That challenge has resulted in our first new work for this year: a piece called WHERE DOES THE MONEY GO?
You can read Esquire's article on the project and see the other contributions.
By combining migration data from the Internal Revenue Service (IRS) with route information from MapQuest Open, we were able to show population movement alongside income flow, resulting in thousands of different images of the 48 contiguous US States, organized left to right by county-level income lost to gained.
Having just joined Stamen, this project was an excellent first crack for me for a few different reasons:
1) The project passed around the studio like a plate in a kitchen, with different people preparing different parts of the whole;
2) I was able to get a sudden, detailed look at the geography of the USA, useful given that I'm Australian! Did you know there are 3,143 counties? Or that Michigan looks like it's cut in two?
4) I'm finally on the inside (mwahaha!), so was able to participate in the delicious evolution of a project, watching sketches being made with data, being tossed or surviving, being refined and perfected.
To help prepare this post, I asked around the studio to uncover any insights or stories people found as they worked on the project. It's these observations that mould the shape of the end result, but often lay silent once the project goes live. This is something I hope to deliberately listen to and share in future projects at the studio, because I think it's a glimpse into the minds of the craftspeople here.
Eric was struck by how this map can show you the USA from so many different angles. Noticing that different people use different routes to travel across the country, he remarked that
"if you live in one of these places where a lot of people are moving into, you're probably around people from a lot of different places, so your map of the country is really different."
As I moved the slider from one side to the other, it became obvious how migration across the country was oddly parallel to the movement of money. At each outer edge, people seem to travel further, and in the middle of the country, paths from old to new home seem shorter, and the map starts to break into more abstract archipelagos.
At the mid-point, where in some cases neither income nor population shifted, the map is quiet:
Mike picked up on the fact that most of the big cities in the US - Los Angeles, Chicago, New York - are in counties that lost the most income overall. "In a way, these cities aren't really losers at all. It's more like they're the major exporters of the country." When we use language like loss or gain, it implies some kind of failure, but in this case, we're describing money movement, not necessarily the "biggest losers."
There were a couple of stories that stood out for me. Florida appears to be gaining both income and population. Of the 80 wealthiest counties on the scale (that's the top 2%), 19 of those are in Florida, and the average income per household moving there is about $53,000. The national poverty threshold for a four-person household in 2009 was $21,954. (The source for that number is this census.gov "Poverty Thresholds by Size of Family Unit: 1980 to 2009" PDF report.)
I also spotted Orleans Parish in Lousiana, where Hurricane Katrina hit so hard back in 2005, but was experiencing substantial growth in 2010, with an influx of $146 million in income, and 10,594 new souls.
All in all, a great start to an exciting year. Enjoy exploring this different America!