For the past few months, we’ve been quietly working under a grant from the Knight Foundation to do something that no one seems to have done before: publicly compiling as many open-licensed, locally-created terrain data sets as possible to stitch together a global set.
Esri, Mapbox, NASA,Google, Mapquest and others already make terrain maps available. The challenge our project is intended to address, in addition to bringing a different set of design values to the maps themselves, is access to what’s under the hood. Access to the underlying data is often proprietary (like Esri’s terrain data) or otherwise restricted. This means that if someone wants to experiment with this data, they’re pretty much SOL. Low-resolution data exists for the whole world, thanks to NASA and their freely available SRTM 90m dataset, and the ongoing release of 30m data. The challenge is that once you start zooming in, even at 30m resolution, the visual impact of the data starts to break down:
The free and open medium- to high-resolution data that we can find is often limited to a single country or region, and it’s scattered all over the place. We want to use this project to change that by creating a public repository containing lists of these open datasets, a merged “best available open data” view, and creating instructions and processes to stitch these datasets together.
To make things more complicated, most raw digital elevation data requires some amount of cleanup to fill in holes in the data or otherwise remove artifacts. Everybody who uses the same open data sources has to fix the same errors, resulting in a lot of unnecessarily duplicated effort. Our workflow will include these cleanup steps (which typically involve using data sourced elsewhere), providing a seamless and ready-to-use service sourced from various datasets all over the world.
Yesterday we put out a call for these datasets on Twitter, and have been amazed and delighted by the response. In less than 24 hours, we’ve been given leads to data sets that we were unaware of from Norway (thx @andershartmann), Australia (thx @AllanBarger), Germany (thx @cartocalypse), and more. We’ve also gotten pointers to bathymetry data, too! We’ll take it!
This is a great start, but we need your help to make this the best data collection it can be.
Here’s how you can help:
1. Send us open terrain data sets.
There’s a few different ways to get this to us:
If you know the license and the resolution of the data, please include that too. Licensing is important, and is one of the major hurdles encountered when we work with terrain data. Keeping track of all data sources regardless of license is incredibly helpful, although we intend to stick to non-ambiguously licensed open data when producing the worldwide map.
2. Follow what we’re doing at openterrain.tumblr.com. It’s our hub for public project updates, which are sometimes super nerdy and often really pretty.
3. Spread the word! You might not know where to find this data, but you might know someone who does. Send them this post, or even just retweet this Tweet.
Once we have this data, we’ll update our in-house terrain style and create terrain-enhanced versions of the Positron and Dark Matter basemaps we designed for CartoDB. These basemaps are free for anyone to use in their own mapping projects!
Plus, we can make more beautiful experiments like these, for everywhere in the world:
What’s even better, is that you’ll be able to, too.
All around the world, people are getting together for Instameets, where Instagrammers gather to take lovely, fabulously filtered square photos of the world around them. It had been a challenge to effectively and elegantly map this community activity, so Instagram reached out to us to help to them out with a new map of community activity around the world.
The new, now mobile-friendly Instameet map foregrounds photos instead of standard icons, and it distinguishes between gatherings old and new. Red dots and photo borders reflect Instameets in the future, while blue dots and photo borders reflect meetings in the past. Photos along the bottom of the map are organized chronologically, too, so that it’s easy to see what’s happening, when.
Photo selection is also much clearer and cleaner:
And you can search for locations of Instameets as well:
As a cherry on top, for people who are looking to just see what’s out there, we implemented a simple animation cycling between featured Instameet photos.
Finding water data is harder than I thought. Like detective Gittes in the movie Chinatown, I’m poking my nose around and asking everyone about water. Instead of murder and slimy deals, I am scouring the internet and working with city government. I’ve spent many hours sleuthing and learning about the water system in our city.
In San Francisco, where this story takes place, we have three primary water systems. Here’s an overview:
The Sewer System is owned and operated by the SFPUC. The DPW provides certain engineering services. This is a combined stormwater and wastewater system. Yup, that’s right, the water you flush down the toilet goes into the same pipes as the the rainwater. Everything gets piped to a state-of-the art wastewaster treatment plant. Amazingly the sewer pipes are fed almost entirely by gravity, taking advantage of the natural landscape of the city.
The Auxiliary Water Supply System (AWSS) was built in 1908 just after the 1906 San Francisco Earthquake. It is an entire water system that is dedicated solely to firefighting. 80% of the city was destroyed not by earthquake itself, but by the fires that ravaged the city. The fires rampaged through the city mostly because the water mains collapsed. Just afterwards, the city began construction on a separate this infrastructure for combatting future fires. It consists of reservoirs that feed an entire network of pipes to high-pressure fire hydrants and also includes approximately 170 underground cisterns at various intersections in the city. This incredible separate water system is unique to San Francisco.
The Potable WaterSystem, a.k.a. drinking water is the water we get from our faucets and showers. It comes from the Hetch Hetchy — a historic valley but also a reservoir and water system constructed from 1913-1938 to provide water to San Francisco. This history is well-documented, but what I know little about is how the actual drinking water gets piped into San Francisco. homes Also, the San Francisco water is amongst the most safe in the world, so you can drink directly from your tap.
Given all of this, where is the story? This is the question that I asked folks at Stamen, Autodesk and Gray Area during a hyper-productive brainstorming session last week. Here’s the whiteboard with the notes. The takeaways, as folks call it are, are below and here I’m going to get nitty-gritty into process.
(whiteboard brainstorming session with Stamen)
(1) In my original proposal, I had envisioned a table-top version of the entire water infrastucture: pipes, cisterns, manhole chambers, reservoirs as a large-scale sculpture, printed in panels. It was kindly pointed out to me by the Autodesk Creative Projects team that this is unfeasible. I quickly realized the truth of this: 3D prints are expensive, time-consuming to clean and fragile. Divide the sculptural part of the project into several small parts.
(2) People are interested in the sewer system. Someone said, “I want to know if you take a dump at Nob Hill, where does the poop go?” It’s universal. Everyone poops, even the Queen of England and even Batman. It’s funny, it’s gross, it’s entirely human. This could be accessible to everyone.
(4) Think about focusing on making a beautiful and informative 3D map / data-visualization of just 1 square mile of San Francisco infrastructure. Hone on one area of the city.
(5) Complex systems can be modeled virtually. Over the last couple weeks, I’ve been running code tests, talking to many people in city government and building out an entire water modeling systems in C++ using OpenFrameworks. It’s been slow, deliberate and arduous. Balance the physical models with a complex virtual one.
I'm still not sure exactly where this project is heading, which is to be expected at this stage. For now, I’m mining data and acting as a detective. In the meantime, here is the trailer for Chinatown, which gives away the entire plot in 3 minutes.
Today marks the official launch of parks.stamen.com, a project designed to highlight social media from parks and open spaces across California, created in partnership with Electric Roadrunner Lab.
Stories pour out of our parks every day. This project is a first step towards visualizing Twitter, Flickr, Instagram, and Foursquare content using the actual boundaries of our parks, so that we can start to understand how people feel about their favorite open spaces. Taken together these stories make plain that parks are integral to the lives of all Californians. They are evidence for the argument that access to open space must be protected—and expanded—for all and that parks need our support. This is also a tool that park rangers, managers, and advocates can use to understand how people are using parks and to connect with their customers and supporters.
The project's original concept came from writer and educator Jon Christensen, editor of BOOM! A Journal of California and his Electric Roadrunner Lab. Jon sees this work as a gift to the parks and park visitors, an experiment about learning who is using the parks and how. It's the start of a conversation about how we use open spaces in this screen-mediated age. Parks, it turns out, aren't just places where you get away from your everyday life and connect with nature (if they ever were)—they're also places of deep social meaning and engagement with others, and this project is a way of taking a new look at how we do that in the age of mobile everywhere. Our hope is that this project helps park rangers and staff to see your stories, and to clearly show how people are using and enjoying parks across California.
Project initiator Jon Christensen leading a workshop with Stamen and parks stakeholders about social media use in California Parks.
Search for your favorite California park and find the park's hashtag, like #YOSE for Yosemite National Park, or #ALCA for Alcatraz Island.
Include that hashtag when you share content on Twitter, Instagram, Flickr or Foursquare
Look for the wander link too. It'll take you to a random open space and hopefully bring you to new parks that you didn't even know existed!
Not in California? Don't have any pics? Wish you had a project like this in your state? We'd still love to hear from you on Twitter, joining the conversation on #caliparks and following @parks_stamen. You can also send us a good old-fashioned email to let us know your thoughts.
The updated data included several new elective procedures like breast cancer treatment and spinal procedures, which we highlighted in the "menu" of procedures:
This update also allowed us to make some improvements in the readability of the data, like a dynamically updating graph at the bottom of the map that shows the range of performance per procedure for all the regions currently visible on the map:
And a new chart per region that allows you to glance at that region's performance by age relative to the California average, for example, you can see that more gallbladder-ectomies happen in Red Bluff comparative to other regions, in each age group:
It's always good to have the opportunity to try to improve on work we've done before, and we were very pleased to have this opportunity with the folks at CHCF. Making this healthcare-related information more accessible to Californians is important work, and we're proud to have been involved. The study has made the news too - you can see the press release on Reuters or this article in Silicon Valley's Mercury News.
The Chesapeake Bay is the largest estuary in the United States. (An estuary is a body of water where fresh and salt water mix.) It's about 200 miles long, with a total surface area of about 4,480 square miles, an average depth of about 21 feet, and the Bay and its tributaries have a shoreline longer than the entire U.S. west coast. Thanks to its watermen, the Bay produces about 500 million pounds of seafood per year, despite increasing pollution levels. You can read lots more interesting facts and figures on the Chesapeake Bay Program website.
Here's a (west-facing) map of what it looked like in the 1600s.
And a more detailed view of the Bay created in 1840 by Fielding Lucas Jr. in Baltimore:
As you can see, humans have been interested in the Bay for ages. In addition to the fisherpeople, citizens and pirates (!) that live or have lived on its shallow waters, there is also a ton of scientific research going on, much of which is lead by the CBP. The Bay is home to some 80,000 acres of bay grasses, and bay grass density over time is a great indicator of the overall health of the system. We worked with CBP and, in particular, Dr. Robert Orth from the Virginia Institute of Marine Science to create a map that helps show grass density across the Bay from 1984-2012. There's a general challenge for scientists around the world to mine their gigantic research datasets for insight and stories, and this project is a first round at help the CBP do that.
Knowing that the project was focussed on a body of water, and not the land ended up being a strong influence on the cartography we designed. But first, we had to make sure we could "operate" the remarkably comprehensive bathymetry data that CBP has. Here are a few of Seth's first drawings, where we were trying to work out why there were odd "water-y alien crop circles" appearing in the middle of the bay:
Turns out that the alien-y circles are probably data artifacts (dartifacts?) from a long-term Bay soundings dataset that had been interpolated into a raster. Nathaniel ended up fixing this more or less by hand, cutting and pasting non-alien pieces back into the larger picture.
In addition to a bunch of public datasets online at ftp://ftp.chesapeakebay.net, CBP partner, NOAA also has a selection of bathymetry datas available, if you're interested to experiment with it.
We gave the roads a very light-handed treatment because the water is the main feature of the map. There are enough roads to help locals find their way, and we labelled only a few of the major cities in the area, again to help people orient themselves, but never to overwhelm the water.
Tangier Island has been inhabited for years and years by fishermen and women. Crabs grow happily in the shallows around the islands, and you can see by comparing year on year, that the bay grasses where the crabs frolic have covered the area for the last 20 or so years. Dr. Orth explained it like this:
Tangier Island is home to one of the biggest grass beds in the Chesapeake Bay, and many island residents continue to make their livelihoods fishing and crabbing among the eelgrass and widgeongrass that grows here. While both of these species are typically found in high-salinity areas, they continue to thrive in the medium to low salinity of the region.
We're excited that the CBP now host the tiles themselves, and look forward to seeing this bathymetry used to help tell other stories with their data into the future.
Economic geography: adds global roads, railroads, ports, airports, and time zones to show how people are interconnected and goods route (read Richard Florida on airports, full legal document about time zones and international date line shifts, and background on the E-Road network).
Remastered geometries: fixes topological errors at 1:10 to 1:1,000 scales in the basic coastline, ocean, land, admin-0, and admin-1 related themes for files in the the 1:10m scaleset. By removing self-intersections, sliver polygons, and adjusting offset polygons, Natural Earth imports into more GIS software (like PostGIS) and will be easier to maintain. The coastline is adjusted to better conform to ~1:3,000,000 satellite imagery. Because of all these changes, some raster themes are also updated. Land, ocean, and minor islands all build topologically by scripting ingredients, as do the admin-0 and admin-1 cultural themes.
Introduce Gray Earth rasters. Worldwide terrain depicted monochromatically in shades of gray. It combines shaded relief and regionally adjusted hypsography that emphasizes both high mountains and the micro terrain found in lowlands. View new raster »
New file name and field name schemas. Full adoption of ne_10m_theme_name.shp file names with `ne_` prefix to allow better import into GeoDB and PostGIS storage, lowercase field (column) names instead of MiXeD and UPPER cased names, and use of consistent `name` field (versus name1).
Address user submitted bug reports, ~25 since the 1.4 release, and earlier.
All themes now include README and VERSION files. The admin-0 attributes have more veracity and now includes nested disputed areas (was a sidecar). Adds continent, region, subregion codes. Adds versions of country and admin-1 without boundary lakes. All places and parts of places have population and GDP estimates. The populated places pop_max and pop_min attributes are now fully built out for all records (pop max is for the metropolitan area, pop_min is for the incorporated city of the same name). populated places now include rank_max and rank_min for simple town size grading. All instances of name1 have been changed to name, name to name, name2 to name_alt. Vertexes were added to many themes to allow them to project into conics smoothly (they’re back!). All field (column) names are now generally in the order of: scalerank, featurecla, name, name_alt, natscale, labelrank, *.
Many thanks to the individuals who contributed over the last year of development: Tom, Nathaniel, Alex Tait, Hans van der Maarel, Scott Zillmer, Mike Migurski, Daniel Huffman, Xan Gregg, Peter Bispham, Drew Noakes, Miguel Angel Vilela, Matthew Toro, Kevin Pickell, Shawn Allen, Robert Coup, Iain, Leo, and more! Thanks also to Stamen thru the Knight Foundation Citytracking grant for sponsoring a portion of this work including remastering geometries for better PostGIS import, the move to Github, and adopting semantic versioning.
UPDATED:NE_ADMIN_0 - Updated for South Sudan, map colors (now with 7, 8, 9 and 13 options), population figures, removed () from notes, and more. note: diffs between sov, adm0, map units, map subunits, and new breakaways are all calculated on the a3 codes now, no longer mix of names and a3 codes. Added and split note_adm0 and note_brk to note which countries are parts of which sovereignties and who’s breaking away or disputing. One spurious “county” feature code fixed to “country” (finland). Added labelrank on all. Added new mapcolors (7, 8, 9 and old 13). Includes new detail on Caribbean Netherlands map unit. Adds more detail to Bhutan disputed areas. Now includes continent codes, and future region code placeholder columns. Added name_len to know when to abbreviate labels. Added label ranks.
UPDATED:NE_10M_ADMIN_0_BOUNDARY_LINES_LAND - Minor updates to alignment of boundary lines (and topology fixes), additional coding to allow official US gov’t view of same. better disputed coding, including Kosovo. Densified vertex along lines to allow smooth projection into conics. Moved Omani exclave Madha to correct location. Adds left and right labels and codes. Fixes: N96NSYPAPV, ZQNTN5VGDD, Z8ZYYUQZVS.
UPDATED:NE_50M_ADMIN_1_STATES_PROVINCES_SHP – Added some new ISO coding, other minor changes. Fixes topology errors. Adds admin-1 for brazil and australia. Uses same coding as 10m files. Derived from new scale rank version.
**NEW**:NE_10M_ADMIN_0_ANTARCTICA_CLAIMS – Although countries have paused their claims to the southernmost continent, they haven’t suspended them. Thanks, Hans!
**NEW**:NE_10M_ADMIN_0_ANTARCTICA_CLAIM_LIMIT_LINES – Although countries have paused their claims to the southernmost continent, they haven’t suspended them. Thanks, Hans!
UPDATED:NE_10M_POPULATED_PLACES – A couple name corrections (Morelia, Mexico spelling fixed. Mazatlan, Mexico spelling fixed. Clarified confusion around Tabatinga / Leticia on the Colombian / Brazilian border. On the Brazil / Bolivia border, clarified Brasileia / Cobija. Fixed spelling of Shuozhou, China), many population max values, mostly in China, India, rift valley (Africa), Nigeria, and other countries in east Asia, but some elsewhere. Made sure cities in Switzerland are coded admin-0 of CH and China are CN. Moved Amundsen Base to 176° so it’s in the -12 timezone. Also moved Peter I Island. Vatican City is also moved to be contained by it’s admin-0 polygon. Same for San Marino. Added a poprank column with 0 to 14 numerical classes. Deleted spurious Extra Eureka town in Canada near Greenland. Delete duplicate town Urengoy in RUS, rename the real one Novy Urengoy. Fixes: 4SUAZ7BB49, D459XT1Z6Y.
UPDATED:NE_10M_COASTLINE – Better matches modern satellite imagery to zoom 8-ish. The earlier coastline could have been several kilometers off (like in Gibraltar). Several large new islands added. Includes densified vertex along lines to allow smooth projection into conics.
UPDATED:NE_10M_RIVERS_LAKE_CENTERLINES – See changelog for ne_10m_rivers_lake_centerlines_scale_ranks for details.
UPDATED:NE_10M_RIVERS_LAKE_CENTERLINES_SCALE_RANKS – Updated river names, few new rivers, splits. added river connector in Sweden between lake near Stockholm and Baltic Sea. Fixes in France and Netherlands. Fixes Mackenzie river at it’s confluence with Dawson river in Australia. Names the Mahakam in Borneo (Rivernum 544). Changes scalerank on Nelson river in Canada. Fixes: SHAWNZQJ3B, 5J47B13PJ7, W9X539LBUT, 35YLBL2W9Z.
UPDATED:NE_10M_RIVERS_LAKE_CENTERLINES_NORTH_AMERICA_SUPPLEMENT – Updated river names, few new rivers, splits. Fixes: SHAWNZQJ3B, 5J47B13PJ7.
UPDATED:NE_10M_RIVERS_LAKE_CENTERLINES_EUROPE_SUPPLEMENT – Updated river names, few new rivers, splits. fixed topology errors. Fixes: SHAWNZQJ3B.
UPDATED:NE_10M_LAKES – Removed major lake groupings (Great Lakes, Finger Lakes, etc) to geography label areas instead. Title cased the feature class values. Added Swedish lake near Stockholm (had been extension of Baltic Sea in ocean theme). Fixed topology errors. Fixed a few reservoir and salt lake codes (thanks Craig!).
UPDATED:NE_10M_LAKES_NORTH_AMERICA_SUPPLEMENT – Name1 have been changed to name, name to name, name2 to name_alt. Fixes 4VA9P9UGQE.
UPDATED:NE_10M_GEOGRAPHIC_LINES – New int’l date line, thanks Alex! Also densified linework for smoother projection into conics.
UPDATED:NE_10M_LAND – A dissolved version of the original 1.x file, now renamed “ne_10m_land_scale_rank”, see that changelog for full details. Fixes XAWXTN54GT.
**NEW**:NE_10M_LAND_SCALE_RANKS – Renamed our original land file to this. Incorporates new coastline. Includes densified vertex along lines to allow smooth projection into conics. Fixes XAWXTN54GT.
UPDATED:NE_10M_OCEAN – A dissolved version of the original 1.x file, now renamed “ne_10m_ocean_scale_rank”, see that changelog for full details. Fixes XAWXTN54GT.
**NEW**:NE_10M_OCEAN_SCALE_RANKS – Renamed our original ocean file to this. Incorporates new coastline. Removed Swedish lake near Stockholm (had been extension of Baltic Sea in ocean theme) to lakes layer. Incorporates new coastline. Includes densified vertex along lines to allow smooth projection into conics. Fixes XAWXTN54GT.
We've launched a new project for PMC, a consulting firm that advises municipalities on things like transit policy and energy use. Energy Efficiency in San Gabriel Valley looks at a variety of cities in southern California and reports how much electricity and natural gas people used, how far they drove, how much waste they generated, and other metrics. We compare each city to the others in the Valley, to LA and SoCal as a whole when we can, and plot these metrics on an interactive map and series of charts below. It works best in Google Chrome, as PMC's initial use for the project is a conference on energy efficiency and climate change, held last week in Monrovia.
We're using terrainbackground tiles, with terrain-lines overlaid), for the base maps. The outlines are loosely based on the municipality boundaries, and fill up and empty out based on whatever metric's being compared. La Cañada, Flintridge and Irwindale have the highest Vehicle Miles Traveled counts, and we think this is because they're furthest away from the center of the Valley:
Measuring like this shows outliers pretty clearly. "Rock quarries dominate the small community of Irwindale, but the city is planning to attract more diverse land uses as some of the mines begin to close," and you can see this reflected in the Waste per Job statistics for that city. It also has many more jobs than it does residents, by about 13 to 1.
Further down on the page, the whole site is clickable, sortable, and otherwise interactive. Selecting a stat down below pivots all the rest of the data, so every location becomes a potential jumping off point for more comparisons.
Compare the more industrial cities to largely residential ones, and the wealthier cities start to emerge, like Bradbury. With the smallest population, but highest residential electricity use, larger homes are implied.
Or, you can infer that La Puente is fairly "self-reliant", and people tend to stay nearby, since they travel the least:
The editorial is pretty terrific as well: who knew that Baldwin Park is the home of the first Drive-In restaurant in California (In-N-Out), and the best performing city in the Residential Gas category?
Stamen alumSha Hwang and I shared a stage last night at Arkitektura's Design Assembly in their lovely Soma showroom. Besides the obvious awesomeness of sharing a stage with Sha (whose work at Trulia is up there with the best), it's always fun to talk to an audience of designers; their focus on how things look and the kinds of questions they ask bring a certain kind of energy. I also feel like I can let my hair down a bit (what's left of it), talk about the cultural aspects of what the studio does and explore some new, not-entirely-fleshed out ideas.
One of the ways I've been tricking myself into thinking new thoughts is to look at writings about other forms of expression and substitute the medium that's being discussed—painting, photography, architecture etc.—and replacing that with "data visualization." So if you take a look at what Group f/64 (Ansel Adams' cohort) said about photography:
"The members of Group f/64 believe that photography, as an art form, must develop along lines defined by the actualities and limitations of the photographic medium, and must always remain independent of ideological conventions of art and aesthetics that are reminiscent of a period and culture antedating the growth of the medium itself."
and drop "data visualization" in there:
"The members of Stamen believe that data visualization, as medium, must develop along lines defined by the actualities and limitations of the data visualization medium, and must always remain independent of ideological conventions of art and aesthetics that are reminiscent of a period and culture antedating the growth of the medium itself."
you wind up with some things to consider that we can toss into the mix of data visualization manifestos and what all this work is "for." Who would take seriously a manifesto about what photography if "for" that was this restrictive now? It's a way to jump the conversation into a more interesting place and start to anticipate a world where these kinds of visualizations are as common as photographs are now; maybe more so.
Another fun one is "fashion." I've been talking about with Ben Cerveny and others for a while now about the idea that Stamen's approach to mapping and data visualization is more like that of a fashion house than like a graphic design studio or a web development shop. Fashion, far from being superficial fluff on top of real culture, in this view is is highly technical (the Gaultier show at the de Young convinced me of this), an endeavor where innovation and new material is key, and is deeply embedded in and often leading aspects of culture.