Finding water data is harder than I thought. Like detective Gittes in the movie Chinatown, I’m poking my nose around and asking everyone about water. Instead of murder and slimy deals, I am scouring the internet and working with city government. I’ve spent many hours sleuthing and learning about the water system in our city.
In San Francisco, where this story takes place, we have three primary water systems. Here’s an overview:
The Sewer System is owned and operated by the SFPUC. The DPW provides certain engineering services. This is a combined stormwater and wastewater system. Yup, that’s right, the water you flush down the toilet goes into the same pipes as the the rainwater. Everything gets piped to a state-of-the art wastewaster treatment plant. Amazingly the sewer pipes are fed almost entirely by gravity, taking advantage of the natural landscape of the city.
The Auxiliary Water Supply System (AWSS) was built in 1908 just after the 1906 San Francisco Earthquake. It is an entire water system that is dedicated solely to firefighting. 80% of the city was destroyed not by earthquake itself, but by the fires that ravaged the city. The fires rampaged through the city mostly because the water mains collapsed. Just afterwards, the city began construction on a separate this infrastructure for combatting future fires. It consists of reservoirs that feed an entire network of pipes to high-pressure fire hydrants and also includes approximately 170 underground cisterns at various intersections in the city. This incredible separate water system is unique to San Francisco.
The Potable WaterSystem, a.k.a. drinking water is the water we get from our faucets and showers. It comes from the Hetch Hetchy — a historic valley but also a reservoir and water system constructed from 1913-1938 to provide water to San Francisco. This history is well-documented, but what I know little about is how the actual drinking water gets piped into San Francisco. homes Also, the San Francisco water is amongst the most safe in the world, so you can drink directly from your tap.
Given all of this, where is the story? This is the question that I asked folks at Stamen, Autodesk and Gray Area during a hyper-productive brainstorming session last week. Here’s the whiteboard with the notes. The takeaways, as folks call it are, are below and here I’m going to get nitty-gritty into process.
(whiteboard brainstorming session with Stamen)
(1) In my original proposal, I had envisioned a table-top version of the entire water infrastucture: pipes, cisterns, manhole chambers, reservoirs as a large-scale sculpture, printed in panels. It was kindly pointed out to me by the Autodesk Creative Projects team that this is unfeasible. I quickly realized the truth of this: 3D prints are expensive, time-consuming to clean and fragile. Divide the sculptural part of the project into several small parts.
(2) People are interested in the sewer system. Someone said, “I want to know if you take a dump at Nob Hill, where does the poop go?” It’s universal. Everyone poops, even the Queen of England and even Batman. It’s funny, it’s gross, it’s entirely human. This could be accessible to everyone.
(4) Think about focusing on making a beautiful and informative 3D map / data-visualization of just 1 square mile of San Francisco infrastructure. Hone on one area of the city.
(5) Complex systems can be modeled virtually. Over the last couple weeks, I’ve been running code tests, talking to many people in city government and building out an entire water modeling systems in C++ using OpenFrameworks. It’s been slow, deliberate and arduous. Balance the physical models with a complex virtual one.
I'm still not sure exactly where this project is heading, which is to be expected at this stage. For now, I’m mining data and acting as a detective. In the meantime, here is the trailer for Chinatown, which gives away the entire plot in 3 minutes.
Outside our own kitchens, gas leaks aren’t something we’d ever thought much about here at Stamen. But being a studio filled with progressive San Francisco do-gooders and all, we do think about climate change and sustainability. We also think an inordinate amount about all the sensors out in the world gathering data – satellites, airplanes, cellphones, Jawbones, and, of course, those Google Street View cars.
Stamen worked with EDF and Google Earth Outreach to design and build a system that EDF staff could use to crunch complex data files used by scientists into easily digestible maps that give an immediate sense of the scale of leaks in various cities around the country. For now, the maps cover only Boston, Indianapolis, and Staten Island, but Google will keep gathering data in more cities and the system we built for EDF allows them to easily add more maps as more cities come online.
Here’s New York, which is totally covered in methane leak occurrences:
As is Boston:
You might ask, is this normal? Well, take a look at Indianapolis:
There’s a lot less going on.
It was important to EDF and Google that we show the paths the cars drove, as well as the leaks detected, since the absence of leaks could mean sound pipes, or just an area that hadn’t yet been sampled. We took our cue from the familiar blue lines of Google Street View maps to show the drive paths, and then desaturated the base map to make sure the focus stayed on the leaks and sample areas.
Among our design goals for the project was to balance the visual presentation against the nature of the data. The leaks detected are really something called “verified peaks” — elevated methane levels detected frequently enough and at high enough levels to be significant. But, given winds and the potential for leaks to be contained in buildings and a thousand other variables, a measured peak doesn’t necessarily mean there’s a leak at an exact location out to n-decimals of latitude and longitude. So we opted for larger dots and diffuse edges.
And all these leaks are much lower than anything that would cause your neighborhood to erupt in flames. So the overall palette is a bit muted — nothing’s going to explode!
But our vast natural gas system — from wells getting drilled across the county to the networks of pipes in our cities to the knob on your kitchen stove — does make a huge difference for the future of our climate. We’re delighted to have helped EDF and Google begin mapping the part of that infrastructure that most closely touches tens of millions of people across the country.
We’ve been quiet about something cooking here at Stamen, in part because it’s new and experimental, and in part because we’ve been so busy doing it that we haven’t made time to write yet. That something is our Education program, run by Beth.
Stamen has had internships before, but this summer we wanted to take it a step further. Rather than having students work on bits of client projects, we decided that we really wanted to create a program that affords both research and design on a single project. This approach gives fellows the opportunity to complete a work with us that they feel proud of and that advances the current state of knowledge in their area of interest.
Although each fellow is working on their own project, what unites them all is a focus on urban data and making something invisible about the city of San Francisco truly visible.
Meet the Fellows
Andreas Viglakis // Uncovering Bay Area Transit
Andreas is a recent graduate of the Harvard Graduate School of Design, and has worked as an architect and urban designer in Shanghai, New York, and Hong Kong. His work this summer, in partnership with urban planning think tank SPUR, focuses on San Francisco’s fragmented transit system and how it's affected by the changing economic climate of the Bay Area.
A scale study comparing the size of the Bay Area to the size of other major cities. Shown above: Bay Area over New York.
Kristin Henry // Storytelling with Sounds in San Francisco
Kristin Henry, a fellow in partnership with Gray Area, is a generative artist and computer scientist specializing in science and data visualization. She also founded GalaxyGoo.org, a small non-profit for science literacy, where she leads a group of volunteer tech + art + science enthusiasts. During mid-career graduate studies in Computer Science, she developed an Android application that collects sound and some Python scripts to analyze and visualize the resulting collection of audio recordings. This summer, she’s using this technology to develop a sound map of San Francisco.
Sketches of walking paths and recordings.
Scott Kildall // Water Works
Scott Kildall is a visual artist who writes algorithms that transform various datasets into 3D sculptures and installations, which imagine “what data looks like.” His project for the Creative Code Fellowship is called Water Works and will include a 3D-printed data visualization of the San Francisco water system, which includes drinking water, sewer and storm pipes and emergency fire-fighting systems, all of which interact in various ways to provide a dynamic city infrastructure. His fellowship this summer is in partnership with Gray Area and Autodesk's Pier 9 Workshop, who is providing access to their workshop and advanced machine training in conjunction with their Artist in Residence program.
Screenshot of a mesh of manholes of the San Francisco water system, z-axis exaggerated.
Moira Forberg is a current MBA student at Harvard Business School and is interested in the intersection of design and technology. Prior to school, she worked on hardware design for 3D printing projects at MIT and Disney Research, and worked as a strategy consultant at OC&C. This summer, she is helping Stamen design best practices for intellectual property (IP) management around internal research projects and productization.
About the classes
So this is all great, but why is Stamen getting involved in education in the first place?
Stamen has always been a place for experimentation and research, typically through our research projects. We’ve also been involved in teaching people how to do this kind of work for a number of years now, historically with a focus on developers. While that focus continues to be important, the world of visualization has changed and what’s new is a focus on mapmaking and data visualization literacy for all skill levels, and an effort to support education around truly creative coding (i.e. the beautiful universe that exists beyond charts and graphs).
Inspiration for developing this practice around education came from Maptime, which Beth started here at Stamen last year. In addition to helping even more people learn how to make maps, it’s also building a community through education. We aspire for Stamen’s education program to have a similar community building aspect to it as well, using educational and research activities as a way to engage the public, use public data, and when possible, to collaborate with some of the talented visualizers and cartographers and mathematicians and artists on our network map and beyond.
Additionally, we believe it’s important for businesses in our space to pay it forward. This sentiment has its roots in the spirit of open source software development, and in practice it also gives us an opportunity to meet new people and try new things. Thanks to our fellowship program, for example, we’re learning so much about visualization-based installations, physical computing and non-web visualization. It gives us a chance to be students again, as well as teachers.
Eric’s always talking about how he never wanted to work for a cigarette company during the day to pay the bills, and for Greenpeace at night to salve his conscience, but instead wanted to commit a studio towards relevant work that also pays for everyone’s 401(k). Beth’s always talking about how she dreams of starting her own experimental, forward-thinking school that focuses on art, design, technology, and sustainability. Together, we foresee some lovely opportunities and collaborations in our future. It’s starting here in the Bay, with Gray Area, Autodesk, and SPUR. We’re excited to see where this experiment goes next.
Interested in collaborating with Stamen on a educational activity? Want to get more information about next year’s fellowships and internships? Drop a line to education [at] stamen [dot] com.
Stamen has long aspired to make it easier for people everywhere to visualize data, particularly on and with maps. In our recent partnership with Tableau, we’ve helped to improve a tool that does just that. Tableau’s latest version, 8.2, comes complete with a mapping suite designed by us. Suddenly it’s that much easier for people all over to make beautiful maps with their data.
The maps come in three varieties: light, dark, and blue water.
Light maps are meant to be as subdued as possible, using hints of terrain to provide texture.
The dark maps are designed for bright colors, barely hinting at the outline of country borders.
Blue Waters are meant to provide a similar subtlety as the light maps, only with more natural tones.
Our goals for the project were:
A clean, modern look and feel.
Consistency across the three maps. They should be strong enough to stand alone but work together as a set.
That the maps be beautiful in themselves, but primarily built to display data.
Finding the right shades for the background, shades that work well with all of Tableau’s standard palettes, brought a level of science to the color process that we had simply never practiced before. Typically Stamen's approach to color is an intuitive, experimental and evocative one; color theory is present but not primary. Tableau’s maps, in contrast, required rigorous testing and iteration for even the most incremental tweaks in order to retain contrast. Tableau’s Visual Analyst, Maureen Stone, was our guide deep into the world of color science. In doing so, she taught us more about color calibration than we ever would have thought to learn. We now know that color is not just about trusting the eyes; it’s also about trusting the numbers. The numbers are what allow a wide range of colors – important for data layers – to have consistent levels of contrast and pizazz.
This connection of art and science is part of what made the project come out so well. The resulting map design system has a consistency and intensity to it that we haven’t seen with any other out-of-the-box analytic tools.
We applaud Tableau’s use of Leaflet and OpenStreetMap for creating and presenting these maps. We're proud every time we can help one of our clients make the switch to designing with open data.
This year, people across California began getting health insurance coverage in new ways, thanks to President Barack Obama’s signature Affordable Care Act, often called Obamacare.
As the new marketplaces, subsidies, and penalties got discussed and debated in the media this spring, we at Stamen were hard at work with our frequent client the California Healthcare Foundation, creating a system to visualize and communicate a range of key measures to assess the impact and effectiveness of Obamacare.
It was initially a bit daunting to develop a systematic, dynamic way to display all this raw data, but we were determined to help make the ACA understandable by applying some delight to the figures.
Within each indicator, the data is sliced in several ways: types of care, providers, demographics, enrollment, quality of care, income levels, even geographic averages. With so many categories, the data can give you a general overview or a very specific peek into health coverage. After relentless explorations, we’ve developed a system we think works pretty darn well. And it’s a solution flexible enough to accommodate new data in years to come.
The data is broken down into three categories: Domain, Topic, and Indicator. Once the user makes their selection, they are taken to the indicator’s page, which then has a variety of data points to look at.
By default, the page opens to the Total datapoint and the most recent year of data collected, with trending data displayed on the left. Indicators with data from more than one year also let you browse through the different years. Once you click on another tab, you have the ability to compare those specific data points to the trended totals, as well as the ability to export, share, or embed the data you’re looking at.
The combination of data points results in thousands of potential exports. Here are some examples of other charts you’ll find in the project:
The technical challenges in the project were in how to make the whole system run off a simple set of spreadsheets that anyone could edit in Excel. We worked hard to make sure that CHCF can manage the project into the future, adding new indicators and assigning chart types to them, all via simple text files they can publish through their existing content management system.
We’re looking forward to seeing those trend lines fill out for years to come — and, we hope, measure ever better health coverage and health care for California.
Climate change is one of those things that can be so invisible, and in some cases so subtle, that it can be hard to conceptualize. Over the years, our work with Climate Central has sought to make one symptom of this looming problem – sea level rise – much easier to see. The maps we've made together reveal what's lost as water makes its way upon our shores, rather than the mass of land that's left behind. Our most recent work with them, in partnership with New America Media, shows us very clearly what we'd lose in California: the homes of hundreds of thousands of people.
90,000 of those people live in San Mateo County alone, 50,000 of whom are ethnic and racial minorities. Climate Central estimates the cost of this damage to be in the realm of $22 billion. Cities all along California's coast will face similar losses.
This version of the map, released today, lets us see a very poignant and personal part of the story, one more specific and far more real than a population density map could reveal. Inspiration for this method came by way of the by the Cooper Center's Racial Dot Map.
It's not just the numbers that are blowing our minds. It's the difference that we're seeing in each city and state, each of which has different a different datasets, and different levees for that matter.
We can see in Hayward that a large Black and Asian population will be affected:
Here are some specific numbers:
And here is the overall social vulnerability for the area:
I especially love parks. I spent almost a decade telling stories of local parks and wildlife for Bay Nature Institute, and toward the end of my time there I was constantly trying to push the envelope of what a small nonprofit could do with technology.
Now that I’m at Stamen I’ve got a slightly different view of how park agencies and nonprofits can and should engage with technology.
Back at Bay Nature, I thought, “I’ll figure out precisely what we want and do all the research, and then I’ll just hire someone to make that exact thing!”
I’ve now seen this same thought process at work in a few RFPs for mobile websites and interactive maps for parks and trails. Figure out every possible feature you and your colleagues might want, put them in a giant list, throw in some house-made wireframes, and then send out the RFP for precisely what you know you need.
Makes perfect sense. Except:
It doesn’t work.
What you think you know at the beginning of the process may well turn out to be dead wrong, and, more importantly, you lose so much of the big win of hiring professional designers when you try to solve all the design problems ahead of time.
When Stamen took on the effort to redesign the maps of the Golden Gate National Parks Conservancy, we were fortunate to have flexible, and all-around awesome, project leads at the Conservancy. Michael Norelli and Mason Cummings were ready and willing to hear that the enormous feature set they’d assembled was too big to do well in one scope of work.
Instead, we focused on the key goal: Get people to the parks.
Being able to focus that way was incredibly important to making sure the effort was successful. It meant the Trip Planner was the most important thing to build out fully, complete with bespoke directions to obscure locations and special warnings if parking at Muir Woods was impossible.
We made some awesome Trail Views (and those weren’t in the RFP), but the key to success here was understanding that getting people to the parks was the top priority and the main measure of success.
What about the original RFP’s several other pages of requirements, from hawk tracking to editable management zones? We had to let those go for future phases of work. If we’d tried to do them all, we wouldn’t have finished anything well. Check out the Golden Gate National Parks Conservancy's open source code repository if you want to do a deep dive into the code that made that project possible.
Better Scoping Principles
Prioritize: define must-haves, then cut your list in half
Start simple, move quickly
Release early, release often
Be ready to change your priorities
Favor open data, standard formats, and existing tools
Avoid letting data management dictate design
A common thread in parks and mapping RFPs is that data pipelines, GIS workflows, and general data management sometimes receive a lot more attention than the ultimate goal, which is usually something like getting people to parks or helping people discover new trails or find programs in parks.
That's natural, since the people writing the RFPs are the ones who most often wrangle all that data. But unless you’re really doing a pure infrastructure project, resist the temptation to let data management and your internal needs dictate the thing you’re getting built.
Which leads to:
Use the kinds of software you aim to hire someone to build!
This might seem obvious, but it’s surprisingly rare, especially if you’re a longtime parks pro. Hey, you already know tons of trails and parks and birds and nature lore, why do you need to use an app or website?
Well, first, you might discover some things you don’t know or new ways of looking at things you do know. But more importantly: Just trying out existing tools and really using them is a huge source of insight. Heck, you might discover something that works so well that you want to just hire the people who built it to make a special version for your park or agency.
Keep it simple! (That is hard.)
You might notice that the best existing services and applications are often the simplest. Whether through dumb luck or, more likely, previous mistakes combined with persistence, they’ve figured out what people want most and they’ve given it to them. And they’ve made it easy.
You can do the same in your search for a technology firm to build your stuff: Keep it simple. Favor qualifications and past work over ability to navigate complex documents and matrices. Find a firm with which you can have a relationship based on collaboration and trust. Hire them, take some risks, be ready for small failures in the service of big wins. And have fun.
Recently the fine folks at WIRED's Map Lab asked for a checklist for maps. We have a list of interaction design-related things we like to ensure across all of our map projects which we thought you might be interested to see. Here goes!
URLs should contain and maintain state, by default. This feature means that you can deep-link into any place, any zoom and even any data layer's state.
Clicking UI elements affects the URL too, e.g. show/hide legend should also affect the URL.
The map should be integrated with any external displays, so one affects the other in real time. You can see this at work on the Parks Conservancy Trails Page.
The application should know when all required tiles are loaded, and act accordingly.
e.g. Make sure tiles are loaded before any animation begins.
e.g. Don't necessarily display data overlays before the tiles have loaded, unless they're easy to place geographically, which is rare.
Mouse scroll should not affect map zoom level when the map is embedded in a page. There's little more irritating than scrolling quickly down a page and getting stuck way zoomed out of a map you hadn't intended to interact with.
Double click should zoom into the click point on zoom, not the center of the map.
No flickering (unless it's intentional). Work hard to make map transitions as smooth as silk.
All data and tiles should load smoothly. Tiled data is good for this, as are compression and content delivery networks.
Wait until the new data is ready and loaded before hiding older data. (Avoid showing grey tiles.)
e.g. switching between map styles. Don't throw the old stuff away and show grey tiles in between the switch.
If you can, make a full screen version, or at least make the map as BIG AS POSSIBLE, like on the Chesapeake Bay Grasses project.
What's on your list? Anything you'd like to see added to ours? Tweet @stamen + @wiredmaps + #maplist, or send an email to info [at] stamen [dot] com.
When I first moved to San Francisco from Atlanta a few years ago, the first thing that struck me about the city (aside from the expensive everything) was the pace. People here go so fast, how can they possibly stop and smell the roses? Our speed at Stamen often reflects the pace of this city, and amid all the work, sometimes we forget to stop too. I think that's why we started ordering in lunches, so we wouldn't forget to eat, either.
So today I want to pause, to take a moment to reflect on some of the lovely things that have been happening round these parts just in the past couple weeks.
Eric gave a talk at TEDx Market Street. In addition to showing some Crimespotting maps of the Tenderloin, he spoke about some of the great work he's doing with the Kenneth Rainin Foundation and the Community Arts Stabilization Trust (CAST) in the Tenderloin. Much of that work is kept out of the studio (for reasons both legal and practical) so it was a first for me to see him speak so personally and passionately about turning alleyways into gardens, empty storefronts into art galleries, and buying buildings to create rent-controlled artist studios in a city where artists (and Outreach Managers, fwiw) can barely afford to eat, much less live.
We released an open call for our first ever Fellowship! We've done internships before, but this is the first time we'll be working with a student to get them to sink their teeth into a datavis project of their very own. We couldn't be more excited to get this started next month just after EYEO, and to work with friends at Gray Area, Helios, and Obscura over the summer.
Dan and I hung out with California Open Spaces staff and advocates at the Bay Area Open Spaces Conference to talk with them about parks.stamen.com. Social media usage in the parks is a hot topic, and a controversial one: some parks administrators are delighted about seeing social media usage around their parks, while others are less enthused about encouraging technology use in open spaces. Learning about all sides and perspectives is key to the project, so we loved being there and collecting 140-character comments on post-it notes, instead of Tweets and pics.
George and Heather made us some more postcards! We were totally out of postcards. And now we're not! Hooray! #outreachmanagersdreamcometrue
Top it all off with meeting amazing artists U-Ram Choe and Theo Jansen within days of each other (one met off the clock; the other one on it during Scott Kildall's artist talk at Autodesk), I've gotta say the past few weeks have been filled with some loveliness. Let's hope for more like this in the weeks to come.
This is a real opportunity for someone to come and work with us and Gray Area for the summer. The project comes with a modest stipend and will give selected Fellow(s) the opportunity to hone your research and visualization skills with guidance from a world-class team.
The project focus, which we'll decide on together, will focus on data visualization and civic engagement. It should address a contemporary issue (that’s not the Google Bus) within the city of San Francisco, involve data of some kind, and be visually represented within the Fellow’s technical capabilities. Ideally it will make visible something that was previously invisible or not well understood.
Don't have the coding chops yet for a fellowship but want to get started? Fret not, dear artist-to-be! Gray Area is offering a Creative Code Intensive this summer, designed to get you from 0-60 in 10 short weeks. Follow the links below to learn more. If you've got questions not covered on the Creative Code site, feel free to email beth [at] stamen [dot] com.
Best of luck with your application, and happy creative coding!