This is a followup to yesterday's post on the visualization of a day's worth of trading data on the NASDAQ stock exchange. We've taken another look at the same dataset a bit more closely. In the examples that follow, each of which represents a single minute of trading, the image on the left uses a unique color to represent each trader, and the image on the right uses a unique color to represent each stock. So on the left hand (trader) side, a big grouping of the same color means that a single trader is buying or selling stocks. And on the right hand (stock) side, a big color block means a single stock being purchased in lots of different transactions.
In this first example, we see that a single trader (UBS, in this case) is responsible for the majority of the shares in this minute:
and that they're regularly trading a single stock at a single price at the same amounts (the yellow dots in a row):
Here we see that UBS buys a fixed amount of a stock at a fixed price, very steadily, stops abruptly, and then starts trading a different stock at a slightly lower price (the dark green and then blue dots on the right):
Here we see a single trader (the orangish square at left) perform a burst of concentrated activity within precisely deliniated margins, making small trades across a wide range of stocks (the kaleidosopic square on the right)
And here we see a similarly trader-centric burst (in blue, at left) spread across a multiplicity of small stock trades, just before the market closes for the day.
Earlier this year Zach Watson and I spent some time visualizing financial data. It's time to make that work public. The following images represent visualization of buy/sell data during a single day of NASDAQ trades.
We mapped a small subset of the variables for each transaction:
- time of the transaction, to the second
- whether it was buy or sell
- price of the transaction
- number of shares traded
Each of these variables is represented in the diagrams below. Each image represents a minute of time, and shows every trade that happens in that minute. Each trade is shown as a circle:
- Every vertical row is a second in time. So the left hand side of the screen is the beginning of the minute, the middle of the screen is 15 seconds in, and the right hand side of the screen is the end of the minute, with 60 seconds in between.
- Blue dots are buys, yellow dots are sells
- The vertical axis is the price of the transaction; the top of the screen is cheaper stocks and the bottom is more expensive stocks.
- The size of the dot is the number of shares traded; small dots are for a few shares and larger dots are for a larger number of shares.
NASDAQ opens for pre-trading hours at 7am, and for public trading at 9:30am.
The market opens. Fairly light activity in the first minute. Most of it is contained within the middle band.
Slightly more trades are happening, and they're for smaller amounts.
Someone seems to be buying shares at a low and high price, simultaneously - hence the lines at the top and bottom of the screen that match each other perfectly. We're not representing who's making these simultaneous buy/sell moves, but it would be easy to find that out or build it in.
This is about the pace we see for the next 2 hours, with the exception of a few bursts like this one right at 8:30am.
There's an incredible burst of activity just before public trading starts. It's completely unlike anything that comes before it. Our theory is that these are algorithms getting in one last set of tiny flurrying trades before the great unwashed masses come on board.
Right at launch, there's a giant burst of selling and trading, within seconds of the bell ringing.
And then the day starts:
Here's a video of what the data looks like when it's animated:
Untitled from Stamen on Vimeo.
There are literally thousands more where these came from. It's surprising us how much loveliness is in this financial data, which is generally perceived dry and boring, only interesting to bean counters.
What I like most about what's come out of this exercise is this idea that you can visually start to detect a difference between normal and anomalous data, even for what's normally considered data that lay people can't understand. If we could find a way to make it easier to understand what's happening in the markets, there's potential here for a kind of literacy in financial data that could help to offset some of the damage done by unscrupulous experts over the past few years.
The first thing I thought after we hung a copy of London's Kerning, a printed map showing only the street names in London, in the studio, was: I want one of those for the rest of the world. "How hard can it be to just (people here love that) show the streets?"
As part of the CityTracking project, we've released six new map flavors that get pretty close: Terrain and Toner both now include layers with streets only, labels only, and background only.
Toner: just streets
Those finding Toner
a little too, well, stark
for their purposes (it can be a bit heavy on the printer cartridges, we've heard) will hopefully find Toner Lite
a bit easier to work with.
Terrain: just streets
Terrain: just labels
Terrain: just background
We'll follow up with some more detail about how to incorporate these into existing projects soon—many of the styles have transparent backgrounds, for example, so they can be used as layers on other maps—but for now: enjoy!
We've got some new additions to Toner, the black and white style that Geraldine started and that Nathaniel and Mike have been gradually improving this year. There are some fairly significant changes to the cartography stack all the way through, which you can read about in detail at the project's visual changelog on GitHub. And of course everything's open source and available for download as per the terms of the Citytracking grant.
We promised to do this work in public, so here goes. One significant thing we've decided to do is to keep older versions of the project around, so that we (and, hopefully, you) can compare the different versions of the maps. So when Nathaniel talks in his post about "shaving San Francisco's Mohawk" from how it looked in 2010:
to how it looks in 2012, with a better coastline:
you can see it in situ. It's pretty simple to change the urls:
The changes can be fairly visually dramatic, as in the addition of non-Roman scripts to places like Tokyo:
The thing about designing maps is that you're never designing just one view. For one thing, it's important to account for all the different zoom levels: it's about showing more as you zoom in, but it's also about showing different things at different scales. Choices need to be made at every level about the thicknesses of streets, which buildings to show, which city name to show, and so forth. Different places have different characteristics spatially; some are more dense than others, and you have to keep the whole system in mind. These two versions of the zoom into DC, from different years, gives a sense of the range of choices involved:
I'm not aware of any other mapping projects that let you look back in time as a design evolves this way.
We're the cover story (!) of this month's Icon Magazine, featured alongside my longtime heroes at the Center for Land Use Interpretation as part of the maps issue. Ari Messer did a terrific job on the interview, and R.C. Rivera spent an afternoon photographing our plant- and map-filled studio with some lovely results. Unfortunately Shawn wasn't here for the photographing (I think he got married or something, whatever) so it's just Mike and I on the front spread, but overall I'm super happy with the resulting portrait of where the studio is now: 13 people, working in a garden in the middle of a vibrant city, a strong ethic, and maps and visualizations in active use by the public.
There's a lot in the article, but this bit is one I like enough to want to post here. I think it was me, who said:
Stamen finds inspiration everywhere, but Rodenbeck hopes that the public will stop conflating infographics with data visualization. "The rise of the infographic as a genre is a little depressing. Back when desktop publishing started, people were worried that there would be no more room for designers, that computers would do all the work for you. But this clearly didn't turn out to be the case." While someone without design training [or skill -- E] could make use of desktop publishing to create a holiday card or office leaflet or company newsletter, the band at the top for good designers actually grew. In a similar way, he says, "infographics have become the mother's day cards - the company newsletters - of data visualization."
Luckily, my office was a total wreck when R.C. took some pictures of it, but it gives a pretty good sense of what it's like around here these days:
Nathaniel showcased nearly two years of City Tracking in Boston last Monday at the MIT/Knight Story and the Algorithm conference. You can read about this years' winners here, as well about the changing nature of the grant: shorter cycles, more opportunities to apply, that kind of thing. It's worth a look.
The City Tracking project started off with Dotspotting.org, which allows mapping of data spreadsheets. The project still sees active use, but after we made the Toner tiles available for download we started to notice that more people were expressing interest in the background maps than in the ability to put dots on them.
Responding to public feedback while the project happened was something we were interested in from the start, and http://maps.stamen.com is the result: a browsable, embeddable, and otherwise immediately usable map of the whole world that can be used in Google Maps, Modest Maps, and other mapping APIs without having to download all of OpenStreetMap or tinkering with servers and technical code.
Nathaniel closed with a project we launched in March showing how climate change can be made personal on the street level, instead of the usual course brush strokes, with Climate Central's surging seas project.
These thin slices of big data are bite size morsels of aha. We hope you like them!
We've been working on a travel planner for the Weather Channel that tells you a bit more than just how long it'll take to get from point A to point B. This one predicts what the weather's going to be like along your drive, when you get there.
So let's say you're driving from New York to San Francisco, and you're trying to decide whether to go straight across or loop up or down a bit; this will give you a sense for whether it's going to be rainy or sunny when you plan to be in the middle of Nebraska. You can drag around the rainy bits if you like, and also along the way maybe you'd like to stop for a bite to eat, so we're hitting the Yelp API to give you a sense of where to go and what to see.
As is often the case, the research that goes into building one of these pieces can be as interesting as the final product. As an early stage proof of concept, we put together some images showing all the predictions for weather searches offset by time and location, with some (I think) lovely results:
They start to put me in mind of an early project we did here at Stamen, a set of travel time maps for MySociety that plotted the time it would take to get from one place in London to every other place in London:
I start thinking of weather maps that flow and ebb across the country, where different sliders open and ebb various kinds of other axes: time for sure, but maybe population density, maybe altitude, maybe temperature, maybe how many farms there are, maybe distance from a weather station or a McDonald's—all the different kinds of things that affect people's sense of place and space and time, organized by what's above our heads.
There are cupcakes with Stamen maps on them. Each one has a single tile printed on it. Cups and Cakes Bakery baked and prepared them. They were made for the Where 2.0 conference, where Mike is talking about how Old Is The New New and I am talking about Drawing Outside the Lines, which taken together is a pretty good indication of where Stamen's collective thoughts are.
THERE ARE CUPCAKES WITH STAMEN MAPS ON THEM.
maps.stamen.com, the second installment of the City Tracking project funded by the Knight News Challenge, is live. These unique cartographic styles and tiles, based on data from Open Street Map, are available for the entire world, downloadable for use under a under a Creative Commons Attribution 3.0 license, and free.
*takes deep breath*
There are three styles available: toner, terrain, and watercolor:
- Toner is about stripping online cartography down to its absolute essentials. It uses just black and white, describing a baseline that other kinds of data can be layered on. Stripping out any kind of color or image makes it easier to like focus on the interactive nature of online cartography: when do different labels show up for different cities? what should the thickness of freeways be at different zoom levels? and so forth. This project is the one that Nathaniel is hacking on at all hours, and it's great to be seeing Natural Earth data get more tightly integrated into the project over time.
- Terrain occupies a middle ground: "shaded hills, nice big text, and green where it belongs." In keeping with City Tracking's mandate to make it easier for people to tell stories about cities, this is an open-source alternative to Google's terrain maps, and it uses all open-source software like Skeletron to improve on the base line cartographic experience. Mike has been heading up this design, with help from Gem Spear and Nelson Minar.
- Watercolor pushes through to the other side of normal, bending the rules of traditional legibility in order to explore some new terrain. It incorporates hand-painted textures and algorithmic rule sets into a design that looks like it's been done by 10,000 slaves in the basement, but is rendered on the fly. Geraldine and Zach did the lion's share of the design and development on this one. This design is a mixed bag for me: I'm delighted to see it out in the world, but it's the thing that's pretty much kept me from looking at anything else for the last month and a half.
The code that runs Toner and Terrain is available for download and use at the Citytracking GitHub repository; watercolor we're going to wait on a little while until we can get some of the kinks ironed out. We talked about waiting to launch until watercolor was all buttoned up but what with all the attention that Open Street Map has been getting we decided to just bite the bullet and go for it.
We'll follow up this week with some posts on how everything works and how the sausage is made, and I've got a lot more to say about what I think this implies for what can be done with online maps and data visualization. In the meantime, have you seen how awesome Los Angeles, Washington DC, The Forbidden City, Massachussets Bay, Key West, London, New Orleans, New York, Versailles, and every other city in the cotton-pickin' world look when you point this thing at it? Holy heck.
The Forbidden City
We've been working with the smarties and do-gooders at Climate Central for the better part of a year now, designing and programming and planning and rendering and otherwise embiggening the idea of a map that could bring the reality of climate change to people's doorsteps. As of last week, the project is available at http://sealevel.climatecentral.org.
We started with two ideas:
- most maps of sea level rise are generic. How could we bring home the idea that this isn't about "those people" but about you and your neighborhood? and
- most maps of sea level rise are about the shrinking land. As a percentage of total land area, these always feel small and unsatisfying. Why don't we focus on the land that's going to be underwater, and try to make it clear that this is the land that we're going to lose?
The context for this work is: while there are a great many papers, scientific studies, meteorological surveys and other things that fall under the rubric of things that normal people accept as true, there remains a persistent and nagging unreality to the idea that, in something like a normal human timescale, we'll see and have to reckon with large-scale changes to the world as we know it. It's one thing to say "the world is changing and all of us will have to deal with it." It's quite another to say "7.6% of the people and 9.1% of the homes may very well be underwater in Boston, and so you'll need to start thinking about that pretty damn soon, is that cool?"
The political reluctance is certainly predictable — telling people about a long-term existential threat like this is just not a job that your average politician, elected for a term of a few years, may want to tackle. For that matter, it’s a hard subject for the rest of us to think about intelligently.
Although the risk of coastal flooding is slowly worsening year by year, it’s true that the worst consequences of sea-level rise, if they ever materialize, are still a long way off. Most people simply have trouble contemplating risks to their great-grandchildren. We’re a lot more interested in our own skins!
(As proof, I will tell you that two Times editors came by my desk this morning wondering about the near-term risks to their homes in low-lying areas of Long Island.)...
The Web site of Climate Central, where these sea-level studies were mainly done, has a great deal more useful information, including the ability to search by ZIP code and get a sense of your own risk.
(Justin Gillis, The New York Times)
Which is to say: if you're doing science and you want to have an impact on public discourse, pubish a rigorous scientific paper in the New York Times, and no mistake. But also: let people search for their own zip code, because what's personal matters.
Also: when the New York Times puts you on their front page, a couple dozen other news outlets do the same, and both Al Gore and Tim O'Reilly link to your stuff, it's probably a sign of two things: a) the alpha nerds are paying attention, and b) you're about to have a big traffic day.
At times on Wednesday the Climate Central computer servers were overloaded, so if you have trouble getting the search to work, try again a while later. (NYT)