Big data, little farmers markets Part 2: The minefield of analyzing Big Data

In the first installment of this series, I introduced the idea of Big Data, the Internet of things (IoT) and what social media has promised and what it has delivered. I promised some thoughts on analysis next. here goes:

•Big Data is partly defined by its resistance to analysis. The volume, velocity and variety of Big Data makes problems for easy collection and analysis. This story on the struggle among safe street advocates to find good data speaks to that issue.

•Big Data is probably more appealing to advertisers than to our often shadowy government at this point but still, we should keep an eye on both of them and their analysis/use of Big Data.

•Lastly, as put so well by the author of Dataclysm: Who We Are (When We Think No One is Looking), much of behavioral science research is based on WEIRD research: White, educated, industrialized, rich and democratic nation’s subjects. Big Data may help to offset that issue.

Markets already intersect with Big Data across many different sectors, such as health care, the public sector, agriculture and retail. So let’s think about how this could play out for markets:
What if a researcher used the total dollars spent at markets on SNAP and compared it to grocery store SNAP sales on a map, not adjusting for hours open or the number of goods or markets available or fixed costs to offer those goods? Or how about the decrease in certification for organic farmers among market vendors – What if that was just a graph showing the decrease year after year, without the analysis that many farmers stated that they feel they do not need certification while they sell directly to shoppers and are therefore able to explain their practices? What if those maps/graphs were what influenced policymakers?

Some scenarios to ponder:

    •Market A (which runs on Saturday morning downtown) is asked by its city to participate in a traffic planning project that will offer recommendations for car-free weekend days in the city center. The city will also review the requirement for parking lots in every new downtown development and possibly recalibrate where parking meters are located. To do this, the city will add driving strips to the areas around the market to count the auto traffic and will monitor the meters and parking lot uses over the weekend. The market is being asked for its farmers to track their driving for all trips to the city and ask shoppers to do Dot Surveys on their driving experiences to the market on the weekend. Public transportation use will be gathered by university students.

    •Market B is partnering with an agricultural organization and other environmental organizations to measure the level of knowledge and awareness about farming in the greater metropolitan area. For one summer month, the market and other organizations will ask their supporters and farmers to use the hashtag #Junefarminfo on social media to share any news about markets, farm visits, gardening data or any other seasonal agricultural news.

    •Market C is working with its Main Street stores to understand shopping patterns by gathering data on average sales for credit and debit users. The Chamber of Commerce will also set up observation stations at key intersections to capture visual data on visitor behavior.

    •Market D has a grant with a health care corporation to offer incentives and will ask those voucher users to track their personal health care stats and their purchase and consumption of fresh foods. The users will get digital tools such as cameras to record their meals, voice recorders to record their children’s opinions about the menus (to upload on an online log) with their health stats such as BP, exercise regimen. That data will be compared to the larger Census population.

In all of these cases, the data to be collected crosses sectors and systems, meaning that no one entity has all of the raw data at their disposal at all times. That boils down into Analysis Issue #1

In all of these cases, the data to be collected has many ways to be interpreted, based on which entity is interpreting the data. Analysis Issue #2

In most of these cases, the data collected requires some self-reporting. Analysis Issue #3

In some of these cases, privacy controls must be strictly managed and will affect how much analysis can be done. Analysis Issue #4

from the New York Times:
“The first thing to note is that although big data is very good at detecting correlations, especially subtle correlations that an analysis of smaller data sets might miss, it never tells us which correlations are meaningful (italics added). Analysis Issue #5

Check out this site for fun examples of how matching correlations doesn’t always add up to good conclusions.

The thing we should be able to agree on: all partners should be involved with the analysis and should receive access to the raw data. That means markets participating in just the data collection piece is not enough. They need to be involved in the analysis because if not, the context of markets will be lost.
Yet we know that just collecting the data is be a massive undertaking for low-capacity markets (even assume some funding is offered in all of these cases for the partners to staff the collection of the data), not even adding in the time and effort it takes to analyze it. What might help is to have some analysis prepared ahead of time and to prepare the market community for participation.
1. This means that every market association, or group of markets or markets themselves should keep information about each market’s history, size, structure and staffing in separate PDFs. This, by the way, is a resource that Farmers Market Coalition (for whom I am a consultant) is working on with one of their university partners, the University of Wisconsin to pilot for their AFRI Indicators for Impact project . Hopefully, the Market Profile will be available online for all markets to test in 2015- stay tuned!
2. Markets need to know the area’s current demographic and other relevant details. Check the census to know what the larger population’s stats are and make friends with real estate professionals to keep up on trends in the neighborhood.
3. Do a Dot Survey or Bean Poll a few times a year asking shoppers to tell you what zip code they live in, how they come to the market, things like that and keep track of that data. Maybe a big dry wipe calendar on the wall to add all data collected?
4. Market boards and advisors should keep any data already collected and the Profile information to be able to share it as needed in any meeting they happen to attend in their own professional lives.
5. When researchers do come to your market with an offer to help with data collection, be ready to ask for data you want. How about asking for focus group data so that a market can begin to build “persona profiles” of those who come to the market? Or ask for added analysis for numbers that you think might be important for the market: those who know me have heard my song about finding a way to track the number of return SNAP shoppers and how I think it that metric is so useful for markets and possibly even more useful than total SNAP dollars, in terms of analysis.

5. Encourage city or county public health agencies to offer a semi-annual breakfast for those entities that work on community interventions (like markets, health clinics, social service entities, university programs, youth outreach etc) to share news about what they are seeing in their field and to share any data informally. If meetings are impossible, then a regular email would work. In other words, stay in touch with other data collection efforts in your community.

I’ll end this post with some of the lovely words of Dataclysm author Christian Rudder who was talking about the Vietnam Memorial’s physical self versus its online database self:

“A web page can’t replace granite. It can’t replace friendship or love or family either. But what it can do – as a conduit for our shared experience – is help us understand ourselves and our lives. The era of data is here; we are now recorded. That, like all change is frightening, but between the gunmetal gray of the government and the hot pink of product offers we just can’t refuse, there is an open and ungarish way. To use data to know yet not manipulate, to explore but not to pry, to protect but not to smother, to see yet never expose, and, above all, to repay that priceless gift we bequeath to the world when we share our lives so that other lives may be better – and to fulfill for everyone that oldest of human hopes, from Gilgamesh to Ramses to today:that our names be remembered not only in stone but as part of memory itself.”
I think I’ll adopt that bit as my mantra.


Big data and little farmers markets, Part 1

Recently, I have been reading a few books and articles on the new world looming over the next bend. This new world is called many things and includes shiny named ideas and tools to make it so. Here are some of those titles in case anyone needs some bedside reading:

•Collaborative Commons (Rifkin, (The Zero Marginal Costs Society)
•Disruption (Next City 2012, Fortune 2014 “Next up for disruption: The grocery business”, Urbanophile 2014, Disrupting the Disruptors )
•Flattened economy (Friedman The World Is Flat: A Brief History of the Twenty-First Century 2005)
•Spiky Economy (Florida, “The World Is Spiky” 2005)
•Alternative Economics, Community-Supported Industry (Anderberg 2012, Schumacher Center for New Economics)
•Social impact bonds (Jacobin Magazine Issue 15–16 “Friendly Fire”)
•Placemaking/Livable Places (PPS, Tactical Urbanism, CityLab)
•Human-Centered Design (LUMA, Ideo)

and then bunches on how to measure this stuff:
•Measuring Urban Design: Metrics for Livable Places (Ewing, Clemente 2013)
•3 Keys To Better Data-Driven Decisions (Technology Evaluation Centers)
•Five Borough Farm II: Growing the Benefits of Urban Agriculture in NYC (Design Trust for Public Space 2014)
•Data Infoactive (Chiasson, Gregory 2013)
•Disruption Index (Next City 2012)
•Livability Index ( 2014)

and so on. (and please feel free to send me any that you find useful).

Much of this discussion of the new economy and its infrastructure centers around the use of technology to allow data (usually known as Big Data) produced by every system, sensor, and mobile device to be shared across sectors and users – aka the Internet of Things (IoT). Big Data and IoT are representative of what is both good and bad about the new world; they pressure public entities to adopt private sector characteristics and measures, and conversely, ask private entities to add public sector transparency as a mode of operating in this new world. Additionally, both sectors must respond immediately to any trends or innovations. This can be good and bad.
 (The intersection of public and private is what the non-profit sector is supposed to exist and, increasingly how it participates in Big Data, is a measure of its ability to do just that. I’ll come back to that very idea later in this series.)

Examples of Big Data:
Think of how that grocery store loyalty card transmits information about what, when and where customers purchase goods. Or citizen used tools to measure and report pollution, or how that electronic parking card tells the city the peak parking hours, letting planners know the need for more (or less) parking facilities. Or, the sensors that are timed to go off for irrigation to start for food production.
For food system advocates, the connection to data sharing is mostly through the public health sector at this point, but the planning and design sector of governments will be wanting data from us too and then, you can expect the line to form from other sectors after that.

Social media is not the center of Big Data, but it’s already helping to study the behavior of its millions of users. In the interdisciplinary Cornell University course entitled “Networks, Crowds and Markets” taught by professors David Easley, Jon Kleinberg, and Éva Tardos, they use data from online networks to talk about “strong and weak ties” and “bridges” and to map the patterns of why, how and when connections are made and what impact those connections have in the fields of economics, social sciences, and public health, among others. Since social media is mostly networking, informal updates, and chatter, (constant and sometimes as cheerfully mindless as an acquaintance’s wave from across the street), it may seem without value, but it is certainly changing the way that we communicate.
Social media can also power revolutions, allow for professional development and offer small businesses appealingly designed, low-cost online faces for their already-developed customer base. This blog you are reading is part of social media and as such, is written to be ephemeral and chatty opinion with links to other information sources rather than hosting peer-reviewed reports.

Recently, I had the good fortune (thanks to the Farmers Market Coalition) to be invited to a Knight Foundation technology gathering of social entrepreneurs and so heard many ideas for leashing the power of Twitter and other social media platforms to better aggregate data or reorganize news feeds. No doubt as new platforms are built on top of the first tier, there will be more usability and versatility, but for now, many people view it as a multi-platform address book to keep track of friends, colleagues, and friends of friends.

The ease of using social media is what was beguiling to many at first but the gossamer veil of privacy means that if not careful, one’s identity may be stolen or become the target of a bully. At that point, that once-enticing open entry can drive plenty away and that very fact is what is being argued about sites like Airbnb and Uber: 1) that the lack of regulation at city halls or public agencies allows for exemption of rules that their counterparts with physical outlets are not able to sidestep and 2) since there is often no face to face meeting between buyer and user, the perceived opportunity for criminal activity increases. My feeling is that the regulation needed for the IoT and online sites must be a new system rather than asking for adherence to the old since the old grey mare of city hall or the federal government is not suited for managing these (which sounds like what the community food system has been saying for the last few decades!)
The European Commission has already published a report outlined some best practices for architectural, ethics and governance of the IoT, highlighting social justice, privacy and opting out concerns (“consent activities” in designer language). Their early conclusions encourage better credential exchange systems and a deeper awareness of “reliance versus trust” parameters. In short, make sure most online relationships include a requirement for sharing some sort of identification and create some active boundaries between systems. Maybe the U.S. community food system can jump on these ideas, thereby leaping ahead in confidence levels to be able to share useful data more rapidly than other sectors.

Yet, even with the perception of these systems as being hackable, an increasing number of people in the Western world still participate regularly even while others hoot it down while they cling to their wall phone and postal stamps as their talismans against the new world of constant updates. Those folks are not likely to let us forget that social media is just a part of the communication sector and only the ephemeral part of it. We still read newspapers and books, meet people face to face and still have postal carriers and grocery store corkboards with lists of apartments to rent.
Therefore, how we use social media within community food systems has to be balanced far better than we early adopters have done so far. Plenty of markets and other food system initiatives use social media brilliantly within its limited use, but others often ignore traditional media entirely by not factoring in that those reached with social media are only a tiny portion of the audience that might be found. Or conversely for the Luddites among us, the need to adapt their thinking to understand that social media has worth for a low-capacity, face-to-face entity like a Saturday morning market.
What I have noticed is that social media helps drive farmers market or CSA sales for a single or a few products on a single day extremely well. It also does a passable to good job reminding its users that they are members of a larger community of doers and thinkers, which can extend the social and human capital of a market. It can connect producers to shoppers on non-market days (although I think less well than promised) and can do something akin to the Dot Survey method pioneered at market by Stephenson, Brewer and Lev: allow for an easy mood of the day give and take between market organizers and users. It also is that friendly wave from across the street that in our sped up world can stand in for reminder of community on a bad day and add a layer of connection. Let’s just not build our world entirely on chance meetings or depend on a small number of tools.

update This morning, I am sitting in a farmer workshop at Southern SSAWG conference listening to a 5th generation farmer talk about the open source free crop planning software system, sensors, and apps that he uses to run his direct marketing farm business; clearly, for some, the IoT is already here.

Coming Soon
Part 2 The minefield of analyzing Big Data
Part 3 Connecting farmers markets and food systems to Big Data
Part 4 Managing face to face and online communities in farmers markets

Coming Soon
Part 2 The minefield of analyzing Big Data
Part 3 Connecting farmers markets and food systems to Big Data
Part 4 Managing face to face and online communities in farmers markets

104 Fascinating Social Media and Marketing Statistics for 2014 (and 2015)

some of the ones that caught my eye for food organizers:

There are 76 million millennials (born between 1981 and 2000) in the U.S. — 27% of the total population.

63% of millennials say they stay updated on brands through social networks; 51% say social opinions influence their purchase decisions; and 46% “count on social media” when buying online.

37% of marketers say blogs are the most valuable content type for marketing.

Pinterest grabs 41% of the e-commerce traffic compared to Facebook’s 37%. Food is the top category of content on Pinterest with 57% of its user base sharing food-related content

“Interesting content” is one of the top three reasons people follow brands on social media.

17% of marketers plan to increase podcasting efforts this year.

47% of Americans say Facebook is their #1 influencer of purchases.

70% of marketers used Facebook to gain new customers.

104 Fascinating Social Media and Marketing Statistics for 2014 (and 2015).

10 Surprising social media statistics that might make you rethink your social strategy – The Buffer Blog

here’s a sneak peek at one of those stats:
YouTube reaches more U.S. adults aged 18–34 than any cable network

10 Surprising social media statistics that might make you rethink your social strategy – The Buffer Blog.

Breakdown of how people use social media

How many markets are using Pinterest or Instagram, I wonder?

Social media demographics

Social media demographics

11 “Donate Now” Best Practices for Nonprofits

A great article with tips for using donate now badges on your site. Everyone needs to find ways to increase their earned income so that they can add long term supporters and reduce the annual grant writing frenzy. The best way to do that is to make sure that your listings and website send the right professional message and reach as many supporters as possible.

11 Donate Now Best Practices for Nonprofits « Nonprofit Tech 2.0 Blog :: A Social Media Guide for Nonprofits.

HOW TO: Calculate the ROI of Your Social Media Campaign

Great information as to measuring your “Return on Investment” for all of that time the market spends on social media. With these calculations alongside your one time a year SEED survey, you would have some incredible data!

HOW TO: Calculate the ROI of Your Social Media Campaign.