Indicators (sick of them yet?)

With the announcement of the 2018 FMPP/LFPP RFA this week – tucked into the Specialty Crop Block Grant announcement- I wanted to alert you to this 2017 post below about the indicators that are included in the proposal.

There is also a shorter version on FMC’s website.  Here is the link to it. )

Congratulations to everyone who got their FMPP/LFPP grants in by the deadline yesterday. I talked or emailed with a few of you throughout that process and was impressed by the well-crafted strategies that I read and heard about.

As you can imagine, a lot of the calls I was on focused on the new prescribed indicators (performance/outcome measures) that were included with the RFP for the first time. Those were the same for FMPP as for LFPP projects and were:


 OUTCOME 1: TO INCREASE CONSUMPTION OF AND ACCESS TO LOCALLY AND REGIONALLY PRODUCED AGRICULTURAL PRODUCTS.

Indicators 1. Of the [insert total number of] consumers, farm and ranch operations, or wholesale buyers reached, a. The number that gained knowledge on how to buy or sell local/regional food OR aggregate, store, produce, and/or distribute local/regional food b. The number that reported an intention to buy or sell local/regional food OR aggregate, store, produce, and/or distribute local/regional food c. The number that reported buying, selling, consuming more or supporting the consumption of local/regional food that they aggregate, store, produce, and/or distribute

2. Of the [insert total number of] individuals (culinary professionals, institutional kitchens, entrepreneurs such as kitchen incubators/shared-use kitchens, etc.) reached, a. The number that gained knowledge on how to access, produce, prepare, and/or preserve locally and regionally produced agricultural products b. The number that reported an intention to access, produce, prepare, and/or preserve locally and regionally produced agricultural products c. The number that reported supplementing their diets with locally and regionally produced agricultural products that they produced, prepared, preserved, and/or obtained

OUTCOME 2: INCREASE SALES AND CUSTOMERS OF LOCAL AND REGIONAL AGRICULTURAL PRODUCTS.

Indicator 1. Sales increased from $________ to $_________ and by ______ percent ( n final – n initial/n initial (100) =% change), as result of marketing and/or promotion activities during the project performance period. 14 | Page 2. Customer counts increased from [insert total number of] to [insert total number of] customers and by _____percent ( n final – n initial/n initial (100) =% change) during the project performance period.

OUTCOME 3: DEVELOP NEW MARKET OPPORTUNITIES FOR FARM AND RANCH OPERATIONS SERVING LOCAL MARKETS.

Indicators 1. Number of new and/or existing delivery systems/access points of those reached that expanded and/or improved offerings of: a. ______farmers markets. b. ______roadside stands. c. ______community supported agriculture programs. d. ______agritourism activities. e. ______other direct producer-to-consumer market opportunities. f. ______local and regional Food Business Enterprises that process, aggregate, distribute, or store locally and regionally produced agricultural products. 2. Number of local and regional farmers and ranchers, processors, aggregators, and/or distributors that reported: a. an increase in revenue expressed in dollars: _____ b. a gained knowledge about new market opportunities through technical assistance and education programs: ______

3. Number of: a. new rural/urban careers created (Difference between “jobs” and “careers”: jobs are net gain of paid employment; new businesses created or adopted can indicate new careers): _______ b. jobs maintained/created:_______ c. new beginning farmers who went into local/regional food production: _____ d. socially disadvantaged famers who went into local/regional food production: ______ e. business plans developed:____

OUTCOME 4: IMPROVE THE FOOD SAFETY OF LOCALLY AND REGIONALLY PRODUCED AGRICULTURAL PRODUCTS.

Indicator(s) – Only applicable to projects focused on food safety. 1. Number of individuals who learned about prevention, detection, control, and intervention through food safety practices:_____ 2. Number of those individuals who reported increasing their food safety skills and knowledge:______ 3. Number of growers or producers who obtained on-farm food safety certifications (such as Good Agricultural Practices or Good Handling Practices): _____

The applicant is also required to develop at least one project-specific outcome(s) and indicator(s) in the Project Narrative and must explain how data will be collected to report on each applicable outcome and indicator.



These confounded many,  while others knew exactly how to use these to define their grant’s outcomes. I hope that  USDA calls in some of those who do a bang up job in setting and achieving their numbers to talk with the newbies in future years.

Because of the previous work on the trans•act tools (which include the SEED tool) while at Market Umbrella, and the more recent and engrossing Farmers Market Metrics (FMM) work I have been doing with FMC and their partners these last few years, I have become very familiar with this language and these indicators.  Most are included in the metrics chosen by FMC to be collected starting in 2016 with FMM through their own projects and through offering support to networks that area ready to embed evaluation systems in their projects.

Since I spent some time working with various project leaders on this, I thought I’d give my two cents here as to how I’d approach these if I was the lead.In this post, I’m going to talk about my general theory of data at the grassroots level and the first two outcomes; I’ll tackle #3 and #4 and unique indicators in upcoming posts.

Some may disagree with my assessment of how to handle these indicators which to me is actually a good thing since by tackling this in varying ways,  we are likely to hit on the best  methods of establishing these baseline numbers and for collecting the data.

The first thing that confounded some proposal writers is how every indicator could be met by the varied projects: of course, they cannot and are not expected to. Since some projects are focused only on increasing sales at a market and not on increasing the number of outlets, some indicators are more relevant than others and should be used in more detail. Remember, these indicators are for both FMPP and LFPP projects which covers a wide spectrum and so are meant to support the general outcomes for all. It is my opinion that the unique indicators asked for at the end are likely to be the most useful for reviewers to read closely in order to match to the narrative or budget. I’d expect though that those proposals that could not reasonably answer a majority of the indicators with numbers will suffer in that reviewing process, as did USDA it seems, as they recommended in their webinar that everyone explain those that they couldn’t answer. Or if possible, add a piece to their project to address that indicator. And I think you can assume that USDA was being firm in saying that this pot of money should result in changes of these kinds, so if your project cannot reasonably do any of them, maybe look elsewhere for support.

I think the best way to really make these outcomes accurate is for the project lead to write them with the vision of using them as a banner to fly throughout the term of the project for the team to hit, surpass or to discuss why they cannot be met and what that means. And that the numbers should be slightly lofty-it is better to extend the reach at the outset and urge the team to do their best work to reach or even surpass it. However, don’t just throw some outrageous numbers in there or you will be telling the reviewers and your team that you have no intention of achieving them. So even though I used the word lofty, there is something in being efficient with your project through establishing very precise numbers too.

Efficiency is a good plan for our tiny organizations in order to conserve ours and our vendors’ energy for the long haul and to be there for another day. And that how well we plan and how we address our assumptions about those we hope to reach has a lot to do with setting numbers and meeting or achieving them.

Okay let’s look at the first two outcomes now:

Outcome 1: Increase consumption and access.

The indicators that are clustered with this outcome are related, meaning that once you have established the  (a) the number of buyers and or producers that gained knowledge, you can then estimate the number (b) of those that then report an intention and then finally, the number (c) that reported actually buying, selling, aggregating etc. The second part of this outcome is related to those professionals like chefs or incubator-users who, if the project is expecting to reach that audience, then they are also going to be measured for knowledge, intention and actual activity.

I think this one was written out particularly well done as it takes a project step by step through the process of establishing their reach. This should have been relatively easy for most projects, as knowing how many people you plan on reaching is sort of 101 for FMPP or any USDA grant!

USDA’s suggestion was to write them out in a mathematical formula writing a beginning number, then the number you want to hit and then calculating the percentage of increase. It may be helpful to do that in 2 columns and consider both the direct and indirect ways that your project will reach people. Certainly, if you are doing training or workshops you can estimate your attendance, but how about those who just read about your training or workshop and track down the info that way? How about through the media that your project uses to gain attendees? Is it reasonable to think that others will hear about the market or outlet and begin to attend because of it? And never forget the vendors and including them into any project outcome, even if it is a straight up new shopper project; the vendors also can learn about the marketing and use it in their own sales reach if it is shared properly.  And of course, how about the project partners and their reach?

Once you set the number who will gain knowledge (and I think that your project should plan that just about everyone that gets your materials or attends your workshop will gain knowledge) you then think about who will change their behavior because of it. I wonder if I had a group of market managers and a group of vendors in one room and asked them to gauge that if 1,000 people are reached through materials or training, how many they think will actually intend to use it, and then how many will actually use that knowledge to buy, sell aggregate etc what differences we’d see. Because that estimate can vary, based on the perspective and experience of those setting the number.

My feeling would be that the vendors would assume that more people will intend to come but would think that less will actually buy. I say that because they deal with everyone directly and know painfully well how many pass by their table without eye contact or a deep perusal of what is for sale. So they know firsthand how getting people to actually do something is hard. I’d say that managers would be more likely to think more people will be reached but that less would report an intention to come to a market, but that once they are there, that a higher percentage will purchase. My assumption may be entirely wrong and maybe someday I can test it and readjust it. The most important thing is to test your project assumptions by asking everyone for numbers and adjusting them accordingly to their bias and experience and according to your plan.

I also think percentages without numbers can be difficult to be realistic about, so I often suggest that people start on the wrong end: if the project is for increasing shoppers to a single market, how many more shoppers could that market actually handle per week? 100? 200? 1000? Think about the vendors and your space and your Welcome Booth and visualize adding that number every week. Would it overwhelm the market? Do you have enough parking or access to transportation to make it happen? How many added shoppers per hour would that mean to your anchor vendors? Is that worth it?

Remember that the average shopper in most markets spends between 10-30 dollars so using those numbers above, the market would add another $1000 -$30,000 week in sales. Pretty cool huh? Or if you hope to add another market day: Maybe your Saturday market has 45 vendors on average, you might estimate that since your new market is smaller and has less parking, that you hope 25 or so can use this new outlet. In both cases, your initial outreach has to be wider than the final number, as some will not get to your market or have the ability to add market days even when told of the opportunity.

Outcome 2: To increase sales

Couldn’t be simpler as, in most cases,  FMPP projects are still chiefly attempting to increase sales. It may be true that at some later date, sales increases are not the primary indicator of the success of our work, but with the small reach that alternative food outlets currently have with food shoppers, I agree that this should still be the main goal. Even so, this indicator stymied more people (and I would imagine contributed to some not writing a grant at all) and since it is a common metric for FMM, I’m going to attempt to reason why it is necessary and how we can capture this.

Measuring an increase of sales for a project that is going to do marketing or outreach for a single sales outlet is pretty standard.  The issue is that you need a baseline number (starting point) and that is the thing many markets do not have yet. So how do you find the baseline?

Everyone knows that the majority of markets ask for standard stall fees which are not based on vendors’ sales percentages and because of that, many markets have never asked for sales data from their vendors*. What USDA, FMM, Wholesome Wave and others are now saying is that we need to know the impact of our work whether you collect this data for the market’s fee rates or not. So, for those who do already collect it, you are ahead of the curve and probably have a lot to teach the rest of us about how to do it well.

So how do the rest of us do it? Well, the simplest way is to ask vendors directly, either every market day, every month or every season. As you can imagine, the longer you wait to ask this, the more difficult it becomes for the vendor to separate the numbers from your market from the other outlets he/she sells at. However, it also is difficult for multi-tasking vendors to stop at the end of the day to count their money and get that number to you. So what works best? My answer is one that some people hate hearing: whatever works best for your community and your management level is what works best- as long as it gives you accurate data in increments acceptable to those using it.

I’ll talk your ear off about accurate data whenever discussing market evaluation because it is my experience that markets rely too much on anecdotal information and estimates that probably are better described as guesstimates as they have almost no basis in real numbers. I can hear many of you yelling at me through your computer that you are not evaluators and cannot be expected to gather data. My answer to that is as soon as you create projects that use the resources of partners and promise your community some change in behavior because of these efforts, you are both. Meaning as soon as you decided to run a market. (You like how I run the entire argument on my own and that I get the last word?)

However, I am in agreement with many market leaders and vendors that too much data is often asked of markets or vendors that is never used or not shared back with those who offered it. And of course, that collecting the data and the costs associated are almost never added to the cost of any project, and usually, partners just assume that overworked market communities will just throw that added work in their long list and get it to them toot sweet.

Yeah, don’t get me started on data collection challenges here.

Additionally, sales data is at the top of the sensitive information asked presently and I often ask managers or market partners to tell me how much is in their bank account right now as an example of how asking for information without context or reason is alarming to say the least. That is, if you even know a precise number! So I say first be the change you want to see by sharing market data with vendors regularly: token sales for debit are going up but SNAP is steady? What do you think that means? And then ask them what they think it means.

Asking for it in anonymous sales slips is the  way FMM suggests it is collected, but I assume that there are other good methods to test. And that it helps all of those methods when the raw data is shared with the vendors and it is used to advocate for their needs. It must be said that to be able to use it in aggregate means it has to be collected in the same way for the same time period and a lot more data is needed to get to any collective contribution, so we do need to hit upon some common methods sooner rather than later. Here are two more possibilities:

And as many of you know, the SEED tool  asks shoppers to estimate their purchases and then calculates overall sales from those numbers.  Many feel this method of getting sales is better, but it does require more surveying of shoppers more often which means added staff and volunteers.

Another way may come as some markets grow their token systems. Depending on your market, it might be possible to estimate how many of your shoppers use that system and whether it is representative of the type of overall shopper you have and use the data to estimate sales.

The main point is we have to agree that we need some data and it should be as precise as possible without violating privacy or exposing weaknesses in one business over another- after all, this is a competitive place. The data you can use for internal analysis as to the market’s impact on its vendors and shoppers can be a lot less and a lot less specific than the data your research partners will need when they start to calculate economic numbers. And that until you have actual data, how you calculated your starting point for these indicators says a lot about your circle of advisors, your experience and your knowledge of the target population.

Whew; enough for now. I’d love to hear how some of you did calculate both of these outcomes and especially sales, both in systems you had baselines and ones that did not. I expect that some of you will disagree with much of my unscientific approach to measurement but hope you know that I welcome your opinions.

Counting public gatherings in 2017-Washington Post article

The point of this post is to show how complex and grassroots public gatherings can be counted and measured. The two main researchers quoted in these Washington Post articles are Erika Chenoweth and Jeremy Pressman, both respected analysts of the details of large-scale civil movements and gatherings. As a data junkie, I have followed this effort with a great deal of interest (and have even counted some of these gatherings in my own town to check others’ counts) and look forward to more of the analysis of both the methodology and the actual count data. The analysis included not just the number who gathered but who and what was being protested or being supported, where these events were held, what symbols were used, how many arrests were made.

For March 2017, we tallied 585 protests, demonstrations, marches, sit-ins and rallies in the United States, with at least one in every state and the District. Our conservative guess is that 79,389 to 89,585 people showed up at these political gatherings, although it is likely that there were far more participants.

Certainly, food and farming systems should note some of the systems used for collection and analysis. For example, the Crowd-Counting Consortium may be something that national entities involved in any grassroots data collection systems like food systems should discuss creating for their own use.

Here is their counting method:

We arrived at these figures by relying on publicly reported estimates of march locations and the number of participants involved in each. We started a spreadsheet and called for crowdsourced information about the location and number of participants in marches. Before long, we had received thousands of reports, allowing us to derive low and high estimates for each event. We carefully validated each estimate by consulting local news sources, law enforcement statements, event pages on social media, and, in some cases, photos of the marchers. When reports were imprecise, we aimed for conservative counts; for example, if observers reported “hundreds” of participants, we reported a value of 200 (“thousands” was 2,000, “tens of thousands” was 20,000, etc).

An example of their public data set.

https://www.washingtonpost.com/news/monkey-cage/wp/2017/04/24/in-trumps-america-whos-protesting-and-why-heres-our-march-report/?utm_term=.ce99baecf0b6

Communicating community

Hopefully, all of you who read this blog are okay with my use of Vermont as one of this blog’s recurring examples of food system work. I will caution my readers to refrain from assuming that the Vermonters think they have it all figured out just because their residents are rightly proud of its glorious revival of small-acreage farming and its rep as an organic stronghold. I’d say the state food and farming leaders are very honest about the issues that they continue to face and their assessment of what remains to do. For example, there are still no full-time market managers at all and the average market manager makes less than 10,000 per year (really, it’s likely much less but I am trying to not overstate it here) and, in a state with only 625,000 residents (the most rural state in the union with 82.6 percent of its population living in either rural areas or small cities, and many of them poor*), the 80 or so markets are always struggling with maintaining attendance and sales amid strong competition from co-ops and other well-regarded outlets. And of course, like everywhere else, the state’s farmers are pulled in so many directions trying to serve every outlet at once while dealing with weather and regulatory woes and the typical small business challenges that many are not profitable.

What is exciting is that they all try to work collaboratively at the network level to seek appealing ways to showcase producers and organizers’ hard work. The pictures below are an example of that. My home team there (NOFA-VT) has an artist who does lovely work that hang on their walls and whose art is used by NOFA in many other ways. During my last visit, Erin Buckwalter showed me this bowl in their office of farming “affirmations” from that artist some of which also include actual data. She encouraged me to take a handful and so I have been asking people at markets to reach in to my bag of cutouts and take one. What a simple way to display the difference in our system from the one that reduces everything in a store to a place for purchased advertising. So if you see me, ask to dip in and see what you get…

IMG_4714.JPGIMG_4855.jpg

IMG_4854.jpg

This one, from NOFA-VT’s imaginative and thoughtful Executive Director, will remain on my board to inspire me.

 

 

• Income and Poverty in Vermont

iMedian household income (in 2015 dollars), 2011-2015 $55,176 (US $53,889)
iPer capita income in past 12 months (in 2015 dollars), 2011-2015 $29,894 (US $28,930)
iPersons in poverty, percent Warning Sign 10.2% (US 13.5%)
Fifteen states have more than half their populations living in rural areas or in towns under 50,000 population.

Bring your food waste to the library for composting

Food waste collection programs are being phased in at New Orleans public libraries.

I’m glad they also mention Greenmarket and their innovative compost collection program. What is significant about the NYC market program is that Greenmarket does not occupy their market spaces constantly, so managing programs like composting require added logistics for the staff.

In data collection terms for markets, this program can be measured for its ecological, economic, social and intellectual capital benefits.

Bring your food waste to the library for composting: Yes, really | NOLA.com

2016 SNAP retailer report

Fascinating.
 This is from the 2016 SNAP retailer data report from USDA/FNS, found here. This is the kind of data that we need to pore over and gather data at the market level to compare and contrast.
Some quick data and my thoughts:
• 53 percent of the 366,972 households that shopped at a farmers’ market or direct marketing farmer made one purchase; another 18 percent made two purchases; and 29 percent made three or more purchases within the year. These percentages have remained relatively unchanged over the last five fiscal years.
So 53% only made a single purchase using their benefit card… This may correspond with the analysis from my 2013 VT Market Currency report that markets were then using a whole bunch of money and time to attract shoppers who are either at the end of their time on the SNAP rolls, or are new to SNAP and need more assistance in using their benefits, or that markets have not yet made enough changes to be welcoming for that shopping base to use their benefits at markets. So this data should be crunched with what part of that 53% has been enrolled on SNAP for ___ period of months.
The other way we should analyze this would be compared to our cash/credit shoppers too, which means we need to collect that data too in the same way. That is the kind of comparison that would tell us what the best initial strategy is for SNAP shoppers.

• In fiscal year 2016, program recipients made 1,095,107 purchases at farmers’ markets and direct marketing farmers nationwide. The average purchase amount was $18.60.

Almost 1.1 million purchases has a nice ring to it. Of course so does 10 million.That would be a great goal for the market field: 10,000,000 purchases at markets and with dmfs in a single FY by 2019. Sometimes the obsession over only measuring total dollar amount spent limits the strategy for increasing actual uses of benefits in ways that are more useful in retail terms…

•  366,972 SNAP households made at least one purchase at a farmers’ market in fiscal year 2016. Households shopping at farmers’ markets spent $55.51 on average over the course of the year.

 • More SNAP benefits were redeemed at farmers’ markets and direct marketing farmers in fiscal year 2016 during August than any other month of the year.

 

Some of this data needs to be released per state, as the August spike is likely not true in the Southern states. It’d also be great to see here which week or days of the month it spiked as well.

One business shares a gratitude index

In what has to be the most common topic in this blog for 2016, I offer another illustration of how data collection can be simply managed, yet offer meaningful data for a wide variety of audiences.
The elegant metrics from a small value-added business in Charlottesville VA are below. It seems important to note that Stacy was my comrade for the original creation of the FMC Farmers Market Metrics work (and FMC’s founding executive director before our current excellent Jen Cheek) , and did a incredible consulting  job in 2016 assisting FMC with metric refinement, data collection research and devising the workbooks that are at the core of the FMM program. Those tasks are in addition to what the output shows in the metrics below, her devotion to fitness as a student and a leader and (along with her ever cheerful husband Joe) raising their energetic son.

2016 was good phyte foods’ first full calendar year of operation, after our first humble sale at Charlottesville City Market in May 2015. Since January this year, the business has made from scratch and sold 57,024 crackers, 1,003 bars, 157 pounds of granola, 978 pumpkin muffins, 533 kale cookies, and a motley assortment of odd-looking experimental snacks. In so doing, we have bought sustainably grown (and often certified organic) vegetables and herbs from at least twelve farms² in Central Virginia. We are grateful not only to our loyal and patient customers at City Market, but to our enduring local partners at The Juice Laundry, Random Row Brewing, Blue Ridge Country Store, and ACAC Downtown. Thank you. Thank you. Thank you. It is a humbling honor to be part of this evolving local food economy.

And for those who think cottage industry producers only care about their own cash flow:

In the coming year, we want to expand our reach to more retail partners, reinvest in new equipment, train some cracker apprentices, experiment with early summer cracker flavors, and give back to community organizations that are helping kids eat more vegetables. I have recently been appointed to the Advisory Council of the Urban Agriculture Collective of Charlottesville, which, among other things, teaches children of limited economic means how to grow, use, and appreciate fresh organic vegetables.

Go Stacy Miller; Keep the goodphyte going.

Source: getting specific: gratitude counts in our 2016 index