Understanding Small Firms: Complementing big data with financial diaries
- Tracy Cole
- Sep 4
- 4 min read
Updated: Sep 9
In our last blog, we introduced the Small Firm Diaries USA, which explores the realities of America's smallest employer firms. These firms are everywhere — from your local flower shop, to your favorite family-run restaurant, to the electrician you call when the power goes out. In fact, firms with 1-20 workers make up the majority of employer firms in the US.
Despite their importance, firms in this category are under-examined in existing research. Much of the current data on small business comes from surveys like those conducted by the Federal Reserve, the U.S. Census Bureau, or the National Federation of Independent Business.
These large-scale, national surveys are an essential part of the picture. They offer, among other things:
Statistical significance: With large numbers of respondents, these surveys can reveal powerful insights across regions and industries.
Breadth: They provide high-level information on wide-ranging topics like revenue, credit use, business confidence, employment, and others.
Comparability: Regular, periodic surveys make it easy to compare responses over time to shape national policy and monitor economic trends at scale.
However, the benefits of doing large-scale surveys of businesses come with trade-offs. Firms may be consolidated into broad categories, such as those with between 1 and 500 employees, consistent with the widely-adopted definition used by the Small Business Administration. While this makes the data easier to analyze at scale, it blurs the different realities faced by the smallest firms at the lower end of those ranges. This means a catering company with three workers and a manufacturer with 200 workers both qualify as “small businesses,” despite clear differences in their financial lives and vulnerabilities.
Surveys also tend to go wide rather than deep. They can provide valuable insights into whether firms are using credit, the number of people they employ, and whether they expect to grow — but are limited in their ability to explore the “why” underpinning these decisions. It would be difficult for survey data to provide a full picture of how owners juggle competing financial obligations from week to week, what risks they weigh before taking on debt, or how they decide whether to adopt new technology. These daily trade-offs, coping strategies, and decision-making processes shape whether a firm survives, grows, or closes its doors, which in turn affects the livelihoods of millions of owners and workers.
Aggregating all small businesses together affects how we see them. If policymakers, lenders, and support organizations rely on the “average” small business profile to design interventions, they may skew solutions towards the needs of larger firms. As a result, the unique needs of the smallest employer firms could be overlooked, leaving critical blind spots in how we understand and support them. Moreover, surveys can struggle to capture insights from hard-to-reach groups such as firms owned by minorities or immigrants, located in low income areas, and employing low income workers.
This is where SFD USA aims to contribute. Our mixed-methods, financial diaries approach is designed to capture the day-to-day lived realities of small firms, including those that large surveys struggle to reach.
The financial diaries approach
The financial diaries approach is designed to provide richer insights on small employer firms through:
A focused sample: We target firms that have been in operation for more than 2 years, have at least 2 but not more than 20 paid workers (1099 or W2), and operate in low income communities and employ low income workers. This deliberate design aims to uncover rich data on underserved businesses that traditional surveys may miss.
A longitudinal design: We don’t just ask once; we collect data bi-weekly across a 12 month period. This creates a wealth of knowledge on how they handle seasonality, shocks, challenges, and opportunities.
A mixed-methods approach: Alongside quantitative data through bookkeeping records, we will conduct regular surveys with firm owners to explore topics such as growth aspirations, financial access and use, regulatory compliance, and much more.
Capturing data not traditionally examined. SFD USA aims to capture insights on topics that don’t show up in standard economic datasets; for example, the hidden burdens of administrative work, the creative strategies small firms use to cope with cash flow gaps, and the everyday experiences of workers in these firms.
This approach allows us to capture not just what firms do, but why they do it. For example, we may learn that a firm avoided taking a loan not because it was denied, but because the owner feared risking their family’s home as collateral. Or we might see how a sudden repair bill forced a business to delay paying workers, creating ripple effects for low-income households relying on a salary from the firm.
What will this research contribute?
Understanding the nuances of small firms in the U.S. is not just an academic issue. When we lack detailed information on these firms, policy can miss the mark, financial products may fail to meet firms’ needs, and support services risk overlooking the most pressing gaps. The implications of this are significant, particularly for Black-owned, Asian-owned, Latino-owned, rural firms, and those operating in low-income areas. If their stories are not accurately represented in the data, time and resources can be wasted promoting products and services that don’t meet their needs.
The stories revealed by Small Firm Diaries can complement the high-level view provided by larger, national surveys to:
Shape policy design: Diaries data can inform policies that are responsive to the specific challenges of small employer firms like smaller budgets and highly volatile cash flows.
Inform more supportive financial products: By revealing how small firms actually use (or avoid) credit, our data can inform underwriting models and product design that better align with their needs.
Enable better targeted support services: Nonprofits and business associations provide valuable training and technical assistance, but these resources can miss the mark if they aren’t targeted. Diaries highlight where the real gaps lie—whether in digital adoption, bookkeeping, or workforce management—so that support can be tailored to what firms actually need.
Strengthen local economic development: For cities and states looking to strengthen neighborhoods and spur revitalization, small firms are essential. By uncovering how these businesses make decisions and navigate constraints, diary data can guide smarter, more effective economic development strategies.