Caveat to AI for knowledge management

In an earlier posting, I argued that knowledge management might come close to being the “killer app” needed to spark a major boom in AI use.

Daron Acemoglu writing a piece in Project Syndicate has laid out a slightly contradictory argument. He notes (and many others have also noted) that productivity increases are coming from the automation of routine cognitive tasks.

“Early adoption of generative AI has naturally occurred where it performs reasonably well, meaning tasks for which there are objective measures of success, such as writing simple programming subroutines or verifying information. Here, the model can learn on the basis of outside information and readily available historical data.”

But “evaluating applications, diagnosing health problems, providing financial advice – do not have such clearly defined objective measures of success, and often involve complex context-dependent variables (what is good for one patient will not be right for another). In these cases, learning from outside observation is much harder, and generative AI models must rely instead on the behavior of existing workers.” As a result, productivity gains from AI will be lower than many expect.

Such cases are certainly more of a challenge. But relying on the behavior of existing workers is half of what knowledge management is all about. Capturing the expertise of workers is what makes knowledge management so important and so difficult. It calls for articulating the unarticulated. Often referred to as Polanyi’s Paradox (“we know more than we can say”), tacit knowledge underpins much of what we refer to as expertise.

David Autor argues that the ability to capture and share expertise is what makes AI so powerful. In an article published earlier this year, he contends that “By providing decision support in the form of real-time guidance and guardrails, AI could enable a larger set of workers possessing complementary knowledge to perform some of the higher stakes decision-making tasks currently arrogated to elite experts like doctors, lawyers, coders and educators. This would improve the quality of jobs for workers without college degrees, moderate earnings inequality, and — akin to what the Industrial Revolution did for consumer goods — lower the cost of key services such as healthcare, education and legal expertise.”

However, Autor forcefully articulates the need for AI, like any tool, to be grounded in foundational expertise. “By shortening the distance from intention to result, tools enable workers with proper training and judgment to accomplish tasks that were previously time consuming, failure-prone or infeasible. Conversely, tools are useless at best — and hazardous at worst — to those lacking relevant training and experience. A pneumatic nail gun is an indispensable time-saver for a roofer and a looming impalement hazard for a home hobbyist.” Getting knowledge management right is critical, as a recent story on Boeing in the Wall Street Journal points out.

I noted above that capturing tacit knowledge is half of the task of knowledge management. The other half is sharing learning. Expertise is not fixed. It changes and evolves. There is not a set pool of knowledge waiting to be uncovered. Learning is a process, not an end point. Business and governments will have to understand that deploying AI is a constant process. The temptation of a train-once-and-done approach is appealing—and dangerous.

In the Industrial Economy, managers and workers understood that tools wore out. In the age of AI, we need to recognize that expertise and knowledge can become obsolete. A train-once-and-done approach would lock in the existing “good” practices. It would freeze expertise and undercut development of judgement. Thus, any AI and knowledge management system need to have a built-in mechanism for dynamic renewal. That mechanism must be built on a process of constant monitoring and accurate evaluation.

There is another reason why AI needs constant monitoring, evaluation, and adjustment. AI is, as one research put it mildly, often dumb as a rock. A recent Washington Post story recounts some of the more hilarious examples of how Google got it wrong. The author, Shira Ovide, advises that “With the generative AI from Google, OpenAI’s ChatGPT and Microsoft’s Copilot, you should assume that they’re wrong until proved otherwise.” Others have pointed out that the problems of “hallucinations” (made up answers) and incorrect information may be a built-in feature, not a fixable bug. Referred to now as “slop”, this trait in AI will require ongoing human oversight.

Bottom line is that AI has tremendous potential, especially as a tool for knowledge management. But realizing that potential may be harder than many originally believed. And humans will, as far as we can see, remain an important part of the system.

Corporate disclosure of information on human capital

Twenty years ago I wrote a paper Reporting Intangibles: A Hard Look at Improving Business Information in the US where I argued that investors, managers, regulators, and policymakers are essentially flying blind. In that and other publications I advocated that the SEC mandate certain disclosures of basic data on intangibles, including human capital. Now the topic is heating up.

Late last year, the SEC’s Investor Advisory Committee recommended that companies be required to disclose basic data about their human capital:

1. The number of people employed by the issuer, broken down by whether those people are full-time, part-time, or contingent workers;

2. Turnover or comparable workforce stability metrics;

3. The total cost of the issuer’s workforce, broken down into major components of compensation; and

4. Workforce demographic data sufficient to allow investors to understand the company’s efforts to access and develop new sources of talent, and to evaluate the effectiveness of these efforts.

Action on mandatory disclosure of this type of information is sorely needed. According to the 2024 JUST Jobs Scorecard, only 46% of the Russell 1000 companies disclose retention or turnover rates and only 11% of companies disclose their internal hiring rates. Only 40% of companies disclose the average annual hours of training and only 11% offer more than 30 hours of training per employee.

The SEC has taken some steps toward expanding disclosure in the MD&A (Management Discussion and Analysis) section of SEC-required financial statements in 2020. However, more needs to be done. Disclosure needs to be mandatory, not subject to the companies’ discretion as to whether or not the information is material. And the information needs to be standardized so as to allow for cross-company comparisons.

The SEC is reportedly working on proposed rules on Human Capital Management, supposedly to come out soon. Those rules were scheduled to come out in April. Now, they are expected by the end of the June.

We shall see.

Labor market continues to surprise, again

Today’s employment data showed US employment surged this past month. According to the Bureau of Labor Statistics (BLS), non-farm payrolls were up by 272,000 in May. Economists had expected an increase of 180,000 to 190,000.

Once again there was strong employment growth across the board in the intangible-producing sectors. Payrolls in intangible-producing sectors was up by 172,800. Tangible services was up by 73,800, lead by Trade, Transportation & Utilities and Accommodation & Food Services. And goods producing sectors were up by 25,000, mostly Construction and Mining.

And the growth was concentrated in the Big 5. Intangible Educational & Health Services employment was up by 72,800. Governments payrolls (excluding the Postal Service) grew by 42,300. Professional and Business Services reversed its decline in April to increase by 25,300 in May. Employment in Trade, Transportation and Utilities was up by 27,000. Payrolls in the tangible services area of Accommodation and Food Services grew by 25,300.

I would note that Arts, Entertainment, and Recreation payrolls were also up by 16,600.

It is somewhat worrisome however that the Educational & Health Services dominates the labor market, along with Government. One wonders how long these sectors con continue to grow. That said, payroll growth in May was a little more balanced than in previous months.

April was not good month for trade: Intangibles trade surplus declines

They say that April is the cruelest month. For US trade, that was true. According to data released today by the Bureau of Economic Analysis (BEA), the overall trade deficit increased by $6 billion to $74.6 billion as imports surged. More disturbing was that the trade surplus in intangibles declined, again as imports grew and exports stagnated.

The really bad news is that the trade surplus in Financial Services dropped, net charges for the use of Intellectual Property were down, and deficit in Insurance Services grew. The only good news was in Business Services, where the surplus increased slightly after 8 months of steady decline. (see charts below)

I would remind everyone however, as the first chart below shows, the deficit in goods far exceeds the surplus intangibles (and tangible services).

Frontier Firms and Universities

Picking up on an intriguing study on “The Effect of Public Science on Corporate R&D” by Ashish Arora, et al. presented a couple of weeks ago to the NASEM Innovation Forum (presentation here, slides here). In the study, the authors look specifically at the impact of knowledge (publications), inventions (patents), and human capital in the form of Ph.D.’s (people) on the level of internal corporate R&D. Their findings are somewhat surprising, but only somewhat:

• firms respond to an increase in public invention by producing fewer corporate patents and publications
• firms respond to an increase in human capital by producing more corporate publications and patents
• firms do not respond to an increase in public knowledge

In other words, traditional academic research publications had little impact on corporate R&D; university patents had a negative impact on internal corporate R&D; and availability of human capital had a positive impact on the amount of internal corporate R&D.

From that, they conclude the following: abstract research has little immediate value to corporations; university patents are substitutes for corporate R&D; and, hiring more Ph.D. spurs more internal corporate R&D.

Their overall assessment of the university-company relationship is blunt:

“Our findings suggest that university research is most relevant for corporate innovation not as abstract, non-rivalrous ideas, but rather as embodied, market-supplied inputs. Incumbent corporations appear to have a limited ability to absorb and use abstract ideas produced by universities. It is only when those ideas are developed into inventions that they become relevant to firms, reducing the demand for internal invention by incumbent corporations and hence also reducing the demand for internal research.”

At first blush, it seems to undercut the case for publicly supported university-based research. If commercialization (and economic development more broadly) is the goal, then it appears that university research does not produce the biggest bang-for-the-buck. But on closer examination, such a reading of the paper’s findings misses the nuances.

The overall conclusions of the paper are based on the average of all companies. However, one of the paper’s major findings is that there is a difference between firms on the technological frontier and the so-called follower firms: “Frontier firms tend to continue investing in internal research and invention, even in the presence of abundant public science.” The authors speculate that “frontier firms may benefit more from public knowledge and skilled PhDs to fuel their internal research and inventions than followers.”

I did note one area of clarification and possible future research. The authors argue frontier firms maintain higher levels of internal R&D in areas with less public invention. The glaring counterpoint is the life sciences where there are high levels of internal corporate R&D and high levels of public knowledge (publications), public invention (patents), and human capital. It would be interesting to see the relationship between levels of internal R&D, availability of public invention, and the frontier-follower difference for various industries.

For me, one of the most important conclusions of the paper was the impact of public science on follower firms, specifically on the problem of absorption. As the authors state,

“Our findings confirm that firms, especially those not on the technological frontier, appear to lack the absorptive capacity to use externally supplied ideas unless they are embodied in human capital or inventions. The limit on growth is not the creation of useful ideas but rather the rate at which those ideas can be embodied in human capital and inventions, and then allocated to firms to convert them into innovations.”

They go on to argue:

“The loss of absorptive capacity is partly related to the growing specialization and division of innovative labor in the U.S. economy. Not only do universities and public research institutes produce the bulk of scientific knowledge, but over the past three decades, publicly funded inventions and startups have grown in importance as sources of innovation. Concomitantly, many incumbent firms have substantially withdrawn from performing upstream scientific research. The withdrawal of many companies from upstream scientific research may have reduced their absorptive capacity—their ability to understand and use scientific advances produced by public science. If so, the division of innovative labor between universities and firms, wherein the former produce knowledge and the latter apply the knowledge to invent, appears to work much better for frontier firms. Non-frontier firms instead require universities or startups to convert ideas into inventions.”

The study offer a lesson for policymakers, but which should be approached carefully. We need to look beyond the average findings of the study and specifically address the differences between frontier and follower firms. First, while on average increased production of public knowledge has no effect on the overall level of internal company research, it is important for frontier firms. One of our policy goals should be the promotion of frontier firms. This requires continued university R&D funding.

Second, while increased public invention results in overall lower levels of internal research, this points to the importance of technology transfer. The technology transfer process needs to understand the difference abilities of frontier and follower firms. Follower firms have a lower ability to absorb public knowledge and therefore rely on universities to undertake the translational research required to develop a commercializable product.

Third, while much emphasis is placed on universities’ research activities, development of human capital is critical. We should never forget (or minimize) the important role of universities as educational institutions and the contribution that activity makes to the economy.

Finally, our policy should be to increase the absorption capacity of the follower firms. As the paper notes, “productivity growth may have slowed down because the potential users—private corporations—lack the absorptive capacity to understand and use those ideas.” Helping all firms improve their ability to absorb public knowledge is a key to continued economic prosperity.