استفاده از ماشین یادگیری در تحقیقات حسابداری

Quite often, in order to derive meaningful insights, accounting researchers have to analyze large bodies of text. Usually, this is done manually by several human coders, which makes the process time consuming, expensive, and often neither replicable nor accurate. In an attempt to mitigate these problems, we perform a feasibility study investigating the applicability of computer-aided content analysis techniques onto the domain of accounting research. Krippendorff (1980) defines an algorithm’s reliability as its stability, reproducibility and accuracy. Since in computer-aided text classification, which is inherently objective and repeatable, the first two requirements, stability and reproducibility, are not an issue, this paper focuses exclusively on the third requirement, the algorithm’s accuracy. It is important to note that, although inaccurate classification results are completely worthless, it is surprising to see how few research papers actually mention the accuracy of the used classification methodology. After a survey of the available techniques, we perform an in depth analysis of the most promising one, LPU (Learning from Positive and Unlabelled), which turns out to have an F-value and accuracy of about 90%, which means that, given a random text, it has a 90% probability of classifying it correctly.

Quite often, in the hope of discovering new relations and insights, research involves analyzing and classifying large quantities of unstructured information, such as news items or annual reports. So, to accurately extract the relevant information from the entire, and possibly huge, set of available raw data is an important challenge. Since this can easily mean analyzing multiple thousands documents, manual analysis is often not feasible, unless off course, one limits himself to a very small subset of the total amount of available data. Therefore, researchers typically delegate this very time consuming task to a series of human coders who will look through the entire set of news items and perform the classification. There is however a problem. You know, either implicitly or explicitly, which characteristics will make a document interesting, or positive, for you. The difficulty is conveying these criteria to your coders. But often, you will classify those documents, at least partly, on intuition. When you attempt to explain these intuitive criteria to your coders, some nuances will be lost and their results will never completely match your intents. Or, put more scientifically, the accuracy with which they classified the documents the way you intended it, will almost never be 100%. This is only achievable if you do the job yourself. Furthermore, repeatability is a major concern as a different group of coders will almost certainly produce a different outcome. In an attempt to decrease the cost and further increase the speed of classification, recent research delegates this highly time consuming process to computers instead of human coders. Without actually being able to argument the process, humans are able to make fussy decisions, like for example, whether a news item is favourable or unfavourable. Classification algorithms basically try to autonomously discover the implicit, underlying decision structure used by their human counterpart. As such, one can say that these algorithms try to mimic human decision making behaviour. However, this introduces an important limitation. Most computer aided classifications are limited to the use of naïve, heuristic algorithms such as word counting and frequency analysis. The problem with these techniques is that they are not very accurate as they are not capable of handling the many syntactical and semantic nuances one typically finds in natural languages. And because of this, they suffer from a rather large number of false positives and false negatives (Nigam, Mccallum, Thrun, & Mitchell, 2000). A more modern, and better, approach is the use of machine learning techniques. In this case, a self-learning algorithm tries to discover the underlying characteristics on which a human classifier classifies documents as either relevant or not. In a second phase it utilized this newly acquired knowledge to autonomously classify the entire set of documents. In this contribution, we present a methodology for using these advanced machine learning techniques in accounting research. More specifically, we demonstrate that the use of a text classification algorithm will greatly improve the classification speed of unstructured information while maintaining a very high accuracy. The paper is structured as follows: Section 2 describes the major text classification methodologies; Section 3 describes a series of text classification application, both general and specific to accounting research. Next, Section 4 presents an overview of the most important text classification approaches. Section 5 maps these approaches to two candidate algorithms and selects the most appropriate one, in this case LPU (Learning from Positive and Unlabelled) which is discussed in detail in Section 6. Next, Section 7 describes the complete process from raw data to classified information. Section 8 presents three possible algorithms for automatically building test sets. Section 9 describes the software program written to support this process and Section 10 evaluates the proposed classification process. To wrap up, in Section 11 we summarize the complete process in four practical steps and finally present a real-life application in Section 12.

In this contribution we attempted to make a case for applying machine learning in accounting research. We gave a concise overview of some important text classification algorithms and selected the most applicable one, in this case LPU. Furthermore, a four stage text classification process, three semi-automated training set builders and three fully automated test set builders were proposed. Given that, using CAQDAS, coding is done on an objective and repeatable way, stability and reproducibility are no issues. The accuracy of LPU, the third requirement of reliable text classification, as identified by Krippendorff (1980), still needed to be thoroughly tested. An elaborate analysis showed that LPU achieves an F-value and accuracy of about 90% and as such, can be used to reliably and efficiently classify large numbers of unstructured documents. Although, in this study, we only considered classification based on the degree of favourableness of a news item, the proposed algorithm is adjustable to almost every kind of content analysis.