Specialization is a hallmark of capitalist modernity. The division of labor into specialisms has propelled efÏciency and productivity in economic, administrative, and scientific contexts [10]. Politics is no exception. In the past decades, decision-making processes have been delegated to specialist experts, and politicians increasingly operate as specialists [37]. In the context of highly technical policy challenges, they can no longer afford to be generalists. While such specialization can enhance depth of knowledge and decision-making efÏciency, it also harbors intrinsic risks within democratic politics, as thinkers from John Dewey to Jürgen Habermas have pointed out [9,13]. The worldview of specialists and politicians may differ substantially from that of citizens. In fact, many scholars consider recent populist upsurges a response to a growing dominance of a class of specialized experts [30]. This signals a more fundamental tension: the process of specialization is driven by a search for efÏciency and productivity. Specialists value skill and "truth", whereas democratic politics, revolves around conflicting viewpoints, popular sovereignty, and regular debate. As an "empty space", it defies desiderata of efÏciency and truth [18,35]. Specialization and democracy, in other words, are in tension [36].
The specific risk posed to democracy by specialization can be called epistemic capture [39]. Such capture manifests when an individual's or group's specialized expertise in a specific domain results in a constrained perspective that dominates their analytical and decision-making processes and is taken by non-specialists as "truth" [12]. This narrowed focus can act as an intellectual filter, obscuring the broader context and impeding deliberation over diverse insights and alternative approaches. Consequently, this expertise-induced myopia can significantly hinder comprehensive problem-solving and balanced policy formulation [5]. The democratic debate is thus "captured" by specialists.
These captive effects of specialization also matter to the historical study of ideas. Intellectual historians commonly focus on the eighteenth and nineteenth centuries as eras of intellectual creativity, of nascent ideologies and emergent public spheres. The twentieth century, as some historians observe, appears markedly different. As the previously fertile soil for intellectual and ideological evolution, political debate became arid through specialization and "scientification" [11]. Historians are only at the start of understanding these structural changes in twentieth-century political debate. In this challenge, they face the enormous scale and complexity of twentieth-century intellectual production and circulation, one that is hard to grasp with traditional "close reading" methods. It is hard to imagine how a survey of lexicons or even exploratory keyword analysis would probe into a structural transformations such as specialization. Nevertheless, understanding specialization through the lens of language and concepts appears a crucial complement to prevailing institutional approaches to the emergence of specialists. Politicial scientist have closely studied specialist bodies such as parliamentary committees. Yet, the "informal" and "epistemic" aspects-crucial for understanding the impact of specialization on political debate-are hard to grasp through the lens of so-called "distributive" approaches, that consider specialization primarily as politics by other means [31]. At the same time, political scientists have long recognized the importance of specialists in interpreting reality for policy-makers at large [7]. Specialist communities have come to dominate policy-making in their area of expertise. In the Dutch case, the "Green Front" (a community of agricultural specialists) is a known example of such a political and epistemic hegemony [16]. These examples invite us to study specialization beyond procedural reform and inquire into the epistemic capture of policy areas by specialist perspectives.
Historians have long endeavored to reconstruct the web of ideas, worldviews, and cognitive constructs that have shaped human culture throughout history. Traditional methods, such as close reading and contextualization of texts drawn from archival research, have provided invaluable insights. However, these methods have always struggled with the scale of the his-torical record. Especially in democratic societies, where ideas develop through public debate, it is difÏcult for historians to grasp their complex circulation, production, and reception. In recent years, computational analysis has opened up new avenues for historians studying ideas and worldviews at scale. By applying algorithms and statistical techniques to large corpora of historical texts, researchers can discern patterns and trends that may be invisible to the human eye, using language-use as a reflection of collectively held ideas [6,20]. These "distant reading" methods-computational approaches to analyzing large volumes of text-often rely on keywords and their distribution over various contexts [24]. While these methods offer new insights, they also have limitations. Distant reading tends to revolve around singular keywords and ideas, with historians taking the former as an index to the latter. These words are usually explored through frequencies, collocations, and distributional vectors [34]. While this mode of lexical exploration boosts the efÏciency of the research process, its impact on the way historians study and assess the history of ideas is limited.
Recently, scholars have begun employing more sophisticated methods to transcend the study of individual concepts and ideas. By prioritizing theory-driven modeling over lexical exploration, they have tackled questions and concepts in the history of ideas that were long regarded too complex or big to empirically study. For instance, Ryan Heuser tests several long-standing hypotheses regarding general conceptual transformations during the "Sattelzeit" (saddle time), a transitional era between early modern and modern history [14]. Recent work by Vicinanza et al. [38] shows how ideas tend to emerge in peripheral spaces, illustrating how various studies have recently taken information-theoretic measures to quantify the novelty and resonance of intellectual developments. These approaches differ from "distant reading" by focusing on structures and processes [28]. Instead of merely analyzing word distributions, they utilize lowdimensional numerical representations of texts to explain complex dynamics. This allows these scholars to make more novel and ambitious claims about the production, circulation, and contestation of ideas in the past. In this paper, we build on this new direction in computational intellectual history, by looking at specialization as an intellectual and epistemic process.
The Dutch Lower House exemplifies the trend toward specialization in the twentieth century. Gradually, members of parliament came to believe that specialist deliberation was more efÏcient and productive. This marked a significant departure from the nineteenth century, when specialism was often equated with narrow-mindedness [15]. During the interwar period, however, politicians observed a widening gap between the expert bureaucracies of the expanding state and parliamentary generalists. Consequently, parliamentary specialists emerged in specific domains such as foreign policy and agriculture. After the Second World War, this trend gained an institutional footing. Similar to other Western European contexts, so-called permanent committees were established in which specialists could wage a more technical debate [32,2]. The Dutch multiparty system and the desire of the main parties to depoliticize sensitive issues only stimulated such specialization [19]. Previously, parliament had met in randomly allocated subgroups, known as "departments", reflecting the generalist assumptions of the time. The new Rules of Order drafted in 1966, a decade after the experimental introduction of permanent committees, abolished the departments [15]. Since then the Dutch Lower House has grown into a legislative institution that is marked by high levels of specialization [1].
Given this historical context and the broader implications of specialization in democratic institutions, our study aims to address the following research question:
1. How did specialization in the Dutch Lower House evolve between 1945 and 1995 2. To what extent did it lead to epistemic capture in parliamentary debates?
To answer these questions, we employ a novel computational approach that combines topic modeling, network analysis, and dynamic community detection. This method allows us to trace the evolution of specialized knowledge domains in the Dutch Lower House over five decades, offering insights into the process of specialization and its effects on parliamentary discourse.
By applying a dynamic network analysis to links between topics in debates, we aim to contribute to a deeper understanding of how ideas and cognitive constructs shape political realities. This study also seeks to shed light on the tensions between specialization, pluralism, and contingency in democratic decision-making, offering a new perspective on the challenges facing modern democracies.
This study utilizes the digitized parliamentary proceedings of the Dutch Lower House from 1946 to 1995. The proceedings consist of speeches held in the Lower House that are transcribed, edited, and published. In recent years, the proceedings have been digitized and linked with metadata on speakers, parties, and dates. The quality of the Optical Character Recognition is known to improve considerably for postwar proceedings [17]. To prepare the data for analysis, we conducted several preprocessing steps. First, we removed stop words to eliminate common but uninformative words. Next, we filtered out speeches shorter than ten words, as these are unlikely to contain substantial content. We also limited our analysis to nouns, verbs, adjectives, and adverbs, as these parts of speech are most informative for semantic analysis.
In addition to text preprocessing, we also excluded several types of speeches. First, we restrict our analysis to plenary debates. With the omission of committee meetings (also present in our data), we prevent our models from reflecting merely committee language, instead forcing them to measure the degree of specialization in plenary debates. Second, we excluded speeches by the House chair, which often contain procedural and repetitive language not directly relevant to substantive debates. Including these speeches could disproportionately affect the topic model and subsequent linkage scores, introducing noise and potentially obscuring meaningful patterns in the data. Thus, excluding the chair's speeches serves both conceptual and methodological purposes, ensuring our analysis focuses on the most informative and relevant content.
Our data comprises 52,396,073 tokens and 495,053 types. The data size varies unevenly over the period; after 1967, the size of the parliamentary text gradually expands. Whereas an average year in the 1950s contains 1.25% of the tokens, this number rises to 4% in 1979.
Our methodological approach consists of four main steps: 1) pre-processing of parliamentary proceedings, 2) topic modeling to identify thematic content, 3) network analysis to map relationships between topics, and 4) dynamic community detection to identify clusters of specialized domains (see Figure 1). This multi-step process allows us to trace the evolution of specialized knowledge domains in the Dutch Lower House over five decades.
To examine specialization as an epistemic phenomenon, we analyzed the dynamic structure of the content of parliamentary debates using topic modeling techniques. We employed Latent Dirichlet Allocation (LDA) using the Mallet toolkit to generate low-dimensional representations of the semantic content of the speeches [3,21]. 1 We chose LDA over more recent techniques such as top2vec and BERTopic because the vector-based similarity scores of the latter yielded relatively unclear linkage networks. Specifically, we configured the LDA model with 250 topics to capture a wide range of potential themes in the parliamentary debates. Each speech was segmented into 50-word chunks to account for the declining speech lengths over the study period. After training, we averaged the topic distributions for each member within a single session to facilitate subsequent network analysis. The trained topic model captured both thematic categories aligned with policy areas and procedural and rhetorical categories, such as filing motions or citing newspapers. We chose 250 initial topics to capture a wide range of potential themes in the parliamentary debates. (Appendix Topic Models). After manual inspection, we identified and removed 116 topics that primarily represented procedural or rhetorical aspects of the debates rather than substantive policy areas based on domain knowledge. This reduction allows us to focus our analysis on the thematic content most relevant to our research questions about specialization in policy domains. We then (re)normalized the remaining topic distributions to ensure comparability across sessions. We consider this filtering justified because it forces our network-based approach to focus on connections between themes. Also, the metrics we use in the subsequent analysis appear robust to this filtering.
To quantify the connectedness of parliamentary speech topics, we leverage topic linkage scores based on Pointwise Mutual Information (PMI), building on the work by Perry and DeDeo [27] (See Appendix Linkage function). PMI measures the degree of co-occurrence of topics in a period beyond random chance, with higher scores indicating stronger associations between topics. We calculate PMI scores between pairs of topics and use this to create network structures of topics that have a PMI higher than 0 (See Figure 1 as an example). Network metrics enable us to investigate the evolving architecture of linkages over time. To address PMI's sensitivity to rare topics, we employ a smoothing function based on joint frequency [26]. This approach prevents low-frequency topics from becoming highly connected and central in the networks, resulting in more accurate representations of topical connections. We construct PMI-based networks on time periods of 6 months, acknowledging changes across parliamentary years and differences between budgetary debates (that commonly dominate August-December) and other types of debates between January and July [27].
After generating the networks, we applied the Louvain community detection algorithm to identify clusters of related topics [4]. This algorithm optimizes modularity, a measure of the density of links inside communities compared to links between communities, potentially reflecting the emergence of specialized epistemic domains within parliamentary debates. We tested multiple resolution parameters, which control the size and number of detected communities, and found that a resolution of 3 provided the most meaningful and interpretable results based on our domain knowledge of Dutch parliamentary history. This choice was robust across different time periods and yielded communities that align well with known policy domains and historical developments in Dutch politics.
To analyze the properties and evolution of these topic networks and communities, we rely on several key metrics.
1. Network Metrics (Modularity, Network Density, Node Degree). Modularity assesses the degree of community structure within a network [25]. High modularity values indicate that the network has dense connections between nodes within communities and sparse connections between nodes in different communities, suggesting a strong partition into specialized domains. Network Density quantifies the overall "connectedness" of a network, calculated as the ratio of the actual number of edges to the maximum possible number of edges [23]. Node Degree, conversely, measures the "connectedness" of a sin-gle node by counting the number of edges attached. These metrics help identify highly connected topics that may play central roles in structuring parliamentary debates. 2. Clustering Metrics (Normalized Mutual Information (NMI)). To evaluate the similarity between clusterings in different periods we use Normalized Mutual Information. This metric calculates the mutual information between two distributions of nodes over communities and provides a normalized score between 0 and 1 that quantifies the degree of mutual information [22]. 3. Set Similarity Metrics (Overlap CoefÏcient). To compare node neighborhoods and track the evolution of communities over time, we employ the overlap coefÏcient to calculate the similarity (node neighborhood similarity) between two sets [33]. This measure quantifies the size of the intersection between two groups relative to the size of the smaller group, highlighting the extent to which the smaller group is contained within the larger one. This makes the overlap coefÏcient suitable for quantifying "epistemic capture", more so than the common Jaccard Similarity. Because the latter compares the size of the set intersection to the size of the set union, it neglects the possibility that a set is fully present in another. The overlap coefÏcient is particularly useful for detecting significant overlap when one group is substantially smaller than the other, a common scenario in longitudinal studies where group sizes fluctuate. By calculating the overlap coefÏcient between neighbors in period 𝑃 and neighbors in period 𝑃 −1, and aggregating these stability scores, a coarse picture of stability can be mapped. 4. Community Paths. To capture the formation and change of communities over time, we employed the CDlib (Community Discovery Library) to perform temporal community analysis. Using the Temporal Network Clustering algorithm, we match communities in different time periods [29]. Specifically, we use the aforementioned Overlap CoefÏcient to calculate the similarity of communities based on their topic composition. The algorithm links communities across adjacent time steps based on node overlap, providing insights into the stability and dynamics of specialized knowledge domains in the Dutch Lower House. This analysis reveals persistent topic clusters and volatile areas, potentially reflecting changes in political priorities, the emergence of new issues, or the restructuring of existing policy domains.
Our analysis starts with an attempt to gain a macro-level perspective on the extent of specialization in the Lower House. We expect a gradual increase in modularity-the extent to which networks exhibit community structure-to reflect this specialist compartmentalization of debates. Indeed, temporal networks show a notable increase in modularity between 1950 and 1975, followed by stabilization that persists until the 1990s. The initial increase in modularity contrasts with a declining network density (the overall "connectedness" of a network). The divergence between modularity and density trends (Figure 2a) suggests that while topics are becoming more specialized (increasing modularity), they are also becoming more isolated from other topics (decreasing density). Modularity and density thus point at increased community structure. However, specialization would not only entail community structure, but also a crystallization or stabilization of communities. To assess the extent of stabilization, we measure stability using Normalized Mutual Information (NMI), which quantifies the similarity between clusterings from different periods, with higher NMI values indicating greater stability. From 1946 to 1994, we calculate NMI scores for each pair of consecutive six-month periods. The upward trend (Figure 2b) and considerable increase (from about .5 to .6) in these scores shows that topic clusters-specialized communities-are becoming increasingly stable over time. This indicates that specialized topics in parliamentary debates are not only forming but also becoming more entrenched, reflecting a growing stability in how topics are grouped and discussed.
The extent and stability of community structure as measured above suggest the gradual advance of specialization in the Lower House. The signals roughly correspond to the known solidification of the committee system in the 1950s and 1960s. In this period, these institutions became the focal actors in parliament. Our analysis shows, however, that this institutional advance also left an imprint on the way politicians thought and talked: the breadth of debate declined, while its depth increased.
The macro-level analysis confirms the gradual specialization of the Lower House, especially in the decades between 1950 and 1975. However, they say little about driving factors and local dynamics that are clearly present in the previous figures. To investigate specialization beyond linear increase, we take a meso-level look at the specific nodes in the networks. We begin with studying the changing neighbors of nodes. Overall, node neighborhood similarity increases over time, confirming the macro-level trend of specialization (see Figure 3). The overlap coefficient, which measures the similarity of a node's neighbors between periods, shows a slight upward trend after a decline around 1965. This suggests that topics tend to maintain their associations over time, reinforcing the stability of specialized communities. Cabinet changes appear to influence neighborhood stability, with sharp declines followed by gradual recovery during a cabinet's tenure. This pattern highlights the interplay between political change and epistemic shifts in parliamentary discourse, suggesting that-amid general institutional change-political changes also manifest as epistemic changes. The reshufÒe in committee membership at the start of a cabinet period translates into a epistemic "reset" followed by gradual crystallizations of networks. If specialist communities would persist without any sensitivity to political change, the variation would not be visible in the figure.
The interplay of (institutional) macro-level specialization and (political) meso-level dynamics can be illustrated by looking at individual topic neighborhood change (see the three figures at the top in Figure 3). First, to the right, the topic of "broadcasters" shows low overall neighborhood stability. This signifies the role of public broadcasting as a volatile topic, often involved in disparate political conflicts. By contrast, the topic of international conflict shows high overall stability. Foreign affairs was the domain of specialists who forced the topic into a fixed epistemic mold. Neighborhood stability did fluctuate, likely due to different international conflicts, but overall, stability remained high. Lastly, the topic of inflation shows another motif. Stability declines from 1955 onward, as uncertainty grew about surging inflation rates and the best way of combating them. After 1966, stability increases gradually until 1982, reflecting a consolidating perspective on inflation. The breakthrough of this (neoliberal) interpretation of inflation, and the subsequently drawn links between inflation and a plethora of other issues, demonstrates as a sharp drop in 1982. As such, these three topics show how, despite the overall rise of specialization, events and conflicts produced variation in the stability of a topic's neighborhood. In other words, the epistemic capture induced by specialization is dependent on political events and ideological change. Topics could be epistemically captured, but also break free from solidified structures through political contestation. This means that macro-level epistemic capture was no teleological force, but rather a gradual transformation that allowed substantial variation driven by distinct political factors.
Specialization and its captive consequences can be further contextualized and understood by looking at specialist communities. By connecting communities in different time periods, "paths" of specialist communities can be traced. These paths consist of communities, that, in turn, consists of topics. In Appendix Community Paths, we show how the paths begin, dissolve, or persist. We highlighted specialist community paths that relate to foreign policy, education, and agriculture: the most specialized areas. 2Paths of specialist communities shed yet another light on postwar specialization and epistemic capture. They offer the possibility to study specialization not only as intensification-in the form of an increasingly focused debate following procedures and party politics, but also as the epistemic capture of new topics and communities, integrating them in existing specialist chains. The overview of the chains (Appendix Community Paths) points to moments where multiple chains were born, and where the majority of communities fell within a chain. Three moments, in particular, stand out. First, the late 1960s appear as a moment where several chains emerged. New issues, such as public housing, municipal reorganization, wage policy, and social benefits are captured in communities that persist into the early 1970s. This aligns with historiographical depictions of the era as one of renewed labor militancy. Second, the "captivity" of topics peaks again around 1980. The zenith of new polarization in the Lower House, the turn of the decade forms the stage for emergent specialist chains, around constitutional reform, nuclear waste, and business subsidies. The third and final peak occurs around 1990. The salience of social benefits, organized consultation, and public transport reflects in new chains, reflecting the political agenda.
Chains of connected communities signify relatively stable specialisms. However, the question remains how these epistemic structures bear a footing in the daily work of individual politicians. To analyze this dimension, we turn to the relationship between politicians and topics. Politician-topic interaction reveals the connection between epistemic structures and political reality. By calculating the conditional probability 𝑃(𝑡𝑜𝑝𝑖𝑐, 𝑚𝑒𝑚𝑏𝑒𝑟) in each period, we identify specialists connected to chained specialisms. Specifically, we z-score normalize the conditional probabilities and assign politicians to communities based on topics where the probability is larger than 1.
The connections between politicians and specialist communities can be used to validate the linkage networks as grounded in parliamentary practice. Filtering communities with strong connections to specific members points to, for example, financial specialists such as Anton Lucas or agricultural specialists such as Jur Mellema. Generally, the predictability of topicsexpressed through entropy-in a specialist chain closely correlates with the predictability of associated politicians. Figure 4a shows this correlation: paths with predictable (or stable) topics also have predictable members. Paths with unpredictable topics (corresponding to unstable specialisms) have unpredictable members. This shows that the epistemic dimension measured through topic linkage corresponds to the actorial dimension of parliamentary work.
Path-politician connections also demonstrate how the politicians that link to specialist paths also tend to stabilize. Figure 4b shows the overlap between the "members" of a specialist community in a period, and those in the "same" community in the previous period. It shows that the longer a specialist community exists, the more stable its members are. This shows that epistemic capture is not only a matter of increasingly crystallized linkages on the level of topics, but also reflects in the crystallization of members around epistemic structures.
Micro-level dynamics, such as the persistence or breakdown of specialist topic paths, or the stabilization of specialists around particular communities points to the complexities of specialization at the micro-level. Epistemic capture is visible in the recurrence and persistence of communities in time and the sudden proliferation of specialist paths at specific points in twentieth-century debates. It also manifests as the stabilization of individual politicians as specialists in specific areas. As such our network-based approach also shows the contingencies of epistemic capture.
Our analysis of the Dutch Lower House proceedings from 1945 to 1995 reveals the complex dynamics between epistemic capture and specialization in parliamentary debate. By combining analytical techniques, we demonstrate how specialization and epistemic capture unfold at the macro, meso, and micro levels. These levels map surprisingly well onto institutional, political, and epistemic factors, respectively, revealing the multifaceted nature of these processes. At the macro-level, we find a declining network density, a growing modularity, and a similarly increasing clustering stability. These trends are particularly visible in the 1950s and 1960s, which suggests topic specialization to develop in tandem with institutional specialization the form of permanent committees. However, given our focus on plenary debates, we show that this is not just a matter of procedural reform: specialisms become visible as epistemic structures in parliamentary language.
At the meso-level, we find an interplay between political change and epistemic shifts, as cabinet changes appear to act as moments of reconfiguration for specialist communities. The sharp declines in neighborhood stability following cabinet changes, followed by gradual recovery during a cabinet's tenure, suggest that political transitions can disrupt the continuity of specialist communities. Yet, the fact that these patterns are visible in the topic linkage networks, which are not explicitly informed by procedural or institutional factors, underscores the epistemic dimensions of political change.
At the micro level, we identify distinct specialist chains that shed light on the dynamics of epistemic capture within the Dutch Lower House. The increasing share of communities belonging to these chains over time points to both an intensification of existing specializations and an extension of specialization to new topics. The close correlation between the predictability of topics and politicians within specialist chains validates the link between epistemic structures and individual actors, while the stabilization of politician membership in longer-lasting specialist communities suggests a reinforcing cycle of expertise and influence. This finding highlights how epistemic capture can become self-perpetuating, potentially limiting the diversity of perspectives in policy debates over time.
Our multi-level analysis offers contributions to three fields: political science, intellectual history, and computational humanities. First, we contribute to the understanding of how specialization and epistemic capture shape the dynamics of knowledge production and decisionmaking in democratic institutions. The multi-level approach highlights the complex interplay between institutional, political, and epistemic factors in driving these processes. We, thereby, make the (often theorized) tensions between pluralism, diversity, expertise, and efÏciency explicit and show how twentieth-century political debate is faced with a trade-off between breadth and depth. Further applications of our method could reveal the extent to which epis-temic capture has persisted in democratic institutions, manifested in other contexts, and faced a populist backlash in recent decades.
Second, these findings have consequences for the way historians approach the computational study of ideas and worldviews. Our analysis shows that the production, circulation, and contestation of ideas itself changes as a consequence of specialization. It matters if ideas are articulated in ideologically diverse and open debates, or in technical and specialist settings. To understand these environments, computational analysis can be used beyond the level of lexical exploration. We show that using a combination of established methodologies (topic modeling and network analysis) can yield fresh insights in the uncharted territories of intellectual history.
Third, we have used network analysis to shed light on complex processes such as specialization. As such, this paper intends to show that iterating between distant and close reading is not the only digital approach. Networks can elucidate regularities and contingencies at different levels. They do not prefigure a choice between "close" and "distant", but offer a versatile means of differentiating between factors and contexts. However, network analysis is by no means the only or best way to understand complex dynamics and layers of context and causality. In the future, we aim to explore the use of agent-based modeling (ABM) could provide deeper insights into the processes of epistemic capture and specialization. By incorporating agents representing politicians, policies, and institutional rules, ABM can offer a nuanced understanding of how individual actions and interactions lead to macro-level phenomena, thus enhancing our comprehension of the dynamics observed in this study.
Instead of using single (information-theoretical) metrics to measure specialization, we employ topic linkage networks to enable a multi-level analysis of specialization in parliamentary debates. Global network statistics reveal long-term developments, while similarities between communities, topics (nodes), and linkages (edges) point to local dynamics and contextual factors. We identify specialization and its effects on the macro, meso, and micro-level. Through specific metrics and signals at each level, we integrate different explanations, allowing us to differentiate between various causal forces and contextual factors that shape specialization.
The full code and data for replicating our analysis will be made available from the following Github repository upon the publication of the paper: anonymized url
The table in this appendix contains the labels and top terms for a rhetorical, procedural, and "thematic" topic.
Given a matrix 𝜃 ∈ ℝ 𝑁 ×𝐾 , where 𝑁 is the number of documents and 𝐾 is the number of topics, the function computes mutual information between topics.
• 𝜃 𝑖𝑗 : Document-topic mixture value for document 𝑖 and topic 𝑗.
1. Joint Probability Calculation:
𝑃(𝑖) = ∑ 𝑑 𝜃 𝑑𝑖 ∑ 𝑖 ∑ 𝑑 𝜃 𝑑𝑖
𝑅 𝑖𝑗 = log 2 ( 𝑃(𝑖, 𝑗) 𝑃(𝑖) ⋅ 𝑃(𝑗) )
weight(𝑖, 𝑗) = ( 𝑃(𝑖, 𝑗) 𝑃(𝑖, 𝑗) + 1
) ( min(𝑃(𝑖), 𝑃(𝑗)) min(𝑃(𝑖), 𝑃(𝑗)) + 1 )
SMI 𝑖𝑗 = weight(𝑖, 𝑗) × 𝑅 𝑖𝑗
Community paths are consecutive links (paths) between similar communities in topic linkage networks. Lines indicate connections between communities in different periods. Paths related to foreign policy (blue), education (red), and agriculture (yellow) are highlighted.
͓͖͐͘ ͔͐͘͏ ͔͐͑͘ ͔͔͐͘ ͔͖͐͘ ͕͐͘͏ ͕͐͑͘ ͕͔͐͘ ͕͖͐͘ ͖͐͘͏ ͖͐͑͘ ͖͔͐͘ ͖͖͐͘ ͐͗͘͏ ͐͗͑͘ ͔͐͗͘ ͖͐͗͘ ͐͘͘͏ ͐͑͘͘