Information gathering through open sources is a crucial aspect of cybersecurity activities in today's digital landscape. This process relies on publicly available sources, such as social networks, websites, and blogs. It encompasses data mining, data gathering techniques, data extraction, and data analysis activities. Open source intelligence (OSINT) methods are now widely utilized across various domains. In the context of penetration testing activities, OSINT processes are typically executed manually and managed by human operators. The integration of OSINT with machine learning (ML) elements can be beneficial in various sectors, such as everyday users, offices, and industrial or corporate environments. Combining data collection processes with general security regulations can yield better results for both production and end users. Platforms like TensorFlow leverage artificial intelligence to analyze massive amounts of data, as seen in Gmail and Google Translate platforms. In addition, numerous approaches are employed in social networks for friend suggestions and video streaming platforms for content recommendations. Utilizing artificial intelligence in OSINT activities can enhance the efficiency and quality of the search process, and consequently, the overall results of cybersecurity activities, including penetration testing. AI can automate various processes, such as web crawling, data collection, and pattern analysis. This opens up a wide range of new possibilities for building cybersecurity activities and exploring different vectors for data collection and its application in security events or penetration testing. Incorporating artificial intelligence in OSINT activities can also strengthen assistant mechanisms like Amazon Echo or Siri, enabling them to collect more relevant and rich data about a subject based on specific requirements.
Information gathering through open sources is a critical aspect of cybersecurity activities in today's digital landscape. This process relies on publicly available sources, such as social networks, websites, and blogs. It involves data mining and gathering techniques, as well as data extraction and analysis activities. Open source intelligence (OSINT) methods are widely employed across various domains. In the context of penetration testing activities, these processes are typically executed manually and managed by human operators.
Transitioning from manual to automated processes in OSINT is essential, particularly when dealing with real-world operations. A comprehensive system should be developed to facilitate automated open
2023 Copyright for this paper by its authors. CEUR
ceur-ws.org source-based activities, in conjunction with training simulations for machine learning. The architecture of the machine learning approach is contingent upon specific requirements.
• Based on our research we can obtain the following requirements: • The data used in previous user activities; • Gathering information using web crawlers and / or scrapers; • Processing activities like pattern recognition and events detection; • Vision of the automated mechanisms in the processes; • Analytics based on matching the patterns and data visualization; • Automation of the responses, error messages;
Within the scope of our research, we employ an approach that focuses on the aforementioned aspects, which can be implemented using automated process mechanisms. Combining open source intelligence with machine learning elements can be advantageous in various fields, such as everyday users, offices, and industrial or corporate sectors. Integrating data collection processes with general security regulations can yield improved results for both production and end users.
We can highlight the following key fields from data gathering point of view: • • • • • • • • •
Law firms; IT security personnel; Special investigators; IT oriented corporations Financial and insurance sector; Ethical hackers; Black hat hackers;
Hacktivist groups; Open data: social media and sources:
Information about domains; Media lookup; Contact details; Files online;
Location information;
Automated data collection mechanisms are extensively employed across various sectors today. Even
technology giants like Google utilize AI in their search mechanisms to generate predictive search
results. Platforms such as TensorFlow leverage artificial intelligence to analyze vast amounts of data,
as seen in Gmail and Google Translate platforms. Additionally, numerous approaches are implemented
in social networks for friend suggestions and video streaming platforms for content recommendations.
In these instances, user activity and preference data are collected and processed by AI algorithms,
providing end users with improved recommendations and, consequently, enhancing the overall system's
efficiency. Figure 1 illustrates the intelligence cycle and its stages [
For better understanding of the process, we need to go through the entire intelligence process. It consist of the following steps:
1. Requirements identification: clarification of the requirements and search area. Choosing the right direction, working with the requirements is an important step to start the process correctly and not lose time to back to the beginning repeatedly.
2. Data gathering process: defining the vectors of the work and planning activities. On this stage should be defined the appropriate informational sources. Data collection process also can be performed in different manner of the activities, such active and passive data gathering.
3. User-friendly format generation: gathered information needs to be sorted and represented in understandable format. It can be special spreadsheet or graphical representation, or even database with prepared data.
4. Information extracting process: the collected data needs to be processed and exploitation activities should be performed based on the relevant information;
5. Distribution of the information: gathered information should be shared with the trusted parties to achieve more global results in the frame of the OSINT activities;
6. Intelligence and requirements: the final feedback should be check to meet the requirements. The accuracy of the entire process should be assessed.
Together with the possibilities of artificial intelligence today, there are different limitations when it goes to data gathering process. During such activities, we need to take into account the fact, that not everything is reachable using open sources and in some cases on the internet itself. During the building of AI-based OSINT working system, we need to consider the limitations and scenarios every time. As in cyber security and attacks almost everything in based on working environment. Together with the technical limitations, the ethical part is also should be considered.
Cyber intelligence process automation refers to the implementation of automated technologies and
tools to gather, analyze, and disseminate cyber threat intelligence [
Automation allows for the efficient and continuous collection of data from various sources, such as open-source intelligence, social media, dark web, internal logs, and threat feeds. This reduces the workload of analysts and ensures a more comprehensive and diverse set of data.
Automated tools can aggregate and normalize data from multiple sources, which improves the accuracy and relevance of the information. This helps organizations to gain a better understanding of the overall threat landscape.
Automated analysis of the collected data can reveal patterns, trends, and anomalies that may not be apparent to human analysts. Machine learning and artificial intelligence can be used to uncover hidden relationships, predict potential threats, and assess the likelihood of specific attacks.
Automation can help prioritize threats based on the potential impact on the organization and the level of risk involved. This enables security teams to focus on the most pressing issues and allocate resources accordingly.
Automated systems can recommend appropriate countermeasures and response strategies based on the analyzed data. In some cases, they can even implement these measures, such as patching vulnerabilities, blocking malicious IP addresses, or updating firewall rules.
Automation enables continuous monitoring of the threat landscape, allowing organizations to adapt their security strategies as new threats emerge. This ensures a proactive and agile approach to cybersecurity.
Automated tools can disseminate relevant and actionable intelligence to stakeholders throughout the organization, fostering a more informed and collaborative security environment.
Automation reduces the likelihood of human errors in the cyber intelligence process, such as missed threats, data entry mistakes, or incorrect analysis.
By automating repetitive tasks, organizations can reduce the need for additional staff, increase the efficiency of existing staff, and ultimately reduce costs related to cybersecurity.
Cyber intelligence process automation can be easily scaled to meet the changing needs and growth of an organization. As more data and resources become available, automated systems can adapt and expand accordingly.
From the points above we can say that automating the cyber intelligence process offers numerous
advantages, such as improved efficiency, accuracy, and scalability. It enables organizations to
proactively address potential threats, reduce human error, and save costs [
Use of artificial intelligence in OSINT activities can increase the efficiency and quality of the search process and consequentially the results of the entire cyber security activity, for example penetration testing. Using an artificial intelligence a lot of processes like web crawling, collection of the data, analysis of the patterns can be automate. This fact gives a wide range of new possibilities in building of cyber security activities and different vectors for data collecting and it’s usage for security events or during penetration testing. As artificial intelligence algorithms learn based on the previews experience, it might have an impact on the entire process. A good example would be the suggestions of the videos on modern video streaming services. Recommendation works based on different parameters, like userpreferred categories, music genre or timing. The process of the open source intelligence works as follows [5-7]: 1. Work with sources of information to get the relevant ones; 2. Data gathering process using appropriate informational sources; 3. Processing of the information based on the sources and search requirements; 4. Data collection and analysis based on multiple sources; 5. Generation of the results and reporting activities;
AI systems will be able to combine OSINT data with geospatial and temporal information to provide a more contextualized understanding of events and threats, improving situational awareness and decision-making.
Network and Relationship Analysis:
By leveraging machine learning and graph-based algorithms, AI can uncover hidden relationships, patterns, and networks within OSINT data, helping to identify key influencers, threat actors, or potential collaborators.
Automated Threat Assessment and Prioritization:
AI systems will automatically assess and prioritize threats based on factors such as impact, likelihood, and relevance to the organization, helping security teams focus their efforts on the most pressing issues.
Predictive Analytics:
Advanced AI models will be able to predict future events, trends, or threats based on historical data and current patterns, providing organizations with valuable foresight and strategic planning capabilities.
Customization and Personalization:
AI-powered OSINT platforms will offer tailored intelligence feeds based on an organization's specific interests, needs, and risk profile, ensuring that relevant and actionable information is readily available.
Improved Collaboration and Knowledge Sharing:
AI-driven OSINT tools will facilitate better collaboration and knowledge sharing among analysts, organizations, and other stakeholders, resulting in a more informed and unified approach to security and intelligence.
In conclusion, the future of AI use in OSINT will likely involve a more advanced, integrated, and efficient approach to intelligence gathering and analysis. This will empower organizations to better understand and navigate the complex and ever-evolving threat landscape, making more informed decisions and taking proactive measures to ensure their security [8,9].
A practical example of AI use in OSINT is sentiment analysis for tracking public opinion on a specific topic, such as political events or product launches. Sentiment analysis is a technique that uses NLP and machine learning to identify and classify the sentiment of text data into positive, negative, or neutral categories. One common mathematical model used in sentiment analysis is the Naïve Bayes classifier.
We can describe the example of how the Naïve Bayes classifier can be used for sentiment analysis in OSINT:
Gather text data from public sources such as social media, news articles, and forums related to the topic of interest.
Clean and preprocess the data by removing irrelevant information (e.g., URLs, special characters), converting text to lowercase, and tokenizing words.
Split the dataset into a training set and a testing set. Manually label a portion of the training set with the sentiment categories (positive, negative, or neutral). Extract features from the text, such as word frequencies or n-grams, to create a feature matrix.
The Naïve Bayes classifier is a probabilistic model that calculates the probability of a given sentiment category (e.g., positive) based on the features extracted from the text. The model is based on Bayes' theorem, which is formulated as: ( _ | ) = ( | _ ) ∗ ( _ ) / ( ) (1) • • • •
P(C_k|X) is the posterior probability of class (sentiment category) C_k given the features X. P(X|C_k) is the likelihood of the features X given the class C_k.
P(C_k) is the prior probability of class C_k.
P(X) is the probability of the features X.
The Naïve Bayes classifier makes a "naïve" assumption that the features are conditionally independent given the class, which simplifies the likelihood calculation. For text data, the likelihood is typically calculated using the bag-of-words representation: ( | _ ) = ( _1| _ ) ∗ ( _2| _ ) ∗ . . .∗ ( _ | _ ) (2)
Using the labeled training data, calculate the likelihoods and prior probabilities for each sentiment category. This will form the basis for the Naïve Bayes classifier.
Apply the trained Naïve Bayes classifier to the testing set to predict the sentiment of the unlabeled text data. Choose the sentiment category with the highest posterior probability as the final prediction.
Evaluate the performance of the classifier by comparing its predictions to the actual sentiment labels. Analyze the results to gain insights into public opinion on the topic of interest.
In this example, the Naïve Bayes classifier is used as the mathematical model for sentiment analysis in OSINT. This practical application demonstrates how AI can be utilized to gather and analyze public opinion data from open sources, providing valuable insights for decision-making and strategic planning [10-12].
For textual data processing we can use Python library for processing textual data - TextBlob. It provides a simple API for common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more. TextBlob is built on top of the NLTK (Natural Language Toolkit) library, making it an easy-to-use interface for working with textual data.
TextBlob's sentiment analysis functionality is based on a pre-trained Naïve Bayes classifier. This classifier has been trained on a dataset of movie reviews, with the goal of determining whether the reviews are positive or negative. The sentiment analysis in TextBlob returns two values: polarity and subjectivity. Polarity is a float between -1 and 1, representing the sentiment's negativity or positivity, while subjectivity is a float between 0 and 1, representing the level of subjectivity or objectivity of the text.
The following pseudocode represents the Python script for gathering and analyzing public opinion data from Twitter using TextBlob's sentiment analysis:
This pseudocode outlines the main steps for fetching tweets based on a search query and analyzing their sentiment using TextBlob's sentiment analysis feature. The results are then displayed as percentages of positive, neutral, and negative sentiments [13,14].
Enhancing cyber intelligence capabilities through process automation has emerged as a critical aspect of contemporary cybersecurity strategies. While it offers numerous advantages in terms of scalability, speed, and efficiency, there are several potential difficulties and challenges associated with this endeavor. This paper aims to discuss these challenges, which include:
Complexity of cyber threats: The constantly evolving nature of cyber threats, along with their increasing complexity, poses a challenge to the effectiveness of process automation. Developing automated systems that can keep pace with the rapid advancements in threat techniques necessitates continuous adaptation and updates.
Integration challenges: A seamless integration of process automation tools with existing cyber intelligence systems is crucial for optimal performance. This may prove to be difficult, given the wide array of systems, platforms, and applications used by organizations. The integration process may be time-consuming and costly, further complicating the adoption of process automation.
False positives and negatives: The automated systems may generate false positives and false negatives, potentially leading to missed threats or unnecessary alerts. This challenge necessitates the development of sophisticated algorithms that can accurately identify true threats while minimizing false detections.
Human factor: The reliance on automation may lead to a reduction in human involvement, potentially undermining the role of human expertise in cyber intelligence. A balance between automated and human-driven processes is essential to ensure that valuable insights from human analysts are not lost.
Legal and ethical considerations: As process automation tools become more sophisticated, the potential for misuse and abuse increases. Ensuring that these tools are deployed ethically and in accordance with relevant laws and regulations is a critical challenge that needs to be addressed.
Data privacy and security: Process automation tools may require access to vast amounts of sensitive data, which raises concerns about data privacy and security. Ensuring that the data used by these tools is stored and processed securely is a vital challenge.
Skills shortage: Implementing process automation in cyber intelligence requires a workforce skilled in both cybersecurity and automation technologies. The current shortage of such skilled professionals may hinder the widespread adoption of process automation.
Cost considerations: The development, implementation, and maintenance of process automation tools can be costly. Organizations must weigh the benefits of automation against the financial investment required to implement and maintain these systems.
In conclusion, the future of AI in the OSINT sphere appears promising and poised for significant advancements. As AI technologies, such as natural language processing, machine learning, and computer vision, continue to improve, their applications in OSINT will become more sophisticated, accurate, and efficient. The integration of AI in OSINT will lead to the following key developments:
Enhanced data collection and analysis: AI-powered tools will be able to gather and analyze data from a broader range of sources, such as social media, news outlets, blogs, forums, and the deep/dark web. This will provide a more comprehensive and diverse dataset for organizations to analyze and respond to.
Real-time monitoring and response: AI-driven OSINT platforms will enable real-time analysis of data, allowing organizations to identify emerging events, trends, and threats rapidly. This will facilitate more proactive responses to potential issues and improve overall situational awareness.
Improved accuracy and efficiency: AI algorithms will be able to identify patterns, trends, and anomalies more accurately, reducing false positives and false negatives in the analysis process. Automation will also increase efficiency, allowing organizations to process large amounts of data more quickly and effectively.
Predictive analytics: Advanced AI models will enable organizations to predict future events, trends, or threats based on historical data and current patterns. This will provide valuable foresight and support strategic planning and decision-making.
Customization and personalization: AI-powered OSINT platforms will offer tailored intelligence feeds based on an organization's specific interests, needs, and risk profile. This will ensure that relevant and actionable information is readily available to stakeholders.
Enhanced collaboration and knowledge sharing: AI-driven OSINT tools will foster better collaboration and knowledge sharing among analysts, organizations, and other stakeholders, resulting in a more informed and unified approach to security and intelligence.
The adoption of AI in the OSINT sphere will continue to grow as technologies evolve and organizations recognize the value of AI-driven intelligence gathering and analysis. The future of AI in OSINT will be characterized by increased efficiency, accuracy, and effectiveness in processing and understanding vast amounts of publicly available information, ultimately leading to better-informed decision-making and more robust security strategies.
This work was supported by Shota Rustaveli National Foundation of Georgia (SRNSFG)[NFR-2214060].