Online webinar and Q&A-5 technical lessons learned from the interruption of AWS, Google and Microsoft (online webinar, December 9, 2021) Register now

Promote knowledge dissemination and innovation in professional software development

The continuous documentation method is a useful paradigm that helps ensure that high-quality documentation is created, maintained, and readily available. The code walkthrough leads the reader on a "walk"—visiting at least two sites in the code—describes the process and interactions, and usually contains code snippets.

Gwen Shapira and Vivek Sharma discussed some of the architectural highlights of building, developing, and extending the control plane for thousands of Kafka clusters, as well as some of the challenges they encountered.

In the podcast, Rosaria Silipo talked about emerging trends in deep learning, focusing on low-code visual programming to help data scientists apply deep learning techniques without writing solutions from scratch.

The amazing progress made in technology has led to blindly following technical requirements at the expense of social and human dimensions. Social science can help us create a working environment that makes people feel more at home and feel proud of their products. An organization designed using the theory of an open society technology system will be a more humane organization, and people will be more engaged.

This article discusses three kinds of biases that people may have when trying to quickly build application security, attitudes that may cost the organization in the future, show how to spot biases, and provide suggestions on how to deal with them.

Learn from practitioners who drive software innovation and change. Participate in person from April 4 to 6, 2022.

A monthly guide to all topics, techniques and techniques every professional needs to know. Free subscription.

InfoQ Homepage News What machine learning can do for safety

Machine learning can be applied to the security field in various ways, such as malware analysis, prediction, and security event clustering. It can also be used to detect previously unknown attacks that have not established a signature.

Wendy Edwards, a software developer interested in the intersection of cybersecurity and data science, talked about applying machine learning to security in the Diana Plan 2021.

Artificial Intelligence (AI) can be used to detect abnormalities by finding abnormal patterns. But as Edwards explained, being unusual does not necessarily mean malicious:

For example, maybe your web server is experiencing higher traffic than usual because of some trends on social media. You might be able to check things related to traffic to make this decision. For example, are there many HTTP requests that set the "user agent" to something that is not normally related to normal web browsing? Is there a large amount of unexplainable traffic originating from a single IP or IP range? A sequence of abnormal accesses to the endpoint may indicate that fuzzing has been performed.

Through artificial intelligence and machine learning, there are techniques that can process a large number of input variables and draw conclusions. Edwards gave an example of how forecasting allows you to use time series data to predict the future, and supports trends, seasons, and cycles:

This can be useful for measuring CPU utilization or total web server access. The system is likely to be busiest at certain times of the day. Perhaps the number of clicks on the new website is gradually rising. Statistical indicators are also useful, such as mean and standard deviation. This can help us determine how much "abnormal" activity from a single IP or IP range actually is.

Edwards shows how to use machine learning to cluster security events:

Clustering is a machine learning technique used to create groups of data points that are more similar than external points. A security incident is a series of incidents, usually the same group of incidents with the same root cause appear in multiple locations.

For example, a Trojan horse may attack multiple machines, but the root cause and remedy are the same.

Clusters can help Security Operations Center (SOC) analysts identify similar events that often require the same response. Edwards mentioned that this can save time by eliminating a lot of tedious work.

InfoQ interviewed Wendy Edwards on how machine learning can be applied to security.

InfoQ: What is the status of the application of artificial intelligence in IT security?

Wendy Edwards: It is steadily improving, although I think there is always a need for skilled practitioners; artificial intelligence and machine learning are unlikely to replace people. Artificial intelligence has developed significantly in the past 15 years, and due to the increasing complexity of computing, network security has become more challenging.

At this point, there has been extensive research and development related to the potential applications of artificial intelligence in network security, including intrusion detection, malware analysis, phishing detection, and finding robot accounts on social media. Natural language processing also played a role, most notably in spam detection and in identifying malicious code in obfuscated scripts.

See how many vendors tell you how their products use machine learning! However, there is currently no widely accepted best practice regarding artificial intelligence and cybersecurity.

InfoQ: You mentioned in your speech that anomaly-based detection may detect previously unknown attacks with no definite signature. How does this work?

Edwards: This involves determining what is normal and what is malicious. A signature is a set of rules related to known attacks, so there won't be any attacks that have never been seen before.

When we see something abnormal without a benign explanation, something may be wrong. For example, if some content on your website is popular on social media, you may see an increase in activity, which is okay. However, if you see a lot of activity that does not match normal user behavior, you may be under attack.

InfoQ: What AI tools are available and how do we use them?

Edwards: There are many established freely available tools; for example, Python has scikit-learn. Google and Facebook released Tensorflow and PyTorch libraries respectively.

Scikit-learn provides many useful tools, including regression, clustering, classification, etc.

Tensorflow and PyTorch support more complex tasks, such as deep learning. Generally speaking, PyTorch is considered easier for experienced Python programmers to use, while TensorFlow is considered more suitable for use in a production environment.

InfoQ: In terms of AI and IT security, what do you expect to bring in the future?

Edwards: I think opponents will also use artificial intelligence in their attacks. The Internet of Things (IoT) and other evolving technologies will create an increasingly larger attack surface, and attackers may use artificial intelligence to find ways to exploit this. According to the National Academy of Sciences report on the impact of artificial intelligence on network security, the use of artificial intelligence and machine learning to discover and weaponize new vulnerabilities is in the conceptualization and development stage in the United States, and it is also possible in China and Israel.

Adversarial machine learning refers to attempts to deceive machine learning algorithms. For example, a spammer may try to evade filtering by misspelling the "bad" word and including the "good" word that is not normally related to the filter. If the operational data is used to train future systems, attackers may try to contaminate the data.

An example is Microsoft's "Tay" robot. After being bombarded by racist and sexist messages from trolls, Tay started posting offensive content and was eventually shut down after about 16 hours.

Learn everything you need to save time and deliver higher quality software and applications through test automation.

A summary of last week's InfoQ content is released every Tuesday. Join a community of more than 250,000 senior developers. View example

You need to register for an InfoQ account or log in or log in to post comments. But there is still a lot to do after registration.

Take advantage of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

A summary of last week's InfoQ content is released every Tuesday. Join a community of more than 250,000 senior developers. View example

Discover emerging trends and practices from the most innovative software professionals in the world. QCon London is a conference for senior software engineers, architects and team leaders. In-depth discussions with world-class software leaders about the patterns, practices, and use cases used by the world's most innovative software professionals.

InfoQ.com and all contents are Copyright © 2006-2021 C4Media Inc. InfoQ.com is hosted at Contegix, which is the best ISP we have worked with. Privacy Statement, Terms and Conditions, Cookie Policy