Government agencies collect a lot of data, and have access to even more of it in their archives. The trick has always been trying to tap into that store of information to improve decision-making, which is a major focus in government these days. The President’s Management Agenda, for example, emphasizes the importance of data-driven decision-making to improve federal services.
The volume of data that most agencies are working with is such that humans can’t easily tap into it for help with that decision-making. And even if they can perform searches into that data, the process is slow. Plus, because humans can’t get an overview of the entire pool of data, many of the interrelationships, causalities and influences hidden within it are lost. As such, government agencies at all levels are increasingly tapping into artificial intelligence, machine learning, robotic process automation and other similar tools to help sort, classify and mine their data to produce actionable results.
For the most part, these programs have proven to be extremely successful for the agencies that deploy them. Artificial intelligence is very good at analyzing data, and tends to get even more accurate as more data is added to a system.
The future certainly looks bright for AI and related technologies within government. However, a new study conducted by the NCC Group, a global security consulting firm, cautions that there are quite a few hidden dangers when employing AI and machine learning that agencies should know about. NextGov talked with NCC Group Chief Scientist Chris Anley about the results of their study, the specific risks associated with AI and machine learning, and ways that agencies can protect their data and their users from exploits that target those systems.
NextGov: Can you first tell us a little bit about your background and the NCC Group?
Anley: I'm the Chief Scientist at NCC Group, which means I collaborate with colleagues on projects and conduct my own research. We look into attacks and defenses for IT systems, networks and computing devices of all kinds, and publish research in these areas. NCC Group is one of the largest and most respected security consultancies in the world, with over 2,000 employees, 35 offices around the world and 14,000 clients.
My own background is in IT security and software development. I co-founded a company, NGS Software, in 2001 which was bought by NCC Group and I've been associated with NCC ever since.
NextGov: And what led you to research the specific vulnerabilities associated with AI and machine learning systems?
Anley: We started to notice ML applications becoming much more prevalent around five years ago. They present a whole new set of security challenges, so we've been actively researching attacks and defenses since then. In terms of applications, ML used to be a fairly niche activity, but we are increasingly seeing it used for routine tasks like suggesting actions to users in web applications, handling customer support queries and so on.
And attackers are starting to exploit those situations.
NextGov: Are the kinds of attacks being made against AI and ML systems different from the typical kinds of attacks made against government agencies and their networks?
Anley: Yes, there are a range of new types of attacks that apply specifically to ML systems, which is what the bulk of our paper is about, although the traditional security issues like patching, credential management and application security issues leading to traditional data breaches all still apply.
NextGov: The paper you produced details dozens of real-world attacks and successful attack techniques made against AI and ML systems. I want to talk about those, but one of the most striking findings is that you said training ML systems with sensitive or secret information should be considered an especially dangerous practice. Can you explain why you made that statement?
Anley: ML systems perform better when trained on larger amounts of data, so it follows that if the training data is sensitive in some way—say, financial, medical or other types of personal data—then there's an increased potential for security and even regulatory issues. Curating training data can be difficult and time consuming even without the security challenges of privacy, access control and complex configurations.
NextGov: And the kinds of attacks you demonstrated were sometimes able to gather the information that was used to train the system, so loading it up with sensitive information makes the situation worse. Can you talk about some of the other kinds of attacks that are made against AI systems?
Anley: Privacy attacks allow criminals to retrieve fragments of training data from the trained model by submitting inputs in the “normal” way; if the model was trained on sensitive data, some portion of this sensitive training data can be retrieved.
Poisoning attacks allow an attacker to modify the behavior of a model during training, to change the decisions it makes. For instance, if the model was involved in financial decisions, this might allow the attacker a financial advantage, or if the model was making security decisions—perhaps a facial recognition system—then it might allow the attacker to bypass the security check. In some cases the attacker can insert malicious code into the model itself, which could then do anything the attacker wants—install ransomware, mine cryptocurrencies or provide backdoor access.
Adversarial perturbation attacks allow an attacker to change the decision a system makes by making small, carefully chosen changes to inputs. Image classification systems are now a matter of life and death, so it's important to ensure they're robust. For example, issues have been found that relate to road signs and other physical objects in the real world. There are also examples of a 3D printed turtle that is mistaken for a gun or a 3D printed baseball that's mistaken for a cup of coffee.
NextGov: Your report is a fascinating read about how those kinds of attacks can happen. But what about defenses? Is there anything that agencies can do to help protect their AI systems?
Anley: For each attack, we’ve suggested mitigations in the taxonomy section of the paper. Additionally, each of the attacks are referenced in the categorized references section where the academic papers relating to that attack are listed.
There are no silver bullets to defend against these attacks, but the traditional precautions like vigilance, authentication, access controls, rate limiting, careful handling of sensitive data and periodical review by external security professionals are, as always, the best ways to avoid unpleasant surprises.
NextGov: Thanks for your time today. Because defenses are so important, can you explain some specific actions that agencies should take? And given what you learned while researching the paper, do you think that government agencies can safely deploy AI and ML systems without taking on significant additional risks?
Anley: Like any new technology, machine learning brings new opportunities and new risks. There are certainly things that organizations can do when developing and deploying ML systems that will help reduce their risk. Some good general advice includes:
And specifically in terms of machine learning or AI systems, do all of the aforementioned activities plus the following:
John Breeden II is an award-winning journalist and reviewer with over 20 years of experience covering technology. He is the CEO of the Tech Writers Bureau, a group that creates technological thought leadership content for organizations of all sizes. Twitter: @LabGuys
NEXT STORY: Log4j Vulnerability Prompts Lawmakers to Examine Agency Cyber Measures
Do Not Sell My Personal Information
When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.
Strictly Necessary Cookies - Always Active
We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.
Sale of Personal Data, Targeting & Social Media Cookies
Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link
If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.
Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.
Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.
If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page. Save Settings
A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:
We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.
We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.
We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.
We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.
We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.
We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.
Help us tailor content specifically for you: