Should the government have the right to troll through your thoughts and memories? That seems like a question for a "Minority Report" or "Matrix" future, but legal precedent is being set today. This is what is really at stake in an emerging tussle between Washington DC and Silicon Valley.
The Internets are
all abuzz with Apple's refusal to hack an iPhone belonging to an
accused terrorist. The FBI has served a court order on Apple, based
on the All Writs Act of 1789, requiring Apple to break the
lock that limits the number of times a passcode can be tried. Since
law enforcement has been unable to crack the security of iOS on its
own, it wants Apple to write special software to do the job. Here is Wired's summary. This NYT story has additional good background. The short version: should law enforcement and intelligence agencies be able to compel corporations to hack devices owned by citizens and entrusted with their sensitive information?
Apple CEO Tim Cook published a letter saying no, thank you, because weakening the security of iPhones would be bad for his customers and "has implications far beyond the legal case at hand". Read Cook's letter; it is thoughtful. The FBI says it is just about this one phone and "isn't about trying to set a precedent," in the words of FBI Director James Comey. But this language is neither accurate nor wise — and it is important to say so.
Once the software is written, the U.S. government can hardly argue it will never be used again, nor that it will never be stolen off government servers. And since the point of the hack is to be able to push it onto a phone without consent (which is itself a backdoor that needs closing), this software would allow breaking the locks on any susceptible iPhone, anywhere. Many commentators have observed that any effort to hack iOS this once would facilitate repetitions, and any general weakening of smartphone security could easily be exploited by governments or groups less concerned about due process, privacy, or human rights. (And you do have to wonder whether Tim Cook's position here is influenced by his experience as a gay man, a demographic that has been persecuted, if not actually prosecuted, merely for thought and intent by the same organization now sitting on the other side of the table. He knows a thing or two about privacy.) U.S. Senator Ron Wyden has a nice take on these issues. Yet while these are critically important concerns for modern life, they are shortsighted. There is much more at stake here than just one phone, or even the fate of a one particular company. The bigger, longer term issue is whether governments should have access to electronic devices that we rely on in daily life, particularly when those devices are becoming extensions of our bodies and brains. Indeed, these devices will soon be integrated into our bodies — and into our brains.
Hacking electronically-networked brains sounds like science fiction. That is largely because there has been so much science fiction produced about neural interfaces, Matrices, and the like. We are used to thinking of such technology as years, or maybe decades, off. But these devices are already a reality, and will only become more sophisticated and prevalent over the coming decades. Policy, as usual, is way behind.
My concern, as usual, is less about the hubbub in the press today and instead about where this all leads in ten years. The security strategy and policy we implement today should be designed for a future in which neural interfaces are commonplace. Unfortunately, today's politicians and law enforcement are happy to set legal precent that will create massive insecurity in just a few years. We can be sure that any precedent of access to personal electronic devices adopted today, particularly any precedent in which a major corporation is forced to write new software to hack a device, will be cited at least decades hence, when technology that connects hardware to our wetware is certain to be common. After all, the FBI is now proposing that a law from 1789 applies perfectly well in 2016, allowing a judge to "conscript Apple into government service", and many of our political representatives appear delighted to concur. A brief tour of current technology and security flaws sets the stage for how bad it is likely to get.
As I suggested a couple of years ago, hospital networks and medical devices are examples of existing critical vulnerabilities. Just in the last week hackers took control of computers and devices in a Los Angeles hospital, and only a few days later received a ransom to restore access and functionality. We will be seeing more of this. The targets are soft, and when attacked they have little choice but to pay when patients' health and lives are on the line. What are hospitals going to do when they are suddenly locked out of all the ventilators or morphine pumps in the ICU? Yes, yes, they should harden their security. But they won't be fully successful, and additional ransom events will inevitably happen. More patients will be exposed to more such flaws as they begin to rely more on medical devices to maintain their health. Now consider where this trend is headed: what sorts of security problems will we create by implanting those medical devices into our bodies?
Already on the market are cochlear implants that are essentially ethernet connections to the brain, although they are not physically configured that way today. An external circuit converts sound into signals that directly stimulate the auditory nerves. But who holds the password for the hardware? What other sorts of signals can be piped into the auditory nerve? This sort of security concern, in which networked electronics implanted in our bodies create security holes, has actually been with us for more than a decade. When serving as Vice President, Dick Cheney had the wireless networking on his fully-implanted heart defibrillator disabled because it was perceived as a threat. The device contained a test mode that could exploited to fully discharge the battery into the surrounding tissue. This might be called a fatal flaw. And it will only get worse.
DARPA has already limited the strength of a recently developed, fully articulated bionic arm to "human normal" precisely because the organization is worried about hacking. These prosthetics are networked in order to tune their function and provide diagnostic information. Hacking is inevitable, by users interested in modifications and by miscreants interested in mischief.
Not content to replace damaged limbs, within the last few months DARPA has announced a program to develop what the staff sometimes calls a "cortical modem". DARPA is quite serious about developing a device that will provide direct connections between the internet and the brain. The pieces are coming together quickly. Several years ago a patient in Sweden received a prosthesis grafted to the bone in his arm and controlled by local neural signals. Last summer I saw Gregoire Courtine show video of a monkey implanted with microfabricated neural bridge that spanned a severed spinal cord; flip a switch on and the monkey could walk, flip it off and the monkey was lame. Just this month came news of an implanted cortical electrode array used to directly control a robot arm. Now, imagine you have something like this implanted in your spine or head, so that you can walk or use an arm, and you find that the manufacturer was careless about security. Oops. You'll have just woken up — unpleasantly — in a William Gibson novel. And you won't be alone. Given the massive medical need, followed closely by the demand for augmentation, we can expect rapid proliferation of these devices and accompanying rapid proliferation of security flaws, even if today they are one-offs. But that is the point; as Gibson has famously observed, "The future is already here — it's just not evenly distributed yet."
When — when — cortical modems become an evenly distributed human augmentation, they will inevitably come with memory and computational power that exceeds the wetware they are attached to. (Otherwise, what would be the point?) They will expand the capacity of all who receive them. They will be used as any technology is, for good and ill. Which means they will be targets of interest by law enforcement and intelligence agencies. Judges will be grappling with this for decades: where does the device stop and the human begin? ("Not guilt by reason of hacking, your honor." "I heard voices in my head.") And these devices will also come with security flaws that will expose the human brain to direct influence from attackers. Some of those flaws will be accidents, bugs, zero-days. But how will we feel about back doors built in to allow governments to pursue criminal or intelligence investigations, back doors that lead directly into our brains? I am profoundly unimpressed by suggestions that any government could responsibly use or look after keys to any such back door.
There are other incredibly interesting questions here, though they all lead to the same place. For example, would neural augmentation count as a medical device? If so, what does the testing look like? If not, who will be responsible for guaranteeing safety and security? And I have to wonder, given the historical leakiness of backdoors, if governments insist on access to these devices who is going to want to accept liability inherent in protecting access to customers' brains? What insurance or reinsurance company would issue a policy indemnifying a cortical modem with a known, built-in security flaw? Undoubtably an insurance policy can be written that exempts governments from responsibility for the consequences of using a backdoor, but how can a government or company guarantee that no one else will exploit the backdoor? Obviously, they can do no such thing. Neural interfaces will have to be protected by maximum security, otherwise manufacturers will never subject themselves to the consequent product liability.
Which brings us back to today, and the precedent set by Apple in refusing to make it easy for the FBI to hack an iPhone. If all this talk of backdoors and golden keys by law enforcement and politicians moves forward to become precedent by default, or is written into law, we risk building security holes into even more devices. Eventually, we will become subject to those security holes in increasingly uncomfortable, personal ways. That is why it is important to support Tim Cook as he defends your brain.
On the 16 of July 1945, after the first nuclear test at large scale in New Mexico (called trinity) the American physicist Kenneth Bainbridge, head of the shooting, told Robert Oppenheimer, head of the Manhattan Project, "Now we are all sons of bitches ".
In your discipline, do you feel that the time the searchers might have the same revelation has been reached ? Will it be soon?
I think this analogy does not apply to biotechnology. It is crucially important to distinguish between weapons developed in a time of war and the pursuit of science and technology in a time of peace. Over the last thirty years, biotechnology has emerged as a globally important technology because it is useful and beneficial.
The development and maintenance of biological weapons is internationally outlawed, and has been for decades. The Trinity test, and more broadly the Manhattan Project, was a response to what the military and political leaders of the time considered an existential threat. These were actions taken in a time of world war. The scientists and engineers who developed the U.S. bombs were almost to a person ambivalent about their roles – most saw the downsides, yet were also convinced of their responsibility to fight against the Axis Powers. Developing nuclear weapons was seen as imperative for survival.
The scale of the Manhattan Project (both in personnel and as a fraction of GDP) was unprecedented, and remains so. In contrast to the exclusive governmental domain of nuclear weapons, biotechnology has been commercially developed largely with private funds. The resulting products – whether new drugs, new crop traits, or new materials – have clear beneficial value to our society.
Do you have this feeling in other disciplines? Which ones ? Why?
No. There is nothing in our experience like the Manhattan Project and nuclear weapons. It is easy to point to the participants’ regrets, and to the long aftereffects of dropping the bomb, as a way to generate debate about, and fear of, new technologies. The latest bugaboos are artificial intelligence and genetic engineering. But neither of these technologies – even if they can be said to qualify as mature technologies – is even remotely as impactful as nuclear weapons.
What could be the impact of a "Hiroshima" in your discipline?
In biosecurity circles, you often hear discussion of what would happen if there were “an event”. It is often not clear what that event might be, but it is presumed to be bad. The putative event could be natural or it could be artificial. Perhaps the event might kill many people as Hiroshima. (Though that would be hard, as even the most deadly organisms around today cannot wipe out populated cities in an instant.) Perhaps the event would be the intentional use of a biological weapon, and perhaps that weapon would be genetically modified in some way to enhance its capabilities. This would obviously be horrible. The impact would depend on where the weapon came from, and who used it. Was it the result of an ongoing state program? Was it a sample deployed, or stolen, from discontinued program? Or was it built and used by a terrorist group? A state can be held accountable by many means, but we are finding it challenging to hold non-state groups to account. If the organism is genetically modified, it is possible that there will be pushback against the technology. But biotechnology is producing huge benefits today, and restrictions motivated by the response to an event would reduce those benefits. It is also very possible that biotechnology will be the primary means to provide remedies to bioweapons (probably vaccines or drugs), in which case an event might wind up pushing the technology even faster.
After 1945, physicists, including Einstein, have committed an ethical reflection on their own work. has your discipline done the same ? is it doing the same today ?
Ethical reflection has been built into biotechnology from its origins. The early participants met at Asilomar to discuss the implications of their work. Today, students involved in the International Genetically Engineered Machines (iGEM) competition are required to complete a “policy and practices” (also referred to as “ethical, legal, and social implications” (ELSI)) examination of their project. This isn’t window dressing, by any means. Everyone takes it seriously.
Do you think it would be necessary to rase the public awarereness about the issues related to your work?
Well, I’ve been writing and speaking about this issue for 15 years, trying to raise awareness of biotechnology and where it is headed. My book, “Biology is Technology”, was specifically aimed at encouraging public discussion. But we definitely need to work harder to understand the scope and impact of biotechnology on our lives. No government measures very well the size of the biotechnology industry – either in terms of revenues or in terms of benefits – so very few people understand how economically pervasive it is already.
What is, according to you, the degree of liberty of scientists face to political and industrial powers that will exploit the results of the scientific works?
Scientists face the same expectation of personal responsibility as every other member of the societies to which they belong. That’s pretty simple. And most scientists are motivated by ideals of truth, the pursuit of knowledge, and improving the human condition. That is one reason why most scientists publish their results for others to learn from. But it is less clear how to control scientific results after they are published. I would turn your question in another direction, and say politicians and industrialists should be responsible for how they use science, rather than putting this all on scientists. If you want to take this back to the bomb, the Manhattan Project was a massive military operation in a time of war, implemented by both government and the private sector. It relied on science, to be sure, but it was very much a political and industrial activity – you cannot divorce these two sides of the Project.
Do you think about accurate measures [?] to prevent further Hiroshima?
I constantly think about how to prevent bad things from happening. We have to pay attention to how new technologies are developed and used. That is true of all technologies. For my part, I work domestically and internationally to make sure policy makers understand where biotechnology is headed and what it can do, and also to make sure it is not misused.
But I think the question is rather off target. Bombing Hiroshima was a conscious decision made by an elected leader in a time of war. It was a very specific sort of event in a very specific context. We are not facing any sort of similar situation. If the intent of the question is to make an analogy to intentional use of biological weapons, these are already illegal, and nobody should be developing or storing them under any circumstances. The current international arms control regime is the way to deal with it. If the intent is to allude to the prevention of “bad stuff”, then this is something that every responsible citizen should be doing anyway. All we can do is pay attention and keep working to ensure that technologies are not used maliciously.
Last week brought news of a truly interesting advance in porting opioid production to yeast. This is pretty cool science, because it involves combining enzymes from several different organisms to produce a complex and valuable chemical, although no one has yet managed to integrate the whole synthetic pathway in microbes. It is also potentially pretty cool economics, because implementing opiate production in yeast should dramatically lower the price of a class of important pain medications to a point that developing countries might finally be able to afford.
Alongside the scientific article was a Commentary – with images of drug dens and home beer brewing – explicitly suggesting that high doses of morphine and other addictive narcotics would soon be brewed at home in the garage. The text advertised “Home-brew opiates” – wow, just like beer! The authors of the Commentary used this imagery to argue for immediate regulation of 1) yeast strains that can make opioids (even though no such strains exist yet), and 2) the DNA sequences that code for the opioid synthesis pathways. This is a step backward for biosecurity policy, by more than a decade, because the proposal embraces measures known to be counterproductive for security.
The wrong recipe.
I'll be very frank here – proposals like this are deep failures of the science policy enterprise. The logic that leads to “must regulate now!” is 1) methodologically flawed and 2) ignores data we have in hand about the impacts of restricting access to technology and markets. In what follows, I will deal in due course with both kinds of failures, as well as looking at the predilection to assume regulation and restriction should be the primary policy response to any perceived threat.
There are some reading this who will now jump to “Carlson is yet again saying that we should have no regulation; he wants wants everything to be available to anyone.” This is not my position, and never has been. Rather, I insist that our policies be grounded in data from the real world. And the real world data we have demonstrates that regulation and restriction often cause more harm than good. Moreover, harm is precisely the impact we should expect by restricting access to democratized biological technologies. What if even simple analyses suggests that proposed actions are likely to make things worse? What if the specific policy actions recommended in response to a threat have already been shown to exacerbate damage from the threat? That is precisely the case here. I am constantly confronted with people saying, "That's all very well and good, but what do you propose we do instead?" The answer is simple: I don't know. Maybe nothing. Maybe there isn't anything we can do. But for now, I just want us to not make things worse. In particular I want to make sure we don't screw up the emerging bioeconomy by building in perverse incentives for black markets, which would be the worst possible development for biosecurity.
Policy conversations at all levels regularly make these same mistakes, and the arguments are nearly uniform in structure. “Here is something we don't know about, or are uncertain about, and it might be bad – really, really bad – so we should most certainly prepare policy options to prevent the hypothetical worst!” Exclamation points are usually just implied throughout, but they are there nonetheless. The policy options almost always involve regulation and restriction of a technology or process that can be construed as threatening, usually with little or no consideration of what that threatening thing might plausibly grow into, nor of how similar regulatory efforts have fared historically.
This brings me to the set up. Several news pieces (e.g., the NYT, Buzzfeed) succinctly pointed out that the “home-brew” language was completely overblown and inflammatory, and that the Commentary largely ignored both the complicated rationale for producing opioids in yeast and the complicated benefits of doing so. The Economist managed to avoid getting caught up in discussing the Commentary, remaining mostly focussed on the science, while in the last paragraph touching on the larger market issues and potential future impacts of “home brew opium” to pull the economic rug out from under heroin cartels. (Maybe so. It's an interesting hypothesis, but I won't have much to say about it here.) Over at Biosecu.re, Piers Millet – formerly of the Biological Weapons Convention Implementation Support Unit – calmly responded to the Commentary by observing that, for policy inspiration, the authors look backward rather than forward, and that the science itself demonstrates the world we are entering requires developing completely new policy tools to deal with new technical and economic realities.
Stanford's Christina Smolke, who knows a thing or two about opioid production in yeast, observed in multiple news outlets that getting yeast to produce anything industrially at high yields is finicky to get going and then hard to maintain as a production process. It's relatively easy to produce trace amounts of lots of interesting things in microbes (ask any iGEM team); it is very hard and very expensive to scale up to produce interesting amounts of interesting things in microbes (ask any iGEM team). Note that we are swimming in data about how hard this is to do, which is an important part of this story. In addition to the many academic examples of challenges in scaling up production, the last ten years are littered with startups that failed at scale up. The next ten years, alas, will see many more.
Even with an engineered microbial strain in hand, it can be extraordinarily hard to make a microbe jump through the metabolic and fermentation hoops to produce interesting/useful quantities of a compound. And then transferring that process elsewhere is very frequently its own expensive and difficult effort. It is not true that you can just mail a strain and a recipe from one place to another and automatically get the same result. However, it is true that all this will get easier over time, and many people are working on reproducible process control for biological production.
That future looks amazing. I've written many times about how the future of the economy looks like beer and cows – in other words, that our economy will inevitably be based on distributed biological manufacturing. But that is the future: i.e., not the present. Nor is it imminent. I truly wish it were imminent, but it is not. Whole industries exist to solve these problems, and much more money and effort will be spent before we get there. The economic drivers are huge. Some of the investments made by Bioeconomy Capital are, in fact, aimed at eventually facilitating distributed biological manufacturing. But, if nothing else, these investments have taught me just how much effort is required to reach that goal. If anybody out there has a credible plan to build the Cowborg or to microbrew chemicals and pharmaceuticals as suggested by the Commentary, I will be your first investor. (I said “credible”! Don't bother me otherwise.) But I think any sort of credible plan is years away. For the time being, the only thing we can expect to brew like beer is beer.
FBI Supervisory Special Agent Ed You makes great use of the “brewing bad” and “baking bad” memes, mentioned in the Commentary, in talking to students and professionals alike about the future of drug production. But this is in the context of taking personal responsibility for your own science and for speaking up when you see something dangerous. I've never heard Ed say anything about increasing surveillance and enforcement efforts as the way forward. In fact, in the Times piece, Ed specifically says, “We’ve learned that the top-down approach doesn’t work.” I can't say exactly why Ed chose that turn of phrase, but I can speculate that it is based 1) on his own experience as a professional bench molecular biologist, 2) the catastrophically bad impacts of the FBI's earlier arrests and prosecutions of scientists and artists for doing things that were legal, and 3) the official change in policy from the White House and National Security Council away from suppression and toward embracing and encouraging garage biology. The standing order at the FBI is now engagement. In fact, Ed You's arrival on the scene was coincident with any number of positive policy changes in DC, and I am happy to give him all the credit I can. Moreover, I completely agree with Ed and the Commentary authors that we should be discussing early on the implications of new technologies, an approach I have been advocating for 15 years. But I completely disagree with the authors that the current or future state of the technology serves as an indicator of the need to prepare some sort of regulatory response. We tried regulating fermentation once before; that didn't work out so well .
Badly baked regulatory policy.
So now we're caught up to about the middle of the Commentary. At this point, the story is like other such policy stories. “Assume hypothetical thing is inevitable: discuss and prepare regulation.” And like other such stories, here is where it runs off the rails with a non sequitur common in policy work. Even if the assumption of the thing's inevitability is correct (which is almost always debatable), the next step should be to assess the impact of the thing. Is it good, or is it bad? (By a particular definition of good and bad, of course, but never mind that for now.) Usually, this question is actually skipped and the thing is just assumed to be bad and in need of a policy remedy, but the assumption of badness, breaking or otherwise, isn't fatal for the analysis.
Let's say it looks bad – bad, bad, bad – and the goal of your policy is to try to either head it off or fix it. First you have to have some metric to judge how bad it is. How many people are addicted, or how many people die, or how is the crime rate affected? Just how bad is it breaking? Next – and this is the part the vast majority of policy exercises miss – you have to try to understand what happens in the absence of a policy change. What is the cost of doing nothing, of taking no remediating action? Call this the null hypothesis. Maybe there is even a benefit to doing nothing. But only now, after evaluating the null hypothesis, are you in a position to propose remedies, because only now you have a metric to compare costs and benefits. If you leap directly to “the impacts of doing nothing are terrible, so we must do something, anything, because otherwise we are doing nothing”, then you have already lost. To be sure, policy makers and politicians feel that their job is to do something, to take action, and that if they are doing nothing then they aren't doing their jobs. That is just a recipe for bad policy. Without the null hypothesis, your policy development is a waste of time and, potentially, could make matters worse. This happens time and time again. Prohibition, for example, was exactly this sort of failure, and cost much more than it benefited, which is why it was considered a failure .
We keep making the same mistake. We have plenty of data and reporting, courtesy of the DEA, that the ongoing crackdown on methamphetamine production has created bigger and blacker markets, as well as mayhem and violence in Mexico, all without much impact on domestic drug use. Here is the DEA Statistics & Facts page – have a look and then make up your own mind.
I started writing about the potential negative impacts of restricting access to biological technologies in 2003 (PDF), including the likely emergence of black markets in the event of overregulation. I looked around for any data I could find on the impacts of regulating democratized technologies. In particular, I happened upon the DEA's first reporting of the impacts of the then newly instituted crackdown on domestic methamphetamine production and distribution. Even in 2003, the DEA was already observing that it had created bigger, blacker markets – that are by definition harder to surveil and disrupt – without impacting meth use. The same story has played out similarly in cocaine production and distribution, and more recently in the markets for “bath salts”, aka “legal highs”.
That is, we have multiple, clear demonstrations that, rather than improving the world, restricting access to distributed production can instead cause harm. But, really, when has this ever worked? And why do people think going down the same path in the future will lead anywhere else? I am still looking for data – any data at all – that supports the assertion that regulating biological technologies will have any different result. If you have such data, bring it. Let's see it. In that absence of that data, policy proposals that lead with regulation and restriction are doomed to repeat the failures of the past. It has always seemed to me like a terrible idea to transfer such policies over to biosecurity. Yet that is exactly what the Commentary proposes.
Brewing black markets.
The fundamental problem with the approach advocated in the Commentary is that security policies, unlike beer brewing, do not work equally well across all technical and economic scales. What works in one context will not work in another. Nuclear weapons can be secured by guns, gates, and guards because they are expensive to build and the raw materials are hard to come by, so heavy touch regulation works just fine. There are some industries – as it happens, beer brewing – where only light touch regulation works. In the U.S., we tried heavy touch regulation in the form of Prohibition, and it failed miserably, creating many more problems than it solved. There are other industries, for example DNA and gene synthesis, in which even light touch regulations are a bad idea. Indeed, light touch regulation of has already created the problem it was supposed to prevent, namely the existence of DNA synthesis providers that 1) intentionally do not screen their orders and 2) ship to countries and customers that are on unofficial black lists.
For those who don't know this story: In early 2013, the International Council for the Life Sciences (ICLS) convened a meeting in Hong Kong to discuss "Codes of Conduct" for the DNA synthesis industry, namely screening orders and paying attention to who is doing the ordering. According to various codes and guidelines promulgated by industry associations and the NIH, DNA synthesis providers are supposed to reject orders that are similar to sequences that code for pathogens, or genes from pathogens, and it is suggested that they do not ship DNA to certain countries or customers (the unofficial black list). Here is a PDF of the meeting report; be sure to read through Appendix A.
The report is fairly anodyne in describing what emerged in discussions. But people who attended have since described in public the Chinese DNA synthesis market as follows. There are 3 tiers of DNA providers. The first tier is populated with companies that comply with the various guidelines and codes promulgated internationally because this tier serves international markets. There is a second tier that appears to similarly comply, because while they serve primarily the large internal market these companies have aspirations of also serving the international market. There is a third tier that exists specifically to serve orders from customers seeking ways around the guidelines and codes. (One company in this tier was described to me as a "DNA shanty", with the employees living over the lab.) Thus the relatively light touch guidelines (which are not laws) have directly incentivized exactly the behavior they were supposed to prevent. This is not a black market, per se, and cannot be accurate described as illegal, so let's call it a "grey market".
I should say here that this is entirely consistent with my understanding of biotech in China. In 2010, I attended a warm up meeting for the last round of BWC negotiations. After that meeting, I chatted with one of the Chinese representatives present, hoping to gain a little bit of insight into the size of the Chinese bioeconomy and the state of the industry. My query was met with frank acknowledgment that the Chinese government isn't able to keep track of the industry, does't know how many companies are active, or how many employees they have, or what they are up to, and so doesn't hold out much hope of controlling the industry. I covered this a bit in my 2012 Biodefense Net Assessment report for DHS. (If anyone has any new insight into the Chinese biotech industry, I am all ears.) Not that the U.S. or Europe is any better in this regard, as our mechanisms for tracking the biotech industry are completely dysfunctional, too. There could very well be DNA synthesis providers operating elsewhere that don't comply with the recommended codes of conduct: we have no real means of broadly surveying for this behavior. There are no physical means either to track it remotely or to control it.
I am a little bit sensitive about the apparent emergence of the DNA synthesis grey market, because I warned for years in print and in person that DNA screening would create exactly this outcome. I was condescendingly told on many occasions that it was foolish to imagine a black market for DNA. And then we have to do something, right? But it was never very complicated to think this through. DNA is cheap, and getting cheaper. You need this cheap DNA as code to build more complicated, more valuable things. Ergo, restrictions on DNA synthesis will incentivize people to seek, and to provide, DNA outside any control mechanism. The logic is pretty straightforward, and denying it is simply willful self-deception. Regulation of DNA synthesis will never work. In the vernacular of the day: because economics. To make it even simpler: because humans.
So the idea that people are still suggesting proscription of certain DNA sequences is a viable route to security just rankles. And it is demonstrably counterproductive. The restrictions incentivize the bad behavior they are supposed to prevent, probably much earlier than might have happened otherwise. The take home message here is that not all industries are the same, because not all technologies are the same, and that our policy approaches should take into account these differences rather than papering over them. In particular, restricting access to information in our modern economy is a losing game.
Where do we go from here?
We are still at the beginning of biotech. This is the most important time to get it right. This is the most important time not to screw up and make things worse. And it is important that we are at the beginning, because things are not yet screwed up.
Conversely, we are well down the road in developing and deploying drug policies, with much damage done. To be sure, despite the accumulated and ongoing costs, you have to acknowledge that it is not at all clear that suddenly legalizing drugs such as meth or cocaine would be a positive step. I am not in any way making that argument. But it is abundantly clear that drug enforcement activities have created the world we live in today. Was there an alternative? If the DEA had been able to do cost/benefit analysis of the impacts of its actions – that is, predict the emergence of DTOs and their role in production, trafficking, and violence – would the policy response 15 years ago have been any different? If Nixon had more thoughtfully considered even what was known 50 years about about the impacts of proscription, would he have launched the war on drugs? That is a hard question, because drug policy is clearly driven more by stories and personal politics than by facts. I am inclined to think the present drug policy mess was inevitable. Even with the DEA's self-diagnosed role in creating and sustaining DTOs, the national conversation is still largely dominated by “the war on drugs”. And thus the first reaction to the prospect of microbial narcotics production is to employ strategies and tactics that have already failed elsewhere. I would hate to think we are in for a war on microbes, because that is doomed to failure.
But we haven't yet made all those mistakes with biological technologies. I continue to hope that, if nothing else, we will avoid making things worse by rejecting policies we already know won't work.
 Pause here to note that even this early in the set up, the Commentary conflates via words and images the use of yeast in home brew narcotics with centralized brewing of narcotics by cartels. These are two quite different, and are perhaps mutually exclusive, technoeconomic futures. Drug cartels very clearly have the resources to develop technology. Depending on whether you listen to the U.S. Navy or the U.S. Coast Guard, either 30% or 80% of the cocaine delivered to the U.S. is transported at some point in semisubmersible cargo vessels or in fully submersible cargo submarines. These 'smugglerines', if you will, are the result of specific technology development efforts directly incentivized by governmental interdiction efforts. Similarly, if cartels decide that developing biological technologies suits their business needs, they are likely to do so. And cartels certainly have incentives to develop opioid-producing yeast, because fermentation usually lowers the cost of goods between 50% and 90% compared to production in plants. Again, cartels have the resources, and they aren't stupid. If cartels do develop these yeast strains, for competitive reasons they certainly won't want anyone else to have them. Home brew narcotics would further undermine their monopoly.
 Prohibition was obviously the result of a complex socio-political situation, just as was its repeal. If you want a light touch look at the interaction of the teetotaler movement, the suffragette movement, and the utility of Prohibition in continued repression of freed slaves after the Civil War, check out Ken Burns's “Prohibition” on Netflix. But after all that, it was still a dismal failure that created more problems than it solved. Oh, and Prohibition didn't accomplish its intended aims. Anheuser-Busch thrived during those years. Its best selling products at the time were yeast and kettles (see William Knoedleseder's Bitter Brew)...
- Riffyn, which is building software that provides experimental process design and analytics software to improve reproducibility and tech transfer in life science and materials R&D;
- Synthace, which is increasing the reliability, quality, and scale of biological science;
- RoosterBio, which is is creating exponential advances in stem cell manufacturing to provide raw materials for cell-based therapies, biofabrication, and cellular ink for 3D BioPrinting.
Part 2. From natural security to neural security
Humans are fragile. For most of history we have lived with the expectation that we will lose the use of organs, and some of us limbs, as we age or suffer injury. But that is now changing. Prostheses are becoming more lifelike and more useful, and replacement organs have been used to save lives and restore function. But how robust are the replacement parts? The imminent prospect of technological restoration of human organs and limbs lost to injury or disease is cause to think carefully about increasing both our biological capabilities and our technological fragilities.
Technology fails us for many reasons. A particular object or application may be poorly designed or poorly constructed. Constituent materials may be faulty, or maintenance may be shoddy. Failure can result from inherent security flaws, which can be exploited directly by those with sufficient technical knowledge and skill. Failure can also be driven by clever and conniving exploits of the overall system that focus on its weakest link, almost always the human user, by inducing them to make a mistake or divulge critical information. Our centuries of experience and documentation of such failures should inform our thinking about the security of emerging technologies, particularly as we begin to fuse biology with electronic systems. The growing scope of biotechnology will therefore require constant reassessment of what vulnerabilities we are introducing through that expansion. Examining the course of other technologies provides some insight into the future of biology.
We carry powerful computers in our pockets, use the internet to gather information and access our finances, and travel the world in aircraft that are often piloted and landed by computers. We are told we can trust this technology with our financial information, our identities and social networks, and, ultimately, our lives. At the same time, technology is constantly shown to be vulnerable and fragile at a non-trivial rate -- resulting in identity theft, financial loss, and sometimes personal injury and death. We embrace technology despite well-understood risks; automobiles, electricity, fossil fuels, automation, and bicycles all kill people every day in predictable numbers. Yet we continue to use technology, integrating it further into multiple arenas in our lives, because we decide that the benefits outweigh risks.
Healthcare is one arena in which risks are multiplying. The IT security community has for some years been aware of network vulnerabilities in medical devices such as pacemakers and implantable defibrillators. The ongoing integration of networked medical devices in health care settings, an integration that is constantly introducing both new capabilities and new vulnerabilities, is now the focus of extensive efforts to improve security. The impending introduction of networked, semi-autonomous prostheses raises obvious similar concerns. Wi-fi enabled pacemakers and implantable defibrillators are just the start, as soon we will see bionic arms, legs, and eyes with network connections that allow performance monitoring and tuning.
Eventually, prostheses will not simply restore "human normal" capabilities, they will also augment human performance. I learned recently that DARPA explicitly chose to limit the strength of its robotic arm, but that can't last: science fiction, super robotic strength is coming. What happens when hackers get ahold of this technology? How will people begin to modify themselves and their robotic appendages? And, of course, the flip side of having enhanced physical capabilities is having enhanced vulnerabilities. By definition, tuning can improve or degrade performance, and this raises an important security question: who holds the password for your shiny new arm? Did someone remember to overwrite the factory default password? Is the new password susceptible to a dictionary attack? The future brings even more concerns. Control connections to a prosthesis are bi-directional and, as the technology improves, ever better neural interfaces will eventually jack these prostheses directly into the brain. "Tickling" a robotic limb could take on a whole new meaning, providing a means to connect various kinds of external signals to the brain in new ways.
Beyond limbs, we must also consider neural connections that serve to open entirely novel senses. It is not a great leap to envision a wide range of ensuing digital-to-neural input/output devices. These technologies are evolving at a rapid rate, and through them we are on the cusp of opening up human brains to connections with a wide range of electromechanical hardware capabilities and, indeed, all the information on the internet.
Just this week saw publication of a cochlear implant that delivers a gene therapy to auditory neurons, promoting the formation of electrical connections with the implant and thereby dramatically improving the hearing response of test animals. We are used to the idea of digital music files being converted by speakers into sound waves, which enter the brain through the ear. But the cochlear implant is basically an ethernet connection wired to your auditory nerve, which in principal means any signal can be piped into your brain. How long can it be before we see experiments with a cochlear (or other) implant that enables direct conversion of arbitrary digital information into neural signals? At that point, "hearing" might extend into every information format. So, again we must ask, who holds the password to your brain implant?
Hacking the Bionic Man
As this technology is deployed in the population it is clear that there can be no final and fixed security solution. Most phone and computer users are now all too aware that new hardware, firmware, and operating systems always introduce new kinds of risks and threats. The same will be true of prostheses. The constant rat race to chase down security holes in new products upgrades will soon extend directly into human brains. As more people are exposed to medical device vulnerabilities, security awareness and improvement must become an integrated part of medical practice. This discussion can be easily extended to potential vulnerabilities that will arise from the inevitable integration into human bodies of not just electromechanical devices, but of ever more sophisticated biological technologies. The exploration of prosthesis security, loosely defined, gives some indication of the scope of the challenge ahead.
The class of things we call prostheses will soon expand beyond electromechanical devices to encompass biological objects such as 3D printed tissues and lab-grown organs. As these cell-based therapies begin to enter human clinical trials, we must assess the security of both the therapies themselves and the means used to create and administer them. If replacement organs and tissues are generated from cells derived from donors, what vulnerabilities do the donors have? How are those donor vulnerabilities passed along to the recipients? Yes, you have an immune system that does wonders most of the time. But are your natural systems up to the task of handling the biosecurity of augmented organs?
What does security even mean in this context? In addition to standard patient work-ups, should we begin to fully sequence the genomes of donor tissues, first to identify potential known health issues, and then to build a database that can be re-queried as new genetic links to disease are discovered? Are there security holes in the 3D printers and other devices used to manipulate cells and tissues? What are the long term security implications of deploying novel therapeutic tissues in large numbers of military and civilian personnel? What are the long-term security implications of using both donor and patient tissue as seeds of induced pluripotent stem cells, or of differentiating any stem cell line for use in therapies? Do we fully understand the complement of microbes and genomes that may be present in donor samples, or lying dormant in donor genomes, or that may be introduced via laboratory procedures and instruments used to process cells for use as therapies? What is the genetic security of a modified cell line or induced pluripotent stem cell? If there is a genetic modification embedded in your replacement heart tissue, where did the new DNA come from, and are you sure you know everything that it encodes? As with information technologies, we should expect that these new biological technologies will sometimes arrive with accidental vulnerabilities; they may also come with intentionally introduced back doors. The economic motivation to create new protheses, as well as to exploit vulnerabilities, will soon introduce market competition as a factor in biosecurity.
Competition often drives perverse strategic decisions when it comes to security. Firms rush to sell hardware and software that are said to be secure, only to discover that constant updates are required to patch security holes. We are surrounded by products in endless beta. Worse yet, manufacturers have been known to sit on security holes in the naive hope that no one else will notice. Vendors sometimes appear no more literate about the security of hardware and software than are their customers. What will the world look like when eletromechanical and biological prostheses are similarly in constant states of upgrade? Who will you trust to build/print/grow a prosthesis? Are you going to place your faith in the FDA to police all these risks? (Really?) If you decide to instead place your faith in the market, how will you judge the trustworthiness of firms that sell aftermarket security solutions for your bionic leg or replacement liver?
The complexity of the task at hand is nearly overwhelming. Understanding the coming fusion of technologies will require competency in software, hardware, wetware, and security -- where are those skill sets being developed in a compatible, integrated manner? This just leads to more questions: Are there particular countries that will have a competitive advantage in this area? Are there particular countries that will be hotbeds of prosthesis malware creation and distribution?
The conception of security, whether of individuals or nation states, is going to change dramatically as we become ever more economically dependent upon the market for biological technologies. Given the spreading capability to participate and innovate in technology development, which inevitably amplifies the number and effect of vulnerabilities of all kinds, I suspect we need to re-envision at a very high level how security works.
[Coming soon: Part 3.]
Part 1. The ecosystem is the enterprise
We live in a society increasingly reliant upon the fruits of nature. We consume those fruits directly, and we cultivate them as feedstocks for fuel, industrial materials, and the threads on our backs. As a measure of our dependence, revenues in the bioeconomy are rising rapidly, demonstrating a demand for biological products that is growing much faster than the global economy as a whole.
This demand represents an enormous market pull on technology development, commercialization, and, ultimately, natural resources that serve as feedstocks for biological production. Consequently, we must assess carefully the health and longevity of those resources. Unfortunately, it is becoming ever clearer that the natural systems serving to supply our demand are under severe stress. We have been assaulting nature for centuries, with the heaviest blows delivered most recently. Nature, in the most encompassing sense of the word, has been astonishingly resilient in the face of this assault. But the accumulated damage has cracked multiple holes in ecosystems around the globe. There are very clear economic costs to this damage -- costs that compound over time -- and the cumulative damage now poses a threat to the availability of the water, farmland, and organisms we rely on to feed ourselves and our economy.
I would like to clarify that I am not predicting collapse, nor that we will run out of resources; rather, I expect new technologies to continue increasing productivity and improving the human condition. Successfully developing and deploying those technologies will, obviously, further increase our economic dependency on nature. As part of that growing dependency, businesses that participate in the bioeconomy must understand and ensure the security of feedstocks, transportation links, and end use, often at a global scale. Consequently, it behooves us to thoroughly evaluate any vulnerabilities we are building into the system so that we can begin to prepare for inevitable contingencies.
Revisiting the definition of biosecurity: from national security to natural security, and beyond
Last year John Mecklin at Bulletin of the Atomic Scientists asked me to consider the security implications of the emerging conversation (or, perhaps, collision) between synthetic biology and conservation biology. This conversation started at a meeting last April at the University of Cambridge, and is summarized in a recent article in Oryx. What I came up with for BAS was an essay that cast very broadly the need to understand threats to all of the natural systems we depend on. Quantifying the economic benefit of those systems, and the risk inherent in our dependence upon them, led me directly to the concept of natural security.
Here I want to take a stab at expanding the conversation further. Rapidly rising revenues in the bioeconomy, and the rapidly expanding scope of application, must critically inform an evolving definition of biosecurity. In other words, because economic demand is driving technology proliferation, we must continually refine our understanding of what it is that we must secure and from where threats may arise.
Biosecurity has typically been interpreted as the physical security of individuals, institutions, and the food supply in the context of threats such as toxins and pathogens. These will, of course, continue to be important concerns: new influenza strains constantly emerge to cause human and animal health concerns; the (re?)emergent PEDS virus has killed an astonishing 10% of U.S. pigs this year alone; within the last few weeks there has been an alarming uptick in the number of human cases and deaths caused by MERS. Beyond these natural threats are pathogens created by state and non-state organizations, sometimes in the name of science and preparation for outbreaks, while sometimes escaping containment to cause harm. Yet, however important these events are, they are but pieces of a biosecurity puzzle that is becoming ever more complex.
Due to the large and growing contribution of the bioeconomy, no longer are governments concerned merely with the proverbial white powder produced in a state-sponsored lab, or even in a 'cave' in Afghanistan. Because economic security is now generally included in the definition of national security, the security of crops, drug production facilities, and industrial biotech will constitute an ever more important concern. Moreover, in the U.S., as made clear by the National Strategy for Countering Biological Threats(PDF), the government has established that encouraging the development and use of biological technologies in unconventional environments (i.e., "garages and basements") is central to national security. Consequently, the concept of biosecurity must comprise the entire value chain from academics and garage innovators, through production and use, to, more traditionally, the health of crops, farm animals, and humans. We must endeavor to understand, and to buttress, fragility at every link in this chain.
Beyond the security of specific links in the bioeconomy value chain we must examine the explicit and implicit connections between them, because through our behavior we connect them. We transport organisms around the world; we actively breed plants, animals, and microbes; we create new objects with flaws; we emit waste into the world. It's really not that complicated. However, we often choose to ignore these connections because acknowledging them would require us to respect them, and consequently to behave differently. But that change in behavior must be the future of biosecurity.
From an enterprise perspective, as we rely ever more heavily on biology in our economy, so must we comprehensively define 'biosecurity' to adequately encompass relevant systems. Vulnerabilities in those systems may be introduced intentionally or accidentally. An accidental vulnerability may lie undiscovered for years, as in the case of the recently disclosed Heartbleed hole in the OpenSSL internet security protocol, until it is identified, when it becomes a threat. The risk, even in open source software, is that the vulnerability may be identified by organizations which then exploit it before it becomes widely known. This is reported to be true of the NSA's understanding and exploitation of Heartbleed at least two years in advance of its recent public announcement. Our biosecurity challenge is to carefully, and constantly, assess how the world is changing and address shortcomings as we find them. It will be a transition every bit as painful as the one we are now experiencing for hardware and software security.
(Here is Part 2.)