Postmortem: Zettlr’s first Security Incident
This postmortem has been crossposted to hendrik-erz.de.
On Friday, May 7th, 2021, Zettlr 1.8.8 has been released containing a fix that addresses several CVEs within the Electron framework. There was a security hole that potentially allowed malicious actors to gain access to your computer using a specifically crafted HTML document. One week later, on May 13th, 2021, another hotfix – Zettlr 1.8.9 – has been released, addressing yet another security issue that allowed something similar, but with much less effort. All it required was a maliciously crafted Markdown document containing an insecure iFrame element.
In this postmortem, I will first lay out the timeline of the incident, explain what went wrong and why it went wrong, and close by indicating what steps I will take in the future in order to minimise this problem. To be upfront with you: The second hotfix could’ve been avoided, so I take full responsibility for simply being negligent in this regard. But let us tackle the incident one step at a time.
Was I affected? It is very likely that you were not affected by this security issue. If you disabled the “Render iFrame” setting in the display preferences, you were definitely not affected. Otherwise, ask yourself these questions: (a) Did I open a Markdown document that I received from someone else? (b) Did this document contain an iFrame-element, e.g., an embedded video? (c) Did I copy a video-embed code (or the embed-code for something else) from a lesser known site (not, e.g., the YouTube or Vimeo share button)? (d) Did any of these iFrames contain the
srcdoc-attribute? If you can answer any of these questions with “Yes”, it might be that you have been affected. In this case, immediately disable the option “Render iFrames” in your display preferences, then open the Markdown documents in question and double-check the code. As soon as you upgrade to Zettlr 1.8.9, you can safely re-enable the “Render iFrames” preference.
Timeline: What Went Wrong?
In Q2 2021, the CVE-number (Common Vulnerabilities and Exposures) CVE-2021-21222 was being assigned to publicly announce a security hole in the Chromium browser. This security hole was located in the so-called V8-engine. The V8-engine is the piece of Chromium that enables JavaScript-support for the browser. The issue was due to a fairly common bug in software (a heap buffer overflow) and allowed an attacker to circumvent the security measures of the browser that are supposed to make sure that no website can execute malicious code to access your computer. This security measure is called sandboxing and prevents code from some website to interact with your computer in any way.
In short succession, a total number of eight CVE-numbers were being publicly released, all concerning the same issue:
- CVE-2021-21222
- CVE-2021-21223
- CVE-2021-21225
- CVE-2021-21226
- CVE-2021-21227
- CVE-2021-21230
- CVE-2021-21231
- CVE-2021-21233
Following the public release of this issue, all affected developers began fixing their Chromium version. In this regard, it is necessary to state the importance of bugs in Chromium. While you may not have heard of Chromium before, its importance to the software industry cannot be overstated: Chromium is an Open Source browser. Chromium is being used by Google to build their Chrome browser. Google Chrome is – in simple terms – the Chromium browser with custom elements (such as settings synchronisation and Google account management), wrapped and packaged to be released as Google Chrome.
Chromium is also being used as an engine to render web-content across many products. For example, the Electron framework takes the Chromium browser in order to allow Electron apps to use web-content to display their user interface. Thus, the Electron framework was also affected by the above mentioned CVEs. And, since Zettlr is built on top of the Electron framework, Zettlr was also affected. The new Electron version 12.0.6, which fixed the above CVEs was released in the first week of May and I immediately updated the Electron version of Zettlr’s development branch to that one and knew that I need to update the current release of Zettlr 1.8 as well. That was on Thursday, May 6th, 2021.
However, the whole situation is not as simple as that. On that same day, I received an email that alerted me that something else was going on. The email came from the Japanese cybersecurity organisation JPCERT, one of the major agencies that monitors the security of software and works closely with the Japanese government in maintaining the safety of publicly available software products. A Japanese security researcher has found a security issue in Zettlr and has reported this to JPCERT/CC (JPCERT’s Coordination Center), which in turn contacted me for a disclosure.
The report containing a detailed description of the security hole in Zettlr was made available to me on May 10th, four days after the initial contact. Upon looking at it, I realised it was what I feared it was: A security issue that allowed malicious actors to gain access to your computer via a crafted Markdown document. To make the matter more embarrassing, a corresponding fix was already in Zettlr’s code base, specifically at commit 53b544b, which was already 22 days old at the time of this writing. That commit was in response to issue #1716, opened in early February 2021. You might already begin to fathom what the problem here is.
Nevertheless, on May 13th, 2021, I backported[^1] 53b544b to Zettlr 1.8 as commit 15a8d60 and released the second hotfix within the span of a single week. Zettlr can since be regarded as safe again.
Analysis: Why did it go wrong?
However, the main use of articles like this – so-called Postmortems – is to analyse how it could come to the security issue and identify what can be done better. As Pete Cheslock explains:
A post-mortem is held after an incident has taken place (in this case, a security breach of some type). The security team sits down with the rest of the organization (or the affected team) and talks through what happened, identifies causes, lessons learned, and how to move forward.
We already did the “what” — now it is time to identify causes and lessons learned. And to that belongs a shameless analysis of what I personally did wrong. The problem were not the CVEs I mentioned above: I always keep Electron updated and I did so shortly after 12.0.6 was released. However, I did not initially plan on backporting that fix to Zettlr 1.8.
In a similar vein, while I already knew about the problem regarding iFrame rendering in Zettlr since early February 2021, I did not prioritise that issue until authority came crashing down on me, and I had to acknowledge that one of the central security organisations has unveiled a severe hole in my app. When I received the email from JPCERT/CC, I was frightened something bad could happen, and started to evaluate what the cause was (at this moment, all I knew was that there was a security issue, but not which one exactly). I looked at the release notes of Electron 12.0.6 and found the above mentioned row of CVE numbers, indicating there was something bad happening. So I assumed that this might’ve been the cause for the email I had received. After all, it made sense to me that, if there is a severe bug in one of the core frameworks Zettlr makes use of, it would make sense that security researchers would reach out to all affected developers and notify them to update immediately.
Still being in shock, I quickly updated Zettlr 1.8’s Electron version and released a hotfix on Friday, May 7th, 2021 and urged users to update. But that might’ve been too quick. Thinking about the CVEs, it is pretty unlikely that anyone could gain access to your computer using these CVEs, because they require loading a rogue HTML document. However, Zettlr is pretty much locked down, and you cannot load any HTML file except the one containing the graphical user interface. So I already suspected that there was something else incoming.
And it did. When I received the full security report and read through it, it became obvious that someone obviously was fed up with me ignoring issue number 1716 that already pretty plainly told me: “Dude, there’s something seriously wrong, and you need to fix that!” However, I chose not to fix the bug just then. What are the reasons for this? I can think of the following:
- For starters, there are few pages that offer iFrames, since you only need iFrames for very specific purposes – to embed videos, for example.
- Furthermore, I suspected that people would be smart enough not to add an iFrame of some weird rogue page that didn’t seem trustworthy. The rationale behind this was: “I wouldn’t fall for that, so nobody else would” – which is a big problem.
- Also, I regarded Zettlr 1.8 more or less as “finished” and wanted to focus my energy on creating Zettlr 2.0, which is due to my 40hrs full-time PhD job, which is my main concern
As you can see, it is a mixture of negligence and time constraints. While I cannot alter the time constraints, the negligence is fully on me. So what did I learn?
Lessons Learned: How it Could’ve Been Avoided
I already knew that issue 1716 was serious, and indeed could harm users. However, I assumed the issue could only occur during the writing of a document. I did not consider the case when a user gets sent a Markdown document and opens it without prior inspecting it in another editor. Furthermore, since iFrame rendering was enabled by default, a secure Zettlr installation required users to change seemingly unrelated settings.
It is important that an app – in its default state with no settings altered – is secure. Potential security issues should only open up when certain settings are activated by a user on purpose, similar to setting the “executable” flag on Bash script files. Zettlr did not fulfil this requirement.
To add insult to injury: I knew about this issue even before February 2021. When writing Zettlr, I sometimes look at the source code of other programs to get a sense of how they have solved certain engineering problems I face. Sometimes, I just look at other people’s code out of interest. I did so in 2020 or even 2019, if I remember correctly, and looked at the source code of Abricotine, yet another Markdown editor I have used before starting to create Zettlr. Abricotine also enables users to preview certain elements like images or iFrames. Thomas Brouard, the creator of Abricotine, was already aware of the security implications of iFrames and implemented a whitelisting approach: If the hostname of the source of an iFrame element was not in the whitelist, Abricotine did not render that iFrame. Users thus had to explicitly consent to viewing iFrames from certain websites.
So I cannot state that I didn’t know about those security implications. However, I still didn’t implement this. As to why, I have no excuse. I could’ve done it, but I didn’t. Up until today, Zettlr would simply take the full iFrame element (including problematic attributes such as srcdoc) and rendered it as-is, thereby ripping a massive hole into the security of Zettlr.
In the end of April 2021 I finally took the time to fix issue 1716, but at that point, the security hole in Zettlr was already more than three years old, and it is pure luck that nothing bad happened during that time.
As of now, Zettlr takes two crucial precautions in order to prevent users from harm. First, it will by default only render iFrames from YouTube and Vimeo. Any other video-streaming site (or other sites that allow you to embed things) must be whitelisted manually. Instead of rendering the iFrame’s contents, Zettlr will now first render a security warning and give you two options: You can either render an iFrame once, or you can render it and additionally add the hostname of the iFrame to the whitelist so that in the future, iFrames from that websites will be rendered automatically.
However, this alone wouldn’t fix the security issue raised by JPCERT/CC. For that, a second measurement was implemented: From now on, Zettlr will extract the src-attribute from the iFrame you pasted and create a new iFrame element that only contains that source. Other, potentially dangerous attributes such as srcdoc will be omitted and not be part of the rendering. So as long as you trust the hostname, even if someone wants to take over your computer by providing a valid source from YouTube while hiding malicious code in other attributes, this will not work anymore.
Wrapping up: Future Actions
What can be done in the future to avoid this security cataclysm? There are several steps that I can and will take.
First, I am going to set up a security policy on the Zettlr repository so that newly found critical security issues will not be opened on the issue tracker. The reason for this is that if you open an issue, that will be publicly visible, and that means that malicious actors can also look at that issue and immediately begin exploiting the issue. If the details of this issue are only shared with the maintainers and thus remain hidden from the public view, maintainers have time to implement a security fix and only then make it public. You never hear about security issues before a corresponding fix has been released. And this is also why I did not publicly talk about this issue before having released Zettlr 1.8.9 only a few minutes ago. Only after the fix has been released is it safe to publicly disclose the problem.
Second, I will in the future more carefully monitor potentially security-relevant issues on the tracker, possibly even deleting them quickly in order to make sure those will not be publicly known before I had the time to implement a fix. You can also help me with this: If you see that an issue is security-relevant or if you discover one such thing yourself, please immediately notify me via mail or Twitter DM so I can fix the issue asap! In this regard, even if something is not, in fact, security relevant, still inform me. It’s better for me to see that an issue is not a security-issue than having an actual bug in the app because you weren’t sure – better safe than sorry.
Third, I have set up a legacy branch for Zettlr 1.x, and I will keep this practice in the future. So whenever a new version is released, I will keep a legacy branch of the latest released Zettlr version to quickly implement security fixes if need be. Note that this still doesn’t mean that older versions will receive new features or enhancements, since I still don’t have the time to maintain two versions of Zettlr at the same time. But security fixes must be published asap, even if I’m working on some shiny new 2.0-version.
Lastly, let me use this article to reiterate the need for Open Source projects to receive support by volunteers. And I don’t mean financial support, I mean plain and simple coding support. Every security issue that is being addressed by someone in a PR is one less I have to take care of. And since I have a full-time job, I am not flexible enough to start fixing a security issue right away. This week it was possible because I have a little less work and above that a public holiday, but I still spent about five hours implementing the bug and addressing the issues mentioned above.
Conclusion
For the past three years I have betrayed your trust – your trust that I keep you, the users, safe from harm. I would see unsafe iFrames and know about the implications, but this is not a valid reason not to fix security breaches. Specifically, it can even be seen as arrogance on my side because I assumed Zettlr’s users would have the same amount of technical knowledge as I do. As a maintainer of a product that is being used by tens of thousands of users around the world, I now have a responsibility towards these users – whether I like it or not. And I must live up to this fact. I cannot hide behind the “No warranty”-statement of the GNU GPL v3-license I use to publish Zettlr. Because yes, legally I am not liable if anything bad does happen.
However, society doesn’t work like this. Society builds on trust; trust that doesn’t care about the legal statutes. Even though I’m legally off the hook, I’m not morally. And that is something important, especially given that I do a PhD on these issues. A PhD! Everyday I read texts about how society builds on trust, mutual understanding, and communication, and it literally takes me ten seconds of search in my reading notes to find multiple explanations for how trust works. It would certainly be dishonest for me if I didn’t live up to this knowledge.
So, let me finish by saying sorry. Sorry that I did not fix this issue earlier. I knew about this security issue, and potential malicious actors did so too for at least three months, and I still didn’t fix it. I let you down. So, please accept my apologies, and I vow to never let something like this happen ever again.
But now, enjoy the rest of the day, and have a great weekend.
[^1]: "Backporting" refers to the practice of adding a bugfix, a security fix, or a new feature to an older version of a software product after it has been added to the latest version.