August 11, 2014
Robert O'Callahan -- Milestones On The Road To Christianity
Around the age of 20 I found myself struggling with some fairly deep philosophical questions. The most important was this: assuming (as I did) naturalism is true, then what should I do?
It seemed clear to me then (and still does) that if naturalism is true, the is-ought problem is insurmountable. There can be no objective moral truths or goals. The best we can do is identify commonly held moral values and pursue them. Unfortunately --- if honesty is one of those values --- we cannot tell others that their behavior is, in any objective sense, wrong. For example, we observe that Hitler's moral opinions are different from ours, but we could not claim that our moral opinions are intrinsically more valid. All we could do is wage war against him and hope our side prevails. Might makes right.
That doesn't make naturalism incoherent, but it opens a chasm between what naturalists can really believe about moral statements and the way almost everyone uses them in practice. The more die-hard naturalists are prone to say things like "naturalism is true, and therefore everyone should ... (stop believing in God, etc)" without respecting the limitation that the consequent ought-statements are subjective opinions, not objectively rational facts. It's really very difficult to be a proper moral relativist through-and-through!
Making this all much more difficult was my awareness of being able to reshape my own moral opinions. The evolutionary-psychology approach of "these are the values imbued by my primate brain; work them out" seems totally inadequate when the rational part of my brain can give priority to any subset of values (or none) and use that as justification for rewriting the others. Given a real choice between being a hero and a monster, on what grounds can one make that decision? It seemed a bit narrow-minded to reject monstrosity simply because it was less popular.
This all made me very dissatisfied with naturalism as a worldview. If it's true, but is powerless to say how one should live --- indeed, denies that there can be any definitive guidance how to live --- it's inadequate. Like a scientific theory that lacks predictive power, whether it's true or not, one has to keep looking for more.
(OK, I was a weird kid, but everyone thinks about this, right?)
August 11, 2014 10:16 AM
August 08, 2014
Robert O'Callahan -- cf1e5386ecde9c2eb9416c9b07416686
August 08, 2014 05:39 AM
Choose Firefox Now, Or Later You Won't Get A Choice
I know it's not the greatest marketing pitch, but it's the truth.
Google is bent on establishing platform domination unlike anything we've ever seen, even from late-1990s Microsoft. Google controls Android, which is winning; Chrome, which is winning; and key Web properties in Search, Youtube, Gmail and Docs, which are all winning. The potential for lock-in is vast and they're already exploiting it, for example by restricting certain Google Docs features (e.g. offline support) to Chrome users, and by writing contracts with Android OEMs forcing them to make Chrome the default browser. Other bad things are happening that I can't even talk about. Individual people and groups want to do the right thing but the corporation routes around them. (E.g. PNaCl and Chromecast avoided Blink's Web standards commitments by declaring themselves not part of Blink.) If Google achieves a state where the Internet is really only accessible through Chrome (or Android apps), that situation will be very difficult to escape from, and it will give Google more power than any company has ever had.
Microsoft and Apple will try to stop Google but even if they were to succeed, their goal is only to replace one victor with another.
So if you want an Internet --- which means, in many ways, a world --- that isn't controlled by Google, you must stop using Chrome now and encourage others to do the same. If you don't, and Google wins, then in years to come you'll wish you had a choice and have only yourself to blame for spurning it now.
Of course, Firefox is the best alternative :-). We have a good browser, and lots of dedicated and brilliant people improving it. Unlike Apple and Microsoft, Mozilla is totally committed to the standards-based Web platform as a long-term strategy against lock-in. And one thing I can say for certain is that of all the contenders, Mozilla is least likely to establish world domination :-).
August 08, 2014 01:33 AM
July 16, 2014
Robert O'Callahan -- Multiverses And Anthropic Reasoning
I liked this article summarizing the current state of science regarding multiverse theories. It's very clear and well-illustrated, and, as far as I know, accurate.
This quote is particularly interesting:
So as appealing as the idea is that there are other Level 1 Multiverses out there with different constants than our own, we have good physical reasons based on observable evidence to think it’s unlikely, and zero good reasons (because wanting it to be so is not a good reason) to think it’s likely.
He doesn't mention why anyone would "want it to be so", i.e. believe that other universes of a "Level 1 Multiverse" could have different constants to our own. However, I'm pretty sure he had in mind the selection-bias explanation for the anthropic coincidences. That is, if we accept that only a narrow range of possible values for the fundamental physical constants are compatible with the existence of intelligent life (and most scientists do, I think), then we would like to be able to explain why our universe's constants are in that range. If there are an abundance of different universes, each with different values for the physical constants, then most of them would be dead but a lucky few would sustain intelligent life, and naturally we can only observe one of those latter.
This reasoning relies on the assumption that there are abundance of different universes with different values for the physical constants. Scientists obviously would prefer to be able to deduce this from observations rather than pull it out of thin air. As discussed in the above article, theories of chaotic inflation --- which are reasonably well-grounded in observations of our own universe --- predict the existence of alternate universes. If those universes could have different values for physical constants (or even different physical laws), we'd have an observationally-grounded theory that predicts exactly the kind of multiverse needed to power the selection-bias explanation for the anthropic coincidences. Unfortunately for proponents of that explanation, the science isn't working out.
Of course, the selection-bias explanation could still be valid, either because new information shows that chaotic-inflation universes can get different constants after all, or because we assume another level of multiverse, whose existence is not due to chaotic inflation. However, many scientists (such as in the article above) find the assumption of a higher-level multiverse quite unsatisfactory.
Unsurprisingly, I'm comfortable with the explanation that our universe was intentionally created for intelligent life to live in. Incidentally, you don't need to be a classical theist to adopt this explanation; some atheist philosophers argue with varying degrees of seriousness that we are (probably) living in a simulation.
July 16, 2014 01:29 PM
July 02, 2014
Robert O'Callahan -- Implementing Scroll Animations Using Web Animations
It's fashionable for apps to perform fancy animations during scrolling. Some examples:
- Parallax scrolling
- Sticky headers that scroll normally until they would scroll out of view and then stop moving
- Panels that slide laterally as you scroll vertically
- Elements that shrink as their available space decreases instead of scrolling out of view
- Scrollable panels that resist scrolling as you get near the end
There are a few possible approaches to fixing this. One is to not provide any new API, hope that skilled developers can avoid blowing the latency budget, and carefully engineer the browser to minimize its overhead. We took this approach to implementing homescreen panning in FirefoxOS. This approach sounds fragile to me. We could make it less fragile by changing event dispatch policy during a touch-drag, e.g. to suppress the firing of "non-essential" event handlers such as setTimeouts, but that would add platform complexity and possibly create compatibility issues.
Another approach would be to move scroll animation calculations to a Worker script, per an old high-level proposal from Google (which AFAIK they are not currently pursuing). This would be more robust than main-thread calculations. It would probably be a bit clumsy.
Another suggestion is to leverage the Web's existing and proposed animation support. Basically we would allow an animation on an element to be use another element's scroll position instead of time as the input to the animation function. Tab Atkins proposed this with declarative CSS syntax a while ago, though it now seems to make more sense as part of Web Animations. This approach is appealing because this animation data can be processed off the main thread, so these animations can happen at 60Hz regardless of what the main thread is doing. It's also very flexible; versions of all of the above examples can be implemented using it.
One important question is how much of the problem space is covered by the Web Animations approach. There are two sub-issues:
- What scrolling effects depend on more than just the scroll position, e.g. scroll velocity? (There certainly are some, such as headers that appear when you scroll down but disappear when you scroll up.)
- For effects that depend on just the scroll position, which ones can't be expressed just by animating CSS transforms and/or opacity as a function of the scroll position?
July 02, 2014 09:10 PM
May 25, 2014
Robert O'Callahan -- Unnecessary Dichotomy
As the scope of the open Web has expanded we've run into many hard issues such as DRM, support for patented video codecs, and applications needing APIs that increase fingerprintability. These issues are easily but incorrectly framed as choices between "principles" and "pragmatism" --- the former prioritizing Mozilla's mission, the latter prioritizing other things such as developer friendliness for the Web platform and Firefox market share. This framing is incorrect because the latter are essential components of our strategy for pursuing our mission.
For example I believe the optimal way to pursue our goal of unencumbered video codecs is neither to completely rely on platform video codecs (achieving nothing for our goal) nor to refuse to support all patent-encumbered codecs (in the current market, pushing almost all users and developers to avoid Firefox for video playback). Instead it is somewhere in between --- hopefully somewhere close to our current strategy of supporting H.264 in various ways while we support VP8/VP9 and develop an even better free codec, Daala. If we deliberately took a stance on video that made us irrelevant, then we would fail to make progress towards our goal and we would have actually betrayed our mission rather than served it.
Therefore I do not feel any need to apologize for our positions on encumbered codecs, DRM and the like. The positions we have taken are our best estimate of the optimal strategy for pursuing our mission. Our strategy will probably turn out to be suboptimal in some way, because this is a very difficult optimization problem, requiring knowledge of the future ... but we don't need to apologize for being unomniscient either.
A related problem is that our detractors tend to view our stance on any given issue in isolation, whereas we really face a global optimization problem that spans a huge range of issues. For example, when developers turn away from the Web platform for any reason, the Web as a whole is diminished, and likewise when users turn away from Firefox for any reason our influence of most issues is diminished. So the negative impact of taking a "hyper-principled" stand on, say, patent-encumbered video codecs would be felt across many other issues Mozilla is working on.
Having said all that, we need to remain vigilant against corruption and lazy thinking, so we need keep our minds open to people who complain we're veering too much in one direction or another. In particular I hope we continue to recruit contributors into our community who disagree with some of the things we do, because I find it much easier to give credence to contributors than to bystanders.
May 25, 2014 11:29 PM
May 23, 2014
Robert O'Callahan -- Against The "Internet Of Things"
I was lucky to be at the Berkeley "programming systems" retreat in Santa Cruz a couple of weeks ago. One topic that came up was programming in the world of the "Internet of Things" --- the anticipated future where everyone has dozens of very small, very low-power devices with CPUs, sensors and radios. There's certainly a lot of interesting work to be done in that area, but nobody seems to have asked the question "do we really want an Internet of Things?" Because I don't, not until we've addressed the pervasive problems we already have with security, privacy, and general untrustworthiness of our computing infrastructure. It's obvious that these buy-and-forget devices won't get timely security updates (if updates are even adequate for security), so governments and criminal organizations will be battling for control over them, out of sight and mind of their nominal owners. So we're talking about making the surveillance network of the NSA (and even worse actors) a lot more pervasive. These devices will be able to run for years without a battery change, so when you buy a new house or car, you will inherit a miasma of unseen machines doing everyone's bidding but yours.
There's going to be a big market for scanning and cleaning tools. A small handheld EMP unit would be great. Someone should get on this problem.
May 23, 2014 05:04 AM
May 04, 2014
Robert O'Callahan -- Milford Track
I took last week off to walk the Milford Track with some family members (plus a few days in Queenstown). The Milford Track is the most famous multi-day "Great Walk" in New Zealand and you have to book months in advance to do it during the regular season. It deserves every bit of its hyping as the finest walk in the world.
This three-night, four-day walk can be done "guided" or "independently". You pay a lot of money for the "guided" version and in return you stay in lodges with clean sheets, hot showers, and catered meals. For a lot less money the "independent" version has you staying at fairly standard (but slightly fancier than average) Department of Conservation huts, which means no showers and you carry your own food and bedding. We did the independent version. With both versions you must walk in the westward direction and stop for three nights in the designated hut/lodge each night. Each DOC hut has a resident ranger, and they're all brilliant to talk to.
You start by taking a boat to the north end of Lake Te Anau, then walking for just over an hour to the first hut. It's such a short distance you can carry whatever luxuries you can consume on that first night --- sausages, bacon and eggs were popular items amongst our fellow trampers. Then you work your way up to the end of the Clinton Valley, cross Mckinnon pass on the third day, and walk down Arthur Valley to Milford Sound to be transported by boat to the village of Milford itself.
The scenery is, needless to say, astonishing --- mountains, lakes, trees, rivers, waterfalls, and all that. The environment is pristine; every stream and river is crystal clear where not colored by beech tannins, and all are perfectly drinkable. There's fascinating wildlife --- trout, keas, wekas, black robins. The weather is highly variable --- sunshine, rain and snow are all quite common and we had them all in our four days (very little snow though). Having rain near the beginning of the walk, like we did, is excellent because it powers up all the streamlets and waterfalls.
One curious fact is that the Mckinnon Pass area was first explored by Europeans specifically to find a route for tourists to cross overland to Milford. Almost immediately after the pass was first crossed, tourists started using the Milford Track, often in rather appalling conditions by today's standards. For many years there was no road out of Milford so track walkers arriving in Milford generally had to walk straight back out again.
May 04, 2014 11:58 AM
April 09, 2014
Robert O'Callahan -- Getting Back To Work
... is what we need now. So let me give a brief summary of what's happening with me work-wise.
Last year I fully divested my direct reports. No more management work to do, yay! I think they probably all ended up with a better manager than they had.
I've been involved in a lot of different projects and a lot of helping other people get their work done. In between I managed to carve out a few things of my own:
- I built reftests for async panning to try to stem the tide of regressions we were encountering there. Unfortunately that's not quite done because most of those tests aren't running on TBPL yet.
- I worked on CSS scroll snapping with an intern. Unfortunately the spec situation got bogged down; there's an impasse between us and Microsoft over the design of the CSS feature, and Google has decided not to do a CSS feature at all for now and try to do it with JS instead. I'm skeptical that will work will, and looking forward to their proposal, but it's taking a while.
- I landed an implementation of the CSS OM GeometryUtils interface, described in hacks.mozilla.org blog posts here and here. This fixes a functionality gap in the Web platform and was needed by our devtools team. Almost everything you would need to know about the CSS box geometry of your page is now easy to get.
- Lately I've been doing work on rr. People are trying to use it, and they've been uncovering bugs and inconveniences that Chris Jones and I have been fixing as fast as we can. Terrence Cole used it successfully to help fix a JS engine bug! I'm using it a lot myself for general-purpose debugging, and enjoying it. I want to spend a few more days getting some of the obvious rough edges we've found filed off and then make a new release.
Looking forward, I expect to be working on making our async scrolling fully generic (right now there are some edge cases where we can't do it), and working on some improvements to our MediaStreamGraph code for lower latency video and audio.
April 09, 2014 11:54 AM
Fighting Media Narratives
- Almost all Mozilla staff supported keeping Brendan Eich as CEO, including many prominent LGBT staff, and many made public statements to that effect. A small number of Tweeters calling for him to step down got all the media attention. The narrative that Mozilla staff as a group "turned against Brendan" is false. It should go without saying, but most Mozilla staff, especially me, are very upset that he's gone. I've known him, worked with him and fought alongside him (and sometimes against him :-) ) for fourteen years and having him ripped away like this is agonizing.
- The external pressure for Brendan to step down was the primary factor driving the entire situation. The same issue exploded in 2012 but there was less pressure and he got through it. No doubt Mozilla could have handled it better but the narrative that blames Mozilla for Brendan's departure misses the big picture. Boycotting Mozilla (or anyone for that matter) for cracking under intense pressure is like shooting a shell-shocked soldier.
- As a Christian, Mozilla is as friendly a workplace as any tech organization I've known --- which is to say, not super friendly, but unproblematic. Because of our geographic spread --- not just of headcount, but of actual power --- and our broad volunteer base I think we have more real diversity than many of our competitors. The narrative that Mozilla as a group has landed on one side of the culture war is false, or at least no more true than for other tech organizations. In fact one thing I've really enjoyed over the last couple of weeks is seeing a diverse set of Mozilla people pull together in adversity and form even closer bonds.
I'll also echo something else a lot of people are saying: we have to fix Internet discourse somehow. It's toxic. I wrote about this a while back, and this episode has made me experience the problem at a whole new level. I'll throw one idea out there: let's communicate using only recorded voice+video messages, no tweets, no text. If you want to listen to what I have to say, you have to watch me say it, and hopefully that will trigger some flickers of empathy. If you want me to listen to you, you have to show me your face. Want to be anonymous, do it the old-fashioned way and wear a mask. Yeah I know we'd have to figure out searchability, skimmability, editing, etc etc. Someone get to work on it.
April 09, 2014 11:31 AM
How much does the world need Mozilla? A useful, if uncomfortable, thought experiment is to consider what the world would be like without Mozilla.
Consider the world of Web standards. Microsoft doesn't contribute much to developing new Web features, and neither does Apple these days. Mozilla and Google do. Google, per Blink's own policy (mirroring our own), relies on feedback and implementation by other browser vendors, i.e. usually us. If you take Mozilla out of the equation, it's going to be awfully hard to apply the "two independent implementations" test for new Web features. (Especially since Webkit and Blink still have so much shared heritage.) For example it's hard to see how important stuff like Web Audio, WebGL and WebRTC would have become true multi-vendor standards without us. Without us, most progress would depend on unilateral extensions by individual vendors. That has all the downsides of a single-implementation ecosystem --- a single implementation's bugs become the de-facto standard; the standards process, if there even is one, becomes irrelevant; and even more power accrues to the dominant vendor.
In the bigger picture, it would be dangerous to leave the Web --- and the entire application platform space --- in the hands of three very large US corporations who have few scruples and who each have substantial non-Web interests to protect, including proprietary desktop and mobile platforms. It has always been very important that there be a compelling vendor-neutral platform for people to deploy content and apps on, a platform without gatekeepers and without taxes. The Web is that platform ---- for now. Mozilla is dedicated to preserving and enhancing that, but the other vendors are not.
Mozilla has plenty of faults, and lots of challenges, but our mission is as important as ever ... probably more important, given how computing devices and the Internet penetrate more and more of our lives. More than ever, pursuing our mission is the greatest good I can do with the talents God has given me.
April 09, 2014 10:34 AM
April 06, 2014
Robert O'Callahan -- Responsible Self-Censorship
People may be wondering why I, as one of the most notorious Christians at Mozilla, have been silent during the turmoil of recent days. It is simply because I haven't been able to think of anything to say that won't cause more harm in the current environment. It is not because I am afraid, malicious, or suffering from a lack of freedom of speech in any way. As soon as I have the environment and the words to say something helpful, I will.
By "the current environment" I mean the situation where many of the culture warriors of all stripes are picking over every utterance looking for something to be angry against or something to fuel their anger against others.
Update Actually, right after posting that, I thought of something to say.
I have never loved the people of Mozilla as much as I do right now. With just a few exceptions, your commitment and grace under fire is inspiring and (in the old-fashioned sense) awesome. I'm glad I'm here for this.
April 06, 2014 09:27 PM
March 26, 2014
Robert O'Callahan -- Conflict
I was standing in a long line of people waiting to get their passports checked before departing Auckland to the USA. A man pushed into the line ahead of me. People around me sighed and muttered, but no-one did anything. This triggered my "if no-one else is willing to act, it's up to me" instinct, so I started a conversation:
Roc: Shouldn't you go to the back of the line?
Man: I'm in a hurry to get to my flight.
There are only two flights to the USA in the immediate future and AFAIK neither of them is imminent.
Roc: Which flight are you on?
Man: What's the big deal? I'm hardly going to slow you down.
Roc: But it's rude.
Man: So what do you want me to do?
Roc: I want you to move to the back of the line.
Man: You're going to have to make me.
At that point I couldn't think of anything to say or do so I let it go. It turned out that he was on my flight and it would have been unpleasant if he'd ended up sitting next to me.
I was a bit shocked by this incident, though I probably shouldn't have been. This kind of unapologetic rudeness is not something I encounter very often from strangers in face-to-face situations. I guess that means I'm fortunate. Surprisingly I felt quite calm during the conversation and only felt rage later.
Embarrassingly-much-later, it dawned on me that Jesus wants me to forgive this man. I am now making the effort to do so.
March 26, 2014 09:46 PM
Bugs that reproduce intermittently are hard to debug with traditional techniques because single stepping, setting breakpoints, inspecting program state, etc, is all a waste of time if the program execution you're debugging ends up not even exhibiting the bug. Even when you can reproduce a bug consistently, important information such as the addresses of suspect objects is unpredictable from run to run. Given that software developers like me spend the majority of their time finding and fixing bugs (either in new code or existing code), nondeterminism has a major impact on my productivity.
Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.
Here are some of our key design parameters:
- We focused on debugging Firefox. Firefox is a complex application, so if rr is useful for debugging Firefox, it is very likely to be generally useful. But, for example, we have not concerned ourselves with record and replay of hostile binaries, or highly parallel applications. Nor have we explored novel techniques just for the sake of novelty.
- We prioritized deployability. For example, we deliberately avoided modifying the OS kernel and even worked hard to eliminate the need for system configuration changes. Of course we ensured rr works on stock hardware.
- We placed high importance on low run-time overhead. We want rr to be the tool of first resort for debugging. That means you need to start getting results with rr about as quickly as you would if you were using a traditional debugger. (This is where my previous project Chronomancer fell down.)
- We tried to take a path that would lead to a quick positive payoff with a small development investment. A large speculative project in this area would fall outside Mozilla's resources and mission.
- Naturally, the tool has to be free software.
I believe we've achieved those goals with our 1.0 release.
There's a lot to say about how rr works, and I'll probably write some followup blog posts about that. In this post I focus on what rr can do.
rr records and replays a Linux process tree, i.e. an initial process and the threads and subprocesses (indirectly) spawned by that process. The replay is exact in the sense that the contents of memory and registers during replay are identical to their values during recording. rr provides a gdbserver interface allowing gdb to debug the state of replayed processes.
Here are performance results for some Firefox testsuite workloads: These represent the ratio of wall-clock run-time of rr recording and replay over the wall-clock run-time of normal execution (except for Octane, where it's the ratio of the normal execution's Octane score over the record/replay Octane scores ... which, of course, are the same). It turns out that reftests are slow under rr because Gecko's current default Linux configuration draws with X11, hence the rendered pixmap of every test and reference page has to be slurped back from the X server over the X socket for comparison, and rr has to record all of that data. So I also show overhead for reftests with Xrender disabled, which causes Gecko to draw locally and avoid the problem.
I should also point out that we stopped focusing on rr performance a while ago because we felt it was already good enough, not because we couldn't make it any faster. It can probably be improved further without much work.
Debugging with rr
The rr project landing page has a screencast illustrating the rr debugging experience. rr lets you use gdb to debug during replay. It's difficult to communicate the feeling of debugging with rr, but if you've ever done something wrong during debugging (e.g. stepped too far) and had that "crap! Now I have to start all over again" sinking feeling --- rr does away with that. Everything you already learned about the execution --- e.g. the addresses of the objects that matter --- remains valid when you start the next replay. Your understanding of the execution increases monotonically.
rr has limitations. Some are inherent to its design, and some are fixable with more work.
- rr emulates a single-core machine. So, parallel programs incur the slowdown of running on a single core. This is an inherent feature of the design. Practical low-overhead recording in a multicore setting requires hardware support; we hope that if rr becomes popular, it will motivate hardware vendors to add such support.
- rr cannot record processes that share memory with processes outside the recording tree. This is an inherent feature of the design. rr automatically disables features such as X11 shared memory for recorded processes to avoid this problem.
- For the same reason, rr tracees cannot use direct-rendering user-space GL drivers. To debug GL code under rr we'll need to find or create a proxy driver that doesn't share memory with the kernel (something like GLX).
- rr requires a reasonably modern x86 CPU. It depends on certain performance counter features that are not available in older CPUs, or in ARM at all currently. rr works on Intel Ivy Bridge and Sandy Bridge microarchitectures. It doesn't currently work on Haswell and we're investigating how to fix that.
- rr currently only supports x86 32-bit processes. (Porting to x86-64 should be straightforward but it's quite a lot of work.)
- rr needs to explicitly support every system call executed by the recorded processes. It already supports a wide range of syscalls --- syscalls used by Firefox --- but running rr on more workloads will likely uncover more syscalls that need to be supported.
- When used with gdb, rr does not provide the ability to call program functions from the debugger, nor does it provide hardware data breakpoints. These issues can be fixed with future work.
We believe rr is already a useful tool. I like to use it myself to debug Gecko bugs; in fact, it's the first "research" tool I've ever built that I like to use myself. If you debug Firefox at the C/C++ level on Linux you should probably try it out. We would love to have feedback --- or better still, contributions!
If you try to debug other large applications with rr, you will probably encounter rr bugs. Therefore we are not yet recommending rr for general-purpose C/C++ debugging. However, if rr interests you, please consider trying it out and reporting/fixing any bugs that you find.
We hope rr is a useful tool in itself, but we also see it as just a first step. rr+gdb is not the ultimate debugging experience (even if gdb's backtracing features get an rr-based backend, which I hope happens!). We have a lot of ideas for making vast improvements to the debugging experience by building on rr. I hope other people find rr useful as a building block for their ideas too.
I'd like to thank the people who've contributed to rr so far: Albert Noll, Nimrod Partush, Thomas Anderegg and Chris Jones --- and to Mozilla Research, especially Andreas Gal, for supporting this project.
March 26, 2014 01:59 AM
March 23, 2014
Robert O'Callahan -- Mozilla And The Silicon Valley Cartel
This article, while overblown (e.g. I don't think "no cold calls" agreements are much of a problem), is an interesting read, especially the court exhibits attached at the end. One interesting fact is that Mozilla does not appear on any of Google's do-not-call lists --- yay us! (I can confirm first-hand that Mozilla people have been relentlessly cold-emailed by Google over the years. They stopped contacting me after I told them not to even bother trying until they open a development office in New Zealand.)
Exhibit 1871 is also very interesting since it appears the trigger that caused Steve Jobs to fly off the handle (which in turn led to the collusion at the heart of the court case) was Ben Goodger referring members of the Safari team for recruitment by Google, at least one of whom was formerly from Mozilla.
March 23, 2014 10:42 PM
March 16, 2014
Robert O'Callahan -- Taroko National Park
This weekend, in between the media and layout work weeks, I had time off so I thought I'd get out of Taipei and see a different part of Taiwan. On Alfredo's recommendation I decided to visit Taroko National Park and do some hiking. I stayed at Taroko Lodge and had a great time --- Rihang Su was an excellent host, very friendly and helpful.
On Saturday I took the train from Songshan station in Taipei to Xincheng. From there I went via the lodge to the park's eastern headquarters and did a short afternoon hike up to near Dali village and back down again. This was short in distance --- about 3.5 km each way in two and half hours --- but reasonably hard work since it's about a 900m vertical climb.
On Saturday night Rihang took the lodge guests to Hualien city for dinner at a hotpot buffet place. The food was reasonable and it was a fun dinner. There were instructions in English and in general I was surprised by how much English signage there was in Hualien (including at the supermarket) --- I had expected none.
On Sunday (today) I got up fairly early for the main goal of my trip: Zhuilu Old Trail. Rihang dropped me off at the western end around 8:15am and I walked the 10km length in a bit under four hours. You're supposed to allow for six, but that's probably a bit conservative, plus I'm fairly fast. It is hard work and tricky in places, however. This trail is quite amazing. There's one stretch in particular where the path winds along the cliff face, mostly about a meter wide, and there's nothing between you and a sheer drop of hundreds of meters --- except a plastic-clad steel cable on the cliff side of the track to hold onto. It reminded me of the National Pass track in Australia's Blue Mountains, but more intense. Much of the track has spectacular views of the Taroko Gorge --- sheer cliffs, marble boulders, lush forest, and mountain streams. In one place I was lucky enough to see macaque monkeys, which was thrilling since I've never seen monkeys in the wild before. The trail has an interesting history; it was originally constructed by the Japanese military in the early 20th century (probably using slave labour, though it wasn't clear from the signage), to improve their control over the native people of the area. Later the Japanese turned it into a tourist destination and that's what it is today. Along the trail there are rest areas where fortified "police stations" used to be.
After finishing that trail, since I was ahead of schedule I continued with the Swallow Grotto walk along the road westward up the gorge. This walk was also amazing but offers quite different views since you're near the bottom of the gorge --- and it's much easier. The walk ends at the bottom of the Zhuilu cliff, which is neck-craningly massive. After that I got picked up and went to the train station to head straight back to Taipei.
This was definitely a very good way to see a bit of Taiwan that's not Taipei. Shuttling around between the train station, the lodge, and hiking destinations, I got a glimpse of what life is like in the region --- gardens, farming, stray dogs, decrepit buildings, industry, narrow roads, etc. But the natural beauty of Taroko gorge was definitely the highlight of the trip. I enjoy hiking with friends and family very much, but it's nice to hike alone for a change; I can go at my own pace, not worry about anyone else, and be a bit more relaxed. All in all it was a great weekend getaway.
March 16, 2014 04:41 PM
I've been in Taiwan for a week. Last Sunday, almost immediately after checking into the hotel I went with a number of Mozilla people to Maokong Gondala and did a short hike around Maokong itself, to Yinhe Cave and back. I really enjoyed this trip. The gondala ride is quite long, and pretty. Maokong itself is devoted to the cultivation and vending of tea. I haven't seen a real tea plantation before, and I also got to see rice paddies up close, so this was quite interesting --- I'm fascinated by agricultural culture, which dominated so many people's lives for such a long time. Yinhe Cave is a cave behind a waterfall which has been fitted out as a Buddhist temple; quite an amazing spot. We capped off the afternoon by having dinner at a tea house in Maokong. A lot of the food was tea-themed --- deep-fried tea leaves, tea-flavoured omelette, etc. It was excellent, a really great destination for visitors to Taipei.
March 16, 2014 10:12 AM
March 10, 2014
Robert O'Callahan -- Introducing Chaos Mode
Some test failures are hard to reproduce. This is often because code (either tests or implementation code) makes unwarranted assumptions about the environment, assumptions that are violated nondeterministically. For example, a lot of tests have used setTimeout to schedule test code and assumed certain events will have happened before the timeout, which may not be true depending on effects such as network speeds and system load.
One way to make such bugs easier to reproduce is to intentionally exercise nondeterminism up to the limits of API contracts. For example, we can intentionally vary the actual time at which timers fire, to simulate the skew between CPU execution time and real time. To simulate different permitted thread schedules, we can assign random priorities to threads. Since hashtable iteration is not defined to have any particular order, we can make a hashtable iterator always start at a randomly chosen item.
I tried applying this to Gecko. I have patches that define a global "chaos mode" switch, and in several different places, if we're in chaos mode, we choose randomly between different valid behaviors of the code. Here's what the patches currently do:
- Sometimes yield just before dispatching an XPCOM event. This gives another thread a chance to win an event-dispatch race.
- On Linux, give threads a random priority and pin some threads to CPU 0 so they contend for CPU.
- Insert sockets in random positions in the list of polled sockets, to effectively randomize the priority of sockets in poll results.
- Similarly, when putting HTTP transactions into the HTTP transaction queue, randomly order them among other transactions with the same specified priority.
- Start hashtable iteration at a random entry.
- Scale timer firing times by random amounts (but don't vary the order in which timers fire, since that would violate the API contract).
- Shuffle mochitests and reftests so they run in random order.
Note that it can be valuable to make a single random choice consistently (for the same object, thread, etc) rather than making lots of fine-grained random decisions. For example, giving a thread a fixed low priority will starve it of CPU which will likely cause more extreme behavior --- hopefully more buggy behavior --- than choosing a random thread to run in each time quantum.
One important source of nondeterminism in Gecko is XPCOM event (i.e. HTML5 task) dispatch. A lot of intermittent bugs are due to event timing and ordering. It would be nice to exploit this in chaos mode, e.g. by choosing the next event to fire randomly from the set of pending events instead of processing them in dispatch order. Unfortunately we can't do that because a lot of code depend on the API contract that firing order follows dispatch order. In general it's hard to determine what the valid alternative firing orders are; the first item on my list above is my best approximation at the moment.
Does this find bugs? Yes:
Which chaos features are the most helpful for producing test failures? I don't know. It would be a very interesting experiment to do try pushes with different patches enabled to figure out which ones are the most important.
Does it help reproduce known intermittent bugs? Sometimes. In bug 975931 there was an intermittent reftest failure I could not reproduce locally without chaos mode, but I could reproduce with chaos mode. On the other hand chaos mode did not help reproduce bug 791480. Extending chaos mode can improve this situation.
Isn't this just fault injection? It's similar to fault injection (e.g. random out-of-memory triggering) but different. With fault injection typically we expect most tests to fail because faults like OOM are not fully recoverable. Chaos mode should not affect any correctness properties of the program.
Wasn't this already done by <insert project name>? Probably. I don't claim this is a new idea.
When is this going to land and how do I turn it on? It has already landed. To turn it on, change isActive() to return true in mfbt/ChaosMode.h. Shuffling of reftests and mochitests has to be done separately.
OK, so this can trigger interesting bugs, but how do we debug them? Indeed, chaos mode makes normal debugging workflows worse by introducing more nondeterminism. We could try to modify chaos mode to reproduce the random number stream between runs but that's inadequate because other sources of nondeterminism would interfere with the order in which the random number stream is sampled. But we are working on a much better solution to debugging nondeterministic programs; I'll be saying more about that very soon!
March 10, 2014 12:46 AM
March 09, 2014
Robert O'Callahan -- My Linkedin Account Is Dead, And Why Is Google Being Stupid?
I tried to delete my Linkedin account a while back, but I still get a lot of "invitation to connect on Linkedin" emails. I plan to never connect to anyone on Linkedin ever again, so whoever wants to connect, please don't be offended when it doesn't happen --- it's not about you.
Linkedin, I despise you for your persistent spam.
PS, I'm visiting Taiwan at the moment and wondering why Google uses that as a cue to switch its Web interface to Chinese even when I'm logged into my regular Google account. Dear Google, surely it is not very likely that my change of location to Taiwan indicates I have suddenly learned Chinese and forgotten English.
March 09, 2014 11:56 PM
March 07, 2014
Robert O'Callahan -- Fine-Tuning Arguments
I've been doing a little background reading and stumbled over this very interesting summary of arguments for fine-tuning of the universe. The cosmology is a bit over my head but as far as I understand it, it matches other sources I've read. It doesn't reach any conclusions about what causes the fine-tuning but it does pose some interesting challenges to the multiverse explanation.
March 07, 2014 12:50 AM
March 05, 2014
Robert O'Callahan -- Internet Connectivity As A Geopolitical Tool
A lot of people are wondering what Western countries can do about Russian's invasion of Ukraine. One option I haven't seen anyone suggest is to disconnect Russia from the Internet. There are lots of ways this could be done, but the simplest would be for participating countries to compel their exchanges to drop packets sourced from Russian ISPs.
This tactic has several advantages. It's asymmetric --- hurts Russia a lot more than the rest of the world, because not many services used internationally are based in Russia. It's nonviolent. It can be implemented quickly and cheaply and reversed just as quickly. It's quite severe. It distributes pain over most of the population.
This may not be the right tactic for this situation, but it's a realistic option against most modern countries (other than the USA, for obvious reasons).
If used, it would have the side effect of encouraging people to stop depending on services outside their own country, which I think is no bad thing.
March 05, 2014 04:05 AM
March 02, 2014
Robert O'Callahan -- Te Henga Walkway
On Saturday we finally did the Te Henga Walkway from Bethell's Beach to Goldie's Bush. I've wanted to do it for a long time but it's not a loop so you need multiple cars and a bit of planning. This weekend we finally got our act together with some friends and made it happen.
We dropped a car off at the end of Horseman Rd, at Goldie's Bush, and everyone continued to the start of the track at Bethell's Beach. The track goes over the hills with quite a bit of up-and-down walking but consistently excellent views of the ocean, beaches, cliffs and bush. It's a bit overgrown with gorse in places. After reaching Constable Rd, we walked along a bit and entered the west end of Goldie's Bush, walking through to the carpark on the other side. All the drivers got into our car and we dropped them off at the Bethell's end so they could bring their cars back to Horseman Rd to pick up everyone else.
Oddly, we were a bit slower than the nominal time for the Te Henga Walkway (signs say 3-4 hours, but we took a bit over 4 including our lunch break), but actually considerably faster than the nominal time for Goldie's Bush (signs say 2 hours, we took less than an hour). I would have expected to be slower on the later part of the walk when we were more tired.
March 02, 2014 04:56 AM
Q&A Panel At ACPC This Friday
This Friday evening I'm part of the panel for an open Q&A; session at Auckland Chinese Presbyterian Church. It should be a lot of fun!
March 02, 2014 03:23 AM
February 20, 2014
Robert O'Callahan -- 3 Mile Limit
Tonight I went the the premiere of the movie 3 Mile Limit. It's a New Zealand movie based on a true New Zealand story, the story of how in the 1960s four young men set out to break the government's monopoly on radio broadcasting by setting up a "pirate" radio station on a boat in international waters in the Hauraki Gulf. Lack of funds and government obstruction made their project extremely difficult but they were ultimately successful. This story is special to me because my father, Denis "Doc" O'Callahan, was one of the four and I am immensely proud of him. My father played an important role in the project, being the original radio engineer and also the first skipper of the boat, the Tiri. The movie's "Morrie" character is based on him.
You can tell from the trailer that the movie's a bit cliched, but I enjoyed it more than I expected. I found the production design evoking 1960s Auckland quite lovely --- parts of it reminded me of early childhood memories, and some parts remind me of Auckland today. A lot of the story has been rewritten for dramatic purposes, and the changes mostly work. The core theme of young rebels fighting to bring rock music to the public is completely true. The movie opens on March 6 and I hope it has a good run.
February 20, 2014 10:43 AM
February 18, 2014
Robert O'Callahan -- World Famous In Newmarket
The Newmarket Business Association has made a series of videos promoting Newmarket. They're interesting, if a little over-produced for my taste. Their Creativity in Newmarket video has a nice plug for the Mozilla office. Describing our office as a "head office" is a bit over the top, but there you go.
February 18, 2014 10:01 PM
February 17, 2014
Robert O'Callahan -- Implementing Virtual Widgets On The Web Platform
Some applications need to render large data sets in a single document, for example an email app might contain a list of tens of thousands of messages, or the FirefoxOS contacts app might have thousands of contacts in a single scrolling view. Creating explicit GUI elements for each item can be prohibitively expensive in memory usage and time. Solutions to this problem often involve custom widgets defined by the platform, e.g. the XUL tree widget, which expose a particular layout and content model and issue some kind of callback to the application to populate the widget with data incrementally, e.g. to get the data for the currently visible rows. Unfortunately these built-in widgets aren't a good solution to the problem, because they never have enough functionality to handle the needs of all applications --- they're never as rich as the language of explicit GUI elements.
Another approach is to expose very low-level API such as paint events and input events and let the developer reimplement their own widgets from scratch, but that that's far too much work.
I think the best approach is for applications to dynamically create UI elements that render the visible items. For this to work well, the platform needs to expose events or callbacks informing the application of which part of the list is (or will be) visible, and the application needs to be able to efficiently generate the UI elements in time for them to be displayed when the user is scrolling. We want to minimize the "checkerboarding" effect when a user scrolls to a point that app hasn't been able to populate with content.
I've written a demo of how this can work on the Web. It's quite simple. On receiving a scroll event, it creates enough items to fill the viewport, and then some more items within a limited distance of the viewport, so that browsers with async scrolling will (mostly) not see blank space during scrolling. Items far away from the viewport are recycled to reduce peak memory usage and speed up the item-placement step.
The demo uses absolute positioning to put items in the right place; this is more efficient than moving elements around in the DOM to reorder them vertically. Moving elements in the DOM forces a significant amount of restyling work which we want to avoid. On the other hand, this approach messes up selection a bit. The correct tradeoff depends on the application.
The demo depends on the layout being a simple vertical stack of identically-sized items. These constraints can be relaxed, but to avoid CSS layout of all items, we need to build in some application-specific knowledge of heights and layout. For example, if we can cheaply compute the height of each item (including the important case where there are a limited number of kinds of items and items with the same kind have the same height), we can use a balanced tree of items with each node in the tree storing the combined height of the items to get the performance we need.
It's important to note that the best implementation strategy varies a lot based on the needs and invariants of the application. That's one reason why I think we should not provide high-level functionality for these use-cases in the Web platform.
The main downside of this approach, IMHO, is that async scrolling (scrolling that occurs independently of script execution) can occasionally and temporarily show blank space where items should be. Of course, this is difficult to avoid with any solution to the large-data-set problem. I think the only alternative is to have the application signal to the async scroll thread the visible area that is valid, and have the async scroll thread prevent scrolling from leaving that region --- i.e. jank briefly. I see no way to completely avoid both jank and checkboarding.
Is there any API we can add to the platform to make this approach work better? I can think of just a couple of things:
- Make sure that at all times, an application can reliably tell which contents of a scrollable element are going to be rendered. We need not just the "scroll" event but also events that fire on resize. We need some way to determine which region of the scrolled content is going to be prerendered for async scrolling.
- Provide a way for an application to choose between checkerboarding and janking for a scrollable element.
February 17, 2014 01:38 AM
February 06, 2014
Robert O'Callahan -- Mozilla At Motuihe
Today was Waitangi Day in New Zealand, a public holiday. Since we haven't had our usual annual Mozilla office excursion, several people from the Mozilla office decided to go to Motuihe Island for the day with families/partners. Motuihe is a fairly small island between Waiheke Island and Auckland, recently cleared of predators and subject to a large reforestation and native bird repopulation project. In particular, little spotted kiwi have been reintroduced. This means I can see, from my desk, a place where kiwi live in the wild :-).
It's a short ferry ride from the city --- about 30 minutes --- but the ferry doesn't run most of the year; after this weekend it won't run again until December. (I've been to Motuihe lots of times but in the past, always on my parents' boat.) The weather today wasn't as good as forecast --- very windy, lots of cloud, and a spot of rain. Nevertheless I think we all had a good time. We walked around almost the whole island in a few hours, having a picnic lunch along the way at one of the smaller beaches (Calypso Bay). Then we walked on Ocean Beach a bit --- a very pleasant beach, generally, although today it was a bit wild with the wind blowing in hard. A few of us went for a swim, which was surprisingly fun, since the water was warm and the waves made it interesting. Auckland has also been experiencing a sequence of very high tides and the high tide today was no exception, which made it fun for the kids swinging on rope swings over the water.
Definitely worth doing if you're an Aucklander and haven't been there before. You can camp overnight on the island, and it's interesting to learn about the history of the island (which includes quarantine station and WW1 POW camp, which hosted the famous Count Felix von Luckner --- a most extraordinary man.)
February 06, 2014 11:28 AM
February 05, 2014
Robert O'Callahan -- Camels
The NZ Herald published a story (reprinted from the Daily Mail) about lack of evidence for camel domestication in the pre-1000BC Levant casting doubt on Biblical references to Abraham and his family using camels before that time. The report seems a bit off to me for a couple of reasons... Contrary to the implication of the article, this isn't by any means a new issue. People have been arguing about this precise question for over fifty years, albeit with a little less archeological data. (See this for example.) Secondly, the obvious rejoinder is that Abraham wasn't actually from the Levant; according to the Biblical narrative he was from Mesopotamia --- where camels appear to have been domesticated much earlier --- and travelled to what is now Israel, bringing his (considerable) household and possessions with him. It would have made complete sense for him to bring some camels with him for the journey, and for that small herd to be maintained and handed down for at least a few generations.
I admit it's a bit foolish to try to analyze data that's passed through the filters of press release and newspaper publication.
February 05, 2014 12:04 PM
January 24, 2014
Robert O'Callahan -- Lake Waikaremoana
This week I did the Lake Waikaremoana "Great Walk" with my children. This is a three-to-four day walk in Te Urewera National Park on the east coast of the North Island --- a range of low but rugged mountains covered in cold rainforest. We did it in four days, staying at Department of Conservation huts on the three nights --- Waiharuru, Waiopaoa and Panekire. These huts are cheap and shared with 20 to 40 other trampers depending on the size of the hut and how busy the track is. Since it's the school holidays and the middle of the summer, it was quite busy but the huts don't get overcrowded since they must be booked in advance.
This is the first "Great Walk" I've done, and the first tramp I've done over more than one night. Everything went very well. Tuesday was wet and also the longest day (eight hours including a side trip to Korokoro Falls, well worth it!) but everyone coped fine. Needless to say the lake and surrounding scenery are very beautiful.
The walk is around the western arm of the lake. Most trampers do it from the southern end to the northern end of the lake but we did it in the other direction, in the hope our packs would be lighter for the climb over the Panekire bluff at the southern end. It probably didn't make much difference.
For some reason I find it very easy to talk to strangers staying at tramping huts. Maybe it's because we're cooking, eating, talking and sleeping at close quarters, maybe it's because of the isolation or the shared challenge of the walk, or maybe it's something else, but it's the easiest place --- other than church maybe --- to strike up a conversation with someone I don't know at all. I've met a lot of interesting people from different countries and backgrounds. I've learned a lot about tramping from them, since of course many of them are more experienced than me. It's a good environment to learn to be more outgoing.
Everything was great and I highly recommend the walk. Panekire Hut is outstanding, perched on the top of the bluff with an incredible view over the lake, yet somehow in a pocket sheltered from the strong gusty winds just a few paces away. As it turned out, we could have easily done the walk in three days by ascending and descending Panekire on the same day, skipping staying at the hut, but I'm glad we didn't. I do wish we'd brought togs so we could swim in the lake while at the lakeside huts --- many of the other trampers did.
January 24, 2014 10:17 AM
January 12, 2014
Robert O'Callahan -- Web Specifications And The Frame Problem
I've noticed a pattern of mistakes when some smart people think about Web specifications. They think that a general rule in some spec does not apply to some specific situation that they're focused on, because the spec does not mention their specific situation. One example: someone suggested that we could make typed arrays returned by Web Audio APIs not be neuterable, arguing that this did not alter already-specified behavior because "nothing currently specifies how typed arrays work with Web Audio". Another example: someone argued that CSS does not specify how to determine the size of CSS content, padding and border boxes when the 'overflow' property is not 'visible'.
I hope it's clear that these are mistakes because when a Web spec states a general rule, that rule applies in all situations unless there is specific text contradicting it elsewhere (in which case, some spec should clearly indicate which rule prevails ... and I recognize that's not always true). Any other approach leads to madness; modularity and extensibility would be destroyed if, any time a feature is added to the Web platform, we had to explicitly reaffirm the behavior of all other features.
I think this is a form of the "frame problem" in AI, or "default logic". A kind of intertia requires previously specified rules to remain true until explicitly changed by another spec.
In typical programming languages, we tackle these issues by making every modifiable rule an explicit extension point (which could be implemented in lots of ways --- a virtual function call, a function value, an enum value, etc). Modifications explicitly utilize the extension point. This is sometimes done in specs, but rarely. One reason is that it makes your specs verbose and hard to read. Another reason is that it's hard to know the set of extension points ahead of time, so when you discover later you need to make modifications, you have to go back and add those extension points. That's getting easier now that more specs are using the "living standard" model, but it used to be hard.
There are programming languages that support this kind of dynamic extensibility. Languages with externally defined multimethods have a taste of it. CSS itself, ironically, orders rules by specificity and allows more-specific rules to override less-specific rules.
One problem I've been interested in for a long time is encoding the semantics of board games. Board game rules are often designed like Web specs, with some rules overriding others in special cases. Board games have extensibility problems, when expansions come out which lead to ambiguities in the context of the original game. (Munchkin makes sport of the problem.)
January 12, 2014 09:14 PM
Last updated: August 22, 2014 11:30 PM