April 09, 2014
Robert O'Callahan -- Getting Back To Work
... is what we need now. So let me give a brief summary of what's happening with me work-wise.
Last year I fully divested my direct reports. No more management work to do, yay! I think they probably all ended up with a better manager than they had.
I've been involved in a lot of different projects and a lot of helping other people get their work done. In between I managed to carve out a few things of my own:
- I built reftests for async panning to try to stem the tide of regressions we were encountering there. Unfortunately that's not quite done because most of those tests aren't running on TBPL yet.
- I worked on CSS scroll snapping with an intern. Unfortunately the spec situation got bogged down; there's an impasse between us and Microsoft over the design of the CSS feature, and Google has decided not to do a CSS feature at all for now and try to do it with JS instead. I'm skeptical that will work will, and looking forward to their proposal, but it's taking a while.
- I landed an implementation of the CSS OM GeometryUtils interface, described in hacks.mozilla.org blog posts here and here. This fixes a functionality gap in the Web platform and was needed by our devtools team. Almost everything you would need to know about the CSS box geometry of your page is now easy to get.
- Lately I've been doing work on rr. People are trying to use it, and they've been uncovering bugs and inconveniences that Chris Jones and I have been fixing as fast as we can. Terrence Cole used it successfully to help fix a JS engine bug! I'm using it a lot myself for general-purpose debugging, and enjoying it. I want to spend a few more days getting some of the obvious rough edges we've found filed off and then make a new release.
Looking forward, I expect to be working on making our async scrolling fully generic (right now there are some edge cases where we can't do it), and working on some improvements to our MediaStreamGraph code for lower latency video and audio.
April 09, 2014 11:54 AM
Fighting Media Narratives
- Almost all Mozilla staff supported keeping Brendan Eich as CEO, including many prominent LGBT staff, and many made public statements to that effect. A small number of Tweeters calling for him to step down got all the media attention. The narrative that Mozilla staff as a group "turned against Brendan" is false. It should go without saying, but most Mozilla staff, especially me, are very upset that he's gone. I've known him, worked with him and fought alongside him (and sometimes against him :-) ) for fourteen years and having him ripped away like this is agonizing.
- The external pressure for Brendan to step down was the primary factor driving the entire situation. The same issue exploded in 2012 but there was less pressure and he got through it. No doubt Mozilla could have handled it better but the narrative that blames Mozilla for Brendan's departure misses the big picture. Boycotting Mozilla (or anyone for that matter) for cracking under intense pressure is like shooting a shell-shocked soldier.
- As a Christian, Mozilla is as friendly a workplace as any tech organization I've known --- which is to say, not super friendly, but unproblematic. Because of our geographic spread --- not just of headcount, but of actual power --- and our broad volunteer base I think we have more real diversity than many of our competitors. The narrative that Mozilla as a group has landed on one side of the culture war is false, or at least no more true than for other tech organizations. In fact one thing I've really enjoyed over the last couple of weeks is seeing a diverse set of Mozilla people pull together in adversity and form even closer bonds.
I'll also echo something else a lot of people are saying: we have to fix Internet discourse somehow. It's toxic. I wrote about this a while back, and this episode has made me experience the problem at a whole new level. I'll throw one idea out there: let's communicate using only recorded voice+video messages, no tweets, no text. If you want to listen to what I have to say, you have to watch me say it, and hopefully that will trigger some flickers of empathy. If you want me to listen to you, you have to show me your face. Want to be anonymous, do it the old-fashioned way and wear a mask. Yeah I know we'd have to figure out searchability, skimmability, editing, etc etc. Someone get to work on it.
April 09, 2014 11:31 AM
How much does the world need Mozilla? A useful, if uncomfortable, thought experiment is to consider what the world would be like without Mozilla.
Consider the world of Web standards. Microsoft doesn't contribute much to developing new Web features, and neither does Apple these days. Mozilla and Google do. Google, per Blink's own policy (mirroring our own), relies on feedback and implementation by other browser vendors, i.e. usually us. If you take Mozilla out of the equation, it's going to be awfully hard to apply the "two independent implementations" test for new Web features. (Especially since Webkit and Blink still have so much shared heritage.) For example it's hard to see how important stuff like Web Audio, WebGL and WebRTC would have become true multi-vendor standards without us. Without us, most progress would depend on unilateral extensions by individual vendors. That has all the downsides of a single-implementation ecosystem --- a single implementation's bugs become the de-facto standard; the standards process, if there even is one, becomes irrelevant; and even more power accrues to the dominant vendor.
In the bigger picture, it would be dangerous to leave the Web --- and the entire application platform space --- in the hands of three very large US corporations who have few scruples and who each have substantial non-Web interests to protect, including proprietary desktop and mobile platforms. It has always been very important that there be a compelling vendor-neutral platform for people to deploy content and apps on, a platform without gatekeepers and without taxes. The Web is that platform ---- for now. Mozilla is dedicated to preserving and enhancing that, but the other vendors are not.
Mozilla has plenty of faults, and lots of challenges, but our mission is as important as ever ... probably more important, given how computing devices and the Internet penetrate more and more of our lives. More than ever, pursuing our mission is the greatest good I can do with the talents God has given me.
April 09, 2014 10:34 AM
April 06, 2014
Robert O'Callahan -- Responsible Self-Censorship
People may be wondering why I, as one of the most notorious Christians at Mozilla, have been silent during the turmoil of recent days. It is simply because I haven't been able to think of anything to say that won't cause more harm in the current environment. It is not because I am afraid, malicious, or suffering from a lack of freedom of speech in any way. As soon as I have the environment and the words to say something helpful, I will.
By "the current environment" I mean the situation where many of the culture warriors of all stripes are picking over every utterance looking for something to be angry against or something to fuel their anger against others.
Update Actually, right after posting that, I thought of something to say.
I have never loved the people of Mozilla as much as I do right now. With just a few exceptions, your commitment and grace under fire is inspiring and (in the old-fashioned sense) awesome. I'm glad I'm here for this.
April 06, 2014 09:27 PM
March 26, 2014
Robert O'Callahan -- Conflict
I was standing in a long line of people waiting to get their passports checked before departing Auckland to the USA. A man pushed into the line ahead of me. People around me sighed and muttered, but no-one did anything. This triggered my "if no-one else is willing to act, it's up to me" instinct, so I started a conversation:
Roc: Shouldn't you go to the back of the line?
Man: I'm in a hurry to get to my flight.
There are only two flights to the USA in the immediate future and AFAIK neither of them is imminent.
Roc: Which flight are you on?
Man: What's the big deal? I'm hardly going to slow you down.
Roc: But it's rude.
Man: So what do you want me to do?
Roc: I want you to move to the back of the line.
Man: You're going to have to make me.
At that point I couldn't think of anything to say or do so I let it go. It turned out that he was on my flight and it would have been unpleasant if he'd ended up sitting next to me.
I was a bit shocked by this incident, though I probably shouldn't have been. This kind of unapologetic rudeness is not something I encounter very often from strangers in face-to-face situations. I guess that means I'm fortunate. Surprisingly I felt quite calm during the conversation and only felt rage later.
Embarrassingly-much-later, it dawned on me that Jesus wants me to forgive this man. I am now making the effort to do so.
March 26, 2014 09:46 PM
Bugs that reproduce intermittently are hard to debug with traditional techniques because single stepping, setting breakpoints, inspecting program state, etc, is all a waste of time if the program execution you're debugging ends up not even exhibiting the bug. Even when you can reproduce a bug consistently, important information such as the addresses of suspect objects is unpredictable from run to run. Given that software developers like me spend the majority of their time finding and fixing bugs (either in new code or existing code), nondeterminism has a major impact on my productivity.
Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.
Here are some of our key design parameters:
- We focused on debugging Firefox. Firefox is a complex application, so if rr is useful for debugging Firefox, it is very likely to be generally useful. But, for example, we have not concerned ourselves with record and replay of hostile binaries, or highly parallel applications. Nor have we explored novel techniques just for the sake of novelty.
- We prioritized deployability. For example, we deliberately avoided modifying the OS kernel and even worked hard to eliminate the need for system configuration changes. Of course we ensured rr works on stock hardware.
- We placed high importance on low run-time overhead. We want rr to be the tool of first resort for debugging. That means you need to start getting results with rr about as quickly as you would if you were using a traditional debugger. (This is where my previous project Chronomancer fell down.)
- We tried to take a path that would lead to a quick positive payoff with a small development investment. A large speculative project in this area would fall outside Mozilla's resources and mission.
- Naturally, the tool has to be free software.
I believe we've achieved those goals with our 1.0 release.
There's a lot to say about how rr works, and I'll probably write some followup blog posts about that. In this post I focus on what rr can do.
rr records and replays a Linux process tree, i.e. an initial process and the threads and subprocesses (indirectly) spawned by that process. The replay is exact in the sense that the contents of memory and registers during replay are identical to their values during recording. rr provides a gdbserver interface allowing gdb to debug the state of replayed processes.
Here are performance results for some Firefox testsuite workloads: These represent the ratio of wall-clock run-time of rr recording and replay over the wall-clock run-time of normal execution (except for Octane, where it's the ratio of the normal execution's Octane score over the record/replay Octane scores ... which, of course, are the same). It turns out that reftests are slow under rr because Gecko's current default Linux configuration draws with X11, hence the rendered pixmap of every test and reference page has to be slurped back from the X server over the X socket for comparison, and rr has to record all of that data. So I also show overhead for reftests with Xrender disabled, which causes Gecko to draw locally and avoid the problem.
I should also point out that we stopped focusing on rr performance a while ago because we felt it was already good enough, not because we couldn't make it any faster. It can probably be improved further without much work.
Debugging with rr
The rr project landing page has a screencast illustrating the rr debugging experience. rr lets you use gdb to debug during replay. It's difficult to communicate the feeling of debugging with rr, but if you've ever done something wrong during debugging (e.g. stepped too far) and had that "crap! Now I have to start all over again" sinking feeling --- rr does away with that. Everything you already learned about the execution --- e.g. the addresses of the objects that matter --- remains valid when you start the next replay. Your understanding of the execution increases monotonically.
rr has limitations. Some are inherent to its design, and some are fixable with more work.
- rr emulates a single-core machine. So, parallel programs incur the slowdown of running on a single core. This is an inherent feature of the design. Practical low-overhead recording in a multicore setting requires hardware support; we hope that if rr becomes popular, it will motivate hardware vendors to add such support.
- rr cannot record processes that share memory with processes outside the recording tree. This is an inherent feature of the design. rr automatically disables features such as X11 shared memory for recorded processes to avoid this problem.
- For the same reason, rr tracees cannot use direct-rendering user-space GL drivers. To debug GL code under rr we'll need to find or create a proxy driver that doesn't share memory with the kernel (something like GLX).
- rr requires a reasonably modern x86 CPU. It depends on certain performance counter features that are not available in older CPUs, or in ARM at all currently. rr works on Intel Ivy Bridge and Sandy Bridge microarchitectures. It doesn't currently work on Haswell and we're investigating how to fix that.
- rr currently only supports x86 32-bit processes. (Porting to x86-64 should be straightforward but it's quite a lot of work.)
- rr needs to explicitly support every system call executed by the recorded processes. It already supports a wide range of syscalls --- syscalls used by Firefox --- but running rr on more workloads will likely uncover more syscalls that need to be supported.
- When used with gdb, rr does not provide the ability to call program functions from the debugger, nor does it provide hardware data breakpoints. These issues can be fixed with future work.
We believe rr is already a useful tool. I like to use it myself to debug Gecko bugs; in fact, it's the first "research" tool I've ever built that I like to use myself. If you debug Firefox at the C/C++ level on Linux you should probably try it out. We would love to have feedback --- or better still, contributions!
If you try to debug other large applications with rr, you will probably encounter rr bugs. Therefore we are not yet recommending rr for general-purpose C/C++ debugging. However, if rr interests you, please consider trying it out and reporting/fixing any bugs that you find.
We hope rr is a useful tool in itself, but we also see it as just a first step. rr+gdb is not the ultimate debugging experience (even if gdb's backtracing features get an rr-based backend, which I hope happens!). We have a lot of ideas for making vast improvements to the debugging experience by building on rr. I hope other people find rr useful as a building block for their ideas too.
I'd like to thank the people who've contributed to rr so far: Albert Noll, Nimrod Partush, Thomas Anderegg and Chris Jones --- and to Mozilla Research, especially Andreas Gal, for supporting this project.
March 26, 2014 01:59 AM
March 23, 2014
Robert O'Callahan -- Mozilla And The Silicon Valley Cartel
This article, while overblown (e.g. I don't think "no cold calls" agreements are much of a problem), is an interesting read, especially the court exhibits attached at the end. One interesting fact is that Mozilla does not appear on any of Google's do-not-call lists --- yay us! (I can confirm first-hand that Mozilla people have been relentlessly cold-emailed by Google over the years. They stopped contacting me after I told them not to even bother trying until they open a development office in New Zealand.)
Exhibit 1871 is also very interesting since it appears the trigger that caused Steve Jobs to fly off the handle (which in turn led to the collusion at the heart of the court case) was Ben Goodger referring members of the Safari team for recruitment by Google, at least one of whom was formerly from Mozilla.
March 23, 2014 10:42 PM
March 16, 2014
Robert O'Callahan -- Taroko National Park
This weekend, in between the media and layout work weeks, I had time off so I thought I'd get out of Taipei and see a different part of Taiwan. On Alfredo's recommendation I decided to visit Taroko National Park and do some hiking. I stayed at Taroko Lodge and had a great time --- Rihang Su was an excellent host, very friendly and helpful.
On Saturday I took the train from Songshan station in Taipei to Xincheng. From there I went via the lodge to the park's eastern headquarters and did a short afternoon hike up to near Dali village and back down again. This was short in distance --- about 3.5 km each way in two and half hours --- but reasonably hard work since it's about a 900m vertical climb.
On Saturday night Rihang took the lodge guests to Hualien city for dinner at a hotpot buffet place. The food was reasonable and it was a fun dinner. There were instructions in English and in general I was surprised by how much English signage there was in Hualien (including at the supermarket) --- I had expected none.
On Sunday (today) I got up fairly early for the main goal of my trip: Zhuilu Old Trail. Rihang dropped me off at the western end around 8:15am and I walked the 10km length in a bit under four hours. You're supposed to allow for six, but that's probably a bit conservative, plus I'm fairly fast. It is hard work and tricky in places, however. This trail is quite amazing. There's one stretch in particular where the path winds along the cliff face, mostly about a meter wide, and there's nothing between you and a sheer drop of hundreds of meters --- except a plastic-clad steel cable on the cliff side of the track to hold onto. It reminded me of the National Pass track in Australia's Blue Mountains, but more intense. Much of the track has spectacular views of the Taroko Gorge --- sheer cliffs, marble boulders, lush forest, and mountain streams. In one place I was lucky enough to see macaque monkeys, which was thrilling since I've never seen monkeys in the wild before. The trail has an interesting history; it was originally constructed by the Japanese military in the early 20th century (probably using slave labour, though it wasn't clear from the signage), to improve their control over the native people of the area. Later the Japanese turned it into a tourist destination and that's what it is today. Along the trail there are rest areas where fortified "police stations" used to be.
After finishing that trail, since I was ahead of schedule I continued with the Swallow Grotto walk along the road westward up the gorge. This walk was also amazing but offers quite different views since you're near the bottom of the gorge --- and it's much easier. The walk ends at the bottom of the Zhuilu cliff, which is neck-craningly massive. After that I got picked up and went to the train station to head straight back to Taipei.
This was definitely a very good way to see a bit of Taiwan that's not Taipei. Shuttling around between the train station, the lodge, and hiking destinations, I got a glimpse of what life is like in the region --- gardens, farming, stray dogs, decrepit buildings, industry, narrow roads, etc. But the natural beauty of Taroko gorge was definitely the highlight of the trip. I enjoy hiking with friends and family very much, but it's nice to hike alone for a change; I can go at my own pace, not worry about anyone else, and be a bit more relaxed. All in all it was a great weekend getaway.
March 16, 2014 04:41 PM
I've been in Taiwan for a week. Last Sunday, almost immediately after checking into the hotel I went with a number of Mozilla people to Maokong Gondala and did a short hike around Maokong itself, to Yinhe Cave and back. I really enjoyed this trip. The gondala ride is quite long, and pretty. Maokong itself is devoted to the cultivation and vending of tea. I haven't seen a real tea plantation before, and I also got to see rice paddies up close, so this was quite interesting --- I'm fascinated by agricultural culture, which dominated so many people's lives for such a long time. Yinhe Cave is a cave behind a waterfall which has been fitted out as a Buddhist temple; quite an amazing spot. We capped off the afternoon by having dinner at a tea house in Maokong. A lot of the food was tea-themed --- deep-fried tea leaves, tea-flavoured omelette, etc. It was excellent, a really great destination for visitors to Taipei.
March 16, 2014 10:12 AM
March 10, 2014
Robert O'Callahan -- Introducing Chaos Mode
Some test failures are hard to reproduce. This is often because code (either tests or implementation code) makes unwarranted assumptions about the environment, assumptions that are violated nondeterministically. For example, a lot of tests have used setTimeout to schedule test code and assumed certain events will have happened before the timeout, which may not be true depending on effects such as network speeds and system load.
One way to make such bugs easier to reproduce is to intentionally exercise nondeterminism up to the limits of API contracts. For example, we can intentionally vary the actual time at which timers fire, to simulate the skew between CPU execution time and real time. To simulate different permitted thread schedules, we can assign random priorities to threads. Since hashtable iteration is not defined to have any particular order, we can make a hashtable iterator always start at a randomly chosen item.
I tried applying this to Gecko. I have patches that define a global "chaos mode" switch, and in several different places, if we're in chaos mode, we choose randomly between different valid behaviors of the code. Here's what the patches currently do:
- Sometimes yield just before dispatching an XPCOM event. This gives another thread a chance to win an event-dispatch race.
- On Linux, give threads a random priority and pin some threads to CPU 0 so they contend for CPU.
- Insert sockets in random positions in the list of polled sockets, to effectively randomize the priority of sockets in poll results.
- Similarly, when putting HTTP transactions into the HTTP transaction queue, randomly order them among other transactions with the same specified priority.
- Start hashtable iteration at a random entry.
- Scale timer firing times by random amounts (but don't vary the order in which timers fire, since that would violate the API contract).
- Shuffle mochitests and reftests so they run in random order.
Note that it can be valuable to make a single random choice consistently (for the same object, thread, etc) rather than making lots of fine-grained random decisions. For example, giving a thread a fixed low priority will starve it of CPU which will likely cause more extreme behavior --- hopefully more buggy behavior --- than choosing a random thread to run in each time quantum.
One important source of nondeterminism in Gecko is XPCOM event (i.e. HTML5 task) dispatch. A lot of intermittent bugs are due to event timing and ordering. It would be nice to exploit this in chaos mode, e.g. by choosing the next event to fire randomly from the set of pending events instead of processing them in dispatch order. Unfortunately we can't do that because a lot of code depend on the API contract that firing order follows dispatch order. In general it's hard to determine what the valid alternative firing orders are; the first item on my list above is my best approximation at the moment.
Does this find bugs? Yes:
Which chaos features are the most helpful for producing test failures? I don't know. It would be a very interesting experiment to do try pushes with different patches enabled to figure out which ones are the most important.
Does it help reproduce known intermittent bugs? Sometimes. In bug 975931 there was an intermittent reftest failure I could not reproduce locally without chaos mode, but I could reproduce with chaos mode. On the other hand chaos mode did not help reproduce bug 791480. Extending chaos mode can improve this situation.
Isn't this just fault injection? It's similar to fault injection (e.g. random out-of-memory triggering) but different. With fault injection typically we expect most tests to fail because faults like OOM are not fully recoverable. Chaos mode should not affect any correctness properties of the program.
Wasn't this already done by <insert project name>? Probably. I don't claim this is a new idea.
When is this going to land and how do I turn it on? It has already landed. To turn it on, change isActive() to return true in mfbt/ChaosMode.h. Shuffling of reftests and mochitests has to be done separately.
OK, so this can trigger interesting bugs, but how do we debug them? Indeed, chaos mode makes normal debugging workflows worse by introducing more nondeterminism. We could try to modify chaos mode to reproduce the random number stream between runs but that's inadequate because other sources of nondeterminism would interfere with the order in which the random number stream is sampled. But we are working on a much better solution to debugging nondeterministic programs; I'll be saying more about that very soon!
March 10, 2014 12:46 AM
March 09, 2014
Robert O'Callahan -- My Linkedin Account Is Dead, And Why Is Google Being Stupid?
I tried to delete my Linkedin account a while back, but I still get a lot of "invitation to connect on Linkedin" emails. I plan to never connect to anyone on Linkedin ever again, so whoever wants to connect, please don't be offended when it doesn't happen --- it's not about you.
Linkedin, I despise you for your persistent spam.
PS, I'm visiting Taiwan at the moment and wondering why Google uses that as a cue to switch its Web interface to Chinese even when I'm logged into my regular Google account. Dear Google, surely it is not very likely that my change of location to Taiwan indicates I have suddenly learned Chinese and forgotten English.
March 09, 2014 11:56 PM
March 07, 2014
Robert O'Callahan -- Fine-Tuning Arguments
I've been doing a little background reading and stumbled over this very interesting summary of arguments for fine-tuning of the universe. The cosmology is a bit over my head but as far as I understand it, it matches other sources I've read. It doesn't reach any conclusions about what causes the fine-tuning but it does pose some interesting challenges to the multiverse explanation.
March 07, 2014 12:50 AM
March 05, 2014
Robert O'Callahan -- Internet Connectivity As A Geopolitical Tool
A lot of people are wondering what Western countries can do about Russian's invasion of Ukraine. One option I haven't seen anyone suggest is to disconnect Russia from the Internet. There are lots of ways this could be done, but the simplest would be for participating countries to compel their exchanges to drop packets sourced from Russian ISPs.
This tactic has several advantages. It's asymmetric --- hurts Russia a lot more than the rest of the world, because not many services used internationally are based in Russia. It's nonviolent. It can be implemented quickly and cheaply and reversed just as quickly. It's quite severe. It distributes pain over most of the population.
This may not be the right tactic for this situation, but it's a realistic option against most modern countries (other than the USA, for obvious reasons).
If used, it would have the side effect of encouraging people to stop depending on services outside their own country, which I think is no bad thing.
March 05, 2014 04:05 AM
March 02, 2014
Robert O'Callahan -- Te Henga Walkway
On Saturday we finally did the Te Henga Walkway from Bethell's Beach to Goldie's Bush. I've wanted to do it for a long time but it's not a loop so you need multiple cars and a bit of planning. This weekend we finally got our act together with some friends and made it happen.
We dropped a car off at the end of Horseman Rd, at Goldie's Bush, and everyone continued to the start of the track at Bethell's Beach. The track goes over the hills with quite a bit of up-and-down walking but consistently excellent views of the ocean, beaches, cliffs and bush. It's a bit overgrown with gorse in places. After reaching Constable Rd, we walked along a bit and entered the west end of Goldie's Bush, walking through to the carpark on the other side. All the drivers got into our car and we dropped them off at the Bethell's end so they could bring their cars back to Horseman Rd to pick up everyone else.
Oddly, we were a bit slower than the nominal time for the Te Henga Walkway (signs say 3-4 hours, but we took a bit over 4 including our lunch break), but actually considerably faster than the nominal time for Goldie's Bush (signs say 2 hours, we took less than an hour). I would have expected to be slower on the later part of the walk when we were more tired.
March 02, 2014 04:56 AM
Q&A Panel At ACPC This Friday
This Friday evening I'm part of the panel for an open Q&A; session at Auckland Chinese Presbyterian Church. It should be a lot of fun!
March 02, 2014 03:23 AM
February 20, 2014
Robert O'Callahan -- 3 Mile Limit
Tonight I went the the premiere of the movie 3 Mile Limit. It's a New Zealand movie based on a true New Zealand story, the story of how in the 1960s four young men set out to break the government's monopoly on radio broadcasting by setting up a "pirate" radio station on a boat in international waters in the Hauraki Gulf. Lack of funds and government obstruction made their project extremely difficult but they were ultimately successful. This story is special to me because my father, Denis "Doc" O'Callahan, was one of the four and I am immensely proud of him. My father played an important role in the project, being the original radio engineer and also the first skipper of the boat, the Tiri. The movie's "Morrie" character is based on him.
You can tell from the trailer that the movie's a bit cliched, but I enjoyed it more than I expected. I found the production design evoking 1960s Auckland quite lovely --- parts of it reminded me of early childhood memories, and some parts remind me of Auckland today. A lot of the story has been rewritten for dramatic purposes, and the changes mostly work. The core theme of young rebels fighting to bring rock music to the public is completely true. The movie opens on March 6 and I hope it has a good run.
February 20, 2014 10:43 AM
February 18, 2014
Robert O'Callahan -- World Famous In Newmarket
The Newmarket Business Association has made a series of videos promoting Newmarket. They're interesting, if a little over-produced for my taste. Their Creativity in Newmarket video has a nice plug for the Mozilla office. Describing our office as a "head office" is a bit over the top, but there you go.
February 18, 2014 10:01 PM
February 17, 2014
Robert O'Callahan -- Implementing Virtual Widgets On The Web Platform
Some applications need to render large data sets in a single document, for example an email app might contain a list of tens of thousands of messages, or the FirefoxOS contacts app might have thousands of contacts in a single scrolling view. Creating explicit GUI elements for each item can be prohibitively expensive in memory usage and time. Solutions to this problem often involve custom widgets defined by the platform, e.g. the XUL tree widget, which expose a particular layout and content model and issue some kind of callback to the application to populate the widget with data incrementally, e.g. to get the data for the currently visible rows. Unfortunately these built-in widgets aren't a good solution to the problem, because they never have enough functionality to handle the needs of all applications --- they're never as rich as the language of explicit GUI elements.
Another approach is to expose very low-level API such as paint events and input events and let the developer reimplement their own widgets from scratch, but that that's far too much work.
I think the best approach is for applications to dynamically create UI elements that render the visible items. For this to work well, the platform needs to expose events or callbacks informing the application of which part of the list is (or will be) visible, and the application needs to be able to efficiently generate the UI elements in time for them to be displayed when the user is scrolling. We want to minimize the "checkerboarding" effect when a user scrolls to a point that app hasn't been able to populate with content.
I've written a demo of how this can work on the Web. It's quite simple. On receiving a scroll event, it creates enough items to fill the viewport, and then some more items within a limited distance of the viewport, so that browsers with async scrolling will (mostly) not see blank space during scrolling. Items far away from the viewport are recycled to reduce peak memory usage and speed up the item-placement step.
The demo uses absolute positioning to put items in the right place; this is more efficient than moving elements around in the DOM to reorder them vertically. Moving elements in the DOM forces a significant amount of restyling work which we want to avoid. On the other hand, this approach messes up selection a bit. The correct tradeoff depends on the application.
The demo depends on the layout being a simple vertical stack of identically-sized items. These constraints can be relaxed, but to avoid CSS layout of all items, we need to build in some application-specific knowledge of heights and layout. For example, if we can cheaply compute the height of each item (including the important case where there are a limited number of kinds of items and items with the same kind have the same height), we can use a balanced tree of items with each node in the tree storing the combined height of the items to get the performance we need.
It's important to note that the best implementation strategy varies a lot based on the needs and invariants of the application. That's one reason why I think we should not provide high-level functionality for these use-cases in the Web platform.
The main downside of this approach, IMHO, is that async scrolling (scrolling that occurs independently of script execution) can occasionally and temporarily show blank space where items should be. Of course, this is difficult to avoid with any solution to the large-data-set problem. I think the only alternative is to have the application signal to the async scroll thread the visible area that is valid, and have the async scroll thread prevent scrolling from leaving that region --- i.e. jank briefly. I see no way to completely avoid both jank and checkboarding.
Is there any API we can add to the platform to make this approach work better? I can think of just a couple of things:
- Make sure that at all times, an application can reliably tell which contents of a scrollable element are going to be rendered. We need not just the "scroll" event but also events that fire on resize. We need some way to determine which region of the scrolled content is going to be prerendered for async scrolling.
- Provide a way for an application to choose between checkerboarding and janking for a scrollable element.
February 17, 2014 01:38 AM
February 06, 2014
Robert O'Callahan -- Mozilla At Motuihe
Today was Waitangi Day in New Zealand, a public holiday. Since we haven't had our usual annual Mozilla office excursion, several people from the Mozilla office decided to go to Motuihe Island for the day with families/partners. Motuihe is a fairly small island between Waiheke Island and Auckland, recently cleared of predators and subject to a large reforestation and native bird repopulation project. In particular, little spotted kiwi have been reintroduced. This means I can see, from my desk, a place where kiwi live in the wild :-).
It's a short ferry ride from the city --- about 30 minutes --- but the ferry doesn't run most of the year; after this weekend it won't run again until December. (I've been to Motuihe lots of times but in the past, always on my parents' boat.) The weather today wasn't as good as forecast --- very windy, lots of cloud, and a spot of rain. Nevertheless I think we all had a good time. We walked around almost the whole island in a few hours, having a picnic lunch along the way at one of the smaller beaches (Calypso Bay). Then we walked on Ocean Beach a bit --- a very pleasant beach, generally, although today it was a bit wild with the wind blowing in hard. A few of us went for a swim, which was surprisingly fun, since the water was warm and the waves made it interesting. Auckland has also been experiencing a sequence of very high tides and the high tide today was no exception, which made it fun for the kids swinging on rope swings over the water.
Definitely worth doing if you're an Aucklander and haven't been there before. You can camp overnight on the island, and it's interesting to learn about the history of the island (which includes quarantine station and WW1 POW camp, which hosted the famous Count Felix von Luckner --- a most extraordinary man.)
February 06, 2014 11:28 AM
February 05, 2014
Robert O'Callahan -- Camels
The NZ Herald published a story (reprinted from the Daily Mail) about lack of evidence for camel domestication in the pre-1000BC Levant casting doubt on Biblical references to Abraham and his family using camels before that time. The report seems a bit off to me for a couple of reasons... Contrary to the implication of the article, this isn't by any means a new issue. People have been arguing about this precise question for over fifty years, albeit with a little less archeological data. (See this for example.) Secondly, the obvious rejoinder is that Abraham wasn't actually from the Levant; according to the Biblical narrative he was from Mesopotamia --- where camels appear to have been domesticated much earlier --- and travelled to what is now Israel, bringing his (considerable) household and possessions with him. It would have made complete sense for him to bring some camels with him for the journey, and for that small herd to be maintained and handed down for at least a few generations.
I admit it's a bit foolish to try to analyze data that's passed through the filters of press release and newspaper publication.
February 05, 2014 12:04 PM
January 24, 2014
Robert O'Callahan -- Lake Waikaremoana
This week I did the Lake Waikaremoana "Great Walk" with my children. This is a three-to-four day walk in Te Urewera National Park on the east coast of the North Island --- a range of low but rugged mountains covered in cold rainforest. We did it in four days, staying at Department of Conservation huts on the three nights --- Waiharuru, Waiopaoa and Panekire. These huts are cheap and shared with 20 to 40 other trampers depending on the size of the hut and how busy the track is. Since it's the school holidays and the middle of the summer, it was quite busy but the huts don't get overcrowded since they must be booked in advance.
This is the first "Great Walk" I've done, and the first tramp I've done over more than one night. Everything went very well. Tuesday was wet and also the longest day (eight hours including a side trip to Korokoro Falls, well worth it!) but everyone coped fine. Needless to say the lake and surrounding scenery are very beautiful.
The walk is around the western arm of the lake. Most trampers do it from the southern end to the northern end of the lake but we did it in the other direction, in the hope our packs would be lighter for the climb over the Panekire bluff at the southern end. It probably didn't make much difference.
For some reason I find it very easy to talk to strangers staying at tramping huts. Maybe it's because we're cooking, eating, talking and sleeping at close quarters, maybe it's because of the isolation or the shared challenge of the walk, or maybe it's something else, but it's the easiest place --- other than church maybe --- to strike up a conversation with someone I don't know at all. I've met a lot of interesting people from different countries and backgrounds. I've learned a lot about tramping from them, since of course many of them are more experienced than me. It's a good environment to learn to be more outgoing.
Everything was great and I highly recommend the walk. Panekire Hut is outstanding, perched on the top of the bluff with an incredible view over the lake, yet somehow in a pocket sheltered from the strong gusty winds just a few paces away. As it turned out, we could have easily done the walk in three days by ascending and descending Panekire on the same day, skipping staying at the hut, but I'm glad we didn't. I do wish we'd brought togs so we could swim in the lake while at the lakeside huts --- many of the other trampers did.
January 24, 2014 10:17 AM
January 12, 2014
Robert O'Callahan -- Web Specifications And The Frame Problem
I've noticed a pattern of mistakes when some smart people think about Web specifications. They think that a general rule in some spec does not apply to some specific situation that they're focused on, because the spec does not mention their specific situation. One example: someone suggested that we could make typed arrays returned by Web Audio APIs not be neuterable, arguing that this did not alter already-specified behavior because "nothing currently specifies how typed arrays work with Web Audio". Another example: someone argued that CSS does not specify how to determine the size of CSS content, padding and border boxes when the 'overflow' property is not 'visible'.
I hope it's clear that these are mistakes because when a Web spec states a general rule, that rule applies in all situations unless there is specific text contradicting it elsewhere (in which case, some spec should clearly indicate which rule prevails ... and I recognize that's not always true). Any other approach leads to madness; modularity and extensibility would be destroyed if, any time a feature is added to the Web platform, we had to explicitly reaffirm the behavior of all other features.
I think this is a form of the "frame problem" in AI, or "default logic". A kind of intertia requires previously specified rules to remain true until explicitly changed by another spec.
In typical programming languages, we tackle these issues by making every modifiable rule an explicit extension point (which could be implemented in lots of ways --- a virtual function call, a function value, an enum value, etc). Modifications explicitly utilize the extension point. This is sometimes done in specs, but rarely. One reason is that it makes your specs verbose and hard to read. Another reason is that it's hard to know the set of extension points ahead of time, so when you discover later you need to make modifications, you have to go back and add those extension points. That's getting easier now that more specs are using the "living standard" model, but it used to be hard.
There are programming languages that support this kind of dynamic extensibility. Languages with externally defined multimethods have a taste of it. CSS itself, ironically, orders rules by specificity and allows more-specific rules to override less-specific rules.
One problem I've been interested in for a long time is encoding the semantics of board games. Board game rules are often designed like Web specs, with some rules overriding others in special cases. Board games have extensibility problems, when expansions come out which lead to ambiguities in the context of the original game. (Munchkin makes sport of the problem.)
January 12, 2014 09:14 PM
Tiritiri Matangi Island
On Saturday our family took the ferry to Tiritiri Matangi Island for a day trip. Over the last 30 years farming stopped on this island and it was reforested, with intensive help from volunteers planting hundreds of thousands of trees and other plants. The island was also cleared of predators such as rats, never had many of the common New Zealand predators such as stoats and weasels, and is now an amazing habitat for native birds, many species of which are endangered on the mainland. Various rare species from other sanctuaries, such as takahes and little spotted kiwi, and even tuataras, have been introduced to Tiritiri Matangi. It is also an important nesting island for little blue penguins.
Our trip was excellent. We took a guided tour with one of the guides from the "Friends of Tiritiri Matangi" volunteers society. She was very knowledgeable and observant and the guided tour was well worth having. We saw a lot of rare birds, particularly saddlebacks, robins and whiteheads, and the birdsong was cacophonous. The bush is regenerating well. In open grass near the lighthouse, a highlight was seeing three takahe --- about 1% of the world's entire takahe population!
One of the best things about Tiritiri Matangi is that it is being used as a template for other Hauraki Gulf islands. In recent years, Rangitoto, Motutapu, Rotoroa and Motuhie have all been cleared of predators and reforestation of Motutapu, Rorotora and Motuhie (all formerly farmed) is in progress. Takahes have been released on Motutapu and kiwi on Motuhie. This will provide redundancy for the island sanctuaries and amazing places for people to visit when the forest grows up.
One interesting tidbit is that apparently rats can swim up to 3km and hence the 4km gap between Tiritiri Matangi is fortuitous. I wonder what would cause a rat to swim 3km towards an island ... desperation, insanity, or wanderlust and very good judgement of distance and tides?
Tiritiri Matangi is definitely a worthwhile day trip for anyone based in Auckland.
January 12, 2014 08:38 PM
For my entire life I've had the privilege of boating on the Hauraki Gulf. My parents are part-owners of a classic wooden launch built over 80 years ago, and throughout my childhood we spent weeks every year cruising the Gulf in it. They still take our family out with them for several days every year, for which we are very grateful. A lot of people, even those who live in Auckland, don't know what this entails so I thought I'd write about it.
The experience is a bit hard to describe to those who haven't done anything like it. The perimeter of the Hauraki Gulf is roughly the east coast of Auckland, running south and east to the Firth of Thames, then north along the inside of the Coromandel Peninsula, then north and west through the Great and Little Barrier Islands and rejoining the mainland at Tawharanui Peninsula. Its area is about 4000 km2. Inside that perimeter are many islands of various sizes, and a great many harbours and bays on the islands and the mainland. Even relatively small craft can go anywhere in this area. Often pleasure boats travel from bay to bay on a near-daily basis, anchoring in bays that are sheltered from any wind that's blowing. Common activities are fishing, swimming, going ashore via dinghy for swimming or walking, and generally just about any activity you can imagine doing on a boat, on a beach, or in a park --- much of the coastal land reachable by boat is public land of some kind, and you can almost always use the beach.
These boats are often equipped with everything you need to live for weeks --- sleeping, cooking, hot water, washing, etc. They're all different shapes and sizes, with the biggest difference being between the yachts that sail and the launches that rely on an engine, although yachts typically have a small backup engine too. These boats would have electricity generated from the engine and hence electric lighting, a refrigerator, and charging for your mobile devices. Most of the Gulf has pretty good 3G coverage.
Those sorts of boats are definitely an upper-middle-class luxury, but smaller boats down to large dinghies with outboard motors are used for day trips, especially for fishing. I read an estimate that one in three Auckland households have some kind of boat. A lot of Aucklanders live close enough to the water to make sailing classes a common extra-curricular activity for children; children learn to sail dinghies from the age of around 11. This partly explains why New Zealand produces so many world-class sailors.
If you don't have a boat, or have friends or relatives with a boat, it's very difficult to even approximate the pleasure cruising experience. If you're into sea kayaking you can kayak and camp at many of the closer islands. Many of the Gulf islands are reachable by ferry, but ferries to many of the less popular ones only run on specific days during the summer. Regardless, they're all worth visiting:
- Waiheke is basically a suburb at its western end, but a lovely island suburb with nice beaches and excellent walking tracks. Its eastern end is difficult to access, being largely private farmland, and you need a boat to really appreciate the bays of the "bottom end". People often visit Waiheke for the food and wine at its vineyards.
- Rangitoto is the newest volcano of the Auckland volcanic field. No more than several hundred years old, it's a unique ecosystem of plants colonizing bare lava, and the view from its top is exceptional. You can take the ferry from downtown, walk up and down, and be back in Auckland for a late lunch. This is my top recommendation if you have a half-day in Auckland.
- Motutapu is an older island next to Rangitoto, formerly used for farming. Has a campsite.
- Motuhie is a small island used as a POW camp in World War One. It has a campsite and the north-facing Ocean Beach is excellent.
- Kawau is at the northern end of the Hauraki Gulf. It has a lot of private land but the public area around Governer Grey's Mansion House is lovely.
- Rotoroa was a treatment center for alcoholics run by the Salvation Army which has recently reverted to a public reserve. It's nice.
- Tiritiri Matangi is a bird sanctuary. More about that in a future blog post.
- Great Barrier is a large island, reachable by light plane as well as ferry. Visitors will probably want to rent a car and it also has a network of walking tracks. It's at the edge of the Gulf, faces the Pacific and is more remote than these other islands. Highly recommended.
January 12, 2014 11:32 AM
December 26, 2013
Robert O'Callahan -- We Need A "Dumb Device" Movement
I'm habitually pessimistic about many things, and this year Snowden reinforced my habit. In the narrow sense of his obligations to the US government, he's a traitor, but to the human race as a whole he's a hero and a role model; he personally is inspiring, but what he revealed is depressing.
I think his most important lesson is that total surveillance is an explicit goal of the US and UK governments (and by extension other governments), and there's no real restraint in how that goal is being pursued, especially for those of us who aren't US citizens. Combine that with the cold truth that we are incapable of securing complex systems, and we're in a very bad situation. We have to start assuming that mass-market computing devices are compromised, or can be compromised at will.
When people talk about the "Internet of things", they're implying the situation is going to get much worse. Every device that is network-accessible and supports updateable software is a surveillance device ... if not all the time, then as soon as someone decides to turn it on. (Let's ignore for now devices that can be programmed to take hostile action against their users!) I am not in favor of the Internet of things in the present climate.
Unfortunately, factors of cost, convenience and cool will keep driving general-purpose, network-accessible computation into every nook and cranny of our world. It may help if a significant subset of customers (I hate the word "consumers", it's demeaning) prefer devices that don't have unnecessary computation jammed into them. I want to buy "dumb devices" --- meaning they are not unnecessarily smart, and don't talk about me behind my back. My refrigerator, clothes, and bicycle do not need network access or upgradeable software, and I don't want them. Of course, if my market segment's population is me, it's not economically viable. Therefore I need a mass movement.
One interesting product segment is cars. The computerization of cars is truly terrifying, and there is some great work detailing how modern cars can be subverted. I would pay a decent premium for a car that lacks any kind of over-the-air communications. A potential problem is that safety regulations require new cars to have sophisticated computers, and sooner or later a computationally secure car may become effectively illegal, if it isn't already.
I don't know what to do from here. Does this movement already exist? If not, I hope someone starts it, since I'm rather busy.
December 26, 2013 09:46 AM
December 16, 2013
Robert O'Callahan -- Blood Clot
I'm tagging this post with 'Mozilla' because many Mozilla people travel a lot.
In September, while in California for a week, I developed pain in my right calf. For a few days I thought it was a muscle niggle but after I got back to New Zealand it kept getting worse and my leg was swelling, so I went to my doctor, who diagnosed a blood clot. A scan confirmed that I had one but it was small and non-threatening. I went to a hospital, got a shot of the anticoagulant clexane to stop the clot growing, and then went home. The next day I was put on a schedule of regular rivaroxaban, an oral anticoagulant. Symptoms abated over the next few days and I haven't had a problem since. No side-effects either. I cut myself shaving at the Mozilla Summit and bled for hours, but that's more of an intended effect than a side effect :-).
I had heard of the dreaded DVT, but have never known anyone with it until now, and apparently the same is true for my friends. Fortunately I didn't have a full-blown DVT since the clot did not reach a "deep vein". Still, I can confirm these clots are real and some of the warnings about them are worth paying attention to :-).
I saw a specialist for a followup visit today. Apparently plane travel is not actually such a high risk so in my case it was probably just a contributing factor, along with other factors such as sitting around too much in my hotel room, possibly genetic factors, and probably some bad luck. (Being generally healthy is no sure protection; ironically, I got this clot when I'm fitter than ever before.) So when flying, getting up to walk around, wiggling your toes, and keeping fluid intake up are all worth doing. I used to do none of them :-).
My long term prognosis is completely fine as long as I take the above precautions, wear compressing socks on flights and take anticoagulant before flights. I will be given a blood test to screen for known genetic factors.
I did everything through the public health system and it worked very well. Like health systems everywhere, New Zealand's has its good and bad points, but overall I think it's good. At least it doesn't have the obvious flaws of the USA's system (the only other one I've used). In this case I had basically zero paperwork (signed a couple of forms that staff filled out), personal costs of about $20 (standard GP consultation), good care, reasonable wait times, and modern drugs. I was pleased to see sensible things being done to deliver care efficiently; for example, my treatment program was determined by a specialist nurse, who checked it out with a doctor over the phone. New Zealand has a central drug-buying agency, Pharmac, which is a great system for getting good deals from drug companies (which is why they keep trying to undermine it, via TPP most recently ... of course they managed to make the Pharmac approach illegal in the USA :-(.) Rivaroxaban is relatively new and not yet funded by Pharmac, but Bayer has been basically giving it away to try to encourage Pharmac to fund it.
Overall, blood clots can be nasty but I got off easy. Thanks God!
December 16, 2013 10:35 AM
December 07, 2013
Robert O'Callahan -- Why I Don't Worry About Global Warming (Much)
I don't worry about global warming or any other threat whose most important effects are several decades out. Technology is going to change everything by then: either we'll kill ourselves in more immediate ways, or at least destroy most of civilization (which would do a lot to reduce carbon emissions!) --- or we'll make a lot more technological progress, probably developing brain uploading, strong AI, or other game-changing capabilities we can't forsee yet. Probably the former.
People who worry about what will happen when the sun burns out in five billion years are the worst.
December 07, 2013 09:51 AM
One Day The Luddites Will Be Right
Whenever a person proposes that technological advances might reduce human job opportunities in the long term, someone responds with the Luddite Argument: "the Luddites thought the Industrial Revolution would destroy their jobs, and they were wrong, so you're wrong too" . Some go further and explain that the Luddites were wrong because technological productivity improvements are balanced by finding new uses for human labour. Wikipedia has a good summary. However, it seems obvious to me that at some point technological advance will --- or at least could --- be a net destroyer of jobs. All you have to do is imagine a world where robots can do everything a human can do, at lower cost than maintaining a human life. Clearly, there are no economically rational job opportunities for humans in that state , so at some point of technological advance short of that state, there's net job destruction.
The only question is whether and when we will reach that point. It seems inevitable we'll reach it unless something halts technological progress or some very strong flavor of Cartesian dualism holds. Economic arguments that human labour will still be worth something at that point are just wrong.
 Actually the Luddites were right; the Industrial Revolution did destroy their jobs, and drove them into misery. But they were wrong in that they did not forsee the net benefits to future generations.
 There could be sinecures to keep humans occupied, but they would not be economically motivated.
December 07, 2013 09:34 AM
December 06, 2013
Robert O'Callahan -- WebRTC And People-Oriented Communications
Tantek had an interesting blog post about making people rather than protocols the organizing principle of communication apps. I like his vision quite a lot. One neat extension of his post would be to introduce WebRTC. With WebRTC it would be relatively easy to have the "Robert O'Callahan" app check if I'm currently logged into the appropriate receiver app, at any WebRTC-capable endpoint, and if I am, establish a voice or voice+video session with me (with peer-to-peer transmission, naturally). If I'm not logged in, WebRTC lets you record a message for later delivery. This would be very cool.
December 06, 2013 01:53 AM
December 03, 2013
Robert O'Callahan -- Another Knee-Jerk Reaction To International Rankings
Predictably the OECD "Pisa" report ranking countries' education results has caused a stir in New Zealand. New Zealanders, or at least their news media, love international rankings of all kinds --- especially if they can be portrayed negatively for New Zealand. As often, the latest report, and the discussion around it, has some major problems.
For starters it's interesting to compare the initial NZ Herald story with the more nuanced reporting from the AP wire story. The Herald chose the completely fallacious headline Significant drops in NZ educational achievement --- fallacious because a drop in ranking does not necessarily mean a drop in actual achievement (and in this case, there is no evidence of a drop in achievement). (PPTA president Angela Roberts gets this right here.) The Herald story (and its followups) simply ignores the issues with the Pisa report which are touched on by the AP story, and better explained in Slate (mainly, it's invalid to compare city-states with entire countries, especially including Shanghai but no other part of China!).
Apart from that, some of the reactions to the report are ridiculous. There's this:
But Labour says any drop in the rankings should be sheeted home to an excessive focus by National on "testing" over the past five years. Although National Standards does not actually involve national testing, Labour's education spokesman Chris Hipkins said, "It shows that the last five years' focus on test-taking has been a disaster and it has actually narrowed the focus of our system and it has actually decreased the level of achievement within the education system".NZ's ranking drop is mainly due to Asian countries increasing theirs. Those countries' education systems are far more focused on testing than NZ's has ever been, which is probably why their ranking is increasing: if you focus on testing (and teach to the test), you do really well on tests, which is of course what studies like Pisa measure, since tests produce data and other educational activities don't. Chris Hipkins, if you really think Pisa is important you should advocate a big increase in national testing.
I actually think national standardized testing is important, but it's not the only important thing and the Asian education systems ranked highly in the Pisa report have massive problems despite producing good test results. My wife and many other people I know went through those systems and describe how they're focused on rote memorization and discourage any kind of learning other than school and after-school coaching on their core subjects. For example, I taught myself computer programming in my copious spare time after school, but for most children in Hong Kong that simply wouldn't have been possible. People talk about how in exams you "give it back to the teacher" --- cram for exams, do well, and then forget it as you prepare for the next one. I think it would be very interesting to re-test children three years after they left school to see what they've retained.
It's really important to avoid over-optimizing for the things we can measure at the cost of the things we can't as easily measure. It's also really important to not overreact to every international ranking report. We have to think critically about these things. I wish the media would.
December 03, 2013 10:33 PM
Last updated: April 20, 2014 03:15 PM