No fault of your own…

If you’ve ever taken a kernel trace of an application starting up on a kernel that is 6.3.2 or later, you might have noticed a thread state in your application called STATE_WAITPAGE.

To understand what is going on here, we have to first look at mmap() and how it allocates and initializes memory.

When you allocate memory with mmap(), it first allocates a virtual address range, and then (unless you’ve passed MAP_LAZY), it will allocate the physical memory needed to back that object.

What it doesn’t do, though is setup all the page table mappings for the mapping (although in actual fact it will setup a certain amount, depending on heuristics based on the type of mapping).

In actual fact it will wait until the program first accesses a page before setting up a page table entry for it.

This is a change from pre-6.3.2 kernels, where all mappings were setup immediately. You can re-enable the old behaviour by using the procnto option -mL in your buildfile.

The benefits of this scheme, known as demand paging, are a speedup when the application doesn’t actually access all of the pages in a mapping. This is quite useful when mapping executable objects, since often there are quite a few pages in the text section which may never be referenced.

In some patterns of usage, though, this can induce a performance penalty, especially when there are many threads running at the same priority.

This brings us back to STATE_WAITPAGE. When a process is executing and it access a page that has not been mapped, or tries to write to a page that is marked read-only, it will induce a page fault. This is caught by the kernel.

The fast path in the kernel will peek at the processes pagetables, and if found it will bang it into the TLB (this is done in hardware on some architectures, such as the x86). If it can’t find it then it needs to defer the work of handling the fault to the process manager, since it may need to do some complex work such as communicating with device drivers, and also the also the necessary structures may not be accessible/consistent.

This, then, is where the thread gets blocked in STATE_WAITPAGE, and a pulse is sent to procnto, to wake up a thread to handle the request.

The procnto worker does the dirty work of initializing the memory (it may need to read a page from a disk driver for example) and setting up the pagetables.

The first time a page is read it is setup with a read-only mapping, even if the mapping is writeable, unless the underlying page has already been marked as modified. For MAP_LAZY mappings, this is the point at which physical memory is actually allocated for the page. If the allocation fails, then the application will have a SIGBUS signal delivered to it (this was the same in pre 6.3.2 kernels).

This delayed initialisation supports two handy schemes.

This first is Copy On Write semantics (COW). This is where we don’t bother making a private copy of a page that was originally shared with another mapping until the page is modified.

The second is support for writeable mappings to files. Prior to 6.3.2 you could only map a file readonly, or with the extra flag, MAP_NOSYNCFILE. This was because there was no tracking of modified pages.

When you call msync() on a shared mapping to a file, all the modified pages are written to the backing store. Now the pages can be marked read-only again, and the modified indicator can be turned off.

Now all this is great, but what about that performance penalty? Well all that context switch can make the page fault processing take quite a while if there are lots of threads in STATE_READY at the same priority. The procnto thread will be placed at the back of the queue, wait for the others to run, the it will run, potentially talk to a device driver, and then make the original thread ready again, which will be placed at the end of the queue.

Another thing to think of in some circumstances you want to take the hit of talking to the backing store all at once for determinism reasons.

A way to control this behaviour is via the mlock() and mlockall() functions. These tell procnto that you want the some (or all) pages made memory resident right now.

This means that the mapping will have been fully read in from disk by the time the mmap() call returns to your program.

You still get page faults on the first write to a page, though. In that case we setup a read-only mapping (or if the underlying pages have already been modified, a read-write mapping).

If you TRULY don’t want page faults for mappings, then there are two options open to you. Since device drivers may have their code run in an interrupt handler, and they may also be running code with interrupt disabled, they can’t take page faults at all. So when you request IO privity with the ThreadCtl() call, you get marked as being super locked. This means that all mappings are setup and ready to go.

Of course, only processes running with superuser privilege can request this special status. To have all processes be marked as super locked, you can pass the -mL option to procnto. To have all processes simply be locked (the equivalent of calling mlockall(MCL_FUTURE|MCL_CURRENT), you can pass the -ml option.

Well, this post has gone all over the map <groan>, so I’m going to sign off.

Later,

Colin

It’s alive!

Well sorry for not having written for so long. We’ve all been very busy working on the 6.4 kernel.

BTW – if you’ve been wondering about the being CoreOS source repository out of date, the sync process between the internal and external repositories broke down. :-(

The good news though is that it’s fixed now, so you can get all the goodness straight from the source. :-)

Colin

Work your Pocketgeek!

In the spirit of open development, QNX has created a new product – the Pocket Geek! He works in transparent cube (get it, transparent development!!)

Head over to http://thepocketgeek.com for some recreational fun, trying to drive your little pocketgeek to complete his tasks without exploding. There’s also a contest at the end of it all.

Have fun!

Colin

Happy Holidays! Colin’s in Charge!

Most of QNX is off this week on a yearly holiday break that lasts from Christmas to New Years.  

Most of the developers end up being busy playing with new toys (thanks Santa) or tinkering with pet projects (CDT integration with Mylar) that they’ve had brewing on the side, so January ends up being a pretty exciting time to be back in the office finding out what everyone’s been up to.

However, this year I’m going to miss out on that.  Starting in January I’m turning main responsability of SRR over to my partner in crime Colin as I’m starting three months of parental leave to hang out with my 9 month old son and teach him the essentials of life:

• Recursion, can’t program effectively without thinking it!
• Snow fort construction, can’t live in Ottawa without it!
• Appreciation for solid foods … just makes sense!

So as this is my last post for a while (likely, but not certainly) I want to wish you all a happy holiday and a wonderful new year!

Thomas

Why wait for the next release …

OK … so I promise that this will be my last post about foundry27for a while and I’ll get back to talking about QNX technology.  But I couldn’t help myself, I needed to put out one more plug for some of the QNX Neutrino software that is being made more accessible to developers.

This time it is the availability of the IDE, also commonly known as QNX Momentics, although that statement will get me in trouble since QNX Momentics is more than just the IDE, but it is the complete assembly of a development environment for QNX Neutrino … but I digress.

The IDE is not part of the current roadmap for source publishing, but the QNX tools development team is shifting to performing its development work more transparently.  To that end there are two community projects, the Core Development Tools project and the the Integrated Development Environment (IDE) project.  From the Core Development Tools project you can get the bleeding edge versions of the GNU toolchain as they are developed … which is interesting, but most commercial projects currently under development aren’t likely to switch to a non-released (by QNX) compiler for their production work.   The IDE also offers bleeding edge integration build downloads and since the IDE pretty much sits on top of any core tool or runtime components, it can be updated with far less risk to a production development environment.   While I hate to compare the IDE to an editor, updating your IDE is like updating your editor:  In theory new features and functionality should only make you more productive!

The IDE team is tracking what significant new and noteworthy features are being added to each of the integration builds so you can get an immediate idea of what areas you might want to explore.  A more visual page is alsotracking the general IDE 4.5 features that you might also want to take a look at.  There is a lot of goodness in these downloads, ranging from the source navigation and usability improvements in the C/C++ development environment to the brand new Application Profiler functionality and a range of smaller, but equally rewarding, fixes and enhancements.

The other benefit to trying it out early is that the IDE team is actively looking for feedback and there is still time in the schedule before the next major release … codenamed Athens … to respond to feedback and feature requests.

Of course I’m a bit biased since I’ve had a hand in putting some of the new IDE features in, but I’ve switched for my C/C++ development and I’m not going back!

Thomas