After many months of hard work the developers at Crank Software (which includes myself) are ready to release Storyboard Suite 1.0, a multi-operating system, multi-platform designer driven graphical application builder.

At this point you may be either bored or intrigued … in either case you should go and start downloading your free version of Storyboard Suite now!  If you are curious and don’t want to read … go watch a Storyboard video.

Still reading?  Great … now let’s talk about why this is great for QNX developers.  Up to this point if you wanted to build single mode graphical applications on top of Advanced Graphics TDK (aka GF aka Core Graphics) you had a few different options:

1. Use Flash.  Flash is an awesome platform/tool and you can build incredibly creative and eye popping applications using all of the power and features that Adobe has provided, with a ready pool of skilled Flash developers at the ready … but sometimes Flash is just too big a solution when all you want are a couple of screens and some data displays.
2. Use the rich C API’s and a (ported) widget library.  This isn’t impossible, but it also means that your software developers are the ones doing the user interface development (and porting) and re-interpreting the graphical design that some artist has created for the product.  This is time consuming and in my experience the software developer rarely gets the vision the graphical designer had in mind when he build the application screens.

Crank Storyboard Suite is composed of two pieces that give you an awesome third alternative:

Storyboard Designer is an Eclipse based graphical application development environment.  It is based on a application hierarchy composed of screens, layers and controls and is driven by a simple, but extensible, event model.  Visual content comes directly from designers as either direct image imports including a sweet PSD file importer that can transform Photoshop layer elements directly into an application’s screen, layers and controls.

The tool exports a data bundle which is used as input for the  Storyboard Embedded Engine, the runtime component that lives on your embedded system.  The engine is easily configurable and plugin based so that it is extensible to any new technologies or trends that appear in the market.

… did I mention that there are videos that you can watch?

By now your download should be finished, so go and give it a whirl.  A few simple examples are included along with a simulator that allows you to run applications you build on your native system if you don’t have an embedded target handy, and if you’ve got questions … well we’ve got forums to ask them in.

With 1.0 now out the door 1.1 (or 1.0.1?) is coming along quick with a focus on user driven feedback and enhancements!  Things like providing additional  WebKit integration for various customers and making simple graphical transforms even easier .. oh and I thought I saw an OpenGLES renderer running in the office too!

Thomas

Note: I didn’t include Photon in the above list, since really if you are using Photon you’ve got a ready solution for building applications, widgets and an application builder at the ready.  It isn’t likely that you graphics designers will find it an environment they want to work in, but they could learn it if they wanted to.

One of the ways in which we highlighted this performance issue was to take advantage of the System Profiler’s Trace Search and the new Path Manager trace events that are available in the 6.4.0 Neutrino kernel.   This search gives us a quick view into what files are being opened up

These search results give us very usefull information, but they are kind of isolated.  One of the neat things that the Trace Search dialog allows you to do is to combine search conditions together using the && and || operators to AND and OR conditions which can give your search results a bit more of an impact.  

For example, if we wanted to get a rough estimate of the kind of effect that each of these file opens was having on our system, then we might do a compound search joining both the FileOpen information with the number of times that we perform a ConnectAttach to actually establish a connection to a server:

Then enter a compound condition:

Now when we do our search, we get a much more interesting set of results that gives us an indication of just how many different servers might be involved when we are doing all of this additional file access:

I’ve cut out some of the columns here to try and fit the content in, but we can now see the FileOpen results with the ConnectAttach() calls inter-mingled.  The handy thing with this information is that you can now quickly establish a few cost metrics based on the observed results.  This can be handy when you are trying to determine what areas may be worth persuing for further optimization gains.

Happy Optimizing.

Thomas

As a follow up to the Crank WebKit performance “shark week” we decided that since runtime performance was starting to look pretty good we should turn our sights to the memory footprint that the WebKit port was taking on our systems.  

Due to the fact that this was a brand new browser engine (WebKit), with a minimal custom front end (Crank Launcher), ported to a brand new graphics framework from QNX (AG/GF), there was really no “apples to apples” comparable to use (though Crank does have many internal versions of Firefox).  This isn’t as bad as it seems … in the world of software, attempting to “match” what previous software generations did rarely helps to advance the state of the technology.  Our goal was to take a fixed block of time and make the biggest impact that we could.

We took base measurements off many different sites, but here I’ll use the plain old www.google.com as the reference.  When we loaded that site up, after all of the shared objects were loaded and we finished rendering the page, we found that we would often hit close to 30M of memory being used!  That was a pretty outrageous number.

There are lots of different tools for memory analysis, most of which don’t scale appropriately or provide accurate enough results when used on an application as disjoint as WebKit:  It uses both C and C++ object smart pointers, it mixes custom allocators and the system allocator (malloc) and the usage and behaviour of memory allocation is completely different for each of the shared libraries used (ie libc, libxml, freetype, icu, libjpeg, …).

Since we were coming off the week of CPU performance analysis we had lots of different trace files hanging around so we decided to take a crank at them with the System Profiler.   In addition to the new Path Manager event in 6.4.0, there are two system events for tracking mmap events … one for named events and one for un-named events. 

We set up some traces, using the same technique as with the Path Manager events, and leveraged that information to give us some insight to what major allocations (those that routed to mmap()) were occuring and correspondingly to see what shared libraries, files and other objects were being mmap’ed by WebKit. 

This tuning once again pointed us to the font handing  and some less than optimal code in the way that WebKit uses the FreeType and FontConfig API’s.  After last week’s CPU tuning experience with the font configuration, I’m rapidly coming to the conclusion that fonts and internationalization is nothing but a work generator!

Just for confirmation, we dropped the entire application into the debugger, put a breakpoint on the mmap() function and stepped through the major WebKit operations all while double verifying using pidin mem  to see the memory effect on the system.

To make a short story long, we concluded that there is no reason to map in Asian fonts, especially at 6M a pop, if you have no characters on screen that require such functionality!  Not a revelation that most people would be knocked over by, but the effect on this software was.  With a little bit of tuning around the fonts and a few other memory areas, our memory footprint rocketted down to a very respectable (and stable) 4M when we target www.google.com. 

… and of course to speak to the title, when you drop from 32M to 4M (the less) we pick up a few extra CPU cycles (the more) in the general operation.  We’ve got a few more tricks up our sleeve but we’re getting very close to our GF release point!

Thomas

If you are running an embedded system though, filesystem access is your nemesis.  Not only does the media that you are interfacing with have generally slower access times, but the environment much more demanding.  You get the double whammy of fewer CPU cycles to spare and little or no additional cache memory to compensate for slow device access.  All that, and you’ve got some pesky power monitor that  keeps trying to get the system to idle so it can sleep and save a few milliwatts of power.

The customers that have approached Crank software about WebKit integration all have embedded systems that they are running, so as part of my WebKit optimization week I started out with the easy targets: Excessive filesystem access. 

Using the System Profiler (a handy hammer for all my problems) and QNX 6.4.0 it is dead easy to highlight those expensive pathname based filesystem operations  (open,opendir,stat etc):

1. Open a trace log file and open the search dialog; Search > Search or just  Ctrl+H in the editor. 
2. Select the Trace Search tab icon and select Add to create a new condition
3. Create the condition using the System class and the Path Manager code

Now if you use this condition you will see all of the path based file operations.  You can use the data fields in the event such as process or pid to filter the events down to what you are interested in.  Running this query, focusing in on the the WebKit based application as it launched against a test website yeilded:

Unbelievable!  Considering that this was a trace that lasted only 30 seconds, 2100file accesses seems to be a little bit unreasonable and either a bug or an area ripe for optimization.  The pathname provided us with lots of insight into what was happening (lots and lots of font access).  Our current port uses FontConfig to manage the font mappings and FreeType to perform the rendering. 

Our first change was to create a more ‘embedded friendly’ font configuration file.  By default the font configuration is scanned every few seconds to support dynamic font addition and removal.  Usefull for desktops, but not needed by most embedded systems.  Doing that, we picked up a few seconds of improvement, but were still churning. 

Time to dive into the code and correlate it with our trace results:

The traces showed repetitive file access for the same font, so we added a simple filesystem name cache and used FreeType’s internal cache to avoid this churn.  This dropped the file accesses by about an eighth and picked up another few seconds of improvement.  Not bad for a bit of effort, but not the big gain we were looking for.

The traces still showed that we were hitting the font configuration directories several times over, definitely not the intention of the source as far as we could tell.  After a day of code inspection, the culprit turned out to be an innocent looking routine that was responsible for cleaning up temporary font resources … unfortunately the cleanup also destroyed a static font configuration, causing it to be re-created each time a new font request was made!  Fixing that bug and re-running our test load:

Now that’s looking better!  While the number of file accesses is still high (318) it isn’t completely ridiculous (2100) and even better, most of those initial accesses are the shared library loading and don’t occur during the steady state operations of the browser.  The even better news is the time savings that came with this reduction … it dropped a full ten (10) seconds off the general load time!  We’ve totally moving from Super Sloppy to Super Shiny!  (thanks Mario and Paul!).

If you want to get more details on the WebKit port, we’ve just opened up the Crank Software blog where we’re planning on posting updates directly there as we make progress and make it available.

Thomas

Since we’re building WebKit ourselves, I was able to add in a number of user trace events that had specific meaning for the WebKit performance metrics we were looking at.  With a few events, it made repetitive measurements of things like application load time, page load time and network latency a snap to calculate.

Prior to the 6.4.0 release, if you wanted to add in your own custom trace events you had to use the TraceEvent() API.  This API does way more than just insert trace events, it is the swiss army knife of calls to configure and control the entire kernel instrumentation system.  At the end of the day, I was always having to go back to the documentation just to double check the arguments required to push out an event with a string in it:

TraceEvent(_NTO_TRACE_INSERTUSRSTREVENT, <id>, <string>)

Of course if you wanted anything more than a simple string, you had to fiddle with sprintf’s, allocate buffers etc.  A number of times I simply would insert five or six trace events in a row rather than doing all that extra work … which of course meant havin log files cluttered with extra events I eventually needed to filter out to see what I wanted.

With 6.4.0 a whole slew of trace_* functions were added into the <sys/trace.h> header.  Now instead of having to remember the specific define for the TraceEvent() call, I can just do:

 trace_logf(<id>, <printf style format string>);

That is way easier and far more convenient … makes tracing almost as much fun as using my ShamWow! 

Unfortunately the functions aren’t documented yet, but they were modelled using the same style as the slog_* functions, and you have to be carefull about their use within interrupt handlers but things like the trace_here() function can offer interesting insight using the other optimization tools.

… and in case you were wondering about the WebKit optimization results, they are looking good.  We’ve eliminated most of the superfluous work that hit embedded systems (especially non-x86 targets) harder than desktops and hope to be making a very usable GF based version available soon!

Happy Tracing!

Thomas

Lots has gone on since my last post on SRR.  After finishing up my parental leave last year  I decided that after working at QNX for nearly 10 years it was time for a change and took on a role at Crank Software.

We’re doing a lot of interesting things at Crank, mostly to do with graphics and improving the way people integrate graphical content into their embedded products. 

Crank is doing a lot of work with customers who base their user interface designs on what Apple’s iPhone and iTouchcan do, but want to do it on systems with less powerful CPU’s, smaller memory capacity and less capable graphics engines.

Oh yeah … and for those devices that are network connected they also want to run WebKit, the engine under Apple’s Safari web browser. 

As part of addressing these needs (better, faster, smaller) Crank has ported WebKit to QNX Neutrino, and since web browsers and graphical applications go hand in hand these days, we plan to provide assistance and support on this technology. 

If you are keen, you can try out an advance pre-release version of the WebKit engine by downloading it from  http://www.cranksoftware.com/pre-release/.

This is a snapshot of our development from a month ago after we got the initial port running and passing the basic browser tests, lots of improvements have been made since then and we’ll update it when we hit a good stability point.

Our initial ports run on both Neutrino 6.3.2 and 6.4.0.  We currently have Photon microGUI versions and expect to have an AGTDK based version out in the coming weeks.

The development on WebKit is very active, with lots of work going on in all areas from user interface to scalability to scripting performance.  Getting the initial port up and running wasn’t a trivial amount of work, and we hope to work with the WebKit developers to back port our changes. 

In the mean time, if you run Neutrino and wanted to be like all the other cool kids out there running WebKit go check out the pre-release and let us know how it works for you! 

Thomas

You may recall that ARM v5 chips have a somewhat icky cache implementation, resulting in an unfortunate limititation in the virtual address range per process, and also a limitation in the number of processes in the system.  It’s all described here, if you are interested.

The really cool news is that the v6 architecture doesn’t have this problem (it has a physically tagged cache), and thus the limitation has been removed (read all about it here!).

Not only that, but the gcc 4.2.4 compiler is much better for ARM, resulting in much better code generation.

QNX 6.4.0 now also supports the Vector Floating Point unit – which can make a big difference – read about it here.

Last but not least, the ARM kernel and Image Filesystem are now mapped with large 1Mb pages.  This really boosts the performance of kernel calls – since we are using way fewer TLBs.

The fact that 6.4.0 supports large pages is also important for userland programs, especially on ARM – I’ll get into that in another post.

Colin

I’ll be posting about some of the new features in 6.4.0 – but how about first you go and download it!

http://www.qnx.com/products/index_sdp_640.html

Yes, it’s really “bleeding edge”, though a lot of benifit and fun of running the HEAD branch stack, one of the disadvantage is, while in it’s early stage, the stack “crashes”. The good thing is I have the core dump I could look at, but the bad thing is, that’s also when my kids started shutting at me.

Those of you who had been managed a home network, would really understand how stressful this is. :) Fortunatly, soon my kids find out the “engineering way” to fix the problem. They went down to the basement, press the little reset button on the old Pentinum, give it a couple of minutes, and wola, everything comes back.

This works well for a while, but one day while I was at home along, the stack on gateway gone again. I have to get out of my comfort couch, went down to the basement and reset it myself. I said to myself, “why can’t I just write a program to resetart the network if it’s crashed”, after all, QNX is all about Micro Kernel and Modular System, isn’t it?

That’s where my “sockmon” program cames from. Once started, it will keeps on monitoring if TCPIP stack is still running, if it disappered, “sockmon” will try to execute a shell script you gave it on command line, to re-start the network. If the restart somehow failed after some try, then it will just reboot the system.

You may wonder “how do you know if TCPIP stack is there or not”? Well, QNX resource manager have builtin notification to all connected clients if their server disappeared. So all you need is to establish a connection to the tcpip stack (by call socket()), and setup to waitfor the notification events.

I have include the source here, the “notification” thing above is true for ALL resource manager (unless the manager is written in such way that turned off this feature), so you can easilly extended my program to any resource manager. Just give it a config file to read about which resource manager (what namespace you care) to watch out, and what to do (which script to execute) if the manager went away.

I will leave this for reader exercise, but if you did that, you would realiz you just got yourself a simple, basic, HA program.

 -xtang

#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <process.h>
#include <signal.h>
#include <string.h>
#include <syslog.h>
#include <sys/procmgr.h>
#include <sys/sysmgr.h>
#include <sys/neutrino.h>
#include <sys/socket.h>

int main(int argc, char **argv)
{
        char *script;
        int sd, chid, fcount;
        struct _pulse pulse;

        if (argc < 2) {
                fprintf(stderr, “sockmon <re-start script>\n”);
                return -1;
        }

        script = argv[1];
        if (access(script, X_OK) != 0) {
                fprintf(stderr, “access(‘%s’): %s\n”, script, strerror(errno));
                return -1;
        }

        /* creat a channel for accept COIDDEATH pulse */
        if ((chid = ChannelCreate(_NTO_CHF_COID_DISCONNECT)) == -1) {
                perror(“ChannelCreate”);
                return -1;
        }

        /* don’t care about the child */
        signal(SIGCHLD, SIG_IGN);

#ifdef NDEBUG
        if (procmgr_daemon(0, PROCMGR_DAEMON_NOCLOSE) == -1) {
                perror(“procmgr_daemon”);
                return -1;
        }
#endif

        openlog(“sockmon”, LOG_PID, LOG_DAEMON);
        setlogmask(LOG_UPTO(LOG_INFO));

        for (;;)
        {
                fcount = 0;

                /* connect to tcpip to monitoring, give it 30 seconds, if still can’t
                 * connect, reboot the system
                 */
                while ((sd = socket(AF_INET, SOCK_DGRAM, 0)) == -1) {
                        if (++fcount >= 6) {
                                syslog(LOG_ERR, “Can’t connect to socket after 3 minutes, reboot…”
);
                                spawnl(P_NOWAIT, “/bin/slay”, “slay”, “-f”, “syslogd”, NULL);
                                sleep(1);
                                sysmgr_reboot();
                                return 0;
                        }
                        sleep(5);
                        syslog(LOG_INFO, “Connect to Socket failed: %m”);
                }

                syslog(LOG_INFO, “Connected to Network, start monitoring…”);
                if (MsgReceivePulse(chid, &pulse, sizeof(pulse), NULL) == -1) {
                        syslog(LOG_ERR, “MsgReceivePulse(): %m”);
                        return -1;
                }

                if (pulse.code != _PULSE_CODE_COIDDEATH) {
                        syslog(LOG_ERR, “MsgReceivePulse(): %m”);
                        return -1;
                }

                if (pulse.value.sival_int != sd) {
                        syslog(LOG_ERR, “COIDDEATH pulse for %d\n”, pulse.value);
                        continue;
                }

                syslog(LOG_INFO, “Network gone, restarting…”);
                spawnl(P_WAIT, “/bin/ksh”, “/bin/ksh”, script, NULL);
        }

        return 0;
}

I’m slowing working on a more thorough port, but the prototype is a nice forum to play around in.

Have fun…

Colin

To understand what is going on here, we have to first look at mmap() and how it allocates and initializes memory.

When you allocate memory with mmap(), it first allocates a virtual address range, and then (unless you’ve passed MAP_LAZY), it will allocate the physical memory needed to back that object.

What it doesn’t do, though is setup all the page table mappings for the mapping (although in actual fact it will setup a certain amount, depending on heuristics based on the type of mapping).

In actual fact it will wait until the program first accesses a page before setting up a page table entry for it.

This is a change from pre-6.3.2 kernels, where all mappings were setup immediately. You can re-enable the old behaviour by using the procnto option -mL in your buildfile.

The benefits of this scheme, known as demand paging, are a speedup when the application doesn’t actually access all of the pages in a mapping. This is quite useful when mapping executable objects, since often there are quite a few pages in the text section which may never be referenced.

In some patterns of usage, though, this can induce a performance penalty, especially when there are many threads running at the same priority.

This brings us back to STATE_WAITPAGE. When a process is executing and it access a page that has not been mapped, or tries to write to a page that is marked read-only, it will induce a page fault. This is caught by the kernel.

The fast path in the kernel will peek at the processes pagetables, and if found it will bang it into the TLB (this is done in hardware on some architectures, such as the x86). If it can’t find it then it needs to defer the work of handling the fault to the process manager, since it may need to do some complex work such as communicating with device drivers, and also the also the necessary structures may not be accessible/consistent.

This, then, is where the thread gets blocked in STATE_WAITPAGE, and a pulse is sent to procnto, to wake up a thread to handle the request.

The procnto worker does the dirty work of initializing the memory (it may need to read a page from a disk driver for example) and setting up the pagetables.

The first time a page is read it is setup with a read-only mapping, even if the mapping is writeable, unless the underlying page has already been marked as modified. For MAP_LAZY mappings, this is the point at which physical memory is actually allocated for the page. If the allocation fails, then the application will have a SIGBUS signal delivered to it (this was the same in pre 6.3.2 kernels).

This delayed initialisation supports two handy schemes.

This first is Copy On Write semantics (COW). This is where we don’t bother making a private copy of a page that was originally shared with another mapping until the page is modified.

The second is support for writeable mappings to files. Prior to 6.3.2 you could only map a file readonly, or with the extra flag, MAP_NOSYNCFILE. This was because there was no tracking of modified pages.

When you call msync() on a shared mapping to a file, all the modified pages are written to the backing store. Now the pages can be marked read-only again, and the modified indicator can be turned off.

Now all this is great, but what about that performance penalty? Well all that context switch can make the page fault processing take quite a while if there are lots of threads in STATE_READY at the same priority. The procnto thread will be placed at the back of the queue, wait for the others to run, the it will run, potentially talk to a device driver, and then make the original thread ready again, which will be placed at the end of the queue.

Another thing to think of in some circumstances you want to take the hit of talking to the backing store all at once for determinism reasons.

A way to control this behaviour is via the mlock() and mlockall() functions. These tell procnto that you want the some (or all) pages made memory resident right now.

This means that the mapping will have been fully read in from disk by the time the mmap() call returns to your program.

You still get page faults on the first write to a page, though. In that case we setup a read-only mapping (or if the underlying pages have already been modified, a read-write mapping).

If you TRULY don’t want page faults for mappings, then there are two options open to you. Since device drivers may have their code run in an interrupt handler, and they may also be running code with interrupt disabled, they can’t take page faults at all. So when you request IO privity with the ThreadCtl() call, you get marked as being super locked. This means that all mappings are setup and ready to go.

Of course, only processes running with superuser privilege can request this special status. To have all processes be marked as super locked, you can pass the -mL option to procnto. To have all processes simply be locked (the equivalent of calling mlockall(MCL_FUTURE|MCL_CURRENT), you can pass the -ml option.

Well, this post has gone all over the map <groan>, so I’m going to sign off.

Later,

Colin

BTW – if you’ve been wondering about the being CoreOS source repository out of date, the sync process between the internal and external repositories broke down. :-(

The good news though is that it’s fixed now, so you can get all the goodness straight from the source. :-)

Colin

I was dismayed, however, to read this blog entry by one of dtrace’s inventors, Adam Leventhal, on a discovery he had made about the Leopard port.

I won’t repeat the details of it here, but the gist of it is that Apple allows for certain processes to be hidden from any debug/tracing systems. While you may or may not agree on whether that is Truly Evil Behaviour or not, it doesn’t really matter – what it means for sure is that the Leopard dtrace port is severely broken.

When a system wide tracer cannot trace the whole system, it pretty much ceases to be of any use!

You might as well pack up, and go home, folks!

The really sad thing is the way they broke it! They basically disable any probes from running at the same time as a protected process, the prime example being iTunes.

This means, for example, that anything based on the timer tick is useless, since you lose the regular heartbeat, and that any kernel activity that may have been deferred (say disk io) is lost, unnoticed by your dtrace probes!

It really means that you cannot trust any of the data that dtrace is supplying, as it may be at best incomplete, at worst incorrect.

I shake my head, I really do.

Colin

Head over to http://thepocketgeek.com for some recreational fun, trying to drive your little pocketgeek to complete his tasks without exploding. There’s also a contest at the end of it all.

Have fun!

Colin

Most of the developers end up being busy playing with new toys (thanks Santa) or tinkering with pet projects (CDT integration with Mylar) that they’ve had brewing on the side, so January ends up being a pretty exciting time to be back in the office finding out what everyone’s been up to.

However, this year I’m going to miss out on that.  Starting in January I’m turning main responsability of SRR over to my partner in crime Colin as I’m starting three months of parental leave to hang out with my 9 month old son and teach him the essentials of life:

• Recursion, can’t program effectively without thinking it!
• Snow fort construction, can’t live in Ottawa without it!
• Appreciation for solid foods … just makes sense!

So as this is my last post for a while (likely, but not certainly) I want to wish you all a happy holiday and a wonderful new year!

Thomas

This time it is the availability of the IDE, also commonly known as QNX Momentics, although that statement will get me in trouble since QNX Momentics is more than just the IDE, but it is the complete assembly of a development environment for QNX Neutrino … but I digress.

The IDE is not part of the current roadmap for source publishing, but the QNX tools development team is shifting to performing its development work more transparently.  To that end there are two community projects, the Core Development Tools project and the the Integrated Development Environment (IDE) project.  From the Core Development Tools project you can get the bleeding edge versions of the GNU toolchain as they are developed … which is interesting, but most commercial projects currently under development aren’t likely to switch to a non-released (by QNX) compiler for their production work.   The IDE also offers bleeding edge integration build downloads and since the IDE pretty much sits on top of any core tool or runtime components, it can be updated with far less risk to a production development environment.   While I hate to compare the IDE to an editor, updating your IDE is like updating your editor:  In theory new features and functionality should only make you more productive!

The IDE team is tracking what significant new and noteworthy features are being added to each of the integration builds so you can get an immediate idea of what areas you might want to explore.  A more visual page is alsotracking the general IDE 4.5 features that you might also want to take a look at.  There is a lot of goodness in these downloads, ranging from the source navigation and usability improvements in the C/C++ development environment to the brand new Application Profiler functionality and a range of smaller, but equally rewarding, fixes and enhancements.

The other benefit to trying it out early is that the IDE team is actively looking for feedback and there is still time in the schedule before the next major release … codenamed Athens … to respond to feedback and feature requests.

Of course I’m a bit biased since I’ve had a hand in putting some of the new IDE features in, but I’ve switched for my C/C++ development and I’m not going back!

Thomas

Having ported software on a self-hosted system is pretty much essential for a software developer to work effectively.  There are so many essential software bits and pieces that QNX would never include with the Neutrino SDK because while they are useful, the SDK is targeted specifically at what is needed to build Neutrino applications and needs to be portable (and consistent) for use under Windows, Linux, Solaris as well as Neutrino self-hosted x86.   That doesn’t mean that the extra software bits aren’t useful, and historically a 3rd party CD was provided for Neutrino self-hosted developers that came with loads of useful extras, from scripting languages (perl, python, tk) to web servers (apache) and editors (vim).

A couple of developers here at QNX, Sean B. and Xiaodan T., with the help of some QNX community members have been working on a wicked awesome project that pretty much replaces the 3rd party CD with something even better!  They have been working to port the BSD pkgsrc tool/project to support the QNX Neutrino self-hosted environment.   Just the other day I got a message in my mailbox indicating that the project had hit a critical milestone and that QNX Neutrino was now considered a full fledged supported platform:

It is even listed on the projects main documentation page … now that is super cool news!  There are 7000+ different software packages part of this collection, and to have all of them available pre-ported and on-demand is a pretty awesome feat.

So if you are running Neutrino self-hosted, (and why wouldn’t you be!) and you want more than just the stock installed software, then you should definitely take a look at the pkgsrc community project. 

Congratulations and thanks to Sean, Xiaodan and all the others who’ve been working on getting this up and running!

Thomas

After looking at the trace it seemed to be sane, but the behaviour shown was somewhat puzzling. However it is explainable, and it has to do with our implementation of mutexes.

No kernel calls?

One of the first things you must realize, when looking at a QNX kernel trace, is that a pthread_mutex_lock() doesn’t always result in a SyncMutexLock() kernel call. This is because we implement an optimisation where an uncontested mutex can be aquired without entry to the kernel. Likewise, unlocking a mutex on which no-one is waiting will not result in a SyncMutexUnlock() kernel call. This is implemented using atomic compare-and-exchange operations. How they are implemented depends on which cpu you are using. Take a look at $QNX_TARGET/usr/include/<cpu>/smpcmp.h if you are interested in the various ways of doing this.

Not our man though…

State changes without seeing kernel calls might point us in that direction then, but the fact is that we are seeing a change to STATE_MUTEX, which would indicate a contested mutex.

The next clue though is that we ARE, in fact, seeing these state changes while in the kernel. The kernel call was a MsgReceivev(). This led me to the answer.

When we unlock a mutex that other threads are being waited on, we call SyncMutexUnlock(), which checks a priority ordered list of waiting threads. It then will remove the first thread from the list and mark it runnable (ie it will transition to STATE_READY). It also sets a thread flag, _NTO_TF_ACQUIRE_MUTEX, which indicates that the thread should aquire the mutex it was waiting on upon exit from the kernel. The address of the mutex was saved in the thread kernel calls arguments when it called SyncMutexLock(), some time in the past.

The point is, though, that the thread has not yet aquired the mutex. This can lead to some interesting behaviour.

In the log that was sent to me, the holding thread unlocked the mutex, and then immediately tried to relock it. The first unlock makes a waiting thread ready. SyncMutexUnlock() is not a blocking call though, so the thread remains running.

At this point, no-one holds the lock, although the lock indicates that people are waiting on it.

The following call to SyncMutexLock detects that there are threads waiting, and readies another thread (it has no knowledge that there was already a thread readied) to attempt to acquire the mutex. It then blocks the calling thread, and at this point the first thread that wanted to aquire the mutex will be made running.

Time passes, at at some point the second thread that was marked ready may actually be made running. As it exits the kernel, the kernel exit code processing notices the aquire flag is set on the thread, and attempts to aquire the mutex. Of course it is still held by the earlier thread, and thus we need to block this thread and choose another.

Capice?

And there you have it. I’m sure it’s all a little odd without better knowledge of how kernel is implemented. The good news is that you can checkout the source and see for yourself. Also don’t forget to checkout the evolving docs on the project wiki page.

The moral of this story is to not get too upset by multiple thread state changes when we are within the kernel. The kernel can change it’s mind several times on which thread will ultimately make it out of the kernel and become running. The last state change before you see a kernel exit is the one to believe.

Colin

I’m jazzed about it too … but for a different reason all together. 

Colin and I have been following the work of Bryan, Mike and Adamon dtrace since it was released as part of the Open Solaris.  We head up most of the kernel instrumentation and system profilingdevelopment on Neutrino and have always felt that dtrace, with its focus on runtime safety and use in a production environment, would be a perfect match for embedded developers and compliment our existing instrumentation and tracing strategy for Neutrino very nicely.  

So why were we jazzed by the Max OS X release? The more platforms that dtrace is ported to, the more generic the code base will become.  It is exciting to us because now dtrace is running on Solaris, NetBSD, Max OS X and oh yeah … Neutrino!

You can see here that this is a system that is running the 6.3.0 release of Neutrino and this is the basic dtrace command help.  The initial port to Neutrino that Colin and I have done is slightly different from the other OS ports.  Since Neutrino is an micro-kernel, we wanted to see how far we could go keeping the dtrace module out  of the kernel … and if possible even outside of the process manager.  The current implementation has all of the dtrace system code, normally found in the kernel or a kernel module, encapsulated in a resource manager, io-dtrace, and the utility is a straight port of the Solaris utility.

We’ve only implemented three sets of providers currently. 

• dtrace:  This is the standard dtrace provider with the BEGIN, END and ERROR probes
• instr: This is a provider based on the existing kernel instrumentation hook and can provide kernel calls, thread state, interrupt information
• profile: This provider is pretty much the same as the Solaris provider and provides statistical sampling at a user controlled rate

Even with this minimal set of providers, you can extract some pretty interesting information.  For example, we can re-write the hogs utility, that tracks the runtime of the top few processes, with a simple dtrace script:

We are missing a lot of the standard dtrace built-ins that allow you to get much deeper into the application space, such as providing the user stack information, but even by recording the instruction pointer we can get some usefull information about where an application is spending its time … expanding on the idea of an application specific statistical profiler, we can gather information about the whole system or target just one process or thread:

Since dtrace can work directly with C constructs, we can pull in code directly from the Neutrino system headers.  If you wanted to take a look at the kind of messaging operations that were going on in a process, you could write a script to catch all of the MsgSend’s via the instrumentation hook and decode the first few bytes of data based on the message header to determine what an application is doing:

Pretty cool huh! It is a nice extension to the capabilities of the System Profiler that provides the graphical analysis capabilities, but in a development environment back on the host.

There is still have a lot to do and the port is far from complete.  Before we can go much further, we are likely going to have to do the kernel and/or process manager integration to be able to efficiently provide things like the equivalent of the pid provider and the function boundary tracing.  The good news is that as a proof of concept this is already usefull … and so far architecturally independent (running on MIPS, SH, ARM, PPC and of course x86), although the hard bits are still ahead of us. 

Hats off to Bryan, Mike and Adam for a wicked awesome technology and to Sun to making it available to the software development community.  Since the port isn’t quite complete, certainly not production ready, Colin and I are hoping that we can get our prototype posted up to the Foundry27 QNX community site so that we aren’t the only ones who get a chance to play around with this. 

Stay tuned …

Thomas (& Colin who is onsite with a customer … likely using the dtrace prototype right now)

It is nice to be a hero … but sometimes things just get lost in translation.

Since it used to be that I couldn’t just share the code with the customer’s engineers, we spend a lot of time going back and forth discussing topics in the “abstract” until I can paint enough of a mental picture for them about a particular implementation of technology.  By the time that they understand what to do about the problem they are facing, I’ve pretty much given them a tour through our code.  Kind of a  waste of time considering that most of the software developers I end up working with can read through a blob of C, C++ or Java code just as easily as a paperback novel.

This is why I think that Foundry27 is an awesome idea.  We are working to get to the point where for nearly all of the Neutrino technology that has been developed, I’ll be able to talk customers through a problem and bring them right into the guts of the code during the explanation.   It is going to take some time to get all of that source out, but we have already pushed out the C library source code and the kernel/process manager code and that represents quite a bit of important glue bits that help you understand how to get the most out of your applications.

Even better than the fact that the source is being published, is the fact that the different development teams within QNX are starting to take on much more active public roles themselves and helping others to interpret and navigate through the source code.  If you missed it, you should really check out the  webinar that Colin and Sebastien presented as a bit of a guided tour through some of the source code included in the QNX Operating System project.  Also we’ve got most of the kernel developers (and a large number of other QNX developers) online now and monitoring the forums so there is lots of good information flowing around.

Next up on the source release roadmap is the networking stack and driver source and I’m also going to see if I can get a set of missing OS utilities out the door as well before Christmas!

The funny thing is that a few days later the customer was then starting to get all sort of spurious errors in their system.  Normally this isn’t a challenge to track down, but in this case the system was tightly coupled and composed of hundreds of threads with tens of processes and close to a dozen active interrupts.  There was enough other system activity going on that a small failure in one path took a long time to ripple through to a point where the error was noticeable.  The gut feeling was that some part of the client software was being unblocked as a result of the recent server unblock changes and the client wasn’t ready to deal with the consequences.

… but we needed to prove it …

The good news is that the system was running the instrumented kernel and we could take some traces and take a look at the entire system, so traces we took and then we loaded them into the System Profiler.

The question was, how can we quickly check and see if the unblock behaviour is at fault?  Well the first thing to do is to see if there is any unblock activity in the log.  This is easily accomplished with using the Trace Search capability,  Search > Search… (or CTRL+H for the keyboard addicts).  Click Add to create a new search condition.  The condition we want to look for is the Communication class events with the code of Unblock Pulse.  This will pull up all of the unblock pulses generated in the trace.  In this case we didn’t have very many to consider:

We could have gone through and taken a look at each instance, but in this case we knew that we had just put the unblock handling code into the mme server.  The unblock pulse event contains the name of the target process, so we could take a look specifically to see if we had targeted our server.  We can do this by going back into the Trace Search and either editing the existing condition or creating a new condition.  We will still specify the Communication class and the Unblock Pulse event, but this time we will also add in a specific search for instances where the processdata is equal to the mme server.

Now when we run the search BINGO!we get just one single hit.  From this point it is easy now to look at the Timeline editor pane and track the behaviour of the client process and thread that the mme unblocks and what happens after the unblock (bad things it turns out).

Also, with the system trace we can take a look at the specific cause of the signal.  It turns out in this case that there was a timer firing a signal that got dropped on the thread which caused it to unblock.  This in turn exposed a general flaw in the customer’s design since they were running a multi-threaded application where any thread could have potentially been blasted by this signal instead of the process having a dedicated signal handling thread.

Hopefully this quick example gives you some more ideas how you can effectively navigate through the instrumentation trace files, Trace Search is a powerful tool.

Thomas

The snippet of code we were looking at looked something like the following (code changes made to protect those who should know better):

/*
** Sleep for a little while.  Return the amount of time slept.
** The argument is the number of milliseconds we want to sleep.
*/
int OSSleep(int ms){
  usleep(ms*1000);
  return ms;
}

Can you see what the flaw in the logic is here? Perhaps if we put the code into a context the problem will become clearer. The OSSleep function is being called in order to walk our way through a series of progressively longer timeouts .. based on the return value of how long this function has slept.

If the application containing this source code was the only thing running on the system, then there would be no problem. However, in most modern operating systems, QNX Neutrino counted among them, there are multiple tasks that are all running concurrently and being managed by the OS.  This means that while surely the code above will delay for N milliseconds, it could in fact be much longer than that if some other thread pre-empts this thread before or after it makes it into (or out of) the system call.

So .. what could we do to make it better?  Well we could snapshot the time on input and snapshotting it on exit and returning the difference.   This makes things incrementally better by narrowing down the window of error, but the window is still there. 

The reality is that this kind of API, and this application’s use of it, is fundamentally flawed.  What the application is trying to emulate is allowing multiple threads of execution the ability to acquire a resource.  If the resource can’t be acquired, then the thread backs off or may choose to cancel the operation all together. 

This is a perfect scenario for a condition variable … something available under Neutrino, but not necessarily under the other Unix flavours that this source was ported to.  By switching to a condition variable, we can be much more precise with respect to when we signal client applications to “try again” … which can result in a surprisingly large reduction in overall CPU consumption and system churn (due to all the extra context switches and cycles through the scheduling READY queue).

So in these days of thinking about multiple threads of execution running on separate CPU cores, we should also remind ourselves that using “time” as a basis for control needs to be carefully planned on systems where there is more than a single thread of execution.

Thomas

Well, today I had a few minutes, and so I went of in search of a more complete solution.

Switching Modes

As you may remember, one of the issues was that the beep control sequence, was only understood by devc-con when it was running QNX4 Terminal emulation mode, which is not the default.

I was telling this to my colleague, Peterv, and he said – well just emit the code to switch to QNX emulation and back.

Well, it was one of those ‘Duh’ moments. :-)

Of course, the fun part was figuring out the codes to emit. I couldn’t find a decent doc of the QNX4 Terminal Emulator codes, and the ANSI code to switch to QNX4 was a private vendor extension.

After a bit of mucking around though, I figured it out…

The Dark Majik

And yeah shall chant…

#!/bin/sh

# switch to QNX4 emulation
printf "%c[?+q" 0x1b

# set beep tick count = 1, frequency divider to 1

printf "%cs%c%c%c" 0x1b 33 33 32

# switch back to ANSI emulation
printf "%c?1+q" 0x1b

Now, make sure you put this output to /dev/con1, because that is the device that actually talks to the PC beep port. Photon pterm terminals do understand these codes, but they just send a bell character to /dev/con1!

Enjoy a beepless world, my friends.

BTW – you can muck around with the parameters. You might find you like a beep but just a bit shorter – each tick is 50ms.

One day I realized all “ditto” doing (or need to do), is to ”sniff” characters between applications and the character device. Just like a network packet sniffer. Applications are going to write() into, for example, /dev/con1; All Ineed to do, is to find out what has benn written in, buffer them, for someone else to read out.

So how do I do this? QNX6 resource manager allow you register already exist path name, to, un, hijack it. So all I need to do, is write my own resrouce manager (called devc-ditto :D), it also attach to /dev/con1; The default behavior of resource manager pathname attach, is the ”latest attched got message first”, which works very well. So once devc-ditto attached to /dev/con1, any one open(“/dev/con1″, ..) it will end up with a connection from application to devc-ditto. All the applications write() would comes to me, I buffer those data it is written, then relay it to the resource manager who is really managing /dev/con1 (devc-con); Done!

Well almost done, this only allow me to sniff out the datas, how do I pass it to another application? This is easy, if I take over /dev/con1 for example, I would then create a /dev/con1.ditto, read from this device, you got all it buffered.

So here is how you use it:

On a console 2, try 

  # devc-ditto /dev/con1
  # cat /dev/con1.ditto

Then go back to console 1, do “ls >/dev/con1“, and come back, you should see all the “ls” output (from cat).

Wait, this is not what ditto doing right? Why is the 2 character “ls” not showing up in “cat” ? Think what happened, you hit “l” key, devc-con got it, give it to your shell, youe shell will echo it, by sending it to /dev/con1… but hey devc-ditto doesn’t see this “l”, why?

Because, the shell made a connection directly to devc-con, not to me. OK, so if I exit the shell, and re-login, that would do it, right? Sorry, no, because “login” spawned “sh”, the shell inherite stdin, stdout, stderr from “login”, in other words, the shell never open() the /dev/con1 device. Hm, so I have to run devc-ditto before “login” running? Yes, you can do that, by put devc-ditto in /etc/rc.d/rc.local, it actually works….

But, but I don’t like that. I want to be able to start/stop/restart devc-ditto freely. After talk to some expert, I have put this in my .kshrc.

  alias reopen ‘exec 0<$TTY 1>$TTY 2>$TTY’

Now, exit and login again on console 1, and type

   # reopen

Wohoo, after that, anything you type in concole 1, you should be able to see it in “cat”.

“But that’s now all ‘ditto’ doing, I also want to type in from ‘the other window’”, yes, I hear you. To allow inject key, I made it so any character write() into /dev/con1.ditto device, would be read() out from the /dev/con1. So what do you need to inject key? Yes, use “qtalk”.  Now there are a few gocha of “qtalk”, first of all, you need a qtalk support “-O” option, which ignores file open count; Also qtalk tend to open a character device in “edit” mode, we want “raw” mode.  So here is what you do, (assume you are on /dev/con2)

    # qtalk -O -m /dev/con1.ditto
    <in qtalk>
    # stty raw </dev/con2

After this, whatever you type in qtalk, would show up in /dev/con1, and processed by the shell running there. This ultimatlly gives you the same “ditto” experence. And of cause, all these are standard utilities (cat, qtalk), so they work over QNET just fine.

But this is not all of it. Since the principle is just hijack a device name, devc-ditto not limited to only console, you can use it against any character device, /dev/ser1, or maybe /dev/ttypX. Wait, “/dev/ttypX”, isn’t that means I can ”ditto” into a telnet session? Exactly. I will leave the practice for you, to figure out how to ”ditto” into a telnet session.

That’s all I have to say.  come to ostech forum if you want to discuss more, enhance it if you have more idea. Oh, one more thing, I made the devc-ditto also handle mount/umount, so you don’t need to start a lot of them if you want to monitor multiple interfaces.

   # mount -Tditto /dev/ttyp5

Now you can tap into /dev/ttyp5.ditto, after you done:

   # umount /dev/ttyp5.ditto

xtang

The good news is that a QNX developer, Xiaodan Tang, took it upon himself to help out and solve this ditto dilemma.  Xiaodan has written a new Neutrino service, aptly called devc-ditto, that will allow character buffer replication and general terminal emulation.

While solving a lingering problem is always good news, the next question people will inevitably ask is “When can I get devc-ditto?“. 

The new devc-ditto is a good thing.  As a general rule, we like to include good things in our product distribution.  Here is where the bad news used to have to be delivered.  Our software releases are “steady and stable” which at times can be an euphamism for “glacially paced“.  This means that to get devc-ditto you would either have to wait for the commercial release (potentially a long time) or give your QNX sales rep a call and figure out a way to get this excellent piece of accellerated technology into your hands.

… but the times they are a changin’

QNX is shifting its entire R&D team to a new Transparent Development model.  In this model the source code for our products is being published live and the work on those products is taking place publicly.    Still commercial software development but with a flavour that should make open source developers, who like probing the product source and interacting with the authors/maintainers the source, much more at home.

I can hear you thinking “Yeah yeah … neat.  Now how does this help me get devc-ditto?!“

Well QNX has a lot of source to release, and it will take some time to clean it up into publically presentable form, but working with Xiaodan and the Core OS team we have been able to get devc-ditto included in the first OS source release, which means you can get it, build it, use it, change it all to your hearts content!

1. Get a community username (aka your MyQNX username)
2. Get the latest development tools and distribution (if you just want devc-ditto, and you have a dev seat, you can skip this)
3. Get the source: svn co –username=<community_username> http://community.qnx.com/svn/repos/coreos_pub/trunk/services/devc-ditto
4. Build it: make

I’ll leave the use-it up to you to figure out now … if in doubt, post to the OS forums with your questions!  If you want to get more and build more then you should really check out the whole Core OS project as well as the other community projects.

Thomas 

More often that not bug reports would be greeted with a friendly post saying – “Thanks! Try this and let me know if it fixes your problem!”, and you’d be downloading a binary straight from the desk of Real Live Kernel Engineer.

Ah, the thrill of finding hidden paths and filenames embedded in that debug binary.

What it must be like to be a PART of that select club, to be involved directly in the creation of the most successful micro-kernel RTOS out there…

Wouldn’t you want to be part of that?

Well, I’ve been there over 10 years now, and all I can say is… I can’t wait until Wednesday!!!!!

:-D

What I found in my mailbox I found rather amusing.  I had four separate requests for help in getting the QNX debugger up and running using the IDE. 

I’m not doing (much) IDE development these days but these types of requests still manage to find their way to me because of my dogged insistance that I use our Momentics tools for all of my development and also because I think that printf  debugging is for dinosaurs.  If the IDE can’t get the job done for you, then we need to fix it so that it can … feel free to tell us so!

Anyway … back those e-mails.  They all were stating that they just couldn’t get the IDE debugger to work.  The platforms were all different (arm, sh, x86) and the environments were all different (Windows, Linux and Neutrino) but the problem was the same.  The application would launch, but the IDE debug session would fail with a cryptic connection error.

Why is this happening?  For most IDE services, the qconn target agent is all that needs to be up and running.  However, the debugger also makes use of the pdebug debugger target agent.  This is launched automatically for you when you start debugging. 

… unless it isn’t. 

The problem is that since the IDE is telling qconn to launch the pdebug program, pdebug needs to be in one of two places:

• Accessible via qconn’s PATH environment variable
• Located in /usr/bin/pdebug

If the pdebug debugger target agent can’t be launched, then the debug session will fail … and the resulting error message provides absolutely no indication of what is going on and why the debug session failed.

While this bit of information is recorded in the documentation for the IDE, it isn’t the first thing you would find when things go wrong.

So … if debugging wasn’t working for you and you never understood why and flipped back to printf’s take a close look at your configuration (and/or make sure that pdebug is in /usr/bin) and try it again … I think you’ll like it!

Thomas

The unfortunate side is that it only works if you are running devc-con in QNX mode, so you will either have to do that (by passing the -Q option) or suffer with the loud beeps (for now)

Ok, so go change your build file, and make sure your TERM env var is qnx

So we need to emit an escape sequence, which can be done thusly…

# printf “%cs%c%c%c” 0x1b, <c1>, <c2>, <c3>

Where c1 c2 and c3 are the parameters…

Here’s a snippet of the state machine…

case STATE_QNX4_11:     /* esc_beep_set */
s->beep_ticks = (c <= ' ') ? 6 : c - ' ';
s->esc_flag = STATE_QNX4_12;
break;
case STATE_QNX4_12:
s->xpos = c - ' ';
s->esc_flag = STATE_QNX4_13;
break;
case STATE_QNX4_13:
n = (s->xpos % 192) + ((c - ' ') % 192) * 192;
if(!n)
s->beep_counter = 0x0606;
else
s->beep_counter = (n &lt; 19) ? 0xffff :
(unsigned)(1193180L / (long)n);
break;

The default values for beep_ticks and beep_counter are 6 and 0×0606, respectively.

I haven’t quite worked out the optimimum, but basically if you screw it up, the beep is gone anyways, so I’m alright with that. You guys can have fun playing.

I’ll try and work a nobeep facility into the next version of devc-con.

Sorry this is brief, I’m running out of time today. I’ll try and elaborate more on exactly WHAT these things mean later (or maybe Thomas will…)

Enjoy!

Colin

What is up with those send receive reply guys? They haven’t posted anything in weeks!

True … it has been a few weeks, but we haven’t been idle.  There are exciting things a brew at QNX during the summer heat … air conditioning being one of them that tends to only make sporadic appearances unfortunately.

One of the more interesting features that is has worked its way into the Neutrino kernel for the next release (slated for the not too distant future) is going to be the ability to report out the mapping of physical memory of a system to virtual addresses used by individual processes. 

“But we live in a virtual address world“,  you say “why would we want a physical map of a systems memory?”

That was certainly my first impression.  Particularly with the fact that you can get a complete virtual address mapping for a process using the /proc filesystem and the appropriate devctl() and the entries for <sys/procfs.h> and <sys/debug.h>, you should be able to paint a fairly complete memory story all using virtual address information.

Turns out that that isn’t quite the case.  You can get close, but there are still some missing details when it comes to how particular shared libraries and executables are mapped within other mapped areas (for example using execute in place within secondary image file systems).  Getting access to the physical address allows you to resolve that ambiguity quite nicely.

Also, while I mentioned in the article on fragmentation (yes … we will get back to that issue =;-) that the OS nominally hands out memory uniformly in ~4K chunks, it does have the ability to fragment over time.  In general this isn’t a problem since the MMU for the processor provides the mapping giving you virtual continuity over fragmented physical memory.

Unless you need (or want) physically contiguous memory.  This can lead to improved performance in some device drivers that can leverage DMA and direct IO data transfers.  It is also a requirement by some tooling like the instrumented kernel, that arranges its data buffers as one contiguous ring.

The interface for extracting this additional memory information will be through the /proc interface and is a multi-stage procedure, but the fact that the information is now available is great news.  

Now all we need is a tool to showcase this new information … nothing like writing a handy utility to help beat the heat!

 Thomas

When I started MIPS, PPC ports were well underway with ARM and SH support starting up.   I’m not an expert in any particular processor architecture, so I ended up working through a lot of the general system bugs and polishing some of the rough edges of this new OS.  By necessity this meant spending a lot of time interacting with the very active set of QNX developers, who at that time were steeped in QNX 4 knowledge and trying to migrate their systems and software to this new OS.   There are a lot of basic similarities between QNX 4 and QNX 6 but there are enough differences that migration wasn’t a zero effort affair, despite a fairly comprehensive software migration kit.

About once a month I’d receive a request from QNX4 developers:

“When will ditto be ported to Neutrino?”

Of course, the first time I heard this comment I was intrigued since ditto is not a standard *nix utility, so I thought I’d look into it.  It turned out that porting this particular utility (which provides terminal replication and remote terminal access) to Neutrino was going to take some signficiant work and more or less be an entire re-write since the utility took specific advantage of how QNX 4 character device drivers were written.  Under Neutrino the serial driver code was completely different,  so at the time I turned my focus to other areas where fixes could be made more short term with a plan to come back to do the ditto re-write.

Here we are, nine years later and there are still people asking for ditto for Neutrino. 

Boy do I feel bad about never getting back to that work =;-(

Well since I still haven’t ported ditto yet, the first thing I’m going to do is suggest an alternative that may be a more mainstream solution in these days of Linux fervor.  The GNU Screen utility provides much of the same terminal multiplexing and remote or shared control that ditto did.  The source compiles, with a few minor changes, out of the box for Neutrino. While I haven’t taken it for a full test drive yet, this looks like it might be the best immediate solution and maybe even long term since there are several screen resources available in the community.

However, ditto ain’t totally dead yet.  The other day walking through our source tree on a related project I noticed that one of our aspiring developers has done some work to make some of the ditto remote monitoring (no interaction) capabilities available for Neutrino.  This is still internal goods, and not in the next OS release, but perhaps this article might nudge it along if there is some competition.

Thomas

Mysterious … and incredibly annoying!

I’ve come to the conclusion that what is happening is that the Exchange meeting rooms are suffering from fragmentation. When the system is cleared, then large groups book re-occuring meetings. Time goes on, and the re-occuring meeting needs to be shifted and bumped, rooms get borrowed (or stolen) and when the meeting organizer changes the room, the room doesn’t officially get free’ed up until all participants clear their bookings of the rooms. Over time things get so fragmented that until everyone clears their calendars you can’t book anything.

Fragmentation is a pain. There are lots of schemes to avoid it in filesystems and in memory allocators, but eventually it comes down to the fact that you need to make arbitrary decisions about how to carve up a resource into pieces to satisfy a particular usage pattern. In the case of operating system memory, this granularity is of page sizes which are generally* 4K chunks.

For most embedded applications 4K of memory is an awfull lot so there is a secondary, application level, allocator that sits behind the malloc/free/realloc calls that applications use to get at the memory. This memory allocator takes the pages provided by the operating system and then further carves them into bands of memory of smaller fixed block sizes as well as a list of “bigger blocks” that are non-uniform in size to satisfy the abnormal requests. A great, more detailed, description of what the QNX memory allocator is doing is in the QNX IDE User’s Guide in the Memory Analysis section.

If you snoop the header file then you can see the struct malloc_stats structure that can provide you with information about how your system is using application allocated memory. You will need to include the _malloc_stats external global variable to get at the numbers.

While not a direct measure of fragmentation, these statistics give you some insight into the overhead that the memory allocator is taking and how much memory is being handed out to your application (and thus shows up in a pidin mem listing as application heap) but may not be used directly. Note however that these statistics don’t tell the whole story and in the allocator it is possible for “big blocks” to be borrowed for “small block” use but for now we’ll pretend that those values tell you most of what you need to know to get started.

So now that you can see a bit of detail of how the application allocator is carving up and managing memory … what can you do when you find out that the allocator’s decision about small blocks or big blocks isn’t the right one for you but you don’t want to write your own memory allocator? We’ll talk about that kind of customization (and how you know you need to do it) in my next post!

In the meantime, I need to go and find out which meeting room is suffering from fragmentation and grab it for a meeting!

Thomas

If the path we were looking for was attached in between the stat() and the procmgr_event_notify() call (admittedly a small window) then we would end up waiting the entire timeout duration before noticing that the path had appeared.

Here’s a new version that uses a pulse that closes that hole, by using a pulse as the notification event.

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <time.h>
#include <sys/netmgr.h>
#include <sys/neutrino.h>

#include <sys/procmgr.h>
#include <sys/siginfo.h>

#define PULSE_CODE_TIMEOUT      _PULSE_CODE_MINAVAIL
#define PULSE_CODE_PATHSPACE    PULSE_CODE_TIMEOUT+1

int main( int argc, char *argv[] )
{
    struct itimerspec   timeout;
    int                 timer_id;
    struct sigevent     event;
    struct stat         sbuf;
    int                 chid, coid;
    char                *path;
    struct _pulse       pulse;

    path = argv[1];
    memset( &timeout, 0, sizeof(timeout) );
    timeout.it_value.tv_sec = atoi(argv[2]);

    chid = ChannelCreate(0);
    coid = ConnectAttach( ND_LOCAL_NODE, 0, chid, _NTO_SIDE_CHANNEL, 0 );

    SIGEV_PULSE_INIT( &event, coid, getprio(0), PULSE_CODE_TIMEOUT, 0 );
    timer_create( CLOCK_REALTIME, &event, &timer_id );

    /* make sure PATHSPACE event has higher priority */
    SIGEV_PULSE_INIT( &event, coid, getprio(0)+1, PULSE_CODE_PATHSPACE, 0 );
    procmgr_event_notify( PROCMGR_EVENT_PATHSPACE, &event );

    /* Check to make sure we don't wait for something already there... */
    if ( stat( argv[1], &sbuf ) == 0 ) {
        printf("Found %s\\n", path );
        return EXIT_SUCCESS;
    }

    timer_settime( timer_id, 0, &timeout, NULL );

    while( MsgReceivePulse( chid, &pulse, sizeof(pulse), NULL ) == 0 ) {
        switch(pulse.code) {
        case PULSE_CODE_PATHSPACE:
            if ( stat( argv[1], &sbuf ) == 0 ) {
                printf("Found %s\\n", path );
                return EXIT_SUCCESS;
            }
            break;
        case PULSE_CODE_TIMEOUT:
            printf("Timed out waiting for %s\\n", path);
            return EXIT_FAILURE;
        default:
            printf("Unknown pulse code %d waiting for %s\\n", pulse.code, path);
            return EXIT_FAILURE;
        }
    }
    return EXIT_FAILURE;
}

Note that I use a timeout pulse too, and make the timeout pulse have a lower priority than the notification pulse. This is because pulses queue in priority order, and I want to make sure that I receive the path event pulse first.

Cheers,

Colin

If you have used Neutrino for any length of time, then you have probably done the former and if it was a trivial system, then the latter thought has probably crossed your mind.

QNX’s message passing strategy is all about following the Send/Receive/Reply mantra where clients talk to servers and block until they are replied to with a message.  Under QNX4 there were officially no threads so when you were sending a message, you only needed to supply the node (transparent network message passing!) and the process identifier as routing information.  When Neutrino (QNX6) was introduced, threads were made first class citizensand to make the targeting of message operations completely unambiguous, message channels were added.  This meant that instead of just a node and process id, you also had to specify the channel identifier.

 How does a client get these three pieces of information about the server it wants to talk to?  There are lots of different schemes to accomplish this depending on the structure of the system you are building; ranging from sticking the information into a piece of shared memory to passing it from parent to child processes via a fork/exec/spawn type of operation.   In a simple system, you can have simple sharing policies to get at this data, however as a system gets more complicated with more clients and servers, then things get out of hand quickly.

The preferred mechanism however, would be to let the OS do the heavy lifting and to replace the node, process id, channel id with a symbolic entry that is easy for clients to use and discover and let the resolution of that be handled by the operating system.

Enter the raison d’etre for the resource manager.  It was introduced to help facilitate that binding on both the client (name resolution) and on the server (node, process id, channel id binding) as well as providing dispatching capabilities to help with the demultiplexing of common messages to specific handlers.  The Neutrino Programmer’s Guide tells you all about this, and Rob Krten’s Neutrino books do a great job of providing further enlightenment.

About every six months or so I run into someone who is stuck in a very specific situation.  They have created their own channel via ChannelCreate(), perhaps to turn off priority inheritance or because they need to avoid using certain flags, and would like to use it with the resource manager framework.   The resource manager framework does such a  good job of abstracting and hiding the channel that it seems that it isn’t possible.

 Alas … do not lose hope!  There is an undocumented function that allows this to occur:

dispatch_t *_dispatch_create(int chid, unsigned flags)

The function above allows you to pass in a channel identifier that you have created yourself and serves as a replacement for the general dispatch initialization function, dispatch_create(), that is the preferred initialization.  As always, when you are using something that is undocumented (but publicly exposed) there are some caveats to consider:

• The flags field should be passed as 0 at this time
• In order for proper operation of the resource manager, you must specify _NTO_CHF_UNBLOCK | _NTO_CHF_DISCONNECT as flags for the channel.  Without these flags, your resource manager may fail to properly close off connections from clients that abnormally disconnect.

One of the situations where this function is required, is if you are trying to have a resource manager as part of an application that also uses Photon.  While this joining of server and interface functionality is generally frowned upon (a clean separation is always preferred) there are some situations where it simply isn’t practical.  In this case, Photon generally grabs the _NTO_CHF_COID_DISCONNECT flag for itself, which the current resource manager framework assumes it will be able to use.  Since this flag can be set on only one channel per process, your resource manager will end up failing during initialization with a mysterious error … a future release will clean this up and allow everyone to play nice, but for the current 6.3.x releases of Neutrino, you will need to use this function to work around that situation.

 Thomas

# pidin
     pid tid name               prio STATE       Blocked
...
       20   1 server_app             255o RECEIVE    1
       20   1 server_app             10o  RUNNING
       20   1 server_app             255o RECEIVE    1
...
       32   1 client_app              10o REPLY         20
       32   2 client_app              10f NANOSLEEP
...

He was concerned because most of his system generally operated at priority 10 and if there were tasks that were at priority 255, then wouldn’t those other tasks be starved out and never get a chance to run!?

Of course if you take a few minutes to reflect on what is really being displayed then peace and calm return quite quickly to the world. 

The threads that are all marked as being high priority are all in blocking states.  That is to say that they are not RUNNING, nor are they wanting to run and in the READY state.  In fact they are all blocked waiting for some other sort of event to occur or non-CPU resource to become available.  While their reported priorities are high they are not going to interfere with the operation of the system … at least not until they transition to a RUNNING or READY state.   In the case of threads which are RECEIVE blocked such as is the case here, unless the communication channels were created specifically to avoid priority inheritance, the receive thread will end up READY/RUNNING at the priority of the sending client.

If that is the case you may wonder, why have pidin report the priority information on blocking states at all if it is just going to cause people this sense of panic and trigger these types of false alarms?

Well the information can be insightfull since in most blocking scenarios, the thread was READY or RUNNING prior to being blocked and as such the priority gives a hint to what they might have been doing.  Also in the case of priority inheritance states (MUTEX, SEND) then you can better see who is causing priorities to be automatically shifted in the system.

In this case, 255 was a curious number to be seeing and we eventually traced it back to their code where they were generating asynchronous pulse events to a server, but the pulses were not being properly initialized and were picking up junk off the stack and setting bad priorities … hence the 255 value. 

QNX Coding Tip of the Day:  Don’t initialize sigevent‘s using the member values, use the SIGEV_* macros!

 Thomas

There are many things you can do to squeeze a few precious seconds out of your startup sequence, but one of the simplest may not be immediately obvious.

The sequence

devb-eide
waitfor /dev/hd0 30
…

is an all too common one, but it’s surprisingly damaging if you are trying to not waste cycles.

For one thing, it means that you can’t really start anything after the waitfor until devb-eide has gone through its sometimes lengthy initialization sequence (sorry devb, don’t mean to pick on you…).

Really all the system can do is to finish off the initialization of any servers you already started and poll for the device to appear.

Wait a second – POLL??!!!

Hmmm, in RTOS speak, Poll=BAD, WaitForAnEvent=GOOD!

Yes, my friends, it’s sad to say that /bin/waitfor (actually the amazingly versatile /bin/on) does actually poll, with a period of 100ms to boot, as does the builtin version that procnto provides when running your image’s startup script.

There are a number of nasties associated with this, one of the most annoying being that if your device appears 1ms after that 100ms poll (which is just a stat()) then you’re going to wait around for ANOTHER 99ms before waitfor notices!

And unless you have plenty else going on in the meantime then that means the dreaded idle thread will happily be using up your precious cpu. Another good reason to have plenty of servers started BEFORE you do the waitfor.

So how about a remedy?

Well, the basic version of waitfor looks something like this…

while we haven’t reached the max limit
do
stat the device
if it’s there, break out
delay 100ms
done

Wouldn’t it be nice if we could do something more like

while we haven’t reached the max limit
do
sleep for the remaining timeout, but wakeup if the device appears?
done

Well, as it happens, if you are running QNX6.3.0 SP2 or later, then there is actually a facility you can use. It’s a new flag to procmgr_event_notify, PROCMGR_EVENT_NAMESPACE,which will see a sigevent winging it’s way to you whenever the pathspace changes, which is basically whenever someone attaches or detaches a pathname.

So here’s a revised version of waitfor that waits until something changes, then hopefully stats the device.

#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/procmgr.h>
#include <sys/siginfo.h>

int main( int argc, char *argv[] )
{
struct sigevent event;
struct stat sbuf;
int timeout;
int total;
char *path;
time_t start, now;

path = argv[1];
timeout = atoi(argv[2]);

if ( stat( argv[1], &sbuf ) == 0 ) {
printf(“Found %s!\n”, path );
return EXIT_SUCCESS;
}

SIGEV_UNBLOCK_INIT( &event );
procmgr_event_notify( PROCMGR_EVENT_PATHSPACE, &event );

start = time(NULL);
do {
sleep( timeout );
if ( stat( argv[1], &sbuf ) == 0 ) {
printf(“Found %s!\n”, path );
return EXIT_SUCCESS;
}
now = time(NULL);
timeout -= (now – start);
start = now;
} while( timeout > 0 );
return EXIT_FAILURE;
}

If 99ms of idle time upsets you, this might help you sleep better.

Colin

PS – yes, this will be fixed in the next release of the kernel/utils, if I have anything to do with it.

PPS – I should mention that there is yet another location that waitfor is defined, that is the ewaitfor builtin in /bin/fesh. This version at least lets you set the poll period, as a third argument to ewaitfor. Again, that should be in 6.3.0 SP2 or later.

The problem, other than the not so elegant API required by this third party library, was that while the library needed a filename to work with, his plugin needed to send drive level control commands in the form of custom devctl()‘s.

 This reminded me of some control commands that had been put in to the block/filesystem level in the early days of Neutrino.  Hunting around I found the devctl() commands I was looking for in <sys/dcmd_blk.h>:

#define DCMD_FSYS_MOUNTED_ON  __DIOF(_DCMD_FSYS,  16, char[256])
#define DCMD_FSYS_MOUNTED_AT  __DIOF(_DCMD_FSYS,  17, char[256])
#define DCMD_FSYS_MOUNTED_BY  __DIOF(_DCMD_FSYS,  18, char[256])

When you have a file descriptor that points to something handled by of of the block oriented filesystems or block driver (ie devb-*), then you can use these handy commands to dig down or up or to determine where an entry is mounted in the filesystem.

DCMD_FSYS_MOUNTED_ON
This command will return the pathname of the underlying device, partition or file the current entry was mounted on. For example a file descriptor to a filesystem file will generally point back to a partition (or the raw block device), a partition file descriptor will point back to the raw device. In this way you can burrow your way down to the controlling device.

DCMD_FSYS_MOUNTED_BY
This command tries to root out and determine where the file descriptor you are referencing has been mounted if it is a singular mount point such as a partition. This is an upward version of MOUNTED_ON, but since multiple users can exist for a single device, it isn’t as generally useful as the MOUNTED_ON command.

DCMD_FSYS_MOUNTED_AT
This command tells you where the object that you are referencing was mounted at. This is similar to the type of information that you get back from the mount utility, but in this case the MOUNTED_AT command is a specific command for the block filesystem.

Due to the overlay nature of the Neutrino pathname space, these commands are really only applicable for file entries since directory entry resolution isn’t as explicit as for files*, but can come in handy if you are in a pinch like we were with this silly third party library API.

Thomas

#./hello
Process 5861407 (hello) terminated SIGSEGV code=1 fltno=11 ip=b0348cec mapaddr=00048cec. ref=00000000

Oops, that’s not the start to this blog that I was hoping to make. Two lines in and already my efforts are riddled with bugs. Oh my.

In actual fact though, I wanted to start of by sharing some a tidbit I found the other day, while staring at the termination code in procnto.

The message you see above is generated by the termer thread, which runs some procnto code on behalf of your dearly departed process to cleanup leftover resources, such as file descriptors, memory mappings etc. And one of the things it does, if you have added a -v to procnto in your build file, is to print out information about the exit status of processes.

What it tells us is that your process died of a SIGSEGV, and the code=1 (SEGV_MAPERR from sys/siginfo.h) and fault=11 (FLTPAGE from sys/fault.h) tell me that it was trying to access a page that wasn’t mapped.

It died at the address 0xb0348cec, which is all very well, but if I load my hello binary into gdb then this address is clearly not in my application.

The clue is mapaddr=00048cec. This tells me that the object that contains the offending code was relocated when it was mapped into memory, and thus we are crashing in a shared object. But where/what is 0x48cec? The long winded approach is to figure out which shared object contains 0xb0348cec (using pidin mem) and then load that shared object into either gdb or objdump. But that’s all rather grungy.

Which brings me to the paydirt – there is code in the termer thread to actually fish out the name of the shared library and the function name in which you crashed!

So why weren’t we seeing this? Well a little investigation revealed that a linked list that was supposed to point to the process’ shared objects was not being initialised (Note this bug has been fixed now and will be available in a newer version of libc.so).

The initialisation function is an internal function, but you can force it to be re-called by trying to figure the address of a symbol with dladdr().

So now, here’s my hello world with dladdr invocation added. You might already note my error, but let’s run it just in case…

#include <stdio.h>
#include <dlfcn.h>

int main(int argc, char **argv)
{
char *world = NULL;
Dl_info info;

dladdr( main, &info );

printf( “Hello %s!\n”, strdup(argv[1]) );

world = “World”;

return 0;
}

#./hello
Process 5951519 (hello) terminated SIGSEGV code=1 fltno=11 ip=b0348cec(libc.so.2@strdup+0×20) mapaddr=00048cec. ref=00000000

Well silly me – I simply forgot to pass a command line (not to mention some nice error checking? Serves me right for writing such a contrived example… )

#./hello World
Hello World!

Process 6004767 (hello) exited status=0.

Ah, that’s better…

The developer’s initial thought was to use a semaphore since that is what the name of the callout implied, and other ports had used named semaphores … and QNX/POSIX has semaphores, both named and un-named, but the comment about the use of the callout to provide mutual exclusion seemed to imply that despite the naming, a mutex would be a better choice for performance which is when he asked me about it.

Assuming that you are using a binary semaphore and not a counting semaphore then purely from a performance point of view here is your ranking from worst to best choices:

Named Semaphore

This is a semaphore that trades off API convenience* (a pathname style location) for performance since you are going to be going through a resource manager (procnto in later versions of Neutrino or mqueue in earlier versions of Neutrino) who will be serializing the access to the semaphore count. While operationally it is the same, this extra messaging is going to impose an extra bit of overhead.

Normal Semaphore
This is a semaphore that is not identified by name, and whose operation is managed by the kernel. For each semaphore operation (sem_wait/sem_post) a kernel call is made to handle the management of the semaphore data. This is a smaller overhead than then message passing and server operations required for a named semaphore, but is still a kernel call for each semaphore call.

Mutex
While a mutex doesn’t provide the same semantics as a counting semaphore, is can be a great high performance alternative to a binary semaphore. Under Neutrino, mutexes are highly optimized such that they use the processor’s atomic operations to do an in place compare and exchange. Only if the mutex is contested is there any requirement to enter into the kernel. This means that in most cases where there is only minimal contention for the synchronization primitive there is no additional kernel call overhead

Inline Mutex
For most operations, the standard mutex is going to provide much better performance than a binary named semaphore and also better performance than the normal semaphore. However, if you really want to crank performance (and still use standard primitives) then you might have noticed that there are also inline mutex operations defined in pthread.h.

So in general, I tend to favour mutex operations over semaphores just because they are so nicely optimized. If you need to have a counting semaphore, then you might want to consider using a condition variable instead (which relies on a mutex base) for greater flexibility and potentially higher throughput. If that is the case, reading through the post a condvar is not a semaphore might be usefull.

However, since that documentation is not always at hand when you want to run a quick command* each target binary (ls, find, pidin etc) also contains an embedded text segment that holds a description on how to use the utility. In order to get at this information, you use the imaginatively named use utility.

For example, I never remember all the options to the find utility. If I were on a target then I could type in:

 % use find
find  - find files (POSIX)
find  ... [operand_expression]
Operand expressions:
...

In some ways this information is equivalent to unix style man pages, but since it is embedded directly into the binaries you never have to do any fancy configuration to prune information in (or out). Compared to rooting through and finding the utilities you use and then pruning the man pages appropriately, this is wicked easy.

In addition to the usage type of information, you can also print out build version information that is included with QNX binaries:

 % use -i find
NAME=find
DESCRIPTION=find files
DATE=2004/12/16-00:01:41-EST
STATE=Stable
HOST=uberbuild
USER=toolsbuild
VERSION=6.3.0SP1
TAGID=329

If you are ever contacting QNX support, this is great stuff to send them so that they can match up your binaries directly with the source for a given release if need be.

Under Neutrino the usage information is stuffed into two different ELF sections of the binary.

 % ntox86-objdump -h find
/usr/bin/find:     file format elf32-i386
Sections:
Idx Name          Size      VMA       LMA       File off  Algn
...
 19 QNX_usage     00004fb7  00000000  00000000  0000d304  2**0
                  CONTENTS, READONLY
 20 QNX_info      00000085  00000000  00000000  000122bb  2**0
                  CONTENTS, READONLY

This approach is a double edged sword. On the one hand it means you don’t pay a memory penalty when loading the process since the QNX loader knows that these sections aren’t executable and basically ignores them. Since they are standard ELF sections, you can use all of the standard binary manipulation tools to work with them. The downside of this is that these sections can be removed by utilities such as strip, although the Neutrino image build process, using mkifs will preserve these sections in the OS filesystem image.

The cool thing is that this information stuffing isn’t something just reserved for QNX utilities and shared objects! It can be added to any Neutrino binary via the usemsg utility. There is no set format for the general usage section (QNX_usage), so you can put whatever you like there. The convention is to put command line parameter explanations and example invocations in this free form text section. The information section (QNX_info) is a structured key=value area and has a date and time field that is automatically updated when information is added. It is easy to use this technique to integrate notes that can be easily retrieved on both host and target systems.

% use -i ./a.out
No info available in ./a.out.
% usemsg -i qnx=cool ./a.out
% use -i ./a.out
NAME=a.out
DATE=2007-03-25EDT-20:06:24
QNX=cool

Use messages are definitely a handy bit of infrastructure that make things just a bit easier for you to work with and manage your system software.

One of the particularly religious debates that we occasionally get into revolves around the use of the goto statement. There are many people who use the goto statement considered harmful to argue that there is no rightfull place for a goto in our code.

This attachment to a particular dogma means that for complex structure initialization code developers often end up running a series of if clauses initializing each member of the structure in turn. If the code properly deals with errors and rollsback to undo the initialization, you end up with one of two scenarios:

For this reason, I’ve always appreciated the goto statement as acceptable for these types of “go to a cleanup” scenarios where you have a large initialization dependancy (as might happen in a QNX resource manager) and want to group the cleanup so as to improve the readability and reduce the chance of an accidental resource leak. Unlike the use of goto resulting in spaghetti code flow control, this is a much more tempered use of goto statements in C code.

However, the other day I was participating in a code review and came across an interesting use of the do {} while
loop to achieve a similar early exit, but without all of the contempt of those who think that goto should be removed from the language all together:

  //Structure declaration and initial allocation here
  do {
    // ... Member initialization here
    if(failure) {
      break;
    }
    ...
   //Return successfully allocated and initialized member here
   } while(0);

   //Cleanup and failure reporting all goes here

Here you get a nice linear, easy to read and maintain flow of code initialization without being burdened by the error condition flow control that a nested if requires or the deep unrolling of initialization that a chained if requires.

In any case a nice escape away from goto’s; even though Dijkstra never said goto was considered harmfull =;-).

Oddly enough, my first memory of using a debugger had nothing to do with traditional debugging. I was a co-op student working on some speech processing software using the now defunct Mac CodeWarrior development tools.

I was trying to figure out how to insert timing measurements into a large piece of algorithmic software to determine how to best optimize it.

“Simple” said one of my colleagues at the time. He fired up the debugger and started stepping through the project:

step step, step ………… step

“That’s the function you should look at” indicating the one with the long step time delay.

A true KISS principle that I’ve used over and over to do basic analysis of where a program is spending its time. Simple and effective when you are considering software where delays and latency is measured in seconds.

The cool thing about working with QNX systems is that you can use the same debugger and techniques on just about any piece of system software. Services, drivers and applications are all outside the kernel so there are very few occasions where you can’t attack a problem using a process level debugger.

Neutrino uses gdb as its main debugger (the IDE drives gdb under the covers) and one of the great facilities that gdb enables is the ability to do post-mortem analysis on crashed binaries using core files.

 % ntox86-gdb my-x86-binary my-x86-binary-corefile

or if you are in the IDE then create a post-mortem debug session and select the binary and the core file as part of the launch configuration.

Capturing core files on a target Neutrino system requires that you run the dumper program. If (when?) your program abnormally terminates, assuming dumper is running, a core file will be generated and a diagnostic message logged. You can then load up the debugger with the core file and get right to the root of the problem.

The dumper program has a bunch of usefull options, but one of the most usefull is the -p option that allows you to generate a core file for any running process without terminating that process.

 % dumper -p process_id -d dump_directory

This is a great way to capture program snapshots for those situations where you can’t attach to the running program due to potential latency
or you just want to get an idea of the program state (ie the program seems to be hung up or burning CPU) and don’t have a local debugger connection.

Once you have the core file, you load up into the debugger the same way as if you had received a core file from a crash.

Simple, easy and totally pain free debugging!

 copy no more than n bytes over from dst to src. … yeah yeah …

then whammo the next line from the docs jumped right out at me:

If the string pointed to by src is shorter than n characters, null characters are appended to the copy in the array pointed to by dst, until n characters in all have been written.

Wow! So strncpy() is always padding out the full size of the buffer with null characters. I don’t know why this hadn’t struck me before, but this time I was looking at some code that was working with a large maximum buffer (4K) but was most often dealing with strings that were only 10 or 20 characters long.  Since this was being done repeatedly, it seemed like a code change was going to be in order, something along the lines of (using the same src,dst and n):

int len = strlen(src) + 1;  

if(len > n) {
   len = n - 1;
   dst[len] = '';
}   

memcpy(dst, src, len);

This was kind of a drag since it was more code to write, but it was going to be a bit more efficient since we weren’t going to iterate over all 4K of the buffer padding it in with bytes we didn’t need … although we were passing twice through the string. Wanting to double check the order of the src and dst arguments for memcpy I headed back to the docs. I use the HTML index instead of the search so I jumped to the ‘M’ functions. Looking through the list I came across a library function I had never heard of before … memccpy so I thought I’d take a quick read of it:

Copy bytes between buffers until a given byte is found

Cool! That is exactly what I was looking for. The call does an efficient one pass iteration copying only the data bytes until it hit the matching character or it runs out of space to copy to.  I still needed to deal with ensuring the destination buffer was going to be null terminated (as with strncpy() ) but at least now I wasn’t spending any time filling in extra bytes as padding:

if(memcchr(dst, src, '', n) == NULL) {
dst[n-1] = '';
}

So, now I get the security of a bounded string buffer copy and efficiency to boot and I picked up a new standard C API for my toolbox … all because I like to browse the API instead of using the indexed search!  Of course, perhaps at some point this will get even more straightforward if the strlcpy() function becomes part of POSIX/ANSI C standard.

 struct _data {
  pthread_mutex_t m;
  pthread_condvar_t c;
  ... other stuff ...
 };
void * writer_thread(void *arg) {
  struct _data *d = (struct _data *)arg;
  ...
  pthread_mutex_lock(&d->m);
  pthread_cond_signal(&d->c);
  pthread_mutex_unlock(&d->m);
  ...
 }
void * reader_thread(void *arg) {
  struct _data *d = (struct _data *)arg;
  ...
  pthread_mutex_lock(&d->m);
  pthread_cond_wait(&d->c, &d->m);
  pthread_mutex_unlock(&d->m);
  ...
 }

The _intention_ here is that the writer_thread is signaling to the reader_thread that things are good and it can continue to operate. If this code were using sem_post()/sem_wait() the code would like operate as intended (assuming proper initialization values).However, imagine the following execution sequence (cause by context switches):

writer_thread:pthread_mutex_lock()
reader_thread:pthread_mutex_lock() > blocks (mutex held)
writer_thread:pthread_cond_signal()
writer_thread:pthread_mutex_unlock()
reader_thread:pthread_cond_wait() > blocks waiting
...

struct _data {
  pthread_mutex_t m;
  pthread_condvar_t c;
  int data;
  ... other stuff ...
};  

void * writer_thread(void *arg) {
  struct _data *d = (struct _data *)arg;
  ...
  pthread_mutex_lock(&d->m);
  d->data++;
  pthread_cond_signal(&d->c);
  pthread_mutex_unlock(&d->m);
   ...
}  

void * reader_thread(void *arg) {
  struct _data *d = (struct _data *)arg;
  ...
  pthread_mutex_lock(&d->m);
   while(d->data <= 0) {
  pthread_cond_wait(&d->c, &d->m);
  }
  d->data--;
  pthread_mutex_unlock(&d->m);
  ...
 }

void * reader_thread(void *arg) {
  ...
  while(d->data <= 5) {
  pthread_cond_wait(&d->c, &d->m);
  }
  d->data -= 5;
  ...

What we hope to be bringing you from this site in the coming weeks and months (years?) is the equivalent of the type of interesting QNX development tidbits that old QNX users will remember from the days of QNX News and “Ask Bruce” columns. 

As someone who interacts with QNX customer systems more often than not in an debugging and analysis capacity I hope to be able to share lots of great “lessons from the trenches” that will make your programs faster and your development life more pain free!

 Thomas