The gateway of my home network, is not one of those “broadband router”. Instead, it’s an old Pentium 200Hz machine in my basement, running, of cause, QNX. Why am I doing this? I think it’s one of those “because I can” thing. Since I compiled my own TCPIP stack, I can really know every detail of the packets in and out of my gateway. Another reason is, of cause, I like to “live on the edge”.

Yes, it’s really “bleeding edge”, though a lot of benifit and fun of running the HEAD branch stack, one of the disadvantage is, while in it’s early stage, the stack “crashes”. The good thing is I have the core dump I could look at, but the bad thing is, that’s also when my kids started shutting at me.

Those of you who had been managed a home network, would really understand how stressful this is. :) Fortunatly, soon my kids find out the “engineering way” to fix the problem. They went down to the basement, press the little reset button on the old Pentinum, give it a couple of minutes, and wola, everything comes back.

This works well for a while, but one day while I was at home along, the stack on gateway gone again. I have to get out of my comfort couch, went down to the basement and reset it myself. I said to myself, “why can’t I just write a program to resetart the network if it’s crashed”, after all, QNX is all about Micro Kernel and Modular System, isn’t it?

That’s where my “sockmon” program cames from. Once started, it will keeps on monitoring if TCPIP stack is still running, if it disappered, “sockmon” will try to execute a shell script you gave it on command line, to re-start the network. If the restart somehow failed after some try, then it will just reboot the system.

You may wonder “how do you know if TCPIP stack is there or not”? Well, QNX resource manager have builtin notification to all connected clients if their server disappeared. So all you need is to establish a connection to the tcpip stack (by call socket()), and setup to waitfor the notification events.

I have include the source here, the “notification” thing above is true for ALL resource manager (unless the manager is written in such way that turned off this feature), so you can easilly extended my program to any resource manager. Just give it a config file to read about which resource manager (what namespace you care) to watch out, and what to do (which script to execute) if the manager went away.

I will leave this for reader exercise, but if you did that, you would realiz you just got yourself a simple, basic, HA program.

 -xtang

#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <process.h>
#include <signal.h>
#include <string.h>
#include <syslog.h>
#include <sys/procmgr.h>
#include <sys/sysmgr.h>
#include <sys/neutrino.h>
#include <sys/socket.h>

int main(int argc, char **argv)
{
        char *script;
        int sd, chid, fcount;
        struct _pulse pulse;

        if (argc < 2) {
                fprintf(stderr, “sockmon <re-start script>\n”);
                return -1;
        }

        script = argv[1];
        if (access(script, X_OK) != 0) {
                fprintf(stderr, “access(’%s’): %s\n”, script, strerror(errno));
                return -1;
        }

        /* creat a channel for accept COIDDEATH pulse */
        if ((chid = ChannelCreate(_NTO_CHF_COID_DISCONNECT)) == -1) {
                perror(”ChannelCreate”);
                return -1;
        }

        /* don’t care about the child */
        signal(SIGCHLD, SIG_IGN);

#ifdef NDEBUG
        if (procmgr_daemon(0, PROCMGR_DAEMON_NOCLOSE) == -1) {
                perror(”procmgr_daemon”);
                return -1;
        }
#endif

        openlog(”sockmon”, LOG_PID, LOG_DAEMON);
        setlogmask(LOG_UPTO(LOG_INFO));

        for (;;)
        {
                fcount = 0;

                /* connect to tcpip to monitoring, give it 30 seconds, if still can’t
                 * connect, reboot the system
                 */
                while ((sd = socket(AF_INET, SOCK_DGRAM, 0)) == -1) {
                        if (++fcount >= 6) {
                                syslog(LOG_ERR, “Can’t connect to socket after 3 minutes, reboot…”
);
                                spawnl(P_NOWAIT, “/bin/slay”, “slay”, “-f”, “syslogd”, NULL);
                                sleep(1);
                                sysmgr_reboot();
                                return 0;
                        }
                        sleep(5);
                        syslog(LOG_INFO, “Connect to Socket failed: %m”);
                }

                syslog(LOG_INFO, “Connected to Network, start monitoring…”);
                if (MsgReceivePulse(chid, &pulse, sizeof(pulse), NULL) == -1) {
                        syslog(LOG_ERR, “MsgReceivePulse(): %m”);
                        return -1;
                }

                if (pulse.code != _PULSE_CODE_COIDDEATH) {
                        syslog(LOG_ERR, “MsgReceivePulse(): %m”);
                        return -1;
                }

                if (pulse.value.sival_int != sd) {
                        syslog(LOG_ERR, “COIDDEATH pulse for %d\n”, pulse.value);
                        continue;
                }

                syslog(LOG_INFO, “Network gone, restarting…”);
                spawnl(P_WAIT, “/bin/ksh”, “/bin/ksh”, script, NULL);
        }

        return 0;
}

At long last, the dtrace project at Foundry27 is public. It’s really still in the formative stages, but if you want to download the source and hack around a bit, or even just laugh at the dirty hacks, feel free!

I’m slowing working on a more thorough port, but the prototype is a nice forum to play around in.

Have fun…

Colin

If you’ve ever taken a kernel trace of an application starting up on a kernel that is 6.3.2 or later, you might have noticed a thread state in your application called STATE_WAITPAGE.

To understand what is going on here, we have to first look at mmap() and how it allocates and initializes memory.

When you allocate memory with mmap(), it first allocates a virtual address range, and then (unless you’ve passed MAP_LAZY), it will allocate the physical memory needed to back that object.

What it doesn’t do, though is setup all the page table mappings for the mapping (although in actual fact it will setup a certain amount, depending on heuristics based on the type of mapping).

In actual fact it will wait until the program first accesses a page before setting up a page table entry for it.

This is a change from pre-6.3.2 kernels, where all mappings were setup immediately. You can re-enable the old behaviour by using the procnto option -mL in your buildfile.

The benefits of this scheme, known as demand paging, are a speedup when the application doesn’t actually access all of the pages in a mapping. This is quite useful when mapping executable objects, since often there are quite a few pages in the text section which may never be referenced.

In some patterns of usage, though, this can induce a performance penalty, especially when there are many threads running at the same priority.

This brings us back to STATE_WAITPAGE. When a process is executing and it access a page that has not been mapped, or tries to write to a page that is marked read-only, it will induce a page fault. This is caught by the kernel.

The fast path in the kernel will peek at the processes pagetables, and if found it will bang it into the TLB (this is done in hardware on some architectures, such as the x86). If it can’t find it then it needs to defer the work of handling the fault to the process manager, since it may need to do some complex work such as communicating with device drivers, and also the also the necessary structures may not be accessible/consistent.

This, then, is where the thread gets blocked in STATE_WAITPAGE, and a pulse is sent to procnto, to wake up a thread to handle the request.

The procnto worker does the dirty work of initializing the memory (it may need to read a page from a disk driver for example) and setting up the pagetables.

The first time a page is read it is setup with a read-only mapping, even if the mapping is writeable, unless the underlying page has already been marked as modified. For MAP_LAZY mappings, this is the point at which physical memory is actually allocated for the page. If the allocation fails, then the application will have a SIGBUS signal delivered to it (this was the same in pre 6.3.2 kernels).

This delayed initialisation supports two handy schemes.

This first is Copy On Write semantics (COW). This is where we don’t bother making a private copy of a page that was originally shared with another mapping until the page is modified.

The second is support for writeable mappings to files. Prior to 6.3.2 you could only map a file readonly, or with the extra flag, MAP_NOSYNCFILE. This was because there was no tracking of modified pages.

When you call msync() on a shared mapping to a file, all the modified pages are written to the backing store. Now the pages can be marked read-only again, and the modified indicator can be turned off.

Now all this is great, but what about that performance penalty? Well all that context switch can make the page fault processing take quite a while if there are lots of threads in STATE_READY at the same priority. The procnto thread will be placed at the back of the queue, wait for the others to run, the it will run, potentially talk to a device driver, and then make the original thread ready again, which will be placed at the end of the queue.

Another thing to think of in some circumstances you want to take the hit of talking to the backing store all at once for determinism reasons.

A way to control this behaviour is via the mlock() and mlockall() functions. These tell procnto that you want the some (or all) pages made memory resident right now.

This means that the mapping will have been fully read in from disk by the time the mmap() call returns to your program.

You still get page faults on the first write to a page, though. In that case we setup a read-only mapping (or if the underlying pages have already been modified, a read-write mapping).

If you TRULY don’t want page faults for mappings, then there are two options open to you. Since device drivers may have their code run in an interrupt handler, and they may also be running code with interrupt disabled, they can’t take page faults at all. So when you request IO privity with the ThreadCtl() call, you get marked as being super locked. This means that all mappings are setup and ready to go.

Of course, only processes running with superuser privilege can request this special status. To have all processes be marked as super locked, you can pass the -mL option to procnto. To have all processes simply be locked (the equivalent of calling mlockall(MCL_FUTURE|MCL_CURRENT), you can pass the -ml option.

Well, this post has gone all over the map <groan>, so I’m going to sign off.

Later,

Colin

Well sorry for not having written for so long. We’ve all been very busy working on the 6.4 kernel.

BTW - if you’ve been wondering about the being CoreOS source repository out of date, the sync process between the internal and external repositories broke down. :-(

The good news though is that it’s fixed now, so you can get all the goodness straight from the source. :-)

Colin

As some of you may know, Thomas and I have been working on a port of dtrace, mostly in our spare time. I was intrigued and enthused to see that it had also been ported to FreeBSD and Mac OS X Leopard.

I was dismayed, however, to read this blog entry by one of dtrace’s inventors, Adam Leventhal, on a discovery he had made about the Leopard port.

I won’t repeat the details of it here, but the gist of it is that Apple allows for certain processes to be hidden from any debug/tracing systems. While you may or may not agree on whether that is Truly Evil Behaviour or not, it doesn’t really matter - what it means for sure is that the Leopard dtrace port is severely broken.

When a system wide tracer cannot trace the whole system, it pretty much ceases to be of any use!

You might as well pack up, and go home, folks!

The really sad thing is the way they broke it! They basically disable any probes from running at the same time as a protected process, the prime example being iTunes.

This means, for example, that anything based on the timer tick is useless, since you lose the regular heartbeat, and that any kernel activity that may have been deferred (say disk io) is lost, unnoticed by your dtrace probes!

It really means that you cannot trust any of the data that dtrace is supplying, as it may be at best incomplete, at worst incorrect.

I shake my head, I really do.

Colin

In the spirit of open development, QNX has created a new product - the Pocket Geek! He works in transparent cube (get it, transparent development!!)

Head over to http://thepocketgeek.com for some recreational fun, trying to drive your little pocketgeek to complete his tasks without exploding. There’s also a contest at the end of it all.

Have fun!

Colin

Most of QNX is off this week on a yearly holiday break that lasts from Christmas to New Years.  

Most of the developers end up being busy playing with new toys (thanks Santa) or tinkering with pet projects (CDT integration with Mylar) that they’ve had brewing on the side, so January ends up being a pretty exciting time to be back in the office finding out what everyone’s been up to.

However, this year I’m going to miss out on that.  Starting in January I’m turning main responsability of SRR over to my partner in crime Colin as I’m starting three months of parental leave to hang out with my 9 month old son and teach him the essentials of life:

• Recursion, can’t program effectively without thinking it!
• Snow fort construction, can’t live in Ottawa without it!
• Appreciation for solid foods … just makes sense!

So as this is my last post for a while (likely, but not certainly) I want to wish you all a happy holiday and a wonderful new year!

Thomas

OK … so I promise that this will be my last post about foundry27for a while and I’ll get back to talking about QNX technology.  But I couldn’t help myself, I needed to put out one more plug for some of the QNX Neutrino software that is being made more accessible to developers.

This time it is the availability of the IDE, also commonly known as QNX Momentics, although that statement will get me in trouble since QNX Momentics is more than just the IDE, but it is the complete assembly of a development environment for QNX Neutrino … but I digress.

The IDE is not part of the current roadmap for source publishing, but the QNX tools development team is shifting to performing its development work more transparently.  To that end there are two community projects, the Core Development Tools project and the the Integrated Development Environment (IDE) project.  From the Core Development Tools project you can get the bleeding edge versions of the GNU toolchain as they are developed … which is interesting, but most commercial projects currently under development aren’t likely to switch to a non-released (by QNX) compiler for their production work.   The IDE also offers bleeding edge integration build downloads and since the IDE pretty much sits on top of any core tool or runtime components, it can be updated with far less risk to a production development environment.   While I hate to compare the IDE to an editor, updating your IDE is like updating your editor:  In theory new features and functionality should only make you more productive!

The IDE team is tracking what significant new and noteworthy features are being added to each of the integration builds so you can get an immediate idea of what areas you might want to explore.  A more visual page is alsotracking the general IDE 4.5 features that you might also want to take a look at.  There is a lot of goodness in these downloads, ranging from the source navigation and usability improvements in the C/C++ development environment to the brand new Application Profiler functionality and a range of smaller, but equally rewarding, fixes and enhancements.

The other benefit to trying it out early is that the IDE team is actively looking for feedback and there is still time in the schedule before the next major release … codenamed Athens … to respond to feedback and feature requests.

Of course I’m a bit biased since I’ve had a hand in putting some of the new IDE features in, but I’ve switched for my C/C++ development and I’m not going back!

Thomas

A couple of weeks ago I wrote about the port of dtrace that Colin and I have got up and running under Neutrino.   For the most part, porting software to Neutrino is a pretty straightforward matter.  Since Neutrino runs self-hosted (ie you work and build in the same environment that you deploy) it means that all of the ‘configure/make’ style projects work without too much hassle.  In fact most of the time if the source is coming from Linux, *BSD, Solaris or some other unix flavour you can get it up and running in a couple of hours.  The main challenge is almost always figuring out the build system and making sure that our compiler front end (qcc) is used to pick up all of the required environment bits and that hacks for other systems are really required for Neutrino (which is pretty well behaved).

Having ported software on a self-hosted system is pretty much essential for a software developer to work effectively.  There are so many essential software bits and pieces that QNX would never include with the Neutrino SDK because while they are useful, the SDK is targeted specifically at what is needed to build Neutrino applications and needs to be portable (and consistent) for use under Windows, Linux, Solaris as well as Neutrino self-hosted x86.   That doesn’t mean that the extra software bits aren’t useful, and historically a 3rd party CD was provided for Neutrino self-hosted developers that came with loads of useful extras, from scripting languages (perl, python, tk) to web servers (apache) and editors (vim).

A couple of developers here at QNX, Sean B. and Xiaodan T., with the help of some QNX community members have been working on a wicked awesome project that pretty much replaces the 3rd party CD with something even better!  They have been working to port the BSD pkgsrc tool/project to support the QNX Neutrino self-hosted environment.   Just the other day I got a message in my mailbox indicating that the project had hit a critical milestone and that QNX Neutrino was now considered a full fledged supported platform:

It is even listed on the projects main documentation page … now that is super cool news!  There are 7000+ different software packages part of this collection, and to have all of them available pre-ported and on-demand is a pretty awesome feat.

So if you are running Neutrino self-hosted, (and why wouldn’t you be!) and you want more than just the stock installed software, then you should definitely take a look at the pkgsrc community project. 

Congratulations and thanks to Sean, Xiaodan and all the others who’ve been working on getting this up and running!

Thomas

I recently received an interesting kernel trace from a customer. It showed several threads going from STATE_RUNNING to STATE_MUTEX without an intervening SyncMutexLock kernel call. He wondered if perhaps the trace was corrupted.

After looking at the trace it seemed to be sane, but the behaviour shown was somewhat puzzling. However it is explainable, and it has to do with our implementation of mutexes.

No kernel calls?

One of the first things you must realize, when looking at a QNX kernel trace, is that a pthread_mutex_lock() doesn’t always result in a SyncMutexLock() kernel call. This is because we implement an optimisation where an uncontested mutex can be aquired without entry to the kernel. Likewise, unlocking a mutex on which no-one is waiting will not result in a SyncMutexUnlock() kernel call. This is implemented using atomic compare-and-exchange operations. How they are implemented depends on which cpu you are using. Take a look at $QNX_TARGET/usr/include/<cpu>/smpcmp.h if you are interested in the various ways of doing this.

Not our man though…

State changes without seeing kernel calls might point us in that direction then, but the fact is that we are seeing a change to STATE_MUTEX, which would indicate a contested mutex.

The next clue though is that we ARE, in fact, seeing these state changes while in the kernel. The kernel call was a MsgReceivev(). This led me to the answer.

When we unlock a mutex that other threads are being waited on, we call SyncMutexUnlock(), which checks a priority ordered list of waiting threads. It then will remove the first thread from the list and mark it runnable (ie it will transition to STATE_READY). It also sets a thread flag, _NTO_TF_ACQUIRE_MUTEX, which indicates that the thread should aquire the mutex it was waiting on upon exit from the kernel. The address of the mutex was saved in the thread kernel calls arguments when it called SyncMutexLock(), some time in the past.

The point is, though, that the thread has not yet aquired the mutex. This can lead to some interesting behaviour.

In the log that was sent to me, the holding thread unlocked the mutex, and then immediately tried to relock it. The first unlock makes a waiting thread ready. SyncMutexUnlock() is not a blocking call though, so the thread remains running.

At this point, no-one holds the lock, although the lock indicates that people are waiting on it.

The following call to SyncMutexLock detects that there are threads waiting, and readies another thread (it has no knowledge that there was already a thread readied) to attempt to acquire the mutex. It then blocks the calling thread, and at this point the first thread that wanted to aquire the mutex will be made running.

Time passes, at at some point the second thread that was marked ready may actually be made running. As it exits the kernel, the kernel exit code processing notices the aquire flag is set on the thread, and attempts to aquire the mutex. Of course it is still held by the earlier thread, and thus we need to block this thread and choose another.

Capice?

And there you have it. I’m sure it’s all a little odd without better knowledge of how kernel is implemented. The good news is that you can checkout the source and see for yourself. Also don’t forget to checkout the evolving docs on the project wiki page.

The moral of this story is to not get too upset by multiple thread state changes when we are within the kernel. The kernel can change it’s mind several times on which thread will ultimately make it out of the kernel and become running. The last state change before you see a kernel exit is the one to believe.

Colin

Anyone who has been plugged into the OS scene is surely aware of the latest Mac OS X Leopardrelease.  It is hard to miss since the news is pretty much splashing up all over the place in news articles and product reviews and in my case filling my mailbox with Google Alerts.  I don’t have an alert set for Max OS X, but I do have one set for DTrace and this tracing functionality has now been included in the latest Mac OS X release and people seem to be pretty jazzed about it.

I’m jazzed about it too … but for a different reason all together. 

Colin and I have been following the work of Bryan, Mike and Adamon dtrace since it was released as part of the Open Solaris.  We head up most of the kernel instrumentation and system profilingdevelopment on Neutrino and have always felt that dtrace, with its focus on runtime safety and use in a production environment, would be a perfect match for embedded developers and compliment our existing instrumentation and tracing strategy for Neutrino very nicely.  

So why were we jazzed by the Max OS X release? The more platforms that dtrace is ported to, the more generic the code base will become.  It is exciting to us because now dtrace is running on Solaris, NetBSD, Max OS X and oh yeah … Neutrino!

You can see here that this is a system that is running the 6.3.0 release of Neutrino and this is the basic dtrace command help.  The initial port to Neutrino that Colin and I have done is slightly different from the other OS ports.  Since Neutrino is an micro-kernel, we wanted to see how far we could go keeping the dtrace module out  of the kernel … and if possible even outside of the process manager.  The current implementation has all of the dtrace system code, normally found in the kernel or a kernel module, encapsulated in a resource manager, io-dtrace, and the utility is a straight port of the Solaris utility.

We’ve only implemented three sets of providers currently. 

• dtrace:  This is the standard dtrace provider with the BEGIN, END and ERROR probes
• instr: This is a provider based on the existing kernel instrumentation hook and can provide kernel calls, thread state, interrupt information
• profile: This provider is pretty much the same as the Solaris provider and provides statistical sampling at a user controlled rate

Even with this minimal set of providers, you can extract some pretty interesting information.  For example, we can re-write the hogs utility, that tracks the runtime of the top few processes, with a simple dtrace script:

We are missing a lot of the standard dtrace built-ins that allow you to get much deeper into the application space, such as providing the user stack information, but even by recording the instruction pointer we can get some usefull information about where an application is spending its time … expanding on the idea of an application specific statistical profiler, we can gather information about the whole system or target just one process or thread:

Since dtrace can work directly with C constructs, we can pull in code directly from the Neutrino system headers.  If you wanted to take a look at the kind of messaging operations that were going on in a process, you could write a script to catch all of the MsgSend’s via the instrumentation hook and decode the first few bytes of data based on the message header to determine what an application is doing:

Pretty cool huh! It is a nice extension to the capabilities of the System Profiler that provides the graphical analysis capabilities, but in a development environment back on the host.

There is still have a lot to do and the port is far from complete.  Before we can go much further, we are likely going to have to do the kernel and/or process manager integration to be able to efficiently provide things like the equivalent of the pid provider and the function boundary tracing.  The good news is that as a proof of concept this is already usefull … and so far architecturally independent (running on MIPS, SH, ARM, PPC and of course x86), although the hard bits are still ahead of us. 

Hats off to Bryan, Mike and Adam for a wicked awesome technology and to Sun to making it available to the software development community.  Since the port isn’t quite complete, certainly not production ready, Colin and I are hoping that we can get our prototype posted up to the Foundry27 QNX community site so that we aren’t the only ones who get a chance to play around with this. 

Stay tuned …

Thomas (& Colin who is onsite with a customer … likely using the dtrace prototype right now)

One of the hardest things about working through problems with some of our customers is the fact that since I work here at the QNX mothership office I have full and complete visibility into all source from our products … and they do not.  This of course means that I can almost always come out looking like a hero since for any hard question a customer has about how something works (or appear not to as the case may be) I can generally root through the source repository, figure out what is going on and explain it back to them.

It is nice to be a hero … but sometimes things just get lost in translation.

Since it used to be that I couldn’t just share the code with the customer’s engineers, we spend a lot of time going back and forth discussing topics in the “abstract” until I can paint enough of a mental picture for them about a particular implementation of technology.  By the time that they understand what to do about the problem they are facing, I’ve pretty much given them a tour through our code.  Kind of a  waste of time considering that most of the software developers I end up working with can read through a blob of C, C++ or Java code just as easily as a paperback novel.

This is why I think that Foundry27 is an awesome idea.  We are working to get to the point where for nearly all of the Neutrino technology that has been developed, I’ll be able to talk customers through a problem and bring them right into the guts of the code during the explanation.   It is going to take some time to get all of that source out, but we have already pushed out the C library source code and the kernel/process manager code and that represents quite a bit of important glue bits that help you understand how to get the most out of your applications.

Even better than the fact that the source is being published, is the fact that the different development teams within QNX are starting to take on much more active public roles themselves and helping others to interpret and navigate through the source code.  If you missed it, you should really check out the  webinar that Colin and Sebastien presented as a bit of a guided tour through some of the source code included in the QNX Operating System project.  Also we’ve got most of the kernel developers (and a large number of other QNX developers) online now and monitoring the forums so there is lots of good information flowing around.

Next up on the source release roadmap is the networking stack and driver source and I’m also going to see if I can get a set of missing OS utilities out the door as well before Christmas!

Dan posted recently on his blog about implementing an unblock handler in one of our media servers.  Unblock handling is tricky and it can take a bit of fiddling to get an implementation that makes the appropriate trade off between unblock response time and the complexity of implementation.  In this case a significant customer needed the work done in short order so it was done.

The funny thing is that a few days later the customer was then starting to get all sort of spurious errors in their system.  Normally this isn’t a challenge to track down, but in this case the system was tightly coupled and composed of hundreds of threads with tens of processes and close to a dozen active interrupts.  There was enough other system activity going on that a small failure in one path took a long time to ripple through to a point where the error was noticeable.  The gut feeling was that some part of the client software was being unblocked as a result of the recent server unblock changes and the client wasn’t ready to deal with the consequences.

… but we needed to prove it …

The good news is that the system was running the instrumented kernel and we could take some traces and take a look at the entire system, so traces we took and then we loaded them into the System Profiler.

The question was, how can we quickly check and see if the unblock behaviour is at fault?  Well the first thing to do is to see if there is any unblock activity in the log.  This is easily accomplished with using the Trace Search capability,  Search > Search… (or CTRL+H for the keyboard addicts).  Click Add to create a new search condition.  The condition we want to look for is the Communication class events with the code of Unblock Pulse.  This will pull up all of the unblock pulses generated in the trace.  In this case we didn’t have very many to consider:

We could have gone through and taken a look at each instance, but in this case we knew that we had just put the unblock handling code into the mme server.  The unblock pulse event contains the name of the target process, so we could take a look specifically to see if we had targeted our server.  We can do this by going back into the Trace Search and either editing the existing condition or creating a new condition.  We will still specify the Communication class and the Unblock Pulse event, but this time we will also add in a specific search for instances where the processdata is equal to the mme server.

Now when we run the search BINGO!we get just one single hit.  From this point it is easy now to look at the Timeline editor pane and track the behaviour of the client process and thread that the mme unblocks and what happens after the unblock (bad things it turns out).

Also, with the system trace we can take a look at the specific cause of the signal.  It turns out in this case that there was a timer firing a signal that got dropped on the thread which caused it to unblock.  This in turn exposed a general flaw in the customer’s design since they were running a multi-threaded application where any thread could have potentially been blasted by this signal instead of the process having a dedicated signal handling thread.

Hopefully this quick example gives you some more ideas how you can effectively navigate through the instrumentation trace files, Trace Search is a powerful tool.

Thomas

I was discussing with one of the other developers here some source code that comes from a very popular open source database that has been ported to QNX and that is going to be shipping with the next release of the OS. It is funny the way that developers get into this mindset that the code they are writing will be the only code in the entire universe running.

The snippet of code we were looking at looked something like the following (code changes made to protect those who should know better):

/*
** Sleep for a little while.  Return the amount of time slept.
** The argument is the number of milliseconds we want to sleep.
*/
int OSSleep(int ms){
  usleep(ms*1000);
  return ms;
}

Can you see what the flaw in the logic is here? Perhaps if we put the code into a context the problem will become clearer. The OSSleep function is being called in order to walk our way through a series of progressively longer timeouts .. based on the return value of how long this function has slept.

If the application containing this source code was the only thing running on the system, then there would be no problem. However, in most modern operating systems, QNX Neutrino counted among them, there are multiple tasks that are all running concurrently and being managed by the OS.  This means that while surely the code above will delay for N milliseconds, it could in fact be much longer than that if some other thread pre-empts this thread before or after it makes it into (or out of) the system call.

So .. what could we do to make it better?  Well we could snapshot the time on input and snapshotting it on exit and returning the difference.   This makes things incrementally better by narrowing down the window of error, but the window is still there. 

The reality is that this kind of API, and this application’s use of it, is fundamentally flawed.  What the application is trying to emulate is allowing multiple threads of execution the ability to acquire a resource.  If the resource can’t be acquired, then the thread backs off or may choose to cancel the operation all together. 

This is a perfect scenario for a condition variable … something available under Neutrino, but not necessarily under the other Unix flavours that this source was ported to.  By switching to a condition variable, we can be much more precise with respect to when we signal client applications to “try again” … which can result in a surprisingly large reduction in overall CPU consumption and system churn (due to all the extra context switches and cycles through the scheduling READY queue).

So in these days of thinking about multiple threads of execution running on separate CPU cores, we should also remind ourselves that using “time” as a basis for control needs to be carefully planned on systems where there is more than a single thread of execution.

Thomas

A few months ago, back in the dark agains, BF (Before Foundry27), I hurriedly blogged about trying to turn off the annoying Beep - the bane of laptop owners anywhere.

Well, today I had a few minutes, and so I went of in search of a more complete solution.

Switching Modes

As you may remember, one of the issues was that the beep control sequence, was only understood by devc-con when it was running QNX4 Terminal emulation mode, which is not the default.

I was telling this to my colleague, Peterv, and he said - well just emit the code to switch to QNX emulation and back.

Well, it was one of those ‘Duh’ moments. :-)

Of course, the fun part was figuring out the codes to emit. I couldn’t find a decent doc of the QNX4 Terminal Emulator codes, and the ANSI code to switch to QNX4 was a private vendor extension.

After a bit of mucking around though, I figured it out…

The Dark Majik

And yeah shall chant…

#!/bin/sh

# switch to QNX4 emulation
printf "%c[?+q" 0x1b

# set beep tick count = 1, frequency divider to 1

printf "%cs%c%c%c" 0x1b 33 33 32

# switch back to ANSI emulation
printf "%c?1+q" 0x1b

Now, make sure you put this output to /dev/con1, because that is the device that actually talks to the PC beep port. Photon pterm terminals do understand these codes, but they just send a bell character to /dev/con1!

Enjoy a beepless world, my friends.

BTW - you can muck around with the parameters. You might find you like a beep but just a bit shorter - each tick is 50ms.

Ever since I get my hands on QNET, the question “where is ditto” seems following me. At some point, I decided to give it a try, to see what’s the problem. Turns out the ditto functionality on QNX4, is mostly built into the console device. The character devices on QNX6 has a different architecture, to do the samething, I would have to “port” those functionality into, right, libio-char; Not only libio-char is owned by Dan Dodge at the time (I am not kidding :D), at the time it already built into a whole lot character devices, including those non-x86s. So basically I am stuck.

One day I realized all “ditto” doing (or need to do), is to ”sniff” characters between applications and the character device. Just like a network packet sniffer. Applications are going to write() into, for example, /dev/con1; All Ineed to do, is to find out what has benn written in, buffer them, for someone else to read out.

So how do I do this? QNX6 resource manager allow you register already exist path name, to, un, hijack it. So all I need to do, is write my own resrouce manager (called devc-ditto :D), it also attach to /dev/con1; The default behavior of resource manager pathname attach, is the ”latest attched got message first”, which works very well. So once devc-ditto attached to /dev/con1, any one open(”/dev/con1″, ..) it will end up with a connection from application to devc-ditto. All the applications write() would comes to me, I buffer those data it is written, then relay it to the resource manager who is really managing /dev/con1 (devc-con); Done!

Well almost done, this only allow me to sniff out the datas, how do I pass it to another application? This is easy, if I take over /dev/con1 for example, I would then create a /dev/con1.ditto, read from this device, you got all it buffered.

So here is how you use it:

On a console 2, try 

  # devc-ditto /dev/con1
  # cat /dev/con1.ditto

Then go back to console 1, do “ls >/dev/con1“, and come back, you should see all the “ls” output (from cat).

Wait, this is not what ditto doing right? Why is the 2 character “ls” not showing up in “cat” ? Think what happened, you hit “l” key, devc-con got it, give it to your shell, youe shell will echo it, by sending it to /dev/con1… but hey devc-ditto doesn’t see this “l”, why?

Because, the shell made a connection directly to devc-con, not to me. OK, so if I exit the shell, and re-login, that would do it, right? Sorry, no, because “login” spawned “sh”, the shell inherite stdin, stdout, stderr from “login”, in other words, the shell never open() the /dev/con1 device. Hm, so I have to run devc-ditto before “login” running? Yes, you can do that, by put devc-ditto in /etc/rc.d/rc.local, it actually works….

But, but I don’t like that. I want to be able to start/stop/restart devc-ditto freely. After talk to some expert, I have put this in my .kshrc.

  alias reopen ‘exec 0<$TTY 1>$TTY 2>$TTY’

Now, exit and login again on console 1, and type

   # reopen

Wohoo, after that, anything you type in concole 1, you should be able to see it in “cat”.

“But that’s now all ‘ditto’ doing, I also want to type in from ‘the other window’”, yes, I hear you. To allow inject key, I made it so any character write() into /dev/con1.ditto device, would be read() out from the /dev/con1. So what do you need to inject key? Yes, use “qtalk”.  Now there are a few gocha of “qtalk”, first of all, you need a qtalk support “-O” option, which ignores file open count; Also qtalk tend to open a character device in “edit” mode, we want “raw” mode.  So here is what you do, (assume you are on /dev/con2)

    # qtalk -O -m /dev/con1.ditto
    <in qtalk>
    # stty raw </dev/con2

After this, whatever you type in qtalk, would show up in /dev/con1, and processed by the shell running there. This ultimatlly gives you the same “ditto” experence. And of cause, all these are standard utilities (cat, qtalk), so they work over QNET just fine.

But this is not all of it. Since the principle is just hijack a device name, devc-ditto not limited to only console, you can use it against any character device, /dev/ser1, or maybe /dev/ttypX. Wait, “/dev/ttypX”, isn’t that means I can ”ditto” into a telnet session? Exactly. I will leave the practice for you, to figure out how to ”ditto” into a telnet session.

That’s all I have to say.  come to ostech forum if you want to discuss more, enhance it if you have more idea. Oh, one more thing, I made the devc-ditto also handle mount/umount, so you don’t need to start a lot of them if you want to monitor multiple interfaces.

   # mount -Tditto /dev/ttyp5

Now you can tap into /dev/ttyp5.ditto, after you done:

   # umount /dev/ttyp5.ditto

xtang

A couple of months ago we wrote about the missing QNX4 ditto for Neutrino and while script is an interesting alternative, the comments coming from some of you indicated that you just weren’t going to be happy until you got your ditto back!

The good news is that a QNX developer, Xiaodan Tang, took it upon himself to help out and solve this ditto dilemma.  Xiaodan has written a new Neutrino service, aptly called devc-ditto, that will allow character buffer replication and general terminal emulation.

While solving a lingering problem is always good news, the next question people will inevitably ask is “When can I get devc-ditto?“. 

The new devc-ditto is a good thing.  As a general rule, we like to include good things in our product distribution.  Here is where the bad news used to have to be delivered.  Our software releases are “steady and stable” which at times can be an euphamism for “glacially paced“.  This means that to get devc-ditto you would either have to wait for the commercial release (potentially a long time) or give your QNX sales rep a call and figure out a way to get this excellent piece of accellerated technology into your hands.

… but the times they are a changin’

QNX is shifting its entire R&D team to a new Transparent Development model.  In this model the source code for our products is being published live and the work on those products is taking place publicly.    Still commercial software development but with a flavour that should make open source developers, who like probing the product source and interacting with the authors/maintainers the source, much more at home.

I can hear you thinking “Yeah yeah … neat.  Now how does this help me get devc-ditto?!“

Well QNX has a lot of source to release, and it will take some time to clean it up into publically presentable form, but working with Xiaodan and the Core OS team we have been able to get devc-ditto included in the first OS source release, which means you can get it, build it, use it, change it all to your hearts content!

1. Get a community username (aka your MyQNX username)
2. Get the latest development tools and distribution (if you just want devc-ditto, and you have a dev seat, you can skip this)
3. Get the source: svn co –username=<community_username> http://community.qnx.com/svn/repos/coreos_pub/trunk/services/devc-ditto
4. Build it: make

I’ll leave the use-it up to you to figure out now … if in doubt, post to the OS forums with your questions!  If you want to get more and build more then you should really check out the whole Core OS project as well as the other community projects.

Thomas 

Back in the day, some 10 years ago, when I was a young Kiwi programmer on the far end of a very long copper wire into 175 Terence Matthews Crescent, being a QNX customer was a lot of fun. QUICS was cool (and I heard they were going to move their BBS system to a Usenet based NNTP reader! WOW!), and you really got a sense of working WITH the team of talented individuals in far off, exotic Ottawa.

More often that not bug reports would be greeted with a friendly post saying - “Thanks! Try this and let me know if it fixes your problem!”, and you’d be downloading a binary straight from the desk of Real Live Kernel Engineer.

Ah, the thrill of finding hidden paths and filenames embedded in that debug binary.

What it must be like to be a PART of that select club, to be involved directly in the creation of the most successful micro-kernel RTOS out there…

Wouldn’t you want to be part of that?

Well, I’ve been there over 10 years now, and all I can say is… I can’t wait until Wednesday!!!!!

:-D

I was just coming off of my holidays and thought that I should run a quick check of my mailbox to see what new and exciting things had transpired while I was away and blissfully ignorant of the comings and goings at the office.

What I found in my mailbox I found rather amusing.  I had four separate requests for help in getting the QNX debugger up and running using the IDE. 

I’m not doing (much) IDE development these days but these types of requests still manage to find their way to me because of my dogged insistance that I use our Momentics tools for all of my development and also because I think that printf  debugging is for dinosaurs.  If the IDE can’t get the job done for you, then we need to fix it so that it can … feel free to tell us so!

Anyway … back those e-mails.  They all were stating that they just couldn’t get the IDE debugger to work.  The platforms were all different (arm, sh, x86) and the environments were all different (Windows, Linux and Neutrino) but the problem was the same.  The application would launch, but the IDE debug session would fail with a cryptic connection error.

Why is this happening?  For most IDE services, the qconn target agent is all that needs to be up and running.  However, the debugger also makes use of the pdebug debugger target agent.  This is launched automatically for you when you start debugging. 

… unless it isn’t. 

The problem is that since the IDE is telling qconn to launch the pdebug program, pdebug needs to be in one of two places:

• Accessible via qconn’s PATH environment variable
• Located in /usr/bin/pdebug

If the pdebug debugger target agent can’t be launched, then the debug session will fail … and the resulting error message provides absolutely no indication of what is going on and why the debug session failed.

While this bit of information is recorded in the documentation for the IDE, it isn’t the first thing you would find when things go wrong.

So … if debugging wasn’t working for you and you never understood why and flipped back to printf’s take a close look at your configuration (and/or make sure that pdebug is in /usr/bin) and try it again … I think you’ll like it!

Thomas

Ok, here you go. I looked into the mystery that is devc-con, and I found a way to modify the beep.

The unfortunate side is that it only works if you are running devc-con in QNX mode, so you will either have to do that (by passing the -Q option) or suffer with the loud beeps (for now)

Ok, so go change your build file, and make sure your TERM env var is qnx

So we need to emit an escape sequence, which can be done thusly…

# printf “%cs%c%c%c” 0×1b, <c1>, <c2>, <c3>

Where c1 c2 and c3 are the parameters…

Here’s a snippet of the state machine…

case STATE_QNX4_11:     /* esc_beep_set */
s->beep_ticks = (c <= ' ') ? 6 : c - ' ';
s->esc_flag = STATE_QNX4_12;
break;
case STATE_QNX4_12:
s->xpos = c - ' ';
s->esc_flag = STATE_QNX4_13;
break;
case STATE_QNX4_13:
n = (s->xpos % 192) + ((c - ' ') % 192) * 192;
if(!n)
s->beep_counter = 0x0606;
else
s->beep_counter = (n &lt; 19) ? 0xffff :
(unsigned)(1193180L / (long)n);
break;

The default values for beep_ticks and beep_counter are 6 and 0×0606, respectively.

I haven’t quite worked out the optimimum, but basically if you screw it up, the beep is gone anyways, so I’m alright with that. You guys can have fun playing.

I’ll try and work a nobeep facility into the next version of devc-con.

Sorry this is brief, I’m running out of time today. I’ll try and elaborate more on exactly WHAT these things mean later (or maybe Thomas will…)

Enjoy!

Colin

So weeks go by and you’re probably thinking

What is up with those send receive reply guys? They haven’t posted anything in weeks!

True … it has been a few weeks, but we haven’t been idle.  There are exciting things a brew at QNX during the summer heat … air conditioning being one of them that tends to only make sporadic appearances unfortunately.

One of the more interesting features that is has worked its way into the Neutrino kernel for the next release (slated for the not too distant future) is going to be the ability to report out the mapping of physical memory of a system to virtual addresses used by individual processes. 

“But we live in a virtual address world“,  you say “why would we want a physical map of a systems memory?”

That was certainly my first impression.  Particularly with the fact that you can get a complete virtual address mapping for a process using the /proc filesystem and the appropriate devctl() and the entries for <sys/procfs.h> and <sys/debug.h>, you should be able to paint a fairly complete memory story all using virtual address information.

Turns out that that isn’t quite the case.  You can get close, but there are still some missing details when it comes to how particular shared libraries and executables are mapped within other mapped areas (for example using execute in place within secondary image file systems).  Getting access to the physical address allows you to resolve that ambiguity quite nicely.

Also, while I mentioned in the article on fragmentation (yes … we will get back to that issue =;-) that the OS nominally hands out memory uniformly in ~4K chunks, it does have the ability to fragment over time.  In general this isn’t a problem since the MMU for the processor provides the mapping giving you virtual continuity over fragmented physical memory.

Unless you need (or want) physically contiguous memory.  This can lead to improved performance in some device drivers that can leverage DMA and direct IO data transfers.  It is also a requirement by some tooling like the instrumented kernel, that arranges its data buffers as one contiguous ring.

The interface for extracting this additional memory information will be through the /proc interface and is a multi-stage procedure, but the fact that the information is now available is great news.  

Now all we need is a tool to showcase this new information … nothing like writing a handy utility to help beat the heat!

 Thomas

This week I’m celebrating my nine year anniversary working with QNX.  When I started at QNX the company was in the throes of a major product development effort to bring QNX 6/Neutrino to market as its next generation operating system with full support for processor families other than x86. 

When I started MIPS, PPC ports were well underway with ARM and SH support starting up.   I’m not an expert in any particular processor architecture, so I ended up working through a lot of the general system bugs and polishing some of the rough edges of this new OS.  By necessity this meant spending a lot of time interacting with the very active set of QNX developers, who at that time were steeped in QNX 4 knowledge and trying to migrate their systems and software to this new OS.   There are a lot of basic similarities between QNX 4 and QNX 6 but there are enough differences that migration wasn’t a zero effort affair, despite a fairly comprehensive software migration kit.

About once a month I’d receive a request from QNX4 developers:

“When will ditto be ported to Neutrino?”

Of course, the first time I heard this comment I was intrigued since ditto is not a standard *nix utility, so I thought I’d look into it.  It turned out that porting this particular utility (which provides terminal replication and remote terminal access) to Neutrino was going to take some signficiant work and more or less be an entire re-write since the utility took specific advantage of how QNX 4 character device drivers were written.  Under Neutrino the serial driver code was completely different,  so at the time I turned my focus to other areas where fixes could be made more short term with a plan to come back to do the ditto re-write.

Here we are, nine years later and there are still people asking for ditto for Neutrino. 

Boy do I feel bad about never getting back to that work =;-(

Well since I still haven’t ported ditto yet, the first thing I’m going to do is suggest an alternative that may be a more mainstream solution in these days of Linux fervor.  The GNU Screen utility provides much of the same terminal multiplexing and remote or shared control that ditto did.  The source compiles, with a few minor changes, out of the box for Neutrino. While I haven’t taken it for a full test drive yet, this looks like it might be the best immediate solution and maybe even long term since there are several screen resources available in the community.

However, ditto ain’t totally dead yet.  The other day walking through our source tree on a related project I noticed that one of our aspiring developers has done some work to make some of the ditto remote monitoring (no interaction) capabilities available for Neutrino.  This is still internal goods, and not in the next OS release, but perhaps this article might nudge it along if there is some competition.

Thomas