Archive for September, 2007|Monthly archive page

More time than you want

I was discussing with one of the other developers here some source code that comes from a very popular open source database that has been ported to QNX and that is going to be shipping with the next release of the OS. It is funny the way that developers get into this mindset that the code they are writing will be the only code in the entire universe running.

The snippet of code we were looking at looked something like the following (code changes made to protect those who should know better):

/*
** Sleep for a little while.  Return the amount of time slept.
** The argument is the number of milliseconds we want to sleep.
*/
int OSSleep(int ms){
  usleep(ms*1000);
  return ms;
}

Can you see what the flaw in the logic is here? Perhaps if we put the code into a context the problem will become clearer. The OSSleep function is being called in order to walk our way through a series of progressively longer timeouts .. based on the return value of how long this function has slept.

If the application containing this source code was the only thing running on the system, then there would be no problem. However, in most modern operating systems, QNX Neutrino counted among them, there are multiple tasks that are all running concurrently and being managed by the OS.  This means that while surely the code above will delay for N milliseconds, it could in fact be much longer than that if some other thread pre-empts this thread before or after it makes it into (or out of) the system call.

So .. what could we do to make it better?  Well we could snapshot the time on input and snapshotting it on exit and returning the difference.   This makes things incrementally better by narrowing down the window of error, but the window is still there. 

The reality is that this kind of API, and this application’s use of it, is fundamentally flawed.  What the application is trying to emulate is allowing multiple threads of execution the ability to acquire a resource.  If the resource can’t be acquired, then the thread backs off or may choose to cancel the operation all together. 

This is a perfect scenario for a condition variable … something available under Neutrino, but not necessarily under the other Unix flavours that this source was ported to.  By switching to a condition variable, we can be much more precise with respect to when we signal client applications to “try again” … which can result in a surprisingly large reduction in overall CPU consumption and system churn (due to all the extra context switches and cycles through the scheduling READY queue).

So in these days of thinking about multiple threads of execution running on separate CPU cores, we should also remind ourselves that using “time” as a basis for control needs to be carefully planned on systems where there is more than a single thread of execution.

Thomas

Advertisements

More on BEEP

A few months ago, back in the dark agains, BF (Before Foundry27), I hurriedly blogged about trying to turn off the annoying Beep – the bane of laptop owners anywhere.

Well, today I had a few minutes, and so I went of in search of a more complete solution.

Continue reading

Devc-ditto, the principle, the usage

Ever since I get my hands on QNET, the question “where is ditto” seems following me. At some point, I decided to give it a try, to see what’s the problem. Turns out the ditto functionality on QNX4, is mostly built into the console device. The character devices on QNX6 has a different architecture, to do the samething, I would have to “port” those functionality into, right, libio-char; Not only libio-char is owned by Dan Dodge at the time (I am not kidding :D), at the time it already built into a whole lot character devices, including those non-x86s. So basically I am stuck.

One day I realized all “ditto” doing (or need to do), is to “sniff” characters between applications and the character device. Just like a network packet sniffer. Applications are going to write() into, for example, /dev/con1; All Ineed to do, is to find out what has benn written in, buffer them, for someone else to read out.

So how do I do this? QNX6 resource manager allow you register already exist path name, to, un, hijack it. So all I need to do, is write my own resrouce manager (called devc-ditto :D), it also attach to /dev/con1; The default behavior of resource manager pathname attach, is the “latest attched got message first”, which works very well. So once devc-ditto attached to /dev/con1, any one open(“/dev/con1”, ..) it will end up with a connection from application to devc-ditto. All the applications write() would comes to me, I buffer those data it is written, then relay it to the resource manager who is really managing /dev/con1 (devc-con); Done!

Well almost done, this only allow me to sniff out the datas, how do I pass it to another application? This is easy, if I take over /dev/con1 for example, I would then create a /dev/con1.ditto, read from this device, you got all it buffered.

So here is how you use it:

On a console 2, try 

  # devc-ditto /dev/con1
  # cat /dev/con1.ditto

Then go back to console 1, do “ls >/dev/con1“, and come back, you should see all the “ls” output (from cat).

Wait, this is not what ditto doing right? Why is the 2 character “ls” not showing up in “cat” ? Think what happened, you hit “l” key, devc-con got it, give it to your shell, youe shell will echo it, by sending it to /dev/con1… but hey devc-ditto doesn’t see this “l”, why?

Because, the shell made a connection directly to devc-con, not to me. OK, so if I exit the shell, and re-login, that would do it, right? Sorry, no, because “login” spawned “sh”, the shell inherite stdin, stdout, stderr from “login”, in other words, the shell never open() the /dev/con1 device. Hm, so I have to run devc-ditto before “login” running? Yes, you can do that, by put devc-ditto in /etc/rc.d/rc.local, it actually works….

But, but I don’t like that. I want to be able to start/stop/restart devc-ditto freely. After talk to some expert, I have put this in my .kshrc.

  alias reopen ‘exec 0<$TTY 1>$TTY 2>$TTY’

Now, exit and login again on console 1, and type

   # reopen

Wohoo, after that, anything you type in concole 1, you should be able to see it in “cat”.

“But that’s now all ‘ditto’ doing, I also want to type in from ‘the other window'”, yes, I hear you. To allow inject key, I made it so any character write() into /dev/con1.ditto device, would be read() out from the /dev/con1. So what do you need to inject key? Yes, use “qtalk”.  Now there are a few gocha of “qtalk”, first of all, you need a qtalk support “-O” option, which ignores file open count; Also qtalk tend to open a character device in “edit” mode, we want “raw” mode.  So here is what you do, (assume you are on /dev/con2)

    # qtalk -O -m /dev/con1.ditto
   
<in qtalk>
    # stty raw </dev/con2

After this, whatever you type in qtalk, would show up in /dev/con1, and processed by the shell running there. This ultimatlly gives you the same “ditto” experence. And of cause, all these are standard utilities (cat, qtalk), so they work over QNET just fine.

But this is not all of it. Since the principle is just hijack a device name, devc-ditto not limited to only console, you can use it against any character device, /dev/ser1, or maybe /dev/ttypX. Wait, “/dev/ttypX”, isn’t that means I can “ditto” into a telnet session? Exactly. I will leave the practice for you, to figure out how to “ditto” into a telnet session.

That’s all I have to say.  come to ostech forum if you want to discuss more, enhance it if you have more idea. Oh, one more thing, I made the devc-ditto also handle mount/umount, so you don’t need to start a lot of them if you want to monitor multiple interfaces.

   # mount -Tditto /dev/ttyp5

Now you can tap into /dev/ttyp5.ditto, after you done:

   # umount /dev/ttyp5.ditto

xtang

Ditto Delivered!

A couple of months ago we wrote about the missing QNX4 ditto for Neutrino and while script is an interesting alternative, the comments coming from some of you indicated that you just weren’t going to be happy until you got your ditto back!

The good news is that a QNX developer, Xiaodan Tang, took it upon himself to help out and solve this ditto dilemma.  Xiaodan has written a new Neutrino service, aptly called devc-ditto, that will allow character buffer replication and general terminal emulation.

While solving a lingering problem is always good news, the next question people will inevitably ask is “When can I get devc-ditto?“. 

The new devc-ditto is a good thing.  As a general rule, we like to include good things in our product distribution.  Here is where the bad news used to have to be delivered.  Our software releases are “steady and stable” which at times can be an euphamism for “glacially paced“.  This means that to get devc-ditto you would either have to wait for the commercial release (potentially a long time) or give your QNX sales rep a call and figure out a way to get this excellent piece of accellerated technology into your hands.

… but the times they are a changin’

QNX is shifting its entire R&D team to a new Transparent Development model.  In this model the source code for our products is being published live and the work on those products is taking place publicly.    Still commercial software development but with a flavour that should make open source developers, who like probing the product source and interacting with the authors/maintainers the source, much more at home.

I can hear you thinking “Yeah yeah … neat.  Now how does this help me get devc-ditto?!

Well QNX has a lot of source to release, and it will take some time to clean it up into publically presentable form, but working with Xiaodan and the Core OS team we have been able to get devc-ditto included in the first OS source release, which means you can get it, build it, use it, change it all to your hearts content!

  1. Get a community username (aka your MyQNX username)
  2. Get the latest development tools and distribution (if you just want devc-ditto, and you have a dev seat, you can skip this)
  3. Get the source: svn co –username=<community_username> http://community.qnx.com/svn/repos/coreos_pub/trunk/services/devc-ditto
  4. Build it: make

I’ll leave the use-it up to you to figure out now … if in doubt, post to the OS forums with your questions!  If you want to get more and build more then you should really check out the whole Core OS project as well as the other community projects.

Thomas 

Anticip………..ation!

Back in the day, some 10 years ago, when I was a young Kiwi programmer on the far end of a very long copper wire into 175 Terence Matthews Crescent, being a QNX customer was a lot of fun. QUICS was cool (and I heard they were going to move their BBS system to a Usenet based NNTP reader! WOW!), and you really got a sense of working WITH the team of talented individuals in far off, exotic Ottawa.

More often that not bug reports would be greeted with a friendly post saying – “Thanks! Try this and let me know if it fixes your problem!”, and you’d be downloading a binary straight from the desk of Real Live Kernel Engineer.

Ah, the thrill of finding hidden paths and filenames embedded in that debug binary.

What it must be like to be a PART of that select club, to be involved directly in the creation of the most successful micro-kernel RTOS out there…

Wouldn’t you want to be part of that?

Well, I’ve been there over 10 years now, and all I can say is… I can’t wait until Wednesday!!!!!

:-D