Planet Debian

I am currently working on a Qt-based app that needs to communicate through the serial port. Apart from all the benefits that a normal lib with a serial port implementation will bring in this case, having a Qt-based serial port lib would make even more sense, as it should be as multiplatform as possible and use the signal/slot mechanism. Also it should have a DFSG-compatible license, so I can package it for Debian, of course :-)

So I have found two libs which seemed to have the above mentioned features: QExtSerialPort and QSerialPort.

QExtSerialPort seems to be the most recommended lib in the web. It features polled and signal-based functionality; it uses Qt's standard types inheriting QIODevice. But it does not states the license in any file within the source code. The original project page at SourceForge says it's in public domain. And the newer project page at Google code says it's under the new BSD license. I have asked in the mailing list for a clarification. So far nothing has changed (although in further threads the authors showed some willing to change this). And then I got to the point of finding a bug, but I don't want to spend time to track it down and make a patch without a clear license.

QSerialPort it's another lib with more or less the same features as QExtSerialPort. It's main LICENSE file says it's under the LGPL2, but licensecheck will say that the present files are LGPL3. Also, on reviewing the code, I found some minor stuff that could be improved. Well, I could contact the author and see if [s]he would receive the patches... but his site seems down. And I could not find a real-person's name in the code so far :-/

So I made a last attempt to try to get QExtSerialPort in a suitable license. If it doesn't suceed, I think I'll have to start writing one myself. The downside: I only use Linux, so there will be no multiplatform features unless someone else contributes it. Of course, if you have another option or any idea to share, I'll be happy to know it :-)

By the way, this should be my first post on Planet Debian in english, so hello planet!

My last post missed an important thing about GHC 7.4's handling of encodings for FileName. It can in fact be safe to use FilePath to write a command like rm. This is because GHC internally uses a special encoding for FilePath data, that is documented to allow "arbitrary undecodable bytes to be round-tripped through it". (It seems to do this by encoding the undecodable bytes as very high unicode code points.) So, when presented with a filename that cannot be decoded using utf-8 (or whatever the system encoding is), it still handles it, and using the resulting FilePath will in fact operate on the right file. Whew!

Moral of the story is that if you're going to be using GHC 7.4 to read or write filenames from a pipe, or a file, you need to arrange for the Handle you're reading or writing to use this special encoding too. I use this to set up my Handles:

import System.IO import GHC.IO.Encoding import GHC.IO.Handle fileEncoding :: Handle -> IO () fileEncoding h = hSetEncoding h =<< getFileSystemEncoding

Even if you're only going to write a FilePath to stdout, you need to do this. Otherwise, your program will crash on some filenames! This doesn't seem quite right to me, but I hesitate to file a bug report. (And this is not a new problem in GHC anyway.) If I did, it would have this testcase:

# touch "me¡" # LANG=C ghc Prelude> :m System.Directory Prelude System.Directory> mapM_ putStrLn =<< getDirectoryContents "." me*** Exception: <stdout>: hPutChar: invalid argument (invalid character)

Since git-annex reads lots of filenames from git commands and other places, I had to deal with this extensively. Unfortunatly I have not found a way to read Text from a Handle using the fileSystemEncoding. So I'm stuck with slow Strings. But, it does seem to work now.

PS: I found a bug in GHC 7.4 today where one of those famous Haskell immutable values seems to get well, mutated. Specifically a [FilePath] that is non-empty at the top of a function ends up empty at the bottom. Unless IO is done involving it at the top. Really. Hope to develop a test case soon. Happily, the code that triggered it did so while working around a bug in GHC that is fixed in 7.4. Language bugs.. gotta love em.

Compilation of Linux kernel takes about an hour.

Assuming you’re using recent kernel and usual hardware available on the market at that time.

Thank you for your attention.

I wonder how many breaches never see the light of day.

Quite late but at least confirmed, VeriSign was hacked and they don’t even know what was the impact of this attack.

Comodo, DigiNotar, Verisign… did I forget some company which CA/SSL infrastructure has been compromised?

In less then 2 hours I'll leave for the Paris Nord station to catch a train headed to Bruxelles Midi. Plan of the week-end: attend and enjoy FOSDEM 2012!.

I haven't submitted any talk for this year FOSDEM edition, but I've been invited (and gladly accepted) to join the round table on working with contributor communities on Sunday. I'm positive it will be a nice occasion to share ideas on how to structure local user groups around the world.

Beside that, I plan to attend several talks of the cross-distribution, legal issues devrooms, hang around the Debian booth, as well as discuss many topics with people and friends from all over the Free Software multiverse.

Too bad I'm still recovering from a recent minor health issue; I won't be able to get the most out of today's beer event. But I'll attend nonetheless, see you there?

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:

In Total:1447 Affecting Wheezy:865 Wheezy only:170 Remaining to be fixed in Wheezy:695

Of these 695 bugs, the following tags are set:

Pending in Wheezy:46 Patched in Wheezy:91 Duplicates in Wheezy:42 Can be fixed in a security Update:18 Contrib or non-free in Wheezy:15 Claimed in Wheezy:0 Delayed in Wheezy:8 Otherwise fixed in Wheezy:44

Ignoring all the above (multiple tags possible) 482 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released.

However, with the view of the Release Managers, 767 need to be dealt with for the release to happen.

Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

In a few hours, I'll be flying to Brussels with Ivan, for a new edition of FOSDEM, undoubtedly the best Free Software conference in Europe.

I'm looking forward to hang out with Debian, GNOME and #dudes people, as well as to explore some other quiet and cool spots in the city with our hosts Raül and Vir.

I'll probably be around the CrossDistro and CrossDesktop rooms most of the time, but before that I'll be at the Delirium café not long after landing in Brussels.

For someone who doesn't enjoy cold weather that much, this is going to be a special edition… oh dear, -10℃, this is fucking crazy!

I'm going to deliver this talk at fosdem 2012, room H.1301 (CrossDistribution Devroom) at 16:30 on Sat. If you are interested, please come by. In particular I'd like to talk with all the developers out there that are using our work (of edos fame) and to discuss with them future plans to migrate their programs to the new generation of mancoosi - powered QA tools. Slides and links will appear in the space next week.

fosdem link : http://fosdem.org/2012/schedule/event/distro_qa_tools

Abstract

FOSS distributions are increasingly over pressure to deliver stable releases including the most up to date upstream software. Time-based release strategies has exacerbated this problem putting even more pressure on QA teams. The recently concluded Mancoosi project has developed a number of tools to automatically analyse large packages collections for a range of problems, from installability checks to speculative analysis of software repositories. In this talk I'll present four command line tools to identify and correct potential problems as soon as possible during the release cycle.

In particular :

• Debcheck: This tools helps to identify all broken packages within a repository and provides a detailed explanation of the problem. This can be used to prevent shipping releases that contain packages that cannot be installed because of missing or malformed dependencies.
• Buildcheck: Given a Sources file and a set of binary repositories, this tool identifies those source packages that cannot be compiled because their build dependencies cannot be satisfied.
• Outdated: This tool identifies those broken packages that need special attention because of outdated meta-data.
• Challenged: This tool performs a speculative analysis of the repository to identify those packages that, if upgraded to a specific version, would break a large number of other packages in the repository. This tool would be particularly useful during the upgrade of a specific component to evaluate its impact on the software archive.

Most of our tools support both rpm (version 4 and 5) and deb based distributions.

The mancoosi team.

It’s only me that is being annoyed by downloading all recommended packages?
How many of you have such file?

someserver:~# cat /etc/apt/apt.conf.d/02recommends APT::Install-Recommends "0"; someserver:~#

Have you ever wondered how to start getting involved in Debian/Ubuntu? Do you enjoy discovering new games and playing them? You might want to come to the games screenshot party! We hope that the party will be a fun, easy, low-commitment way to get involved.

The Debian/Ubuntu games team is organising a half-day screenshots party on the weekend of 25th-26th February for creating screenshots for all the games that are available in Debian/Ubuntu.

If you are interested in attending, please add your availability to the poll linked from the announcement so that we can get some idea of attendance and when is a good time for the people who are interested.

Look forward to lots of game playing, screenshots and a merry time, hope to see you all there!

I've just spent several days trying to adapt git-annex to changes in ghc 4.7's handling of unicode in filenames. And by spent, I mean, time withdrawn from the bank, and frittered away.

In kindergarten, the top of the classrom wall was encircled by the aA bB cC of the alphabet. I'll bet they still put that up on the walls. And all the kids who grow up to become involved with computers learn that was a lie. The alphabet doesn't stop at zZ. It wouldn't all fit on a wall anymore.

So we're in a transition period, where we've all learnt deeply the alphabet, but the reality is much more complicated. And the collision between that intuitive sense of the world and the real world makes things more complicated still. And so, until we get much farther along in this transition period, you have to be very lucky indeed to not have wasted time dealing with that complexity, or at least having encountered Mojibake.

Most of the pain centers around programming languages, and libraries, which are all at different stages of the transition from ascii and other legacy encodings to unicode.

• If you're using C, you likely deal with all characters as raw bytes, and rely on the backwards compatability built into UTF-8, or you go to long lengths to manually deal with wide characters, so you can intelligently manipulate strings. The transition has barely begin, and will, apparently, never end.
• If you're using perl (at least like I do in ikiwiki), everything is (probably) unicode internally, but every time you call a library or do IO you have to manually deal with conversions, that are generally not even documented. You constantly find new encoding bugs. (If you're lucky, you don't find outright language bugs... I have.) You're at a very uncomfortable midpoint of the transition.
• If you're using haskell, or probably lots of other languages like python and ruby, everything is unicode all the time.. except for when it's not.
• If you're using javascript, the transition is basically complete.

My most recent pain is because the haskell GHC compiler is moving along in the transition, getting closer to the end. Or at least finishing the second 80% and moving into the third 80%. (This is not a quick transition..)

The change involves filename encodings, a situation that, at least on unix systems, is a vast mess of its own. Any filename, anywhere, can be in any encoding, and there's no way to know what's the right one, if you dislike guessing.

Haskell folk like strongly typed stuff, so this ambiguity about what type of data is contained in a FilePath type was surely anathama. So GHC is changing to always use UTF-8 for operations on FilePath. (Or whatever the system encoding is set to, but let's just assume it's UTF-8.)

Which is great and all, unless you need to write a Haskell program that can deal with arbitrary files. Let's say you want to delete a file. Just a simple rm. Now there are two problems:

1. The input filename is assumed to be in the system encoding aka unicode. What if it cannot be validly interpreted in that encoding? Probably your rm throws an exception.
2. Once the FilePath is loaded, it's been decoded to unicode characters. In order to call unlink, these have to be re-encoded to get a filename. Will that be the same bytes as the input filename and the filename on disk? Possibly not, and then the rm will delete the wrong thing, or fail.

But haskell people are smart, so they thought of this problem, and provided a separate type that can deal with it. RawFilePath hearks back to kindergarten; the filename is simply a series of bytes with no encoding. Which means it cannot be converted to a FilePath without encountering the above problems. But does let you write a safe rm in ghc 4.7.

So I set out to make something more complicated than a rm, that still needs to deal with arbitrary filename encodings. And I soon saw it would be problimatic. Because the things ghc can do with RawFilePaths are limited. It can't even split the directory from the filename. We often do need to manipulate filenames in such ways, even if we don't know their encoding, when we're doing something more complicated than rm.

If you use a library that does anything useful with FilePath, it's not available for RawFilePath. If you used standard haskell stuff like readFile and writeFile, it's not available for RawFilePath either. Enjoy your low-level POSIX interface!

So, I went lowlevel, and wrote my own RawFilePath versions of pretty much all of System.FilePath, and System.Directory, and parts of MissingH and other libraries. (And noticed that I can understand all this Haskell code.. yay!) And I got it close enough to working that, I'm sure, if I wanted to chase type errors for a week, I could get git-annex, with ghc 4.7, to fully work on any encoding of filenames.

But, now I'm left wondering what to do, because all this work is regressive; it's swimming against the tide of the transition. GHC's change is certainly the right change to make for most programs, that are not like rm. And so most programs and libraries won't use RawFilePath. This risks leaving a program that does a fish out of water.

At this point, I'm inclined to make git-annex support only unicode (or the system encoding). That's easy. And maybe have a branch that uses RawFilePath, in a hackish and type-unsafe way, with no guarantees of correctness, for those who really need it.

Previously: unicode eye chart wanted on a bumper sticker abc boxes unpacking boxes

Sometimes two campaigns that I care about a lot pick the same day to hold an awareness-raising drive. It happened again on Tuesday.

The one I took part in was advertising the Stop ACTA London Protest on Sat 11 Feb. The Anti-Counterfeiting Trade Agreement (#ACTA) is a plurilateral international agreement on enforcement of so-called “intellectual property rights” – copyrights, trademarks and so on. It’ll have major implications for freedom of expression, access to culture and privacy. It will also harm international trade and stifle cooperation. (More background at EDRI or a fairly large AJE page – thanks to Occupy Bristol for the AJE link.)

So the one I didn’t support at the time was the Move Your Money UK launch day. That’s a great idea too, suggesting that if we, the 99%, are actually unhappy with the big banks and their titled leaders, we should move as much as possible out of those banks and into financial institutions that we control. As you might expect for someone whose first memory of mutuals is a trust account at the local building society, I support that too. I still have building society accounts, as well as banking with the co-op bank and recently joining my local credit union. I’ve moved my money. Why don’t you?

I didn’t try to support both campaigns simultaneously on social networks because I thought it would reduce the number of people who saw my message. I backed the ACTA protest because a lot of my networks were already discussing Move Your Money and I thought Stop ACTA would benefit more. Was that the right decision? Who can tell? What would you have done?

On Wednesday, February 1, a new virtual server May First/People Link recently rented went offline. We contracted the virtual server through Server4All because we need their un-metered 100Mbit connection to help us handle the bandwidth for Sahara Reporters, one of the most important independent news sources for Saharan Africa. With the server offline, the web site was down as well.

We scrambled to setup alternative caching servers to handle the bandwidth.

When I logged into our control panel, I saw the message: This virtual server has been suspended by the administrator. Please contact support.

I immediately contacted support and then received the message:

Hello We have received the following complaint associated with your server/service. IP: 76.73.121.164 To prevent any further abuse we have suspended this service. In order to resume, we request you to cooperate with our investigation as promptly as possible. Please respond to us with the following details: (1) What has caused the complaint (2) What is the server used for. Purpose? (3) How can you resolve the complaint and make sure it will not be repeated. Depending on the nature of the complaint and your response, we will put back the server online. Please note, this has violated our Terms Of Service. We expect your response within 24 hours, otherwise your account will be terminated permanently. Thank you

What?? What complaint?? I followed up but had to wait til the next day to get the response.

Here is the full log, An email advertizing the Domain Name: saharareporters.com has been sent to the blacklist.woody.ch spamtrap. This Domain does resolve to IP addresses one of which your are responsible: 76.73.121.164 Please investigate why this Domain has been advertized. Attached you find the headers and reports in ARF for automatic processing. Feedback is appreciated. Actual listing periods: Bounce: 1 Hour in DNS. Whitelisted IP: Not lised in DNS. Spam: 24 hours in DNS. Every Hit: 14 days in evidence DB. For any questions or Feedback, contact abuse@woody.ch From: is intentionally set to a bit-bucket. Kind regards -Benoit Panizzon-

There is no attachment. I went to woody.ch and it was in German. Then tried blacklist.woody.ch, but no luck. Finally I found the Woody's World Blacklist Page. I plugged in our IP address into their checker and I got:

Output from the Check, if empty the IP is not listed. 164.121.73.76.[name of the blacklist] being tested. Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN) Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN) Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN) Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN) Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN) Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN) ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN) Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN) Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN) Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN) Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN) Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN) Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN) Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN) Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN) Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN) 164.121.73.76.blacklist.spambag.org has address 208.91.197.182 164.121.73.76.blacklist.spambag.org descriptive text "v=spf1 -all"

So, Woody's World thinks we are listed in spambag.org. I went to spambag.org and learned that the domain is for sale. I then tried blacklist.spambag.org and got the same page. This page has many links all pointing to advertisements. The "RBL List" link takes me to a page advertising "5 foods you must not eat."

Next, out of curiousity, I tried Mxtoolbox. I got one hit from Barricuda. Barricuda says the reputation of the IP address is "poor". Why? According to Barricuda, the reasons could be:

• Your email server contains a virus and has been sending out spam.
• Your email server may be improperly configured.
• Your PC may be infected with a virus or botnet software program.
• Someone in your organization may have an infected PC with a virus or botnet program.
• You may be using a dynamic IP address which was previously used by a known spammer.
• Your marketing department may be sending out bulk emails that do not comply with the CAN-SPAM Act.
• You may have an insecure wireless network which is allowing unknown users to use your network to send spam.
• In some rare cases, your recipient's Barracuda Spam Firewall may be improperly configured.

Keep in mind, this IP address is not used for sending email. It's just the web site.

I then took a step back and re-read the complaint and noticed that it says that the domain name saharareporters.com was listed in a spam email. Hm. More concerted searching for the terms "woody spamtrap blacklist" and I found a pattern in URLs that suggested I plug in the following:

http://news.scoutnet.org/rblhostlist.php?id=saharareporters.com.uri

And sure enough, there was a result. In short, it was a classic Nigerian Oil scam in which the person claims to be "JAMES IBORI ex-governor of DELTA STATE oil city." The scammer acknowledges that he has been arrested, but promises lots of cash to the person who can help him. In an effort to boost their credibility, the scammer included a link to a Sahara Reporters article about the real James Ibori.

And that, my friends, is enough to have one of the most prominent independent African news organization taken offline.

But, Sahara Reporters shouldn't necessarily feel singled out. Sahara Reporters should feel singled out. They exist to illuminate news from Africa. They were taken offline because a series of individuals don't know the difference between a prominent independent African news service and a criminal scammer. To most of the Internet, Nigeria and email scams are synonymous. That has to change.

Although Sahara Reporters is particularly vulnerable, any site hosted with Server4All can potentially be taken down. All you have to do is write a fake spam/scam email, including a link to the web site you want to be taken offline, and then send that email to: listme@blacklist.woody.ch.

I'm currently following up with both Woody's World and Server4All. However, once this particular issue is resolved, we're left with a much bigger and ominous problem. If your hosting provider (or their upstream provider) takes your site offline when it receives a complaint first and then asks questions second, you have a big problem.

All of our legal fights over our rights to keep content online are moot if our providers, without any legal pressure to do so, still take down our services based on spurious complaints.

I was trying to build the curl hackage package on Debian unstable with ghc 7.0.3, and one of its modules were failing to build. I searched for the error message and found this GHC ticket. As mentioned in the ticket, I had to downgrade binutils to 2.20. The version of binutils in sid is 2.21.

Another possibility would be to change curl to avoid using -fvia-C, but I didn't want to modify the package.

After testing Emacs, Eclipse, KDevelop and CodeBlocks for writing C++ code, I decided to stick with Netbeans. It seems to be the most simple to configure and yet full of features and plugins. I was having one problem with it that when I focused out the window, and then focused on it again, it would not really grab the focus, in the sense that I would not be able to type without first clicking with the mouse. At first I thought the problem was with Netbeans, then with the JRE. I tried using sun-java6-jre, I tried upgrading my openjdk-6-jre, and nothing worked. I searched a little bit more and got to a discussion about this problem in ion3, and that made me think that the problem could be related to the window manager I use, XMonad. After searching a bit about it, I found on the XMonad FAQ some work arounds for problems with Java apps, but they didn't solve my problem. Then, I found the solution on this bug report. I got the darcs version of xmonad and XMonadContrib, included takeTopFocus on logHook, and now it's working!

My problem was: I had a number of databases generated in different machines and I wanted to query them as if they were one, using the database the data came from as a field while querying and while showing results. The databases are SQLite3 files, generated using SQLAlchemy in a Python program.

I solved this by using SQLAlchemy, which was good because I could use the same ORM mapping that the program used. I noticed that the Horizontal Sharding SQLAlchemy extension would fit well the problem, although not perfectly. I had to make some changes in some classes of this extension, and now it works fine.

It was possible to filter the data using database as a criteria, but I couldn't get the database information from each line of a query result. I made a simple patch to SQLAlchemy, which wasn't likely to be introduced in the distribution, but worked for me, and sent it to its bug tracker. The change was included in SQLAlchemy in a very different fashion, as expected, but since I'm using the released version from SQLAlchemy, I kept on using my version of the patch. I don't want to do direct changes in SQLAlchemy source code, so I made the change in my program:

class ShardedSessionShardId(ShardedSession): def __init__(self, *args, **kwargs): super(ShardedSessionShardId, self).__init__(*args, **kwargs) self._query_cls = ShardedQueryShardId class ShardedQueryShardId(ShardedQuery): def _execute_and_instances(self, context): if self._shard_id is not None: result = self.session.connection( mapper=self._mapper_zero(), shard_id=self._shard_id).execute(context.statement, self._params) news = list(self.instances(result, context)) for new in news: new.shard_id = self._shard_id return iter(news) else: partial = [] for shard_id in self.query_chooser(self): result = self.session.connection( mapper=self._mapper_zero(), shard_id=shard_id).execute(context.statement, self._params) news = list(self.instances(result, context)) for new in news: new.shard_id = shard_id partial = partial + news # if some kind of in memory 'sorting' # were done, this is where it would happen return iter(partial) create_session = sessionmaker(class_=ShardedSessionShardId)

Another problem is that I had to make each result be included in the query, even if two results from different DBs have the same primary key. I achieved this by changing two classes: WeakInstanceDict, and Mapper. For using the new WeakInstanceDict, I had again to change the ShardedSession variation:

class WeakInstanceDictNoIdentity(WeakInstanceDict): def add(self, state): # if state.key in self: # if dict.__getitem__(self, state.key) is not state: # raise AssertionError("A conflicting state is already " # "present in the identity map for key %r" # % (state.key, )) # else: dict.__setitem__(self, state.key, state) self._manage_incoming_state(state) class ShardedSessionShardId(ShardedSession): def __init__(self, *args, **kwargs): super(ShardedSessionShardId, self).__init__(*args, **kwargs) self._query_cls = ShardedQueryShardId self._identity_cls = WeakInstanceDictNoIdentity self.identity_map = self._identity_cls()

To start using the new Mapper, I simply replaced each call to mapper with MapperNoIdentity:

class MapperNoIdentity(Mapper): def _instance_processor(self, context, path, adapter, polymorphic_from=None, extension=None, only_load_props=None, refresh_state=None, polymorphic_discriminator=None): """Produce a mapper level row processor callable which processes rows into mapped instances.""" pk_cols = self.primary_key if polymorphic_from or refresh_state: polymorphic_on = None else: if polymorphic_discriminator is not None: polymorphic_on = polymorphic_discriminator else: polymorphic_on = self.polymorphic_on polymorphic_instances = util.PopulateDict( self._configure_subclass_mapper( context, path, adapter) ) version_id_col = self.version_id_col if adapter: pk_cols = [adapter.columns[c] for c in pk_cols] if polymorphic_on is not None: polymorphic_on = adapter.columns[polymorphic_on] if version_id_col is not None: version_id_col = adapter.columns[version_id_col] identity_class = self._identity_class def identity_key(row): return identity_class, tuple([row[column] for column in pk_cols]) new_populators = [] existing_populators = [] load_path = context.query._current_path + path def populate_state(state, dict_, row, isnew, only_load_props): if isnew: if context.propagate_options: state.load_options = context.propagate_options if state.load_options: state.load_path = load_path if not new_populators: new_populators[:], existing_populators[:] = \ self._populators(context, path, row, adapter) if isnew: populators = new_populators else: populators = existing_populators if only_load_props: populators = [p for p in populators if p[0] in only_load_props] for key, populator in populators: populator(state, dict_, row) session_identity_map = context.session.identity_map if not extension: extension = self.extension translate_row = extension.get('translate_row', None) create_instance = extension.get('create_instance', None) populate_instance = extension.get('populate_instance', None) append_result = extension.get('append_result', None) populate_existing = context.populate_existing or self.always_refresh if self.allow_partial_pks: is_not_primary_key = _none_set.issuperset else: is_not_primary_key = _none_set.issubset def _instance(row, result): if translate_row: ret = translate_row(self, context, row) if ret is not EXT_CONTINUE: row = ret if polymorphic_on is not None: discriminator = row[polymorphic_on] if discriminator is not None: _instance = polymorphic_instances[discriminator] if _instance: return _instance(row, result) # determine identity key if refresh_state: identitykey = refresh_state.key if identitykey is None: # super-rare condition; a refresh is being called # on a non-instance-key instance; this is meant to only # occur within a flush() identitykey = self._identity_key_from_state(refresh_state) else: identitykey = identity_key(row) # instance = session_identity_map.get(identitykey) # if instance is not None: # state = attributes.instance_state(instance) # dict_ = attributes.instance_dict(instance) # isnew = state.runid != context.runid # currentload = not isnew # loaded_instance = False # if not currentload and \ # version_id_col is not None and \ # context.version_check and \ # self._get_state_attr_by_column( # state, # dict_, # self.version_id_col) != \ # row[version_id_col]: # raise orm_exc.ConcurrentModificationError( # "Instance '%s' version of %s does not match %s" # % (state_str(state), # self._get_state_attr_by_column( # state, dict_, # self.version_id_col), # row[version_id_col])) # elif refresh_state: if refresh_state: # out of band refresh_state detected (i.e. its not in the # session.identity_map) honor it anyway. this can happen # if a _get() occurs within save_obj(), such as # when eager_defaults is True. state = refresh_state instance = state.obj() dict_ = attributes.instance_dict(instance) isnew = state.runid != context.runid currentload = True loaded_instance = False else: # check for non-NULL values in the primary key columns, # else no entity is returned for the row if is_not_primary_key(identitykey[1]): return None isnew = True currentload = True loaded_instance = True if create_instance: instance = create_instance(self, context, row, self.class_) if instance is EXT_CONTINUE: instance = self.class_manager.new_instance() else: manager = attributes.manager_of_class( instance.__class__) # TODO: if manager is None, raise a friendly error # about returning instances of unmapped types manager.setup_instance(instance) else: instance = self.class_manager.new_instance() dict_ = attributes.instance_dict(instance) state = attributes.instance_state(instance) state.key = identitykey # manually adding instance to session. for a complete add, # session._finalize_loaded() must be called. state.session_id = context.session.hash_key session_identity_map.add(state) if currentload or populate_existing: if isnew: state.runid = context.runid context.progress[state] = dict_ if not populate_instance or \ populate_instance(self, context, row, instance, only_load_props=only_load_props, instancekey=identitykey, isnew=isnew) is \ EXT_CONTINUE: populate_state(state, dict_, row, isnew, only_load_props) else: # populate attributes on non-loading instances which have # been expired # TODO: apply eager loads to un-lazy loaded collections ? if state in context.partials or state.unloaded: if state in context.partials: isnew = False (d_, attrs) = context.partials[state] else: isnew = True attrs = state.unloaded # allow query.instances to commit the subset of attrs context.partials[state] = (dict_, attrs) if not populate_instance or \ populate_instance(self, context, row, instance, only_load_props=attrs, instancekey=identitykey, isnew=isnew) is \ EXT_CONTINUE: populate_state(state, dict_, row, isnew, attrs) if loaded_instance: state._run_on_load(instance) if result is not None and \ (not append_result or append_result(self, context, row, instance, result, instancekey=identitykey, isnew=isnew) is EXT_CONTINUE): result.append(instance) return instance return _instance

I had to include some auxiliary definitions to make the rewrites work:

_none_set = frozenset([None]) _runid = 1L _id_lock = util.threading.Lock() def _new_runid(): global _runid _id_lock.acquire() try: _runid += 1 return _runid finally: _id_lock.release()

It would be good to be able to set these identity requirements as a parameter.

My last problem was selecting more than one database to search. setshard only worked for one, so I created a new field in query, called shards, and checked for it on querychooser:

def query_chooser(query): try: return query.shards except AttributeError: pass return tcs.keys()

So, when I want to look only in a list of shards, I set this field. I'm aware that this is not a recommended python idiom, but, well, it works fine.

I'm using GTK+ in my paid work, which is being cool because I like this library. It's not the first time I use it for paid work, and I like these opportunities to learn more about it and possibly report bugs. It's not easy to find bugs in such a widely used library, so much of them end up being simple cosmetic mistakes, like this one. Some others are only a question of interpretation, like this other, which even when I submitted a patch, it was not applied.

Yesterday I couldn't make popup-menu create a, well, popup menu. It was working with button-press-event, but not with popup-menu. I created a small code sample to illustrate this problem and asked about it in #pygtk@irc.gimp.org --- yes, I'm using Python:

import gtk import gobject def menu(): item = gtk.ImageMenuItem('gtk-add') item.show() menu = gtk.Menu() menu.add(item) return menu def on_view_popup_menu(obj, *data): print 'popup-menu' menu().popup(None, None, None, 0, 0) def on_view_button_press_event(obj, *data): print 'button-press-event' menu().popup(None, None, None, 0, 0) store = gtk.ListStore(gobject.TYPE_STRING) store.append(('teste',)) view = gtk.TreeView(store) view.append_column(gtk.TreeViewColumn('Coluna', gtk.CellRendererText(), text=0)) view.connect('button-press-event', on_view_button_press_event) view.connect('popup-menu', on_view_popup_menu) view.show() window = gtk.Window() window.add(view) window.show() gtk.main()

When I clicked in the window, with any button in this case, 'button-press-event' is printed to stdout and the menu is displayed. When I press the Menu key, or Shift+F10, 'popup-menu' is printed to stdout, but the menu is not displayed. After getting no response in #pygtk, I decided to try #gtk+. Before that, I thought about trying the same problem in C to see if it was not a problem in the bindings, or in how I was using it. The code:

#include <gtk/gtk.h> GtkMenu *menu() { GtkImageMenuItem *image_menu_item = GTK_IMAGE_MENU_ITEM( gtk_image_menu_item_new_from_stock(GTK_STOCK_ADD, NULL)); gtk_widget_show(GTK_WIDGET(image_menu_item)); GtkMenu *menu = GTK_MENU(gtk_menu_new()); gtk_container_add(GTK_CONTAINER(menu), GTK_WIDGET(image_menu_item)); return menu; } void on_tree_view_popup_menu(GtkWidget *widget, gpointer user_data) { printf("popup-menu\n"); gtk_menu_popup( menu(), NULL, NULL, NULL, NULL, 0, gdk_event_get_time(NULL)); } void on_tree_view_button_press_event( GtkWidget *widget, GdkEventButton *event, gpointer user_data) { printf("button-press-event\n"); gtk_menu_popup( menu(), NULL, NULL, NULL, NULL, 0, gdk_event_get_time(NULL)); } int main(int argc, char **argv) { gtk_init(&argc, &argv); GtkListStore *list_store = gtk_list_store_new(1, G_TYPE_STRING); GtkTreeIter iter; gtk_list_store_append(list_store, &iter); gtk_list_store_set(list_store, &iter, 0, "teste", -1); GtkTreeView *tree_view = GTK_TREE_VIEW( gtk_tree_view_new_with_model(GTK_TREE_MODEL(list_store))); gtk_tree_view_append_column( tree_view, gtk_tree_view_column_new_with_attributes( "Coluna", gtk_cell_renderer_text_new(), "text", 0, NULL)); g_signal_connect( tree_view, "popup-menu", G_CALLBACK(on_tree_view_popup_menu), NULL); g_signal_connect( tree_view, "button-press-event", G_CALLBACK(on_tree_view_button_press_event), NULL); gtk_widget_show(GTK_WIDGET(tree_view)); GtkWindow *window = GTK_WINDOW(gtk_window_new(GTK_WINDOW_TOPLEVEL)); gtk_container_add(GTK_CONTAINER(window), GTK_WIDGET(tree_view)); gtk_widget_show(GTK_WIDGET(window)); gtk_main(); }

Wow. The difference between the codes is impressive. I'm not that used to code in GTK+ using C, and I may have missed something here, but the C code is so verbose that I get tired only by looking at it. I'm glad we have other languages, and GTK+ bindings for them. As expected, the behavior was the same, and I asked in #gtk+ and got no answer.

Yes, I did read the documentation before asking, and no, I didn't read it carefully enough. Maybe because of coming from Python, I didn't checked very much the type of the callback function for the signals, and didn't notice the point about the return value: "Returns: TRUE if a menu was activated". And that was not where I was looking for the problem, since I thought the return value was usually related to the treatment of the signal by another functions. Anyway, I changed the type of on_tree_view_popup_menu to return gboolean and returned TRUE. It works! I also updated the type of on_tree_view_button_press_event to do the same, just for compliance. In Python, a simple return True would do it.

I know I could use the event parameter in on_tree_view_button_press_event, but the point was exactly to run the same code and see if it would work in one case and not in the other, as happened.

>>> x = ['a', 'b'] >>> x ['a', 'b'] >>> x += 'c' >>> x ['a', 'b', 'c'] >>> x += '' >>> x ['a', 'b', 'c'] >>> x += [''] >>> x ['a', 'b', 'c', '']

Soile, my partner and love, is a film-maker. She is funding her first feature length documentary via crowd-funding, and has just opened the Kickstarter page. Have a look and see if you like the project and if you do, any donation is most welcome.

There is a Debian bug squashing party (BSP) in Perth, Western Australia being organised for the weekend of 10-11th of March. It will be held at the University of Western Australia in loft of the University Computer Club (UCC) in the student guild building. Sadly it is a bit far to travel for the majority of Debian contributors (and they probably wouldn't enjoy being upside down) but hopefully we can attract some locals and get them addicted to fixing bugs and contributing to Debian and the FLOSS community in general.

Come one, come all, lets squish some bugs and get Debian into better shape for the coming freeze in June for the release of Debian 7 (wheezy).

Thanks go to the UCC folks for the venue and to PLUG folks for helping with organisation and promotion of the event.