It’s been a more than week since the completion of the GSoC and I am happy to announce that I have successfully completed the project. This year’s GSoC couldn’t have been better. I was out of India for the first time to attend the Gnome Users and Developers European Conference(GUADEC) which was held at Strasbourg, France where I met met my mentor and a lot of other cool people who contribute to Gnome. I would like to thank Gnome Foundation for sponsoring me to visit the conference.
I always wondered what motivates someone to keep contributing to a software project. After attending the GUADEC I realised that a sense of belongingness in the community is a really as important if not, more so than the other factors such as sharing the same values and goals as the community. I had interesting conversations with my roommate sri about whats more important, whether adhereing to values or creating software that is contextual and useful and we came to a conclusion that both can be acheived at the same time. After attending GUADEC I have a strong feeling that I will keep contibuting to the project and the community.
Apart from that, I am also happy to announce that I have graduated recently after which I have been travelling which is the reason for the delay in this blog post.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Ubuntu is sponsoring the dConstruct “Living with the network” event on the 5th of September at the Brighton Dome. Stop by for a chat with the team, grab some goodies and enter our competition for a chance to win an Ubuntu Phone.
Ubuntu is one of the best Linux platforms with an awesome desktop for regular users (and soon phone and tablets and more!) and great servers for system administrators and devops. A number of developers are choosing Ubuntu as their primary development system of choice, even if they develop for platforms other than Ubuntu itself, like doing some Android development, web development and so on.
However, even if we fill the basic needs for this audience, we decided a few months ago to start a development and integration effort to make those users completely feel at home. Ubuntu loves developers and we are going to showcase it by making Ubuntu the best available developer platform!Sounds great! What's up then?
We decided to start by concentrating on Android developers. We'll ramp up afterwards on other use cases like Go developers, web developers, Dart… but we want to ensure we deliver a stunning experience for each targeted audience before moving on to the next topic.
After analyzing how to setup an Android development machine on Ubuntu we realized that, depending on the system, it can takes up to 9 different steps to get proper IDE integration and all the dependencies installed. The whole goal was to reduce that to one single command!
Concretely speaking, we created the Ubuntu Developer Tools Center, a command line tool which allows you to download the latest version of Android Studio (beta), alongside the latest Android SDK, and all the required dependencies (which will only ask for sudo access if you don't have all the required dependencies installed already), enable multi-arch on your system if you are on a 64 bit machine, integrate it with the Unity launcher…
As said, we focused on Android Studio (based itself on Intellij IDEA) for now as it seems that’s where Google has been focusing its Android tools development effort for over a year. However, the system is not restrictive and it will be relatively trivial in the near future to add ADT support (Android Development Tools using Eclipse).
Indeed, The Ubuntu Developer Tools Center is targeted as being a real platform for all developer users on Ubuntu. We carefully implemented the base platform with a strong technical foundation, so that it's easily extensible and some features like the advanced bash shell completion will even make more sense once we added other development tools support.Availability
We will always target first the latest Ubuntu LTS version alongside the latest version in development. Yes! It means that people who want to benefit for the extensively tested and strong base experience that a Long Term Support version offers will always be up to date on their favorite developer tools and be first-class citizen. We strongly believe that developers always want the latest available tools on a strong and solid stable base and this is one of the core principle we are focusing on.
For now, the LTS support is through our official Ubuntu Developer Tools Center ppa, but we plan to move that to the backports archive with all the newly or updated libraries. For Utopic, it's already available in the 14.10 Ubuntu archive.Initial available version
Be aware that the Ubuntu Developer Tools Center is currently in alpha. This tool will evolve depending on your feedback, so it's up to you to suggest the direction you want it to go! A blog post on how to contribute will follow in the next days. This initial version is available in English, French and Chinese!
Another blog post will expand as well how we test this tool. For now, just be aware that the extensive test suite is running daily, and it ensures that on all supported platforms we don't break, that the Ubuntu platform itself doesn't break us, or that any 3rd party on which we rely on (like website links and so on) don't change without us spotting it. This will ensure that our tools is always working, with limited downtime.Example: how to install Ubuntu Developer tools and then, Android Studio Ubuntu Developer Tools Center
If you are on Ubuntu 14.04 LTS, first, add the UDTC ppa:
$ sudo add-apt-repository ppa:didrocks/ubuntu-developer-tools-center $ sudo apt-get update
Then, installing UDTC:
$ sudo apt-get install ubuntu-developer-tools-centerHow to install android-studio
$ udtc android
And then, accept the installation path and Google license. It will download, install all requirements alongside Android Studio and latest android SDK itself, then configure and fit it into the system like by adding an Unity launcher icon…
And that's it! Happy Android application hacking on Ubuntu. You will find the familiar experience with the android emulator and sdk manager + auto-updater to always be on the latest.
We welcome any ideas and feedback, as well as contributions as we'll discuss more in the next post. Meanwhile, do not hesitate to reach me on IRC (didrocks on freenode, #ubuntu-desktop as the primary channel to discuss), or on Google+. You can as well open bugs on the launchpad project or github one
I'm excited about this opportunity to work on the developer desktop. Ubuntu loves Developers, and it's all on us to create a strong developer community so that we can really make Ubuntu, the developer-friendly platform of choice!Notes
 android-studio is the default for the android development platform, you can choose it explicitely by executing "$ udtc android android-studio". Refer to --help or use the bash completion for more help and hints
Read more of this story at Slashdot.
Sorry for a bit belated post. I attended GUADEC this summer, in Strasbourg, France. This was my second GUADEC and it was again a great experience.
The keynotes were inspiring and the status updates on GNOME core components were really insightful. I particularly liked the Cheese, Documents/Photos, and GTK+ talks, as I learned a lot of new concepts, along with the ideas behind them. I also found the Python and GNOME talk thought-provoking.
I presented a talk on input-method. It mainly discussed smart input-method features which can be found in other platforms, while exploring ways of improving the current architecture. Afterwards, I had nice chats with Aron and William, working on the same area.
On the BoF days, I was planning I18N BoF, which ended up as a random hacking/chatting session. Sorry about that, but it was fun and there were interesting discussions about translation tools and spell checking.
Today my postal vote in the referendum for Scottish independence was sent off. I usually use my personal blog for non technical bits but I thought some readers of my KDE Blog might be interested in this as it does affect the geopolitics of pretty much the whole world.
What's going on?
300 years ago Scotland was broke and England was getting rich from its empire so England gave Scotland a wad of money and created the United Kingdom. That served Scotland pretty well for some centuries as Scots were able to trade and move freely across the empire and plenty of Scots were happy to take part in that (my old school still has a load of Hong Kong gold in its attic as a result). Time passed and the empire was shut down, the UK joined the EU and some oil was found in the sea off the coast of Scotland. After many decades of Scottish public services being run by some people appointed by the UK government, 15 years ago a new Scottish pariament was reconvened to control public services and laws in Scotland, but not tax or anything international.
In Scottish politics we have not liked the Conservative (and Unionist or Tory) party since Margaret Thatcher destroyed Scottish industry. Then Labour went out of fashion after Tony Blair started some random wars and Scottish administration (not even wanting to call itself a government) decided to market Scotland by calling it small. Then their coalition partner party the Liberal Democrats fell out of fashion by getting into the UK government and dropping all their liberal principles (GCHQ has been doing mass surveilance on the population and they haven't said a word against it). So at the last election the only large party left that had not lost all respect, the Scottish National Party, was voted in with a majority and their Scottish Government is now organising this referendum.
What will happen if the vote is Yes?
Both sides have said they will respect the result. If over 50% of people vote yes then the Scottish Government will start to negotiate with the UK government on the details to make Scotland an independent country. Depending on your point of view this is either a new country (Scotland) and a continuing state (rest of UK) or dissolving the 300 year old union to make two new countries. The proposed timetable is to make Scotland independent by March 2016 (which is done to fit in with the election timetable of the Scottish parliament, it's made somewhat more complex by a UK election happening in 2015).
Why vote for Yes?
There is a democracy in Scotland which has been notably different from England and the rest of the UK for some time. The two parties that make up the UK government (Conservative and Liberal Democrat) are the two least popular parties in Scotland. Political borders are circles on a map which can be arbitrary or based on some nasty tribal allegance but here's a redrawing which makes government better follow the demos of the population.
Is this anti-English?
We can be proud in Scotland that independence is being done through a peaceful political process. That was not the case in Ireland were they had effectively a guerrilla civil war until recently over incomprehensible tribal allegances. And it's not the case that Scottish nationalism is in any way anti any other nation, Scotland would continue to be best friends with England and the other counties of the British Isles.
The Yes campaign has sensibly avoided any call to patriotism, kilts and Irn Bru tend to get old quickly. By contrast the No campaign has tried quite a lot of patriotism talking about shared ties and gosh remember the first world war wasn't it glorious? Meanwhile English politics is getting more and more little-Englander. The Prime Minister has said he will give a referendum on membership of the EU, he wants to pull the UK out of the Court of Human Rights and he is making immigration much harder, all good reasons to vote to stay away from that sort of politics.
But isn't it better to work together?
Yes, and Scotland has benefited from the union with England to not have barriers of trade or movement. Now we have a larger union, the EU, to sort out the boring stuff around trade and movement we have no need of that middle layer of government. Scottish independence is an efficiency drive.
Can Scotland afford to be independent?
This is a common worry. Scots are often not very confident in their own country. We're small and need the help of something larger is a common thought. Of course it's not true, Scotland is exactly in the middle if you place it in a list of countries by size or population. We have plenty of industry and natural resources. And then we have the largest oil reserves in Europe. Currently the UK spends all the money that comes in from the oil while any other well run country with oil creates a soverign wealth fund, a common argument against independence is that the oil will run out sometime, my argument for independence is we need to be independent toot sweet so we can start a soverign wealth fund before it runs out.
Will Scotland be allowed to join the EU?
There's no precendent for this happening and no rules governing this process. Those against independence say Scotland would need to apply to join the EU and the Spanish might block it to stop the Catalans getting ideas above their station. Those for independence point out that Scotland is already a member of the EU, that all the citizens have EU citizenship and there is no rule to kick us out. In the end politics will decide and the EU has a good record of welcoming in people rather than shunning them.
What Currency will Scotland use?
Ah, yes, slightly more tricky this and the No campaign has been playing it to the full. The Scottish Government wants to continue to use the pound sterling with a formal currency union with the rest of the UK. The current UK government says this would not happen but it's not clear why the rest of the UK gets to keep it and Scotland not, this does suggest the last 300 years of union have really been a sham. The Scottish Government in response points out that as a freely tradeable currency Scotland can use the pound if it wants and nobody can stop it (this is what Ireland did when it became independent and what the Isle of Mann and Gibraltar still do) and without a currency union Scotland would have no need to pay the massive national debt the UK has. Personally I'd like to use the Euro but that doesn't seem very fashionable these days for some reason.
What will happen to the BBC.. to the NHS.. won't the terrorists love it.. will Scotland have to join Schengen.. will immigrants steal our jobs and our women... will the UK get to keep its nuclear bombs?
The Scottish Government published a really long book with answers to all these details. They're just starting positions of course, after any Yes vote there will be lots of negotiation to work out how everything will be set up, but there is little that seems insurmountable.
They want to start a Scottish equivalent of the BBC which would mean we would actually get Scottish news in Scotland, currently half the news shown to use is irrelevant. It would make a formal agreement to allow us to keep important stuff like the World Service and Doctor Who. The National Health Service gives us free use of doctors and hospitals and any politician which says anything against it will be voted out quickly, the NHS is separate in Scotland than that in England so there's no change here, although some no campaign adverts have tried to claim otherwise. There has been some nonsense about terrorists loving it and we won't have MI5 and GCHQ to look after us all, to me the mass surveilance of the spy agencies is very much a reason to get away from that. I'd be all for Scotland joining Schengen and having closer ties with Europe but I expect we'll remain part of the British Isles Common Travel Area, it's only sensible politically. Scotland needs more immigrants for various economic reasons but in England there's a large political wish against it, another good reason to vote Yes. The UK nuclear bombs are kept north of Glasgow and sadly this is the only issue that has interested Slashdot. There is no desire to keep these in Scotland and the Scottish Government has promised to get rid of them mucho rapido, what the rest of the UK does with them is a problem for the rest of the UK.
All very exiting..
The vote is on 18th September and the polls show the No campaign stay steady at a bit below 50% and the Yes campaign started low at 30-odd percent but has been gaining ground as undecided voters move to Yes. There's now only a few points separating them and there are still plenty of undecided voters. The trend is for undecideds to move to Yes so it's very much all to play for. It's fun to live in interesting times.
Greets! I'm delighted to be able to announce the release of Pflua, a high-performance packet filtering toolkit written in Lua.
Pflua implements the well-known libpcap packet filtering language, which we call pflang for short.
At this point, regular readers are asking themselves why this Schemer is hacking on a Lua project. The truth is that I've always been looking for an excuse to play with the LuaJIT high-performance Lua implementation.
LuaJIT is a tracing compiler, which is different from other JIT systems I have worked on in the past. Among other characteristics, tracing compilers only emit machine code for branches that are taken at run-time. Tracing seems a particularly appropriate strategy for the packet filtering use case, as you end up with linear machine code that reflects the shape of actual network traffic. This has the potential to be much faster than anything static compilation techniques can produce.
The other reason for using Lua was because it was an excuse to hack with Luke Gorrie, who for the past couple years has been building the Snabb Switch network appliance toolkit, also written in Lua. A common deployment environment for Snabb is within the host virtual machine of a virtualized server, with Snabb having CPU affinity and complete control over a high-performance 10Gbit NIC, which it then routes to guest VMs. The administrator of such an environment might want to apply filters on the kinds of traffic passing into and out of the guests. To this end, we plan on integrating Pflua into Snabb so as to provide a pleasant, expressive, high-performance filtering facility.
Given its high performance, it is also reasonable to deploy Pflua on gateway routers and load-balancers, within virtualized networking appliances.
Pflua compiles pflang expressions to Lua source code, which are then optimized at run-time to native machine code.
There are actually two compilation pipelines in Pflua. The main one is fairly traditional. First, a custom parser produces a high-level AST of a pflang filter expression. This AST is lowered to a primitive AST, with a limited set of operators and ways in which they can be combined. This representation is then exhaustively optimized, folding constants and tests, inferring ranges of expressions and packet offset values, hoisting assertions that post-dominate success continuations, etc. Finally, we residualize Lua source code, performing common subexpression elimination as we go.
For example, if we compile the simple Pflang expression ip or ip6 with the default compilation pipeline, we get the Lua source code:return function(P,length) if not (length >= 14) then return false end do local v1 = ffi.cast("uint16_t*", P+12) if v1 == 8 then return true end do do return v1 == 56710 end end end end
The other compilation pipeline starts with bytecode for the Berkeley packet filter VM. Pflua can load up the libpcap library and use it to compile a pflang expression to BPF. In any case, whether you start from raw BPF or from a pflang expression, the BPF is compiled directly to Lua source code, which LuaJIT can gnaw on as it pleases. Compiling ip or ip6 with this pipeline results in the following Lua code:return function (P, length) local A = 0 if 14 > length then return 0 end A = bit.bor(bit.lshift(P, 8), P[12+1]) if (A==2048) then goto L2 end if not (A==34525) then goto L3 end ::L2:: do return 65535 end ::L3:: do return 0 end error("end of bpf") end
We like the independence and optimization capabilities afforded by the native pflang pipeline. Pflua can hoist and eliminate bounds checks, whereas BPF is obligated to check that every packet access is valid. Also, Pflua can work on data in network byte order, whereas BPF must convert to host byte order. Both of these restrictions apply not only to Pflua's BPF pipeline, but also to all other implementations that use BPF (for example the interpreter in libpcap, as well as the JIT compilers in the BSD and Linux kernels).
However, though Pflua does a good job in implementing pflang, it is inevitable that there may be bugs or differences of implementation relative to what libpcap does. For that reason, the libpcap-to-bytecode pipeline can be a useful alternative in some cases.
When Pflua hits the sweet spots of the LuaJIT compiler, performance screams.
This synthetic benchmark runs over a packet capture of a ping flood between two machines and compares the following pflang implementations:
libpcap: The user-space BPF interpreter from libpcap
bpf-lua: BPF bytecodes, cross-compiled to Lua by Pflua.
pflua: Pflang compiled directly to Lua by Pflua.
To benchmark a pflang implementation, we use the implementation to run a set of pflang expressions over saved packet captures. The result is a corresponding set of benchmark scores measured in millions of packets per second (MPPS). The first set of results is thrown away as a warmup. After warmup, the run is repeated 50 times within the same process to get multiple result sets. Each run checks to see that the filter matches the the expected number of packets, to verify that each implementation does the same thing, and also to ensure that the loop is not dead.
In all cases the same Lua program is used to drive the benchmark. We have tested a native C loop when driving libpcap and gotten similar results, so we consider that the LuaJIT interface to C is not a performance bottleneck. See the pflua-bench project for more on the benchmarking procedure and a more detailed analysis.
The graph above shows that Pflua can stream in packets from memory and run some simple pflang filters them at close to the memory bandwidth on this machine (100 Gbit/s). Because all of the filters are actually faster than the accept-all case, probably due to work causing prefetching, we actually don't know how fast the filters themselves can run. At any case, in this ideal situation, we're running at a handful of nanoseconds per packet. Good times!
It's impossible to make real-world tests right now, especially since we're running over packet captures and not within a network switch. However, we can get more realistic. In the above test, we run a few filters over a packet capture from wingolog.org, which mostly operates as a web server. Here we see again that Pflua beats all of the competition. Oddly, the new Linux JIT appears to fare marginally worse than the old one. I don't know why that would be.
Sadly, though, the last tests aren't running at that amazing flat-out speed we were seeing before. I spent days figuring out why that is, and that's part of the subject of my last section here.
on lua, on luajit
I implement programming languages for a living. That doesn't mean I know everything there is to know about everything, or that everything I think I know is actually true -- in particular, I was quite ignorant about trace compilers, as I had never worked with one, and I hardly knew anything about Lua at all. With all of those caveats, here are some ignorant first impressions of Lua and LuaJIT.
LuaJIT has a ridiculously fast startup time. It also compiles really quickly: under a minute. Neither of these should be important but they feel important. Of course, LuaJIT is not written in Lua, so it doesn't have the bootstrap challenges that Guile has; but still, a fast compilation is refreshing.
LuaJIT's FFI is great. Five stars, would program again.
As a compilation target, Lua is OK. On the plus side, it has goto and efficient bit operations over 32-bit numbers. However, and this is a huge downer, the result range of bit operations is the signed int32 range, not the unsigned range. This means that bit.band(0xffffffff, x) might be negative. No one in the history of programming has ever wanted this. There are sensible meanings for negative results to bit operations, but only if an argument was negative. Grr. Otherwise, Lua shares the same concerns as other languages whose numbers are defined as 64-bit doubles.
Sometimes people get upset that Lua starts its indexes (in "arrays" or strings) with 1 instead of 0. It's foreign to me, so it's sometimes a challenge, but it can work as well as anything else. The problem comes in when working with the LuaJIT FFI, which starts indexes with 0, leading me to make errors as I forget which kind of object I am working on.
As a language to implement compilers, Lua desperately misses a pattern matching facility. Otherwise, a number of small gripes but no big ones; tables and closures abound, which leads to relatively terse code.
Finally, how well does trace compilation work for this task? I offer the following graph.
Here the tests are paired. The first test of a pair, for example the leftmost portrange 0-6000, will match most packets. The second test of a pair, for example the second-from-the-left portrange 0-5, will reject all packets. The generated Lua code will be very similar, except for some constants being different. See portrange-0-6000.md for an example.
The Pflua performance of these filters is very different: the one that matches is slower than the one that doesn't, even though in most cases the non-matching filter will have to do more work. For example, a non-matching filter probably checks both src and dst ports, whereas a successful one might not need to check the dst.
It hurts to see Pflua's performance be less than the Linux JIT compilers, and even less than libpcap at times. I scratched my head for a long time about this. The Lua code is fine, and actually looks much like the BPF code. I had taken a look at the generated assembly code for previous traces and it looked fine -- some things that were not as good as they should be (e.g. a fair bit of conversions between integers and doubles, where these traces have no doubles), but things were OK. What changed?
Well. I captured the traces for portrange 0-6000 to a file, and dove in. Trace 66 contains the inner loop. It's interesting to see that there's a lot of dynamic checks in the beginning of the trace, although the loop itself is not bad (scroll down to see the word LOOP:), though with the double conversions I mentioned before.
It seems that trace 66 was captured for a packet whose src port was within range. Later, we end up compiling a second trace if the src port check fails: trace 67. The trace starts off with an absurd amount of loads and dynamic checks -- to a similar degree as trace 66, even though trace 66 dominates trace 67. It seems that there is a big penalty for transferring from one trace to another, even though they are both compiled.
Finally, once trace 67 is done -- and recall that all it has to do is check the destination port, and then update the counters from the inner loop) -- it jumps back to the top of trace 66 instead of the top of the loop, repeating all of the dynamic checks in trace 66! I can only think this is a current deficiency of LuaJIT, and not with trace compilation in general, although the amount of state transfer points to a lack of global analysis that you would get in a method JIT. I'm sure that values are being transferred that are actually dead.
This explains the good performance for the match-nothing cases: the first trace that gets compiled residualizes the loop expecting that all tests fail, and so only matching cases or variations incur the trace transfer-and-re-loop cost.
It could be that the Lua code that Pflua residualizes is in some way not idiomatic or not performant; tips in that regard are appreciated.
I was going to pass some possible slogans by our marketing department, but we don't really have one, so I pass them on to you and you can tell me what you think:
"Pflua: A Totally Adequate Pflang Implementation"
"Pflua: Sometimes Amazing Performance!!!!1!!"
"Pflua: Organic Artisanal Network Packet Filtering"
Pflua was written by Igalians Diego Pino, Javier Muñoz, and myself for Snabb Gmbh, fine purveyors of high-performance networking solutions. If you are interested in getting Pflua in a Snabb context, we'd be happy to talk; drop a note to the snabb-devel forum. For Pflua in other contexts, file an issue or drop me a mail at email@example.com. Happy hackings with Pflua, the totally adequate pflang implementation!
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.