klickverbot.at by David Nadlinger

getaddrinfo() edge case behavior on Windows, Linux and OS X

2012-01-31T00:00:00+01:00

The POSIX getaddrinfo function returns a list of IP addresses and port numbers for a given hostname and service (resp. port), superseding gethostbyname and getservbyname. Besides some flags, it accepts two string parameters. Either one of them is allowed to be null, representing localhost (or rather 0.0.0.0, depending on AI_PASSIVE) respectively an automatically assigned port. Both parameters being null at the same time, however, is forbidden, and leads to a EAI_NONAME error (WSAHOST_NOT_FOUND on Windows). What happens if the strings are empty ("\0") instead of null, however, is not covered by the spec and not really documented anywhere.

It turns out that there are quite a few differences there between the various operating systems, which is obviously likely to cause issues for Wine (an implementation of the Windows API on Posix/X systems). To get a clear understanding of how the different cases are handled, I put together a little D program which tests a few combinations of host name, port, and flag parameters (see end of post). The snippet could be written in C just the same, as getAddressInfo directly maps to getaddrinfo, I just chose D to avoid platform dependencies and writing more boiler plate code.

The results are summarized in the following table, where »loopback« means that the IP addresses returned were 127.0.0.1 and ::1, »catchall« refers to 0.0.0.0 and ::, »public« means that the actual IP addresses of all available network interfaces were returned, and NONAME refers to a lookup error. »hostname« means that the actual fully qualified name of the host that ran the test was used (care: the host part of the FQDN only does usually not resolve on OS X).

Host	Port	Flags	Windows	Linux	OS X
`null`	`null`	-	`NONAME`	`NONAME`	`NONAME`
		`AI_PASSIVE`	`NONAME`	`NONAME`	`NONAME`
	`""`	-	loopback	loopback	`NONAME`
		`AI_PASSIVE`	catchall	catchall	`NONAME`
	`"0"`	-	loopback	loopback	loopback
		`AI_PASSIVE`	catchall	catchall	catchall
	`"80"`	-	loopback	loopback	loopback
		`AI_PASSIVE`	catchall	catchall	catchall
`""`	`null`	-	public	`NONAME`	`NONAME`
		`AI_PASSIVE`	public	`NONAME`	`NONAME`
	`""`	-	public	`NONAME`	`NONAME`
		`AI_PASSIVE`	public	`NONAME`	`NONAME`
	`"0"`	-	public	`NONAME`	loopback
		`AI_PASSIVE`	public	`NONAME`	catchall
	`"80"`	-	public	`NONAME`	loopback
		`AI_PASSIVE`	public	`NONAME`	catchall
`"localhost"`	`null`	-	loopback	loopback (v4)	loopback
		`AI_PASSIVE`	loopback	loopback (v4)	loopback
	`""`	-	loopback	loopback (v4)	loopback
		`AI_PASSIVE`	loopback	loopback (v4)	loopback
	`"0"`	-	loopback	loopback (v4)	loopback
		`AI_PASSIVE`	loopback	loopback (v4)	loopback
	`"80"`	-	loopback	loopback (v4)	loopback
		`AI_PASSIVE`	loopback	loopback (v4)	loopback
hostname	`null`	-	public	loopback (v4)	public
		`AI_PASSIVE`	public	loopback (v4)	public
	`""`	-	public	loopback (v4)	public
		`AI_PASSIVE`	public	loopback (v4)	public
	`"0"`	-	public	loopback (v4)	public
		`AI_PASSIVE`	public	loopback (v4)	public
	`"80"`	-	public	loopback (v4)	public
		`AI_PASSIVE`	public	loopback (v4)	public

getaddrinfo() behavior on Windows Server 2008 R2, Arch Linux (Kernel 3.1.4, glibc 2.14.1), and OS X 10.7.2 (Lion).

What caused me to investigate the issue in the first place is the behavior when given an empty, non-null host string: Windows returns the public addresses of the present interfaces, while OS X resolves them to ::1/::, but only if a port is given, and Linux doesn’t resolve them at all! Windows generally accepts the most combinations, returning an error only for the explicitly disallowed combination, which is relied on by some applications (e.g. the game League of Legends).

There were also some less significant differences in behavior which are mostly not listed in the table. First, in both of the Linux VMs I tried (an up-to-date Arch box and Ubuntu Oneric), only the IPv4 address of the loopback interface was returned. Second, as in the test no address family, socket type or protocol hints were passed to getaddrinfo(), each address was returned twice on OS X, once with SOCK_STREAM/IPPROTO_TCP and once with SOCK_DGRAM/IPPROTO_UDP set. Linux returned three copies of each address, for STREAM, DGRAM and RAW, with the according protocol types set, whereas Windows only returned a single copy with protocol type IPPROTO_IP and socket type set to 0.

In any case, as a result I have prepared a patch for Wine to emulate at least the succeeding/failing behavior of the Winsock incarnation of getaddrinfo on Linux and OS X, which should solve the bigger part of the related problems. There ideally shouldn’t be any Windows software relying on details beyond that (such as the actual number/layout of addresses returned), but who knows…

import std.algorithm, std.conv, std.range, std.socket, std.stdio;
void main() {
  foreach(host; [null, "", "localhost", Socket.hostName()])
  foreach(port; [null, "", "0", "80"])
  foreach(flags; [cast(AddressInfoFlags)0, AddressInfoFlags.PASSIVE])
  {
    write(
      host ? "'" ~ host ~ "'" : "null", ":",
      port ? "'" ~ port ~ "'" : "null", " (", flags, "): "
    );
    try {
      writeln(
        join(
          map!q{text(a.address, " (", a.protocol, ")")}(
            sort!q{a.family < b.family}(
              getAddressInfo(host, port, flags)
            )
          ),
          ", "
        )
      );
    } catch (Exception e) {
      writefln("[%s]", e.msg);
    }
  }
}

D program used for gathering the data (longer than necessary to get semi-nice output).

D/Thrift GSoC: Performance and other random things

2011-08-01T00:00:00+02:00

This week, I will try to keep the post short, while still informative – I spent way too much time being unproductive due to hard to track down bugs already to be in the mood for writing up extensive ramblings. So, on to the meat of the recent changes (besides the usual little cleanup commits here and there):

Async client design: Yes, even though it took me quite some time to come up with the original one, I had completely missed the fact that it would be unreasonably difficult to extend the support code with resource types other than sockets – long story short, TAsyncSocketManager now inherits from TAsyncManager, instead of being a part of it. Also, I split TFuture into two parts, a TFuture interface for accessing the result, and a TPromise implementation for actually setting/storing it, and only the TFuture part is returned from the async client methods. The thrift.async docs are actually useful now.
Async socket timeouts: Correctly handling the state of the connection after a read/write timeout turned out to be a surprisingly tough problem to solve (allowing other request to be executed on the same connection after a timeout could lead to strange results). In the end, I settled for just closing the connection, which is a simple yet effective solution. To correctly implement this, I also had to finally kill the TTransport.isOpen related contracts and replace them with exceptions in the right places, leading to modified/clarified isOpen semantics.
The non-blocking server now handles one-way calls correctly, and modifying the task pool after it is running no longer leads to undefined results. In the process, I have also turned the static event struct allocations into dynamic ones, since this should have no measurable performance impact, but removes the dependence on the (unstable, per the libevent docs) struct layout.
D now also has a TPipedTransport, which forwards a copy of all data read/written to another transport, useful e.g. for logging requests/responses to disk.
The biggest chunk of time was actually spent on performance investigations: While I was pretty certain that the D serialization code should not perform any worse than its C++ counterpart already, the difference in speed merely being compiler-dependent, I wanted to prove this fact so that I could cross this item from the list. This involved updating LDC to the 2.054 frontend (only to discover that Alexey Prokhin decided to start work on it at the same time I did, the related commits in the main repository are his now), fixing some LDC-specific druntime bugs, etc¹. Unfortunately, I couldn’t test GDC because of issue 6411, but without further ado, here are the results:

	Writing / kHz	Reading / kHz
DMD v2.054, -O -release -inline	2 051	1 030
GCC 4.6.1 (C++), -O2, templates	5 667	1 050
LDC, -O3 -release	2 300	1 077
LDC, -output-ll / opt -O3	5 500	3 150
LDC, -output-ll / opt -std-compile-opts	6 700	1 950

At this point, I will disregard my earlier resolution and again get into the nitty-gritty details – the rest of this post can easily be summarized as the D version is indeed up to par with C++, when it is equally well optimized, but if you are curious about the details, read on.

If you read the performance figures from my last post, the first thing you will probably notice is that the C++ reading performance figure is about four times lower now. This isn’t a mistake; noting the comparatively slim advantage of the C++ version, I made a change to it quite some time ago, which avoids allocating a new TMemoryBuffer instance on every loop iteration (the D version also reuses it). Without really considering the implications, though, I also moved the construction of the OneOfEach struct out of the loop. This seemed like a minor detail to me, but in fact, it enabled reuse of the std::string-internal buffers for the string members of the struct, which is unrealistic (e.g. for a pretty similar situation in the non-blocking server, there is no buffer reuse possible as well).

In a situation where a big part of the time is spent actually allocating and copying around memory, this makes a big difference. To test this assumption about the big influence of memory allocations, I compiled a version of the D benchmark where a static buffer for the strings was used instead of reallocating them every time, and indeed, the reading performance was more than twice as high.

The std::string implementation of the GCC STL seems to be fairly inefficient in this case, because the best D result (which uses GC-allocated memory), is almost three times faster than it for the reading part. It is possible that there are some further optimizations which could improve performance (-O3 didn’t change things for the better, in case you are wondering), but as my goal wasn’t to squeeze every last bit of performance out of this synthetic benchmark, I didn’t investigate this issue any further.

But now to the D results: Simply switching to LDC 2 instead of DMD didn’t give any great speedups, because readAll() wasn’t inlined by it either, thus leaving all the memory copying unoptimized, as discussed in the last post. To see how much of a difference this would really make, I compiled the D code to LLVM IR files and manually ran the optimizer/code generator/linker on them, with the plan being to manually add the alwaysinline attribute to the relevant pieces of IR:

ldc2 -c -output-ll -oq -w -release -I../src -Igen-d ….d
llvm-link *.ll -o benchmark.bc
opt {-O3, -std-compile-opts} benchmark.bc -o benchmark_opt.bc
llvm-ld -native -llphobos2 -ldl -lm -lrt benchmark_opt.bc

I then discovered that the method calls in question were properly inlined by the stand-alone opt without any manual intervention anyway. I am not really sure why this happens; the inliner cost limits could be more liberal in this case, or the optimization passes being scheduled in a different way than inside LDC could have an impact, or maybe it’s connected to the fact that TMemoryBuffer and the caller are in different modules (to my understanding, LTO shouldn’t be required to optimize in this case, but it may well be that I am mistaken here).

The LDC -output-ll rows in the above table correspond to the benchmark compiled this way, with the -std-compile-opts and -O3 flags passed to opt, respectively. This is a nice example of how important compiler optimizations for this, again, synthetic benchmark really are: for the reading part of the benchmark, -O3 gives a nice speed boost because of the more aggressive inlining (-std-compile-opts doesn’t touch TBinaryProtocol.readFieldBegin(), which is called 15 times per loop iteration and contains some code that can completely be optimized out), but for the writing part, its result is actually slower, presumably because of locality effects (the call graphs are identical).

The only change related to benchmark performance I made since the last post was an LDC-specific workaround to stop manifest constants from incorrectly being leaked from the CTFE codegen process into the writing functions. I think the above results are justification enough to stop worrying about raw serialization performance – the results when using the Compact instead of the Binary protocol are similar – and moving on to more important topics².

¹ If you are curious about LDC 2, you can get the source I used from the official hg repo, and the LDC-specific druntime and Phobos source from my clones at GitHub.

² Such as performance-testing the actual server implementations, but I don't expect any big surprises there, and I am not sure how to reliably benchmark the network-related code – running server and clients on the same machine is probably a bad idea?

D/Thrift GSoC: Non-Blocking Server, Async Client, and more

2011-07-15T00:00:00+02:00

First of all, the usual apologies for publishing this post later than I originally planned to. No, seriously, drafting a solid asynchronous client implementation ended up being a lot more work than I originally anticipated, but I wanted to discuss my ideas in this status report. Now, the post turned out way too large anyway, but I guess that’s what I deserve. ;)

Also, a quick notice beforehand: A week ago, DMD 2.054 was released. It is the first version to include, amongst a wealth of other improvements, Don’s necessary CTFE fixes and my std.socket additions. This means that it is no longer necessary to use a Git build to use Thrift with D, you can just go to digitalmars.com and fetch the latest package for your OS.

Small but useful additions

But before discussing the intricacies of non-blocking I/O, to the mundane helper transports that found their way into the D library: The first addition was a simple TInputRangeTransport which, as the name says, just reads data from a generic ubyte input range, with some optimizations for the case where the source is a plain ubyte[] (std.algorithm.put is currently unnecessarily slow if both ranges are sliceable, I didn’t have time to prepare a fix for Phobos yet). It can e.g. be used in cases where want to deserialize some data from a memory buffer, and don’t need to write anything back (which is where TMemoryBuffer would be used).

Another addition is TZlibTransport, which wraps another transport to compress (deflate) data before writing it to the underlying transport, and decompress (inflate) it after reading. This is implemented by directly using zlib (via the C interface) instead of using std.zlib, because the API of the latter would have made it impossible to avoid needlessly allocating buffers all the time. Thankfully, the C++ library already included a zlib-based implementation, saving me from working out the various corner cases.

Some deserialization micro-optimizations

The next thing I worked on were some further optimizations motivated the serialization_benchmark. To recapitulate, it is a trivially simply application which just serializes a struct (OneOfEach from DebugProtoTest.thrift to be precise) to a TMemoryBuffer and then reads the data back into the struct again, repeating both parts a number of times to be able to get meaningful timing results. Here are my related changes:

First, I replaced TMemoryBuffer with the new TInputRangeTransport to avoid copying the data on each iteration of the reading loop. Because the initial copying to the memory buffer took only ~1–2% of the overall time anyway, this didn’t have a great speed impact.
The next change was to provide a shortcut version of TTransport.readAll() for TInputRangeTransport (and TMemoryBuffer as well). Previously, the generic TBaseTransport version which just calls read() in a loop was used – because the method is called about 50 times per reading loop iteration, replacing it with a simple slice assignment gave a ~20% speedup on the reading part of the serialization benchmark.
Furthermore, I nuked the protocol-level »read length« limit implemented for the Binary and Compact protocols. This was not much from an optimization perspective as simply due to the fact that limiting the total amount of data read really belongs at the transport level in my eyes (it was only present because of a, uhm, misguided attempt to draw inspiration from the Java library). Incidentally, this gave another ~15% speedup in the reading benchmark. I will add support for limiting the container and string size Really Soon™ (just as for C++, to be able to somehow cap the amount of memory allocated due to network input), but one more branch per container/string read should have a negligible performance impact.
Finally, I removed a few instances where memory was unnecessarily zero-initialized (only to be completely overwritten later) in the reading code. For the integer buffers (used for byte order conversion) this gave a small but measurable (<5%) performance boost, and for the binary/string reading (which is both larger in size and exercised more often during the benchmark) another ~8% speedup.

Profiling results

So, after all these (de)serialization micro-optimizations (I improved the writing part when first working on performance), how does the D implementation compare to its natural competitor, the C++ one? Well, frankly not too well at this point. Before discussing my findings in more detail, the performance results as measured on an x86_64 Arch Linux VM¹, hosted on my MacBook Pro (Intel Core i7-620M 2.66 GHz, OS X 10.6), by running each part 10 000 000 times and averaging over it (the results are in 1 000 operations per second, so both implementations can perform on the order of a million reads/writes per second):

	Writing / kHz	Reading / kHz
DMD v2.054, -O -release -inline	2 051	1 170
GCC 4.6.1, -O2	4 624	2 053
GCC 4.6.1, -O2, templates	5 667	4 509

The first GCC row shows the result of the vanilla build (what you get by simply doing cd lib/cpp/test; make Benchmark; ./Benchmark), while for the »templates« row, I added the (undocumented?) templates flag to the generator invocation (thrift -gen cpp:templates), which causes the struct reading/writing methods to be templated on the actual protocol type, much like what I implemented for D. In this benchmark, eliminating any indirections naturally has a huge impact on the performance.

So, why has the D version less than half the throughput for writing, and is almost four times slower on reading? Let me first point out that the actual code for the C++ and D implementations is, from a semantic point of view, virtually the same (with the exception of D using garbage collected memory for string/binary data). I think I have arrived at a point where the single largest factor influencing the performance of the serialization code is the compiler used, or to be more exact, how well it optimizes the code.

What follows are a few result from my profiling sessions (Valgrind 3.6.1, visualized using KCachegrind²) which corroborate with my assumption that compiler optimizations are the culprit here. Let’s first have a look at the profiler results for the reading part of the benchmark (this time, the loops were run only a million times each):

C++ reading time profile.

D reading time profile.

I only included the top six functions (by time spent in them) here for the sake of brevity, but for both implementations, the »long tail« of calls in the flat profile are actually runtime helper functions, mostly startup initialization code and memory management-related things used for reading the string functions (for D, GC calls show up prominently, because the benchmark allocates three million strings, which triggers almost 50 collections in between).

This also means that the compiler has done a pretty good job at combining all tiny deserialization functions into the top-level struct reading function by inlining – with one glaring difference: DMD chose not to inline TInputRangeTransport.readAll(), which is ultimately called when deserializing each and every member to read the actual bytes off the wire (or in this case, from memory), thus yielding to 49 million additional function calls. To make matters worse, this also means that the number of bytes requested each time (e.g. 4 for an integer) is not known at compile time, which also means that the generic memcpy implementation has to be called each time. On the other hand, the C++ implementation only calls memcpy in those situations where the number of bytes copied really depends on a runtime value, which is the case for strings which are intrinsically variable-length (the other memcpy calls are called during initialization and initially writing the struct to the buffer).

Profiling the writing part shows similar results:

C++ writing time profile.

D writing time profile.

Again, for the C++ version, everything is inlined into OneOfEach.write(), in which over 80% of the time are actually spent, and just as for the reading part, the only instance where memcpy() is not inlined³ is for strings. On the other hand, the D version is optimized almost as well as the C++ version, with the only exception of TMemoryBuffer.write() not being inlined, which again prevents memcpy from being optimized (the other function showing up, reset(), only resets output buffer once per iteration, this is inlined into main in the C++ version).

So, to recapitulate, I am not sure whether DMD would be able to replace a memcpy() call with optimized asm in the first place, but not knowing the length at compile-time prevents that anyway. I am pretty sure that this difference of about a hundred million function calls and not being able to write optimized text for the short (2, 4, 8, …) byte copies accounts for a large part of the performance gap.

This assumption is supported by data gathered from a case where GCC chose to not inline TBufferBase::write() (which is the common path of TMemoryBuffer::write()). Interestingly, this actually happens at -O3, which is a higher optimization level than -O2 used above (I suppose because of some additional optimizations performed on it, which causes its inlining costs to rise high enough not to be inlined). Just for comparison, here are again the five top functions from the profile:

C++ writing time profile when compiled with -O3, causing TBufferBase::write() to be no longer inlined.

Just as for D, because of this memcpy cannot be optimized away either. And unsurprisingly, this causes the performance to go through the floor as well, the executable only reaches 2 519 thousand operations per second now. The D version is still a bit slower with only 2 051 kHz, but it is on a comparable level now.

So, to finally come to a conclusion, most of the performance gap between C++ and D presumedly comes from DMD not inlining a key function and thus not being able to optimize away memcpy calls as well. An obvious experiment would be to try a different compiler like GDC or LDC, both of which are known to generally optimize better than DMD does. Unfortunately, both of them are currently at front-end version 2.052, but my Thrift code currently requires 2.054.

There are two possible solutions to this, either sprinkle workarounds all over the Thrift code to be able to use the older DMD frontend and Phobos versions for the benchmark, or update the frontend of GDC or LDC to 2.054. While the former would be entirely feasible, I think I update the LDC frontend once I have some time to spare, as this will also be useful for other D projects (choosing LDC because I am already familiar with its codebase).

Libevent-based non-blocking server

If I didn’t lose you during all the talk about micro-optimization above, let me hereby present you the two main additions to the library during the last two weeks: a non-blocking server implementation and a Future-based asynchronous client interface.

I am not sure if I ever stated it explicitly (the timeline only has »event-based I/O Phobos lib?« in parentheses), but I was hoping to be able to come up with a small general-purpose non-blocking I/O library written in D as a by-product of this project. The obvious time to start working on it would have been when implementing the non-blocking server, But after considering several possible designs, I realized that I did not yet know the problem domain well enough to come up with something that is not just a cheap libevent/Boost.Asio rehash, but where I’m still sure that it performs well enough for a production-grade Thrift server implementation.

Thus, I went with simply porting the C++ libevent-based server implementation over to D, which has the benefits of being battle-proved, so that I have something which I can advise people to use in production code without feeling guilty. There are a few instances where I needed to manually add a GC root for some memory passed to libevent, but other than that, the code is reasonably clean, even though it surely could be prettier if a native D »event loop« was used.

A word of warning for Windows users: While libevent is linked dynamically as well, thus making it easy to just use DLL builds on Windows, there are some pieces of the socket code not yet tested for WinSock. Currently, I am not even sure if all of the code compiles on Windows, but I will perform some test on Windows shortly to ensure all the new additions work there as well.

Coroutine-based `TAsyncClient`

Using an asynchronous/concurrent approach for network-related code with its intrinsic I/O latency seems like a very obvious thing to do, but to my knowledge e.g. the C++ libraries currently do not provide a generic async client implementation, which is the part of the reason I did not tackle this earlier.

After getting accustomed with the general idea of non-blocking I/O, it seemed to be a good time to finally work on the topic. What I basically wanted to implement was a way to off-load client-side request/response handling, possibly for multiple connections, to a worker thread, providing a future-based interface to the client code. For multiplexing handling multiple connections per worker thread, I wanted to experiment with a coroutine-based design.

As mentioned in the beginning of this report, coming up with a solid design took me a bit longer than expected, but as of now, thrift.codegen includes a fully functional TAsyncClient implementing such a scheme, also using libevent to have a portable means for handling non-blocking I/O. The new thrift.async package contains the related helper code, such as TAsyncSocket representing a non-blocking socket.

The new code is not yet well-documented or tested, and is still missing some important features like the ability to set timeouts on operations, but I have successfully tested basic use cases.

Plans for the second GSoC half

Which finally brings me to the end of this post: As my project fortunately passed the mid-term evaluations, it is now time to discuss how to go forward during the second part of the Summer of Code.

During the next week, I will work on some of the obviously unfinished things like async client documentation and tests, and will add a few missing utilities such as a tee-like transport which can be used to transparently log requests.

Speaking of documentation, this currently is a big issue for both the D implementation and, to a lesser extent, Thrift in general. However, as of now, I have worked sufficiently long on the code that I am effectively blind for what kinds of documentation a typical user would benefit the most from – more detailed API docs? Simple stand-alone examples with well-commented code? Tutorials? It would be great if you could let me know what you think would be useful.

With the non-blocking server implementation being completed, only the »performance« and »documentation« items from my original timeline remain, besides some general clean-up work being left to do. However, Nitay, my mentor, suggested a few other things which could be worth looking into, such as a generalized client for querying multiple servers, to be used for things like redundancy, load distribution, data verification, etc. I will discuss this in more detail and then update the timeline accordingly.

¹ Why test in a Linux VM (Virtual Box 4.0.8) rather than directly on my OS X development box? Because Linux x86_64 is probably where most of the server-side deployments will end up, only an ancient GCC is available on OS X, DMD is still 32 bit-only there, and Valgrind/Callgrind which I used for profiling is not really usable on OS X 10.6. I am aware that using a VM might skew the results a bit, but I think the impact shouldn't be too large. Incidentally, the tests compiled and ran in the Linux VM generally performed better faster than on the host.

² I patched KCachegrind to elide the middle of the symbol name for better readability in width-limited screenshots, and used my own little demangling tool for the D results.

³ Technically, GCC handles memcpy as compiler built-ins, so inlining might not be precisely the right term, but the effect (avoiding a function call) is the same.

D/Thrift GSoC: Docs, Servers, Tests

2011-07-04T00:00:00+02:00

Dear Reader,

Let me apologize for not being terribly motivated to write a blog post right now, but I was lucky enough to catch the flu last week with temperatures between 25 °C and 35 °C outside, and while the fever has gone by now, I am still depleting my tissue stock at an insanely high rate…

Anyway, back to topic, the usual summary of the recent changes to my Thrift GSoC project:

The most important item on the list from an user point of view are probably the documentation improvements: The project now has a Getting Started page, and I have made a complete pass through all the DDoc docs, a build of which is available here (I still have to whip up a nice design, but it should do for the moment).
More interesting from a coding perspective are the additions of two new thrift.server implementations, TThreadedServer and TTaskPoolServer. The former is a naive implementation of a threaded server which just spawns a new worker thread per client connection, while the latter uses a std.parallelism thread pool to process the queued client connections (the maximum number of active connections is configurable). I also added a D version of the StressTest server for sanity checking.
Another server-related change is the addition of server and processor event handlers, which can be used to hook custom code into various points during the server/request lifecycle, e.g. for collecting diagnostic data. Data can be persisted between the calls by saving them as connection/call »context«, which is a Variant the server code passes around for you (I went with variants over e.g. templating the server code on the context object type simply to avoid adding another layer of complexity for a non-essential feature).
I added a standalone test case exercising the different transport types (socket, file, memory buffer) combined with the various wrapper transports (buffered, framed), modeled after the C++ TransportTest. This has uncovered a number of (sometimes not-so-) subtle defects in the transport implementations which have been fixed (TSocket not handling EINTR, framed/memory buffer borrow() would also return smaller than requested, TFileReaderTransport not tailing files correctly, …).
Build system improvements: The stand-alone test cases are now organized in a much less cumbersome scheme, and DDoc documentation is generated for the library by default. lib/d/README now has instructions on how to generate a self-signed certificate for SSL socket testing.

If you are on OS X, you might want to manually apply Phobos pull request 131 until it is merged into Git master to avoid your servers crashing due to an unhandled SIGPIPE (you can also just set signal(SIGPIPE, …) to SIG_IGN in your startup code).

I have also added a list of not yet scheduled ideas to the project page. Implementing a ZLib compression transport is currently the top item on my list, after which I will start to work on a non-blocking server implementation as planned. An asynchronous version of TClient is something I certainly want to implement, but I am planning to defer work on it until I have tackled the non-blocking server, as I could end up using the same approach (e.g. libevent) for it.

D/Thrift GSoC: Protocols: Compact, JSON, performance

2011-06-22T00:00:00+02:00

Another week of my Google Summer of Code project passed by, and so you are reading another status update. I am not including any core D development-related news this time, first because I didn’t do much DMD/Phobos work last week, and second because it gets tedious to list everything here – feel free to see my GitHub activity stream for more information. But still, thanks to Sean Kelly for quickly fixing the OS X threading/GC race condition I encountered the week before.

One of my targets last week was to do some preliminary performance investigations and using the insights gained to modify the protocol interface accordingly before I implement additional protocols. For this, I used the DebugProtoTest.thrift-based serialization performance test already implemented for C++ and Java (see the D version at GitHub, a more intensive look at performance, including creation of some more extensive benchmarks is planned for later).

Ironically, the change with the biggest impact on the writing performance didn’t have anything to do with the protocol interface at all: When first writing TMemoryBuffer, I simply implemented write() as D array appending operation, because I didn’t want to spend much time on optimizing it yet, and I figured that as long as there would not be too many reallocations, it should be reasonably fast for testing purposes. Array appending translates to a non-inlined and not really cheap D runtime call, however, and TMemoryBuffer.write() unsurprisingly happens to be the single most called function in the whole writing part of the benchmark. After changing TMemoryBuffer to manual malloc/free-style memory management, the writing part finished in less than 30% of the time.

I tried to switch to GC.malloc instead of manual freeing afterwards because it would make getting a buffer content slice safe and the small memory allocation overhead should not really be a problem for typical TMemoryBuffer use cases (it does not matter at all in this benchmark because the required amount of memory is pre-allocated), but I encountered some strange data corruption issues in the other larger test cases I have yet to track down. Most probably, I just missed some subtleties when treating GC.realloc as a drop-in realloc/free replacement, but I just didn’t find a way to pin-point the issue.

For the next step, I tackled the design of the TProtocol interface: When building the first prototype for the library, I had the ad-hoc idea of passing in delegates to the aggregate reading/writing functions for processing their members. I figured that this would make the interface nicer as all the *Begin()/*End() pairs could be collapsed into a single call, the struct member reading loop could be moved into the protocol itself instead of being duplicated over and over again (although this is not a real benefit besides a slight code size reduction because it is generated code anyway), and implementing protocols like JSON would be easier since the structural information would not have been completely lost compared to a »flat« interface.

I was, however, aware of the fact that this could pose a performance problem, and indeed some experimenting showed that DMD generated suboptimal code for delegate literals and was not really able to them, even for scoped delegates. From a compiler point of view, this is not really surprising as generating better code would require a fair bit of analysis to be done, but still I decided to switch to a more simplistic protocol design for the time being – even more so, as I realized that my design idea would not really simplify implementing JSON-like protocols anyway. I chose to go with the C++/Java interface verbatim, as it is proven to work (and having a similar interface across multiple languages has its own merits as well), and with the changes in, I measured a 20% speedup, even though no inlining was possible due to virtual calls all over the place yet. (In hindsight, it might have been better to implement the template mechanism first, so that the actual impact of the protocol API change would have been more visible. Maybe I’ll revert the binary protocol back to the old interface and re-run the test to get precise numbers at some point in the future.)

Finally, I implemented a way to specify the concrete transport/protocol types used in the application at compile-time using templates (similar to C++ and the templates Thrift compiler argument), thus eliminating most virtual calls and enable the compiler to inline calls all over the library. I expected to see a dramatic speedup here as well – when not specifying the protocol/transport type, the writing loop in the C++ benchmark is only half as fast –, but instead I saw »only« a 40% speedup overall, with the C++ version still being significantly faster.

When comparing profiling data for the optimized C++ and D versions, I noticed that in the D version _memcpy gets called ten times as often as in the C++ version – GCC, being able to inline the write() calls, is able to replace the calls with optimized routines for shorter lengths, and since both versions spend most of their time actually copying data around at this point, this yields a huge advantage.

After that, I did not make any further attempts at optimizing the D version, since performance was not my primary goal at this stage anyway – the basic design seems to be solid, and what left are micro-optimizations. When focussing on performance later in the term, I will certainly create more benchmarks, and also try to optimize the languages I will compare D to (C++ and Java, most likely) – for example, the current C++ serialization benchmark from the official HEAD does a lot of unneeded work in the reading loop, moving out the initialization code makes it run twice as fast. I will also have a look at using GDC and LDC instead of DMD for their more sophisticated backends, and document the exact performance findings on various platforms.

Even though I am not going to write that much about it, I spent the bigger part of my time on non-performance related work: Generated structs now have an appropriate toString() and opEquals() implementation, the D ThriftTest client actually checks the data it sends/receives instead of just flooding the console with messages (no idea why this hasn’t already been implemented for C++ and Java), and last but not least, I implemented the Compact and JSON protocols for D. This completes the protocol section, as I do not plan to implement the Dense protocol unless there is much time left to spend during the end of the term (as previously discussed).

During the next (or rather: this) week, I am going to work on documentation, integrate a number of test cases I have already lying around with the repository/build system, and implement a simple multithreaded server.

D/Thrift GSoC: Growing the library

2011-06-14T00:00:00+02:00

First, let me apologize for not posting an update last week – I had a busy time, but regardless I will try to let you know about the state of affairs regularly in the future. Now, what were I working on? I updated my project page based on the timeline previously discussed at my project mailing list, and – besides me being a day late, more below – it is still valid. These were the main points I worked on:

Build system integration: The D library is now integrated with the Thrift Autoconf/Automake build system. If a working D2 compiler is detected, the libthriftd static library containing all the modules is now built on issuing make along with the rest of Thrift. make check runs the unit tests for each D module and builds the standalone test executables (i.e. ThriftTest for now).
Socket transport enhancements: Implemented interrupt() for the server socket which can be used to notify a server waiting on a blocking socket for connections about shutdown, added socket timeouts, properly handle exceptions thrown by std.socket, …
Added a D implementation of TMemoryBuffer, which is widely internally used and a nice tool for writing unit tests as well. Implemented the Framed transport in D.
Implemented TFileReaderTransport and TFileWriterTransport, the D equivalent to the C++ TFileTransport. I separated the two components because I could not really think of a situation in which you would use both at once, and conflating the two would complicate the state space (I am not even sure if the C++ implementation does what it is supposed to if read/write calls are interleaved) and make the implementation unnecessarily complex. The TFileWriterTransport implementation performs the actual file I/O in a separate worker thread, which communicates with the main thread using a message passing approach (leveraging D’s std.concurrency module).
A simple HTTP client/server transport, closely modeled after the C++ implementation.
An SSL client/server socket implementation using the OpenSSL library, which is linked in dynamically (primarily for easy Windows compatibility). The actual implementation is pretty much a direct port of the C++ TSSLSocket, but I had to quickly write the D2 bindings for OpenSSL first. For now, the bindings live in thrift.util.openssl, as I only included the subset of functions I needed for Thrift, but I might move them out in the future.

As always, you can find the changes on my GitHub fork. I also spent a sizable chunk of my time on contributing some improvements and fixed to the D compiler and standard library projects. As for the issues I mentioned two weeks ago, kudos to Don Clugston for promptly fixing CTFE issues 6077 and 6078, and my DMD pull request 77 and Phobos pull request 65 were also merged in the meantime.

During the last two weeks, I worked on Phobos pull requests 73 (adds std.socket.socketpair), 87 (better std.file error messages), 90 (fixes a mailbox handling bug in std.concurrency – took me quite some time to track down as it caused sporadic deadlocks in my unit tests), 99 (adds timeout handling and hostname lookup to std.socket – I still don’t know why WinSock adds 500 ms to the recv() timeout), druntime pull request 28 (adds a Posix netdb.h module) and DMD pull request 118 (finally removes the _DH flag).

Furthermore, I collaborated with Daniel Murphy on fixing the long-standing issue that function pointers are not properly typechecked, resulting in DMD pull request 96 and druntime pull request 26. I have also started to work the dreaded DMD bug 314. While the basic fix is in place – that’s how I found the bug in Phobos pull request 102 – (I adapted the D1/LDC changes by Christian Kamm to D2/DMD), I still need to add some more tests and solve a few more complex cases. Unfortunately, I also hit two new issues which I have not been able to fix yet: 6108, a DMD contract inheritance bug, and 6135, a druntime OSX threading/GC crash.

D/Thrift GSoC: First results

2011-05-29T00:00:00+02:00

The first week of my D/Thrift project as part of the D programming language Google Summer of Code 2011 is over, and I am happy to be able to share some first results. If you are not sure what I am talking about yet: Apache Thrift, originally developed for internal use at Facebook, is both a data serialization/RPC protocol and its reference implementation. In short, it works by defining data types and services interface in a language-agnostic interface definition file. Then, a compiler is used for generating code from that .thrift file (currently written in C++), using target language support libraries which contain the actual serialization protocol/RPC implementation. Currently, Thrift supports a large number of languages including C++, Java, PHP and Python.

It was clear from the beginning that I would stick with this approach for my implementation, not only because the informal project goal is to establish D as an equal target language besides the existing ones, but simply because one of the main strengths of Thrift is that you can use the same interface definition for all target languages, with the compiler doing all the heavy lifting for you. I did, however, want to leverage the powerful metaprogramming capabilities of D (compile-time reflection, CTFE, string mixins) to lift as much work off the »ahead-of-time« C++ code generator as possible, having the option to use the Thrift libraries beyond the traditional scope of the project for ad-hoc extension of existing D data types and interfaces with serialization/RPC functionality at the back of my mind.

My primary goal during the first week was to evaluate the feasibility of this approach by quickly implementing the basic parts of each Thrift component. In more detail, the sub-goals I tackled during the last week were:

Create a preliminary implementation the central parts of the support library (TBinaryProtocol, TBufferedTransport, TSocket, …) using the C++ and Java implementations as reference to be able to directly test the progressing D implementation against other languages.
Implement the general and client-specific parts of compile-time code generation (struct reading/writing, method arguments/result struct generation TClient, …), using a hand-crafted Thrift tutorial interface to test it against the Java server.
Implement TSimpleServer and related basic server functionality (e.g. TServerSocket) to be able to test server code generation.
Complete missing server-side code generation bits (TProcessor, server-side arguments/result structs, …), again using a hand-crafted interface to test it against the Java Thrift calculator tutorial client.
Add D code generation to the Thrift compiler, and run the compiler against all the test interface files coming with Thrift (test/*.thrift) to catch any obvious issues.
Implement a ThriftTest server and client in D to exercise the more advanced serialization code paths and fix any bugs, testing it against the C++ implementation.

So far, no major problems popped up, and I was able to complete the above list as planned. I did, however, hit a few bugs in DMD, which is, on the other hand, doesn’t come as a total surprise because I am heavily using the metaprogramming facilities. I have been able to find workarounds for all of the issues, but it nevertheless took me quite some time to track them down initially: issues 6069, 6077, 6078, DMD pull request 77 and – this one is merely an enhancement – Phobos pull request 65.

If you want to have a look at the code, feel free to head to my GitHub Thrift fork, where I regularly push my work to. And just to give you a short glimpse of the very basic features (a lot more is already implemented), this is how you could implement a simple calculator server/client which adds two numbers using the Thrift library, without using any generated code.

// This could also be generated from a .thrift file and contain
// structs, exceptions, etc.
module calculator;

interface Calculator {
  int add(int lhs, int rhs);
}

module server;

import calculator;
import thrift.codegen;
import thrift.protocol.binary;
import thrift.protocol.processor;
import thrift.server.simple;
import thrift.transport.buffered;
import thrift.transport.serversocket;

class CalculatorHandler : Calculator {
  override int add(int lhs, int rhs) {
    return n1 + n2;
  }
}

void main() {
  // Expose a CalculatorHandler instance at port 9090.
  auto protocolFactory = new TBinaryProtocolFactory();
  auto processor = new TServiceProcessor!Calculator(
    new CalculatorHandler());
  auto serverTransport = new TServerSocket(9090);
  auto transportFactory = new TBufferedTransportFactory();

  auto server = new TSimpleServer(
    processor, serverTransport, transportFactory, protocolFactory);
  server.serve();
}

module client;

import calculator;
import std.stdio;
import thrift.codegen;
import thrift.protocol.binary;
import thrift.transport.buffered;
import thrift.transport.socket;

void main() {
  // Set up a client for the Calculator interface and try to
  // connect to localhost:9090.
  auto socket = new TSocket("localhost", 9090);
  auto transport = new TBufferedTransport(socket);
  auto protocol = new TBinaryProtocol(transport);
  auto client = new TClient!Calculator(protocol);
  transport.open();

  // Call the server's add() method and print the result.
  auto lhs = 2;
  auto rhs = 3;
  auto sum = client.add(lhs, rhs);
  writefln("%s + %s = %s", lhs, rhs, sum);
}

Random D development news

2011-04-26T00:00:00+02:00

During the last couple of weeks, I didn’t really find time to update this blog. Nevertheless, however, I was able to spare some time for work on a couple open source projects related to the D programming language. But first, let me quickly summarize some great changes that will be in the next DMD release:

Don Clugston has basically re-implemented CTFE to fix a whole slew of compile-time function execution bugs, among which is the dreaded bug 1330. There are still some regressions compared to DMD 2.052 (like this one, which breaks QtD), but apart from those, it’s a big step towards getting CTFE out of the »experimental feature« category. The new architecture will also make implementing reference types easier, but this is still a long way off. Then next DMD/Phobos release will also include the new std.parallelism module by David Simcha, some GC optimizations and a large amount of other improvements (among which is the addition of the parent trait, so that QtD doesn’t need a patched DMD any longer) – due to the GitHub migration and the larger part of x86_64 support being done, the perceived development speed in the core community really went up a notch.

As for my own (insignificant, compared to the above) contributions, I did some work on LDC during the last few days, porting it to LLVM 2.9 and bringing the front-end in sync with DMD 1.067 – you can find the changes in the default branch over at Bitbucket. The DMD updates also contained some changes to the varargs ABI on x86_64 and other areas of the runtime interface, which I didn’t merge yet, because it would require an update to Tango as well. I am not aware of any regressions so far (see the DStress results), but feel free to ping me in case of any problems.

There were also some updates and bug fixes to D support in SWIG, most notably support for the nspace feature, which allows you to map C++ namespaces to D packages/modules (it doesn’t work for free functions and global variables yet, but this is a general SWIG restriction that could be easily lifted, just ask me if you need it). There was another SWIG release in the meantime, version 2.0.3, but it was only a »quick backup« by the maintainer before he merged some intrusive Python changes. I was caught pretty much off-guard by it and had no time for real testing and thus, it contains some bugs (mainly related to nspace support when split-proxy mode is not activated, thanks to Jonathan Pfau for the reports) – please use SVN trunk instead.

Another little project I recently worked on is std.units, an units of measurement implementation for D. This topic came up several times on the NG previously, and every time it was suggested to add units support with Phobos, so I have merged the work into my Phobos fork. Please note, however, that this is in no way a formal review request yet. There are still a couple of items left on my to-do list, but before I am tackling the remaining issues, I’d greatly appreciate some feedback (see the thread on the D newsgroup, RFC: Units of measurement for D).

Finally, a personal note: Yesterday, I received notice that I was accepted to work on my Apache Thrift project under the umbrella of Digital Mars as part of the Google Summer of Code 2011 – thanks a lot to everybody who supported my proposals for their trust in me! I know that the expectations are high, and will do my very best to live up to them.

SWIG 2.0.2 with D support released

2011-02-21T00:00:00+01:00

Yesterday, SWIG version 2.0.2 has been officially released. Along with various bug fixes for the other supported languages, this is the first release to support the D programming language. As always, you can get the release from the download area, but here are direct links to the files hosted at SourceForge for your convenience: One for the source tarball, and another for swigwin which includes a pre-built Win32 executable.

Since my first announcement, there were a number of changes and improvements. Along them were some critical fixes to the generated code when compiled on Windows, some minor ones regarding name collision in the D part, and a fix to the »directors« feature where a wrong C++ method would be called silently under certain circumstances (thanks to Jimmy Cao for reporting). Unfortunately, there were also some breaking name changes, as previously mentioned on this blog. Furthermore, I added basic support for operator overloading, please refer to the documentation for details.

If you have any questions or need assistance with using SWIG on a certain library, feel free to contact me directly or to post to the swig-user mailing list. During the next few days, I will be quite busy and cannot promise you to reply quickly, but after that, I will be happy to help. Oh, and it would be great if you could share your personal experiences, common pitfalls and how to overcome them when using SWIG for the first time, since »Getting Started«-style documentation for people new to SWIG is a bit scarce at the moment!

Breaking name changes in SWIG/D

2010-12-01T00:00:00+01:00

Sorry if this notice might come a bit late for some of you, but a few days ago, I have committed a breaking change to D support in SWIG trunk. It was needed to bring the names used in the D module in line with the C# one, the naming scheme of which was intended to be language-independent by the principal maintainer (although it is only used in the C# and D parts right now).

Most of the changes revolve around the term »wrap D module« being replaced with »intermediary D module«, including names derived from it. To adapt your interface files, just perform the following replacements:

s/cwtype/ctype/g
s/dwtype/imtype/g
s/dptype/dtype/g

s/$wcall/$imcall/g
s/$dpcall/$dcall/g

s/wrapdmodule/imdmodule/g

Announcing: D support in SWIG

2010-11-21T00:00:00+01:00

In a nutshell, SWIG is a »glue code« generator, allowing you to access C/C++ libraries from various target languages, including C#, Go, Java, Ruby, Python … and, since I merged my work into SWIG trunk a few days ago, also the D programming language, both version 1 and 2.

Why would D support in SWIG be useful in the first place? After all, D is perfectly able to interface with C on its own, so why bother using a third-party tool?

Well, it turns out that even for »plain old C«, there are reasons why you’d want to use a bindings generator. Besides the obvious problem that you have to convert the C header files to D modules somehow, there is one major inconvenience with directly using C libraries from D: D code usually is on a higher abstraction level than C, and many of the features that make D interesting are simply not available when dealing with C libraries. For instance, you would have to manually convert strings between pointers to \0-terminated char arrays and D strings, and most interesting algorithms from the D2 standard library are simply unusable with C arrays.

While these issues can be worked around relatively easy by hand-coding a thin wrapper layer around the C library in question, there is another issue where writing wrapper code per hand is not feasible: C++ class libraries. D1 does not support interfacing with C++ at all, and even if extern(C++) has been added to D2, the support is quite limited, and a custom wrapper layer is still required in many cases.

Here is, without further ado, a small example of what the D module for SWIG allows you to do. Consider the following (admittedly not very useful) piece of C++ code:

typedef std::pair<float, float> Position;

class Shape {
public:
   Shape( Position pos ) : m_position( pos ) {}
   virtual ~Shape() {}

   virtual std::string getDescription() const = 0;

   Position getPosition() const {
      return m_position;
   }

protected:
   Position m_position;
};

class Circle : public Shape {
public:
   Circle( Position pos ) : Shape( pos ) {}
   virtual ~Circle() {}

   virtual std::string getDescription() const {
      return "A perfect circle.";
   }
};

std::string toString( const Shape& shape ) {
   std::ostringstream result;

   Position p = shape.getPosition();
   result << "A shape at (" << p.first << ", " << p.second << ").";
   result << " It looks like this: " << shape.getDescription();

   return result.str();
}

By using SWIG to generate the necessary glue code, you can easily make the classes available in D, as demonstrated by the following small program:

class Square : Shape {
   this( Position pos ) {
      super( pos );
   }

   override string getDescription() const {
      return "Quite square-ish.";
   }
}

void main() {
   // One of the ugliest bugs currently in D: Type inference does not
   // work correctly for arrays of classes with a common supertype.
   auto shapes = [
      cast( Shape ) new Circle( new Position( 1, 3 ) ),
      new Square( new Position( 2, 1 ) )
   ];

   foreach ( shape; shapes ) {
      writeln( toString( shape ) );
   }
}

A shape at (1, 3). It looks like this: A perfect circle.
A shape at (2, 1). It looks like this: Quite square-ish.

Note that Shape is extended on the D side just as usual and how the C++ call to getDescription() is transparently routed to Square.getDescription(). This mechanism dubbed cross language polymorphism is enabled by a feature of SWIG called »directors«, which causes the extra indirection layer needed for this to be emitted. Also note how the strings are seamlessly converted between their C++ and D representation.

So you want to give the D module in SWIG a whirl? Just head over to the SWIG SVN, grab the sources from there, and build it. If you are planning to run the test suite or the included examples, you might want to specify --with-d1-compiler=<…> and --with-d2-compiler=<…> at the configure command line. In case you want to play around with the small example from above, I also put up a small archive containing the files (for such a small example, the C++ code could be included directly in the SWIG interface file via the %inline directive, but that’s how you would probably want to tackle a real library).

What can you expect to work? The test-suite which covers all the basic features of SWIG should build and run fine, which means that it will probably just work when trying to wrap a library. The source tree includes also a documentation chapter on D (Doc/Manual/D.html) which describes the basic structure and some of the D-specific features. As the D module started out as a fork from the C# one, the documentation on C# could be of considerable use for you as well.

There are still a few areas which need serious work, though. One of them is operator overloading, where both semantics and implementation differ quite a lot between C++ and D. It would probably be not too hard to come up with a solution (maybe using D’s extensive compile-time reflection capabilities to avoid having to add special cases to the SWIG module), but I would really appreciate some help from someone actually needing it here.

The other big one is multithreading support. Since I personally have not needed to use C++ libraries from D in a threaded setting yet, I have not really thought about the problems arising from multiple threads calling the wrapper code. Especially in combination with the garbage collector, I expect quite a lot of issues to pop up in a serious multithreaded environment. There are a few places which include threading-related code (synchronized, shared, …), but these are mostly remnants from the C# module, which may or may not apply to D – once again, I would be happy if somebody needing this would help me out here.

Speaking of C# remnants: As mentioned above, the D module was forked from the C# module, which in turn started out as a fork from the Java one. Due to this heritage, there are a few places where things could be done much easier in D. For example, the code for returning C strings to D without memory leaks is unnecessarily complex at the moment. But the same applies here as well – I would be happy to support anyone wanting to clean this up, but the current implementation did its job for me so far.

Anyway, I would be glad if some of you could go ahead and put SWIG to real-world use, so that any major bug can be fixed before the next SWIG release (not planned so far). If you stumble upon any issues or if any questions should arise, please feel free to contact me, either via mail, on digitalmars.D or in #D on freenode. Besides that, as always, it would also be nice just to hear about what you are doing with this.

In the meantime, two severe bugs in the code generated for Windows have been fixed; please be sure to use the latest version from SVN.

The Joys of OPTLINK

2010-02-13T00:00:00+01:00

As you might know, DMD/Windows (the reference compiler for the D programming language) does not use the standard COFF format for the object files it generates, but the fairly obscure OMF instead. This fact itself causes quite a number of annoyances. For example, the format differences make it unable to link static libraries produced by other compilers to D projects, which is especially annoying since it also applies to DLL import libraries. You also cannot use any tools which expect object files in COFF format and vice versa.

However, all of these issues, as annoying as they may be, do not pose a serious problem, they can all be worked around. But there is another one, and it has seven letters: OPTLINK. OPTLINK, courtesy of Digital Mars, is the linker which comes with DMD. There are quite a number of issues with it:

First, it is proprietary closed-source software. Apart from some people’s idealistic worries, this also poses a serious problem to more pragmatically inclined coders because there are no alternative linkers for OMF, at least no even half-decent ones. This means that if you stumble upon a bug, you can do nothing more than to wait for Walter Bright to fix it.

Second, even if the source code was available, it would probably still be hard to fix bugs, since, according to Walter himself, large parts are written in assembler – a maintainer’s nightmare. This might also explain why it took him so long to fix some serious bugs in the past…

Third, there are bugs. Lots of bugs, compared to other linkers and with the pretty high version number (8.00.2) in mind. If you want to know what I am talking about, just search the D newsgroups; projects which make extensive use of templates seem to be affected more often than others. Until yesterday, I personally had been spared from this kind of issues, but the OPTLINK bug I encountered yesterday almost drove me crazy, because one wouldn’t expect this at all:

After I had worked quite some time on Linux exclusively, I needed to compile a Windows version of a project of mine. So I went ahead and rebooted, updated DMD, Tango and a few other tools. Everything worked fine, the project even built fine, until I needed to build debug symbols into the binary. Every time I just added the -g flag to the compiler invocation, OPTLINK would abort with »Error 118: Filename Expected«. Because I had also upgraded my build tool, my first thought was that the linker commands could really be broken, but on closer inspection, it turned out that the invocation was generated perfectly fine. So I went on and downgraded all of the tools again, but to no avail – again the same error, although debug builds had worked flawlessly in the past.

After having searched for about an hour, I finally found the cause, and I could not really believe it at first: Compared to my previous D/Windows setup, I had added the Notepad++ installation directory to my PATH. You might ask yourself now, »Um, what? How should that break the linker?« Well, it turned out that OPTLINK apparently has problems with handling plus signs in all the lookup paths it uses, including not only the ones passed at the command line, but also those from the environment variables.

For a second I was really tempted to just drop DMD altogether, but unfortunately, there currently is no other D compiler of comparable quality for Windows. In my eyes, it would really help if DMD used COFF for its object files, making it possible to easily switch out OPTLINK, since the maturity of the tool-chain is currently the number one problem of D.

Setting up GDC and Tango on Linux x86

2009-10-26T00:00:00+01:00

Currently, there are three more-or-less working compilers for the D programming language (version 1): The oldest and most mature one is DMD, short for Digital Mars D Compiler, the official reference implementation by Walter Bright, the creator of D. It has grown reasonably stable, but has certain limitations, most of them resulting from using a proprietary back-end. Additionally, not all parts of it are Open Source (starting with a capital letter). The second one is LDC, a rather young, but quick-moving project which aims to port the front-end of DMD to the also fairly recent LLVM compiler infrastructure in order to leverage its advanced code generation and optimization infrastructure. While it still has some bugs to iron out (most notably missing exception support on Windows), it works reasonably well on Linux x86 (32 and 64 bit). The third compiler, and subject of interest for this post, is GDC. Like the other two compilers, it uses the Digital Mars D front-end, but coupled to the very mature GNU Compiler Collection (GCC) back-end, whose C/C++ compiler is widely used on Unix-like systems like Linux, Mac OS X, various flavors of BSD and also Windows through MinGW. Unfortunately, development on it has stalled, making it pretty much unusable due to the many bugs the old DMD front-end it uses contains.

However, there has been an effort started to resurrect GDC recently. Development takes place over at bitbucket (you can also find building instructions for GDC there) and the project has been able to celebrate some first success: The reasonably recent front-end versions 1.040 and 2.015 (for D2) are working with GCC 4.3. This seemed enough of a sign of life for me to decide to give GDC another try. After some initial problems (some of which resulted from bugs which have already been fixed in the official Mercurial repository) I managed to compile a GDC binary (frontend version 1.040 against GCC 4.3.1) which happily compiles the Tango standard library and a personal project of mine. This is what I did (silently omitting quite a few hours of searching and fixing bugs):

First, go to some temporary directory and checkout the GDC sources from the Mercurial repository (at the time of writing, revision 53 was current):

cd ~/tmp
hg clone http://bitbucket.org/goshawk/gdc

Then, download the core of GCC 4.3.1 from a mirror near you (version 4.3.2 should also work, but builds against 4.3.4 are currently known to be broken) and extract it inside the GDC sources:

wget ftp://gd.tuwien.ac.at/gnu/gcc/releases/gcc-4.3.1/gcc-core-4.3.1.tar.bz2
mkdir gdc/dev
cd gdc/dev
tar xjvf ../../gcc-core-4.3.1.tar.bz2

Now, link the GDC sources into the extracted directory and use the provided setup-gcc.sh script to patch GCC to enable D version 1:

cd gcc-4.3.4
ln -s ../../../d gcc/d
gcc/d/setup-gcc.sh --d-language-version=1

After that, you are ready to build and install GCC with D support. For this, go to some build directory and configure and make. You can, of course, choose an arbitrary directory for the build files (for instance, I personally prefer having the build files completely outside the source direcotry):

mkdir build
cd build
../configure --enable-languages=d --disable-multilib --disable-shared --prefix=/opt/gdc
make
sudo make install

Note that I configured GCC/GDC to be installed in /opt/gdc. As the build also includes the C compiler, this avoids any interference with the »normal« GCC installed probably in /usr. After the build has finished – this takes quite long, since GCC is built three times to bootstrap itself – you should have a working GDC executable in /opt/gdc/bin. Now for the second part, Tango:

Start off by fetching the Tango sources from the SVN to a temporary working directory (I worked with revision 5023):

cd ~/tmp
svn co http://svn.dsource.org/projects/tango/trunk tango

Unfortunately, Tango currently does not compile with GDC out of the box, you have to apply a couple of minor changes: The first change adds build/arch files for GDC/Linux:

diff --git a/build/arch/linux-i686-gdc-dbg.mak b/build/arch/linux-i686-gdc-dbg.mak
--- /dev/null
+++ b/build/arch/linux-i686-gdc-dbg.mak
@@ -0,0 +1,6 @@
+include $(ARCHDIR)/gdc.rules
+include $(ARCHDIR)/linux.inc
+
+# -Wall breaks the compilation with wrong errors
+DFLAGS_COMP=-g
+CFLAGS_COMP=-g

diff --git a/build/arch/linux-i686-gdc-opt.mak b/build/arch/linux-i686-gdc-opt.mak
--- /dev/null
+++ b/build/arch/linux-i686-gdc-opt.mak
@@ -0,0 +1,5 @@
+include $(ARCHDIR)/gdc.rules
+include $(ARCHDIR)/linux.inc
+
+DFLAGS_COMP=-O2
+CFLAGS_COMP=-O2

diff --git a/build/arch/linux-i686-gdc-tst.mak b/build/arch/linux-i686-gdc-tst.mak
--- /dev/null
+++ b/build/arch/linux-i686-gdc-tst.mak
@@ -0,0 +1,5 @@
+include $(ARCHDIR)/gdc.rules
+include $(ARCHDIR)/linux.inc
+
+DFLAGS_COMP=-g -fdeprecated -fdebug=UnitTest -funittest
+CFLAGS_COMP=-g

The second change removes the -fversion=Posix flag from the Makefile of the runtime because the DMD frontend GDC currently uses (1.040) does not allow it to be specified as it is set automatically (this restriction has been lifted in later versions):

diff --git a/runtime/compiler/gdc/Makefile.am b/runtime/compiler/gdc/Makefile.am
--- a/runtime/compiler/gdc/Makefile.am
+++ b/runtime/compiler/gdc/Makefile.am
@@ -18,7 +18,7 @@
 # AUTOMAKE_OPTIONS = 1.9.6 foreign no-dependencies

 OUR_CFLAGS=@DEFS@ -I.
-D_EXTRA_DFLAGS=-nostdinc -pipe -I../../.. -I../shared -fversion=Posix
+D_EXTRA_DFLAGS=-nostdinc -pipe -I../../.. -I../shared
 ALL_DFLAGS = $(DFLAGS) $(D_MEM_FLAGS) $(D_EXTRA_DFLAGS) $(MULTIFLAGS)

 host_alias=.
diff --git a/runtime/compiler/gdc/Makefile.in b/runtime/compiler/gdc/Makefile.in
--- a/runtime/compiler/gdc/Makefile.in
+++ b/runtime/compiler/gdc/Makefile.in
@@ -228,7 +228,7 @@ target_vendor = @target_vendor@
 top_builddir = @top_builddir@
 top_srcdir = @top_srcdir@
 OUR_CFLAGS = @DEFS@ -I.
-D_EXTRA_DFLAGS = -nostdinc -pipe -I../../.. -I../shared -fversion=Posix
+D_EXTRA_DFLAGS = -nostdinc -pipe -I../../.. -I../shared
 ALL_DFLAGS = $(DFLAGS) $(D_MEM_FLAGS) $(D_EXTRA_DFLAGS) $(MULTIFLAGS)
 toolexecdir = $(phobos_toolexecdir)
 toolexeclibdir = $(phobos_toolexeclibdir)

The third and last change adds a workaround to Tango’s user library for a bug in the DMD front-end which has been fixed by now (the compiler fails to resolve the type of the template parameter in the templated intpow function):

diff --git a/user/tango/math/internal/BiguintCore.d b/user/tango/math/internal/BiguintCore.d
--- a/user/tango/math/internal/BiguintCore.d
+++ b/user/tango/math/internal/BiguintCore.d
@@ -516,7 +516,7 @@ static BigUint pow(BigUint x, ulong y)
             }
             y0 = y/p;
             finalMultiplier = intpow(x0, y - y0*p);
-            x0 = intpow(x0, p);
+            x0 = intpow!(BigDigit)(x0, p);
         }
         xlength = 1;
     }

After you have applied these patches, you should be ready to build Tango (make sure that you have a cc somewhere in your PATH, if not, create a link to your system’s gcc):

sudo PATH=$PATH:/opt/gdc/bin DC=gdc build/build.sh --lib-install-dir /opt/gdc/lib

However, I had to remove Phobos’ object.d from /opt/gdc/include/d/4.3.1 first. build/build.sh should finish with a note reminding you that the user libraries still have to be installed. To do this, simply copy the contents of the user directory to /opt/gdc/include/d/4.3.1 after removing the old include files which are part of Phobos (you have to keep the gcc and i686-pc-linux-gnu directories though). Congratulations, now you should be able to build your Tango projects with GDC!

A quick tip for DSSS users: You probably have to modify your gdc-posix-tango profile to omit the -version=Posix switch (see above) on gdmd calls and add -L-ltango-base-gdc to the linker flags since Tango was not installed via DSSS in the above instructions.

Since I originally wrote this post, Tango’s build system was modified yet another time (at least, things are much simpler now). Instead of fiddling around with the makefiles, just use the bob tool from the build directory now which should work with GDC out of the box.

Installing DMD, LDC, Tango and DSSS on (K)Ubuntu Jaunty

2009-07-28T00:00:00+02:00

For quite a while now, I am using the D programming language, version 1 (I have not looked at D2 yet, it is said to be still rather unstable). Even though I like it very much for its syntactical quality and the language itself is reasonably mature, I must admit that setting up the toolchain correctly can still be a very cumbersome task to do, especially when you are new to D.

This post describes an installation routine that should provide you with a working D development environment containing DMD, LDC, Tango and DSSS on (K)Ubuntu Jaunty. Please note that it assumes your system to be »clean« – if you have already installed any D-related software, it is probably advisable to remove it completely to prevent any problems with, for instance, stale files.

mkdir -p ~/tmp
cd ~/tmp

wget http://ftp.digitalmars.com/dmd.1.050.zip
unzip dmd.1.050.zip
cd dmd/linux/bin/
chmod +x dmd dumpobj obj2asm rdmd
sudo cp dmd dmd.conf dumpobj obj2asm rdmd /usr/local/bin/

cd ~/tmp
svn co http://svn.dsource.org/projects/tango/trunk tango

cd tango/
sudo DC=dmd build/build.sh --lib-install-dir /usr/local/lib
sudo cp -rf user/object.di user/rt user/std user/tango /usr/local/include/d/

sudo su -c 'echo -e "[Environment]\nDFLAGS=-I/usr/local/include/d -defaultlib=tango-base-dmd -debuglib=tango-base-dmd -L-ltango-user-dmd -version=Tango -version=Posix" > /usr/local/bin/dmd.conf'

sudo su -c 'echo -e "# dsss\ndeb http://ppa.launchpad.net/d-language-packagers/ppa/ubuntu jaunty main" >> /etc/apt/sources.list'
sudo apt-get install dsss
sudo su -c 'echo "profile=dmd-posix-tango" > /etc/drebuild/default'

You should now be able to build your D/Tango programs with DMD and DSSS.

I would suggest giving LDC at least a short glance, a fairly young compiler project which leverages LLVM as its code generating backend. It is maturing very quickly and allows you to make use of the various features the LLVM compiler infrastructure provides, the most noticeable probably being its excellent optimization routines. Fortunately, there are current binary packages available at launchpad, so all that is needed to LDC is:

sudo su -c 'echo -e "# ldc-daily\ndeb http://ppa.launchpad.net/d-language-packagers/ppa/ubuntu karmic main\ndeb http://archive.ubuntu.com/ubuntu karmic main universe" >> /etc/apt/sources.list'
sudo apt-get update

sudo apt-get install ldc-daily libtango-ldc-daily-dev

cd ~/tmp
wget -O ldc-posix-tango http://www.dsource.org/projects/ldc/browser/ldc-posix-tango?format=raw
sudo su -c 'sed "s:ldc.rebuild.conf:/etc/ldc/ldc.rebuild.conf:"  /etc/drebuild/ldc-posix-tango'

sudo su -c 'echo "profile=ldc-posix-tango" > /etc/drebuild/default'

Note that the above commands install a daily snapshot of LDC, which I would recommend to use due to the currently fast development of LDC. In order not to break you Jaunty installation, please do not forget to comment out the official »karmic« repositories (which contain some dependencies for ldc-daily) in your /etc/apt/sources.list and run apt-get update after the installation is completed.

Both compilers are set up to use Tango, do not install Tango via DSSS! If you want to switch compilers, just activate the corresponding profile in /etc/drebuild/default and do not forget to rebuild any D libraries you might have compiled and installed with the old compiler (just run dsss net install … for the ones you installed using DSSS).

Since I wrote this post, Tango received yet another big structural change to its codebase (amongst other changes, the core and user libraries have been merged). Now, you should use the supplied »bob« tool now to build tango. Additionally, Karmic is now stable so you might have to adapt the APT repository-related instructions.

Strange segfaults when compiling with GDC

2009-02-04T00:00:00+01:00

Use -no-export-dynamic to prevent segfaults in external libraries when linking with GDC.

More to come soon…