Most of the posts on the Angle of Attack blog look to the past. "This is how we solved a hard technical problem;" "here's how we designed a product;" "here's how I optimized something." This post takes a different approach, as it speaks more to what FlightAware will do rather than what it has done. My hope is that it will give readers some insight into how significant technical decisions are made at FlightAware.

I recently wrote an internal document that examines the viability of a few different programming languages for writing medium-to-high level volume microservices, which I have adapted for this article. This was one part of a larger assessment performed across my entire division.

The Assessment

When assessing potential languages for implementing new web services in the Backend wing, three languages felt like natural candidates: Python, Go, and Rust. These languages share a few traits: large and enthusiastic communities, strong tooling, and technical advocates already present at FlightAware (sorry, I'm sure Java and .NET are great, but without existing expertise, adoption is an uphill battle). What follows is a broad survey of the languages' strengths and weaknesses and an attempt to capture the tradeoffs between them in the context of web service development. This rubric will be considered at the conclusion of each section, with scores ranging from 1-5 for each category:

• Performance

• Safety

• Learning Curve

• Cloud/Library Support

• Tooling

Python

Python may at first seem like the ideal choice for web services. The language boasts huge popularity in the software industry, resulting in a broad ecosystem of libraries and language tooling. FlightAware has also already invested significant effort in Python education, and our own Python codebases are growing (and will continue do so regardless of the outcome of this assessment). More amorphously, people just like writing in Python. Personally, I've found that Python does a really good job as a language of staying out of the way when trying to map thoughts to code and understand how to solve a problem.

There are two main areas where Python falls short, though, both arising from its dynamic nature: performance and safety.

Performance

As an interpreted language, Python's single-threaded performance has never been terribly good. There are various approaches that try to alleviate this issue, often by JITting hot code paths or by reimplementing them as native extensions. There's also an ongoing effort to improve the primary Python interpreter's performance, which has already born some fruit with its recent 3.11 release. It would probably be best not to make decisions based on hypothetical future performance, though. It's also unlikely that Python will ever get close to the performance of our other two candidates anyway.

I would describe Python's parallelism story as "unspecial" (certainly when compared to our other two choices). It has the usual collection of synchronization primitives (locks, queues, semaphores, etc.). These are made a bit easier to use correctly through Python's context manager syntax (the with keyword), but it's all still a bit fraught. Python is also limited in the sense that it only has process-level parallelism. This makes it difficult to share memory between distinct tasks (though you can always send data through queues and even use the fairly new shared_memory module in a pinch).

The poor single-threaded performance of Python can also contribute to system complexity as approaches like parallelism or queueing must be leveraged more readily to try to sidestep the problem. This ends up making the system harder to reason about and debug.

Safety

This property could perhaps be restated as, "how likely are you to know that code will or won't work before running it?" Ideally, we could have computers help us with this effort. Historically, Python, a dynamically typed language, has been quite unsafe. If you pass the wrong type of object to a function that doesn't expect it, you won't realize it until your code is running - possibly in production.

Tools capable of statically analyzing your code like pylint or pytype can help avoid these issues, but they're not perfect. Recently, optional support for static type annotations in the language has improved safety quite a bit. The key term here is "optional," though. Nothing's stopping you from running code that lacks type annotations, and there are still plenty of libraries that lack annotations. The ergonomics of Python's typing-related APIs were a bit rough at first but have improved greatly with subsequent releases (and are typically backported so you don't have to upgrade Python to get them).

Other stuff

Python's got a lot of good stuff going for it, and since it's effectively the only language of its type in this assessment (we're not looking at other interpreted languages like Ruby, TypeScript, or PHP), I chose to focus on its clearest shortcomings to better motivate the presence of Go and Rust in the comparison. It's worth covering some of Python's other attributes, though, for the sake of completeness.

Learning curve

Python has the gentlest of learning curves, with it often being the first language taught to novice programmers. Its tutorial is strong and I've always felt that its reference documentation is quite comprehensive. There are also myriad online resources covering Python (for better or for worse); I've found that Real Python has some quality in-depth articles on tricky Python features. The presence of modules like json and datetime in Python's standard library also contribute to it feeling like a language you can quickly become productive with; batteries are included!

About those batteries

Though Python has the reputation as a "batteries included" language with a strong standard library, this doesn't quite apply to web services. Due to Python's lack of a production HTTP server and the proliferation of 3rd party options, the WSGI standard was defined as an interface between web servers and Python web frameworks. This brings with it the usual tradeoff of having to spend the effort to pick both sides of the equation (server and framework) in exchange for a more adaptable ecosystem. These days WSGI has been succeeded by its async progeny ASGI, and you tend to have some combination of (uvicorn/nginx unit/daphne/...) x (fastapi/quart/falcon/...). There's also the somewhat unique case of Sanic, which is an ASGI-compatible framework that also has its own server baked in. Regardless of what you choose, you're still running just the one server, but the architecture all feels more complicated due to the proliferation of options.

Cloud / Library support

Due to Python's widespread use in the software industry, it's extremely likely that if there's a service you want to use from Python, a library to use said service already exists. This is true for Cloud platforms (where Python has first class support, both with SDKs and in serverless runtimes) and any given SaaS you can think of.

Tooling

Python has plenty of quality tools for profiling, formatting, linting, etc.; it's just a matter of finding them. The trouble is that much of Python's tooling is 3rd party, and there are often plenty of competing approaches. Consider this dauntingly titled article assessing the state of something like 20 different linters: https://inventwithpython.com/blog/2022/11/19/python-linter-comparison-2022-pylint-vs-pyflakes-vs-flake8-vs-autopep8-vs-bandit-vs-prospector-vs-pylama-vs-pyroma-vs-black-vs-mypy-vs-radon-vs-mccabe/. Even Python package management has at least 3 competing libraries (4 if you include the underlying official pip tool that most rely on, 5 if you count conda, 6 if you...). Speaking of package management, that seems to be a particularly sore spot for the community to harp on, though I haven't experienced frustrating issues with it. Perhaps it's more apparent when working with native libraries.

One area where Python suffers due to its lack of official tooling is in the inconsistent documentation for 3rd party libraries. Although many projects use Sphinx and are found at readthedocs.io, this is not broadly true, which can make it a jarring experience to switch between documentation for various libraries. I never realized how good things could be until I was exposed to Go's https://pkg.go.dev and Rust's https://docs.rs.

Conclusion

So, the rubric for Python:

• Performance: 2

• Safety: 3

• Learning Curve: 5

• Cloud/Library Support: 5

• Tooling: 4

Go(lang)

So, Python's slow and a bit unsafe; what are we to do? Let's throw some compiled, strongly typed code into the mix! Go came to the scene in 2012 touting garbage collection, language-level concurrency, and much more.

Essentially, Go is fast to learn, fast to build, and fast to run. It's become fairly popular for the development of high-performance web services (which makes sense, that's what Google designed it for!), with many companies transitioning to it from slower interpreted languages like Python. You can find various posts online highlighting such transitions. I think Go's approach to structural typing makes it particularly appealing when considering a transition from a dynamic language like Python whose duck typing is quite similar in nature.

Go's philosophy

There's no shortage of talks and posts from Go's creators about why they designed it the way they did (here are but a few: Go at Google: Language Design in the Service of Software Engineering, GopherCon 2015 Keynote, dotGo 2015 - Simplicity is Complicated). The designers make it clear that they were coming from a background of C/C++ and were mainly focused on improving that experience in the context of web services. They explicitly mention the following needs as primary drivers of Go's development:

• Developing "at scale" (referring to both headcount and dependency count)

• Programmer familiarity, so new developers can be brought up quickly

• "Modernity" – specifically, taking advantage of multicore CPUs in a highly networked environment

Out of these needs arose the following language properties:

• A spare (austere, even) syntax that eschews most features of functional programming

• Intentionally minimal ways to perform a given task (just one type of looping construct?!)

• Only one way of formatting your code (with no knobs to tune)

• Documentation as a first-class citizen

• Extremely fast compilation times

• Easy dependency management

• Composable interfaces instead of object orientation

• Green threads (goroutines) as a language construct

• Sizable standard library built with web services in mind

• First-class first party tooling

One thing I'd like to highlight from the above list is how much effort Go's developers put into parts of Go that are adjacent to the language design itself. It's clear that they valued the entire developer experience: writing code, reading code, building it, profiling it, etc. Go's web-focused standard library also makes it easy to get a service up and running without needing to reach for too many dependencies early on. This is a big reduction in cognitive load as developers spend less time surveying 3rd-party libraries to get what they need. I was also struck by some of the other modules you can find in Go's standard library, such as testing/quick for property-based testing, and net/pprof, which can run a server for exporting profile and trace data (this is the magic of including a production grade webserver in your standard library).

Go's ecosystem

Go has taken the cloud by storm. Docker and Kubernetes (the latter having come straight from Google) are probably the highest profile examples of Go's outsized presence in the cloud native ecosystem, but there are oodles of other widely used services implemented in Go (Prometheus, Grafana, Traefik, Caddy - the list is extensive). It's no exaggeration to say that many businesses have been built on Go. Go's prevalence in the cloud means it has first class SDK support for any given cloud vendor, but it also has a pretty good chance of having an official SDK from <insert-SaaS-here>. Chargebee and PagerDuty are two examples of services we use internally that I was able to readily find. Admittedly, most of these language-specific SDKs are just wrapping a standard HTTP API and were likely automatically generated, but I think it speaks to Go's hold in the industry that these platforms explicitly choose to support it.

Go's shortcomings

Go has managed to amass its fair share of detractors. They tend to focus on 3 main categories: Go's opinionated design, Go's weak type system, and Go's resistance to extensibility. There's also a grab bag of papercuts and inconsistencies in the language that tend to stand in stark contrast to Go's mantra of simplicity.

Opinionated design

If you want only one way to do something, you must pick a way and stick with it. There are going to be people who'd much rather do things another way, maybe for very good reasons. Go's designers decided that those reasons typically didn't merit having multiple ways to do something. Go's strong focus on procedural programming over functional programming paradigms is an example. If you're thinking about operating on a collection of data, your only choice is typically going to be with a for loop, probably with a bunch of if statements inside of it. It can be verbose, but it's obvious what's going on, and the developer doesn't need to spend any brainpower choosing between a loop or a functional approach. One of Go's designers, Rob Pike, went so far as to write a filter implementation in Go just to demonstrate that it's possible and he still thinks it's a bad idea.

A weak type system

Until recently, one of the most controversial exclusions from Go was generics. Although generics elicited the most vocal complaints, Go seems like it missed various other opportunities for crafting a more expressive type system that lets developers build safe abstractions (Go in general seems very anti-abstraction). Go's enums are but simple numbers, it has no way to indicate immutability, and it has no concept of (non)-nil-ability (outside of its value types, but then you get to complain about Go's zero values instead). All these features can be found in other languages (even Python's bolted-on type checking has them!), and they are seen as huge wins for writing safe code. In Go, the answer to most of these is to read the documentation carefully and write nil checks.

Lack of language extensibility

Go's designers don't want user code to look too much like native Go code. There's no operator overloading nor is there any way to use the range operator for your own types. Previously, Go's only generic data structures were the builtin map, slice, and channel types. This seems quite in line Go's explicit nature, actually, and some may see it as a boon. It certainly avoids certain clever programming that can harm code readability. Combine this with the lack of macros in Go, and users are frequently left writing and rewriting lots of boilerplate code.

Papercuts

Consider this a lightning round of Go issues that center around no specific theme but serve to irritate its users:

• Interaction of growing slices with underlying arrays can be unexpected, confusing

• Two nil values may not be equal

• Unexpected loop variable capture

• Struct tags feel like a hack, possibility for clashes

• Data races that Go won't catch

• Go zero values can be troublesome (particularly numbers)

None of these are gamebreakers, but when combined they leave me scratching my chin and wondering, "how simple is this language really?"

One more: error handling

You can't read about Go without hearing complaints about how it handles errors. Specifically, the fact that any operation that returns an error requires the additional 3-line (maybe 2) incantation

if err != nil {
    return nil, err
}

Often accompanying a one-line function call above it (or possibly integrated with it), many find Go's error handling distractingly verbose and a pain to wade through when reading code. There's also little in the way of safety rails for handling errors. If you forget to handle an error or outright ignore it with a _ variable name, Go's compiler won't bother you; I hope you remember to run errcheck! Go's error-handling capabilities have expanded slightly with time, with the introduction of error (un)wrapping in 1.13 improving their ergonomics significantly. The boilerplate, it seems, will remain, though.

Conclusion

And Go's rubric

• Performance: 4

• Safety: 3

• Learning Curve: 4

• Cloud/Library Support: 4

• Tooling: 5

Rust

The internet has been set aflame in recent years by the introduction of Rust, a compiled language that makes a lot of big promises with respect to safety, be that memory or types. Imagine writing effortlessly memory-safe code without needing a garbage collector! Well, it's not quite effortless. That hasn't stopped Rust from being the most-loved language in StackOverflow's yearly developer survey for the last 7 years, though (and no, it's not even close). Rust brings a lot more to the table than performance and memory safety. It has an incredibly expressive type system that can prevent entire classes of potential bugs, high quality tooling, and what is most kindly described as an extremely passionate community. Let's start by looking at what makes Rust so great.

The good stuff

Borrow checker

Rust's borrow checker -- combined with its type system -- underpins its memory safety guarantees and enables "fearless concurrency." It does this through the enforcement of a small number of ownership rules at compile-time. By ensuring that variables only have a single owner at a time, Rust's compiler can prevent various errors like dangling references, use-after-free, and double-free (I know, I know, "who cares, Go has a GC!"). Rust essentially takes C++'s best practice of Resource Acquisition Is Initialization (RAII) and makes it an inviolable property of the language. This ownership tracking also prevents data races in concurrent code since data can only have a single owner at a time (hence the "fearless concurrency"). It's worth noting that the power of the borrow checker goes beyond just memory safety; it's RAII, not M(emory)AII. In Rust, there's no need for Go's defer statements or Python's with blocks. Rust implicitly knows when a variable has gone out of scope and can clean up its resources accordingly. All this checking and compile-time safety has given Rust a strong reputation that "if your code builds, it's not going to crash."

Type system

I use the category "type system" here to cover a number of use cases enabled by Rust's type system that I've found particularly refreshing. Obviously, covering the entire type system would be quite an endeavor that none of us have the time for.

Errors as enums

Rust, much like Go, handles errors by returning them directly from functions just like other data. Distinctly from Go, though, Rust encodes errors into a Result enum, which can contain either the result of a successful execution or an error if something went wrong. It’s up to the caller of the function to handle this Result as they see fit. This pattern more accurately models the exclusive nature of errors (there’s no possibility that a function may return both a result and an error). It also encourages dealing with a potential error as soon as possible, preventing the scenario where an error check is forgotten and a bad result flows through your code until something somewhere else blows up.

Option enums

Rust’s notion of “null” differs from that of most other widely used languages. Rather than every type in Rust potentially being null, types in Rust are non-nullable, and there is instead an Option enum, which may contain either a value of a given type or the None sentinel. This allows functions to indicate in their signature whether they might return "nothing," and it lets the compiler check to ensure that you've properly handled that possibility in your calling code.

Tooling

Like Go, Rust's tooling story is quite strong. It has a high-quality formatter, language server, linter, etc. Its dependency management is particularly strong (which is pretty important, considering how many things Rust's standard library leaves out, but I'll get to that). Rust also follows in Go's footsteps with a single website where most library documentation can be found (docs.rs). Rust does rely on the broader native ecosystem for tooling in some cases (for performance profiling it's recommended to use the same tools you might use for profiling a C++ application, like perf or valgrind). The Rust team has also demonstrated a willingness to bring 3rd party tools that outclass their official counterparts into the Rust organization (this recently occurred with rust-analyzer replacing rls as the official Rust LSP server). There are still some tooling voids, though, particularly in the async realm where it can be difficult to inspect how a complex running system is behaving.

Language interop

Rust has a strikingly good language interoperability story. Its Foreign Function Interface (FFI) compatibility is equivalent to C, but it can be written more safely and with its full arsenal of high-level language features brought to bear. This presents an exciting direction to consider when assessing languages: perhaps we can write the bulk of a service in a more readily productive language and write the performance-critical parts as native Rust extensions. This mindset does discount Rust's other strengths like its null-safety, but it's still worth exploring.

The rough edges

It’s not all roses with Rust. Those incredible features granted by the borrow checker that I discussed above do come with a cost in the form of language complexity. The Rust team's painstaking approach to language development has also resulted in a reliance on Rust’s ecosystem of community libraries for functionality that many would expect to see in the language’s own standard library. Let’s explore these issues in more detail.

The learning curve

Rust's learning curve (the "cost" I mentioned above) seems practically legendary at this point. Just the knowledge of it served as a significant mental block to me when trying to learn and assess Rust for this writeup. Unless a developer is already comfortable with C++'s best practices around RAII and ownership and has a strong grasp of the typing concepts originating with the ML programming language, they're going to have a lot of learning to do to become proficient with Rust - and I haven't even mentioned Rust's macros yet! Rust puts its best foot forward with the Rust book, a free online resource for learning Rust that I found to be very approachable, but there remains a gulf between "getting" Rust's concepts and actually putting them to work. There are myriad stories of people having to "fight with the borrow checker" to get their code to compile. While this is ultimately beneficial, as it ensures their code's safety and proper function, it can be quite demotivating to spend potentially hours on your code with little to show for it.

All the features of Rust's type system also bring with them a glut of syntax, making it a visually dense language, and it's tough to learn it all piecemeal. It feels like you need to understand the whole language before you can make much progress with it.

The density of language features in Rust can also make the question of "is this code idiomatic?" hard to answer conclusively. Consider the various ways an error from a function can be handled:

• Match statement

• "If let" statement

• Result methods

• ? operator

The differences between these approaches can differ subtly and become unnecessary points of focus. As a personal anecdote, I was writing some Rust code to handle a Result wrapping an Option wrapping another Option. I painstakingly golfed my way to a 7-line chain of function calls while spending quite a while getting comfortable with Result/Option's large library of functions; I then discovered with clippy's help that there was an even better 6-line approach that I had missed! Now while this was honestly quite fun for me, it wasn't really the best use of my time.

Light standard library

Rust has a fairly small standard library by most standards; certainly no one would describe it as "batteries included." Some see this as a positive due to the strict backwards compatibility expectations of the stdlib ("the standard library is where modules go to die" is a recurring phrase in the Python community highlighting this phenomenon), meaning that non-critical functionality is free to become the best version of itself as a 3rd party library. This approach can create overhead, though, as competing implementations of functionality arise, and developers must spend effort to determine what will best fit their use case. For a few examples, consider time vs chrono, tokio vs async-std, or rustls vs native-tls. This fracturing drags down library developers who need to decide whether to support one or both implementations and produces comical "feature matrices" of possible library combinations.

Another anecdote: when writing some Rust to replicate a test in an existing Go service of ours that checks the color of a couple pixels in a png, I looked to crates.io for a library to handle this. I added the png crate as it seemed like it could do the job, but I found out after attempting to implement the test and digging around the documentation for a while that the crate just wasn't high-level enough to let me do what I wanted succinctly. Instead, it was the image crate that I ended up using (and it was admittedly quite succinct). Go just has a png module built in, along with an interface for getting pixels from images.

Immature ecosystem

This isn't an inherent shortcoming of Rust, but it is a reality that must be accepted: Rust's ecosystem for web services is still young and continues to shift. Full integration of async (which most web frameworks rely on) throughout Rust is still being developed. Rust's async "book", the feature's canonical documentation, is also missing significant sections. Rocket was previously a web framework of choice, but now its sole developer has let it languish. Actix-web's maintainer received so much community backlash for using significant amounts of unsafe code that he stepped down (this was 3 years ago and actix-web quickly got a new maintainer, but it's indicative of the perils of relying on 3rd party projects). Rust also lacks official SDK support from most cloud and SaaS platforms, though AWS has a Rust SDK in developer preview and seems committed to promoting the language more broadly.

Conclusion

And finally, Rust:

• Performance: 5

• Safety: 4

• Learning Curve: 2

• Cloud/Library Support: 2

• Tooling: 5

The above languages at FlightAware

Of course, we can't simply consider Python, Go, and Rust in a vacuum. Each language is already in use at FlightAware in some capacity. As mentioned, we've invested significantly in Python as a general purpose language of choice at FlightAware; Rust has been used by the Flight Tracking Wing to implement multiple high performance data streaming services; and even Go has been used in a more limited fashion to implement some high volume microservices that handle tasks such as proxying weather map tiles through AWS.

Though people often say you should use the best tool for the job at hand, there is also a cost to the proliferation of languages in an organization. It promotes siloing; effort is duplicated when implementing libraries, developing guidelines, and even just learning the languages; and there can be a maintenance burden if your experts in a given language depart.

At FlightAware, Python certainly has the strongest foothold, with Rust coming in second, and Go trailing behind it. Suppose Go really is the best tool for the job of developing performant web services; is it better enough than Rust that it's worth the added burden and risk detailed above? Consider, too, the fact that the Backend wing at FlightAware is going to be investing in Rust for the development of other software like feeds from our data partners.

If we consider the possibility that different web services may merit different languages based on their performance needs, I see the following possible selections we could make:

• Go

  • With its generally very solid performance, strong developer experience, and web service focus, I see Go as a "sweet spot" language. It's productive enough that we likely wouldn't need to reach for Python, and it's performant enough that Rust wouldn't be necessary either (no, I don't expect we'll encounter Discord-level scaling for quite some time).

• Rust

  • Using just Rust for web services could work, but I think Rust's focus on correctness and safety over all else could really slow down development, especially as the wing first trains up. That combined with Rust's still immature web service ecosystem makes it a tough sell.

• Python and Rust

  • Although Python alone won't suffice for our more performance-intensive services, I think Python combined with Rust could be a compelling choice. We could use Rust as needed, enabling a more gradual learning process, and continue to leverage the languages that FlightAware's already invested in.

Overall, I believe that Go would be the best choice for developing web services at FlightAware. Yet it's hard to wholeheartedly recommend it with the awareness that FlightAware's Rust codebases continue to grow, as does the language's community. Rust's costs seem mainly upfront, with large promised dividends.

Ultimately, I see Go as the lowest risk choice. I have confidence that the Backend wing can learn it and become productive with it readily, and I have little doubt that it will serve our needs. I also think it will be an approachable language for the Web wing, who will also need to work with and support these services.