The ottonova Tech Radar is a list of technologies. It’s defined by an assessment outcome, called ring assignment and has four rings with the following definitions:
ADOPT – Technologies we have high confidence in to serve our purpose, also in large scale. Technologies with a usage culture in our ottonova production environment, low risk and recommended to be widely used.
TRIAL – Technologies that we have seen work with success in project work to solve a real problem; first serious usage experience that confirm benefits and can uncover limitations. TRIAL technologies are slightly more risky; some engineers in our organization walked this path and will share knowledge and experiences.
ASSESS – Technologies that are promising and have clear potential value-add for us; technologies worth to invest some research and prototyping efforts in to see if it has impact. ASSESS technologies have higher risks; they are often brand new and highly unproven in our organisation. You will find some engineers that have knowledge in the technology and promote it, you may even find teams that have started a prototyping effort.
HOLD – Technologies not recommended to be used for new projects. Technologies that we think are not (yet) worth to (further) invest in. HOLD technologies should not be used for new projects, but usually can be continued for existing projects.
What do we use it for?
The Tech Radar is a tool to inspire and support engineering teams at ottonova to pick the best technologies for new projects. It provides a platform to share knowledge and experience in technologies, to reflect on technology decisions and continuously evolve our technology landscape.
Based on the pioneering work of ThoughtWorks, our Tech Radar sets out the changes in technologies that are interesting in software development — changes that we think our engineering teams should pay attention to and use in their projects.
When and how is the radar updated?
In general discussions around technology and their implementation is driven everywhere across our tech departments. Once we identify that a new technology is raised, we discuss and consolidate it in our Architecture Team.
We collect these entries and once per quarter the Architecture Team rates and assigns them to the appropriate ring definition.
Disclaimer: We used Zalando’s open source code to create our Tech Radar and were heavily influenced by their implementation. Feel free to do the same to create your own version.
ottonova has a long history of using cross-functional teams for Software Engineering. Very early on we tried to establish them to become more productive and product centred.
It took us a few tries to get it right and oh boy, did we make a lot of mistakes along the way.
I would like to share some of our learnings and put them into basic rules that everyone can use.
Not a band-aid 🩹
Don’t put the cross-functional team structure on top of something else.
Suddenly we had conflicts of interest between the cross-functional teams and the old team structure. Everyone was confused what to focus on. The Team Leads didn’t engage with their engineers on a daily basis anymore. It was messy.
Clarity for everyone should be the first priority when you want to really establish cross-functional teams. No hidden layers, no hidden agendas.
Establish team culture 🗿
A great team shares a connection.
The easiest way to create a connection is to give the team freedom to establish a unique culture and identity. Everything starts with a name.
Give your teams the freedom to chose their own name and celebrate this name with an inauguration ceremony. Everyone in the group should well understand what their purpose and vision is.
Why was the team established?
What are our short, mid and long term goals?
How will we get there?
The more independent this process is, the easier it will be for the team to form a bond and feel connected.
Empower your team 💪
A successful cross-functional team feels empowered to take care of their value stream. They have to be the clear owner of their artefacts and have all resources at hand to complete their tasks.
Take for example a team that is supposed to take care of a customer facing app connected to a backend API. To maintain the app and implement new features without dependencies the team needs a minimum setup like this:
QA engineers for manual and automated testing
Product Owner who takes care of stakeholder communication
Team or Tech Lead
Every dependency to another team causes workflow delays and disruption.
A duo at the helm 🛶
A cross-functional team benefits greatly from a strong and well gelled Team Lead-Product Owner combo.
The Product Owner handles the stakeholder communication, collects the business requirements and pours them into user stories to fill the backlog.
The Team Lead takes care of the software delivery and people management. They unblock tasks, assign experts, make sure that tickets are completed with technical requirements and resolve personal issues.
The harmony of the TL-PO pair can be make-or-break for a team and should always be considered when assembling a cross-functional team.
Here are some words about how the Backend Team goes about finding new team members. We want to do our part and share with the Community, as well as provide a bit more transparency into ottonova and how we are building state-of-the-art software that powers Germany’s first digital health insurance.
This article covers what we are doing in the Backend Team, what we value, and how we ensure we hire people that share our values.
The Backend Team
Our team is responsible for many of the services that power our health insurance solutions at ottonova. This includes our own unique functionality, like document management, appointment timelines, guided signup, as well as interconnecting industry-specific specialized applications.
Under the hood, we manage a collection of independent microservices. Most of them written in PHP and delivering REST APIs through Symfony, but a couple leveraging Node.js or Go. Of course, everything we use is cutting-edge technology, and we periodically upgrade.
As a fairly young company, we spend most of our time adding new functionality to our software in cooperation with product owners. But at the same time we invest fair efforts into continuously improving the technical quality of our services.
Technical excellence is one of our team’s core values. To this end, we are practitioners of domain driven design (DDD). Our services are built around clearly defined domains and follow strict separation boundaries.
Because we created an architecture that allows it, and we have the internal support to focus on quality. We invest a lot into keeping the bar high and whenever needed we refactor and make sure the Domain Layer stays up to date with the business needs, or that the Infrastructure Layer is performant enough and can scale.
Although most of our work is done using PHP, we strongly believe in using the right tool for the job. Modern PHP happens to be a pretty good tool for describing a rich Domain. But we like to be pragmatic and where it is not good enough, maybe in terms of performance, we are free to choose something more appropriate.
Expectations from a new team member
From someone joining our team we first of all expect the right mindset for working in a company that values quality. We are looking for colleagues that are capable and eager to learn as well as happy to share their existing knowledge with the team.
A certain set of skills or the right foundation for developing those skills is needed as well. We are particularly interested in a good mastery of programming and PHP fundamentals, Web Development, REST, OOP, and Clean Code.
As actual coding is central to our work, we require and test the ability to both write code on the spot and to come up with clean design.
These expectations can be grouped into four main pillars that a candidate will be evaluated on:
Mindset – able and willing to both acquire and transfer knowledge inside a team
Knowledge – possesses the core knowledge needed for using the languages and tools we use
Clean Design – capable to employ industry standards to come up with simple solutions that can be understood by others
Coding Fluency – can easily transfer requirements into code and coding is a natural process
The Recruiting Process
To get to work with us, a candidate goes through a process designed to validate our main pillars. All this while giving them plenty of time to get to know us and have all their questions answered.
It starts with a short call with HR, followed by a simple home coding assignment. Next there is a quick technical screening call. If everything is successful, we finish it up with an in-person meeting where we take 1-2 hours to get to know each other better.
The Coding Assignment
Counting mostly for the Clean Design pillar, we start our process with a coding assignment that we send to applicants. This is meant to allow them to show how they would normally solve a problem in their day-to-day work. It can be done at home with little time pressure, as it is estimated to take a couple of hours, and it can be delivered within the next 10 days.
The solution to this would potentially fit into a few lines of code. But since the requirement is to treat it as a realistic assignment, we are expecting something a bit more elaborate. We are particularly interested in how well the design reflects the requirements and the usage of clean OOP and language features, the correctness of the result (including edge cases), and tests.
We value everyone’s time and we don’t want unnecessary effort invested into this. We definitely do not care about features that were not asked for, overly engineered user interfaces or formatting, or usage of design patterns just for the sake of showcasing their knowledge.
It will ideally be complex enough to reflect the requirements in code, but simple enough that anyone can understand the implementation without explanations.
The Tech Screening
To test the Knowledge pillar we continue with a Zoom call. This step was designed for efficiency. By timeboxing to 30 minutes we make sure everyone has time for it, even on short notice. We don’t want to lose the interest of good candidates getting lost in a scheduling maze.
Even if it’s short, this call ensures for us a considerably higher match rate for the in-person interview. In time we found that there really are just a handful of fundamental concepts that we expect a new colleague to already know. Many of the other can quickly be learned by any competent programmer.
All topics covered in this screening are objectively answerable. So at the end of a successful round we can make the invitation for the next step.
This is when we really get to know each other. This is ideally done at our office in central Munich – easier for people already close by, but equally doable for those coming from afar.
In this meeting we start by introducing ourselves to each other and sharing some information about the team and the company in general.
Next we ask about the candidate’s previous work experience. With this and the overall way our dialog progresses we want to check the Mindset pillar and ensure that the potential new colleague fits well into our team.
After that we will go into a new round of “questioning” to deeper test the Knowledge pillar. Similar to the Tech Screening, but this time open-ended. Informed opinions are expected and valued. We definitely want to talk about REST, microservices, web security, design patterns and OOP in general, or even agile processes.
Then comes the fun part. We get to write some code. Well… mostly the candidate writes it, but we can also help. We go through a few mostly straight forward coding problems that can be solved on the spot. We are not looking for obscure PHP function knowledge, bullet-proof code, or anything ready to be released. We just want to see how a new problem is tackled and make sure that writing code comes as something natural to the candidate. With this we cover the Coding Fluency pillar.
Afterwards it’s the interviewee’s turn. We take our time to answer any questions they may have. They get a chance to meet someone from another team and get a tour of the office.
The interviewers consult and if there is a unanimous “hire” decision, we send an offer. In any case, as soon as possible (usually a few days) we inform the candidate of the outcome.
At ottonova, we have taken the challenge of bringing a slowly moving and mostly antiquated business, the one of health insurance in Germany, into the 21st century. For over three years already we’ve been offering our customers not only competitive insurance tariffs, but also the digital products that go along with them. All this to make their lives better and to prove that, yes, insurance can also be easy and fun to use.
It follows naturally that all of us here are innovators at heart. And innovators like to try out new things. “We are brave” – as one of our company values says. In the last years, we have found that a good way to get our creative juices flowing is by holding internal hackathons. The events are organized by our IT department and welcome everyone from the company that has an idea or simply wants to help out. In the first two editions, almost all the engineers participated, joined by many more colleagues from other departments, working in teams. The enthusiasm before and after the event was overwhelming. Everyone was happy to quickly bring a cool new thing live, to try out new tech, or just to hack something up together with some colleagues with whom they would normally not work with.
After such a positive response to the previous instances, we were eagerly awaiting a new installment. What we affectionately call a Hackotton was due to have its third edition this summer. It has already been half a year since the last one. Too long, some would say. We wanted our Hackotton 3.0.
But will a Hackotton be the same if, due to Covid-19, we’re not all in the office together? We have been working remotely since spring, when the social distancing measures were started. We have been doing so successfully, but this time it was not just work, it was a bit more. We did not know how an event that is mostly social in nature would end up if organized remotely. Still, we decided to give it a try.
We planned the event to take about two days. That is enough time to try out an idea, especially when you’re able to hack it and there is no need for the rigorousness of our normal style of work. It all started with a kick-off. If in the last editions this was where ideas were pitched and teams were formed, this time round it was just a very short zoom call to give the start. The idea gathering and team forming was replaced with a Confluence page with a join option for team members. As this was announced well in advance. There was plenty of time to think about topics and decide which teams to join. We ended up having a total of 10 teams and, for the first time, we were joined by external guests.
When the Hackottons where held in the office, all the planning and communication was done in person. There was a constant rustle on the floor, caused by people changing desks to be close to their team, or heated discussion between team members. This time, while the office was still available for the few that chose to go there, most people worked remotely. So the close collaboration mode needed to be emulated somehow. Though some quick planning was done in the beginning to distribute tasks, in the end, we relied mostly on instant messaging (Mattermost) and video calling (Zoom) to get things done.
As we have the good habit of developing only containerized applications, and have quite some practice with this, even “hacked” solutions built during the Hackotton are put into Docker containers. This allows for way easier collaboration and building more complex systems also in hackathon mode. Especially now when working remotely, it feels so good for a team mate to give you an already built container, and you can simply connect it to the part that you are implementing. As a follow up advantage of this setup, we can quickly add these containers to AWS’s ECS in our pre-live environment, so that we demo our ideas under realistic conditions, and later, with the infrastructure already solved, have a smooth path towards a live release.
At the end of an intense second day of hacking, and after hurrying to bring our projects to a functional state and crop up some sort of presentation, we had the chance to show our ideas to the whole company (over Zoom) and see what the other teams managed to build. Everyone was given a 5 minute slot to demo and present their project and impress the audience. A hard task to contain all the passion the teams poured into their projects into 5 short minutes, but the moderators were understanding and allowed for more time when the mark was overstepped.
After all the teams got their turn in the spotlight and demoed their ideas, we had an online voting round to choose a winner. The laurel wreath for the 3rd Hackotton edition went, by a margin, to a very fruitful collaboration between the android developers and one of our designers. They came up with a radically different approach for structuring our mobile apps. They showed how by putting the focus on the doctor appointments, interactions with our apps can be greatly simplified. Their solution takes our current separate Timeline, Documents and Chat sections in the apps, and brings them together under Appointments. This way related invoices, claim settlements, doctor appointments can be reviewed under one place, as well as making it easier to communicate with our customer support. This will allow our app users to have a better overview and control over their interactions with healthcare providers and with us as an insurance company, and consequently them being happier with our service.
Apart from the great mobile app improvement idea, all the other teams brought convincing proposals. In this edition we got to see new tooling for our colleagues in the Sales department, improvements in document processing using OCR, optimizations to our website, internal applications for making our work easier, and even more changes for the better for our customers’ experience using our products. The response from the audience was overwhelmingly positive, and even if the event ended with the presentations and voting, discussions about the projects continued well into the next days.
As any hackathon comes with improvement in morale and team spirit, we can already conclude that organizing them is a long term win for the participants and the company overall. For us at ottonova, it happened that the past Hackottons came with some short term wins as well. Some of the projects ended up being used – either directly, or after planning them into our Roadmap. From the past events we have “Agento”, a very handy internal automation tool that we use to quickly insure children attached to a parent’s account, as well as “ottoPrint” our own PDF generation service, which is much faster and more customizable than the off-the-shelf solution we’ve been using until then. So we are fairly confident that some of this edition’s projects will end up being used. The winner already sparked the interest of the product owners.
With this third edition, and organizing it even remotely, the Hackotton has become part of our company culture and has proven that it can endure. As we are coming to know that “after the Hackotton is before the Hackotton”, we’re now anxiously waiting for the fourth one, and using the time to come up with cool new ideas.
Let’s talk about organization, preparation and venue first. From my point of view, the organizers did a lot to make this conference great, at least they tried to do their best. The conference, same as the workshop took place in the very center of the city, in the biggest public hall. It was quite easy to find it and to get there, either with public transport or by foot if you were staying in the city center. One day in advance I got an email with quite descriptive instruction about everything I should know: how to get there, recommended places to stay, what they prepared for attendees etc.
Unfortunately, I was a bit confused, because I did not figure out how to buy a ticket for the Workshop day if you already bought a conference ticket, when the workshop stream was not announced. Directly at the entrance to the workshop, there was a possibility to buy it, but I decided that it is not worth it and it is a bit expensive. Anyway, I am not sad about this fact, as conference organizers prepared a free of charge tour in the city and it was a good alternative.
On the conference day, everything started with registration, grabbing my personal badge, general community talk and breakfast. I felt pretty comfortable there as organizers always tried to take care of us: there was a lot of drinks and snacks there, lunch was served by a special catering company and in the afternoon they made homemade cakes for us.
And now more about the conference: it had 3 streams in parallel and in the afternoon one of these streams became unConf, where anyone could share something with everyone. The biggest stream had a lot of seats for all attendees, but not every talk assembled so many participants.
You have to know about me, that I do not believe I can learn something from any talk, because most of the things are already known from programming paradigms, web development and PHP in general. Usually, talks at conferences are just a shared experience, exploring new unknown stuff or repeating something like SOLID, caching and other. Everything you want to learn could be easily and faster found on the web, and if you missed some talks you could watch them later on YouTube, moreover, for free. Personally, all these conferences are just community spirit, free baubles and lunch. But this conference managed to absolutely surprise me!
The biggest discovery for me was a talk about modern SQL from Markus Winnand. How much I did not know about SQL in general. Knowing modern relational databases, such as MySQL, PostgreSQL, Oracle DB or SQLite, does not mean you know modern SQL. The most SQL standards and features were introduced since SQL-1999 (recursion), SQL-2003 (schemaless and analytical, like median), SQL-2011 (system versioning, aka time-travelling), SQL-2016 (JSON_TABLE), etc. A lot has happened since SQL-92, SQL has evolved beyond the relational idea. If you use SQL for CRUD operations only, you are doing it wrong.
Do not use self-joins in SQL anymore! Also, avoid OFFSETs from your statements, they are a performance leak!
The saddest conclusion I made: the most popular RDBMSes made themselves compliant with modern SQL only recently, but still, there are some features not ready in all RDBMSes. But what about modern ORMs? When will they be compliant with all the features we have in modern SQL? Or is it the best solution, for now, to avoid ORMs and write custom queries?
By the way, he has a book about SQL performance explained, it is highly recommended to read it. You can find more info on his website or buy his book with stickers and mug.
The conference was worth visiting at least for the sake of this talk, and I was very pleased with the fact that I learned so many new things I can use in my applications to boost performance. Anyway, there were also a few talks worth attending:
Encoding and charset, presented by Andreas Heigl. Worth to know that encoding is not a character set and what is what. How to properly work with UTF-8 in PHP and MySQL. Be aware that utf8 in MySQL is not a real UTF-8 encoding, you have to use utf8mb4 instead for proper UTF-8.
Automated PHP Refactoring, presented by Haralan Dobrev at unConf. He shared a collection of all known tools and showed how they could be implemented together.
Hexagonal Architecture by Nicolas Carlo. It was not that much for me personally as DDD is based on this architecture, but anyway it was a very good structured talk with good examples and real-life cases.
PHP-FIG Panel to describe a stack of standards they have. Be aware that PSR-2 is deprecated right now and PSR-12 should be used instead.
I and the rest of the PHP engineers at ottonova, had the pleasure of attending the PHP.Barcelona Conference in November this year. It was a great experience, spanning over two full days.
I’ve put together some quick and biased notes about the presentations. Here it goes…
Opening Keynote – Rasmus Lerdorf
Really uplifting beginning of the conference with Rasmus going down memory lane. We got to know some of the motivation and the process of creating PHP. All this further strengthening my opinion that it didn’t really start as serious programming language. (Ok. I admit. It did progress since then, so don’t start throwing stones.)
A funny highlight was the explanation that “PHP is vertically consistent” – the PHP functions match the vendor functions they create an API for. Of course, this leaves it inconsistent with itself.
Also, really enlightening, was clarifying why adding things like further type checking, generics, or class modifiers, like “immutable” would be a serious performance hit to the language – so we should really stop hoping that any of that would come soon.
From Helpers to Middleware – Marco Pivetta
Nice practical presentation. It showed how design can incrementally evolve, from what would basically be spaghetti code, into a proper modular and scalable middleware-style architecture. While a good start, I would have hoped to see a deeper dive into this style, because where it stopped it felt like it just scratched the surface.
Microservices gone wrong – Anthony Ferrara
Although this was just a public retrospective of a specific project, which maybe will not resonate with everyone, for me and the work we do at ottonova, there were still a couple of valuable lessons to take home:
Messages that microservices use to synchronize can be differentiated into proper Events and RPCs, and these categories can and maybe should be treated differently. The latter require a response back, while the former don’t really need it. We don’t have this clear separation ourselves yet, but the need for it is definitely starting to show.
Each entity in your domain does not need to have its own service. Larger services are also fine, if they make sense for your use case. Our own setup is using domain-defined microservices, of various sizes, so seeing that splitting everything aggressively may backfire will make us think twice when extracting a new microservice.
He has a cute dog.
Serverless PHP applications with Bref – Matthieu Napoli
I guess it’s nice to see that there is a way to hook up PHP to Lambda, but then again, why take the effort to force it and not just use a language supported directly? Apart from that aspect, interesting to see an intro into AWS Lambda, since I didn’t try it out myself yet.
JSON Web Tokens – Sam Bellen
Not that much to say here: JWTs. We already use them, you should use them too. They’re easy to work with and really useful. At their core they are just signed information in a nice and standard JSON format. Nevertheless still a very powerful concept as it enables you to transfer claims securely from one party to another.
Well, this one was awkward. Especially since I had the pleasure of sitting through this exact presentations some months before. Thijs is a dedicated evangelist, you have to give it to him. He manages to squeeze everything out of what Varnish can do and serve it to his audience on a silver platter.
Now come the buts. The use-cases considered in the presentation are outdated. Really focusing too much on server side rendering and templating. And I particularly did not enjoy instructing an auditorium full of developers (maybe some more impressionable than other) to use their caching layer for keeping application logic.
Nothing wrong with Varnish itself, though. And since we need to keep all our data inside Germany, for legal reasons, maybe we’ll need to consider ourselves an on-premise caching solution in the future.
PHP Performance Trivia – Nikita Popov
Really confident presentation from one of the core PHP contributors, containing a deeper dive into how the OPcache works, and what its limitations are. Not that serious limitations, if you ask me. With a bit of care for how you handle deployments, you should be fine.
Also interesting to see some benchmarks that show that using objects instead of arrays is much more memory-efficient in PHP. Not that the opposite would have made us drop using Value Objects, but still good to see that we’re already using the memory friendly option.
Get GOing with a new language – Kat Zień
While there was nothing spectacular about this presentation, just an intro into Go, I still enjoyed seeing that there is a clear interest from the PHP community to explore other languages. Go is particularly relevant for us at ottonova, since we’re already using it for our messaging setup and we plan to try some more areas where we think it would do a better job than good old PHP.
Advanced Web Application Architecture – Matthias Noback
Nice structured first dive into DDD and the rationale of it. We’re already doing most of what this presentation talks about, and much more, at ottonova, but it helps to see another’s take on it and double-check our approach.
It was reassuring to see that one of the first requirements of DDD is to separate your Domain from your Infrastructure, a thing that we carefully follow, along with some more advanced techniques.
Also really appreciated the general advice that not every project is the same and if something applies somewhere it does not automatically mean that it will apply to your situation. This is, of course, common sense, but it doesn’t hurt to hear some basic common sense from time to time, in a world overflooded with strong opinions.
Working with Webhooks – Lorna Mitchell
Decent structured presentation about Webhooks – an architectural style for async communication. Nothing groundbreaking here, but since this style is not standardly used (or at least not by me), it’s nice to be reminded it exists. Good tip with using ngrok for exposing local stuff.
Supercharge your apps with ReactPHP & PHP-PM – Albert Casademont
Since FPM is already showing its scaling limitations for us, it was particularly interesting to see what other options would be available.
For many of our processed requests, we have the common pattern that we get some input, we process it, and then pass it to another service (maybe external) and wait for a response. Then we do some more work on it and respond to our client. This implies considerable processing power wasted on our side – we wait a lot for HTTP responses ourselves. This is where some concurrent PHP would come in handy. While one request is waiting, other requests can be handled. So we will definitely be looking into either PHP-PM, or Swoole in the future.
It’s all about the goto – Derick Rethans
This was a nice, theoretical, dive into how PHP parses and executes code. For someone with a minimal Computer Science background, I think it was still fairly basic, and I don’t think there was much to take away.
Develop microservices in PHP – Enrico Zimuel
This was an interesting walk-through what are some of the benefits, and specific concerns, of using microservices. Nothing too new to us, since we already heavily rely on microservices both in the PHP group and in our other teams.
One thing we noted down to improve was the standardisation of error responses. Good hint. We’ll definitely look into that one.
Mutation Testing – Better code by making bugs – Théo Fidry
Mutation Testing seems really cool as a theoretical concept, and it’s nice to see that someone is trying it out. Not sure how it would work out in practice. So far, I can see two major downsides. First, it simply takes a lot of time to run such a test. Even with optimisations, this will be something that could take hours. Second, the testing itself seems only as good as your mutators, and I think writing relevant mutators is not a trivial task. Having some that just replace operators could be straight-forward, but how much does that help?
Back to Munich
And the rest we sadly had to skip, to not miss our plane back home.
Overall, as we hoped when booking the tickets, the lineup was a solid one. And they delivered. Kudos to them, as well as to the organizers. We came back with lots of new ideas, some that we will try in the near future, and confirmation that we are on the right track with many of our architectural decisions.
We will definitely have Barcelona on our list for next year as well. Lovely city too.