We recently had our program release party with Jim Webber, where we published our program for GOTO Amsterdam 2016. Over the last weeks we filled every missing spot and the exciting program is now complete. But we haven’t shared our thoughts on the program decisions itself.
Posts by Sven Johann
GOTO Amsterdam already seems a while ago, but I still want to write about two sessions I really enjoyed: the agile mindset (Linda Rising) and a lightning talk about left folds in functional languages (Erik Meijer) unfolded to an object-oriented language.
Unfold your left fold
I never really understood the folding concept in functional programming. At GOTO Amsterdam I thought, I’ll give my brain a chance again and visited the short talk from Erik Meijer, where he explained folding in Haskell and later unfolded his fold in C#.
Two weeks after the talk, I already forgot how it works, so I decided to work through his talk in this blog showing some of his samples with Scala and Java.+
I want to give you a little introduction to my latest InfoQ article on Technical Debt, which I wrote together with my former colleague Eberhard Wolff. We work with new, small and cool systems, but also with large and old systems (= these systems were really valuable over many many years!). Large means 50 to 500 men years resulting in several millions lines of code (LoC) using hundreds of database tables. Old means, that some of these systems have been started in the last century and still use technologies like Visual Basic 6, EJB 2.x, Struts 1.x, PL/SQL or XDoclet, and are connected to COBOL-based systems, just to name a few things which were cool 10-35 years ago, but are now seen as cruelties if you have to add features to such a system.
These systems have also seen many developers come and go, developing good stuff and, especially under time pressure, bad stuff. All these systems have a reasonable amount of Technical Debt (which is unavoidable). While working on these systems, we found that the usual “Technical Debt is a disease” complaining strategy is neither helpful nor correct. Some architects deal very well with this problem, resulting in 10 year old systems, which are still in a really good shape. Other architects didn’t deal well with it, so the development of business-critical systems really slowed down, sometimes whole development organizations come to a stand-still. The problem is then, that you cannot throw away a 2 mio LoC business-critical system and develop it anew. That’s too expensive and too risky. You have to pay the interest of Technical Debt in terms of slow delivery of features. This shows, that not only developers suffer under Technical Debt, but also the business side, who waits for features a way too long. In fast paced environments, that can almost kill businesses!
In our InfoQ article we give you a nice overview what Technical Debt is really about. We found a surprisingly long list of of stakeholders – a software developer is only one of many. After explaining various ways how to identify Technical Debt, we head to the most important part: how to effectively deal with it. We give practical advice by debating the pros and cons of various successful strategies. Finally, we show when it could be useful to just pay the interest, to do debt conversion or to pay back the debt. We think, if you deal with large and successful (=old) systems, you should have a look at the article and leave us and the community your thoughts for further discussion.
After an exciting few days at the QCon conference in London last week, I am slowly recovering from all the new input I got, and decided to do this by writing a little summary of “all things agile” from the Thursday as well as the highlights the other two days too.
Cherry Picking Wednesday
After the keynote, we warmed up with a couple of beers, before we left the conference center to join the conference party at the truly cool Central Hall Westminster.
On Thursday I only visited the presentations in the agile track. Most speakers reported that one thing makes it really difficult to be agile these days: project teams do perfectly implement “Scrum by the book”. Sounds good, but… Having a look at the agile manifesto we see that agile teams should value individuals and interactions over processes and tools. Doing Scrum (or any other agile method) by the book unfortunately leads to situations, where teams value the written down process in the book over interactions with the team mates (“you have to do it like this, because otherwise it’s not Scrum, I read it in the book/heard it in some talk”). Ward Cunningham said during a chat: “Kent Beck and me wrote the extreme programming book, but we’re not doing it like described in the book. It’s just a point to get started. You have to understand what makes you tomorrow better than today. That should influence and drive your process”.
The first talk from the agile track was from Glen Ford, explicitly talking about People over Process. He shared his experience from being a tech lead at a start-up. He recognized that his team’s doing Scrum as a ritual act, without asking why they’re doing certain things. They discovered that a process isn’t a rule of law, but rather a set of concepts. Instead of following rules, they formed a team vision and a why for everything they do. If you don’t find a why, don’t do it. In their specific context, they couldn’t find a why for estimations, so they skipped it. Finding a why also encourages communication and the more communication they had, the less process they needed. The best and most open communication is among team members, which know each others strengths, weaknesses and quirks. So they decided to do not break teams apart, but rather to form long-running teams, which eventually got hyper-productive.
Hyper-performing without the hype by Dan North was seen as the highlight of the day. Indeed, it was the expected hour of entertainment paired with agile expertise. Dan explained the things he saw in the past, which made teams performing extremely well. I won’t mention all, but only those I learned as well in the past: developers should also be absolute domain experts., e.g. you can only be a great team developing trading software, if every developer of the team knows the trading business well. Developers have to participate in trading classes and you should seed your team with domain experts the developers can practice with. In times of Lean Software Development everyone is seeking for value, but we should nevertheless prioritize risk higher than value. Even if a solution promises high value, the question remains how much uncertainty we have to face for that solution. He then moved on to a classic: planning is everything, plans are nothing. Plan as far as you need and adjust along the way. He also strongly recommended to try out technical things regularly, even if you’ll never use them in production: languages, programming concepts, etc (I personally get stomach ache when writing for-loops in Java filtering or mapping list content since I learned functional programming…). Finally he recommends to release often, if possible daily, even if you think the software is not ready. It sounds weird to show the customer something which isn’t ready yet, but if you give the customer the chance to use the software, you’ll get feedback from real use, which is extremely helpful (think about opportunity costs).
Besides the presentations, we also had the opportunity to chat two hours with Ward Cunningham about Technical Debt (beyond the current hype and all the misunderstandings around that) and Agile Software Development (also beyond the hype around Scrum and all the misunderstandings around that). All agilistas completed the day with a fire-site chat with Dan North and Ward Cunningham organized by the Agile London user group.
Big Data and Architectures of the small & beautiful on Friday
The opening keynote from Damian Conway was one of the highlights of this conference. He talked about how to do interesting and fun technical presentations. He gave a great example, because the 45 minutes with him were super-entertaining and we all got very valuable take-a-ways for preparing presentations.
Cool talks about MongoDB, Hadoop and Riak followed the keynote. Since Hadoop and Big Data are a big hype, the speaker Jamie Engesser from HortonWorks pointed out, that we should really, really do Big Data for a reason and not because it’s cool 😉 Matt Asay from 10gen gave a nice talk about the past, present and future of NoSQL. He pointed out, that there is a set of exclusive use cases for document-oriented, column-oriented, key/value-oriented and relational datastores. But: there are many overlaps, where either one of them could be a good solution. He questioned polyglot persistence, because he’s not sure if an organisation can really deal with several different databases in operation. Andy Gross from Basho gave an honest talk about the problems Riak faced the last 5 years and how they solved them.
For me, the absolute highlight of the day was the presentation about the Triposo travel guide architecture. The presenters, former Googlers and ThoughtWorkers are avid travelers and wanted to know, if they can do better then the common travel guides like Lonely Planet & Co. So they started what they learned at Google: crawl the web, aggregate, match, and rank. They send their crawlers to fetch gigabytes of travel related content from all kinds of sources like Wikitravel, Wikipedia, Facebook, Open Street Maps, Flickr and some more.
Once they have all the data, it’s time to parse. From each source they extract information about the places like villages, cities and countries, and the points of interest (restaurants, museums, shops, trees, etc). They’re looking for patterns to create one bucket of information for a particular place from all the various sources they crawled. After this phase they end up with exactly one record for each place or point of interest that has all the information from any of the sources they’ve used. Now it is time to rank and these ideas were pretty cool. Among other things, they extract meta data from Flickr pictures like where and when the pictures were made. That brings them interesting information about possible events, e.g. there are many pictures around 52°38’N 4°45’E, but only from April to September and only Fridays between 10.00–12.30 a.m. There must be something interesting! That’s the cheese market in Alkmaar. So, if your on a trip in Amsterdam, your Triposo travel app proposes you a day trip to Alkmaar on Friday (with my Lonely Planet book I usually see that only when it is already too late). I don’t know if their app will revolutionize the way we travel, but it is an interesting idea how to use the huge amount of publicly available data (=Open Data).
Since not only the idea is nice how to use Open Data, but also the available languages and services they use (Python, Google Spreadsheets, Amazon S3, Amazon Mechanical Turk, automated deployment into the App Store with a browser remote control, etc) we invited them to give a presentation at one of our GOTO nights.
QCon London was absolutely worth it this year and hopefully I’ll be back for more inspiration next year. I was really impressed by the quality of the conference – tracks, speakers, keynotes, chocolate cakes and the selection of international beers. QCon London is one of the best technical conferences I’ve participated in and I recommend it for anyone interested in enterprise software development (It’s almost as good as GOTO Amsterdam ;-)).
There is an interesting approach from the German IT service provider Adesso AG and the Ruhr Institute for Software Technology on how to balance project risks between service providers and their customers within a new contracting model, which is a combination of fixed price and Time & Material approaches. I want to give a brief overview of the work they published.
The problem with contracts within (agile) software development
Projects typically overrun time and budget constraints, missing requirements and quality expectations. These problems mostly arise from insufficient communication between business and technology experts or users and developers. Additionally, customers initially have a rather coarse grained idea of the system they need, but controversially they have to negotiate a fixed price contract or a budget ceiling to control the costs for project, because the purchasing department forces them to do that. This cost estimation can be done best by a service provider based on a complete specification of the system to be build. However, such a complete specification is neither economical (it requires considerable effort on both sides) nor is it helpful (nobody in the world can express all requirements in sufficiently complete and consistent detail up front).
In practice, service providers trying to make a best guess of the price and to balance the expected effort with the price the customer is willing to pay. Additionally, a service provider is sometimes forced to under-bid competing providers, since some customers have only one criterion: the price. This usually results in a too low amount, and the service providers will struggle for sure with the low bid. Many of them became therefore experts in the then following inevitable game of overly expensive change requests (and gone is all the cost control for the customer…). Neither constellation is helpful for a lasting customer relationship.
Fixed price or time & material aren’t the solution, too!
Agile software development replaces voluminous specifications by short iteration and (end-user-) feedback cycles, which makes it virtually impossible to set a fixed price upfront. The scope of the project generally emerges gradually during the project, so the actual effort is not foreseeable. In case of an agile software projects, a fixed-price contract exposes the service provider to the complete project risk. A pure time and material (T&M) contract is on the other side a high risk for the customer, since the service provider can now blow up development effort and neglect quality on the customer’s expense. Since fixed-price and T&M are not satisfactory for both parties, it is desirable to find a contracting model that has a built-in risk limitation mechanism for the service provider and a built-in cost limitation mechanism for the customer.
A fair pricing model?
The adVANTAGE model combines elements of fixed-price and T&M contracting models. It strives to provide some idea of the overall project scope (in terms of requirements, time and budget) as you know it from fixed-price projects. Also, the customer pays the complete effort to the service provider as you know it from T&M projects. The commercial principles behind this are risk distribution and efficiency incentives for both parties for the whole project duration. In the following sections we’ll see how these commercial aspects tie in with the sprints and deliverables of an agile process model.
Step 1: Collect and estimate. To get an initial overview of the project scope, we collect all the customers requirements before the first iteration, typically “must-haves” as well as nice-to-haves, business goals and business ideas with a coarse, non-technical description on the business level. The service provider then estimates the required effort for each requirement. Due to its coarse nature, these estimations have some level of uncertainty. This uncertainty is not expected higher than the uncertainty of a “normal” estimation of a fixed-price bid. Nothing really new here, but in contrast to traditional contracting models, the total of all estimations is not used to calculate a fixed price, but rather serves as a plausible point of orientation for the upcoming steps.
Step 2: Prioritize, plan and implement. Based on the estimate and the customers internal budget ceiling, the customer can now prioritize, eliminate or add requirements. In doing so, he transparently balances the importance of each requirement with his available budget and the time frame. Based on the prioritization, the customer and the service provider can now agree on the contents of the first sprint. At the beginning of a sprint, its requirements are refined together by the service provider and the customer into more detailed specifications. For the subsequent step of inspection and billing it is necessary, that all requirements are thoroughly integrated and tested (= production ready software, automated deployment).
Step 3: Inspect and pay. Now it’s getting interesting! The adVANTAGE model ties the billing very closely to the sprint. Depending on whether all user stories were satisfactory implemented or completed, we’ll have different billing scenarios:
- Underspend sprints are really good for the customer, because only the actual effort will be billed. There’s no additional gain for the service provider of being cheaper then expected.
- In overspend sprints the customer will be billed with the extra effort by a considerably reduced rate that penalizes the service provider. This is fair for both sides, because the customer is only billed for the actual effort and the service provider still gets paid for his extra effort.
- Incomplete/unaccepted requirements can be moved to the next sprint, where the required extra effort will be penalized like described above.
Step 4: Plan the next sprint or terminate. After each sprint, the customer has the option to start the next sprint, where he can re-prioritize all requirements as well as add or remove new requirements to keep the focus on the overall project goal and constraints. Change requests are treated as new requirements. The customer can also terminate the project, when he feels it has reached the required functionality and/or its budget limits. Since every sprint results in a running system, this exit strategy is risk-free for the customer.
Although Trifork is not currently using the adVANTAGE model, it does contain a lot of benefits for both the customer and service provider;
- The model provides what an agile project needs: a fixed scope during sprints and the ability to change the scope of the next sprints. With a fixed price approach this cannot happen without the more/less work discussion;
- No additional fixed price margin is added to the price, which potentially ends up with the customer paying too much;
- The IT service provider is given a fair incentive to deliver the promised result in a timely manner;
- Running out of budget in a fixed price project will always imply cutback on other means, mostly quality. Since the estimations are done on detailed requirements, the risk of cutting corners on quality is much less.
From a customer perspective, it makes sense to give it a try, when you ended up several times in the cheapest-wins-but-they-have-bled-you-dry-with-change-request loop and ask service providers to go for the adVANTAGE model. This kind of project cost control seems to be better than the traditional one. What do you think?
Just to get us started, have you ever had a look on the original paper describing the waterfall model? You really should, have look at it. What you see is surprising; the author shows the famous waterfall diagram on page 2 and directly below he writes “… the implementation described above is risky and invites failure”. Managers (and developers?) just misunderstood the concept over the past 40 years. Maybe even worse, many project managers never actually read the paper and just adapted the waterfall model by “hear-say”. The good ones have been incremental and iterative for the last 40 years!
Even more misunderstandings…
Have you had a look on many so-called agile software projects over the past 24 months? Very likely yes. What did you see? Maybe projects claim to be agile, because they do a stand-up every morning, because they do not do BDUF any more, because they have a product owner and so on. But many things, which are needed to be agile, let’s say trust for example, which is often ignored, because “that doesn’t work for us” (you could also say, we only want to change the parts, which are easy to change). The ScrumBut gained some attention here: <We use Scrum, but> <we have these unique circumstances> <so we have had to modify Scrum so it works here>. That’s not necessarily a bad thing, but many projects have a too simplistic idea of agile adoption (or adaption).
The hip & trendy v’s the bad & evil
Yes, agile is “hip” these days, agile is “trendy” and the rest is “bad & evil”, so everybody claims to be agile. Clear, who doesn’t want to be agile? You have a project where agile doesn’t fit well? What do you say? We’re not agile? Sounds strange, right?
I suspect, agile software development will suffer the same destiny like the waterfall model: misunderstood and misinterpreted. That’s because many people sit around the campfire, eat marshmallows and talk about how great agility is. Agility, the desired silver-bullet in software engineering. No architecture, no documentation, just hacking… wait a minute. That’s XP, Scrum is different: daily stand-up, product owner, bi-yearly releases, full-planned backlog for the next 2 years, quality-assured requirement documents named user stories… wait a minute, that’s wrong, that’s not agile.
Innovators, Imitators, Idiots
Why do we find these interpretations over and over again? We may find a hint for that in Warren Buffetts comment on the natural progression of how good ideas go wrong. He called this the “three Is.” “First come the innovators, who see opportunities and create genuine value. Then come the imitators, who copy what the innovators have done. Sometimes they improve on the original idea; often they tarnish it. Last come the idiots, whose avarice undermines the innovations they are trying to exploit.” (initially from William Taylor and nicely described for agility by Jim Highsmith).
Should we care? No! As stated above, any complex idea will always be misunderstood in the broad. If you want to change that, you have to change mankind. Good luck with that! We have to accept the wrong interpretations of agility and that agility will be blamed in the future for failed projects. Maybe we’re also interpreting it wrong? Maybe a bit? Let’s work on it.
Which memes are spread?
But a lot of things have changed through the agile movement, even the worst misinterpretation cannot change that. Agile software development questioned successfully the possibility of a complete a priori planning for complex (software) projects. We rather want short feedback-cycles with a lot of communication. Projects with fixed scope, fixed quality and fixed price are also successfully questioned. Better price models are on the way. Added value is more important than following a process description. Trust and respect is not something for crybabies, it is rather an engine for productivity and replaces time-consuming finger-pointing.
We have to go on…
There are a couple of innovations we should put an eye on: DevOps, continuous delivery, the conversations over technical debt, Lean, Kanban, Agile Leadership. The Scrum Alliance has a new strategy and they want to make sure, that new ScrumMasters should know that “Scrum Is not Enough” and that there is a whole world of great, agile ideas out there. I am keen to read their upcoming articles about Lean and XP to get new ideas.
Anyway, this is just a little insight into my take on this matter, interested to discuss this further and gain even more insight then join me at our next tech session in December. It’s free, fun and we even serve Pizza & Beer.
Technical Debt (TD) is the gap between perfectly developing software and the reality (ship date, skills of the engineers, tools available, working environment). You get it, when you take shortcuts that fall short on good practices. As in finance, not all debt is bad. Take a debt to buy a house is mostly a good thing, buy something luxury with your credit card knowing your paycheck will not cover it, is stupid .
In finance, measuring debt is easy (principal of debt, interest). But in software development? There are code measurement tools out there like Sonar, which try to measure TD. They are pretty good in measuring the code quality (over time), test coverage, code duplication, debt hot-spots (classes, packages, sprints, etc). They also come up with a number: your TD is $10,000. But: is this a good measurement? When should we start paying back what?