One of the cool things of organizing a conference is dealing with the unexpected. The workshops are busy and tomorrow the conference kicks of with an awesome program. In the last days before the conference we had a few unfortunate cancellations for which we needed to find replacements. We worked really hard but with great pride can announce our latest speakers for GOTO Amsterdam!
Posts Tagged ‘Agile’
You Should GOTO This Conference: Software Development That is Agile, Lean and Rugged
Competitive pressures have pushed speed of software development to become one of the highest priorities for businesses today. Improved tools and techniques have moved the state-of-the-art in agile development from pushing monolithic updates every few weeks to a continuous delivery of “micro-services” several times a day. But this move to “run what you wrote” and developer-driven infrastructure means developers are not only delivering products faster today, but are also responsible for the efficiency and safety of those products once they move into production.
This is a major shift in software development, and it forms a narrative that will underlie programming at the upcoming GOTO London technology conference, which I am helping organize and will be held Sept. 14-18.
A conference is usually about new and exciting stuff and Scrum clearly isn’t new. Scrum is nowadays also adopted across all industries and I personally do not know one software developer who wasn’t part of a Scrum project yet. So, it’s not exciting either. Why the heck have then a Scrum track and even two keynotes about Scrum and Agile Software Development?
People who have worked with me know I’m a bit of a technical conservative. I’m very wary of quickly adopting the latest fads and trends because I’ve seen the collective hype and the following disillusionment too many times, including software being built with the then-latest-hype framework or platform and a year later being stuck with now-obsolete technology that only the original developers and a handful of other people still have any real experience with.
For the same reason I’ve avoided software tech conferences in the past years. A few visits to conferences several years ago on each occasion left me with feeling that I’d heard a lot about a lot, but that it wasn’t really going to improve my daily software development work.
Luckily, Goto Amsterdam 2014 was different.
Many, if not all, of the talks were relevant to my actual, day to day, software development job. I learned about looking at Agile in a different way. I heard people speak on real life problems being solved with actual, current, widely adopted technology. I even listened to talks that weren’t really that much about software development at all.
So let me walk you through my Goto Friday.
GOTO Amsterdam already seems a while ago, but I still want to write about two sessions I really enjoyed: the agile mindset (Linda Rising) and a lightning talk about left folds in functional languages (Erik Meijer) unfolded to an object-oriented language.
Unfold your left fold
I never really understood the folding concept in functional programming. At GOTO Amsterdam I thought, I’ll give my brain a chance again and visited the short talk from Erik Meijer, where he explained folding in Haskell and later unfolded his fold in C#.
Two weeks after the talk, I already forgot how it works, so I decided to work through his talk in this blog showing some of his samples with Scala and Java.+
As a Scrum Master my opinion on doing Scrum in combination with a fixed price, fixed functionality and fixed deadline is somewhat tricky to grasp. However, it still common that in many projects fixed price is just simply the norm. For instance, this is often the case in public tenders for government or education institutions for various projects such as the procurement of a new software system to name an example.
So if you and your company win the tender it’s up to you and your team to deal with the “fixed everything” aspect of the project. Interested in how to deal with some of the ongoing aspects of changes in the requirements, deadlines and how to keep both the customer and the team happy? Read on, in this blog I will share with you our experiences with fixed price projects and Scrum.
Read the rest of this entry »
Westminster Abbey – View from the Queen Elizabeth II conference center
…and now back home
On my desk lies a stack of notepads from the QCon sponsors. I pick up one of them and turn few pages trying to decipher my own handwriting. As I read my notes I reflect back on the conference. QCon had a great line up and awesome keynote speakers: Turing award winner Barbara Liskov, Ward Cunningham, inventor of the Wiki, and of course Damian Conway who gave two highly entertaining keynotes. My colleague Sven Johann and I were at QCon for three days. We attended a few talks together but also went our own way from time to time. Below I discuss the talks I attended that Sven didn’t cover in his QCon blog from last week.
Ideas not art: drawing out solutions – Heather Willems
The first talk I cover has nothing to do with software technology but with communication. Heather Willems shows us the value of communicating ideas visually. She started the talk with an entertaining discussion of the benefits of drawing in knowledge work. Diagrams and visuals help us to retain information and helps group discussion. The short of it: it’s OK to doodle. In fact it is encouraged!
The second part of the talk was a mini-workshop where we learned how to create our own icons and draw faces expressing basic emotions. These icons can form the building blocks of bigger diagrams. Earlier in the day Heather made a graphic recording of Barbara Liskov’s keynote. In real-time: Heather was drawing on-the-spot based on what Barbara was talking about!
Graphic recording of Barbara Liskov’s keynote ‘The power of abstraction’
You are not a software developer! – Russel Miles
Thought provoking talk by Russel Miles about simplicity in problem solving. His main message: in the last decade we learned to deliver software quite well and now face a different problem: overproduction. Problems can often be solved much easier or without writing software at all. Russel argues that software developers find requirements boring, yet they have the drive to code, hence they sometimes create complex, over-engineered solutions.
He also warns of oversimplifying: a solution so simple that the value we seek is lost. His concluding remark relates to a key tenet of Agile development: delivering valuable software frequently. He proposes to instead focus on ‘delivering valuable change frequently’. Work on the change you want to accomplish rather than cranking out new features. These ideas are related to the concepts of impact mapping, which he used to structure the presentation itself, he revealed in the end 🙂
The inevitability of failure – Dave Cliff
In this talk professor Dave Cliff of the Large Scale Complex IT systems group at University of Bristol warns us about the evergrowing complexity in large scale software systems. Especially automated traders in financial markets. Dave mentions recent stock market crashes as failures. These failures did not make big waves in the news, but could have had catastrophic effects if the market did not recover properly. He discusses an interesting concept, normalization of deviance.
Everytime a safety margin is crossed without problems it is likely that the safety margin will be ignored in the future. He argues that we were quite lucky with the temporary market crashes. Because of ‘normalization of defiance’ it’s only a matter of time before a serious failure occurs. Unfortunately I missed an overview of ways to prevent these kind of problems. If they can be prevented at all. A principle from cybernetics, Ashby’s law of requisite variety, says that a system can only be controlled if the controller has enough variety in it’s actions to compensate any behaviour of the system to be controlled. In a financial market, with many interacting traders, human or not, this isn’t the case.
Performance testing Java applications – Martin Thompson
Informative talk about performance testing Java applications. Starts with fundamental definitions and covers tools and approaches on how to do all sorts of performance testing. Martin proposes to use a red-green-debug-profile-refactor cycle in order to really know what is happening with your code and how it performs. Another takeway is the difference between performance testing and optimization. Yes, defer optimization until you need it. But this is not a reason not to know the boundaries of your system. When load testing, use a framework that spends little time on parsing requests and responses. All good points and I’ll have to read his slides again later for all the links to the tools he suggests for performance testing.
Insanely Better Presentations – Damian Conway
Great talk on how to give presentations. Damian shows examples of bad slides and refactors them during his talk. He discusses fear of public speaking, how to properly prepare a talk, a lot of great tips! I won’t do the talk justice by describing it in text. Many of Conway’s ideas have to be seen live to make sense. Nevertheless there is a method to the madness:
- Dump everything you know on the subject
- Decide on 5 main points and create storyline that flows between them
- Toss out everything that does not fit the storyline
- Simplicity – show less content, on more slides
- Use highlighting for code walkthroughs
- Use animations to show code refactorings
- Get rid of distractions
- The most important part of a presentation is person-to-person communication!
- Practice in front of an audience at least 3 times. Even if it is just your cat.
Visualization with HTML 5 – Dio Synodinos
In this tour of technologies for visualizing data, Dio showed everything from CSS3 to SVG, processing and D3js. For each of these he gave a good overview of their pros and cons and made specific animations and demos for all of them. He also mentioned pure CSS3 iOS icons. Lot’s of eye candy and from reading the #QconLondon Twitter stream it seems a few people liked to try out all these frameworks and technologies.
Thankfully, there were plenty of coffee breaks at the conference. During breaks I often bumped into Sejal and Daphne, as well as other Triforkers from both our Zurich & Aarhaus offices. Besides attending talks we went to a nice conference party and went out to dinner a few times. Between talks Sven and I meetup and had a chat about what we saw, whilst we grabbed some delicious cookies here and there. Unfortunately the chocolate chip ones were gone most of the time!
AS HULK WANT MORE CHOCOLATE CHIP COOKIES! #QConLondon
— Hulk User Stories (@HulkUserStories) March 6, 2013
At one point I took the elevator to the top floor. On my right is a large table covered with techy books. Conference goers try to walk by, but look over and can’t help but gravitate to this mountain of tech information. Of course I couldn’t resist either so I browsed a bit and finally bought ‘Team Geek – A software developer’s guide to working well with others’. Later on I visit the web development open space. I listen in on a few conversations and end up chatting with James and Kathy, the camera operators, while they are packing their stuff. They have been filming all the talks for the last three days and we talk a bit about the conference until the place closes down.
All in all QCon London 2013 was a great conference!
There is an interesting approach from the German IT service provider Adesso AG and the Ruhr Institute for Software Technology on how to balance project risks between service providers and their customers within a new contracting model, which is a combination of fixed price and Time & Material approaches. I want to give a brief overview of the work they published.
The problem with contracts within (agile) software development
Projects typically overrun time and budget constraints, missing requirements and quality expectations. These problems mostly arise from insufficient communication between business and technology experts or users and developers. Additionally, customers initially have a rather coarse grained idea of the system they need, but controversially they have to negotiate a fixed price contract or a budget ceiling to control the costs for project, because the purchasing department forces them to do that. This cost estimation can be done best by a service provider based on a complete specification of the system to be build. However, such a complete specification is neither economical (it requires considerable effort on both sides) nor is it helpful (nobody in the world can express all requirements in sufficiently complete and consistent detail up front).
In practice, service providers trying to make a best guess of the price and to balance the expected effort with the price the customer is willing to pay. Additionally, a service provider is sometimes forced to under-bid competing providers, since some customers have only one criterion: the price. This usually results in a too low amount, and the service providers will struggle for sure with the low bid. Many of them became therefore experts in the then following inevitable game of overly expensive change requests (and gone is all the cost control for the customer…). Neither constellation is helpful for a lasting customer relationship.
Fixed price or time & material aren’t the solution, too!
Agile software development replaces voluminous specifications by short iteration and (end-user-) feedback cycles, which makes it virtually impossible to set a fixed price upfront. The scope of the project generally emerges gradually during the project, so the actual effort is not foreseeable. In case of an agile software projects, a fixed-price contract exposes the service provider to the complete project risk. A pure time and material (T&M) contract is on the other side a high risk for the customer, since the service provider can now blow up development effort and neglect quality on the customer’s expense. Since fixed-price and T&M are not satisfactory for both parties, it is desirable to find a contracting model that has a built-in risk limitation mechanism for the service provider and a built-in cost limitation mechanism for the customer.
A fair pricing model?
The adVANTAGE model combines elements of fixed-price and T&M contracting models. It strives to provide some idea of the overall project scope (in terms of requirements, time and budget) as you know it from fixed-price projects. Also, the customer pays the complete effort to the service provider as you know it from T&M projects. The commercial principles behind this are risk distribution and efficiency incentives for both parties for the whole project duration. In the following sections we’ll see how these commercial aspects tie in with the sprints and deliverables of an agile process model.
Step 1: Collect and estimate. To get an initial overview of the project scope, we collect all the customers requirements before the first iteration, typically “must-haves” as well as nice-to-haves, business goals and business ideas with a coarse, non-technical description on the business level. The service provider then estimates the required effort for each requirement. Due to its coarse nature, these estimations have some level of uncertainty. This uncertainty is not expected higher than the uncertainty of a “normal” estimation of a fixed-price bid. Nothing really new here, but in contrast to traditional contracting models, the total of all estimations is not used to calculate a fixed price, but rather serves as a plausible point of orientation for the upcoming steps.
Step 2: Prioritize, plan and implement. Based on the estimate and the customers internal budget ceiling, the customer can now prioritize, eliminate or add requirements. In doing so, he transparently balances the importance of each requirement with his available budget and the time frame. Based on the prioritization, the customer and the service provider can now agree on the contents of the first sprint. At the beginning of a sprint, its requirements are refined together by the service provider and the customer into more detailed specifications. For the subsequent step of inspection and billing it is necessary, that all requirements are thoroughly integrated and tested (= production ready software, automated deployment).
Step 3: Inspect and pay. Now it’s getting interesting! The adVANTAGE model ties the billing very closely to the sprint. Depending on whether all user stories were satisfactory implemented or completed, we’ll have different billing scenarios:
- Underspend sprints are really good for the customer, because only the actual effort will be billed. There’s no additional gain for the service provider of being cheaper then expected.
- In overspend sprints the customer will be billed with the extra effort by a considerably reduced rate that penalizes the service provider. This is fair for both sides, because the customer is only billed for the actual effort and the service provider still gets paid for his extra effort.
- Incomplete/unaccepted requirements can be moved to the next sprint, where the required extra effort will be penalized like described above.
Step 4: Plan the next sprint or terminate. After each sprint, the customer has the option to start the next sprint, where he can re-prioritize all requirements as well as add or remove new requirements to keep the focus on the overall project goal and constraints. Change requests are treated as new requirements. The customer can also terminate the project, when he feels it has reached the required functionality and/or its budget limits. Since every sprint results in a running system, this exit strategy is risk-free for the customer.
Although Trifork is not currently using the adVANTAGE model, it does contain a lot of benefits for both the customer and service provider;
- The model provides what an agile project needs: a fixed scope during sprints and the ability to change the scope of the next sprints. With a fixed price approach this cannot happen without the more/less work discussion;
- No additional fixed price margin is added to the price, which potentially ends up with the customer paying too much;
- The IT service provider is given a fair incentive to deliver the promised result in a timely manner;
- Running out of budget in a fixed price project will always imply cutback on other means, mostly quality. Since the estimations are done on detailed requirements, the risk of cutting corners on quality is much less.
From a customer perspective, it makes sense to give it a try, when you ended up several times in the cheapest-wins-but-they-have-bled-you-dry-with-change-request loop and ask service providers to go for the adVANTAGE model. This kind of project cost control seems to be better than the traditional one. What do you think?
Just to get us started, have you ever had a look on the original paper describing the waterfall model? You really should, have look at it. What you see is surprising; the author shows the famous waterfall diagram on page 2 and directly below he writes “… the implementation described above is risky and invites failure”. Managers (and developers?) just misunderstood the concept over the past 40 years. Maybe even worse, many project managers never actually read the paper and just adapted the waterfall model by “hear-say”. The good ones have been incremental and iterative for the last 40 years!
Even more misunderstandings…
Have you had a look on many so-called agile software projects over the past 24 months? Very likely yes. What did you see? Maybe projects claim to be agile, because they do a stand-up every morning, because they do not do BDUF any more, because they have a product owner and so on. But many things, which are needed to be agile, let’s say trust for example, which is often ignored, because “that doesn’t work for us” (you could also say, we only want to change the parts, which are easy to change). The ScrumBut gained some attention here: <We use Scrum, but> <we have these unique circumstances> <so we have had to modify Scrum so it works here>. That’s not necessarily a bad thing, but many projects have a too simplistic idea of agile adoption (or adaption).
The hip & trendy v’s the bad & evil
Yes, agile is “hip” these days, agile is “trendy” and the rest is “bad & evil”, so everybody claims to be agile. Clear, who doesn’t want to be agile? You have a project where agile doesn’t fit well? What do you say? We’re not agile? Sounds strange, right?
I suspect, agile software development will suffer the same destiny like the waterfall model: misunderstood and misinterpreted. That’s because many people sit around the campfire, eat marshmallows and talk about how great agility is. Agility, the desired silver-bullet in software engineering. No architecture, no documentation, just hacking… wait a minute. That’s XP, Scrum is different: daily stand-up, product owner, bi-yearly releases, full-planned backlog for the next 2 years, quality-assured requirement documents named user stories… wait a minute, that’s wrong, that’s not agile.
Innovators, Imitators, Idiots
Why do we find these interpretations over and over again? We may find a hint for that in Warren Buffetts comment on the natural progression of how good ideas go wrong. He called this the “three Is.” “First come the innovators, who see opportunities and create genuine value. Then come the imitators, who copy what the innovators have done. Sometimes they improve on the original idea; often they tarnish it. Last come the idiots, whose avarice undermines the innovations they are trying to exploit.” (initially from William Taylor and nicely described for agility by Jim Highsmith).
Should we care? No! As stated above, any complex idea will always be misunderstood in the broad. If you want to change that, you have to change mankind. Good luck with that! We have to accept the wrong interpretations of agility and that agility will be blamed in the future for failed projects. Maybe we’re also interpreting it wrong? Maybe a bit? Let’s work on it.
Which memes are spread?
But a lot of things have changed through the agile movement, even the worst misinterpretation cannot change that. Agile software development questioned successfully the possibility of a complete a priori planning for complex (software) projects. We rather want short feedback-cycles with a lot of communication. Projects with fixed scope, fixed quality and fixed price are also successfully questioned. Better price models are on the way. Added value is more important than following a process description. Trust and respect is not something for crybabies, it is rather an engine for productivity and replaces time-consuming finger-pointing.
We have to go on…
There are a couple of innovations we should put an eye on: DevOps, continuous delivery, the conversations over technical debt, Lean, Kanban, Agile Leadership. The Scrum Alliance has a new strategy and they want to make sure, that new ScrumMasters should know that “Scrum Is not Enough” and that there is a whole world of great, agile ideas out there. I am keen to read their upcoming articles about Lean and XP to get new ideas.
Anyway, this is just a little insight into my take on this matter, interested to discuss this further and gain even more insight then join me at our next tech session in December. It’s free, fun and we even serve Pizza & Beer.
Technical Debt (TD) is the gap between perfectly developing software and the reality (ship date, skills of the engineers, tools available, working environment). You get it, when you take shortcuts that fall short on good practices. As in finance, not all debt is bad. Take a debt to buy a house is mostly a good thing, buy something luxury with your credit card knowing your paycheck will not cover it, is stupid .
In finance, measuring debt is easy (principal of debt, interest). But in software development? There are code measurement tools out there like Sonar, which try to measure TD. They are pretty good in measuring the code quality (over time), test coverage, code duplication, debt hot-spots (classes, packages, sprints, etc). They also come up with a number: your TD is $10,000. But: is this a good measurement? When should we start paying back what?