Every single line of code ever written was ultimately made with one purpose in mind: to solve problems. No matter what you do, you are solving problems on several scales at once.
A small one-liner solves a problem which makes a function work. The function is needed for a data processing pipeline. The pipeline is integrated to a platform which enables a machine learning driven solution for its users.
Problems are everywhere. Their magnitude and impact might be different, but the general problem solving strategies are the same.
As an engineer, developer, or data scientist, being effective in problem-solving can really supercharge your results and put you before your peers. Some can do this instinctively after years of practice, some have to put conscious effort to learn it. However, no matter who you are, you can and you must improve your problem-solving skills.
Having a background in research-level mathematics, I had the opportunity to practice problem solving and observe the process. Surprisingly, this is not something which you have to improvise each time. Rather,
a successful problem solver has several standard tools and a general plan under their belt, adapting as they go.
In this post, my aim is to give an overview of these tools and use them to create a process, which can be followed at any time. To make the situation realistic, let’s place ourselves into the following scenario: we are deep learning engineers, working on an object detection model. Our data is limited, so we need to provide a solution for image augmentation.
Augmentation is the process of generating new data from the available images by applying random transformations like crops, blurs, brightness changes, etc. See the image above which is from the readme of the awesome albumentations library.
You need to deliver the feature by next week, so you need to get working on it right away. How to approach the problem?
(As a mathematician myself, my thinking process is heavily influenced by the book How to Solve It by George Pólya. Although a mathematical problem is different from real life coding problems, this is a must read for anyone who wishes to get good in problem solving.)
Step 0: Understanding the problem
Before attempting to solve whatever problem you have in mind, there are some questions that need to be answered. Not understanding some details properly can lead to wasted time. You definitely don’t want to do that. For instance, it is good to be clear about the following.
- What is the scale of the problem? In our image augmentation example, will you need to process thousands of images per second in production, or is it just for you to experiment with some methods? If a production grade solution is needed, you should be aware of this in advance.
- Will other people use your solution? If people are going to work with your code extensively, significant effort must be put into code quality and documentation. On the other end of the spectrum, if this is for your use only, there is no need to work as much on this. (I already see people disagreeing with me 🙂 However, I firmly believe in minimizing the amount of work. So, if you only need to quickly try out an idea and experiment, feel free to not consider code quality.)
- Do you need a general or a special solution? A lot of time can be wasted on implementing features no one will ever use, including you. In our example, do you need a wide range of image augmentation methods or just vertical and horizontal flips? In the latter case, flipping the images in advance and adding them to your training set can also work, which requires minimal work.
A good gauge of your degree of understanding is your ability to explain and discuss the problem with others. Discussion is also a great way to discover unexpected approaches and edge cases.
When you understand your constraints and have a somewhat precise problem specification, it is time to get to work.
Step 1. Is there an existing solution?
The first thing you must always do is to look for existing solutions. Unless you are pushing the very boundaries of human knowledge, someone else had already encountered this issue, created a thread on Stack Overflow, and possibly wrote an open source library around it.
Take advantage of this. There are several benefits of using well-established tools, instead of creating your own ones.
- You save a tremendous amount of time and work. This is essential when operating under tight deadlines. (One of my teachers used to say ironically that “you can save an hour of Google search with two months of work”. Spot on.)
- Established tools are more likely to be correct. Open source tools are constantly validated and checked by the community. Thus, they are less likely to contain bugs. (Of course, this is not a guarantee.)
- Less code for you to maintain. Again, we should always strive for reducing complexity, and preferably the amount of code. If you use an external tool, you don’t have to worry about its maintenance, which is a great deal. Every line of code has a hidden cost of maintenance, to be paid later. (Often when it is the most inconvenient.)
Junior developers and data scientists often overlook these and prefer to always write everything from scratch. (I certainly did, but quickly learned to know better.) The most extreme case I have seen was a developer, who wrote his own deep learning framework. You should never do that unless you are a deep learning researcher and you have an idea of how to do significantly better than the existing frameworks.
Of course, not all problems require an entire framework, maybe you are just looking for a one-liner. Looking for existing solutions can be certainly beneficial, though you need to be careful in this case. Finding and using code snippets from Stack Overflow is only fine if you take the time to understand how and why it works. Not doing so may result in unpleasant debugging sessions later, or even serious security vulnerabilities in the worst case.
For these smaller problems, looking for existing solution consists of browsing tutorials and best practices. In general, there is a balance between the ruthless pragmatism and the outside of the box thinking. When you implement something in a way that is usually done, you are doing a favor for the developers who are going to use and maintain that piece of code. (Often including you.)
There is an existing solution. What next?
Suppose that on your path towards delivering image augmentation for your data preprocessing pipeline, you have followed my advice, looked for existing solutions, and found the awesome albumentations library. Great! What next?
As always, there is a wide range of things to consider. Unfortunately, just because you have identified an external tool which can be a potential solution, it doesn’t mean that it will be suitable for your purposes.
- Is it working well and supported properly? There is one thing worse than not using external code: using buggy and unmaintained external code. If a project is not well documented and not maintained, you should avoid it.
For smaller problems, where answers generally can be found on Stack Overflow, the working well part is essential. (See the post I have linked above.)
- Is it adaptable directly? For example, if you use an image processing library that is not compatible with albumentations, then you have to do additional work. Sometimes, this can be too much and you have to look for another solution.
- Does it perform adequately? If you need to process thousands of images per second, performance is a factor. A library might be totally convenient to use, but if it fails to perform, it has to go. This might not be a problem for all cases (for instance, if you are just looking for a quick solution to do experiments), but if it is, it should be discovered early, before putting much work to it.
- Do you understand how it works and what are its underlying assumptions? This is especially true for using Stack Overflow code snippets, for the reasons I have mentioned above. For more complex issues like the image augmentation problem, you don’t need to understand every piece of external code line by line. However, you need to be aware of the requirements of the library, for instance, the format of the input images.
This, of course, is applicable only if you can actually find an external solution. Read on to see what to do when this is not the case.
What if there are no existing solutions?
Sometimes you have to develop a solution on your own. The smaller the problem is, the more frequently it happens. These are great opportunities for learning and building. In fact, this is the actual problem solving part, the one which makes many of us most excited.
There are several strategies to employ, all of them should be in your toolkit. If you read carefully, you’ll notice that there is a common pattern.
- Can you simplify? Sometimes, it is enough to solve only a special case. For instance, if you know for a fact that the inputs for your image augmentation pipeline will always have the same format, there is no need to spend time on processing the input for several cases.
- Isolate the components of the problem. Solving one problem can be difficult, let alone two at the same time. You should always make things easy for yourself. When I was younger, I used to think that solving hard problems is the thing to do in order to get dev points. Soon, I have realized that the people who solve hard problems always do it by solving many small ones.
- Can you solve for special cases? Before you go and implement an abstract interface for image augmentation, you should work on a single method to add into your pipeline. Once you discover the finer details and map out the exact requirements, a more general solution can be devised.
In essence, problem solving is an iterative process, where you pick the problem apart step by step, eventually reducing it to easily solvable pieces.
Step 2. Break the solution (Optional)
There is a common trait which I have noticed in many excellent mathematicians and developers: they enjoy picking apart a solution, analyzing what makes them work. This is how you learn, and how you build robust yet simple code.
Breaking things can be part of the problem solving process. Going from a special case to general, you usually discover solutions by breaking what you have.
When it is done
Depending on the magnitude of the problem itself, you should consider open-sourcing it, if you are allowed. Solving problems for other developers is a great way to contribute to the community.
For instance, this is how I have built modAL, one of the most popular active learning libraries for Python. I started with a very specific problem: building active learning pipelines for bioinformatics. Since building complex methods always require experimentation, I needed a tool which enabled rapid experimentation. This was difficult to achieve with the available frameworks at the time, so I slowly transformed my code into a tool that can be easily adopted by others.
What used to be “just” a solution became a library, with thousands of users.
Contrary to popular belief, effective problem solving is not the same as coming up with brilliant ideas all the time. Rather, it is a thinking process with some well-defined and easy to use tools, which can be learned by anyone. Smart developers use these instinctively, making them look like magic.
Problem-solving skills can be improved with deliberate practice and awareness of thinking habits. There are several platforms where you can find problems to work on, like Project Euler or HackerRank. However, even if you start applying these methods to issues you encounter during your work, you’ll see your skills improve rapidly.