Software monitoring

How to Debug Production Code?

In real-life applications most of the time debugging will not be an option, developers will need data about what happened, when, and where, debugging an application with many flows and states will take forever until found a bug, also sometimes these bugs fixes contain non-technical part (for example refund for orders that failed to deliver).
Application monitoring is divided into two parts: server-side and client-side.

Server-side monitoring

Mainly it is the most essential monitoring you will need to handle because it contains all critical things like databases, security methods, business logic ... etc. there are different types of monitoring: 

Infrastructure monitoring: Target database health, hardware usage, service/micro-service health ... etc., if something happened to these things it will affect the whole business.

Also, it targets some business concepts like response time and the number of requests/secs to certain APIs, which helps in understanding if the application needs to scale up, or if there is a problem that prevents the users from calling APIs.

These incidents are usually triggered by time (application scale, old hardware that needs to change ...etc.), but in rare cases, the problem root could be code or business changes (code reviews and tests should prevent any change that triggers these events).

High Priority incident:  Most applications have invalid states that should be unreachable (like request pay without sign-in, signing without signup, trying to insert a null value at a non-nullable column ... etc.) if any of these states became reachable for any reason it needs to fix immediately. The root of these incidents is recent code changes or bad design that didn't handle this state.

Event Logging

Event Logging is mainly used for debugging purposes to understand what happened, Logging info is stored differently than the normal data as it is much larger, it will be written once (no update operations), and contains more technical information that later (in many cases) will never be used in business, later on, these data will be moved to storage with high latency (an archive) and some applications models will delete these data.

Auto logging: most projects define a framework for auto logging, it provides the developer with standards info like API x called with parameter y, or API a returns result b, while the developer sometimes needs more specific logs, the auto login has a big advantage as it all has the same format style and it doesn't rely on the developer memory (remember to add logs).

The only thing developer needs to be aware of for logging (logging in general) is to never log sensitive data, for example, imagine an application that logs logging info (password) or payment info, while the logging info can be accessed by anyone in the company or the technical team.

Client-side monitoring

Application Monitoring is mainly used to detect a problem ASAP, keep track of what is happening in real usage and understand how the users interact with the app.

While most data collected on the server side is for technical purpose and target only the developers, the client-side data target many teams (developer, product, data analytics, design ...etc) which means this info needs to be written as clearly as possible, assuming someone who never wrote a single line of code will look to these data is the normal case here.

One thing to keep in mind is that any data that come from the client side isn't fully trusted as anyone can edit the APK and send fake data, so, for example, something like if the user paid for the service or not is something you will always look into server-side monitoring data.

Application instrumentation is the process of adding code to your application so you can understand its inner state. It measures what code is doing when it responds to requests by collecting data such as metrics, events, logs ...etc. An instrumented application tracks as much information as possible about the service’s operations and behavior. It provides more detail about what is happening, so you can see relationships between requests. 

Instrumentation is key to ensuring the application is performing up to its best. It focuses more on the whole picture rather than single info, it targets to produce something like flow, relations, charts ... etc. rather than user x sending request y with parameter {a,b,c}.

Design Principles: Encapsulation

I love to talk about one important concept that is easy to apply but greatly improves code readability and makes the code easy to manipulate in future Encapsulation.

The Encapsulation design Principal idea is simple: don't mix dynamic code with static code.

Let's say an example to understand that: imagine you write a code for an online store that sells 3 types of t-shirts, in this example, something like system order flow will be something that will not change frequently while the number of products, their names, and other things that related to the products is highly likely to change by time, so when you tightly coupled them with other logic like the order flow that shouldn't change easily will make the code so hard to manipulation in future or even to understand.

Many fall into a common misunderstanding here (the fake scaling view): in our example, a developer will think "Oh it's just used in one place and it's only 2 or 3 lines" or "It's only duplicated once", take a look at the business model for a minute and you will realize that 3 can grow to hundreds or thousands in few years then imagine a function that handled all these cases and does other business logic and all these cases are repeated multi times in different places in a single class, it will be a nightmare to any developer to try to manipulate or debug these thousands of lines.

Some may hear about over-killing and think it's alright for now to ignore that and push this concept after the product growth, but unlike something like microservices, the Encapsulation design Principle doesn't add complexity to the code it just told the developer to export dynamic code outside the logic, and a long side with single-responsibility design Principle they told the developer to break the logic into small code blocks as much as possible. 

Ignoring this concept will result in a code with a high chance of bugs in the future for two reasons:

• The next developer who wants to manipulate the code will not understand the difference between static code (that shouldn't be changed without deep thought and proper design discussion) and dynamic code that changes frequently (the solution is to move to DB if possible).
• Dynamic code usually exists in multi-places, in most cases that will result in changing in some places and forgetting the rest, and no one will notice that (except the person who wrote the original code), which results in many times inconstant behaviors.

This Principle is also related to other concepts like the (DRY) Principle "Don't repeat yourself" which mainly told the developer to avoid duplicate code.

One of the old great pieces of advice I heard when I was a student is: "When a developer writes a function that exceeds ~15 lines (this value is related to the language) he/she probably can divide it into 2 or more function" while this can be related more to the single-responsibility design Principal concept however understanding how to put the breakpoints in your code like "these 2 blocks of code can/should be separated" is something that also related to Encapsulation Design Principle.

Biometric Authentication

Let's talk about one of the recent sign-in methods that became popular recently, the Biometric login which includes fingerprint, face id, voice recognition, and other methods, but mainly let's talk about the most secure one of them, the fingerprint method.

Why are we talking about fingerprints, not face IDs or other methods?

Mainly the other methods have weakling that to recognize the real user vs a digital copy, you can have any person's voice from online videos or records, you also can get almost any person's face from internet, the sign-in system can ask the user for specific things like making a random specific reaction or say the displayed sentence, however, someone still can break this using train ML model that talks like any person or display any face reaction using public data.

The fingerprint is the only thing that proves it's the real user, but if we use fingerprint doesn't that mean fingerprint will be soon shared data if leaked?

Actually, how fingerprint works are really interesting as it's too similar to the hashed password concept but device-based instead of server-based, what happens these days is there are no public devices that store fingerprint as raw data, all devices hash the fingerprint using a unique key before storing it so if a device got hacked there is no fingerprint to steal, and the hashed value is useless as it is different based on device and the device itself doesn't know what was the real value.

For the password when the company feels it's leaked, they will ask the user to change it, the fingerprint will not have this option, so fingerprint leaking will be something that can never be recovered for the user.

So how does the online business use a fingerprint without knowing the fingerprint of each user?

What happened is the device contains most of the time a token generator and when the device verifies the fingerprint it will just return a new token to use instead of the hashed value, Biometric auth is just a way to verify the user who is using the device right now, there are also applications for auth devices that only activate using Biometric auth.

This explains why the user needs to log in first on the device to enable Biometric to auth on each new device even if the user already enabled it on other devices, Biometric auth support only one user on the device for now because the applications do not have access to hashed values it just asks device and the device return true or false (or auth token if it's auth device).

image references: [link1]