From 135681daff5c84da4a23bb7500b6e46747bcb4be Mon Sep 17 00:00:00 2001 From: Karan Jayachandra Date: Mon, 9 Dec 2024 18:43:30 +0100 Subject: Fixed the file name for the scientific computing post --- content/nano/python_scientific_computing.md | 39 +++++++++++++++++++++++++++++ content/nano/the_turn.md | 39 ----------------------------- 2 files changed, 39 insertions(+), 39 deletions(-) create mode 100644 content/nano/python_scientific_computing.md delete mode 100644 content/nano/the_turn.md diff --git a/content/nano/python_scientific_computing.md b/content/nano/python_scientific_computing.md new file mode 100644 index 0000000..4b7244a --- /dev/null +++ b/content/nano/python_scientific_computing.md @@ -0,0 +1,39 @@ ++++ +title = 'Python for Scientific Computing' +date = "2024-12-09" ++++ + +# Python for Scientific Computing +I am a signal processing engineer and most of my work involves the creation of algorithms that make sense of data. I work with the data that is usually generated from an automotive radar but this necessarily doesn't have to be the case. For me, the data could be from any source but as long as there is information in the data, I can find a way to extract it. The complication that arises with my line of work comes from the fact that the data is usually [noisy](https://en.wikipedia.org/wiki/White_noise). Therefore, I end of doing quite a bit of statistical simulations to verify my work. + +Most engineers and scientists in my field work with MATLAB®. I however prefer to use Python. So if you are someone doing something similar to what I have just described, you might find this post useful. The following are some of the learnings from 2-3 years of consistent use of Python code scientific computing. This is of course a subjective account but I am free to discuss any of the points below via [email](mailto:mail@karanjayachandra.com). + +|![Filtering Noise](https://cdn.prod.website-files.com/5babb9f91ab233ff5f53ce10/608a88ee9c90126452af58d8_SMA-filtering.jpeg)| +|:--:| +| *Filtering Noise (Source: [Catchpoint](https://www.catchpoint.com/blog/signal-vs-noise))* | + +## The don'ts +I will start with a few don't that I believe are counter productive if you want to be good at scientific computing. + +### Don't reinvent the wheel +If you are a software engineer, you can ignore this. But if you are trying to do signal processing, do not reinvent the wheel. Do not write your own `fourier_transform()` function. If you are learning to optimize the fourier transform or trying to understand it, by all means, go ahead. But most of the time, stealing someone else's implementation will be much more productive. Most importantly, your implementation will be slower. This will impact your simulation time. Learn to use [numpy](https://numpy.org/doc/2.1/index.html) and [scipy](https://scipy.org). + +### Don't use the latest version of Python +Python is work in progress. Several versions of python exists and I have found that the sweet spot for the version to use is one lower than the latest. Most packages on PyPI will support it and you will not run into issues where a dependency of a dependency of your package is not supported by the latest version. Python is notoriously difficult at handling this. + +### Don't over engineer your code +I have noticed that most of the code that actually helps make scientific conclusions can be achieved via very little. Keep your code and data simple. You don't need inheritance, you don't need that shiny new data structure. Keep to the basics and things will be much more easy to manage in the long run. It is not about how many lines of code but how few lines of code. Reduce the number of moving parts. A side note: If you are working with a team on a large project. Work on a smaller repository for your idea with a simplified test bench to prove your idea before shipping it to the larger system. + +## The do's + +### Use version control +Even if you are the only person working on a project. Use git. This is crucial. If you want to work with other, it is doubly more important. git is not restricted to software projects but any text files that require frequent revision. Even this blog is under version control. Do not have 10 versions of your code floating around. Work on two branches, `main` and `develop` and keep them regularly in sync. + +### Use a virtual environment +As mentioned earlier, python is known to be difficult to handle with respect to dependencies. Dependencies in python are both a boon and a curse. I find PyPI to be a democracy of sorts with the best package rising to the top for use. This allows for programmers to help each other out and reuse things. However, the version of package that you use might not be the same as the one installed on your colleagues machine. Therefore, for every git repository you have on your machine maintain a [venv](https://docs.python.org/3/library/venv.html). Maintain a `requirements.txt` file in the root folder so that others can easily install the dependencies needed for your code. + +### Automate the testing +Most of the time, simulations require the generation of synthetic data or reading stored data and running it through your code. In my experience this is the most time consuming part of scientific computing. Learn to automate this and it will improve your workflow tremendously. Bonus points if you can make this work with git on pushes to the git server via [Jenkins](https://www.jenkins.io) some other automated testing platform. + +### Document the code +At least, maintain a README.md for your repository. Explain briefly what you code does and how to install dependencies and run it. Start with this and when the experiment starts showing results, start maintain a more long form report containing the ins and outs of the code and the mathematics behind it. I prefer [LaTeX](https://www.latex-project.org) for its superior typesetting and ubiquity in academia. It also exports the same report as a HTML document which you can server as a static page along with your code. \ No newline at end of file diff --git a/content/nano/the_turn.md b/content/nano/the_turn.md deleted file mode 100644 index 4b7244a..0000000 --- a/content/nano/the_turn.md +++ /dev/null @@ -1,39 +0,0 @@ -+++ -title = 'Python for Scientific Computing' -date = "2024-12-09" -+++ - -# Python for Scientific Computing -I am a signal processing engineer and most of my work involves the creation of algorithms that make sense of data. I work with the data that is usually generated from an automotive radar but this necessarily doesn't have to be the case. For me, the data could be from any source but as long as there is information in the data, I can find a way to extract it. The complication that arises with my line of work comes from the fact that the data is usually [noisy](https://en.wikipedia.org/wiki/White_noise). Therefore, I end of doing quite a bit of statistical simulations to verify my work. - -Most engineers and scientists in my field work with MATLAB®. I however prefer to use Python. So if you are someone doing something similar to what I have just described, you might find this post useful. The following are some of the learnings from 2-3 years of consistent use of Python code scientific computing. This is of course a subjective account but I am free to discuss any of the points below via [email](mailto:mail@karanjayachandra.com). - -|![Filtering Noise](https://cdn.prod.website-files.com/5babb9f91ab233ff5f53ce10/608a88ee9c90126452af58d8_SMA-filtering.jpeg)| -|:--:| -| *Filtering Noise (Source: [Catchpoint](https://www.catchpoint.com/blog/signal-vs-noise))* | - -## The don'ts -I will start with a few don't that I believe are counter productive if you want to be good at scientific computing. - -### Don't reinvent the wheel -If you are a software engineer, you can ignore this. But if you are trying to do signal processing, do not reinvent the wheel. Do not write your own `fourier_transform()` function. If you are learning to optimize the fourier transform or trying to understand it, by all means, go ahead. But most of the time, stealing someone else's implementation will be much more productive. Most importantly, your implementation will be slower. This will impact your simulation time. Learn to use [numpy](https://numpy.org/doc/2.1/index.html) and [scipy](https://scipy.org). - -### Don't use the latest version of Python -Python is work in progress. Several versions of python exists and I have found that the sweet spot for the version to use is one lower than the latest. Most packages on PyPI will support it and you will not run into issues where a dependency of a dependency of your package is not supported by the latest version. Python is notoriously difficult at handling this. - -### Don't over engineer your code -I have noticed that most of the code that actually helps make scientific conclusions can be achieved via very little. Keep your code and data simple. You don't need inheritance, you don't need that shiny new data structure. Keep to the basics and things will be much more easy to manage in the long run. It is not about how many lines of code but how few lines of code. Reduce the number of moving parts. A side note: If you are working with a team on a large project. Work on a smaller repository for your idea with a simplified test bench to prove your idea before shipping it to the larger system. - -## The do's - -### Use version control -Even if you are the only person working on a project. Use git. This is crucial. If you want to work with other, it is doubly more important. git is not restricted to software projects but any text files that require frequent revision. Even this blog is under version control. Do not have 10 versions of your code floating around. Work on two branches, `main` and `develop` and keep them regularly in sync. - -### Use a virtual environment -As mentioned earlier, python is known to be difficult to handle with respect to dependencies. Dependencies in python are both a boon and a curse. I find PyPI to be a democracy of sorts with the best package rising to the top for use. This allows for programmers to help each other out and reuse things. However, the version of package that you use might not be the same as the one installed on your colleagues machine. Therefore, for every git repository you have on your machine maintain a [venv](https://docs.python.org/3/library/venv.html). Maintain a `requirements.txt` file in the root folder so that others can easily install the dependencies needed for your code. - -### Automate the testing -Most of the time, simulations require the generation of synthetic data or reading stored data and running it through your code. In my experience this is the most time consuming part of scientific computing. Learn to automate this and it will improve your workflow tremendously. Bonus points if you can make this work with git on pushes to the git server via [Jenkins](https://www.jenkins.io) some other automated testing platform. - -### Document the code -At least, maintain a README.md for your repository. Explain briefly what you code does and how to install dependencies and run it. Start with this and when the experiment starts showing results, start maintain a more long form report containing the ins and outs of the code and the mathematics behind it. I prefer [LaTeX](https://www.latex-project.org) for its superior typesetting and ubiquity in academia. It also exports the same report as a HTML document which you can server as a static page along with your code. \ No newline at end of file -- cgit v1.3.1 From 340147717ab589c1cbe6acc4ae72087f733ed4b6 Mon Sep 17 00:00:00 2001 From: Karan Jayachandra Date: Wed, 25 Dec 2024 17:44:48 +0100 Subject: New post about python packaging of flask application --- content/nano/package_flask.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 content/nano/package_flask.md diff --git a/content/nano/package_flask.md b/content/nano/package_flask.md new file mode 100644 index 0000000..12c2d06 --- /dev/null +++ b/content/nano/package_flask.md @@ -0,0 +1,35 @@ ++++ +title = 'Packaging Your Flask Application' +date = "2024-12-25" ++++ + +# Packaging Your Flask Application + +A while ago, I was working on a hobby project for my Badminton Club, [The Smashing Fellows](https://www.bctsf.nl). The club meets twice a week, Tuesday and Friday. On Tuesday, we play using a system called the "Hussel" system which just means that the players are semi-randomly shuffled to create doubles matches amongst each other. The randomization allows for people to get to know each other while keeping also keeping things a bit competitive in terms of playing level. An archaic application written in Microsoft Access being used for this at the club started unravelling at the seams a few months ago. I volunteered to make a new application. The club didn't want the application to be able to run on Windows. I went down the Windows application development rabbit hole and ended deciding to not have anything to do with that. Since, I was already familiar with the use of Python, I ended up deciding to create a web application using Flask. The unique nature of this request didn't allow for deployment online with user information. Therefore, I ended up packaging the application into a wheel for easier deployment to run as a web server on the local machine. Usually, flask applications don't need to be packaged. They can be deployed as is. So, for people who might be in a similar situation, I am documenting the steps that I followed. I won't go into the details of how to create a package your python code or how to create a flask application. There are several tutorials for that. Here is how you combine. + +## Step 1: Add your static files into the package + +Add the static files that you plan to use into your package via the `pyproject.toml` file. This is usually done via the following lines: + +```bash +[tool.setuptools.package-data] +"templates" = ["*.j2"] +"static" = ["*.css", "*.js"] +``` + +Here, I show an example which uses Jinja2 templates and some CSS and javascript files. This allows for the files you need for your application to run be accessible within the package and also adds it into wheel. + +## Step 2: Reference your files in the code + +Usually flask searches for the `static` and `templates` folder in the root directory from where you run the `app`. But since we have now moved this into the package directory, we would need to let flask where to find this. For this, we can provide the location of these folder when instantiating the `Flask` class with the arguments `static_folder` and `templates_folder` as shown below: + +```python +from flask import Flask +from importlib.resources import path + +app = Flask("", + static_folder=path("", "static"), + template_folder=path("", "templates")) +``` + +Make sure to use the `importlib` package to locate where your package is installed. This allows for the path to be dynamically updated based on where the user installs your package. Now when you run your flask application you should be able to see the static files you need. In case you are interested in seeing the full code base, you can find it [here](https://gitlab.com/KaranJayachandra/match_up). \ No newline at end of file -- cgit v1.3.1